Exploration of issues, challenges and latest developments in autonomous cars

Abstract

Autonomous cars have achieved exceptional growth in the automotive industry in the last century in terms of reliability, safety and affordability. Due to significant advancements in computing, communication and other technologies, today we are in the era of autonomous cars. A number of prototype models of autonomous cars have been tested covering several miles of test drives. Many prominent car manufacturers have started investing huge resources in this technology to make it commercialize in the near future years. But to achieve this goal still there are a number of technical and non-technical challenges that exist in terms of real-time implementation, consumer satisfaction, security and privacy concerns, policies and regulations. In summary, this survey paper presents a comprehensive and up-to-date overview of the latest developments in the field of autonomous cars, including cutting-edge technologies, innovative applications, and testing. It addresses the key obstacles and challenges hindering the progress of autonomous car development, making it a valuable resource for anyone interested in understanding the current state of the art and future potential of autonomous cars.

Introduction

Autonomous cars are becoming more pragmatic from year to year as multi-national companies are racing ahead to produce intelligent vehicles. The projected value for autonomous vehicles in the global market is at $615 Billion by 2026. According to the U.S. Department of Transportation's National Highway Traffic Safety Administration (NHTSA) fully automated or autonomous vehicles are "those in which operation of the vehicle occurs without direct driver input to control the steering, acceleration, and braking and are designed so that the driver is not expected to constantly monitor the roadway while operating in self-driving mode.". The advancement and rise of autonomous cars are due to the significant research results obtained in the arenas of wireless and embedded systems, sensors, communication technologies, navigation, data acquisition, and analysis.

The initial thought of autonomous cars was initiated in the year 1920 with the "phantom autos" concept, which means a remote-control device, which was used to control the vehicle [1]. Later in the 1980s self-managed autonomous cars were developed. Further, NavLab of Carnegie Mellon University contributed majorly in this field by developing an Autonomous Land Vehicle (ALV) [2]. In a major breakthrough in 1987, the “Prometheus project” of Mercedes [3] gave the design of their first automated car with the capability of tracking lanes. At that time, it was not completely autonomous, but it had the ability to automatically switch lanes. In the twenty-first century, there is a huge demand for low-cost, high-performance autonomous cars. There is a fine line of difference between the two terminologies: the automated car and the autonomous car. The term automated car refers to a vehicle with little human intervention, whereas the term autonomous car refers to a vehicle without any human intervention. The autonomous car is a fully computer-controlled car which can instruct (guide) itself, make its own decisions, familiar with its surrounding without any human interference (intervention).

The concept of connected car technology is influenced by autonomous cars [4] as both technologies are related to each other. Layered architectures are being proposed to address challenges faced due to the internet response time and the compatibility of various components that are being used in connected car technology [5]. Autonomous vehicles need to be connected to each other to improve overall autonomy when driving on the road. For example, the connected car works on a vehicular ad hoc network (VANET) technology and a dedicated short-range communication (DSRC) standard protocol [6] using which communication between vehicles is possible when they are in range. The VANET [7] provides 2 types of applications; one regarding safety and the other one related to infotainments. In Autonomous cars, communication related security measures are stringent whereas in connected cars, security measures are moderately relaxed. In the latest developments regarding the connected cars and VANET technologies, many multinational technology companies such as Google along with car manufacturers such as Tesla and Audi are working together and we see a solid collaboration among the technology companies and vehicle manufacturers to facilitate the development and design of cars.

Similarly, Microsoft has begun a coalition with Volvo and Toyota for building autonomous cars. Also, companies like NVidia have shown their dedication to making autonomous cars by launching NVidia Drive PX2, a dynamic supercomputer GPU and a deep learning-based computing platform for autonomous cars [8]. A few more Asian companies such as TATA, KIA, and Hyundai are funding in design, development, and research regarding automated cars. Also, the European auto market (Mercedes and BMW) is in the race for autonomous cars, and their goal is the development of full-fledged commercial versions by 2020. Many companies such as Volkswagen and the French PSA group are focusing on developing autonomous cars and they have started test-drive since 2016. The PSA group brought together many car manufacturers together and drove their autonomous cars covering hundreds of kilometers from Paris to Amsterdam without any driver supervision in 2016 on a level-3 autonomy [9].

In the development of an autonomous system, there are several issues that must be addressed properly by the car manufacturing companies such as governments’ regulations, consumer satisfaction, cost, reliability, and safety. Further, an important role is played by federal regulations in achieving novel technologies, and autonomous cars are no exemption from that. The automatic transmission system plays a vital role in autonomous cars and these days as most automobiles use this technology due to its reduction in cost and improvement in quality and management of a fleet of electric vehicles [10]. In brief, this technology will consume some more time period till it is affordable, and reachable to customers in terms of cost and reliability. This paper highlights all the issues and challenges involved in the development of autonomous cars and conducts a survey on the latest autonomous car technologies that are trying to overcome these issues and challenges in an efficient way.

Related work

Till today, a number of works have been done to explore multiple issues of autonomous car system [11,12,13,14,15,16,17,18,19,20,21]. But the majority of surveys focus only on certain aspects of the autonomous car and none of these surveys present a comprehensive (holistic) method towards autonomous car technology.

Campbell et al. [11] discussed the approaches to challenges faced in urban environments by autonomous vehicles. Okuda et al. [12] did a thorough survey on the usage of advanced driving assistance (ADAS) in autonomous cars and the trends in the technology. Fagnant et al. [13] helped in surveying the required policies to make a nation ready for autonomous vehicles. Moreover, Bagloee et al. [14] focused on a few challenges that such different policies provide to autonomous cars. Other surveys regarding its functionalities include planning and motion control [15], long-term maps’ constructions [18] and visual perception from both implementation and operators’ perspectives [20, 21]. Furthermore, Abraham et al. [16] conducted a survey on consumer trust and also the preferences of the consumer on already available alternatives. Gupta et al. [22] aimed to understand the public's attitude towards autonomous vehicles by analyzing large amounts of data from Twitter without the need for manual labeling. The authors used advanced machine learning techniques to analyze the data and identify patterns in the public's perception of autonomous vehicles. The study found that the majority of tweets about autonomous vehicles were positive in nature, but there were also concerns about safety and privacy. Joy et al. [17] looked over security and privacy issues in autonomous cars. Madhav et al. [23] presented a study on how to improve human trust in autonomous vehicles through the use of Explainable Artificial Intelligence (XAI). The authors focused on bridging the gap between artificial decision-making and human trust by providing an explanation for the decisions taken by autonomous vehicles. Bairy et al. [24] presented a model for explaining the decision-making process of autonomous vehicles. The model is based on integrated formal methods, which are mathematical methods used to model and verify systems. The authors claim that providing explanations for the actions of autonomous vehicles is important for building trust in the technology of autonomous vehicles and for ensuring accountability in the event of an accident.

Parkinson et al. [19] have expansively analyzed cyber threats in autonomous cars and the challenges they pose on the future of connected vehicles. Mazri et al. [25] proposed a self-defense mechanism against security attacks for autonomous vehicles. The mechanism is designed to protect against various types of cyberattacks, including those targeting the communication systems and decision-making processes of the vehicles. The authors presented a detailed analysis of the potential vulnerabilities and risks of autonomous vehicles, and provide a comprehensive evaluation of the proposed mechanism using various simulation scenarios. Li et al. [26] investigated the preferences of drivers when it comes to performing secondary tasks and how the vehicle's level of autonomy affects the driver's task engagement. The study also explored the potential impact of secondary tasks on safety and comfort while driving. The findings of the study can be used to design interfaces and interactions that better support drivers’ needs and preferences in highly autonomous vehicles.

The use of blockchain technology for training autonomous cars has been proposed by G. M. Gandhi et al. [27]. In this process, all connected cars can share their experience with each other. Blockchain can also be used to maintain energy transactions at charging stations [28]. Many recent surveys on autonomous cars have majorly focused on a few topics of autonomous cars. Table 1 presents a summary of related works. Figure 1 depicts the prediction of Autonomous vehicle implementation by 2060.

Table 1 Summary of related works

Fig. 1

Predicted Development of Autonomous Vehicles from 2010 to 2060. This figure illustrates the expected advancement of autonomous vehicle technology over the course of the next several decades. The x-axis represents the years, starting in 2010 and ending in 2060. The y-axis represents the level of autonomy, with 0% being no automation and 100% being fully autonomous. The line represents the projected trajectory of the technology, starting at 0% in 2010 and gradually increasing to 100% by 2060. The graph illustrates the rapid pace of technological advancement in the field of autonomous vehicles and the significant impact it is expected to have on transportation in the coming years

This ground-breaking survey delves deep into the latest developments in the exciting field of autonomous cars. From cutting-edge technologies and innovative applications to in-depth simulations, this paper provides a comprehensive overview of the current state of the art. Additionally, it addresses the key obstacles and challenges that are hindering the progress of autonomous car development, including both technical and non-technical issues.

Organization

Organized in a clear and logical manner, this survey begins by summarizing existing works in the field and outlining the different levels of autonomy according to the standards set by the Society of Automotive Engineers (SAE).

The heart of the paper is dedicated to exploring the underlying technology behind autonomous cars, including detailed descriptions of their design, components, and functionalities. The benefits of autonomous cars are also discussed, along with a survey of state-of-the-art research outcomes.

The survey goes on to explore the implementation and design challenges that must be overcome in order to bring autonomous cars to reality. Finally, the paper concludes with a look at the technology and challenges for autonomous car deployment, making it a must-read for anyone interested in this rapidly evolving field.

Contribution

This survey provides a thorough and up-to-date overview of the latest developments in the field of autonomous cars, including cutting-edge technologies, innovative applications, and in-depth simulations. Additionally, it addresses the key obstacles and challenges that are hindering the progress of autonomous car development, and provides an in-depth look at the technology, benefits, and challenges behind autonomous car deployment. It is a valuable resource for anyone interested in understanding the current state of the art and future potential of autonomous cars.

The autonomous car technology

The autonomous car has been in the spotlight during the last few decades and prototype models have been improved by various car manufacturers. But, the commercial actualization of autonomous cars is a major challenge. As a basic task, gearing of every autonomous car is performed with numerous actuators and sensors that produce vast amounts of data which must be handled and analyzed to take decisions on time. The amount of data that the car must handle depends on the levels of autonomy [38] as shown in Table 2. The key point is the requirement of various sensors and actuators to function autonomously, so that the car should foresee, decide and maneuver cautiously according to some strategy. A number of fundamental characteristics of autonomous car design are outlined in this section. Figure 2 depicts a number of elevated functional components of a typical autonomous car.

Table 2 This table provides a classification of autonomous vehicles based on the levels of autonomy as per the standard set by the Society of Automotive Engineers (SAE)

Fig. 2

This image displays the functional architecture of an autonomous car. The architecture is divided into several functional blocks, each of which is responsible for a specific set of tasks. The main functional blocks of the architecture are Data Acquisition, Data Processing and Actuation – The components that carry out the commands from the control system like acceleration, braking and steering

The functional architecture of an autonomous car is a layered structure which contains data acquisition, data processing and actuation. Data acquisition is performed by the hardware components, such as sensors, radars, cameras, LIDAR, and transceivers [40]. The collected data is processed by a central computer system, which is later implemented by the decision-support system (DSS). The DSS activates the autonomous car and the situational awareness is actualized through both short- and long-range imaging devices [41]. Figure 3 shows the system architecture of the autonomous car, where it shows the areas covered by different components in the car design. Different ranges of situational awareness vary from application to application and it is achieved through multiple components. For instance, prevention of front and rear bumper collisions is done through infrared devices, whereas short-range radars are used for object detection, lane-change cautioning, and traffic view. Equipping autonomous cars with a series of cameras for the surrounding views and LIDAR helps in collision avoidance and emergency brakes. Long-range radars help in cooperative cruise control and long-range traffic view construction. Altogether the aforementioned components are networked and work firmly with each other, as shown in Fig. 2.

Fig. 3

This figure displays the basic architecture of an autonomous car. The architecture consists of various components that work together to enable the car to drive itself. The main components of the architecture are Sensors, Perception and Localization, Planning and Control, Communication and Cloud

For movement of an autonomous car from one point A to B, the car performs a number of important steps in an iterative manner until it reaches its destination such as Perceive and aware about its surrounding environment, Path planning and navigation and Controlled movements on the road [42].

After perceiving its surrounding environment, it makes path planning along with its destination information and then starts navigating to reach the destination. A number of controlled movements are exercised for a smooth, safe ride on the road with the help of actuators and sensors [43]. Electronic Control Units (ECUs) control most of the components electronically. ECUs communicate with each other and with the decision support system through the controller area network (CAN) bus inside each car. During a drive, the autonomous car exhibits a number of manoeuvres which needs both software/hardware support, extensive coordination, and data sharing among different components of the car.

Autonomous car benefits

In spite of several complexities and difficulties, the autonomous car provides security, user-friendliness, comfort and value-added services to its customers.

Safety

Safety is one of the major concerns and is given top priority in the automotive domain. Every year many millions of people lost their lives in road accidents according to the National Highway Traffic Safety Administration (NHTSA). More than 90% of accidents [44] happen due to human errors and these errors are caused by various factors such as carelessness, aggressiveness, intoxication and distraction. So, it’s essential to have an alternative and safer mechanism like autonomous driving cars to eliminate human errors and save lives.

Daniel Zelle et al. [32] developed a security evaluation platform for autonomous driving (SPEAD) to enable researchers to develop, implement and evaluate new security solutions for autonomous vehicles. SPEAD allows us to model realistic autonomous vehicle architectures and test their security mechanisms. There is another aspect of safety that is how to protect the car from thieves. Due to high-end on-board sensors, recognition of the car owner is done by the car itself and it sends the owner an alert in case of any unwanted situation. Unlike ordinary cars, autonomous cars do not require keys, but they operate with finger prints, retina scan and voice recognition software. It is also equipped with finger-print enabled door-lock system to provide more security.

Increase in revenue and business opportunities

Mobility-as-a-Service (MaaS) benefits the customers by saving multiple resources including money, time, and space. Car sharing and car-pooling are two popular applications among customers today. With the advent of autonomous cars, these applications will become more effective by making use of car resources more efficiently. These applications create economic benefits, and also reduce air pollution produced by vehicles in urban areas. This also generates huge business opportunities for the customers. In future, autonomous cars will transform taxi and car rental business. There is no longer requirement of drivers in taxis and rental cars, therefore it will reduce the cost and increase the revenue of owners. Further, it will impact the software industry to develop many smart applications for cars. Hence, autonomous cars will help in increase in revenue by reducing maintenance and labour costs.

User-friendliness and convenience

User-friendliness and convenience are other additional benefits of autonomous cars. In some situations, physically disabled or intoxicated persons can’t drive a car manually. Similarly, for aged, young people without licence, autonomous cars will be a suitable mode of transportation. In such situations, the autonomous car provides a safe and economical way of transportation.

Improving traffic conditions

In autonomous cars, car sharing and car-pooling can be a major advantage, hence it will increment per-vehicle occupancy and decrement the number of vehicles on the road, thereby improving the traffic conditions. Inter-vehicle distance is strictly followed in autonomous cars to improve passenger safety; hence in turn it reduces road traffic. These cars follow traffic rules more precisely, thereby condensing the requirement for more traffic personnel on the road. Fuel efficiency is also improved by automatically choosing the best and shortest path from source to destination [45]. Hence there is also a decrease in air pollution.

Bo Liu et al. [29] proposed a redundancy concept for autonomous vehicle functions using microservice architecture in which vehicle-vehicle and cloud-vehicle communication ensure that important vehicle functions keep executing even in the case of a hardware failure. These cars also make use of a fuel-efficient mode through programming by eliminating unnecessary braking situations on the road. One such example is regenerative-braking [46] in electric autonomous vehicles which uses the kinetic energy of the car to convert it into electrical energy (fuel) until the car stops naturally. Qiao et al. [36] proposed a hybrid model for traffic assignment and control that combines the strengths of both traditional traffic assignment models and control methods for autonomous vehicles. The model uses a combination of traffic simulation, machine learning, and optimization techniques to assign traffic and control the movement of autonomous vehicles in a given road network. The authors evaluated the performance of the proposed model using real-world traffic data and show that it can improve the overall efficiency and safety of traffic flow in comparison to traditional methods.

Autonomous parking

In metropolitan cities, there are a number of issues related to vehicle parking such as getting parking space in peak hours, increased population and maintaining inter-vehicle distance in parking slots. The advantage of autonomous cars is they can park themselves even in a narrow available parking slot, which is very difficult for a driver to park a car manually. Marvy Badr Monir Mansour et al. [47] implemented a project to demonstrate autonomous parallel car parking that can be used efficiently in metropolitan cities. Baramee Thunyapoo et al. [48] proposed a simulation framework for autonomous-car parking for moderate complexity scenarios. Shiva Raj Pokhrel et al. [49] developed (and evaluated) an experience-driven, secure and privacy-aware framework of parking reservations for automated cars. Since autonomous-cars can communicate with each other, they can reduce traffic congestion by coordinating their movement on the road.

Consumer-centric experience

In autonomous cars passengers and drivers can sit, relax and enjoy the ride. They can also simultaneously do their personal work or utilize the car’s entertainment system [50]. When the autonomous car is paired with a smart phone then a passenger can command the car to perform some important tasks automatically like picking up children from school, picking up or dropping someone at the airport. These kinds of sophisticated features called “Summon” of autonomous cars are introduced by Tesla Company in its high-end models [51]. Using the “Summon” application, the car owner can instruct the car to go to the designated parking place even to the basement of the building.

Further research is going on to understand various driving patterns by analysing the behaviour of the drivers by considering drivers’ age, gender, driving experience, personality, emotion, and history of accidents [52]. A number of features of autonomous cars can be customized based on the human behaviour. For example, overtaking and speed while driving depends on gender, age, emotion etc. Young people drive faster than elders, whereas female drivers drive more cautiously. Similarly, people with families and kids drive more carefully than others. So, during autonomous car customization, one should take these factors into consideration. In future, autonomous cars will lead to attract the developers to build a number of customer-driven applications, where customers can personalize their travel experience according to their choice of speed, in-car entertainment, and level of risk Travelling in autonomous vehicles also increases productivity as the user can focus on work instead of driving the vehicle during transit.

Technology behind autonomous cars

In this section, various technological researches conducted so far in the area of autonomous cars are presented. This section also covers the software components and algorithms used in this technology. Broggi et al. [53] discussed regarding a number of tests performed on driverless cars in different scenarios such as roads with less to heavy traffic during 1990–2013. The outcomes witnessed changes in the behaviours of the drivers in different environments such as traffic lights, freeway junctions, pedestrian crossing. This car test was named PROUD (Public Road Urban Driverless) and it was one of the biggest achievements in autonomous car technology. PROUD resulted in a few important observations such as precise route maps, efficient learning and perception mechanism. Jo et al. [54, 55] concentrated on the generalization of the autonomous car development procedure independent of any specific environment. FlexRay was utilized as a communication protocol and software platform to increase the system performance. Traditionally CAN (Controller Area Network) technology is used to communicate among different ECUs (Electronic Control Unit) in normal cars. But this technology was not suitable for autonomous cars due to its low speed and vulnerability to different attacks [56, 57]. FlexRay is a faster and more efficient technology, but it’s expensive. Some technologies used in fully automated cars are Lane Keep Assist (LKA), Park Assist (PA), Automatic Emergency Braking (AEB), Driver Monitoring (DM), Traffic Jam Assist (TJA), Dead Reckoning (DR) [38]. Most of these rely on sensor data and processing of this sensor data using machine learning/deep learning algorithms.

Computer vision

Autonomous cars require two essential and critical features such as object detection and computer vision. Using object detection techniques, autonomous cars should see the road and detect the object in the road. These two important features along with other modules help to drive safely on the road by avoiding unwanted situations. Janai et al. [58] conducted an accurate survey on computer vision algorithms such as perception, object detection and tracking, and motion planning used in autonomous cars. But still there are a number of unpredictable scenarios creating challenges for autonomous cars. Consequently, the success of computer vision algorithms will presumably take longer time. In addition to these algorithms, the decision support system has an important role in analysing the behaviour, learning mechanism of drivers. Also, AI has an important part in the prediction and perception aspects of autonomous cars. Shi et al. [20] studied on computer vision algorithms used for lane detection, object detection and surface detection experimented with GPU (Graphics Processing Unit), FPGA (Field-programmable Gate Array) and ASIC (Application-specific Integrated Circuit) Among these, the performance of ASICs is better than GPUs and FPGAs.

Computer vision is a boundless field of study, where autonomous cars use only object detection and motion estimation algorithms. These cars detect both static and dynamic objects and accurate object detection is always a challenging task due to multiple factors like shadows, identical objects, background lighting, and the size of the object. The detection algorithms accept these multiple factors into consideration and perform object detection with different sensors. Recent Autonomous cars use sensor fusion mechanisms to combine different types of sensors to detect day/night time, living/non-living organisms [59, 60]. Sensor-fusion based object detection techniques have more accuracy than traditional object detection methods. Chen et al. [61] proposed a CNN (Convolutional Neural Network) deep learning approach to detect 3D objects with a single camera, which takes generated date in LIDAR and images as input and predicts 3-D representation of that data. This method first detects objects using dissimilar features and then improves it for the identification of true objects. Then with the help of sensors data, it categorizes the objects into different types. This process of categorization is termed as pixel level semantic segmentation [62]. To support semantic segmentation, shallow and DL approaches are used for classification and prediction [62,63,64]. The approaches used in autonomous cars for object detection, semantic segmentation, and classification give improved accuracy, but have some drawbacks such as algorithm complexity, computational overhead, latency and complex design features. Therefore, compared to the shallow learning approach, DL (deep learning) approaches (such as auto-encoders and CNN) are preferred for object detection and classification due to its automatic feature selection process [65, 66]. DL-based approaches are also used in the construction of 3D images from the 2D image which are used for motion planning and actuation process in autonomous cars [67, 68].

Deep and machine learning approaches in autonomous cars

AI, ML and DL techniques are essential for autonomous cars as the object behaviour and surrounding environment are unpredictable. ML and DL techniques are used by most computer vision mechanisms. A deep Neural Network (DNN) approach learns features automatically from large data sets. Perception is an important aspect of autonomous cars and analysis of huge data collected from sensors for decision making is done by deep learning techniques. Tian et al. [69] introduced a DNN model called “DeepTest” to assess the behaviour of autonomous cars and found erroneous behaviour multiple times during testing when the car comes under erratic traffic and environmental conditions. These test results were not satisfactory for the current challenges and increased the need for more meticulous measures for the accurate functioning of autonomous cars. Chen et al. [70] used a novel mechanism to learn automatically the features from an image to calculate affordance in autonomous cars. Chen computed affordance for each driving action, as a substitute for individual driving scenes, it depends on multiple factors like static and dynamic objects on the road, pedestrian and vegetation Duanfeng Chu et al. [33] proposed a combined trajectory planning and tracking algorithm for autonomous vehicle control. Muhammad Mobaidul Islam et al. [71] proposed an efficient training strategy for pedestrian detection by occlusion handling by classifying body parts. Mohammed Ikhlayel et al. [72] made a car prototype for traffic sign detection using convolutional neural network (CNN).

There are two types of perceptions used in autonomous cars namely mediated perceptron and behaviour reflex mode. In mediated perceptron, current surrounding is not known and in behaviour reflex mode, deep neural networks are used to train the system based on human behaviour [70, 73]. Furthermore, Chen et al. also proposed another direct perception technique which is based on CNN. This system systematically learns mapping from a captured image to various features related to driving actions. TORCS, an open-source car racing simulator was used to test the car. Laddha et al. [74] proposed an algorithm to identify features from road images required for autonomous driving. In this algorithm an automated labelled training dataset was taken to make the process more scalable. It takes multiple sensor inputs which are mounted on vehicles including localization and camera sensors. This algorithm effectively utilized CNN to detect obstacles along the way with a moderately good accuracy. Dairi et al. [74] introduced another DL technique to identify obstacles on the road based on using a hybrid deep autoencoder and stereovision. More than 98% of accuracy was shown on different datasets by this method and it outperformed the DBN (Deep Belief Network) and SDA (Stacked Auto-encoders). Tam et al. [35] proposed a probability-based artificial potential field method for autonomous vehicles to avoid uncertain obstacles. The authors argued that this method is more effective in avoiding uncertain obstacles than traditional methods, and that it can improve the safety of autonomous vehicles.

Qizwini et al. [73] made a system and trained with 5 affordance parameters and tested by simulating with some realistic assumptions. XU et al. [75] used a Long Short-Term Memory Fully Convolutional Network (LSTM-FCN) by training the multi-modal driving-behaviours to predict future ego-motion problem in autonomous driving. The LSTM-FCN model was trained by a large video dataset of vehicular actions by the authors [76]. The author addressed the problems of traditional end-to-end learning by using a very large crowd-sourced dataset and the learning results were promising. Similar to this model, a number of popular DL architectures such as VGG-16, Google LeNet, AlexNet and ResNet are used accurately in semantic segmentation and scene understanding in autonomous cars [77]. Among these models, AlexNet achieved an accuracy of 84.6% and VGG achieved an accuracy of 92.7%, Google LeNet 93.3% and ResNet with 96.4% respectively. So, DL architectures played important roles in the multiple aspects of autonomous cars.

Sensors, communications, and control

Computing unit implement all the logic and it is the heart of an autonomous car. Sensors and actuators are highly needed in autonomous car systems. Basically, both known and unknown situations and environments are dealt with by autonomous cars and it needs ML, DL, AI techniques. These ML/DL techniques are data-intensive and these data are collected using various sensors placed within the car. Therefore, it follows a series of procedures such as data acquisition, data processing, communication and controlling among different modules inside the car and its surroundings. In addition to that it has to take autonomous decisions based on the circumstances and this feature requires a lot of interaction data with neighbours, infrastructure and the Internet. Communication among different modules and the environment is a vital function in autonomous cars and this data helps in sensor data analytics.

Autonomous cars deal with a huge number of sensors which generates a huge volume of data and these data are processed to get maximum utilization of it. Sensor Fusion is one of the commonly used techniques to gather data from multiple sensors intelligently to assist in the decision support systems. A number of algorithms have been proposed to deal with heterogeneous data used in autonomous cars. Oliveria et al. [78] proposed a more accurate technique to visualize a scene from 3D data gathered from sensors by using large scale polygonal primitives. Also, a visual scene may change constantly when obstacles come on the road, so there is a need for a stable mechanism to deal with unanticipated environments. So, the reconstructed scene should be calibrated continuously with the current scene to increase the efficiency as always, the new data from sensors is processed. Polygonal primitives-based scene reconstruction algorithms are incremental in nature and detection of polygon and reconstruction of new scene is performed using old data. Therefore, time required for detecting these polygons increases with an increase in the number of polygons, hence there is an increase in efficiency.

Identifying the road is one of the important aspects in autonomous cars and it is done using different sensors, camera and LIDAR (Fig. 4). Xiao et al. [79] applied a Hybrid Conditional Random Field (HCRF) to overcome the disadvantages of LIDAR and camera sensors. The proposed technique used a multi-modal approach and applied a binary labelling mechanism for separating road and background. This approach showed a maximum degree of accuracy then the existing point-wise CRF. Jo et al. [54, 55] used a number of existing functionalities along with sensor fusion algorithms that deal with sensors used inside the car, communication systems and also stressed upon communication of autonomous cars with the outside world by implementing different algorithms and schemes to establish communication with other entities. There exists efficient coupling between connected vehicles and autonomous cars as they are orthogonal to each other and reinforce towards making of intelligent transportation system. Vehicle-to-vehicle (V2V) and Vehicle-to-Infrastructure (V2I) communication technologies are popularly used to exchange their mobility data in platooning and cooperative traffic information system applications. Hobert et al. [80] worked on applications of cooperative autonomous cars such as convoy management, intersection management, low delay, cooperative sensing and enhanced reliability. It also helps in sharing the information about surrounding environment to operate more efficiently, timely and safely.

Fig. 4

Source – Velodyne. This image shows a 3D map of the surroundings of an autonomous car, constructed using lidar sensor data. The map illustrates the precision and accuracy of the lidar sensor in capturing the geometry of the environment, including buildings, roads, and other obstacles. The use of lidar sensors in autonomous cars is crucial for creating a detailed and accurate map of the car's surroundings, which is then used for navigation and localization

Peng et al. [81] scrutinized the presentation of IEEE 802.11p protocol and considered inter-platoon communication for different platoons in terms of network performance, delay, packet loss and other parameters. Platoon management is an important task in platoon communication, but it is LTE-driven to reduce the communication delay in autonomous cars. In recent years, transmitters and receivers are fixed in the front and back lights of the cars which uses visible light communication (VLC) technology, but it is in its initial stage and structured channel modelling is mandatory to meet the requirements of autonomous cars. VLC technology is also affected by the massive amount of data and network bandwidth. So numerous researchers are working on finding new techniques to reach out the demands of bandwidth requirement. Chang et al. [82] experimented on cooperative motion planning to communicate with pedestrians called “Eyes” on car, which means how the autonomous car makes an eye contact with the walker and evaluate the intention of the walker while crossing the road. The author tested with the real-world users and obtained the results of 86.6% in deciding intention of the users while crossing the road. It is observed that the real users made faster decision around 0.287 seconds then no eyes on the car.

Sensors used in an autonomous- car depends on its autonomy level as per the standards of SAE (table –2). There are a total of 6 levels of autonomy ranging from 0-5. Level 0 (Driver only) has no automation whereas level 5 refers to full autonomy in which the car performs all the Dynamic Driving Tasks (DDT) and also achieves the minimum risk condition (DDT fallback) [38]. A wide range of sensors are used until level 3 of autonomy, from level 4 sensors remain the same but the algorithms and processing of sensor fusion data achieve more autonomy. The U.S. Department of Transportation's National Highway Traffic Safety Administration (NHTSA) has adopted the standards set by the Society of Automotive Engineers (SAE) in the September of 2016.

Autonomous vehicle control (AVC) module is an important feature in autonomous cars that controls the behaviour and environments in various situations by controlling the hardware level component. AVC module is responsible for trajectory during motion by handling both predicted and unforeseen circumstances in autonomous cars. This module is also used in steering wheels to calculate the steering angle for the next action based on the control algorithm. This module is also responsible for speed control, distance travelled and emergency brakes. Apart from this module, the predictive control module is also used in autonomous cars to optimize the predicted motion for inter-vehicular communications. Immense amount of real-time data is required to communicate with neighbours and surroundings to stabilize the operations of autonomous cars. Autonomous cars need a powerful control mechanism to avoid transmission delays and communication errors that arise due to wireless communication. Zeng et al. [83] suggested an integrated control mechanism, where the communication delays were analysed to find the stability in autonomous platoons. The authors suggested to use the adaptive control mechanism to improve the reliability of the communication mechanism. The traditional control system is quite different then the control mechanism used in autonomous cars. The central control mechanism of autonomous cars is context oriented and uses adaptive control system to act immediately as the information of the context is highly required. Liu et al. [84] suggested a joint control communication mechanism to understand different road situations in autonomous cars.

Autonomous decision making

Autonomous decision making is a complex task in autonomous cars and certain cars are known as “ego-vehicles” which focus only on their local surroundings, current speed, direction and destination. But the communication mechanism must be strong enough in autonomous cars, so the concepts of “ego-vehicles” are fading away. So, there is a need of different prediction algorithms which gives predictions of high probability. And the final decision is a combined decision by taking the inputs from sensory data and other modules. But there is a number of other challenges related to environment, noise and limitations in the sensor data, mysterious state of neighbours. Therefore, autonomous cars should have accurate information about the neighbours in order to make predictions with high probability. But sometimes the neighbour information is neither available in public nor shared and this restriction causes serious problems for doing predictions and taking decisions accurately. Mockery of human behaviour in autonomous cars is very difficult, therefore the decision-making process is challenging. Numerous factors affect the decision-making process in autonomous cars such as behaviour of the car, prediction and perception, information about the neighbours, data processing done by sensors, calibration of equipment’s A number of present decision-making mechanisms are based on machine/deep learning, AI (Artificial Intelligence), Markov decision process. Hubmann et al. [85] highlighted the decision-making process for uncertain prediction of the surrounding vehicles from the sensor data. Many such AI based decision making approaches have been discussed by Claussmann et al. [86]. Several external factors impact the decision-making process in autonomous cars and these issues are addressed in a comprehensive way.

Issues such as ego-motion and inter-vehicle communication play a key role in crowd sensing and perception of neighbour’s behaviour. For commercializing autonomous cars in future, a major role will be played by connected car technology and it is seen today in many high-end cars [87]. The cloud infrastructure connected to the autonomous-car can also provide more computational resources for the car to execute its software processing functions, ensuring the car functions even in case of a hardware failure. Cloud infrastructure can also provide many new applications including entertainment [88]. But few issues arise while using the cloud services such as communication delay between cloud infrastructure and autonomous car. Any kind of delay can’t be tolerated by the decision-making module of autonomous cars due to cloud infrastructure, hence mostly decision-making is done locally in real-time environments. Similarly, fog computing techniques are also implemented which gives low delays in providing real-time services [89].

Testing of autonomous cars in real-time

In today’s scenario, numerous tests of autonomous cars have been carried out to evaluate its performance. Campbell et al. [11] took part in the DARPA grand challenge (DGC) and conducted 3 rounds of test for autonomous cars and documented the results from those tests. Campbell tried to incorporate the autonomous car mechanism in a regular car and tested it without any human intervention. In DGC challenge a route network definition file (RNDF) was given for self-driving using a road map to follow [90]. This challenge helped Campbell to understand the problems and issues that need to be addressed and how to commercialize the vehicle. Endsley et al. [91] considered Tesla autonomous car Model S for a period of 6 months and analysed different issues related to situation awareness, response to unanticipated circumstances on the road He also studied a number of other parameters related to consumer behaviour, trust, complexity, interface design and feedback from the customers. Broggi et al. [92] invented a BRAiVE (Brain Drive) system to conduct tests on autonomous car at Intelligent Systems and Artificial Vision Lab. Broggi’s test accumulated a huge amount of test data of real-time driving that is now used for upgrading the existing autonomous cars. Further Broggi tested autonomous cars on streets and he named his project as PROUD. Jo et al. [54, 55] designed an autonomous car and participated in South Korea in 2012. Jo developed the architecture of the car and experimented in various environments.

AUTOSTAR [93] is one of the open standard architectures popularly used in many autonomous cars, but it has its limitations due to high cost and high complexity to implement. Later AUTOSTAR-lite came into market as a replacement to AUTOSTAR. Jo tried a distributed architecture instead of a centralized architecture to reduce the complexity and to group the functional components. This architecture also increased the efficiency and performance through parallel computations. A number of automotive industries along with academia experimented and “Drive Me” is one of the special projects done by Volvo in autonomous cars. This project in Sweden collected information from 100 consumers about their daily routines, driving behaviour, their preferences and other important aspects of driving. This consumer data helped researchers to improve the commercial autonomous cars. Blockchain technology can also be used for shared training of autonomous-cars where cars can share their experience over a blockchain network [27]. Blockchain technology can also be used to charge electric vehicles with no human intervention as shown in [28] with the help of a public ledger recording each transaction.

Design and implementation issues

A number of factors such as safety, robustness, hardware / software designs, customer behaviour will decide the future of autonomous cars. And also, these cars need to be designed with utmost precision, safety and reliability features [94]. The major components of autonomous cars are LIDAR, sensors, radar, positioning systems and various optimized software’s. A number of issues have direct impact on the autonomous car industry such as cost, maps, software complexity and simulation. First, the software/hardware cost is a major barrier in manufacturing autonomous cars. LIDAR is one of the expensive products in the car. Second, the maps used in autonomous cars contain a number of road details and it differs from the normal maps generated by the GPS systems. Memory requirements and processing power are enormous to store all the road details. The log files generated from these cars also requires memory to store and it contains detailed information about localization and mapping. Third, the software program of the car decides the various operations of the car such as move, stop, and lane change and overtake. So, a highly accurate and reliable software program is required which takes inputs from different sensors [95]. The huge data acquired from the sensors, environment and its surroundings create a real challenge for the autonomous car [96]. The algorithms processing this huge amount of sensor data should be efficient as well. An example stated in [38] is the use of stereo-camera over normal camera. A stereo-camera can take 3D images that map the environment accurately. Even though the data generated by stereo-camera is huge, it takes less time to process this data then conventional image processing.

Autonomous cars still need to be tested in adverse conditions such as mist, rainstorms, night-time and densely populated cities. The software used in autonomous cars also records the driving patterns and behaviours such as obstacle management, pedestrians crossing, and overtaking. Simulation technology is becoming the self-driving technology for autonomous car designers and Google is one among the top leading company in the market along with hardware giants like Nvidia announcing the launch of support hardware for autonomous cars [8]. To check the software reliability and safety, large-scale simulation is necessary. After simulation it is tested on real hardware with all built-in functionalities. Simulation tools developed specifically for the requirement of autonomous vehicles are utilized to simulate diverse aspects such as path planning and testing, mobility dynamics, fuel economy in urban scenarios [97]. Sajjad et al. [49] proposed an efficient and scalable simulation model for autonomous vehicles with economical hardware. A reduced reality gap for testing autonomous vehicles has been proposed by Patel et al. [98]. The simulation tool designed by Buzdugan et al. [34] provides a realistic environment for students to learn about the behaviour of autonomous vehicles and how they interact with their surroundings. The authors also evaluated the effectiveness of the tool in teaching the behaviour of autonomous vehicles. The study can help in the development of new methods of teaching autonomous vehicles and to improve the understanding of the behaviour of autonomous vehicles. When such simulation and testing models are combined, the industry can use various models for training autonomous cars both efficiently accurately.

Challenges for autonomous car deployment

A number of challenges still exist and it must be resolved by various stakeholders, manufacturers, developers, academicians, policy makers and designers [99]. These challenges can be categorized into technical, non-technical, social and policy related. A number of technical challenges arise during car deployment such as validation and testing, hardware / software resources, quality, safety, privacy and security. Validation and testing are time consuming process and it varies from model to model and it also depends on the degree of sophistication. Different of testing techniques are used such as simple bug fixing to quality testing. A number of safety and mission-critical testing are performed to fine tune the performance of autonomous cars and to check whether the rigorous necessities are met or not. Koopman et al. conducted widespread overview of the validation and testing trials for autonomous cars when deployed at scale [100]. The author applied ISO 26262 development V process which maps each design of autonomous vehicles to an appropriate testing method while focusing more on specific advanced challenges and divided the work into 3 phases namely phased deployment, monitor architecture, and fault inoculation. This testing framework is popularly used in in automotive industry for validation and testing and it checks the specific requirements of vehicles minutely and test all expected functionalities. Autonomous cars have specific set of complex requirements, and it is different from traditional validation and testing process. The author also discussed the challenges in the testing of autonomous vehicles as vehicle level testing is not thorough enough to ensure systems with ultimate dependencies which, in this case, are the various technologies employed in autonomous cars discussed in the prior sections. Younang et al. [37] examined how the concerns of users and government policies regarding autonomous vehicles compare and contrast. They analysed data from surveys and government policy documents to understand user's concerns and government policies on autonomous vehicles (Fig. 5).

Fig. 5

His image is a flowchart that classifies the challenges faced in the development of autonomous cars into four categories: Technical, Non-technical, Social, and Policy. The chart illustrates the complex nature of autonomous car development and the various factors that must be considered. The Technical challenges include issues related to sensor technology, machine learning, and artificial intelligence. This flowchart is a useful tool to understand and address the different challenges that autonomous car development faces, and it helps to identify the areas that need further research and development

Safety and reliability are the second core aspects to be addressed in autonomous cars. The degree of reliability can be identified to some extent by statistical analysis by using a huge database of distance travelled by the car. Safety is one of the major concerns for autonomous cars, so a number of new methods are required for measuring the reliability parameter. Similarly, legislation is another prime challenge in autonomous cars. Safety is an important and interdisciplinary issue, so a lot amount time and money are spent on safety measures of the car [101].

Software quality is third aspect in autonomous cars as the car is run by complex and sophisticated software. Validation and testing of these software’s play a major role due to mission-critical and complex software system used by these cars. These software’s responds to a number of unanticipated situations and in all situations, it must be fail-safe.

Computational resource requirement is the fourth aspect in autonomous cars due to a number of high-resolution cameras used for monitoring various operations of the car. Apart from that a number of sensors are used for object recognition such as LIDAR. So autonomous cars require GPU processors with FPGA (field-programmable gate array) and SoC (system on a chip) to meet the functional and operational requirements of the car. Optionally these can be connected to a cloud infrastructure which provides the necessary computational power as a service. But in case of a connection timeout, it can cause problems. A number of sophisticated algorithms are used for data processing and provide an efficient and reliable system.

Security requirement is the fifth and one of the important aspects in autonomous cars. The data shared between various components and among other vehicles and infrastructure must be kept safe and inaccessible to unauthorized people. All type of communication information must be safe from hackers, so a number of cryptographic techniques are used to provide internal security and privacy.

Conclusion

Automotive industry is fast changing and a number of auto maker’s companies are now making autonomous cars. A number of new business opportunities are there for car makers along with few challenges such as safety for car and passengers. In this paper, we discussed the current solutions regarding design, operation issues, and forthcoming challenges. A number of major benefits and technical challenges are thoroughly discussed. A number of research areas to be focussed for autonomous cars have been outlined such as computer vision using deep learning, decision-making using machine learning, navigation, planning, control, perception, blockchain technology, cloud infrastructure integration, A number of real-life tests conducted so far are also outlined on autonomous cars. This work has focussed on the recent technologies behind autonomous cars like AI decision-making-based path planning, layered architectures, cyber threat protection mechanism, use of blockchain for transaction management and shared training of autonomous vehicles and also provides a valuable reference to the field of autonomous cars. This paper focussed on a number of technical and non-technical issues that exist in the design and implementation, including the analysis of the consumer and driver preferences, of autonomous cars. A number of issues were reviewed regarding object tracking and detection, data acquisition from sensors, safety and reliability, decision support, security, simulation models and privacy. Even though notable results are achieved in the research and development of autonomous cars and now that we have entered the commercialization phase, but still this field still has several areas of research like user experience and precautions and hardware for efficient decision making on the vehicle. The future of autonomous cars is promising, but further research and development is needed to overcome the remaining obstacles like autonomous vehicle testing for component compatibility, accurate simulations, large scale training and to fully realize their potential. The paper concludes with a call for further research in the field of autonomous cars to address the remaining technical and non-technical issues and to maximize the potential and adoption of autonomous cars in the future.