Human - Artificial Intelligence Teaming for Automotive Applications: A Review

Artificial intelligence (AI) is being utilised as a paradigm to a variety of industries enhancing data processing and decision making actions. In healthcare, diagnoses and treatment plans are assisted with AI being at the forefront [1]. In finance, AI is encapsulated to detect frauds and optimise strategies [2]. In education systems, custom learning experiences are performed [3]. In the transport domain, routing can be safely and efficiently planned [4]. Moreover, manufacturing [5], agriculture [6] and cybersecurity [7] constitute domains whereby AI thrives towards their efficient transformation.

In the automotive domain, AI is utilised to improve safety, efficiency and the driver’s functions in the vehicle. AI is used in autonomous vehicles as well as in driving assistance systems, which change the driver’s state of mind as well as the vehicle’s tasks. The prevention of accidents is performed by the increased perception of the environment and the decision making in complicated traffic scenarios [8]. In addition, predictive maintenance based on AI models predict and prevent vehicle failure, minimising downtime [9]. AI models are also utilised for fuel consumption optimisation [10], routing [11] and customisation of driver custom preferences [12]. It is clear that the use of AI-based technologies leverages the safety, security and pleasant driving experience.

HAIT involves ande combines the human perception advanjtages with the accuracy of the models of AI, and the disadvantages of both are being addressed. There is ongoing reaserch that deals with the opinion of humans on AI-based systems in terms of how reliable explainable and biased they are.

The perceived reliability of AI is crucial for human trust and utilization. For instance, Zhang et al. [13] demonstrated that providing healthcare workers with insights into AI's data transformation process boosted their perception of data quality reliability. Another study found that AI was exhibited better reliability than humans in decision support contexts. AI systems with accountable and explainable features are more likely to gain the trust of human decision-makers (DMs).

The HAIT design distinguishes the authority each team member has over independent control and the individual contribution towards task completion. Endsley and Kaber [14] define levels of automation in systems. The levels of intervention of the individual to a system are given in Figure 1.

The AI-first design paradigm assumes a dynamic circulation of knowledge across the network, enabling decision-makers and autonomous systems to collaborate effectively. Autonomous systems excel in deductive reasoning and solving complex tasks, including automotive applications such as cruise control [15] and lane detection [16]. Operators perform well at applying context and providing input to specific tasks. Automation tasks place when complex tasks' outputs come from sensors and systems, i.e., navigation of vehicles. In particular, the system may identify the best potential road along the path of the vehicle.

1.png

Figure 1. Levels of automation

2.png

Figure 2. IoV with HAIT

IoV [17] has emerged during the last decade whereby the vehicle is connected to the Internet and decisions are thereby made regarding several aspects such as platooning [18]. The IoV can move towards the HAIT, in terms of assisting the driver with decisions coming from Road Side Units (RSUs or other vehicles by an AI system. There the driver will obtain the data wirelessly and the data will be fed to a Machine Learning (ML) and/or AI system to inform the driver regarding specific actions that need to be performed to optimise driving. A basic IoV network with HAIT capabilities is given in Figure 2.

An example of IoV with HAIT is dynamic routing. Taking RSUs in the traffic lights where the traffic is communicated based on the queues of the roads and the speed of the vehicles, the communication system of the vehicle takes the information of the aforementioned metrics and calculates the best route taking into account the current state of the roads. The in-vehicle system informs the driver regarding the proposal of the new route and it is up to the driver to follow it or select a different route. Upon a good decision by the model of the vehicle the driver will acknowledge the optimal decision to the system.

In this paper, the reader is provided with the necessary information to comprehend the notion of HAIT. Using the different ways of representing HAIT an automated IoV system can be built, which will take advantage of the wireless capabilities available and the AI methods that can be utilised to perform some action. The reason behind HAIT is to enable and enhance trust of automated automotive systems, namely the data coming from sensors of the vehicles and RSUs. We present the human in, on, and over the loop concepts and finally, we provide a review of the current academic works that involve HAIT in the automotive domain. The aim is to work closely with the aspects of IoV and AI to integrate them to common solutions. The IoV is a research domain which gets attention due to the dominion of the Internet of Things (IoT). Models that need to be simple to run with minimum delay of response should be available and constructed to maximise the performance of the IoV. Taking, for example, the use of telecommunications to connect a RSU with a vehicle, the response time should not exceed seconds because the nature of the deployment is dynamic and hazardous. Hence, the AI model of the vehicle needs to assist the driver in a fast manner and the driver to assess, agree or not with the suggestion within seconds. This gives a new dimension to the creation of models for HAIT.

This review of academic contributions to HAIT has been primarily undertaken using Google scholar, which has an overlap with other databases. The articles that have been found were selected according to the relevance with the proposed subject and the academic nature of the contribution. The synthesis of the data has been done based on the respective method that has been used to tackle specific issues in the automotive sector.

Primarily, in this section, examples of the different aforementioned models are given. For the HITL, an example can be when the car at front increases its acceleration and the system of the car behind calculates the distance based on some kind of sensor to keep a platooning type of movement. For the HOTL, an automotive application is the throttle control, which can be automated and overridden by the driver. For the HOVTL, course management in vehicles can be automated and further data can be available to the human to make a different call where applicable. In terms of the autonomous peer decision makers, a vehicle may have its own automated cruise control based on RSU and Vehicle to Vehicle (V2V) data while the driver can act independently of the throttle and brakes. A coordination mechanism needs to be present in order to make the teammates act to the benefit of the drive. Here we have different accountability since the processes act independently. Reading in Kumar et al. [20] and references therein, there is a number of applications of autonomous vehicles that use the HITL method.

The market for automotive artificial intelligence is expanding globally, characterized by its size, trends, and industry growth across various segments. It covers offerings such as hardware and software, driven by technologies including deep learning, ML, natural language processing (NLP), context-aware computing, and computer vision. This market analyzes components, processes, applications, and regions, with projections extending to 2028 (https://www.researchcorridor.com/global-automotive-artificial-intelligence-market/).

There exists a number of advancements in the development of intelligent assistants using AI in the automotive industry. These include:

-Audi Autonomous Intelligent Driving (https://www.audi-mediacenter.com/en/audi-ai-9099/download) is actively researching and developing autonomous driving capabilities, integrating AI technologies with human input to create safe and efficient driving solutions.

-Waymo (https://waymo.com) is the self driving car project designed by Google, that has been one of the leaders of autonomous vehicles and it encapsulates AI with human intevenstion in order to create self-driving cars that enhance safety and efficiency.

-Tesla's Autopilot system (https://www.tesla.com/support/autopilot) integrates AI and human tasks in order to assist drivers with driving functions such as lane-keeping, adaptive cruise control, as well as automated parking.

-Uber ATG (https://www.uber.com/en-GR/blog/machine-learning-model-life-cycle-version-control/)integrated AI for navigation purposes and safety while altering the driver to a act in a supervisory role with the ability to intervene.

-BMW (https://www.bmwgroup.com/en/innovation/automated-driving.html) created a campus that concentrates o the development of autonomous driving technologies, integrating AI and human abilities in order to increaser vehicle automation.

-Mercedes-Benz (https://www.mercedesbenzofeaston.com/mercedes-benz-intelligent-drive-overview/) created Intelligent Drive systems, which encpasulate AI-based models in order to increase safety and satisfaction of driving comfort.

-Ford (https://media.ford.com/content/fordmedia/fna/us/en/news/2023/03/02/ford-establishes-latitude-ai-to-develop-future-automated-driving.html) is investing in autonomous vehicle technologies, utilizing AI and human supervision to develop next-generation mobility solutions.

-NVIDIA's DRIVE platform (https://developer.nvidia.com/drive) provides AI computing solutions for autonomous vehicles, supporting human-AI collaboration in the automotive industry.

3.1 Reinforcement learning-based systems

Wu et al. [21] present a research work, which creates a HITL Deep Reinforcement Learning (DRL) system that viably leverages human insights in real-time amid show preparing. A real-time Human-Guidance based DRL (HugDRL) strategy is created and effectively connected to the operator preparing beneath independent driving scenarios. Beneath the proposed design, an energetic learning prepares adaptively apportions weighting variables to human encounter and DRL activity, in arranging to optimize the DRL’s always progressed capacity over human direction amid the by and large preparing handle. Based on this human-in-the-loop direction instrument, a moved forward actor-critic architecture with modified arrangement and esteem systems is created.

The rapid integration of the proposed HugDRL facilitates the real-time integration of human-directed activities within the agent's training cycle, significantly enhancing the efficiency and performance of deep reinforcement learning. This method has been validated through human-in-the-loop tests involving 40 subjects and compared against other state-of-the-art learning approaches. The findings indicate that the proposed approach effectively improves the training efficiency and performance of deep reinforcement learning algorithms under human guidance, without mandating specific expertise or prior experience from participants.

Gopinath et al. [22] focus is directed towards human-AI driving teams, where the AI system interacts closely with the driver through tactile data during critical moments of the driving task. In this scenario, it is assumed that the human maintains continuous control over the vehicle, with the AI system providing recommendations, alerts, or other supportive measures to assist the driver. This approach contrasts with more autonomy-driven integration setups, where the AI system aims to override the driver by assuming control of the vehicle. Such setups are designed to respect the inherent preferences and capabilities of the human partner.

The goal is to demonstrate the flexibility of the proposed system in accommodating various characteristics of human drivers, such as their level of distraction, cautiousness, and preferences towards AI-based interventions. The authors model this scenario as a reinforcement learning (RL) problem, seeking to find a policy—a mapping of observations by the agent to actions taken on behalf of the agent. They conceptualize both the human and AI system in terms of their higher-level objectives and constraints within a framework of rewards. This framework allows for the representation of the agent (the policy) as either a single entity representing both the human and AI systems working together, or as two separate entities interacting with each other. While their approach primarily supports the former (i.e., a joint policy), the system accommodates both configurations to generate results effectively.

The approach learns from real-world perceptions gathered through experiences and continuously improves by maximizing a cumulative reward for each rollout generated by the trained policy. In the context of human-AI collaboration, the state encompasses a simulated environment that includes other vehicles, road conditions, and the human's state. Observations are potentially imperfect measurements of this environment.

The human model is formulated as a Markov Decision Process (MDP), which specifies how the system transitions from one state to another based on actions taken, along with associated rewards (e.g., speed preferences) or penalties (e.g., collisions) incurred during these transitions. To train policies within this framework, a policy gradient method is employed, specifically the proximal policy optimization (PPO) algorithm. This method facilitates the iterative improvement of policies to optimize performance in navigating and interacting within complex driving scenarios.

Huang et al. [23] propose a reinforcement-learning-based approach for designing shared control in semi-autonomous vehicles involving human supervision. The collaborative effort between the assistant pilot controller and the driver enables simultaneous vehicle control. To account for driver reaction time, the human-vehicle system is characterized using differential-difference equations. Real-time data is utilized to develop an adaptive optimal shared controller through adaptive dynamic programming, without requiring complete knowledge of the driver and vehicle models.

The data-driven shared steering controller ensures near-optimal solutions, adaptation, and stability within the human-in-the-loop vehicle system. It effectively manages potential variations and uncertainties in the human-vehicle interaction. The efficacy of this control strategy is substantiated through theoretical proofs and demonstrated with numerical results, affirming its capability to enhance operational efficiency and performance.

Elmalaki [24] states that the number of applications targeting the automotive sector is expected to be high. Hence there is a necessity of a systematic framework that assists to the design and verification of these applications. Furthermore, these applications are built in order to have an interaction with the human; thus, there is the need for human-centered applications. Here, the authors, propose the MAConAuto, which is essentially a framework that encapsulates the preferences of the human behavior and reaction time at the core of the automotive applications. This framework can provide support for two types of applications with respect to the intervention level. These are: Type I, Direct-based interaction and Type II, Monitor-based interaction. MAConAuto is highly adaptive to human variability using a RL engine that tunes the assistance that the two application types perform by addressing certain challenges of human reaction modeling regarding these assistive interventions. The authors elaborated on the design of the system with verification from two applications, namely the context aware Heating Ventilation and Air-Conditioning (HVAC) and context aware Forward Collision Warning (FCW).

Ahadi-Sarkani and Elmalaki [25] investigate the Lane Departure Warning (LDW) systems. These particular systems obtain their data from algorithms that handle sensors rather than encapsulating as well the driving behavior of a driver as well as the physiological data. An Adaptive Driver Assistance System with Reinforcement Learning (ADAS-RL) is suggested which includes the human factor into LDW approaches. The drivers’ response time to functionalities, their driving characteristics and the levels of concentration are taken into account to customize the warning times as well as the frequency. The ADAS-RL is essentially a HITL system, which tweaks the frequency of the warnings to drivers according to the level of concentration in the vehicle. The distraction levels of the drivers are considered to optimize the level of warnings. The proposed system detects the variability in the driver response times; hence it shows the level of adaptation depending on different behaviors of drivers at both simulation and real test cases.

In the study [26], an advanced HITL-RL system is suggested, which aims to construct policies of driving, integrating transportation and robotics characteristics. The Human as AI Mentor (HAIM) is at the heart of this research, whereby humans perform tasks that include supervision, interventions as well as demonstrations to AI models while being in a learning procedure. A HAIM-DRL is proposed, which utilized DRL with interventions by humans to strengthen safety in Autonomous Vehicles (AV)s as well as maximizing the efficacy of the traffic flow. The HAIM-DRL, provides several strengths, which comprise the increased training safety which is vital for real test cases of AV deployment and optimized efficiency in training. Other advantages include the better integrations of AVs to traffic by minimizing disruptions and performing better flow of traffic, as well as supreme generalization to different scenarios. In summary, this paradigm provides a standard for a collaborative setting between humans and AI systems, whereby safety and efficiency are promoting in AVs.

Future research directions will focus on real-world implementation, scalability of the framework, and reducing dependency on perfect human expertise by learning from a wide range of human drivers. Integrating additional transportation domain knowledge will also be critical for advancing the seamless integration of AVs into mixed traffic environments, ultimately contributing to the development of smarter and safer transportation ecosystems.

The RL methods presented offer several advantages: real-time human insights improve training for autonomous driving scenarios, balancing human input with DRL actions optimizes learning, and RL enhances training efficiency and performance without needing specific expertise. Additionally, these methods increase safety, stability, and context-aware interactions, while personalizing outcomes based on individual actions.

3.2 Hidden Markov model-based systems

Dai and Xu [27] explore control augmentation within HITL systems, such as advanced driver assistance systems, where human capabilities impose limitations like visual range and understanding of the Internal Vehicle Model (IVM). The research proposes a novel framework centered on a new human IVM model comprising a Hidden Markov Model (HMM)-based parameter predictor and an augmenting controller. This framework aids in precisely tracking a pre-determined trajectory for the controlled vehicle.

The HMM method includes a specific issue that needs to be addressed which is the humna IVM parameters alignment at the time of the Expectation-Maximisation (EM) procedure. In the case that the IVM parameters are not known, recursion is performs to obtain a closed-form solution which addressed dynamical systems. Simulations of pilot-controlled quadrotor attitude control has been performed to validate the suggested method in terms of how effective it is. Duting the simulation, there was a clear performance increase of the closed-loop system in comparison to the human IVM error not being corrected approach. Already used HITL systems will lack of the advancement of their capabilities comparing to the proposed approach in a diverse number of applications.

Janssen et al. [28], suggest a HMM framework targeting semi-automatic vehicles. The approach aims to formalize the beliefs of the humans in terms of the modes of operation of the aforementioned vehicles. This work takes as a start previous research, which determines different automation levels and the level of expectation of the operator involvement. Moreover, the prior work determines the distinct automation levels of the vehicle as well as the modes of operation, which are timely –evolved. The levels of automation are required to be obtained in an accurate manner, since the driver may get confused regarding the mode of automation of the vehicle. The HMM that is being suggested, is there to address the confusion of the driver which comes as a result of wrong beliefs regarding the modes of automation. This work coincides with other theories and research works targeting automation in vehicles. It serves as a contributor to the design and evaluation of automation in systems as well as future infrastructure of the transport domain. It also provides insights towards understanding and handling issues that arise in real test cased in terms of automation.

HMMs models human behavior by representing the sequential and probabilistic nature of actions and states through hidden states and observable states (actions influenced by hidden states). Transition probabilities model how human states evolve over time, while emission probabilities link hidden states to observable actions. Parameters are learned from data using algorithms like Expectation-Maximization (EM). HMMs capture the evolution of behavior, handle uncertainty, and learn from real-world data for accurate and adaptive modeling.

3.3 Experimental, simulation-based and other systems

Chiang at al. [29] suggest integrating Human-in-the-Loop (HITL) design into a longitudinal automation framework, successfully implemented and validated on a passenger vehicle in real traffic scenarios. This system features a comprehensive architecture comprising an adaptive detection zone, supervisory control, and regulatory control, structured hierarchically for adaptability across different vehicles with minor adjustments. Safety enhancements are achieved through the adaptive detection mechanism, which prevents potential collisions with vehicles ahead during curves.

The supervisory control autonomously selects control modes independent of vehicle-to-vehicle communication, ensuring coherent operation within predefined acceleration limits. Regulatory control for throttle management is designed based on an understanding of human decision-making processes, aiming to minimize operational rules. This automation system assists human drivers by managing speed and inter-vehicle distance, alleviating driving burdens, particularly on long trips.Experimental results in actual traffic environments demonstrate the effective performance and comfort of the longitudinal automation system, validated against ISO 2631-1 standards.

Kuru [30] proposes that while human drivers may eventually cease driving, they will still need to be trained in teleoperating fully autonomous self-driving vehicles (FA-SDVs) equipped with Vehicle-to-Everything (V2X) technologies for location-independent remote control. These capabilities are further enhanced with haptics and Tactile Internet to intervene promptly when AI encounters unforeseen situations beyond its autonomous capability.

To bridge existing gaps in this area, the paper explores establishing an ecosystem aimed at fostering location-independent collaboration between capable Human Teleoperators (HTS) and intelligent FA-SDVs. This concept is embodied in the HOTL-HT-SDV framework, which integrates two parallel worlds: the physical environment and its cyber emulation. The HOTL concept is studied from technological, psychophysical, and philosophical perspectives, emphasizing the critical role of haptic feedback in enabling timely interventions that enhance FA-SDVs' ability to handle uncertainties in real-time. By leveraging the full potential of this framework, the authors argue for accelerating the integration of vehicles into traffic by fostering high levels of trust, even without expecting the technology to achieve perfection. The dual active roles of HTS and FA-SDV are evaluated through Quality of Trust (QoT) metrics, ensuring bidirectional interaction that optimizes Quality of Experience (QoE) and Quality of Vehicle Experience (QoVE).

Different to Humans-Are-Better-At/Machines-Are-Better-At (HABA/MABA), where humans and automation agents are invloved into a competition on who is the owner of a specific task, the suggested Human-Agent-Robot Teamwork (HART)-centric framework sets the priority for the engagement of a human while it gradually moves towards complete automation of FA-SDVs. The change that brings the adaptive automation from human interventipn takes place via the utilisation of a collaborative and behavioral learning and it is shown in studies where proof-of-concept is necessary. This provides evidence of the benefit that may arise from collaborative learning.

Hong and Aparow [31] suggest the utilization of an end-to-end simulator that is implemented to support Level 3 safety systems for autonomous vehicles, as well as for the collection of data related to driving. The simulator comprises an IPG CarMaker with Simulink as well as the Logitech G29 Driving Force Steering Wheel and Pedals kit. Driving in a road network is available at the simulator. Recording of driving data is undertaken, and parsing as well as result clarification is given. Moreover, development and testing of ML algorithms is demonstrated. This simulator’ primary objective is the streamlining of development of autonomous systems as well as the strengthening of reliable safety procedures that undergo testing.

Chen et al. [32] demonstrate a driving assistance system called HITL Connected Cruise Control (hCCC). This system integrates humans and machines towards collaborating between them which gives the flexibility to a non-automated but connected vehicle to be part in a platooning setting, while ensuring and improving string stability. The driver has control of the vehicle’s functions, while the hCCC is more of an assistant which performs control modifications in order to keep the vehicle stable.

The controller of the hCCC is caefully designed with a bi-level architecture in order to accomplish control linearization. It utilises specific features taken from existing and established Cooperative Adaptive Cruise Control (CACC), including a feedback-feedforward control structure and a zero-spacing-error rule. Validation came as a comparison of the hCCC having drivers with existing CCC and it demonstrated that the hCCC showed supreme stability takin as input a significant variety of human behaviour as well as uncertainties. Moreover, the hCCC was validated in a simulator in order to investigate its performance when real conditions are given. The results showed that the hCCC exhibited quite large improvements in comparison to human drivers, including acceleration reduction, less fuel dissipation and time-gap variation decrease. In contrast to traditional systems, the hCCC is flexible to the adjustment of parameters regarding users. This comeas as a result of the enhanced string stability abilities that can encompass different driving behaviours.

Ropelato et al. [33] showed the Virtual Reality (VR) methods’ efficacy to enhance vehicle driving experience.They simulated the physical properties affecting car driving characteristics, including the behavior of the engine and transmission system. Their software integrates with a 6 degrees of freedom (DoF) motion system to simulate acceleration during driving, utilizing input devices in a cockpit mockup to control the virtual car. AI-driven vehicles operate within a city environment, adhering to predefined traffic rules and interacting with both other AI vehicles and the driver’s vehicle.

Additionally, the authors proposed five types of driving-related activities that can be automated and evaluated through an Intelligent Transportation System (ITS). They adapted the zpdes algorithm to drive vehicles, demonstrating how personalized teaching sequences can be developed. Challenges such as limited computational performance, integration of motion hardware, and effective simulation of city-wide traffic were acknowledged. A user study revealed that most participants experienced a high level of presence in the virtual world and demonstrated proficiency in operating the car within the VR driving simulator.

Ai et al. [34] suggest an enhancement of autonomous transportation in open-pit mines, the iMAPeM paradigm which introduces an intelligent mining system that incorporates humans in the loop, providing a reliable, efficient, and universal operating framework. The iMAPeM architecture features three categories of miners—biological, digital, and robotic—each with specific roles to improve safety and efficiency while reducing the workload on human miners. It also includes three models: autonomous, parallel, and emergence/expert, which can be adjusted based on the severity of production issues to ensure stable transportation policies.

The iMAPeM-based parallel mining system, YUGONG, has been implemented in various open-pit mines, demonstrating stability and efficiency in real-world scenarios. As the system's operating time and mileage increase, its stability improves, reducing reliance on the Expert/Emergency mode and lessening the tasks assigned to human miners. YUGONG is also evolving to incorporate human-machine interactive functions and a multimodal model, aiming to create a human-centered operation system that provides intelligent services for the diverse transportation needs of open-pit mines.

Wang et al. [35] introduce a human-in-the-loop multi-indices fusion decision framework (HITL-FD) for predicting Mars rover trafficability. The framework uses fuzzy theory to address the uncertainty of trafficability boundaries and integrates multiple evaluation indices through a fuzzy decision tree algorithm. Human-assisted decision-making is included in the decision loop, and a reward function based on decision error is proposed to create a closed-loop system. Results indicate that HITL-FD increases the performance of the trafficability prediction and decreases the uncertaintly of the terrain in comarison to decision method made only by machine.

Kaufman et al. [36] suggest a framework who’s main objective is the cultivation of situational awareness of human AV systems by the joint action theory. This framework emphasizes on the significance of personalised, shared and de-centralised situational awareness uo to a threhold, in order to accomplish objectives related to joint action in the teams between humans and Avs. An example is the assurance of safety in transportation as well as the learning process from the behaviours of the AVs. There are also other goals that need to be satisfied such as the establishment of trust. The efficient collaboration is an outcome of the human and the AV having accurate representations of each other and processing them along with the driving context. There the situational awareness of the individual agent is mandatory in ordrer to set and promote their actions.

The situational awareness is quite significant in order to allow the team to achieve its objectives. Explainable AI (XAI) is performed to be incorporated by humans as well as human input in real-time as well as AVs sensing retaed tasks. This is crucial for the exhange of data. The AVs can incorporate data for training to predict and personalise the communication depending to the different contaxt. There is the necessity of the understanding of the interactions between four components, namely the characteristics of the AV, the objactives of each action, the individual conditions and the environment of the driving task. The comprehension of these core components is mandatory to enhance situational awareness and to secure efficient joint action. The successful joint action shows the necessity of the prediction and the adoption of the shared and individual situational awareness behavior; thus, assuring the fact that there is the need to coincide with the tema’s objectives.

The approaches discussed in this section have several limitations. HITL into automation frameworks can face adaptability issues in real-world conditions and safety assurance challenges across various driving environments. Teleoperation of fully AVs heavily depends on V2X communication and haptic feedback systems. Transitioning from user intervention to full autonomy in HART-centric frameworks can be complex due to collaborative learning modes. End-to-end simulator platforms may not accurately mimic the unpredictability of real-world driving. Human-in-the-loop Connected Cruise Control systems need thorough validation to ensure robustness for different driving behaviors. Virtual Reality simulations for car driving face computational performance challenges and issues in simulating realistic city-wide traffic. Autonomous mining transportation relies significantly on human oversight and expertise. Additionally, situational awareness in human-AV systems is compromised by the unpredictability of human behavior and limitations in current AI explainability methods.

3.4 Deep learning and neural networks

Usman et al. [37] propose a framework integrating a probabilistic Convolutional Neural Network (CNN) and fuzzy logic to predict accidents in vehicular networks. This approach emphasizes human-in-the-loop intervention through classification of driver emotions—such as heartbeat and facial expressions—complemented by traffic status analysis. A probabilistic graph-based inference model is employed to estimate accident probabilities using data classified during the initial learning phase. Subsequently, a fuzzy rule-based mapping method maps the severity of accidents based on this model. Alerts are then generated for drivers to prompt appropriate actions. Validation of the framework was conducted using Kaggle data suitable for accident prediction, incorporating emotional states and road traffic data. Experimental results demonstrate superior performance compared to benchmarking methods in accurately predicting accidents. This improvement is evident in metrics such as accuracy, precision, and F1 score, underscoring the effectiveness of integrating human-in-the-loop considerations alongside other critical factors.

Schmidt et al. [38] explore learning in an industrial context focusing on object detection for autonomous driving. They propose distinct approaches for calculating uncertainty using ensembles. Additionally, they evaluate two training strategies for 2D object detection networks: continuous training and active class weighting. Continuous training is shown to reduce training time by approximately 55% and data requirements by 15% compared to training from scratch. They further demonstrate that active learning combined with dataset balancing methods enhances data efficiency. Beyond 2D object detection, the authors implement a proof of concept using a more sophisticated neural network for 3D object detection. This approach facilitates more efficient development of object detectors tailored for the automotive industry.

Deng et al. [39] investigate how speech impacts the measurement of drivers' trust in autonomous vehicles (AVs). Seventy-five participants were randomly assigned to high-trust and low-trust groups based on different AV performance scenarios. The high-trust group experienced AVs with perfect accuracy (100%), no crashes, and received visual-auditory system messages. In contrast, the low-trust group encountered AVs with 60% accuracy, a 40% crash rate, and received visual-only system messages. During driving tasks, voice interaction was employed to collect speech data. The study successfully induced states of both trust and distrust among participants. Speech features extracted from both trust groups were utilized to train a back-propagation neural network for predicting trust levels, achieving a maximum accuracy of 90.80%. This research presents a method for reliably assessing trust in AVs using voice recognition technology.

Deep learning empowers vehicles with advanced perceptual capabilities, intelligent decision-making, and enhanced interaction with drivers, thereby advancing the efficacy and safety of Human-AI Teaming in automotive applications. Sensor fusion and other methods to correlate the readings, encapsulating a variety of sensors such as cameras, lidar, On -Board diagnostics (OBD) are often utilised.

This paper aims to elucidate the concept of HAIT, providing readers with essential insights for understanding its implications. HAIT serves to foster trust in automated automotive systems, particularly in the data sourced from vehicle sensors and RSUs. The concepts of human in, on, and over the loop are introduced, elucidating their significance. Furthermore, a comprehensive review of current academic research incorporating HAIT is conducted within the automotive domain. The main objective of this paper is to closely align IoV and AI elements, facilitating their integration into cohesive solutions.

HAIT in the automotive industry is quite important since it may assist the driver to make the correct decision on specific tasks. Moreover, the driver’s mental and physiological state may be monitored by sensors that will feed the AI model to intervene to decisions on the roard. In general, HAIT is gaining a lot of attention since, the ML models that are used do not provide black box decisons, but engage the driver with explanations to the process. In this manner the driver is more aware of the new systems of the vehicle and the interconnection with other systems, like the RSSs, and attention is maximised. Explainability which is at the heart of HAIT will provide the driver with the means to trust the AI and acknowledge the decisions. Explainable AI (XAI) does exactly that. It enhances the trust between the AI and the driver.

HAIT is recently addressed in aviation and maritime, whereby the cooperation between the human and the AI will be evident. The next generation of HAIT will evolve with the contributions in Brain Computer Interfaces (BCI)s as well as the prevalence of level 5 automation, which means a completely automated vehicle. With this in mind, the driver will play a supervisory role rather than actively engaging in the driving and will be able to communicate with the vehicle using her thoughts. Comparing to aviation HAIT is therefore considered as a promising methodology. The HAIKU project aims to include HAIT, where 6 use cases that are given, including startle effect, UAVs and COVID spreading prevention to airports. Moreover, to our knowledge, several EU projects aim to include HAIT in the maritime industry where sensor fusion and connectivity is a major issue. Automation in maritime is evident and an inclusion of HAIT is a great step towards the future ships.