Objective This article is a response to Wickens et al.'s (2019) critique of Jamieson and Skraaning (2019). Background Wickens et al. (2019) offer a five-point critique of Jamieson and Skraaning (2019) that they claim tempers the strength of our conclusions. Approach We first correct a misrepresentation in the critique and then respond to each of the criticisms. Results We preserve the strength of our skeptical conclusions about the applicability of the lumberjack model to complex work settings. Applications We continue to caution system designers about the lack of evidence supporting the lumberjack model in the context of complex work systems.
Objective Test the automation transparency design principle using a full-scope nuclear power plant simulator. Background Automation transparency is a long-held human factors design principle espousing that the responsibilities, capabilities, goals, activities, and/or effects of automation should be directly observable in the human–system interface. The anticipated benefits of transparency include more effective reliance, more appropriate trust, better understanding, and greater user satisfaction. Transparency has enjoyed a recent upsurge in use in the context of human interaction with agent-oriented automation. Method Three full-scope nuclear power plant simulator studies were conducted with licensed operating crews. In the first two experiments, transparency was implemented for interlocks, controllers, limitations, protections, and automatic programs that operate at the local component level of the plant. In the third experiment, procedure automation assumed control of plant operations and was represented in dedicated agent displays. Results Results from Experiments 1 and 2 appear to validate the human performance benefits of automation transparency for automation at the component level. However, Experiment 3 failed to replicate these findings for automation that assumed control for executing procedural actions. Conclusion Automation transparency appears to yield expected benefits for component-level automation, but caution is warranted in generalizing the design principle to agent-oriented automation. Application The automation transparency design principle may offer a powerful means of compensating for the detrimental impacts of hidden automation influence at the component level of complex systems. However, system developers should exercise caution in assuming that the principle extends to agent-oriented automation.
Objective The objective of this study was to test the predictions of the routine-failure trade-off (or lumberjack) model in a full-scope simulator study with expert operators performing realistic control tasks. Background A meta-study of degree of automation (DOA) studies concluded that DOA predicts task performance under both routine and automation failure conditions, workload, and situation awareness. Empirical support for this conclusion appears to be weak for complex work situations. Method A full-scope nuclear power plant simulator experiment was conducted in which licensed operating crews completed realistic procedure execution tasks. Dependent measures selected from the lumberjack model were collected and analyzed for systematic effects. Results Situation awareness increased with increasing DOA, which contradicts the lumberjack model. Anticipated workload and failure task performance effects were not observed. Conclusion The experimental results add further evidence challenging the applicability of the lumberjack model to complex work situations. Application Practitioners should use caution when extending the predictions of the lumberjack model based on data from simple work situations to complex work situations. Researchers should invest more resources in testing the predictive power of the lumberjack model in complex work situations.
Objective The objective of this study was to develop a machine learning classifier to infer attentional tunneling through behavioral indices. This research serves as a proof of concept for a method for inferring operator state to trigger adaptations to user interfaces. Background Adaptive user interfaces adapt their information content or configuration to changes in operating context. Operator attentional states represent a promising class of triggers for these adaptations. Behavioral indices may be a viable alternative to physiological correlates for triggering interface adaptations based on attentional state. Method A visual search task sought to induce attentional tunneling in participants. We analyzed user interaction under tunnel and non-tunnel conditions to determine whether the paradigm was successful. We then examined the performance trade-offs stemming from attentional tunnels. Finally, we developed a machine learning classifier to identify patterns of interaction characteristics associated with attentional tunnels. Results The experimental paradigm successfully induced attentional tunnels. Attentional tunnels were shown to improve performance when information appeared within them, but to hinder performance when it appeared outside. Participants were found to be more tunneled in their second tunnel trial relative to their first. Our classifier achieved a classification accuracy similar to comparable studies (area under curve = 0.74). Conclusion Behavioral indices can be used to infer attentional tunneling. There is a performance trade-off from attentional tunneling, suggesting the opportunity for adaptive systems. Application This research applies to adaptive automation aimed at managing operator attention in information-dense work domains.
In this article, we propose the application of a control-theoretic framework to human-automation interaction. The framework consists of a set of conceptual distinctions that should be respected in automation research and design. We demonstrate how existing automation interface designs in some nuclear plants fail to recognize these distinctions. We further show the value of the approach by applying it to modes of automation. The design guidelines that have been proposed in the automation literature are evaluated from the perspective of the framework. This comparison shows that the framework reveals insights that are frequently overlooked in this literature. A new set of design guidelines is introduced that builds upon the contributions of previous research and draws complementary insights from the control-theoretic framework. The result is a coherent and systematic approach to the design of human-automation-plant interfaces that will yield more concrete design criteria and a broader set of design tools. Applications of this research include improving the effectiveness of human-automation interaction design and the relevance of human-automation interaction research.
Objective The objective of this study was to develop and evaluate an adaptive user interface that could detect states of operator information overload and calibrate the amount of information on the screen. Background Machine learning can detect changes in operating context and trigger adaptive user interfaces (AUIs) to accommodate those changes. Operator attentional state represents a promising aspect of operating context for triggering AUIs. Behavioral rather than physiological indices can be used to infer operator attentional state. Method In Experiment 1, a network analysis task sought to induce states of information overload relative to a baseline. Streams of interaction data were taken from these two states and used to train machine learning classifiers. We implemented these classifiers in Experiment 2 to drive an AUI that automatically calibrated the amount of information displayed to operators. Results Experiment 1 successfully induced information overload in participants, resulting in lower accuracy, slower completion time, and higher workload. A series of machine learning classifiers detected states of information overload significantly above chance level. Experiment 2 identified four clusters of users who responded significantly differently to the AUIs. The AUIs benefited performance, completion time, and workload in three clusters. Conclusion Behavioral indices can successfully detect states of information overload and be used to effectively drive an AUI for some user groups. The success of AUIs may be contingent on characteristics of the user group. Application This research applies to domains seeking real-time assessments of user attentional or psychological state.
Objective: We examined the effects of aid reliability and reliability disclosure on human trust in and reliance on a combat identification (CID) aid. We tested whether trust acts as a mediating factor between belief in and reliance on a CID aid. Background: Individual CID systems have been developed to reduce friendly fire incidents. However, these systems cannot positively identify a target that does not have a working transponder. Therefore, when the feedback is "unknown", the target could be hostile, neutral, or friendly. Soldiers have difficulty relying on this type of imperfect automation appropriately. Method: In manual and aided conditions, 24 participants completed a simulated CID task. The reliability of the aid varied within participants, half of whom were told the aid reliability level. We used the difference in response bias values across conditions to measure automation reliance. Results: Response bias varied more appropriately with the aid reliability level when it was disclosed than when not. Trust in aid feedback correlated with belief in aid reliability and reliance on aid feedback; however, belief was not correlated with reliance. Conclusion: To engender appropriate reliance on CID systems, users should be made aware of system reliability. Application: The findings can be applied to the design of information displays for individual CID systems and soldier training.
Objective: We investigated the effects of automatic target detection (ATD) on the detection and identification performance of soldiers. Background: Prior studies have shown that highlighting targets can aid their detection. We provided soldiers with ATD that was more likely to detect one target identity than another, potentially acting as an implicit identification aid. Method: Twenty-eight soldiers detected and identified simulated human targets in an immersive virtual environment with and without ATD. Task difficulty was manipulated by varying scene illumination (day, night). The ATD identification bias was also manipulated (hostile bias, no bias, and friendly bias). We used signal detection measures to treat the identification results. Results: ATD presence improved detection performance, especially under high task difficulty (night illumination). Identification sensitivity was greater for cued than uncued targets. The identification decision criterion for cued targets varied with the ATD identification bias but showed a "sluggish beta" effect. Conclusion: ATD helps soldiers detect and identify targets. The effects of biased ATD on identification should be considered with respect to the operational context. Application: Less-than-perfectly-reliable ATD is a useful detection aid for dismounted soldiers. Disclosure of known ATD identification bias to the operator may aid the identification process.
Objective: The aim of this study was to evaluate display formats for an automated combat identification (CID) aid. Background: Verbally informing users of automation reliability improves reliance on automated CID systems. A display can provide reliability information in real time. Method: We developed and tested four visual displays that showed both target identity and system reliability information. Display type (pie, random mesh) and display proximity (integrated, separated) of identity and reliability information were manipulated. In Experiment 1, participants used the displays while engaging targets in a simulated combat environment. In Experiment 2, participants briefly viewed still scenes from the simulation. Results: Participants relied on the automation more appropriately with the integrated display than with the separated display. Participants using the random mesh display showed greater sensitivity than those using a pie chart. However, in Experiment 2, the sensitivity effects were limited to lower reliability levels. Conclusion: The integrated display format and the random mesh display were the most effective displays tested. Application: We recommend the use of the integrated format and a random mesh display to indicate identity and reliability information with an automated CID system.
Objective: The authors seek to characterize the behavioral costs of attentional switches between points in a network map and assess the efficacy of interventions intended to reduce those costs. Background: Cybersecurity network operators are tasked with determining an appropriate attentional allocation scheme given the state of the network, which requires repeated attentional switches. These attentional switches may result in temporal performance decrements, during which operators disengage from one attentional fixation point and engage with another. Method: We ran two experiments where participants identified a chain of malicious emails within a network. All interactions with the system were logged and analyzed to determine if users experienced disengagement and engagement delays. Results: Both experiments revealed significant costs from attentional switches before (i.e., disengagement) and after (i.e., engagement) participants navigated to a new area in the network. In our second experiment, we found that interventions aimed at contextualizing navigation actions lessened both disengagement and engagement delays. Conclusion: Attentional switches are detrimental to operator performance. Their costs can be reduced by design features that contextualize navigations through an interface. Application: This research can be applied to the identification and mitigation of attentional switching costs in a variety of visual search tasks. Furthermore, it demonstrates the efficacy of noninvasive behavioral monitoring for inferring cognitive events.
Objective: We determine whether an ecological interface display for nuclear power plant operations supports improved situation awareness over traditional and user-centered displays in a realistic environment. Background: Ecological interface design (EID) has not yet been fully evaluated with real operators facing realistic scenarios. Method: Ecological displays were evaluated alongside traditional and user-centered "advanced" displays in a full-scope nuclear power plant simulation. Licensed plant operators used the displays in realistic scenarios that either had procedural support or did not have procedural support. All three displays were evaluated for their ability to support operator situation awareness. Results: A significant three-way interaction effect was observed on two independent measures of situation awareness. For both measures, ecological displays improved situation awareness in scenarios that did not have procedural support, primarily in the detection phases of those scenarios. No other pronounced effects appeared across both measures. Conclusions: The observed improvement was sufficiently large to suggest that EID could improve situation awareness in situations where procedures are unavailable. However, the EID displays did not lead to improved situation awareness in the other conditions of the evaluation, and participants using these displays occasionally underperformed on single measures of situation awareness. This suggests that the approach requires further development, particularly in integrating EID with procedural support. Application: This research has important findings for the ongoing development of the EID approach, the design of industrial operator displays, and design to support situation awareness.
