AIRLINE PILOT SITUATION AWARENESS: PRESENTING A CONCEPTUAL MODEL FOR META-COGNITION, REFLECTION AND EDUCATION

The dissertation research summarized here, utilized the Grounded Theory Method to develop a conceptual model of pilot situation awareness from 223 Aviation Safety Reporting System (ASRS) narratives. The application of Latent Semantic Analysis aided the theoretical sampling of ASRS reports. A multistage model was developed involving attention, perception, interpretation, decision making, and action in support of goal-driven behavior. Narrative report coding identified several categories of situation awareness elements that pilots direct their attention to in building and maintaining situation awareness. Internal to the aircraft, flight crews directed their attention to the aircraft’s flight state and automation state. They also directed their attention to the condition of the aircraft, the functioning of the crew, and the status of the cabin. External to the aircraft, flight crews directed their attention to airport conditions, air traffic control, terrain, traffic, and weather. Pilots were also aware of the passage of time. Twelve characteristics of situation awareness were identified from narrative report coding which were subsequently compared with existing theoretical perspectives of situation awareness.


Introduction
The primary aim of the dissertation research summarized here was to provide a greater understanding of the process of situation awareness from a qualitative perspective (Irwin, 2017). This research utilized the grounded theory method to explore how airline pilots understand their situation and structure their awareness. A greater understanding of how pilots construct their awareness of a situation has implications for research by improving our understanding of pilot cognitive processes. A comprehensive model of situation awareness, grounded in the experiences of airline pilots, improves pilot education and professional practice, by providing a tool for reflection, education, and event analysis. This article presents a conceptual model of pilot situation awareness intended to provide a simple and intuitive framework for reflection and analysis, useful to practitioners. A central focus was to describe how airline pilots maintain situation awareness. This research sought to understand the situation models pilots use to maintain situation awareness, how these situation models are developed and organized, and how they impact broader crew processes.

Literature review
A literature review of pilot situation awareness suggests that several perspectives exist for understanding situation awareness. Uhlarik and Comerford (2002) described four approaches to explaining situation awareness, suggesting the perspectives of information processing models, perception/action cycle models, expert decision making models, and phenomenon description models. Similarly, Stanton et al. (2017) and Nguyen et al. (2019) suggest three perspectives of situation awareness focusing on the individual, team, and system levels. Synthesizing these concepts identifies the five situation awareness perspectives discussed in this section, including individual information processing models, perception action models, expert decision making models, shared team models, and systems phenomenon models. These perspectives have a degree of overlap and are not meant to be mutually exclusive. Instead, they are intended to provide a mechanism for understanding the underlying concepts inherent to each perspective. Endsley (1995) described situation awareness as "the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future" (p. 36). Endsley (1995) describes situation awareness as the first component in a three-step process involving situation awareness, decision making, and performance. Each step is influenced by information processing mechanisms, including memory, automaticity, knowledge, abilities, experience, and training. The system is affected by system capabilities, system design, stress, workload, complexity, and automation. Goals, objectives, preconceptions, and expectations further influence situation awareness and decision making. Wickens et al. (2003), and Wickens et al. (2008) describe a computational model of situation awareness grounded in attention and belief. The attention module is driven by the salience of events and the expectancy of seeing valuable events within the environment, while inhibited by the effort or workload required (Wickens et al., 2008). Over time events of varying salience occur with the bandwidth along each channel determining the degree of attention. The attention module provides evidence to the belief module which anchors or adjusts the model in relation to beliefs about the current state of affairs. Incoming evidence may support, refute or be neutral to the current belief, with the belief decaying over time. Fracker (1991) suggested that situation awareness involves combining new information with existing knowledge in working memory to develop a picture of the situation, including projections of the future state and appropriate courses of action. Dominguez (1994) described situation awareness as the extraction and integration of environmental information into a coherent mental picture, guiding the perception and anticipation of future events. Sarter and Woods (1995) described situation awareness as a label for a variety of cognitive processes including attention, mental simulation, mental bookkeeping, and attention to multiple threads of sub-problems (Sarter & Woods, 1995). Bell and Lyon (2000) suggested that situation awareness is knowledge in working memory concerning elements of the environment. Tenney et al. (1992) describe situation awareness as a perpetual cycle of activity where actors notice and respond to changes in the environment, anticipating the situation to focus attention on gathering relevant, available information, which updates their knowledge of the environment. They suggest pilot situation awareness should consider the pilot task being performed, the nature of information available, constraints on human processing capabilities, and the nature of expertise. Situation awareness involves concepts of memory (focus), attention (mental shifting of topics), contingency planning, knowledge, and schema (Tenney et al., 1992). Smith and Hancock (1995) expanded on this research, providing a holistic view of situation awareness, where information cues from the environment form or modify a picture of the world, directing action. From Smith and Hancock's (1995) perspective situation awareness involves building mental models of the situation from environmental cues, allowing for the anticipation of situational events, facilitated by knowledge and experience. Individuals update their model of the situation from environmental cues to verify the evolving situation matches their expectations, driving further evaluation and action (Salmon et al., 2009). Bedny and Meister (1999) presented a theory of situation awareness grounded in activity theory. From this perspective, individuals possess goals representing ideal situations, and are motivated to achieve these goals through their actions (Salmon et al., 2009). This process involves building a model of the current situation, taking action through decision making and task execution to reach the desired goal, then assessing the newly formed situation, leading to a feedback-based revisiting of the prior stages. Consistent with a naturalistic view of decision making, the knowledge and experience of the individual informs their assessment of the current situation, impacting subsequent decision making (Salmon et al., 2009). Landman et al. (2017), presented a model of startle and surprise on the flight deck grounded in perceptual cycle theory, illustrating the usefulness of frames in describing situations and determining appropriate action. Frames are thought to be created through prior experience and aid in selecting, filtering, and assigning meaning to incoming information, and are a useful construct for understanding how pilots develop and maintain their situation awareness from environmental cues (Landman et al., 2017). Frames may be active or inactive, with inactive frames becoming activated through reframing as a result of incoming environmental information.

Expert decision making models
Expert decision making describes a process of decision making grounded in situation assessment. Dekker (2015) suggests a shift in perspective from rationalism to naturalism reinvented decision making by shifting the point of focus from the decision moment to the point of situation assessment. Lipshitz (1993) identified several characteristics of naturalistic decision making, including a reliance on situation assessment, the use of mental imagery, and context dependence. Naturalistic decision making relies on expert judgment to assess the situation and to evaluate alternatives heuristically through mental simulations. Experts efficiently recognize situations in relation to previous encounters, by recognizing the most important cues, rather than the most salient ones, based on their domain specific knowledge (O'Hare & Wiggins, 2004). Rasmussen (1983) described information processing as relying on skill, rule and knowledge-based decision making behavior. Skill level behavior relies on signals from the environment resulting in automated sensori-motor responses. Rule based behavior relies on situation recognition leading to rule-based responses. Knowledge level behavior relies on mental representations constructed from environmental cues supporting decision making. These models operate at different levels of abstraction and rely on different levels of attention to the environment. Klein (1993Klein ( , 2008 described an expert process of decision making based on situation recognition, the serial evaluation of options, and mental simulation leading to action in pursuit of achievable goals. Alternatives are determined based on the recognition of critical information in the environment and the application of prior experience (Klein, 1993). Once alternatives are determined, they are serially evaluated, based on comparisons with previously experienced events in order to determine the most appropriate response (Klein, 1993). Naturalistic decision making may occur at the individual and team levels (Klein, 2008). Salas et al. (1995) described team situation awareness as "the shared understanding of a situation among team members at one point in time" (p. 131). Salas et al. (1995) suggested that situation awareness is a complex team process where team members build individual models of situation awareness, which are subsequently shared through the communication of team objectives, individual task requirements and roles, team capabilities, and team performance factors. As individual situation awareness models are shared through communication, coordination, and collaboration, a common picture of the situation is developed by the team, which allows individual team members to update their awareness of the situation through ongoing team processes (Salas et al., 1995). Endsley (1995) described team situation awareness, as individual situation awareness which is shared and overlapping. Shared situation awareness is built upon individual situation awareness as each person operates within their roles and responsibilities, and is shared through coordination, communication and duplicate information displays. Kaber and Endsley (1998) describe team situation awareness as the sum of each individual situation awareness independent of overlapping situation awareness requirements. Multiple team members with similar responsibilities may possess similar situation awareness. Team situation awareness is dependent on system complexity, the physical distribution of systems, feedback motivated by situation awareness, and the need to manage multiple tasks. Barriers to team situation awareness include poor information, a lack of information sharing, a lack of teamwork, interpersonal conflicts, and poor information system reliability. Shu and Furata (2005) built on the concept of shared and team situation awareness, focusing on mutual understanding among teams within broader distributed systems. From the perspective of Shu and Furata (2005), team situation awareness exists as parallel individual awareness that is different, but complementary and redundant. Mutual awareness includes an awareness of other team member activities, beliefs and intentions, operating at a mental level different than the domain level of individual situation awareness. Team members work cooperatively on the shared belief they will carry out specific tasks respectively, rather than on their possession of shared knowledge. Mutual understanding allows team members to have different knowledge and situation awareness, while still maintaining good team situation awareness.

Systems phenomenon models
Several authors advocate that situation awareness exists as a system level phenomenon. Artman and Garbis (1998) applied the concept of distributed cognition to situation awareness. Stanton et al. (2006) and Salmon et al. (2009) applied the concepts of distributed cognition and team situation awareness to develop the concept of distributed situation awareness. From a distributed perspective, situation awareness goes beyond a shared understanding of team members to a unique cognitive characteristic of the systems itself (Salmon et al., 2009). Stanton et al. (2006) asserted that in a distributed environment, situation awareness exists in human and non-human participants in the system, each of which has their own view or perspective. Situation awareness may be overlapping or non-overlapping, with different participants in the system representing different levels of situation awareness. Each participant in the system can compensate for losses in situation awareness by other participants through verbal or non-verbal communication.
A key distinction of distributed situation awareness surrounds shared goals, suggesting that any sharing of goals among team members stem from their need to perform required tasks, rather than for the sake of building shared situation awareness (Salmon et al., 2009). The individual goals, roles, task responsibilities, experience, and training possessed by each element of the system result in different perceptions of the situation, which are shared within the distributed environment (Salmon et al., 2009). Task factors (workload and complexity), system factors (system goals, system design, and procedures), individual factors (goals, roles, experience, training, and situation awareness requirements), and team factors (team attributes and processes) all contribute to the unique team level understanding of the situation (Salmon et al., 2009).
An analysis of the aforementioned perspectives of situation awareness suggest several shared characteristics. Multiple perspectives of situation awareness hold the view that situation awareness (a) involves active attention to the environment; (b) relies on the perception of environmental cues to reach an understanding of the situation; (c) is coordinated through communication; (d) relies on conceptual models of the situation; (e) is limited by cognitive ability, time, and workload; (f) involves judgment and prediction as precursors to decision making; (g) relies on the application of knowledge and experience; (h) is goal driven in nature. Individual perspectives suggest situation awareness involves a perpetual cycle of activity, results in naturalistic decision making, and is distributed in human and non-human participants within the system. These characteristics provide a baseline set of criteria for evaluating any proposed model of situation awareness.

Methodology
The grounded theory method was selected for this research project based on its consistency with the pragmatic worldview undergirding this research, its compatibility with naturalistic human factors theories, and the aim of this project to develop a model of pilot situation awareness grounded in pilot experience. Qualitative research methods grounded in pragmatism, such as grounded theory, provide a method of studying situation awareness in situ, while avoiding reductionism, where isolated concepts and processes are examined in isolation from their situation and context (Shalin, 1986;Dillon et al., 2000). Grounded theory is well suited for this study, given its compatibility with a socially constructed view of reality where "representation is viewed as a distributed, systems phenomenon" (Bryant, 2002, p. 25). Glaser andStrauss (1964, 1967) first developed the grounded theory method to demonstrate how theory could be discovered from the systematic collection and analysis of data. Grounded theory was intended to generate sociological theories from collected data, which could predict and explain behavior, could be verified or operationalized in future research studies, and were useful in practical situations. Glaser and Strauss (1967) intended grounded theories to be understandable by laymen within a field and applicable to a multitude of daily situations, allowing users to adapt the theory to manage the situational realities they encounter. Further, grounded theories were intended to be compatible with existing theory when applied to situations encountered in practice.

Grounded theory method
In developing the grounded theory method, Glaser and Strauss (1967) identified a process using the simultaneous collection and analysis of data and multiple levels of data coding to generate and elaborate on emerging theories. The constant comparative method is used to identify, verify, and saturate emerging categories and relationships, further elaborating emerging theories through theoretical sampling and memo-writing. Systematic data collection and analysis result in the development of substantive and formal theories as (a) emergent categories are specified, verified and elaborated upon; (b) relationships between categories are better defined; and (c) gaps in the theory are identified (Charmaz, 2014). It is widely accepted that the process of grounded theory begins with the collection and open, line-by-line coding of multiple sources of data, such as interview, observational, and archival data (Glaser & Strauss, 1967;Strauss & Corbin, 1998;Charmaz, 2014). Several variations of grounded theory have evolved since the original publication of Glaser and Strauss (1967), each differing in their process and research assumptions, approach to data coding, and use of existing theory (Charmaz, 2014). The original method suggested by Glaser and Strauss (1967) and promoted by Glaser (1978) was used consistent with the pragmatic view of reality underlying this research project and the desire to generate a theory that is trustworthy and widely accepted by professionals.

Data collection
This research analyzed 223 narrative reports from the National Aeronautics and Space Administration Aviation Safety Reporting System (ASRS) database occurring between January and December 2014. The ASRS reports analyzed in this study are provided in the Appendix by reports number (ACN), and are publicly accessible from the ASRS database available online at https://asrs.arc. nasa.gov/. The ASRS database contains a collection of deidentified, voluntary reports from aviation persons, such as pilots, mechanics, and air traffic controllers, typically submitted within 24 to 48 hours of an event. ASRS reports include a narrative where reporters are asked to describe the event and provide their perception of surrounding circumstances and cause. ASRS reports may be filtered by numerous criteria including reporter organization, reporter function, reporter provided human factors, and ASRS safety analyst assessed primary cause and contributing factors.
Data collection primarily focused on narrative reports submitted by flight crewmembers of Federal Aviation Regulation Part 121 air carriers based on the wide availability of aviation safety narrative reports, the distributed nature of the airline crew environment, and the complexity which exists in air carrier operations. Initially, one calendar year of reports between January 2014 and December 2014 were downloaded from the ASRS database to avoid biases introduced by limiting data collection to a particular season or month. To focus on airline pilot situation awareness the dataset was limited to reports submitted by flight crewmembers of Federal Aviation Regulation Part 121 air carriers, where the reporting person indicated situation awareness, confusion or distraction as an involved human factor, and ASRS safety analysts indicated human factors as a primary cause or contributing factor. To minimize the impact of inconsistencies in ASRS taxonomy coding resulting from missing, improper or ambiguous report classification, Latent Semantic Analysis (LSA) matches to reports in the dataset were added using the LSA process described by Robinson et al. (2015). Robinson et al. (2015) demonstrated the application of Latent Semantic Analysis as a technique for overcoming the limitations of taxonomy coding in aviation safety report databases. LSA is a mathematical technique for inferring relations between words within bodies of text, where individual report narratives may be characterized as a multi-dimensional vector based on multiple LSA topics statistically generated from a word frequency matrix. LSA was originally developed by Deerwester et al. (1990), and applied to aviation safety narrative reports in Robinson et al. (2015). As suggested by Robinson et al. (2015), report narratives with high LSA cosine values are thematically similar.

Data coding and analysis
From the aforementioned dataset, a total of 48 narrative reports were randomly selected, evenly distributed across each month, for coding and analysis. Report coding followed classic grounded theory, using a process of open, substantive, and theoretical coding to generate the resulting theory. Open coding involved the line by line coding of narrative reports into small distinct concepts, which were subsequently grouped into relevant categories during data analysis. Substantive coding used the constant comparative method to identify, verify, and saturate emerging categories and relationships through theoretical sampling, as individual data elements were compared within and across categories. Theoretical coding identified how the substantive codes and categories were related to the core category of situation awareness, resulting in a coherent, integrated model.
Open coding initially began with the line by line coding of 24 randomly selected narrative reports, using pilot situation awareness elements adapted from Table 3 of Endsley et al. (1998) as sensitizing concepts, allowing for additional open codes as necessary. An additional 24 narrative reports were randomly selected and coded after a determination was made that new categories of situation awareness were still emerging from the coding and analysis process. Table 1 provides examples of narrative segments coded as automation mode awareness and automation programming.
Substantive coding organized the coding terms generated during the open coding process into thematically similar categories. For example, during substantive coding the coding themes in Table 1 were combined with other similar themes under the broader category of aircraft automation. Each category was systematically analyzed in similar fashion by isolating and comparing similarly themed narrative segments from the randomly selected reports and their LSA matches to define and delineate the characteristics and themes associated with each category of the emerging taxonomy. Individual themes were compared across categories to ensure that new emerging categories were not overlooked. LSA facilitated theoretical sampling by identifying report narratives similar to the randomly sampled reports. This is analogous to snowball sampling where interview participants are asked for a referral to someone with a similar story.
Theoretical coding occurred as several broad situation awareness themes emerged during the data coding and analysis process including attention, awareness, sensation, perception, interpretation, resample, action, judgment, knowledge, experience, goal, warning system, mental model, and time. The original 48 randomly selected reports, and their LSA matches were subsequently coded using these broad themes, identifying the relationships between the resulting categories and the broader process of situation awareness. These relationships were subsequently used in developing the emerging model.
Although a literature review was conducted prior to this research, the application of existing literature was avoided during data collection and analysis. This is consistent with the grounded theory method and a desire to generate the existing model and themes from the collected data, rather than existing theory. The emergent model was compared to the existing literature post hoc.

Results
Initial narrative report coding identified a taxonomy of ten categories of situation awareness elements pilots use in building or maintaining their situation awareness. In addition, pilots were aware of the passage of time which applied to the other categories. Substantive coding and theoretical sampling expanded and saturated the resulting categories, identifying situational cues, judgments, and predictions pilots used in building their situation awareness. Theoretical coding identified twelve characteristics of situation awareness which inform the relationship of these categories to crew work activities and the work environment.
The proposed model of pilot situation awareness is provided in Figure 1. Within this model, pilots directed their attention to monitoring facets of the environment described by the emerging categories. Pilots became aware of changes to the environment through attention. The MCP was in LNAV, VNAV" (ACN 1204771); "The autopilot was engaged in NAV mode and ALT HOLD" (ACN 1184178); "the FMS ended lateral navigation and the flight guidance went into ROLL mode" (ACN 1181797); and "VOR/LOC and glideslope both turned green in the FMA and the aircraft began to descend" (ACN 1160269).
Automation programming "I intended to enter the 35R RNP Z approach into the FMC but inadvertently selected the 34R RNP Z approach without realizing it" (ACN 1228557); "I realized at that point that the FMC must have been programmed incorrectly" (ACN 1228557); and "our routing took us from MCB to MUURY…I noticed the aircraft turning beyond the proper course… navigating to MURRY" (ACN 1196412).
Sensations from the environment, including warning systems, directed pilot attention to environmental changes. As pilots perceived environmental changes, they began to develop an interpretation of what was happening. As this interpretation built, pilots resampled the environment to gather additional information, until they had a clear enough understanding of what was happening to facilitate decision making. This decision making resulted in some form of action or continued monitoring of the environment. Knowledge and experience facilitated the interpretation of the situation and decision making. Decisions were rational, naturalistic, or automatic in nature. After acting, pilots returned to monitoring the environment, consistent with achieving goals such as safety and compliance.

Situation awareness categories
This study identified several categories of environmental cues airline pilots directed their attention to in developing and maintaining situation awareness. Internal to the aircraft, flight crews directed their attention to flying the aircraft, which required an awareness of the flight state, and the aircraft's automation state. They also directed their attention to the condition of the aircraft, the functioning of the crew, and the status of the cabin. External to the aircraft, flight crews directed their attention to airport conditions, air traffic control, terrain, traffic, and weather. Pilots were also aware of the passage of time. Table 2 indicates the situation cues, judgment and predictions, and requisite knowledge and experience associated with each category which emerged from data coding and analysis. Table 3 provides examples of narrative content analyzed in this research for each situation awareness category, which are illustrative of some of the themes discussed in this section.
The flight state consisted of current information related to the orientation and direct control of the aircraft. Pilots maintained an awareness of the vertical profile or "energy state" of the aircraft, the lateral position of the aircraft, current control inputs, and the aircraft configuration. Perceptions within the flight state were derived from numerous information sources, including the aircraft flight instruments, visual information, kinesthetic cues, and vestibular sensations. Perceptions of aircraft angle of attack, aircraft configuration, engine thrust setting, and control deflection also provided useful information for assessing the flight state.
The automation state consisted of an awareness of the current condition of the autopilot, auto-throttles, and flight management system (FMS) programming. Pilots continually monitored the present state of automation to ensure that the aircraft operated as intended. The automation state included an awareness of the current programming of the aircraft automation, including the level of automation currently being used, the autopilot modes armed or selected, and the current FMS programming. Pilots evaluated the appropriate level of automation based on the cost of using manual control versus the consequences of not using aircraft automation. They also considered the perceived usefulness and known limitations of aircraft automation. These decisions were based in part on their previous automation training and prior experience using automation.
The aircraft state consisted of an awareness of the status and proper operation of all aircraft systems. This   "Shooting the visual to 16L, we encountered virga and light turbulence. We were stabilized by 1,000 FT AGL but not long after that the wind shifted dramatically, it felt like we were sinking, and the pilot flying added a large amount of thrust to arrest the sink rate. There were no windshear warnings from the plane; only the airspeed drop and thrust lever angle. Just as that happened, I noticed the airspeed increase into the red band, up to 170 KTS (5-KT overspeed) and called it out. Ref and target were 131 and 151 for a flaps 30 approach due to gusty conditions. Placard speed is 165. Pilot flying said "bring them up" which I assumed meant to bring the flaps up before the overspeed got worse and we caused damage, and also assumed this meant we would go-around. I brought the flaps to 15 and, after a few seconds, airspeed started to come back down. We were still on glideslope with a descent rate of less than 1,000 FPM. " ACN 1175877 Aircraft automation "Autoflight was on and descent was on profile until ALT* mode entered. I suggested that pilot flying hand fly the level off since we were now trending high. Pilot flying instead deployed speedbrakes. Only after I told him to hand fly did he do so and we still passed FUBBR at FL274, 400 FT high. Two recurring items, again illustrated here. ALT* is an altitude capture mode. Deploying speed brakes will not affect the level off profile.
It will alter your current speed and eventually autothrust will respond but in ALT* you will continue on the level off profile calculated by the autoflight system. If you were going to miss your altitude before you took action, after only deploying speed brakes you will still miss your crossing altitude. Somehow this is not taught or not absorbed by pilots who haven't seen enough of this to understand… And this leads to the second recurring issue this event illustrates, tremendous reluctance to simply disconnect the autopilot and make the airplane do as you would like. Somewhere within the span of my career we have gone from a distrust of autoflight to a distrust of hand flying. " ACN 1156516 Aircraft systems "Hand-flying the departure I noticed controls were stiff after takeoff; something just did not seem right. As soon as we sped up through approximately 220 knots I noticed that the trim switches on my yoke were not functioning. I verified with the First Officer when the last time the trim was seen working normally. The trim was only moved one time from preflight to takeoff, and it was moved less than half a unit. This was done before we left ground metering and the temperature was not even 80 degrees outside, so an overheat seemed very unlikely. While discussing (very quickly I might add) the above I was unable to apply enough forward pressure to push the nose over therefore I disconnected the autothrottles and manually reduced the thrust to about the twelve o' clock position of the throttles to get the nose down and not exacerbate the situation by speeding up any more (even though that is not in our DC-9 book I remember that memory item from the Boeing 737 "Do not increase airspeed"). I asked him if his trim switch would work. His would work but not in the direction we needed. His switch would only trim the nose up--the opposite way of where we needed to trim at the time. " ACN 1178044 Crew functioning "FO was pilot flying, and he had only been 2 weeks out of training… FO appeared to have little or no concept of how to fly the aircraft. I had to intervene by taking control of the aircraft twice during climb to prevent altitude deviations. On descent, the FO became completely confused about how to control the aircraft speed... I have flown with several new-hires in recent months. The FO's performance in this situation is pretty typical of what I have seen. This leads me to believe the problem is inadequate training from the [training center], as there is no way that a dozen pilots are all deficient without some core cause out of their control. Other training problems I have seen include: suction feeding the engines on the ground with no boost pumps. When I asked what they were doing they reply that they were told to do that to shed electrical load on the generators. FMS programming is non-existent. Most FOs ask me to load everything. I later learn that they ask because they received insufficient training on the "box. " No altitude planning. Inadequate autopilot knowledge. " ACN 1197205 Cabin status Cabin Status: "At the gate in LAX just prior to departure I noticed there was the Gate Agent and two Customer Service Managers at forward entry door. I went out to see how the boarding process was going and the CSMs were in the process of shutting the aircraft entry door. I stopped them and told them that there numerous overhead bins open; passengers were up with their bags in hand. I told them that the aircraft was not ready for the door to be closed…I asked my #1 Flight Attendant if she had been pressured by the CSMs and she said that she was being unduly pressured by them to shut the aircraft entry door before all the overheads were closed and bags were stowed. " ACN 1174503 ATC environment "RJAA ILS 16R (Y) and ILS 16R (Z). Since the advent of multiple ILS approaches to the same Runway (YZ) with multiple transitions, the approach IN USE (often) is NOT line selectable and ATC is NOT cooperative to our constrained system. RJAA ATC is NOT flexible and cooperative. The current ATS in the RJAA TRACON has increased the operational complexity for our operations TODAY and increased the potential for an unintentional aircrew ERROR in the TRACON's airspace. I will present the situation that confronted us. Advised ILS 16R (Y) was the approach in use we requested the (Z) 16R, feeder routings are different STARS feed the various (Y) or (Z) approaches, LOC intercept ALTS are different and the missed approach is very complex in its routing and we were concerned that the (Y) and (Z) missed approach tracks might differ. Our request for the (Z) was DENIED, we used the ILS DME to identify the (Y) fixes on the LOC. " ACN 1176560 included an awareness of the current maintenance status of the aircraft and any deferred maintenance items. In assessing the aircraft state, flight crews evaluated the ability of the aircraft to continue operating safely, including the potential impact of system failures. In judging the reliability of the aircraft, flight crews formed opinions regarding maintenance processes and personnel. Flight crews were also aware of the current aircraft loading, including the location of dangerous materials. Awareness of aircraft loading aided in anticipating how the aircraft should perform under normal conditions and to ensuring compliance with published aircraft limitations.
Crew functioning consisted of an awareness of the functioning of the flight crew, including their strengths and limitations, and the reliability of personnel they interacted with. Flight crews were aware of their respective crew roles, the adequacy of their training, knowledge, and experience, and the reliability of standard operating procedures. They also evaluated the training, knowledge, experience, and reliability of others. In making judgments about the functioning of the crew, they were aware of the degree of workload, attention, and situation awareness they possessed, and the quality of their communication, coordination, and decision making. Finally, Airport environment "With approximately 20-30 minutes of flight time remaining we received MU's, with the lowest being 56 averaging 61, leaving us no room for concern ...We CRMed the situation extensively, and planned to treat the airport as a special 'winter ops' airport due to the 6,500 foot runway, unfamiliar station, and night operations. We also planned to treat the runway as it was contaminated, even though we had nothing to suggest it was anything different than wet. … At approximately 40 FT it was evident the runway was worse than wet, but thrust was at idle and we were committed to landing. … The mains touched on/near the 1,000 foot markers, and the nose touched one second later. I immediately applied max (pedals to the floor) braking and max reverse, and full forward control column pressure. It was immediately evident the airplane was sliding, I didn't really feel the antiskid cycling because the tires weren't gripping any pavement. As the airplane began to slow (below 40 KTS), the nose drifted slightly right of center, and I was unable to correct. I utilized max reverse until the airplane came to a complete stop, with approximately 750 FT of runway remaining. Braking action poor. The runways/taxiways/ramp all had at least two inches of slush covering every surface. Not wet as reported, and there is no possible way the MU's were accurate, or even close. " ACN 1149826 Terrain "While descending for ILS 34 AVL at approximately 6,500 MSL and starting the turn to line up with final, we received a single terrain GPWS alert, 'TERRAIN. ' I initiated the terrain avoidance procedure and immediately the warning ended. The runway was in sight and we could see no other threats so the descent was continued and a normal approach and landing was made. The wind was blowing approximately 25-35 KTS over the hills creating a rough ride and some strong up and downdrafts. Descending into a rising terrain airport and high winds can create conditions that cause the GPWS to activate. Even though we were confident we knew where we were and why the alert happened we initiated the terrain avoidance procedure. " ACN 1145583 Traffic "At about ten minutes from landing, we were level at 4,000 feet when I noticed traffic on our TCAS at our one o' clock, six miles out, and four hundred feet above us. I was the pilot flying, and I asked the First Officer if he could see the traffic. I then noticed that the traffic was descending and flying towards us. I picked up the traffic visually at about three miles away which confirmed that it was on a possible collision course with us as it continued to descend into our altitude. By this time we had received a traffic alert from the TCAS. I disconnected the autopilot to begin an evasive maneuver. However, Approach Control mentioned the traffic and issued us a descent to 2,500 feet. I initially began the descent in hopes that we could avoid the traffic via ATC instructions. However, at that very moment, we received a TCAS resolution advisory (RA) to climb. I brought the throttles to almost maximum thrust and executed an aggressive climbing left turn (I made a decision to not only climb but to also climb away from the approaching aircraft). I felt that the bank and the aggressiveness of the maneuver were necessary to avoid the traffic. " ACN 1155642 Weather "Vectored for the approach in VMC. Light to moderate shower over the approach end of the runway extending about a mile from the runway. Radar analysis indicated the precipitation shaft top was probably about 20,000 feet. The runway could easily be seen initially and the rain was light to moderate across the shower. We initially saw no indications of windshear such as a "foot" to the shower or dust rings. About three miles from the runway, 1 mile from the edge of the shower, the intensity of the rain began to increase rapidly. The runway was disappearing and two lightning bolts hit the ground as we entered the shower. Entering the shower visibility dropped to zero. Almost immediately we got a windshear warning at about 400 feet agl and initiated a go around. " ACN 1200906 Passage of time "Both the CA and I felt the dispatcher was in over his head and was struggling to stay ahead of his required work. I believe there was pressure on him to launch and complete this flight. Other operations were also experiencing delays. …All people involved in working this flight felt pressure to complete the mission and I believe this lead to launching a flight [that] was not meant to make it… I wish our dispatcher and us had come up with a plan to delay the flight until the weather would have moved out. However, I was up against a duty issue (we had 40 minutes to launch) I think this further pushed the idea to get the aircraft airborne which was a poor choice. I don't believe it will ever be necessary to launch a flight to 'beat the weather. '" ACN 1228234 End of Table 3 they were aware of the presence of ambiguities or missing information.
The cabin status included passenger cabin related safety and compliance concerns, cabin environmental conditions, and available cabin resources. Flight crews attempted to identify and mitigate threats to cabin safety, attend to passenger needs, and work to promote passenger comfort and customer satisfaction. Communication with flight attendants was an essential component of maintaining an awareness of the passenger cabin, given the limited visibility of the cabin to flight crews. Communication also served to inform flight attendants of threats external to the aircraft such as turbulence.
The airport environment consisted of the current airport conditions, including the active runways and instrument approaches in use, the impact of current weather conditions on ground and flight operations, and the condition of ramp areas, runways, and taxiways. It also included an awareness of the airport facilities and services available, and any hazards in the airport vicinity.
The ATC environment consisted of an awareness of the ATC cleared route. It also included an awareness of ATC capabilities and limitations, an assessment of the reliability of ATC, and an understanding of pilot and air traffic controller needs, roles, and responsibilities. Flight crews worked to anticipate any potential ATC delays or diversions that might impact the safety of flight.
Terrain awareness consisted of an awareness of the location of terrain, terrain height in relation to the aircraft, and closure rate. This was accomplished visually, using aircraft displays, or by comparing aircraft position with navigational charts. Flight crews maintained safe separation from terrain by maintaining an awareness of and compliance with minimum altitudes. Aircraft warning systems, such as ground proximity warning systems assisted crews in avoiding terrain. Crews applied an understanding of how these warning systems function to identify or avoid false or nuisance warnings.
Traffic awareness consisted of knowing the location of threatening traffic and the relative severity of the threat. The location of traffic was inclusive of knowing an aircraft's assigned sequence and traffic to follow. The severity of traffic threats was evaluated based on the proximity of the traffic, closure rate, and projected separation. Flight crews assessed the reliability of traffic information, while attempting to anticipate the actions of other aircraft. Traffic awareness was facilitated through visual avoidance, Traffic Collision and Avoidance System (TCAS) information, and ATC traffic information. Traffic information from TCAS and ATC was limited by aircraft equipment installations.
Weather awareness consisted of knowledge of current and forecast weather conditions. This assessment considered the crew's confidence in weather forecasts, the likelihood of severe weather, and the potential severity of weather conditions likely to be encountered. Similarly, flight crews maintained awareness of the location of hazardous weather in relation to their route, anticipating the movement of weather conditions and potential weather changes. Flight crews also considered alternatives available to avoid or mitigate weather conditions impacting their flight. As with terrain and traffic warnings, aircraft and ATC warning systems played a distributed role in flight crews maintaining awareness of hazardous weather conditions. Finally, flight crews were aware of the passage of time. This awareness was manifest in crews describing their response time to events, perceiving how rapidly events occurred, and comparing the time existing between events with the expected or normal timing of events. This awareness aided crews in evaluating the perceived severity of events, detecting aircraft anomalies, managing time constraints, and adhering to crew flight and duty time limitations. Crews balanced time constraints caused by flight and duty time regulations, passenger considerations, and operational demands for an on-time departure against safety and compliance concerns.

Situation awareness characteristics
This study identified several characteristics associated with the process of situation awareness as presented in Figure  1. Situation awareness (a) involved an active process of attention to the environment; (b) involved activity in response to actively and passively acquired information; (c) was supported by aircraft warning systems; (d) relied on the perception of environmental cues to develop an understanding of the situation; (e) was coordinated among flight crews through communication; (f) relied on conceptual models of the situation; (g) was limited by cognitive abilities, available time, and workload; (h) included judgment and prediction as precursors to decision making; (i) involved the application of knowledge and experience in interpreting environmental cues and making judgments and predictions; (j) led to automatic, naturalistic, or rationalistic decision making strategies; (k) resulted in action, continued monitoring or a reassessment of the situation; (l) was goal driven in nature.
Narrative reports suggested that attention is a central component of situation awareness. Figure 2 provides a word cloud derived from pilot narrative elements coded as attention. Words such as "noticed", "reported", and "encountered" indicate the process of attention, with words such as "fuel", airspeed", and "autopilot" indicating the object of that attention. Attention to environmental cues aided crews in building and maintaining a proper awareness of the situation. Incomplete information led flight crews to an improper assessment of the situation, sometimes resulting in improper action or a delayed response. Situation awareness was formed through information actively sought out from the environment through attention, or passive information received through environmental sensations such as kinesthetic and vestibular cues which directed pilot attention to environmental changes.
Narrative reports provided examples of pilot attention being directed to the environment in three distinct contexts. First, crews queried the environment to build a clearer understanding of what was happening prior to decision making. Second, once a clear enough understanding of the situation emerged, some crews elected to continue monitoring the situation for cues indicating a changing or worsening of the situation, while others elected to take action to remedy the situation. Third, flight crews reassessed the environment after taking action to evaluate the impact of those actions on the newly formed situation, or to identify other situations that undermine primary goals such as safety and compliance.
Narrative reports demonstrated that aircraft warning systems played a distributed role in situation awareness, where visual and aural warnings aided crews in maintaining situation awareness, especially when crew attention was directed elsewhere. Electronic engine and aircraft system displays further aided crews in identifying systems related malfunctions. Report narratives suggested that attention to elements of the environment was context dependent. For example, crew attention to terrain during low threat cruise flight was negligible. As crews descended to lower altitudes their terrain awareness was reactivated, especially in areas where terrain posed a greater threat.
Narrative reports suggested that crews used the perception of environmental cues and communication to facilitate reaching a collective understanding of the situation. This understanding was directed to past, present, or future events. The process of interpretation sought to answer the fundamental questions of "what is happening?", "what has happened?", or "what will happen?". Flight crews used discussion to coordinate their understanding of the situation. As flight crews sought to reach a collective understanding of the situation, they assimilated multiple environmental cues related to the problem at hand. Figure 3 provides a word cloud representing narrative phrases related to perception and comprehension. Terms such as "know", "felt", "seemed" "thought" and "believed" suggest a comprehension or understanding of the situation. In most cases the understanding developed by crews was conceptual rather than visual. This suggests that in most cases a robust model of the situation is not required, instead a more efficient, less visual representation of the situation suffices.
Distributed situation awareness was apparent in emergency situations where crews distributed responsibilities for situation awareness. One crew distributed the workload during a fuel emergency with the flying pilot taking responsibility for monitoring the flight state, automation state, airport environment, and air traffic control. As flying pilot, they were also responsible for monitoring for terrain, traffic and weather threats, although they were likely to be partly dependent on aircraft warning systems due to workload. At the same time, the non-flying pilot took responsibility for the aircraft systems state and cabin status in running checklists and coordinating with flight attendants and dispatch.
Narrative reports further suggested situation awareness involved judgment and prediction as a precursor to decision making. Knowledge and experience aided in the interpretation of environmental cues, and supported judgments and predictions required to develop and maintain situation awareness. Knowledge and experience facilitated situation awareness by aiding crews in determining where to direct their attention, in making meaningful interpretations of the situation, and determining appropriate action. In this context crews assessed what was currently happening, leading to judgments and predictions about what might happen given their interpretation of environmental cues.
Within the flight state pilots assessed environmental cues to conceptualize what the aircraft was currently doing and where it would be at a future time. Their understanding of "what is happening" led to judgments such as whether the approach was stable or the energy state adequate. Similarly, the monitoring of aircraft automation led to judgments evaluating the cost of using automation versus the consequences of not using automation, including perceptions regarding the usability and reliability of aircraft automation systems. Judgment and prediction was illustrated by other themes in this study, including evaluating of the impact of systems failures, the reliability of maintenance personnel and practices, the reliability of other employees, and predicting changes to weather conditions, traffic and airport operating environments.
Narrative reports provided examples of situation awareness, serving as a precursor to decision making, re- sulting in automatic, naturalistic, or rationalistic decision making strategies. Flight crews engaged in situation assessment until a clear enough understanding of the situation formed to support action. That action was manifest in either continued monitoring or action intended to alter the environment. In some circumstances, decision making was automated by standard operating procedures, training, or rules of thumb, such as the decision to execute a wind shear escape maneuver following an aural warning. In the absence of a predetermined decision strategy grounded in procedure or training, some crews responded with a naturalistic decision making strategy (Simpson, 2001;Klein, 2008). Finally, given adequate time and cognitive resources, some decision making followed rationalistic processes described by traditional decision making models. Figure 4 illustrates narrative elements coded as flight crew actions. Finally, narrative reports suggested that situation awareness was goal driven in nature, as pilots monitored their environment to ensure continued safety and compliance until the ultimate goal of a safely completed flight was achieved. Crews established certain reasonable parameters for compliance, intervening when the flight departed from those parameters. As indicated by Figure 5, crews were motivated by safety concerns, compliance with regulations, standard operating procedures, and air traffic control requirements. Secondary goals included compliance with the schedule, passenger comfort, and efficiency. This study suggests that pilot attention to the environment shifted in response to changing circumstances consistent with attaining goals. Typically, flying the aircraft and compliance with ATC instructions had a high priority. Several events involving conflicting traffic, terrain warnings, or wind shear encounters resulted in higher attention to the area of concern, until the conflict was satisfactorily resolved, after which attention returned to the flight state and compliance with ATC instructions.

Discussion
The categories and characteristics of the proposed model of situation awareness, developed through the application of grounded theory to pilot narrative reports, are largely consistent with existing theory. The presented model describes characteristics of situation awareness that mirror the characteristics identified by the literature review. Situation awareness is described by the presented model as an active process of attention, relying on active and passive information, where the perception of environmental cues are used to develop an understanding of the situation. This understanding, grounded in knowledge and experience, is used to inform judgments and predictions leading to automatic, naturalistic or rationalistic decision making in pursuit of desired goals. Situation awareness is a distributed process, supported by aircraft warning systems, where conceptual models of the situation are coordinated through communication. Situation awareness is limited by cognitive ability, available time and workload. These common themes illustrate the general compatibility of the proposed model with the situation awareness perspectives described by the literature review. Differences between the existing perspectives, while important from a theoretical standpoint, may be beyond the scope of practitioner application to meta-cognition, reflection and education.
The situation awareness categories identified in this research provide a framework for describing how pilot attention is divided across various elements of the situation, including the role of situation cues in forming situation awareness and driving pilot attention. Several studies have examined situation awareness in the context of attention divided across active and inactive situation awareness frames. Salmon et al. (2016) reviewed the accident involving Air France 447. In their analysis, they provide detailed situation awareness networks describing the state of situation awareness at various stages of the event. The elements of these highly detailed networks are consistent with the categories identified in this research. As an example, their "phase two" network contained elements which could be categorized within the flight and energy, aircraft automation, aircraft systems, and weather categories.
Similarly, Froger et al. (2018) examined the relationship of pilot attention to flight awareness and traffic awareness by examining heads up and heads down activity. In the context of the model presented here, the study examined the relationship between the flight and energy, and traffic categories of situation awareness. Behrend and Dehais (2020) studied differences in pilot roles and goaround decision making during the approach and landing phase. In the context of the model presented here, the pilot decision to go around was influenced by considerations including the aircraft, automation, ATC, and weather categories.
The model presented by this research provides a simplified framework for examining conflicting information and priorities across situation awareness categories. One particular narrative report illustrates shifting pilot attention to varying situation elements in contemplating the decision to go around. In the context of the presented model, information from the flight and energy, aircraft system, terrain, traffic, and weather categories are illustrated.
"[First approach] On descent we received reports of moderate turbulence and windshear. We briefed the RNAV approach to Runway 24 as this was currently the runway in use. Surface wind was variable 280-300 at 30 gusting to 40...We began to get airspeed fluctuations of plus and minus 20 KTS. The aircraft was very difficult to control. Around three miles from Runway 24 we had an airspeed increase of more than 20 KTS and got the high-speed cue. I was already pulling back the thrust to correct when we got the high-speed cue. As I reduced thrust even more another wind gust hit us and we then received the stick shaker. We immediately executed a go-around… [Third approach] By this time the wind had mostly shifted to 300 at 35 gusting to 40. Considering fuel on board we felt it was safe to try this approach. We felt that this was safe due to the fact that there are not as many peeks on this approach and a Dash 8 was able to land on that Runway 5 minutes earlier. We began the ILS to 34 and it was a much better ride until short final. On short final we received an amber alert for windshear. I felt that it was safe to continue the approach because these gusts were very short. Around 20 FT above the runway we received a red windshear warning. It took me about a second to think about our state. At this point, about 10 FT off the ground, I felt that it was safer to continue the landing since we were so close to the runway instead of doing a low energy go-around with windshear present. We landed safely with no further incident." (ACN 1156858) Consistent with the initial aim of this research, a conceptual model of pilot situation awareness was developed. Grounded theory studies, intended to inductively generate theory from data, provide a starting point for research. Glaser and Strauss (1967) suggested that grounded theories should be evaluated in light of several criteria including their ability to be understood and applied by industry practitioners, their ability to predict and explain behavior in other contexts, their consistency with existing theory, and their described process for collecting, coding, analyzing, and presenting data. This research used a process of data collection, coding, and data analysis consistent with the grounded theory method. The presented model of pilot situation awareness demonstrates common character-istics with other models of situation awareness existing in the literature. This research sought to develop a model of situation awareness useful to practitioners. Ultimately that evaluation will be decided by the degree of acceptance of this model by industry professionals. Similarly, the ability of this model to predict and explain behavior in other contexts will be supported or refuted by future studies. More detailed studies within and across each of the presented situation awareness categories will further understanding of the processes pilots use to assess their situation.

Conclusions
This research sought to identify the situation models pilots use to maintain situation awareness and to describe how these models are developed, organized and relate to broader crew processes. Grounded theory was considered methodologically appropriate in its ability to develop theories grounded in practitioner experiences, useful to professionals, and applicable to the problems of professional practice. As described above, pilots directed their attention to several categories of environmental cues in maintaining situation awareness. Internal to the aircraft, flight crews directed their attention to flying the aircraft, which required an awareness of the flight state, and the aircraft's automation state. They also directed their attention to the condition of the aircraft, the functioning of the crew, and the status of the cabin. External to the aircraft, flight crews directed their attention to airport conditions, air traffic control, terrain, traffic, and weather. Pilots were also aware of the passage of time as they interpreted events and complied with time constraints.
As demonstrated by the narrative reports analyzed by this research and consistent with the existing literature, pilot situation awareness involves complex processes, where pilots actively interact with their environment, as they direct attention to environmental cues, respond to sensory inputs, and actively work to adapt the environment consistent with their goals and expectations. Situation awareness involves communication, judgment and prediction to reach a collective understanding of past, present or future events. Situation awareness is a precursor to decision making, monitoring, and action. Situation awareness is a bounded process, where flight crews are limited in their ability to remain aware of the entire situation, instead focusing attention on aspects of the situation perceived to be relevant. Situation awareness seeks to answer the fundamental question of "what has happened", "what is happening", or "what will happen", with the aim of attaining goals such as safety and compliance. Knowledge and experience facilitate the process of attention, perception, and decision making consistent with naturalistic theories of decision making.
This project primarily studied airline pilots operating in air carrier operations. The study of airline pilots provides a complex crew environment, which includes a diversity of weather and operational environments utilizing sophisticated transport category aircraft. Aviation includes a multitude of missions and operational environments. While large similarities may exist between the situation awareness and human factors processes of airline pilots in comparison to other pilot groups, differences also exist. The models and structures other pilots use to understand and describe their environment may be different than those of airline pilots. The generalizability of this study to other contexts should be explored.
Finally, study is needed to explore the potential application of this model to event analysis and education. A greater understanding of pilot cognitive processes, such as situation awareness, informs professional reflection and education. Improved understanding of pilot situation awareness may lead to changes in aviation education and event analysis. Study of this model in those contexts is necessary to evaluate its usefulness and effectiveness in meeting the needs of industry practitioners.

Author contributions
William Irwin was the primary researcher. Contributions included the conceptual development of the study, development of the research design and methods, preparation of the IRB protocol, the literature review, data collection, data analysis, and manuscript preparation.
Terrence Kelly served as the dissertation chairperson. Contributions included the oversight of the conceptual development, research design, research methods and the IRB protocol, in addition to critical reviews of manuscript drafts.