SEARCH · Search NASA

Results for “scoring system”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

All sources NASA NTRS DOE OSTI

NASA NTRS2006

Airborne Turbulence Detection System Certification Tool Set

A methodology and a corresponding set of simulation tools for testing and evaluating turbulence detection sensors has been presented. The tool set is available to industry and the FAA for certification of radar based airborne turbulence detection systems. The tool set consists of simulated data sets representing convectively induced turbulence, an airborne radar simulation system, hazard tables to convert the radar observable to an aircraft load, documentation, a hazard metric "truth" algorithm, and criteria for scoring the predictions. Analysis indicates that flight test data supports spatial buffers for scoring detections. Also, flight data and demonstrations with the tool set suggest the need for a magnitude buffer.

Hamilton, David W.↗

NASA NTRS2014

Space Launch System Upper Stage Technology Assessment

The Space Launch System (SLS) is envisioned as a heavy-lift vehicle that will provide the foundation for future beyond low-Earth orbit (LEO) exploration missions. Previous studies have been performed to determine the optimal configuration for the SLS and the applicability of commercial off-the-shelf in-space stages for Earth departure. Currently NASA is analyzing the concept of a Dual Use Upper Stage (DUUS) that will provide LEO insertion and Earth departure burns. This paper will explore candidate in-space stages based on the DUUS design for a wide range of beyond LEO missions. Mission payloads will range from small robotic systems up to human systems with deep space habitats and landers. Mission destinations will include cislunar space, Mars, Jupiter, and Saturn. Given these wide-ranging mission objectives, a vehicle-sizing tool has been developed to determine the size of an Earth departure stage based on the mission objectives. The tool calculates masses for all the major subsystems of the vehicle including propellant loads, avionics, power, engines, main propulsion system components, tanks, pressurization system and gases, primary structural elements, and secondary structural elements. The tool uses an iterative sizing algorithm to determine the resulting mass of the stage. Any input into one of the subsystem sizing routines or the mission parameters can be treated as a parametric sweep or as a distribution for use in Monte Carlo analysis. Taking these factors together allows for multi-variable, coupled analysis runs. To increase confidence in the tool, the results have been verified against two point-of-departure designs of the DUUS. The tool has also been verified against Apollo moon mission elements and other manned space systems. This paper will focus on trading key propulsion technologies including chemical, Nuclear Thermal Propulsion (NTP), and Solar Electric Propulsion (SEP). All of the key performance inputs and relationships will be presented and discussed in light of the various missions. For each mission there are several trajectory options and each will be discussed in terms of delta-v required and transit duration. Each propulsion system will be modeled, sized, and judged based on their applicability to the whole range of beyond LEO missions. Criteria for scoring will include the resulting dry mass of the stage, resulting propellant required, time to destination, and an assessment of key enabling technologies. In addition to the larger metrics, this paper will present the results of several coupled sensitivity studies. The ultimate goals of these tools and studies are to provide NASA with the most mass-, technology-, and cost-effective in-space stage for its future exploration missions.

Holladay, Jon↗

NASA NTRS2015

A Heuristic Approach to Correlating ERAM Flight Data from Twenty Centers

Among its many other functions, the Federal Aviation Administration’s En Route Automation Modernization (ERAM) provides external systems with real-time air traffic data for flights in enroute airspace in the National Airspace System. It replaced the En Route Host computer and backup system used at 20 FAA Air Route Traffic Control Centers (Centers) nationwide. Among the new features of ERAM, its output data stream of flight plan and track data includes a unique identifier for a flight originating in any one of the 20 ERAM Centers. The unique identifier, called the Global Unique Flight Identifier (GUFI), is persistent across all the Centers that track the flight. However, certain factors make it difficult to correlate data using the GUFI. First, the value of the GUFI is only unique within a time window of seven days. Second, the GUFI is attached only to flight-plan related data messages. Finally, track positions reported by ERAM do not reference the GUFI. In order to correlate historical as well as real time flight-plan and position related ERAM data, an efficient, heuristic approach was developed, and a prototype was developed. The approach showed that the processing speed, through parallel processing, is sufficient to correlate ERAM data in real-time. As described in this paper, when there are multiple track positions reported from multiple Centers within a few seconds, each position is assigned with a weighted score to indicate the quality of the position relative to its last know position. The weighted score can be used to eliminate potentially duplicate track positions. The approach is database-agnostic, and can be implemented in a Big Data system such as an Apache Hadoop system, as well as in traditional database systems.

Correlating ERAM Flight Data↗

NASA NTRSDate not supplied

Performance Risk Model (PRisM) Proof-of-Concept: An Operational Decision Support Tool to Predict Crew Performance in Space from Available Performance Tests

The Crew Health and Performance-Probabilistic Risk Assessment (CHP-PRA) team at NASA Glenn Research Center has developed a range of tools to evaluate astronaut health during spaceflight and to optimize the medical set required for missions. Among these, the Performance Risk Model (PRisM) represents a novel advancement, extending CHP-PRA’s focus beyond medical systems into the domain of human performance. Such tool could be pivotal in optimizing astronaut capabilities during space travel, thereby enhancing overall mission success. PRisM leverages data from well-established performance assessments conducted during previous crewed space missions and Earth analogs to strategically predict outcomes for planned tasks, even when direct performance testing has not been conducted for those specific tasks. To evaluate performance, PRisM references the various metrics outlined in NASA-STD-3001 and employs a methodology to integrate different performance scales. This framework analyzes the contribution of various human system task categories (HSTCs) to task execution and compares these contributions to the HSTCs’ involvement in other known tests. The model further employs a Monte Carlo simulation to sample performance scores from their distribution in operationally relevant tests such as those in Mulavara et al. (2018) and, by leveraging similarities in HSTC involvement, transfers this knowledge to actual mission tasks, such as those outlined in the "Human Exploration of Mars: Preliminary List of Crew Tasks”. The current PRisM proof-of-concept includes analyses of the impact of exercise and specific medications on astronauts’ performance, with provisions to incorporate additional countermeasures as data becomes available. Furthermore, the tool is customizable to include any system necessary to fully encompass the domain of human systems and can be adapted to evaluate performance for any spaceflight activities as requested by operational stakeholders. PRisM has the potential to assist the Human Research Program in exploring the capabilities trade space for optimized crew performance.

performance modeling↗

NASA NTRS2009

An Anomaly Correlation Skill Score for the Evaluation of the Performance of Hyperspectral Infrared Sounders

With the availability of very accurate forecasts, the metric of accuracy alone for the evaluation of the performance of a retrieval system can produce misleading results. A useful characterization of the quality of a retrieval system and its potential to contribute to an improved weather forecast is its skill, which we define as the ability to make retrievals of geophysical parameters which are closer to the truth than the six hour forecast, when the truth differs significantly from the forecast. We illustrate retrieval skill using one day of AMSU and AIRS data with three different retrieval algorithms, which result in retrievals for more than 90% of the potential retrievals under clear and cloudy conditions. Two of the three algorithms have better than 1 K rms "RAOB quality" accuracy on the troposphere, but only one has skill between 900 and 100 mb. AIRS was launched on the EOS Aqua spacecraft in May 2002 into a 705 km polar sun-synchronous orbit with accurately maintained 1:30 PM ascending node. Essentially uninterrupted data are freely available since September 2002.

retrieval↗

NASA NTRS1969

Improved perceptual-motor performance measurement system

Battery of tests determines the primary dimensions of perceptual-motor performance. Eighteen basic measures range from simple tests to sophisticated electronic devices. Improved system has one unit for the subject containing test display and response elements, and one for the experimenter where test setups, programming, and scoring are accomplished.

Parker, J. F., Jr.↗

NASA NTRS2015

The Ocean Colour Climate Change Initiative: I. A Methodology for Assessing Atmospheric Correction Processors Based on In-Situ Measurements

The Ocean Colour Climate Change Initiative intends to provide a long-term time series of ocean colour data and investigate the detectable climate impact. A reliable and stable atmospheric correction procedure is the basis for ocean colour products of the necessary high quality. In order to guarantee an objective selection from a set of four atmospheric correction processors, the common validation strategy of comparisons between in-situ and satellite derived water leaving reflectance spectra, is extended by a ranking system. In principle, the statistical parameters such as root mean square error, bias, etc. and measures of goodness of fit, are transformed into relative scores, which evaluate the relationship of quality dependent on the algorithms under study. The sensitivity of these scores to the selected database has been assessed by a bootstrapping exercise, which allows identification of the uncertainty in the scoring results. Although the presented methodology is intended to be used in an algorithm selection process, this paper focusses on the scope of the methodology rather than the properties of the individual processors.

Climate Change↗

NASA NTRSDate not supplied

LLMs and GenAI Tools to Depict Contributions of Human Systems to Spaceflight Tasks Execution

Recent advancements in Artificial Intelligence and Machine Learning (AI/ML) technologies, particularly Large Language Models (LLMs) capable of sophisticated syntax analysis, offer substantial potential in automating complex processes, thereby saving time and human resources. This study explores the development of an LLM-driven model designed to analyze and categorize a diverse set of Mars mission tasks into 18 predefined Human System Task Categories (HSTCs) based on their textual descriptions. As part of developing the Crew Health and Performance – Probabilistic Risk Assessment (CHP-PRA projects Performance Risk Model (PRisM) proof-of-concept, we established a framework to project performance scores from small-scale tests onto a preliminary list of Mars tasks. The foundation of our model was a comprehensive spreadsheet populated by NASA experts and clinicians, which detailed each Mars task alongside binary indicators of HSTC involvement. This dataset enabled the initial application of supervised ML, training and testing on existing HSTC labels. The HSTCs were originally defined from a medical system perspective, focusing on task impairments due to deteriorated human health. To expand our model's scope to include categories impacting performance, we face the challenge of generating binary labels (0 or 1) for new categories without pre-existing data. We address this by employing Generative AI (GenAI) software to determine whether a given task involved a new category by asking, "Does task A involve using category B?" We validate our approach by comparing the GenAI's binary classifications with the expert-provided labels for existing HSTCs. Notably, we utilize Ollama [4], a locally hosted GenAI tool that does not require cloud access, thus safeguarding NASA's proprietary data from unauthorized exposure. This study demonstrates the feasibility of leveraging cutting-edge AI tools to advance research, paving the way for automation and rapid decision-making in space exploration.

Mona Matar↗

NASA NTRS1982

The optimal design of involute gear teeth with unequal addenda

The design of a gear mesh is treated with the objective of minimizing the gear size for a given gear ratio, pinion torque, pressure angle, and allowable tooth lengths. Tooth strengths considered include scoring, pitting fatigue, and bending fatigue. Kinematic involute interference is avoided. The design variation on standard spur gear teeth called the long and short addendum system, is considered. In this system the mesh center distance and pressure angle are maintained as is the ability to manufacture the teeth with standard tooling. However, the pinion and gear tooth proportions are altered in order to obtain fewer teeth numbers for the same ratio as standard gears without kinematic involute interference. The effect of this nonstandard gearing geometry with on tooth strengths and gear mesh size are studied. For a 2:1 gearing ratio, the optimal nonstandard gear design is compared with the optimal standard gear design.

Savage, M.↗

NASA NTRS2001

Examination of Automation-Induced Complacency and Individual Difference Variates

Automation-induced complacency has been documented as a cause or contributing factor in many airplane accidents throughout the last two decades. It is surmised that the condition results when a crew is working in highly reliable automated environments in which they serve as supervisory controllers monitoring system states for occasional automation failures. Although many reports have discussed the dangers of complacency, little empirical research has been produced to substantiate its harmful effects on performance as well as what factors produce complacency. There have been some suggestions, however, that individual characteristics could serve as possible predictors of performance in automated systems. The present study examined relationship between the individual differences of complacency potential, boredom proneness, and cognitive failure, automation-induced complacency. Workload and boredom scores were also collected and analyzed in relation to the three individual differences. The results of the study demonstrated that there are personality individual differences that are related to whether an individual will succumb to automation-induced complacency. Theoretical and practical implications are discussed.

Prinzel, Lawrence J., III↗

NASA NTRS1984

The Impact of Pictorial Display on Operator Learning and Performance

The effects of pictorially displayed information on human learning and performance of a simple control task were investigated. The controlled system was a harmonic oscillator and the system response was displayed to subjects as either an animated pendulum or a horizontally moving dot. Results indicated that the pendulum display did not effect performance scores but did significantly effect the learning processes of individual operators. The subjects with the pendulum display demonstrated more vertical internal models early in the experiment and the manner in which their internal models were tuned with practice showed increased variability between subjects.

Miller, R. A.↗

NASA NTRS2007

Feasibility of Turing-Style Tests for Autonomous Aerial Vehicle "Intelligence"

A new approach is suggested to define and evaluate key metrics as to autonomous aerial vehicle performance. This approach entails the conceptual definition of a "Turing Test" for UAVs. Such a "UAV Turing test" would be conducted by means of mission simulations and/or tailored flight demonstrations of vehicles under the guidance of their autonomous system software. These autonomous vehicle mission simulations and flight demonstrations would also have to be benchmarked against missions "flown" with pilots/human-operators in the loop. In turn, scoring criteria for such testing could be based upon both quantitative mission success metrics (unique to each mission) and by turning to analog "handling quality" metrics similar to the well-known Cooper-Harper pilot ratings used for manned aircraft. Autonomous aerial vehicles would be considered to have successfully passed this "UAV Turing Test" if the aggregate mission success metrics and handling qualities for the autonomous aerial vehicle matched or exceeded the equivalent metrics for missions conducted with pilots/human-operators in the loop. Alternatively, an independent, knowledgeable observer could provide the "UAV Turing Test" ratings of whether a vehicle is autonomous or "piloted." This observer ideally would, in the more sophisticated mission simulations, also have the enhanced capability of being able to override the scripted mission scenario and instigate failure modes and change of flight profile/plans. If a majority of mission tasks are rated as "piloted" by the observer, when in reality the vehicle/simulation is fully- or semi- autonomously controlled, then the vehicle/simulation "passes" the "UAV Turing Test." In this regards, this second "UAV Turing Test" approach is more consistent with Turing s original "imitation game" proposal. The overall feasibility, and important considerations and limitations, of such an approach for judging/evaluating autonomous aerial vehicle "intelligence" will be discussed from a theoretical perspective.

Young, Larry A.↗

NASA NTRS2014

Modeling of Blood Lead Levels in Astronauts Exposed to Lead from Microgravity-Accelerated Bone Loss

Human exposure to lead has been associated with toxicity to multiple organ systems. Studies of various population groups with relatively low blood lead concentrations (<10 μg/dL) have indicated associations of blood lead level with lower cognitive test scores in children, later onset of puberty in girls, and increased blood pressure and cardiovascular mortality rates in adults. Cognitive effects are considered by regulatory agencies to be the most sensitive endpoint at low doses. Although ~95% of the body burden of lead is stored in the bones, the adverse effects of lead correlate with the concentration of lead in the blood better than with that in the bones. NASA has found that prolonged exposure to microgravity during spaceflight results in a significant loss of bone minerals, the extent of which varies from individual to individual and from bone to bone, but generally averages about 0.5% per month. During such bone loss, lead that had been stored in bones would be released along with calcium. The effects on the concentration of lead in the blood (PbB) of various concentrations of lead in drinking water (PbW) and of lead released from bones due to accelerated osteoporosis in microgravity, as well as changes in exposure to environmental lead before, during, and after spaceflight were evaluated using a physiologically based pharmacokinetic (PBPK) model that incorporated exposure to environmental lead both on earth and in flight and included temporarily increased rates of osteoporosis during spaceflight.

Garcia, H.↗

NASA NTRSDate not supplied

2024 Nasa Lunabotics University Competition: Site Preparation With Bulk Regolith

Introduction: Lunabotics provides accredited institutions of higher learning students (vocational-technical, college, university) an opportunity to apply the NASA systems engineering process to design and build a prototype robot. This robot would be capable of performing the proposed operations on the Lunar surface in support of future Artemis mission goals. Lunabotics features a systems engineering design challenge to engage students in the next phase of hu-man space exploration supporting the Artemis missions. This two-semester event encourages students to design and build an autonomous or telerobotic robot designed to traverse the simulated Lunar surface and complete the assigned construction tasks. The number of teams accepted into this challenge is not predetermined but is based on the scores and overall quality of the Project Management Plans received and other factors. The culmination of the Lunabotics virtual challenge will be the design, build and operation of a functional prototype Lunar robot. Teams are required to submit the following: (1) Project Management Plan, (2) Systems Engineering Paper, a (3) STEM Engage-ment Report, and a (4) Proof of Life Video. This is an optional item, but to qualify for the grand prize a team must also submit a: (5) Presentation and Demonstration. Background: The NASA Lunabotics University Competition was first held in 2010 as a follow on to the NASA Regolith Excavation Challenge [1]. The high level of interest and participation from over 50 universities each year has led to a sustained annual competi-ion cadence: it has been held every May for the past 14 years [2,3]. Over 6,000 students have participated and been inspired to pursue Science, Technology, En-gineering and Mathematics careers (STEM). 2024 Competition: The necessary lunar surface tasks are evolving to meet the NASA Artemis Mission requirements. In the past Lunabotics challenges we gathered data to support Lunar mining for consuma-bles in the Lunar regolith. Now, in 2024, the task is to gather data on Lunar site preparation and construction by designing and building a robot that will traverse the chaotic Lunar terrain and construct a regolith-based berm. The goal is to build a berm structure which would be useful to the Artemis Mission for blast and ejecta protection during lunar landings and launches, shading cryogenic propellant tank farms, providing radiation protection around a nuclear power plant and other mission critical uses. Lunabotics will consist of three separate events this year. The first event is NASA’s Lunabotics Project Development Challenge, where teams submit various deliverables to be scored by judges. The second event will be the University of Central Florida (UCF) Lunabotics Qualification challenge, in the Exolith laboratory, where teams will put their de-signs to the test. The top ten scoring teams from the Qualification challenge are then invited to the third and final event, NASA’s Lunabotics On-Site Challenge at Kennedy Space Center in Florida. This presentation will summarize the results and lessons learned from the NASA Lunabotics University Competition held in May 2024.

Lunabotics↗

NASA NTRS1988

Optimal solutions for complex design problems: Using isoperformance software for human factors trade offs

A major application of isoperformance is as a trade-off methodology of the three major drivers of system design; equipment, training variables, and user characteristics. The flexibility of isoperformance allows each of these three components to be nearly any rational variation. For example, aptitude may be military Armed Forces Qualification Testing (AFQT) categories, cutoff scores within a selection procedure, or simply dichotomizing high and low scorers (pass/fail). Equipment may be new versus old, 'smart' versus dumb, high versus low resolution, etc. Training may be short versus long or varieties of media types (lecture versus CAI/CBI versus self-paced workbooks). In its final computerized form isoperformance lets the user set an operational level of performance (e.g., a jet pilot in a simulated emergency must take prescribed corrective action and clear the plane in several seconds, pilot astronauts will check out all shuttle flight systems within 30 minutes, or Mission Specialists must handle sucdessfully a required number of job elements). At this point the computer program guides the user through any requested trade-offs of the three components while maintaining the specified operational level of performance through isoperformance curves. A demonstration of the computer program is currently available.

Kennedy, Robert S.↗

NASA NTRS2006

A Countermeasure for Space Motion Sickness

Overall, the results obtained in both the U.S. and the Russian space programs indicate that most space crews will experience some symptoms of motion sickness (MS) causing significant impact on the operational objectives that must be accomplished to assure mission success. At this time the primary countermeasure for MS requires the administration of Promethazine. Promethazine is not a benign drug, and is most frequently administered just prior to the sleep cycle to prevent its side effects from further compromising mission objectives. Clearly other countermeasures for SMS must be developed. Currently the primary focus is on two different technologies: (1) developing new and different pharmacological compounds with less significant side effects, (2) preflight training. The primary problem with all of these methods for controlling MS is time. New drugs that may be beneficial are years from testing and development, and preflight training requires a significant investment of crew time during an already intensive pre-launch schedule. Granted, motion sickness symptoms can be minimized with either of the two methods detailed above, however, it may be possible to develop a countermeasure that does not require either extensive adaptation time or exposure to motion sickness. Approximately 25 years ago Professor Geoffrey Melvill Jones presented his work on adaptation of the vestibuloocular reflex (VOR) using optically reversed vision (left-right prisms) during head rotations in the horizontal plane. It was of no surprise that most subjects experienced motion sickness while wearing the optically reversing prisms. However, a serendipitous finding emerged during this research showing that the same subjects did not experience motion sickness symptoms when wearing the reversing prisms under stroboscopic illumination. The mechanism, by which this side-effect was believed to have occurred, is not clearly understood. However, the fact that no motion sickness was ever noted, suggests the possibility of producing functionally useful adaptation during space flight without the penalty of disabling motion sickness by controlling the rate of the adaptive process by means of an appropriate stroboscopically presented environment. After several recent meetings with Professor Melvill Jones, we were encouraged to repeat the motion sickness portions of his and Mandl's 1981 stroboscopic experiment. In conducting this experiment we used a randomized cross-over design where subjects were randomly assigned to either a stroboscopic flash or no strobe for their first exposure in the experimental design. Twenty subjects (19 subjects completed the study) read a short passage from Treasure Island mounted on the wall approximately 1 m from their eyes while wearing left-right reversing prisms. The strobe on time of 3 microseconds and flash frequency of 4 Hz was set to equal that used in the original study. Motion sickness was scored using a modified Miller and Graybiel scale that we constructed to include symptoms that may be elicited under conditions where reversing prisms are worn. On this scale a score of 5 represented Malaise IIa (mild motion sickness) and a score of 8 or above is approaching frank sickness. Symptoms were tracked and recorded every 5 min during the task. Testing was limited to 30 min unless the subject had reached the MIIa score, at which time the test was terminated. Performance under stroboscopic illumination was significantly better than when the subjects read under normal room illumination while wearing the left-right reversing prisms. Based on these results we developed a goggle system using LCD material that can be strobed. To evaluate the effectiveness of stroboscopic goggles we tested an additional 9 subjects in addition to retesting 10 used in the stroboscopic pilot study described above. These 19 subjects wore a pair of strobing LCD goggles that could be cycled at 4 Hz. These subjects wore the goggles while also wearing left-right reversg prisms. Results while wearing the goggles showed that none of the 19 subjects scored at the MIIa level on the motion sickness rating scale. When the goggles did not flash (no strobe), 11 of the 19 developed symptoms above the MIIa criteria. As a countermeasure the goggles seem to be effective, even with an on time of 10 msec (time the goggles are clear). We have also collected anecdotal data, from our personnel in the Neuroscience Laboratory at the Johnson Space Center, suggesting that the goggles may effective in preventing carsickness.

Reschke, M. F.↗

NASA NTRSDate not supplied

What Went Wrong: A Survey of Wildfire UAS Mishaps through Named Entity Recognition

Increasingly, unmanned aircraft systems (UAS) are being applied to wildfire incidents for tasks such as mapping, aerial ignition, and delivery. As a result, aviation incident reporting systems for wildfires are beginning to accumulate data related to UAS mishaps in wildfire response. In this research, we apply state-of-the-art natural language processing (NLP) techniques to develop a custom Named Entity Recognition (NER) model which extracts entities relevant to safety analysts. The custom NER model is built by fine-tuning an existing Bidirectional Encoder Representations from Transformers (BERT) model, resulting in a generalizable NER model that can extract engineering relevant entities including failure modes, causes, effects, control processes, and recommendations from failure-relevant text. This model performs passably, with a weighted average f1 score of 0.33 across entity types, indicating more labeled training data is needed. Extracted entities are used to form a Failure Modes and Effects Analysis (FMEA)-style survey of wildfire UAS mishaps reported using the SAFECOM system. Similar mishaps are manually clustered and reported as single rows within an FMEA. Foreach cluster, we compute frequency, severity, and overall riskin accordance with FAA standards. This methodology can beapplied as part of a broader safety management system totrack trends in mishaps (e.g., likelihood, severity) and discoverknowledge (e.g., causes, effects) that can be utilized to improvesafety outcomes and system performance.

Machine Learning↗

NASA NTRS2018

Trusted Communication: Utilizing Speech Communication to Enhance Human-Machine Teaming Success

An area of increasing interest for the next generation of aircraft is autonomy and the integration of increasingly autonomous systems into the national airspace. Such an integration requires humans to work closely with autonomous systems, forming teams. Our hypothesis is that a team composed of both humans and autonomous systems will operate better than either entity alone. We have existing procedures for certifying pilots to operate in the national airspace and are currently working on methods for validating the function of autonomous systems, however we have no method in place for assessing the interaction of these two disparate systems. Communication is one avenue. This paper will examine the use of language as a metric for ascertaining human-machine teaming effectiveness. A proof-of-concept of the application of two communication-based analysis techniques, Linguistic Inquiry and Word Count (LIWC) and Latent Semantic Analysis (LSA), for the prediction of success in human/chatbot teaming was conducted. By running these analyses over data from the 2014 and 2015 Loebner Prize competitions of human/chatbot teaming, numerical scores were obtained that can be associated with scores provided by human judges during the competition. Correlating their LIWC and LSA data with the scores provided by the judges, and using linear regression over this correlation, formulae were obtained that predict the score of human/chatbot interaction. These formulae were tested over the 2013 Loebner Prize transcripts, determining that, though there was strong correlation between predicted and actual scores, the predictive success of this method was not strong. However, with specialized topic spaces and lexica, as well as larger data sets, the predictive power of these metrics will improve. Given the importance of providing metrics for human-machine system team success and given the promise shown by the communication-basedLIWCand LSAmethods, continuing research in this area is necessary. After examining the potential for using communication and spoken language as a metric for the success of human/autonomous system teaming, this paper then examines aspects inherent to communication systems that may contribute to unreliability and reduced trust. Modern natural language processing tools rely on deep learning algorithms to create language rules that produce accurate results, but these rules are uninterpretable. The resulting blackbox system lacks transparency necessary for full validation and complete trust. Additionally, speech-based interfaces pose other difficulties to developing coordinated teamwork between humans and autonomous systems. Human communication is infrequently limited to speech only, instead usually relying on a combination of verbal, gestural, and general body language communication. Reducing an analysis of team effectiveness to a study of spoken language alone is problematic as it leaves these other equally important forms of communication out. This paper will examine these problems and the general deficiencies in speech-based metrics for human-machine teaming.

E L Meszaros↗