Search NASA⌕ Search

SEARCH · Search NASA

Results for “Sentiment analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

36 records · Page 2

Understanding and Verifying Neural Networks

Deep Neural Networks (DNNs) have gained immense popularity in recent times and have widespread use in applications such as image classification, sentiment analysis, speech recognition and also in safety-critical applications such as autonomous driving. However, they suffer limitations such as lack of explainability and robustness which raise safety and security concerns in their usage. Further, the complex structure and large input spaces of DNNs act as an impediment to thorough verification and testing. The SafeDNN project at the Robust Software Engineering (RSE) group at NASA aims at exploring techniques to ensure that systems that use deep neural networks are safe, robust and interpretable. In this talk, I will be presenting our technique Prophecy that automatically infers formal properties of deep neural network models. The tool extracts patterns based on neuron activations as preconditions that imply certain desirable output properties of the model. I would be highlighting case studies that use Prophecy in obtaining explanations for network decisions, understanding correct and incorrect behavior, providing formal guarantees wrt safety and robustness, and debugging neural network models. We have applied the tool on image classification networks, neural network controllers providing turn advisories in unmanned aircrafts, regression models used for autonomous center-line tracking in aircrafts and neural network object detectors

Deep Neural Networks↗

Language Model For Earth Science: Exploring Potential Downstream Applications As Well As Current Challenges

The use of deep learning techniques to build transformer language models such as SciBERT and GPT3 have transformed the natural language technology (NLT) landscape. These new NLTs are being used in speech to text and vice versa, auto-mated text classification, sentiment analysis, topic modeling, text summarization, and cognitive assistants. While Earth science has no shortage of unstructured data such as journal and conference papers, little efforts have focused on harnessing NLTs for knowledge extraction and supporting the scientific process. This paper surveys the use of language models in different science. BERT-E, a new Earth science-specific language model, is presented. BERT-E is generated using a transfer learning solution. A language model that has already been trained for general Science (SciBERT) is fine-tuned using abstracts and full text extracted from various Earth science-related articles. A downstream keywords classification application is used for evaluation, and the use of BERT-E shows improved performance. The need to develop a robust set of benchmarks in evaluating the language model such as BERT-E is discussed. Finally, example applications are presented to inspire additional ideas for applications using domain-specific language models.

R Ramachandran↗

Navigating Team Dynamics: Automated Detection of Micro-Behaviors Between Team Members Through Longitudinal Interaction Data

The success in future long term space exploration missions will depend on the cooperation, coordination, and mutual understanding among the crew members. Micro-behaviors are momentary, subtle linguistic and paralinguistic indicators of thinking and feeling toward another member of the team (Cortina et al., 2001; Smith & Griffiths, 2022) that can significantly impact team dynamics and influence the overall team performance (Paromita & Chaspari, 2024). Due to their interactive nature, micro-behaviors have a sender (i.e., the team member expressing the micro-behavior) and a target (the team member impacted by the micro-behavior). Detection of these behaviors can assist in avoiding possible conflict among crew members and promoting the overall team success. Our prior research focused on an initial proof of concept of machine learning (ML) models and natural language processing (NLP) techniques that were used for automatically detect micro-behaviors among crew members of the US National Aeronautics and Space Administration’s (NASA) Human Exploration Research Analog (HERA) Campaigns 4 and 5 missions (Paromita et al., 2023). Results underscored the importance of incorporating contextual information in the ML models in the form of sentiment analysis, type of task, and dyadic interaction among team members. Here, we expand the scope of our prior work in two ways. First, we assess ML/NLP methods on new behavioral annotations coded using an adapted version of Smith & Griffins (2022) theoretical framework in terms of Violation (i.e., presence of valenced behavior, uplifting/positive or discouraging/negative), Intensity (i.e., force of behavior in terms of how uplifting or discouraging is the behavior), and Intent (i.e., motive of the behavior in terms of whether it was deliberate or unintentional). Second, we expand the design of the ML model to preserve information about the role of each team member within the occurrence of the micro-behavior (in contrast to the previous model that only considered the sender and the target without determining the team member role). This allows to consider all team members' contributions in the conversation and model long-term dependencies in the dialogue. Our experiments for this study are conducted on data from 5 teams of the NASA HERA C4 (NASA grant NNX16AQ48G (PI: Bell)). Conversations were extracted from the 1.5 hour Team Interaction Battery (TIB) task that occurred 5 times in-mission per crew. This resulted in a total of 13,058 conversational turns (i.e., 17.8% uplifting, 3.3% discouraging, 75.76% neutral, 3.14% nulls). Our findings with the revised behavioral coding and ML/NLP models indicate a 43.66% macro F1-score (i.e., 38.29% precision (P), 50.8% recall (R)) for a dialog state-tracking model that includes information from the sender only, and a 40.9% F1-score (i.e., 38.7% P, 43.36% R) for the same model that includes information from both the sender and the target of the micro-behavior. These are significantly higher compared to simple random forest models that classify behaviors strictly based on speech content and do not consider iterative team dynamics, achieving a 36.07% F1-score (i.e., 39.04% R, 33.53% P). Our findings demonstrate potential ways to leverage large conversational datasets to better capture complex team dynamics. We will discuss future directions including proposed models that can incorporate additional mission days and tasks beyond the TIB for objectively quantifying team behavior at high temporal resolution in space exploration missions.

Projna Paromita↗

SULI Oral Presentation

Furthering our understanding of the prevalence and severity of issues that customers face when charging their electric vehicles (EVs) is crucial in order to improve the charging experience across the United States. This project utilizes web-scraping, machine leaning (ML), and natural language processing (NLP) techniques to analyze and categorize user-generated reviews. Selenium was used to build a data collection tool that can scrape vast amounts of user review data from the PlugShare website. Sentiment analysis was employed on this dataset in order to filter out negative reviews for further analysis. NLP techniques such as tokenization and word embedding were then used to convert user-written comments into a numerical format that a ML model can interpret. Multiple ML approaches are currently being explored in order to identify and categorize the charging issues being talked about in each review. Ultimately, the results from the ML model will be visualized and explained in a report on customer pain points to be delivered to the ChargeX Consortium, therefore revealing specific areas for improvement in the customer charging experience.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

SULI Oral Presentation

Furthering our understanding of the prevalence and severity of issues that customers face when charging their electric vehicles (EVs) is crucial in order to improve the charging experience across the United States. This project utilizes web-scraping, machine leaning (ML), and natural language processing (NLP) techniques to analyze and categorize user-generated reviews. Selenium was used to build a data collection tool that can scrape vast amounts of user review data from the PlugShare website. Sentiment analysis was employed on this dataset in order to filter out negative reviews for further analysis. NLP techniques such as tokenization and word embedding were then used to convert user-written comments into a numerical format that a ML model can interpret. Multiple ML approaches are currently being explored in order to identify and categorize the charging issues being talked about in each review. Ultimately, the results from the ML model will be visualized and explained in a report on customer pain points to be delivered to the ChargeX Consortium, therefore revealing specific areas for improvement in the customer charging experience.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Water narratives in local newspapers within the United States

Sustainable use of water resources continues to be a challenge across the globe. This is in part due to the complex set of physical and social behaviors that interact to influence water management from local to global scales. Analyses of water resources have been conducted using a variety of techniques, including qualitative evaluations of media narratives. This study aims to augment these methods by leveraging computational and quantitative techniques from the social sciences focused on text analyses. Specifically, we use natural language processing methods to investigate a large corpus (approx. 1.8M) of newspaper articles spanning approximately 35 years (1982–2017) for insights into human-nature interactions with water. Focusing on local and regional United States publications, our analysis demonstrates important dynamics in water-related dialogue about drinking water and pollution to other critical infrastructures, such as energy, across different parts of the country. Our assessment, which looks at water as a system, also highlights key actors and sentiments surrounding water. Extending these analytical methods could help us further improve our understanding of the complex roles of water in current society that should be considered in emerging activities to mitigate and respond to resource conflicts and climate change.

54 ENVIRONMENTAL SCIENCES↗

Self-Admitted Technical Debt in Scientific Software: Prioritization, Sentiment, and Propagation Across Artifacts

Self-admitted technical debt (SATD) impairs scientific software (SSW), yet its prioritization, sentiment, persistence, and propagation remains underexplored. Understanding how SSW developers express, and address SATD is crucial for improving SSW maintenance, and tooling. This study investigates how SATD types and artifacts in SSW are prioritized, how sentiment relates to urgency, SATD removal and resolution rates, and the extent to which SATD propagates across artifacts. We analyzed nine SSW repositories using a SATD classification model and a semantic embedding-based prioritization heuristic. SATD was examined across multiple artifacts, with sentiment assessed via a fine-tuned transformer. Propagation was traced, priority scores compared to static analysis, and removal and resolution rates quantified. SATD in comments, commits, and pull requests receive higher priority than SATD in issues, with negative sentiment amplifying urgency. Resolution and removal rates lag behind open-source software (OSS) averages. Most SATD remains confined to the originating artifact, but longer propagation chains are rare and correlate with higher priority, highlighting persistent and high impact debt. Prioritization is influenced by artifact type and sentiment, while low removal and resolution rates signal persistent debt. Cross-artifact propagation marks high priority, unresolved SATD, providing empirical guidance for targeted monitoring, review prioritization, and tool supported maintenance in SSW.

Melin, Eric [Boise State University]↗

Decarbonizing residential buildings in the United States: A comparative analysis of households and construction professionals

In this study, we present a comparative analysis of surveys distributed to home occupants and construction professionals in the U.S., focused on energy upgrades and electrification retrofits that support residential building decarbonization. The surveys were executed by separate research groups and combined for this study. The study examines the decision-making, sentiments, perceptions, experiences, and practices of both groups by analyzing data from three separate surveys. These surveys assess technologies, attitudes, awareness, motivations, barriers, and opportunities related to energy retrofits and electrification. The analysis highlights key differences in the perceptions and behaviors of households and construction professionals, revealing substantial barriers to achieving decarbonization goals. For example, households cite climate change and sustainability as key motivators for pursuing energy retrofits (89%), while construction industry professionals view these themes as less important for their clients (44%). This suggests an opportunity for the construction industry to align its messaging with the values that households prioritize, helping to advance residential decarbonization. Overall, the study identifies challenges faced by both groups, factors influencing the adoption of energy-efficient practices, and inconsistencies between occupant and construction industry professionals' views. These insights contribute to the development of targeted strategies and policies to accelerate the decarbonization of residential buildings in the U.S.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Sustainable Aviation Fuels from 30,000 feet – Scoping Report

A literature review, first principles analysis, and mapping of the airline-fuel infrastructure were performed to reassess the potential for various means of realizing a sustainable aviation sector. The literature review found a plethora of technical methods to make the airline industry net-zero CO 2 but limited work on the emergent barriers to sustainable aviation implementation such as land use, repurposing of fuel refining assets, and airline cost structures. First principles analyses of the energy, land, and costs of three broad net-zero strategies were performed to determine if sustainable aviation is feasible even in the most optimistic scenarios. Those methods were biomass-derived sustainable aviation fuel (SAF), fuel produced from CO 2 captured from the air and H 2 produced from water electrolysis (air-to-fuel: ATF), and offsetting fossil-derived Jet Fuel with carbon dioxide removal from the atmosphere (direct air capture and storage: DACS). Offsetting CO 2 emissions from using fossil Jet Fuel with DACS was found to consume the least energy, require the smallest quantity of land, and be the cheapest option. Replacing Jet Fuel with SAF was found to be only slightly more expensive than DACS but require substantially more land. ATF was found to be the most expensive and energy intensive option, though it did spare land relative to SAF. We found that calls to utilize waste biomass are well-founded, but wastes alone cannot cover the needs of the global airline industry. This means additional land will need to be allocated for biomass cultivation if SAF becomes the primary pathway to sustainable aviation. While differences were found in the performance of different types of biomass, those differences were small relative to differences with ATF and DACS. An analysis of the pricing structure of the airline industry suggests that some of the cost advantage of DACS may be erased by consumer sentiment which could explain why, to-date, the airline industry is focused primarily on SAF and to a lesser extent ATF. Further work is required to estimate the transportation, capital cost, and financing cost implications of greenfield and brownfield sites for SAF, DACS, and ATF. In the final report, we will analyze the potential to retrofit corn ethanol facilities to operate the ethanol-to-jet fuel process in response to decreasing gasoline demand as the electrification of road transport progresses. We will also construct a prioritization of land use to determine whether each net-zero CO 2 aviation pathway can make claim to the land required by their technology.

09 BIOMASS FUELS↗

Cross-platform analysis of public responses to the 2019 Ridgecrest earthquake sequence on Twitter and Reddit

Online social networks (OSNs) have become a powerful tool to study collective human responses to extreme events such as earthquakes. Most previous research concentrated on a single platform and utilized users’ behaviors on a single platform to study people’s general responses. In this study, we explore the characteristics of people’s behaviors on different OSNs and conduct a cross-platform analysis of public responses to earthquakes. Our findings support the Uses and Gratification theory that users on Reddit and Twitter are engaging with platforms that they may feel best reflect their sense of self. Using the 2019 Ridgecrest earthquakes as our study cases, we collected 510,579 tweets and 45,770 Reddit posts (including 1437 submissions and 44,333 comments) to answer the following research questions: (1) What were the similarities and differences between public responses on Twitter and Reddit? (2) Considering the different mechanisms of Twitter and Reddit, what unique information of public responses can we learn from Reddit as compared with Twitter? By answering these research questions, we aim to bridge the gap of cross-platform public responses research towards natural hazards. Our study evinces that the users on the two different platforms have both different topics of interest and different sentiments towards the same earthquake, which indicates the necessity of investigating cross-platform OSNs to reveal a more comprehensive picture of people’s general public responses towards certain disasters. Our analysis also finds that r/conspiracy subreddit is one of the major venues where people discuss the 2019 Ridgecrest earthquakes on Reddit and different misinformation/conspiracies spread on Twitter and Reddit platforms (e.g., “Big one is coming” on Twitter and “Nuclear test” on Reddit).

58 GEOSCIENCES↗

Understanding the International Space Station Crew Perspective following Long-Duration Missions through Data Analytics & Visualization of Crew Feedback

The International Space Station (ISS) first became a home and research laboratory for NASA and International Partner crewmembers over 16 years ago. Each ISS mission lasts approximately 6 months and consists of three to six crewmembers. After returning to Earth, most crewmembers participate in an extensive series of 30+ debriefs intended to further understand life onboard ISS and allow crews to reflect on their experiences. Examples of debrief data collected include ISS crew feedback about sleep, dining, payload science, scheduling and time planning, health & safety, and maintenance. The Flight Crew Integration (FCI) Operational Habitability (OpsHab) team, based at Johnson Space Center (JSC), is a small group of Human Factors engineers and one stenographer that has worked collaboratively with the NASA Astronaut office and ISS Program to collect, maintain, disseminate and analyze this data. The database provides an exceptional and unique resource for understanding the "crew perspective" on long duration space missions. Data is formatted and categorized to allow for ease of search, reporting, and ultimately trending, in order to understand lessons learned, recurring issues and efficiencies gained over time. Recently, the FCI OpsHab team began collaborating with the NASA JSC Knowledge Management team to provide analytical analysis and visualization of these over 75,000 crew comments in order to better ascertain the crew's perspective on long duration spaceflight and gain insight on changes over time. In this initial phase of study, a text mining framework was used to cluster similar comments and develop measures of similarity useful for identifying relevant topics affecting crew health or performance, locating similar comments when a particular issue or item of operational interest is identified, and providing search capabilities to identify information pertinent to future spaceflight systems and processes for things like procedure development and training. In addition, the comments were scored for sentiment using a polarity scoring algorithm to identify both positive and negative comments for particular groups and clusters, allowing the team to make analytically informed decisions regarding future hardware and operating procedures. The use of polarity scoring with time series analysis was used to provide insight into how crew health and habitability is changing throughout various spaceflight increments or the station lifecycle as a whole. Finally, a visualization framework was developed to address the needs of the end users to search for and analyze comments by user, category or mission. This paper will discuss how the use of an analytical framework in conjunction with the current human interface, improved the understanding of crew perspective and shortened the time for analysis allowing for more informed decisions and rapid development of improvements. These methods are significantly optimizing the way that this valuable data can be assessed and applied to current and future spaceflight design and development. This collaboration allows the FCI OpsHab team to effectively analyze and share data in a more automated and timely fashion. Trends are no longer derived manually and can be illustrated effectively and accurately with these evolving techniques to an ever growing group of human spaceflight end users.

Bryant, Cody↗

User Metrics in NASA Earth Science Data Systems

This presentation the collection and use of user metrics in NASA's Earth Science data systems. A variety of collection methods is discussed, with particular emphasis given to the American Customer Satisfaction Index (ASCI). User sentiment on potential use of cloud computing is presented, with generally positive responses. The presentation also discusses various forms of automatically collected metrics, including an example of the relative usage of different functions within the Giovanni analysis system.

user metrics↗

An Approach to Identifying Aspects of Positive Pilot Behavior within the Aviation Safety Reporting System

The National Airspace System (NAS) is constantly evolving as air traffic continues to ramp up to pre-pandemic numbers and projected to grow to unprecedented levels in the coming years. As well as increasing demand to the current system, emerging operations such as Unmanned Autonomous Systems are also expected to add to complexity in the airspace. To address these issues, the industry and government agencies supporting the NAS will need to rely upon additional automation and new technologies to address future operational requirements, while continuing to be a world-leading safe transportation system. As these new technologies are implemented, the system continues to rely on human pilots and controllers in the loop to monitor the system and intervene in situations the automation cannot handle. The goal of proactively addressing safety is of foremost concern to ensure passenger confidence. The industry has implemented various Safety Monitoring Systems to identify safety risks and proactively address them before they result in a serious incident or accident. One such program is the Aviation Safety Reporting System (ASRS). ASRS is a long-established system where pilots and controllers voluntarily and anonymously report safety incidents they experienced and observed during line operations by providing rich text narratives describing the events, the environment, and conditions leading to the safety event of concern. These narratives provide insight and context around events of interest and can be used to identify emerging problems. They can trigger investigations within Flight Operational Quality Assurance or Flight Data Monitoring programs. However, this process typically focuses on the adverse events and the unsafe aspects of the operations surrounding the reported or detected events. This perspective of investigating factors that went wrong around an adverse event is commonly referred to as Safety I. Alternatively, characterizing successful actions that operators perform every day under varying conditions that keep the system within safe operating bounds is a concept referred to as Safety II. The benefit of the Safety II view is that the scope is much larger than that of Safety I since a vast majority of the operations result in successful flights. Many of the successful techniques used to manage operational threats are not documented in standard operating procedures or taught during training. They are typically acquired over time by working with experienced pilots during line operations or in many cases after experiencing a problem for the first time and reacting to it in situ, drawing from years of experience to manage the threat. In an attempt to quantify these positive actions, we are proposing an approach to extracting key behaviors within ASRS reports that can support the Safety II concept. Our analysis assumes that ASRS reports contain some descriptions of corrective actions that operators performed to prevent a situation from leading to an accident. Leveraging recent advances in Natural Language Process modeling, we have developed an approach to extract positive sentiment from reports, embed these positive statements in a vector space where they can be numerically analyzed, and clustering these statements into similar contextual categories. From these contextualized categories we can attempt to summarized and distilled aspects of the positive behavior. The goal is to identify categories of behavior that describe consistent operator techniques that supports the Safety II concept. With this information, airlines may enable learning from these positive actions, or address procedures that need to be changed to avoid having pilots implement a workaround. These insights can provide a lens into what is “going right” in the operations that may otherwise not be known widely within the community. It is envisioned that this approach can be extended to other narrative programs such as Line Operation Safety Audit or Learning Improvement Team reports where similar observed behavior can be analyzed to extract positive actions and inform the overall operations.

NLP↗

An Approach to Identifying Aspects of Positive Pilot Behavior within the Aviation Safety Reporting System

The National Airspace System (NAS) is constantly evolving as air traffic continues to ramp up to pre-pandemic numbers and projected to grow to unprecedented levels in the coming years. As well as increasing demand to the current system, emerging operations such as Unmanned Autonomous Systems are also expected to add to complexity in the airspace. To address these issues, the industry and government agencies supporting the NAS will need to rely upon additional automation and new technologies to address future operational requirements, while continuing to be a world-leading safe transportation system. As these new technologies are implemented, the system continues to rely on human pilots and controllers in the loop to monitor the system and intervene in situations the automation cannot handle. The goal of proactively addressing safety is of foremost concern to ensure passenger confidence. The industry has implemented various Safety Monitoring Systems to identify safety risks and proactively address them before they result in a serious incident or accident. One such program is the Aviation Safety Reporting System (ASRS). ASRS is a long-established system where pilots and controllers voluntarily and anonymously report safety incidents they experienced and observed during line operations by providing rich text narratives describing the events, the environment, and conditions leading to the safety event of concern. These narratives provide insight and context around events of interest and can be used to identify emerging problems. They can trigger investigations within Flight Operational Quality Assurance or Flight Data Monitoring programs. However, this process typically focuses on the adverse events and the unsafe aspects of the operations surrounding the reported or detected events. This perspective of investigating factors that went wrong around an adverse event is commonly referred to as Safety I. Alternatively, characterizing successful actions that operators perform every day under varying conditions that keep the system within safe operating bounds is a concept referred to as Safety II. The benefit of the Safety II view is that the scope is much larger than that of Safety I since a vast majority of the operations result in successful flights. Many of the successful techniques used to manage operational threats are not documented in standard operating procedures or taught during training. They are typically acquired over time by working with experienced pilots during line operations or in many cases after experiencing a problem for the first time and reacting to it in situ, drawing from years of experience to manage the threat. In an attempt to quantify these positive actions, we are proposing an approach to extracting key behaviors within ASRS reports that can support the Safety II concept. Our analysis assumes that ASRS reports contain some descriptions of corrective actions that operators performed to prevent a situation from leading to an accident. Leveraging recent advances in Natural Language Process modeling, we have developed an approach to extract positive sentiment from reports, embed these positive statements in a vector space where they can be numerically analyzed, and clustering these statements into similar contextual categories. From these contextualized categories we can attempt to summarized and distilled aspects of the positive behavior. The goal is to identify categories of behavior that describe consistent operator techniques that supports the Safety II concept. With this information, airlines may enable learning from these positive actions, or address procedures that need to be changed to avoid having pilots implement a workaround. These insights can provide a lens into what is “going right” in the operations that may otherwise not be known widely within the community. It is envisioned that this approach can be extended to other narrative programs such as Line Operation Safety Audit or Learning Improvement Team reports where similar observed behavior can be analyzed to extract positive actions and inform the overall operations.

NLP↗

An Approach to Identifying Aspects of Positive Pilot Behavior within the Aviation Safety Reporting System

The National Airspace System (NAS) is constantly evolving as air traffic continues to ramp up to pre-pandemic numbers and projected to grow to unprecedented levels in the coming years. As well as increasing demand to the current system, emerging operations such as Unmanned Autonomous Systems are also expected to add to complexity in the airspace. To address these issues, the industry and government agencies supporting the NAS will need to rely upon additional automation and new technologies to address future operational requirements, while continuing to be a world-leading safe transportation system. As these new technologies are implemented, the system continues to rely on human pilots and controllers in the loop to monitor the system and intervene in situations the automation cannot handle. The goal of proactively addressing safety is of foremost concern to ensure passenger confidence. The industry has implemented various Safety Monitoring Systems to identify safety risks and proactively address them before they result in a serious incident or accident. One such program is the Aviation Safety Reporting System (ASRS). ASRS is a long-established system where pilots and controllers voluntarily and anonymously report safety incidents they experienced and observed during line operations by providing rich text narratives describing the events, the environment, and conditions leading to the safety event of concern. These narratives provide insight and context around events of interest and can be used to identify emerging problems. They can trigger investigations within Flight Operational Quality Assurance or Flight Data Monitoring programs. However, this process typically focuses on the adverse events and the unsafe aspects of the operations surrounding the reported or detected events. This perspective of investigating factors that went wrong around an adverse event is commonly referred to as Safety I. Alternatively, characterizing successful actions that operators perform every day under varying conditions that keep the system within safe operating bounds is a concept referred to as Safety II. The benefit of the Safety II view is that the scope is much larger than that of Safety I since a vast majority of the operations result in successful flights. Many of the successful techniques used to manage operational threats are not documented in standard operating procedures or taught during training. They are typically acquired over time by working with experienced pilots during line operations or in many cases after experiencing a problem for the first time and reacting to it in situ, drawing from years of experience to manage the threat. In an attempt to quantify these positive actions, we are proposing an approach to extracting key behaviors within ASRS reports that can support the Safety II concept. Our analysis assumes that ASRS reports contain some descriptions of corrective actions that operators performed to prevent a situation from leading to an accident. Leveraging recent advances in Natural Language Process modeling, we have developed an approach to extract positive sentiment from reports, embed these positive statements in a vector space where they can be numerically analyzed, and clustering these statements into similar contextual categories. From these contextualized categories we can attempt to summarized and distilled aspects of the positive behavior. The goal is to identify categories of behavior that describe consistent operator techniques that supports the Safety II concept. With this information, airlines may enable learning from these positive actions, or address procedures that need to be changed to avoid having pilots implement a workaround. These insights can provide a lens into what is “going right” in the operations that may otherwise not be known widely within the community. It is envisioned that this approach can be extended to other narrative programs such as Line Operation Safety Audit or Learning Improvement Team reports where similar observed behavior can be analyzed to extract positive actions and inform the overall operations.

NLP↗

Usability/Sentiment for the Enterprise and ENTERPRISE

The purpose of the Sentiment of Search Study for NASA Johnson Space Center (JSC) is to gain insight into the intranet search environment. With an initial usability survey, the authors were able to determine a usability score based on the Systems Usability Scale (SUS). Created in 1986, the freely available, well cited, SUS is commonly used to determine user perceptions of a system (in this case the intranet search environment). As with any improvement initiative, one must first examine and document the current reality of the situation. In this scenario, a method was needed to determine the usability of a search interface in addition to the user's perception on how well the search system was providing results. The use of the SUS provided a mechanism to quickly ascertain information in both areas, by adding one additional open-ended question at the end. The first ten questions allowed us to examine the usability of the system, while the last questions informed us on how the users rated the performance of the search results. The final analysis provides us with a better understanding of the current situation and areas to focus on for improvement. The power of search applications to enhance knowledge transfer is indisputable. The performance impact for any user unable to find needed information undermines project lifecycle, resource and scheduling requirements. Ever-increasing complexity of content and the user interface make usability considerations for the intranet, especially for search, a necessity instead of a 'nice-to-have'. Despite these arguments, intranet usability is largely disregarded due to lack of attention beyond the functionality of the infrastructure (White, 2013). The data collected from users of the JSC search system revealed their overall sentiment by means of the widely-known System Usability Scale. Results of the scores suggest 75%, +/-0.04, of the population rank the search system below average. In terms of a grading scaled, this equated to D or lower. It is obvious JSC users are not satisfied with the current situation, however they are eager to provide information and assistance in improving the search system. A majority of the respondents provided feedback on the issues most troubling them. This information will be used to enrich the next phase, root cause analysis and solution creation.

Meza, David↗

North Dakota Integrated Carbon Storage Complex Feasibility Study. Final report

In spring 2017, the Energy & Environmental Research Center (EERC) initiated an effort to determine the feasibility of developing a commercial-scale CO 2 geologic storage complex able to store 50+ million tonnes (Mt) of CO 2 in central North Dakota safely, permanently, and economically. The objective was to fulfill the goals of the U.S. Department of Energy (DOE) Carbon Storage Assurance Facility Enterprise (CarbonSAFE) Initiative and address technical and nontechnical challenges specific to commercial-scale deployment of a CO2 storage project. The findings clearly show that the concept of capturing CO 2 from a lignite-fired electrical generation facility in central North Dakota and safely and permanently storing the CO 2 in the deep subsurface is indeed technically, economically, and socially feasible. This project evaluated two study areas and their respective geologic storage complexes located adjacent to separate coal-fired facilities in North Dakota: the Basin Electric Power Cooperative (BEPC)-owned Great Plains Synfuels Plant (GPSP) and the Minnkota Power Cooperative (Minnkota)-owned Milton R. Young Station (MRYS). These locations, one with CO 2 capture in place and an existing CO 2 pipeline, are bolstered by progressive North Dakota pore space ownership and long-term liability laws. These elements and a motivated team created an ideal synergistic scenario for ensuring success of the CarbonSAFE Initiative and promoting North Dakota’s statewide vision for carbon management. The project included drilling two new geologic characterization wells, integrating an existing 3-D seismic survey, creating a geologic model subsequently used for injection simulation, a risk assessment, public outreach, and generating a site development plan based on results. In addition, the performance of select National Risk Assessment Partnership tools was evaluated. The geologic characterization wells were drilled ~5600 feet deep to the Broom Creek Formation; ~350 feet of core was retrieved from each well. The core included the Broom Creek (targeted injection zone) and a portion of the overlying Opeche Shale (seal). The Flemmer-1 well, west of Beulah, North Dakota, yielded 169 feet of sandstone. The BNI-1 well located south of Center, North Dakota, yielded 124 feet of sandstone. In each case, laboratory analysis of the sandstone showed permeability in the 300–1000-mD range, with porosity of 20%–30%. The Flemmer-1 well was sited within the boundaries of an existing 3-D seismic survey. Colocating the well with the seismic survey maximized the relationship between new and legacy data and developed a first-of-its-kind interpretation of the geologic fabric of the Broom Creek. Geologic characterization data were integrated into a 5544-mi 2 geocellular model that encompassed both new wells and stratigraphy from the surface to the Amsden Formation (underlying the Broom Creek). The model was later expanded vertically to include the deeper Black Island–Deadwood interval to examine its potential viability as a storage target. The geocellular model provided the foundation for dynamic simulation of CO 2 into the Broom Creek. Results of the simulation suggest that the Broom Creek could accept the DOE target rate of 2 Mt/yr of CO 2 into as few as two wells. To bracket the expected capture from MRYS, simulations were also investigated for a 4-Mt/yr rate near MRYS. Although more wells are needed (two additional), the Broom Creek still has the storage resource to accept the CO 2 at the increased rate. A risk assessment exercise was conducted to identify and assess technical and nontechnical risks that could prevent potential candidate storage complexes within the study area from serving as commercial storage sites. The assessment identified and evaluated six technical risk categories: 1) CO 2 injectivity, 2) storage capacity, 3) lateral migration of CO 2 , 4) lateral pressure propagation, 5) vertical migration of CO 2 or formation brine, and 6) induced seismicity. Following two rounds of analysis and scoring, no risks were determined to preclude continued efforts to develop carbon capture, utilization, and storage (CCUS) in central North Dakota. The risk assessment results will be used to guide future site characterization, modeling and simulation, and monitoring activities. A specific nontechnical strategic risk based on challenges that may be realized in amalgamation of pore space resulted in vertically expanding the geologic model to incorporate the potential for stacked storage in multiple saline reservoirs. By including the Black Island–Deadwood interval (the basal sedimentary reservoir in this region), the amalgamated areal extent could be reduced by as much as 45%. Working with a smaller geographic area reduces risks and costs associated with monitoring and pore space leasing. An economic evaluation incorporating capture; transport (<5 mi); Class VI wells; permitting; and monitoring, verification, and accounting suggests implementing commercial-scale CCUS is economically attractive if the federal tax benefits of 45Q are included. This is validated by Minnkota’s continued pursuit of CO 2 capture and geologic storage at MRYS through its Project Tundra initiative, indicating that there is a business case for CCUS in central North Dakota. Currently, North Dakota is the only state with underground injection control (UIC) Class VI primacy. Built into the North Dakota Century Code is a series of regulatory requirements that guide the process to obtain a Class VI CO 2 storage facility permit. As part of this project, a site development plan was compiled to assure compliance with North Dakota’s requirements to permit a commercial-scale CO 2 storage operation and includes a prospective time line encompassing a general breakdown of activities. In total, an estimated 30 months would be needed to execute the necessary steps to attain a North Dakota CO 2 storage facility permit. Outreach was an integral part of the project and encompassed any project-related activity that had contact or exposure beyond the project team. The goals of outreach were to foster an environment from which stakeholders could make informed decisions about the project and gauge community receptiveness to a CCUS project. A consistent set of messages and outreach products were developed in conjunction with an outreach advisory board that integrated project partners and team members. To gauge public acceptability of geologically storing CO 2 , 5611 households in the project area were invited to participate in an online survey. The survey results indicate that the public attitude regarding CCUS is neutral to positive, with strong sentiment that CO 2 capture and storage may be an approach to maintain the economic vitality of the region. To achieve project objectives, critical support in the form of financial backing, engineering evaluations, site access, outreach collaboration, operations data, risk assessment/evaluation, and software access was provided by BEPC, the North Dakota Industrial Commission Lignite Research Council, ALLETE Clean Energy, BNI Energy, North American Coal Corporation, Minnkota, Prairie Public Broadcasting, Computer Modelling Group Ltd., and Schlumberger.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗