Search NASA⌕ Search

SEARCH · Search NASA

Results for “data ingest”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17

Cumulus Lessons Learned: Building, Testing, and Sharing a Cloud Archive

Cumulus is a scalable, extensible cloud-based archive system which is capable of ingesting, archiving, and distributing data from both existing on-prem sources and new cloud-native missions. As we have built and evolved the system with contributions from seven NASA EOSDIS organizations, we have learned several lessons about how to build a robust, broadly-applicable, microservices-based cloud system for geospatial data which we will share in this talk.

Quinn, Patrick↗

Open Source Platform for Live Processing of High Data Rate Electron Microscopy (SBIR Phase I Award Topic 15, Subtopic B, Final Scientific / Technical Report)

4D scanning transmission electron microscopy (STEM) is experiencing a revolution both in terms of data size, and data veracity, as are many other areas of experimental data acquisition. This presents enormous opportunities to make new scientific discoveries, but is also hamstrung by aging software that is no longer fit for purpose. Experimental data acquisition platforms must make the same investments in software infrastructure that the simulation community made decades ago to take advantage of high-performance computing (HPC) if they hope to realize the full potential of the latest advances made in detector technology. This requires rethinking the design and deployment of hardware, moving away from small, fixed commodity computing capabilities with closed interfaces to open, modular, scalable systems designed for parallelism and throughput. The technical approach will leverage best-in-class open source technologies, and develop new approaches strategically to build an open source platform for experimental data acquisition targeting high data rate electron microscopy. The project will develop a software platform that uses the power and performance of modern HPC resources to ingest the large volume of data produced by electron microscopes using direct electron detectors. A set of configurable pipelines will be used to enable the processing to be tailored to the problem being investigated with builtin visualization of data at interactive rates. This project will leverage investments made by the DOE and other agencies to provide a powerful software platform for the experimental data community working with most powerful electron microscopes, such as 4D STEM detectors. It will create a platform that serves as a reference for the community. Permissive open source licensing will make the platform available to all, with a simple deployment strategy offering the ability to keep data close to where it is generated. This advanced software platform will require customization for each deployment, creating opportunities for software services that are typically offered to extend open source software platforms.

97 MATHEMATICS AND COMPUTING↗

Link Winds: A visual data analysis system and its application to the atmospheric ozone depletion problem

The Linked Windows Interactive Data System (LinkWinds) is a prototype visual data exploration system resulting from a NASA Jet Propulsion Laboratory (JPL) program of research into the application of graphical methods for rapidly accessing, displaying, and analyzing large multi variate multidisciplinary data sets. Running under UNIX it is an integrated multi-application executing environment using a data-linking paradigm to dynamically interconnect and control multiple windows containing a variety of displays and manipulators. This paradigm, resulting in a system similar to a graphical spreadsheet, is not only a powerful method for organizing large amounts of data for analysis, but leads to a highly intuitive, easy-to-learn user interface. It provides great flexibility in rapidly interacting with large masses of complex data to detect trends, correlations, and anomalies. The system, containing an expanding suite of non-domain-specific applications, provides for the ingestion of a variety of data base formats and hard -copy output of all displays. Remote networked workstations running LinkWinds may be interconnected, providing a multiuser science environment (MUSE) for collaborative data exploration by a distributed science team. The system is being developed in close collaboration with investigators in a variety of science disciplines using both archived and real-time data. It is currently being used to support the Microwave Limb Sounder (MLS) in orbit aboard the Upper Atmosphere Research Satellite (UARS). This paper describes the application of LinkWinds to this data to rapidly detect features, such as the ozone hole configuration, and to analyze correlations between chemical constituents of the atmosphere.

Jacobson, Allan S.↗

Monitoring Areal Snow Cover Using NASA Satellite Imagery

The objective of this project is to develop products and tools to assist in the hydrologic modeling process, including tools to help prepare inputs for hydrologic models and improved methods for the visualization of streamflow forecasts. In addition, this project will facilitate the use of NASA satellite imagery (primarily snow cover imagery) by other federal and state agencies with operational streamflow forecasting responsibilities. A GIS software toolkit for monitoring areal snow cover extent and producing streamflow forecasts is being developed. This toolkit will be packaged as multiple extensions for ArcGIS 9.x and an opensource GIS software package. The toolkit will provide users with a means for ingesting NASA EOS satellite imagery (snow cover analysis), preparing hydrologic model inputs, and visualizing streamflow forecasts. Primary products include a software tool for predicting the presence of snow under clouds in satellite images; a software tool for producing gridded temperature and precipitation forecasts; and a suite of tools for visualizing hydrologic model forecasting results. The toolkit will be an expert system designed for operational users that need to generate accurate streamflow forecasts in a timely manner. The Remote Sensing of Snow Cover Toolbar will ingest snow cover imagery from multiple sources, including the MODIS Operational Snowcover Data and convert them to gridded datasets that can be readily used. Statistical techniques will then be applied to the gridded snow cover data to predict the presence of snow under cloud cover. The toolbar has the ability to ingest both binary and fractional snow cover data. Binary mapping techniques use a set of thresholds to determine whether a pixel contains snow or no snow. Fractional mapping techniques provide information regarding the percentage of each pixel that is covered with snow. After the imagery has been ingested, physiographic data is attached to each cell in the snow cover image. This data can be obtained from a digital elevation model (DEM) for the area of interest.

Harshburger, Brian J.↗

Use NASA Earthdata in the Cloud

With the impending arrival of new, high-data-volume Earth observation missions, NASA’s ability to effectively ingest, process, and archive large amounts of data requires the most cost-effective, flexible, and scalable data-management architectures and technologies. To meet these demands, NASA's Earth Science Data Systems (ESDS) Program is implementing a strategic vision to develop and operate multiple components of NASA's Earth Observing System Data and Information System (EOSDIS) in a commercial cloud environment. As more earth observing data is moved into the cloud, NASA EOSDIS wants to share our reasoning for moving data into the commercial cloud, as well as demonstrate how end-users can take advantage of both data in the cloud and cloud-deployed services. This presentation explains the enterprise reasons for moving data into the cloud, specifically the scalability of cloud systems, the flexibility to pivot to the most optimal language/system/architecture, and the reduced redundancy of a common architecture. Then we primarily focus on the user-driven reasons for moving to the cloud. Equal access to high performance computing helps all parties, especially users that don't have access to their own on-premise computing infrastructure. Rapid deployment allows users to spin up their own systems without needing the extensive platform previously required for large scale development. Cost effectiveness allows users a wide breadth of computing options, from super computer ability to small scale development, which can help underserved communities. Finally we link to internal resources that users can utilize to begin their cloud based development journey without having to be an expert in the field.

Nicholas Doty↗

Multi-Formalism Modeling for Disaster Resilience, Forecasting, and Response

Disaster response preparedness, risk mitigation, and effective post-event response are critical for national security, public health, and economic integrity. For areas impacted by extreme events, timely and accurate impact assessments of lifeline infrastructures are critical. Traditional disaster management and situational awareness platforms are not capable of handling the massive influx of available information from traditional and non-traditional sources. Next-generation analytic assessment approaches, where use of dynamically sourced, heterogeneous data with near real-time availability are integrated into a continually-updated analytics framework. The objective of this research was to develop a foundational multi-formalism modeling platform to integrate novel data sources for use in improving situational awareness after a disaster event. Specifically, to 1) provide an architecture for machine-learning based data fusion various multi-sensor datasets; 2) classify, geolocate, and temporally bin novel data sources for use in spatially-enabled machine-learning based data fusion; and 3) derive confidence-based observation data with machine-learning based prediction to fill spatial and temporal observation gaps. This project developed a reusable cloud-based pipeline architecture for ingesting, storing, and analyzing relevant data, executing a range of deep-learning models, and visualize the information geographically. The pipeline development was informed by previous PNNL investments (StreamKit) and built upon leading-edge technologies consisting of: 1) a raw datastore; 2) a data enrichment pipeline with custom and third-party analytics; 3) enriched data storage; and 4) a custom UI to display the enriched data. Machine learning/deep learning models were selected and tested to classify social media images for relevancy and presence of flood waters. Our custom trained relevance models were run on a validation dataset and the overall accuracy of the relevance classifiers ranged from 65% to 90%. We found that convolutional neural network (CNN) models were both faster and more accurate than more traditional machine learning models. We applied the relevance classifier to remove non-relevant images and then undertook a limited training of CNN models to detect flood on ~1,800 labeled images. The testing accuracy of the flood detection models were ~88%. To improve the geographic location of both social media and oblique aerial images we tested a multi-step workflow built on a pre-trained deep-learning model. To identify similarity features from test and reference scenes, we applied the Xception and Faiss models to a set of test images with inaccurate spatial locations, as well and precisely-located reference scenes from the general area. Matching reference scenes to the test scenes with the closest similarity index had limited success in improving the precision of test scene location and additional methods have been identified for testing. Data fusion between authoritative geographic data (satellite imaging) and non-authoritative crowdsourced data is an underlying theme throughout this research and methods are tested bring these data to a common geographic form through feature matching and transformations. The automated extraction of supporting information for disaster response from oblique aerial images or ground-level social media images is accomplished through semantic image segmentation and transformation. The non-authoritative data are used to validate the authoritative data collections and processing and optionally, t+0 flood models. In addition, the common form geographic data are combined with other static landscape features to develop spatially-explicit training vectors used in deep generative statistical inference models to develop complex non-linear relationships in static and dynamic data and ultimately derive a spatially continuous and probabilistic assessment of flood damage with the event domain.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Application of Multi-Criteria Decision Analysis Techniques for Informing Select Agent Designation and Decision Making

The Centers for Disease Control and Prevention (CDC) Select Agent Program establishes a list of biological agents and toxins that potentially threaten public health and safety, the procedures governing the possession, utilization, and transfer of those agents, and training requirements for entities working with them. Every 2 years the Program reviews the select agent list, utilizing subject matter expert (SME) assessments to rank the agents. In this study, we explore the applicability of multi-criteria decision analysis (MCDA) techniques and logic tree analysis to support the CDC Select Agent Program biennial review process, applying the approach broadly to include non-select agents to evaluate its generality. We conducted a literature search for over 70 pathogens against 15 criteria for assessing public health and bioterrorism risk and documented the findings for archiving. The most prominent data gaps were found for aerosol stability and human infectious dose by inhalation and ingestion routes. Technical review of published data and associated scoring recommendations by pathogen-specific SMEs was found to be critical for accuracy, particularly for pathogens with very few known cases, or where proxy data (e.g., from animal models or similar organisms) were used to address data gaps. Analysis of results obtained from a two-dimensional plot of weighted scores for difficulty of attack (i.e., exposure and production criteria) vs. consequences of an attack (i.e., consequence and mitigation criteria) provided greater fidelity for understanding agent placement compared to a 1-to-n ranking and was used to define a region in the upper right-hand quadrant for identifying pathogens for consideration as select agents. A sensitivity analysis varied the numerical weights attributed to various properties of the pathogens to identify potential quantitative (x and y) thresholds for classifying select agents. The results indicate while there is some clustering of agent scores to suggest thresholds, there are still pathogens that score close to any threshold, suggesting that thresholding “by eye” may not be sufficient. The sensitivity analysis indicates quantitative thresholds are plausible, and there is good agreement of the analytical results with select agent designations. A second analytical approach that applied the data using a logic tree format to rule out pathogens for consideration as select agents arrived at similar conclusions.

60 APPLIED LIFE SCIENCES↗

Application of multi-criteria decision analysis techniques and decision support framework for informing select agent designation for agricultural animal pathogens

The United States Department of Agriculture (USDA), Division of Agricultural Select Agents and Toxins (DASAT) established a list of biological agents and toxins (Select Agent List) that potentially threaten agricultural health and safety, the procedures governing the transfer of those agents, and training requirements for entities working with them. Every 2 years the USDA DASAT reviews the Select Agent List, using subject matter experts (SMEs) to perform an assessment and rank the agents. To assist the USDA DASAT biennial review process, we explored the applicability of multi-criteria decision analysis (MCDA) techniques and a Decision Support Framework (DSF) in a logic tree format to identify pathogens for consideration as select agents, applying the approach broadly to include non-select agents to evaluate its robustness and generality. We conducted a literature review of 41 pathogens against 21 criteria for assessing agricultural threat, economic impact, and bioterrorism risk and documented the findings to support this assessment. The most prominent data gaps were those for aerosol stability and animal infectious dose by inhalation and ingestion routes. Technical review of published data and associated scoring recommendations by pathogen-specific SMEs was found to be critical for accuracy, particularly for pathogens with very few known cases, or where proxy data (e.g., from animal models or similar organisms) were used to address data gaps. The MCDA analysis supported the intuitive sense that select agents should rank high on the relative risk scale when considering agricultural health consequences of a bioterrorism attack. However, comparing select agents with non-select agents indicated that there was not a clean break in scores to suggest thresholds for designating select agents, requiring subject matter expertise collectively to establish which analytical results were in good agreement to support the intended purpose in designating select agents. The DSF utilized a logic tree approach to identify pathogens that are of sufficiently low concern that they can be ruled out from consideration as a select agent. In contrast to the MCDA approach, the DSF rules out a pathogen if it fails to meet even one criteria threshold. Both the MCDA and DSF approaches arrived at similar conclusions, suggesting the value of employing the two analytical approaches to add robustness for decision making.

60 APPLIED LIFE SCIENCES↗

Basis for the ICRP’s updated biokinetic model for carbon inhaled as CO 2

Here, the International Commission on Radiological Protection (ICRP) is updating its biokinetic and dosimetric models for occupational intake of radionuclides (OIR) in a series of reports called the OIR series. This paper describes the basis for the ICRP's updated biokinetic model for inhalation of radiocarbon as carbon dioxide (CO 2 ) gas. The updated model is based on biokinetic data for carbon isotopes inhaled as carbon dioxide or injected or ingested as bicarbonate $({{{\rm{HCO}}}_{3}}^{-}).$ The data from these studies are expected to apply equally to internally deposited (or internally produced) carbon dioxide and bicarbonate based on comparison of excretion rates for the two administered forms and the fact that carbon dioxide and bicarbonate are largely carried in a common form (CO 2 –H${{{\rm{CO}}}_{3}}^{-})$ in blood. Compared with dose estimates based on current ICRP biokinetic models for inhaled carbon dioxide or ingested carbon, the updated model will result in a somewhat higher dose estimate for 14 C inhaled as CO 2 and a much lower dose estimate for 14 C ingested as bicarbonate.

61 RADIATION PROTECTION AND DOSIMETRY↗

A Modular Framework for Integrating and Visualizing Telemetry for Mars 2020 Rover Mechanism Operations

The analysis of mechanism telemetry requires a wide variety of tools to quickly and effectively assess spacecraft state, capture long-term trends in system performance, and identify and track anomalous events. Such analysis often requires spacecraft telemetry to first be transformed into derived fields and aggregated statistics before operators can begin their analysis. In past missions, aspects of this process have been automated, but operators were expected to use their own tools and procedures to understand and visualize the data, which led to redundant and inconsistent tools and processes. The Mech Data Tools Python library (MDT) was developed to provide a flexible, unified tool set for operators to extract and analyze mechanism telemetry over the life of the Mars 2020 surface mission. MDT consists of a set of configurable components that implement standard interfaces for ingesting input and producing output. Components can be chained together to form a data processing pipeline. Data are ingested from several sources within the greater Mars 2020 cloud infrastructure and stored in pandas DataFrames, which allows users to leverage the data manipulation capabilities present within the widely-used pandas library. Visualization capabilities are provided through the Plotly library, which generates interactive plots for users to interpret. Following the beginning of Mars 2020 surface operations, usage of MDT has spread to all mechanism-focused subsystems and has demonstrated great utility in analyzing early surface activities. This paper describes MDT’s evolution from heritage mechanism telemetry tools, the critical architecture decisions and challenges faced over MDT’s two years of development, and current applications of MDT in support of mechanism operations.

Wolsieffer, Ben↗

Vapor ingestion in Centaur liquid-hydrogen tank

Vapor ingestion phenomena were investigated using scale models of the Centaur liquid hydrogen tank to determine the height of the free surface of the liquid when vapor is intially ingested into the tank outlet. Data are compared with an analysin and, is general the agreement is very good. Predictions are presented for minimum liquid levels required in the Centaur liquid hydrogen tank in order to prevent vapor ingestion when restarting the engines in space and the quantities of liquid remaining in the tank at vapor ingestion during main engine firing.

Symons, E. P.↗

PATHA: Performance Analysis Tool for HPC Applications

Large science projects rely on complex workflows to analyze terabytes or petabytes of data. These jobs are often running over thousands of CPU cores and simultaneously performing data accesses, data movements, and computation. It is difficult to identify bottlenecks or to debug the performance issues in these large workflows. In order to address these challenges, we have developed Performance Analysis Tool for HPC Applications (PATHA) using the state-of-art open source big data processing tools. Our framework can ingest system logs to extract key performance measures, and apply the most sophisticated statistical tools and data mining methods on the performance data. Furthermore, it utilizes an efficient data processing engine to allow users to interactively analyze a large amount of different types of logs and measurements. To illustrate the functionality of PATHA, we conduct a case study on the workflows from an astronomy project known as the Palomar Transient Factory (PTF). This study processed 1.6 TB of system logs collected on the NERSC supercomputer Edison. Using PATHA, we were able to identify performance bottlenecks, which reside in three tasks of PTF workflow with the dependency on the density of celestial objects.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

T.Rex

T.Rex is used to explore tabular data sets containing up to ten million records to help rapidly understand a previously unknown data set. Analysis can quickly identify patterns of interest and the records and fields that capture those patterns. T.Rex contains a growing set of deep analytical tools and supports robust export capabilities that selected data can be incorporated into to other specialized tools for further analysis. T.Rex is flexible in ingesting different types and formats of data, allowing the user to interactively experiment and perform trial and error guesses on the structure of the data; and also has a variety of linked visual analytic tools that enable exploration of the data to find relevant content, relationships among content, trends within the content, and capture knowledge about the content. Finally, T.Rex has a rich export capability, to extract relevant subsets of a larger data source, to further analyze their data in other analytic tools.

Source record↗

Software for Generating Troposphere Corrections for InSAR Using GPS and Weather Model Data

Atmospheric errors due to the troposphere are a limiting error source for spaceborne interferometric synthetic aperture radar (InSAR) imaging. This software generates tropospheric delay maps that can be used to correct atmospheric artifacts in InSAR data. The software automatically acquires all needed GPS (Global Positioning System), weather, and Digital Elevation Map data, and generates a tropospheric correction map using a novel algorithm for combining GPS and weather information while accounting for terrain. Existing JPL software was prototypical in nature, required a MATLAB license, required additional steps to acquire and ingest needed GPS and weather data, and did not account for topography in interpolation. Previous software did not achieve a level of automation suitable for integration in a Web portal. This software overcomes these issues. GPS estimates of tropospheric delay are a source of corrections that can be used to form correction maps to be applied to InSAR data, but the spacing of GPS stations is insufficient to remove short-wavelength tropospheric artifacts. This software combines interpolated GPS delay with weather model precipitable water vapor (PWV) and a digital elevation model to account for terrain, increasing the spatial resolution of the tropospheric correction maps and thus removing short wavelength tropospheric artifacts to a greater extent. It will be integrated into a Web portal request system, allowing use in a future L-band SAR Earth radar mission data system. This will be a significant contribution to its technology readiness, building on existing investments in in situ space geodetic networks, and improving timeliness, quality, and science value of the collected data

Moore, Angelyn W.↗

Integrated Space Asset Management Database and Modeling

Effective Space Asset Management is one key to addressing the ever-growing issue of space congestion. It is imperative that agencies around the world have access to data regarding the numerous active assets and pieces of space junk currently tracked in orbit around the Earth. At the center of this issues is the effective management of data of many types related to orbiting objects. As the population of tracked objects grows, so too should the data management structure used to catalog technical specifications, orbital information, and metadata related to those populations. Marshall Space Flight Center's Space Asset Management Database (SAM-D) was implemented in order to effectively catalog a broad set of data related to known objects in space by ingesting information from a variety of database and processing that data into useful technical information. Using the universal NORAD number as a unique identifier, the SAM-D processes two-line element data into orbital characteristics and cross-references this technical data with metadata related to functional status, country of ownership, and application category. The SAM-D began as an Excel spreadsheet and was later upgraded to an Access database. While SAM-D performs its task very well, it is limited by its current platform and is not available outside of the local user base. Further, while modeling and simulation can be powerful tools to exploit the information contained in SAM-D, the current system does not allow proper integration options for combining the data with both legacy and new M&S tools. This paper provides a summary of SAM-D development efforts to date and outlines a proposed data management infrastructure that extends SAM-D to support the larger data sets to be generated. A service-oriented architecture model using an information sharing platform named SIMON will allow it to easily expand to incorporate new capabilities, including advanced analytics, M&S tools, fusion techniques and user interface for visualizations. In addition, tight control of information sharing policy will increase confidence in the system, which would encourage industry partners to provide commercial data. Combined with the integration of new and legacy M&S tools, a SIMON-based architecture will provide a robust environment that can be extended and expanded indefinitely.

MacLeod, Todd↗

Data Analysis Techniques for Fan Performance in Highly-Distorted Flows from Boundary Layer Ingesting Inlets

The design of a unique distortion-tolerant fan for a high-bypass ratio boundary-layer ingesting propulsion system has been completed and a rig constructed and tested in the NASA Glenn 8’x6’ wind tunnel. Processing the data from the experiment presented some interesting challenges because of the complexity of the experimental setup and the flow through the test rig. The experiment was run in three phases, each of which employed a unique complement of inlet throat and fan face instrumentation to avoid the blockage that would have resulted from simultaneously installing all of the rakes. The measurement from the individual test points were subsequently combined to compute the overall stage performance. A CFD model of the experiment was used to gain understanding of the flow field and to test some of the techniques proposed for interpolating and extrapolating the measurements into regions where measurements were not made. This capability became extremely useful when it was discovered that there was an unexpected total temperature distortion in the tunnel. The CFD model was modified by inserting a total temperature profile at the upstream boundary that mimicked the measured distortion where measurements were available and that CFD solution was used to investigate methods to infer the complete total temperature field at the fan face.

Hardin, Larry W.↗

Satellite-Based Drought Reporting on the Navajo Nation

The Navajo Nation (NN) is the largest reservation in the US, and faces challenges related to water management during long-term and widespread drought episodes. The Navajo Nation is a federally recognized tribe, which has boundaries within Arizona, New Mexico, and Utah. The Navajo Nation has a land area of over 70,000 square kilometers. The Navajo Nation Department of Water Resources (NNDWR) reports on drought and climatic conditions through the use of regional Standardized Precipitation Index (SPI) values and a network of in-situ rainfall, streamflow, and climate data. However, these data sources lack the spatial detail and consistent measurements needed to provide a coherent understanding of the drought regime within the Nation's regional boundaries. This project, as part of NASA's Western Water Applications Office (WWAO), improves upon the recently developed Drought Severity Assessment Tool (DSAT) to ingest satellite-based precipitation data to generate SPI values for specific administrative boundaries within the reservation. The tool aims to: (1) generate SPI values and summary statistics for regions of interest on various timescales, (2) to visualize SPI values within a web-map application, and (3) produce maps and comparative statistical outputs in the format required for annual drought reporting. The co-development of the DSAT with NN partners is integral to increasing the sustained use of Earth Observations for water management applications. This tool will provide data to support the NN in allocation of drought contingency dollars to the regions most adversely impacted by declines in water availability.

Satellite-based↗

dbprocessing

dbprocessing is a python framework for automating data processing pipelines. The dbprocessing package uses a configuration file to define the input, intermediate, and output data products for a particular data type as well as the processes that connect those data products. In addition, dbprocessing uses python scripts called inspectors to determine if a particular file matches a configured data product. If a particular data product is found, it is ingested into the dbprocessing database (sqlite3 or postgres) which then triggers all of the chained processes to the final data product. While the framework is written in python, it can run software in any language but may require a “wrapper” to translate the dbprocessing command line arguments to the form expected by the software.

Walker, Andrew↗