Search NASA⌕ Search

SEARCH · Search NASA

Results for “data ingest”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

NASA SNPP SIPS - Following in the Path of EOS

NASA's Earth Science Data Information System (ESDIS) Project has been operating NASA's Suomi National Polar-Orbiting Partnership (SNPP) Science Data Segment (SDS) since the launch in October 2011. At launch, the SDS focused primarily on the evaluation of Sensor Data Records (SDRs) and Environmental Data Records (EDRs) produced by the Joint Polar Satellite System (JPSS), a National Oceanic and Atmosphere Administration (NOAA) Program, as to their suitability for Earth system science. During the summer of 2014, NASA transitioned to the production of standard Earth Observing System (EOS)-like science products for all instruments aboard Suomi NPP. The five Science Investigator-led Processing Systems (SIPS): Land, Ocean, Atmosphere, Ozone, and Sounder were established to produce the NASA SNPP standard Level 1, Level 2, and global Level 3 products developed by the SNPP Science Teams and to provide the products to NASA's Distributed Active Archive Centers (DAACs) for archive and distribution to the user community. The processing, archiving and distribution of data from NASA's Clouds and the Earth's Radiant Energy System (CERES) and Ozone Mapper/Profiler Suite (OMPS) Limb instruments will continue. With the implementation of the JPSS Block 2 architecture and the launch of JPSS-1, the SDS will receive SNPP data in near real-time via the JPSS Stored Mission Data Hub (JSH), as well as JPSS-1 and future JPSS-2 data. The SNPP SIPS will ingest EOS compatible Level 0 data from the EOS Data Operations System (EDOS) element for their data processing, enabling the continuous EOS-SNPP-JPSS Satellite Data Record.

near real time↗

Counter Unmanned Aircraft System Metrics Tool

SAND2023-05387O The Counter Unmanned Aircraft System (CUAS) Metrics Tool consists of three applications: • The Data Collection Tool is designed to be used during testing and captures details that can be imported into the Figure Creator application. • The Unmanned Aircraft System (UAS) Log Converter converts different UAS logs into a standardized hierarchical data format with a timeline of geospatial data. • The Figure Creator develops relevant images for the test event. The application includes logic to ingest and correlate data from the Data Collection Tool and the UAS Log Converter to produce the chosen figures. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Mayle, Ashley↗

Cost effective data system design approach for EOS AM-1

The question of how to design a cost-effective data system for a space-based application with a high aggregate data rate is addressed. The steps used by the EOS AM-1 design team are defined. A summary of the design as of January 1993 is outlined. The AM-1 Science Data system is comprised of one subsystem that ingests, formats, and routes data - the Science Formatting Equipment (SFE), and another that records the data. Key interfaces are the interfaces to the high rate science instruments, the real-time transmitters, and the internal interface between the SFR and the Solid State Recorder. A cost-effective design approach for the AM-1 data system must take into account impacts on ground systems, data processing facilities, availability of data for real-time users, and future spacecraft designs.

Westmeyer, Paul A.↗

Development of a probabilistic timing model for the ingestion of tap water.

A contamination event in a water distribution system can result in adverse health impacts to individuals consuming contaminated water from the system. Assessing impacts to such consumers requires accounting for the timing of exposures of individuals to tap-water contaminants that have time-varying concentrations. Here we present a probabilistic model for the timing of ingestion of tap water that we developed for use in the U.S. Environmental Protection Agency's Threat Ensemble Vulnerability Assessment and Sensor Placement Tool, which is designed to perform consequence assessments for contamination events in water distribution systems. We also present a statistical analysis of the timing of ingestion activity using data collected by the American Time Use Survey. The results of the analysis provide the basis for our model, which accounts for individual variability in ingestion timing and provides a series of potential ingestion times for tap water. It can be combined with a model for ingestion volume to perform exposure assessments and applied in cases for which the use of characteristics typical of the United States is appropriate.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Exploring the Utility of the Planned CYGNSS Mission for Investigating the Initiation and Development of the Madden-Julian Oscillation

CYGNSS is a planned constellation consisting of multiple micro-satellites that leverage the Global Positioning System (GPS) to provide rapidly updated, high resolution (approx. 15-50 km, approx. 4 h) surface wind speeds (via bi-static scatterometry) over the tropical oceans in any weather condition, including heavy rainfall. The approach of the work to be presented at this conference is to utilize a limited-domain, cloud-system resolving model (Weather Research and Forecasting or WRF) and its attendant data assimilation scheme (Three-Dimensional Variational Assimilation or 3DVAR) to investigate the utility of the CYGNSS mission for helping characterize key convectiveto- mesoscale processes - such as surface evaporation, moisture advection and convergence, and upscale development of precipitation systems - that help drive the initiation and development of the Madden-Julian Oscillation (MJO) in the equatorial Indian Ocean. The proposed work will focus on three scientific objectives. Objective 1 is to produce a high-resolution surface wind dataset resolution (approx. 0.5 h, approx. 1-4 km) for multiple MJO onsets using WRF-assimilated winds and other data from the DYNAmics of the MJO (DYNAMO) field campaign, which took place during October 2011 - March 2012. Objective 2 is to study the variability of surface winds during MJO onsets at temporal and spatial scales of finer resolution than future CYGNSS data. The goal is to understand how sub-CYGNSS-resolution processes will shape the observations made by the satellite constellation. Objective 3 is to ingest simulated CYGNSS data into the WRF model in order to perform observing system simulation experiments (OSSEs). These will be used to test and quantify the potential beneficial effects provided by CYGNSS, particularly for characterizing the physical processes driving convective organization and upscale development during the initiation and development of the MJO. The proposed research is ideal for answering important questions about the CYGNSS mission, such as the representativeness of surface wind retrievals in the context of the complex airflow processes that occur during heavy precipitation, as well as the tradeoffs in retrieval accuracy that result from finer spatial resolution of the CYGNSS winds versus increased errors/noisiness in those data. Research plans and initial progress toward these objectives will be presented.

Lang, Timothy↗

Event Definition for the Automated Detection of Nuclear Proliferation Activities

In FY2020, Savannah River National Laboratory (SRNL) in collaboration with the Discovery Analytics Center (DAC) at Virginia Polytechnic Institute and State University (VT) began developing a demonstration prototype system that uses multiple machine learning and data analytic methods on largescale open data sources to identify new, developing, or undeclared nuclear programs. One of the most challenging aspects of applying machine learning techniques to such a problem is the high likelihood of extremely sparse data from disparate sources. To overcome this challenge, the current work will use a strategic combination of supervised, semi-supervised, and unsupervised learning techniques to ingest and fuse data streams to make a forecast of nuclear activities in a targeted geospatial location. Identifying potential data sources and training supervised learning algorithms is dependent upon the development of a robust foundation of targeted event domains that fundamentally define the nuclear activities of interest. This report documents the definition of a hierarchical structure for both nuclear activity and event domains that will be used to guide the research team in development or use of existing semantic dictionaries that are instrumental to searching, parsing, and categorizing events for the forecasting system’s use.

97 MATHEMATICS AND COMPUTING↗

Descriptor: Infrastructure Perception and Control: Multi-Sensor Object Tracking Dataset (IPC-MSOT)

Traffic intersections are crucial and challenging nodes in transportation networks where multiple lanes of vehicles and pedestrians converge. Traffic accidents often occur at traffic intersections, including a large proportion of traffic fatalities and about one-half of all traffic injuries in the United States. Object detection data were collected in 2024 across three intersections in Colorado Springs, CO, USA, over the course of multiple days and various times to induce a heterogeneous mix of traffic conditions and behaviors. The purpose of the data collection exercises was to learn various attributes about infrastructure sensors and to build a repository of high-resolution, object-level data that can be used for research and development (e.g., to develop multisensor data fusion algorithms). The Infrastructure Perception and Control:Multi-Sensor Object tracking (IPC-MSOT) dataset was collected as part of the U.S. Department of Transportation's Strengthening Mobility and Revolutionizing Transportation (SMART) project, where the city of Colorado Springs, Colorado, and the National Renewable Energy Laboratory collaborated to collect object-level trajectory data from road users using multiple types of infrastructure sensors deployed at different intersections. This dataset allows for testing of late-stage sensor fusion algorithms and their ability to ingest multimodal sensor data, and it can be utilized by traffic engineers to design and evaluate trajectory-based signal control strategies.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Detecting outliers in streaming time series data from ARM distributed sensors

The Atmospheric Radiation Measurement (ARM) Data Center at ORNL collects data from a number of permanent and mobile facilities around the globe. The data is then ingested to create high level scientific products. High frequency streaming measurements from sensors and radar instruments at ARM sites require high degree of accuracy to enable rigorous study of atmospheric processes. Outliers in collected data are common due to instrument failure or extreme weather events. Thus, it is critical to identify and flag them. We employed multiple univariate, multivariate and time series techniques for outlier detection methods and studied their effectiveness. First, we examined Pearson correlation coefficient which is used to measure the pairwise correlations between variables. Singular Spectrum Analysis (SSA) was applied to detect outliers by removing the anticipated annual and seasonal cycles from the signal to accentuate anomalies. K-means was applied for multivariate examination of data from collection of sensors to identify any deviation from expected and known patterns and identify abnormal observation. The Pearson correlation coefficient, SSA and K-means methods were later combined together in a framework to detect outliers through a range of checks. We applied the developed method to data from meteorological sensors at ARM Southern Great Plains site and validated against existing database of known data quality issues.

Lu, Yuping↗

Epi Archive: Automated Synthesis of Global Notifiable Disease Data

ObjectiveLANL has built software that automatically collects global notifiable disease data, synthesizes the data, and makes it available to humans and computers within the Biosurveillance Ecosystem (BSVE) as a novel data stream. These data have many applications including improving the prediction and early warning of disease events.IntroductionMost countries do not report national notifiable disease data in a machine-readable format. Data are often in the form of a file that contains text, tables and graphs summarizing weekly or monthly disease counts. This presents a problem when information is needed for more data intensive approaches to epidemiology, biosurveillance and public health.While most nations likely store incident data in a machine-readable format, governments are often hesitant to share data openly for a variety of reasons that include technical, political, economic, and motivational issues1.A survey conducted by LANL of notifiable disease data reporting in over fifty countries identified only a few websites that report data in a machine-readable format. The majority (>70%) produce reports as PDF files on a regular basis. The bulk of the PDF reports present data in a structured tabular format, while some report in natural language.The structure and format of PDF reports change often; this adds to the complexity of identifying and parsing the desired data. Not all websites publish in English, and it is common to find typos and clerical errors.LANL has developed a tool, Epi Archive, to collect global notifiable disease data automatically and continuously and make it uniform and readily accessible.MethodsWe conducted a survey of the national notifiable disease reporting systems notating how the data are reported and in what formats. We determined the minimal metadata that is required to contextualize incident counts properly, as well as optional metadata that is commonly found.The development of software to regularly ingest notifiable disease data and make it available involves three or four main steps: scraping, detecting, parsing and persisting.Scraping: we examine website design and determine reporting mechanisms for each country/website, as well as what varies across the reporting mechanisms. We then designed and wrote code to automate the downloading of the data for each country. We store all artifacts presented as files (PDF, XLSX, etc.) in their original form, along with appropriate metadata for parsing and data provenance.Detecting: This step is required when parsing structured non-machine-readable data such as tabular data in PDF files. We combined the Nurminen methodology of PDF table detection with in-house heuristics to find the desired data within PDF reports2.Parsing: We determined what to extract from each dataset and parsed these data into uniform data structures, correctly accommodating the variations in metadata (e.g., time interval definitions) and the various human languages.Persisting: We store the data in the Epi Archive database and make it available on the internet and through the BSVE. The data is persisted into a structured and normalized SQL database.ResultsThe Epi Archive tool currently contains national and/or subnational notifiable disease data from twenty nations. When a user accesses the Epi Archive site, they are prompted with four fields: country, subregion, disease of interest, and date duration. Upon form submission, a time series is generated from the users’ specifications. The generated graph can then be downloaded into a CSV file if a user is interested in performing personal analysis. Additionally, the data from Epi Archive can be reached through a REST API (Representational State Transfer Application Programming Interface).ConclusionsLANL, as part of a currently funded DTRA effort, is automatically and continually collecting global notifiable disease data. While 20 nations are in production, more are being brought online in the near future. These data are already being utilized and will have many applications including improving the prediction and early warning of disease events.References[1] van Panhuis WG, Paul P, Emerson C, et al. A systematic review of barriers to data sharing in public health. BMC Public Health. 2014. 14:1144. doi:10.1186/1471-2458-14-1144[2] Nurminen, Anssi. "Algorithmic extraction of data in tables in PDF documents." (2013).

Khalsa, Hari S.↗

The Cumulus Ecosystem: Open Source and Beyond to Foster Collaboration

NASA’s Earth Observing System Data and Information System (EOSDIS) open source Cumulus software is designed as a common set of code and services that can be used to create a pipeline to deliver and manage earth science data in the cloud. Cumulus strives to create an ecosystem on the foundation of open source that unites those with shared problems and goals by encouraging users to contribute solutions back to the platform. Large parts of ingesting and managing data are common and much of what is created can be used by others. Our goal is to maximize collaboration and code reuse while allowing users to design a custom solution that meets their needs without having to take on extraneous functionality. In this talk we will describe how the Cumulus ecosystem works beyond just open source software. We will review the technology, the successes and challenges, and the evolution and future of Cumulus as an ecosystem.

Cumulus↗

Celestial Mapping System Videos

The Celestial Mapping System (CMS) is a software platform to generate virtual 3D globes for celestial bodies within our solar system. Multiple planetary data layers can be added to the virtual globe to provide visualization of high-resolution imagery and elevation data, which enables precise measurements, tools for analytical capabilities and a broad range of other functionalities to assist planetary scientists and mission planners. Third-party planetary data can be ingested into CMS with minimal effort. The present focus of CMS is on developing lunar mapping tools to provide features such as: 3D first person view with zoom and navigational capabilities, realistic terrain visualization based on LRO data, measurement tools, Apollo, CLPS and international mission landing site annotations, 3D Models, stereoscopic view, terrain profiling, line of sight analysis, sunlight shading and many more. The application has been developed to provide situational and domain awareness on the Lunar surface, planning capabilities for equipment placement and traverse path optimization.

Mapping↗

Level 1 Processing of MODIS Direct Broadcast Data at the GSFC DAAC

The GSFC DAAC is working to test and package the MODIS Level 1 Processing software for Aqua Direct Broadcast data. This entails the same code base, but different lookup tables for Aqua and Terra. However, the most significant change is the use of ancillary attitude and ephemeris files instead of orbit/attitude information within the science data stream (as with Terra). In addition, we are working on Linux: ports of the algorithms, which could eventually enable processing on PC clusters. Finally, the GSFC DAAC is also working with the GSFC Direct Readout laboratory to ingest Level 0 data from the GSFC DB antenna into the main DAAC, enabling level 1 production in near real time in support of applications users, such as the Synergy project. The mechanism developed for this could conceivably be extended to other participating stations.

Lynnes, Christopher↗

Transitioning NPOESS Data to Weather Offices: The SPoRT Paradigm with EOS Data

Real-time satellite information provides one of many data sources used by NWS weather forecast offices (WFOs) to diagnose current weather conditions and to assist in short-term forecast preparation. While GOES satellite data provides relatively coarse spatial resolution coverage of the continental U.S. on a 10-15 minute repeat cycle, polar orbiting imagery has the potential to provide snapshots of weather conditions at high-resolution in many spectral channels. Additionally, polar orbiting sounding data can provide additional information on the thermodynamic structure of the atmosphere in data sparse regions of at asynoptic observation times. The NASA Short-term Prediction Research and Transition (SPoRT) project has demonstrated the utility of polar orbiting MODIS and AIRS data on the Terra and Aqua satellites to improve weather diagnostics and short-term forecasting on the regional and local scales. SPoRT scientists work directly forecasters at selected WFOS in the Southern Region (SR) to help them ingest these unique data streams into their AWIPS system, understand how to use the data (through on-site and distance learn techniques), and demonstrate the utility of these products to address significant forecast problems. This process also prepares forecasters for the use of similar observational capabilities from NPOESS operational sensors. NPOESS environmental data records (EDRs) from the Visible 1 Infrared Imager I Radiometer Suite (VIIRS), the Cross-track Infrared Sounder (CrlS) and Advanced Technology Microwave Sounder (ATMS) instruments and additional value-added products produced by NESDIS will be available in near real-time and made available to WFOs to extend their use of NASA EOS data into the NPOESS era. These new data streams will be integrated into the NWs's new AWIPS II decision support tools. The AWIPS I1 system to be unveiled in WFOs in 2009 will be a JAVA-based decision support system which preserves the functionality of the existing systems and offers unique development opportunities for new data sources and applications in the Service Orientated Architecture ISOA) environment. This paper will highlight some of the SPoRT activities leading to the integration of VllRS and CrIS/ATMS data into the display capabilities of these new systems to support short-term forecasting problems at WFOs.

Jedlovec, Gary↗

GUIDE: A Scalable Information Directory Service to Collect, Federate, and Analyze Logs for Operational Insights into a Leadership HPC Facility

In this paper, we describe the GUIDE framework to collect, federate, and analyze log data from the Oak Ridge Leadership Computing Facility's (OLCF), and how we use it derive insights into center operations. We collectsystem logs and extract monitoring data at every level of the various OLCF subsystems, and have developed a suite of pre-processing tools to make the raw data consumable. The cleansed logs are then ingested and federated into a central, scalable data warehouse, Splunk, that offers storage, indexing, querying, and visualization capabilities. We have further developed and deployed a set of analytics tools to analyze these multiple disparate log streams in concert, and derive operational insights. We describe our experience from developing and deploying the GUIDE infrastructure, and deriving valuable insights on the various subsystems, based on two years of operations in the production OLCF environment.

Vazhkudai, Sudharshan↗

Laser Remote Sensing from ISS: CATS Cloud and Aerosol Level 2 Data Products (Heritage Edition)

With the recent launch of the Cloud-Aerosol Transport System (CATS) we have the opportunity to acquire a continuous record of space based lidar measurements spanning from the Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations (CALIPSO) era to the start of the EarthCARE mission. Utilizing existing well-validated science algorithms from the CALIPSO mission, we will ingest the CATS data stream and deliver high-quality lidar data sets to the user community at the earliest possible opportunity. In this paper we present an overview of procedures necessary to generate CALIPSO-like lidar level 2 data products from the CATS level 1 data products.

cloud-aerosol lidar↗

GOES-R AWG GLM Val Tool Development

We are developing tools needed to enable the validation of the Geostationary Lightning Mapper (GLM). In order to develop and test these tools, we have need of a robust, high-fidelity set of GLM proxy data. Many steps have been taken to ensure that the proxy data are high quality. LIS is the closest analog that exists for GLM, so it has been used extensively in developing the GLM proxy. We have verified the proxy data both statistically and algorithmically. The proxy data are pixel (event) data, called Level 1B. These data were then clustered into flashes by the Lightning Cluster-Filter Algorithm (LCFA), generating proxy Level 2 data. These were then compared with the data used to generate the proxy, and both the proxy data and the LCFA were validated. We have developed tools to allow us to visualize and compare the GLM proxy data with several other sources of lightning and other meteorological data (the so-called shallow-dive tool). The shallow-dive tool shows storm-level data and can ingest many different ground-based lightning detection networks, including: NLDN, LMA, WWLLN, and ENTLN. These are presented in a way such that it can be seen if the GLM is properly detecting the lightning in location and time comparable to the ground-based networks. Currently in development is the deep-dive tool, which will allow us to dive into the GLM data, down to flash, group and event level. This will allow us to assess performance in comparison with other data sources, and tell us if there are detection, timing, or geolocation problems. These tools will be compatible with the GLM Level-2 data format, so they can be used beginning on Day 0.

Bateman, Monte↗

Underwater Target Detection Software Demonstration on the RivGen Turbine

This repository contains data and processing scripts necessary to train the object detection models utilized in the underwater target detection software demonstration on the RivGen turbine project and to produce performance metrics (precision, recall, mAP50, mAP50-95). - Contents - Data consist of "images" and "labels". Each image has an associated label, both share the same time string in its file name (e.g., 2024_05_25_09_01_57.98.jpg and 2024_05_25_09_01_57.98.txt). Time strings have the format %yyyy_%mm_%dd_%HH_%MM_%SS.%3f. Images and labels were curated from 2021 and 2024 smolt outmigration periods at the project site in Igiugig, AK. Images are monochrome 8-bit images of objects (smolt, debris, and other) passing through the field of view of the deployed cameras during various operational stages of the RivGen turbine. Labels are text files indicating the class and bounding polygon of each object in an image. The provided labels use the "YOLO" label format. - Requirements - Python3.8+ is required to install and run the train and validation script. The README.md provides instruction for installing the requirements from the requirements.py file. - Instructions - The "example_train.py" file ingests the provided data, trains a model, and produces model performance metrics at completion. NOTE: model performance metrics will vary from run to run as a consequence of the random selection of training and validation data.

16 TIDAL AND WAVE POWER↗

The impact of multi-sensor land data assimilation on river discharge estimation

River discharge is one of the most critical renewable water resources. Accurately estimating river discharge with land surface models (LSMs) remains challenging due to the difficulty in estimating land water storages such as snow, soil moisture, and groundwater. While data assimilation (DA) ingesting optical, microwave, and gravity measurements from space can help constrain theses storage states, its impacts on runoff and eventually river discharge are not fully understood. In this study, by taking advantage of recently published land DA results that jointly assimilate eight different combinations of observations from the Moderate Resolution Imaging Spectroradiometer (MODIS), Gravity Recovery and Climate Experiment (GRACE), and Advanced Microwave Scanning Radiometer for EOS (AMSR-E), we quantify to what degree multi-sensor land DA improves the river discharge simulation skills over 40 global river basins, and investigate the complementary strengths of different satellite measurements on river discharge. To be more specific, river discharge is updated by feeding gridded runoff from the eight multi-sensor DA simulations into a vector-based river routing model named the Routing Application for Parallel computatIon of Discharge (RAPID). Our modeling results, including 7-year simulations at 177,458 river reaches globally, are used to study the seasonal to interannual variability of river discharge. It is found that assimilating GRACE has the greatest impact on global runoff patterns, leading to the most pronounced improvements in spatial river discharge in the middle and high latitudes with the R 2 increased by 0.16. The seasonal variation of spatial discharge is most skillful during the boreal summer. However, our evaluation also shows model and DA still struggle to generate reasonable variability and averaged discharge over permafrost regions. Finally, by assessing how different satellites add value to discharge forecasts, this study paves the way for more advanced multi-sensor satellite data assimilation to predict the terrestrial hydrological cycle.

54 ENVIRONMENTAL SCIENCES↗