Search NASA⌕ Search

SEARCH · Search NASA

Results for “data,”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11

Performance Analysis of Data Processing in Distributed File Systems with Near Data Processing

In the era of big data, the escalating volume and velocity of data generation pose significant challenges in data processing. Traditional systems like Spark and Hadoop manage the increasing amount and velocity of data by improving data placement and processing speeds. However, they face inherent limitations due to the essential data movement required for processing. In this paper, we explore the Skyhook framework, a novel extension of the Ceph distributed system, which significantly reduces the need for data movement. We present an extensive case study using the Skyhook framework, applying it with the TPC-H and K-means clustering algorithms. More specifically, we leverage the TPC-H benchmark to distinguish between CPU-intensive and I/O-intensive tasks. We explore the integration of K-means clustering into SQL, coupled with a near-data processing system to offload the computational burden of the K-means clustering algorithm to storage nodes. We conduct a comprehensive performance evaluation of distributed data processing applications across three processing approaches: traditional layout (baseline), optimized layout, and near-data processing. Additionally, we introduce the use of the FIO tool to simulate real-world system workloads, enabling the measurement of performance metrics such as average latency and CPU utilization. Our research is a significant advance in understanding how to optimize data processing systems to meet the demands of the modern data landscape.

Hou, Shiyue↗

GridDS: Data Science Toolkit for Energy Grid Data

According to the U.S. Energy Information Administration (EIA), the demand for energy is expected to increase 50% by the year 20501. While energy standards, such as the Institute of Electrical and Electronics Engineers (IEEE) Standard 1547, (Basso 2015) and monitoring with wide area management systems (WAMS) (Liu 2017, Zhou 2016) have enabled large scale data collection and storage, the application of this data in mitigating costs associated with increased consumer demand is an ongoing focus for energy research. This ubiquitous data collection presents a promising opportunity for machine learning and data science to improve efficiency of distributed energy resources (DERs). The GridDS software toolkit is designed to leverage advanced metering infrastructure (AMI), outage management systems data (OMS), Supervisory control Data Acquisition (SCADA), and geographic information systems (GIS) to forecast future energy demands and detect incipient grid failures. GridDS is a python software library designed to be modular and generalizable to data recorded by DERs. In adapting to disparate datasets recorded by various WAMS, GridDS provides a range of unique functionality not presently implemented in current WAMS which have highly specific software infrastructure by design. GridDS functionality ranges from data specification and preparation, to training and validation for state of the art machine learning, to interactive data visualization. For data intake, GridDS combines: Pandera: a library for creating data specifications. TimeScaleDB: a postgresSQL database infrastructure for efficient storage of timeseries data. Dataset class: A custom dataset class / interface that ensures modularity between a range of synthetic and live recorded datasets. Is

Ladd, Alexander↗

Real-Time Highly Resolved Spatial-Temporal Vehicle Energy Consumption Estimation Using Machine Learning and Probe Data

Real-time highly resolved spatial-temporal vehicle energy consumption is a key missing dimension in transportation data. Most roadway link-level vehicle energy consumption data are estimated using average annual daily traffic measures derived from the Highway Performance Monitoring System; however, this method does not reflect day-to-day energy consumption fluctuations. As transportation planners and operators are becoming more environmentally attentive, they need accurate real-time link-level vehicle energy consumption data to assess energy and emissions; to incentivize energy-efficient routing; and to estimate energy impact caused by congestion, major events, and severe weather. This paper presents a computational workflow to automate the estimation of time-resolved vehicle energy consumption for each link in a road network of interest using vehicle probe speed and count data in conjunction with machine learning methods in real time. The real-time pipeline can deliver energy estimates within a couple seconds on query to its interface. The proposed method was evaluated on the transportation network of the metropolitan area of Chattanooga, Tennessee. The volume estimation results were validated with ground truth traffic volume data collected in the field. To demonstrate the effectiveness of the proposed method, the energy consumption pipeline was applied to real-world data to quantify road transportation-related energy reduction because of mitigation policies to slow the spread of COVID-19 and to measure energy loss resulting from congestion.

Severino, Joseph↗

Open Source Scalable Data Services and Data Fusion for Biological and Environmental Sciences (SBIR Phase I Final Scientific/ Technical Report)

The overarching goal of the project is to develop an integrated open-source scientific data management system (Apache V2 license), ResonantEco, that meets the need of biological and environmental researchers and developers for data management, curation, and data processing for analyses with a wide range of scale and complexity. ResonantEco will provide web enabled data services with features such as unified data interfaces and federated views of data and metadata for heterogeneous data sources with an interactive web client for data exploration. Our use of the term fusion is taken from geospatial (GIS) domain where data fusion is often synonymous with data integration. In particular, data integration in ResonantEco involves combining data residing in different sources and providing users with a unified view of them.

99 GENERAL AND MISCELLANEOUS↗

Integrating Applied Energy and BER Smart Data Capabilities to Develop a DOE Data Fabric for Energy-Water R&D

Focal Area(s): 1) Data acquisition and assimilation enabled by machine learning, AI, and advanced methods including experimental/network design/optimization, unsupervised learning (including deep learning), and hardware-related efforts involving AI (e.g., edge computing). Science Challenge: DOE R&D, including DOE’s Basic Energy Research (BER)’s Environmental Systems Science Division (EESSD) program and DOE’s applied energy research (AER) programs (EERE, FE, and NE) are producers and consumers of Earth systems datasets. This white paper focuses on the first topic area from the call in relation to how crosscutting resources and innovations from DOE’s EESSD and AER can be brought to bear to mutual benefit and more efficient energy-water, Earth system data resources through improved. The overarching challenge posed by this call focuses on how DOE can directly leverage artificial intelligence (AI) to engineer a substantial (paradigm-changing) improvement in Earth System Predictability? While stemming from DOE BER’s EESSD program, this is a challenge that is faced and also being addressed by DOE’s AER programs. Over the past decade plus, FE, EERE, and NE programs have made important strides towards addressing this need. These strides are in many ways highly complementary to EESSD’s MODEX efforts. Energy water systems spanning metocean to groundwater to surface water systems all are data driven whether for basic energy or applied energy. These are remote, multi-variate, complex natural, and in many cases engineered, systems. Key needs and challenges of both EESSD and AER include developing data-focused tools to enhance data search and discovery to fill in knowledge gaps (address sparse data challenge), and rapidly transform datasets, including disparate and multi-source data. Leveraging DOE on-premise computing (HPC, exascale) infrastructure supports the computing-intensive algorithms required to execute these data acquisition and transformation processes to derive enriched knowledge and data, driving AI/ML and big data analytics for these systems. The opportunity lies in combining BER and AER efforts to provide a more robust, advanced, efficient and complete computing data fabric to address energy-water data acquisition and assimilation needs which currently pose significant impediments to AI/ML predictions and research.

54 ENVIRONMENTAL SCIENCES↗

Super-Resolution for Renewable Energy Resource Data with Wind from Reanalysis Data and Application to Ukraine

With a potentially increasing share of the electricity grid relying on wind to provide generating capacity and energy, there is an expanding global need for historically accurate, spatiotemporally continuous, high-resolution wind data. Conventional downscaling methods for generating these data based on numerical weather prediction have a high computational burden and require extensive tuning for historical accuracy. In this work, we present a novel deep learning-based spatiotemporal downscaling method using generative adversarial networks (GANs) for generating historically accurate high-resolution wind resource data from the European Centre for Medium-Range Weather Forecasting Reanalysis version 5 data (ERA5). In contrast to previous approaches, which used coarsened high-resolution data as low-resolution training data, we use true low-resolution simulation outputs. We show that by training a GAN model with ERA5 as the low-resolution input and Wind Integration National Dataset Toolkit (WTK) data as the high-resolution target, we achieved results comparable in historical accuracy and spatiotemporal variability to conventional dynamical downscaling. This GAN-based downscaling method additionally reduces computational costs over dynamical downscaling by two orders of magnitude. We applied this approach to downscale 30 km, hourly ERA5 data to 2 km, 5 min wind data for January 2000 through December 2023 at multiple hub heights over Ukraine, Moldova, and part of Romania. With WTK coverage limited to North America from 2007–2013, this is a significant spatiotemporal generalization. The geographic extent centered on Ukraine was motivated by stakeholders and energy-planning needs to rebuild the Ukrainian power grid in a decentralized manner. This 24-year data record is the first member of the super-resolution for renewable energy resource data with wind from the reanalysis data dataset (Sup3rWind).

17 WIND ENERGY↗

Pilot climate data system: A state-of-the-art capability in scientific data management

The Pilot Climate Data System (PCDS) was developed by the Information Management Branch of NASA's Goddard Space Flight Center to manage a large collection of climate-related data of interest to the research community. The PCDS now provides uniform data catalogs, inventories, access methods, graphical displays and statistical calculations for selected NASA and non-NASA data sets. Data manipulation capabilities were developed to permit researchers to easily combine or compare data. The current capabilities of the PCDS include many tools for the statistical survey of climate data. A climate researcher can examine any data set of interest via flexible utilities to create a variety of two- and three-dimensional displays, including vector plots, scatter diagrams, histograms, contour plots, surface diagrams and pseudo-color images. The graphics and statistics subsystems employ an intermediate data storage format which is data-set independent. Outside of the graphics system there exist other utilities to select, filter, list, compress, and calculate time-averages and variances for any data of interest. The PCDS now fully supports approximately twenty different data sets and is being used on a trial basis by several different in-house research grounds.

Smith, P. H.↗

Guidelines for submitting data to the National Space Science Data Center

The mission of the National Space Science Data Center (NSSDC) is to disseminate space science data for further analysis beyond that provided by the principal investigators (PIs) or team leaders (TLs) and their coworkers. Consequently, the NSSDC is responsible for the acquisition, organization, storage, retrieval, announcement, and distribution of scientific data obtained mainly from satellites and spacecraft. Any scientist may acquired data from the NSSDC and use them in further studies, either alone or in conjunction with data from ground-based or spacecraft experiments. With the responsibility for archiving data is the concomitant responsibility for distributing the documentation necessary to make those data usable. Since the group most knowledgeable about a particular experiment and its data is the PI or TL and his coworkers, and since the NSSDC cannot possibly supply the qualified personnel needed to write this documentation comprehensively, it is the responsibility of the PI or TL to provide the essential documentation. The NSSDC will support this effort by defining what is needed, by reviewing what is provided, and by reproducing and distributing the resulting documentation with the data. For a high-use data set, the NSSDC may publish the documentation as a Data Users Note; for a low-use data set, the NSSDC may distribute a Xerox, microfilm, or microfiche copy of the documentation.

Source record↗

Residual acceleration data on IML-1: Development of a data reduction and dissemination plan

The main thrust of our work in the third year of contract NAG8-759 was the development and analysis of various data processing techniques that may be applicable to residual acceleration data. Our goal is the development of a data processing guide that low gravity principal investigators can use to assess their need for accelerometer data and then formulate an acceleration data analysis strategy. The work focused on the flight of the first International Microgravity Laboratory (IML-1) mission. We are also developing a data base management system to handle large quantities of residual acceleration data. This type of system should be an integral tool in the detailed analysis of accelerometer data. The system will manage a large graphics data base in the support of supervised and unsupervised pattern recognition. The goal of the pattern recognition phase is to identify specific classes of accelerations so that these classes can be easily recognized in any data base. The data base management system is being tested on the Spacelab 3 (SL3) residual acceleration data.

Rogers, Melissa J. B.↗

Determination of an Optimal Commercial Data Bus Architecture for a Flight Data System

NASA/Marshall Space Flight Center (MSFC) is continually looking for methods to reduce cost and schedule while keeping the quality of work high. MSFC is NASA's lead center for space transportation and microgravity research. When supporting NASA's programs several decisions concerning the avionics system must be made. Usually many trade studies must be conducted to determine the best ways to meet the customer's requirements. When deciding the flight data system, one of the first trade studies normally conducted is the determination of the data bus architecture. The schedule, cost, reliability, and environments are some of the factors that are reviewed in the determination of the data bus architecture. Based on the studies, the data bus architecture could result in a proprietary data bus or a commercial data bus. The cost factor usually removes the proprietary data bus from consideration. The commercial data bus's range from Versa Module Eurocard (VME) to Compact PCI to STD 32 to PC 104. If cost, schedule and size are prime factors, VME is usually not considered. If the prime factors are cost, schedule, and size then Compact PCI, STD 32 and PC104 are the choices for the data bus architecture. MSFC's center director has funded a study from his discretionary fund to determine an optimal low cost commercial data bus architecture. The goal of the study is to functionally and environmentally test Compact PCI, STD 32 and PC 104 data bus architectures. This paper will summarize the results of the data bus architecture study.

Crawford, Kevin↗

Extending the LWS Data Environment: Distributed Data Processing and Analysis

The final stages of this work saw changes to the original framework, as well as the completion and integration of several data processing services. Initially, it was thought that a peer-to-peer architecture was necessary to make this work possible. The peer-to-peer architecture provided many benefits including the dynamic discovery of new services that would be continually added. A prototype example was built and while it showed promise, a major disadvantage was seen in that it was not easily integrated into the existing data environment. While the peer-to-peer system worked well for finding and accessing distributed data processing services, it was found that its use was limited by the difficulty in calling it from existing tools and services. After collaborations with members of the data community, it was determined that our data processing system was of high value and that a new interface should be pursued in order for the community to take full advantage of it. As such; the framework was modified from a peer-to-peer architecture to a more traditional web service approach. Following this change multiple data processing services were added. These services include such things as coordinate transformations and sub setting of data. Observatory (VHO), assisted with integrating the new architecture into the VHO. This allows anyone using the VHO to search for data, to then pass that data through our processing services prior to downloading it. As a second attempt at demonstrating the new system, a collaboration was established with the Collaborative Sun Earth Connector (CoSEC) group at Lockheed Martin. This group is working on a graphical user interface to the Virtual Observatories and data processing software. The intent is to provide a high-level easy-to-use graphical interface that will allow access to the existing Virtual Observatories and data processing services from one convenient application. Working with the CoSEC group we provided access to our data processing tools from within their software. This now allows the CoSEC community to take advantage of our services and also demonstrates another means of accessing our system.

Narock, Thomas↗

Transitioning NPOESS Data to Weather Offices: The SPoRT Paradigm with EOS Data

Real-time satellite information provides one of many data sources used by NWS weather forecast offices (WFOs) to diagnose current weather conditions and to assist in short-term forecast preparation. While GOES satellite data provides relatively coarse spatial resolution coverage of the continental U.S. on a 10-15 minute repeat cycle, polar orbiting imagery has the potential to provide snapshots of weather conditions at high-resolution in many spectral channels. Additionally, polar orbiting sounding data can provide additional information on the thermodynamic structure of the atmosphere in data sparse regions of at asynoptic observation times. The NASA Short-term Prediction Research and Transition (SPoRT) project has demonstrated the utility of polar orbiting MODIS and AIRS data on the Terra and Aqua satellites to improve weather diagnostics and short-term forecasting on the regional and local scales. SPoRT scientists work directly forecasters at selected WFOS in the Southern Region (SR) to help them ingest these unique data streams into their AWIPS system, understand how to use the data (through on-site and distance learn techniques), and demonstrate the utility of these products to address significant forecast problems. This process also prepares forecasters for the use of similar observational capabilities from NPOESS operational sensors. NPOESS environmental data records (EDRs) from the Visible 1 Infrared Imager I Radiometer Suite (VIIRS), the Cross-track Infrared Sounder (CrlS) and Advanced Technology Microwave Sounder (ATMS) instruments and additional value-added products produced by NESDIS will be available in near real-time and made available to WFOs to extend their use of NASA EOS data into the NPOESS era. These new data streams will be integrated into the NWs's new AWIPS II decision support tools. The AWIPS I1 system to be unveiled in WFOs in 2009 will be a JAVA-based decision support system which preserves the functionality of the existing systems and offers unique development opportunities for new data sources and applications in the Service Orientated Architecture ISOA) environment. This paper will highlight some of the SPoRT activities leading to the integration of VllRS and CrIS/ATMS data into the display capabilities of these new systems to support short-term forecasting problems at WFOs.

Jedlovec, Gary↗

NASA's Global Change Master Directory: Discover and Access Earth Science Data Sets, Related Data Services, and Climate Diagnostics

NASA's Global Change Master Directory provides the scientific community with the ability to discover, access, and use Earth science data, data-related services, and climate diagnostics worldwide. The GCMD offers descriptions of Earth science data sets using the Directory Interchange Format (DIF) metadata standard; Earth science related data services are described using the Service Entry Resource Format (SERF); and climate visualizations are described using the Climate Diagnostic (CD) standard. The DIF, SERF and CD standards each capture data attributes used to determine whether a data set, service, or climate visualization is relevant to a user's needs. Metadata fields include: title, summary, science keywords, service keywords, data center, data set citation, personnel, instrument, platform, quality, related URL, temporal and spatial coverage, data resolution and distribution information. In addition, nine valuable sets of controlled vocabularies have been developed to assist users in normalizing the search for data descriptions. An update to the GCMD's search functionality is planned to further capitalize on the controlled vocabularies during database queries. By implementing a dynamic keyword "tree", users will have the ability to search for data sets by combining keywords in new ways. This will allow users to conduct more relevant and efficient database searches to support the free exchange and re-use of Earth science data. http://gcmd.nasa.gov/

Aleman, Alicia↗

Examining Dense Data Usage near the Regions with Severe Storms in All-Sky Microwave Radiance Data Assimilation and Impacts on GEOS Hurricane Analyses

Many numerical weather prediction (NWP) centers assimilate radiances affected by clouds and precipitation from microwave sensors, with the expectation that these data can provide critical constraints on meteorological parameters in dynamically sensitive regions to make significant impacts on forecast accuracy for precipitation. The Global Modeling and Assimilation Office (GMAO) at NASA Goddard Space Flight Center assimilates all-sky microwave radiance data from various microwave sensors such as all-sky GPM Microwave Imager (GMI) radiance in the Goddard Earth Observing System (GEOS) atmospheric data assimilation system (ADAS), which includes the GEOS atmospheric model, the Gridpoint Statistical Interpolation (GSI) atmospheric analysis system, and the Goddard Aerosol Assimilation System (GAAS). So far, most of NWP centers apply same large data thinning distances, that are used in clear-sky radiance data to avoid correlated observation errors, to all-sky microwave radiance data. For example, NASA GMAO is applying 145 km thinning distances for most of satellite radiance data including microwave radiance data in which all-sky approach is implemented. Even with these coarse observation data usage in all-sky assimilation approach, noticeable positive impacts from all-sky microwave data on hurricane track forecasts were identified in GEOS-5 system. The motivation of this study is based on the dynamic thinning distance method developed in our all-sky framework to use of denser data in cloudy and precipitating regions due to relatively small spatial correlations of observation errors. To investigate the benefits of all-sky microwave radiance on hurricane forecasts, several hurricane cases selected between 2016-2017 are examined. The dynamic thinning distance method is utilized in our all-sky approach to understand the sources and mechanisms to explain the benefits of all-sky microwave radiance data from various microwave radiance sensors like Advanced Microwave Sounder Unit (AMSU-A), Microwave Humidity Sounder (MHS), and GMI on GEOS-5 analyses and forecasts of various hurricanes.

Kim, Min-Jeong↗

NASA Earth Science Data Systems: Open Data, Services and Software

Open, Public, Electronic and Necessary (OPEN) Government Data Act, which requires all non-sensitive government data to be made available in open and machine-readable formats by default is part of the overall Foundations for Evidence-Based Policymaking (FEBP) Act passed in late 2018. This town hall will bring together data officers and policy makers from NOAA, EPA, NASA and others to discuss the impact of the act on data management strategies going forward. In 1994 NASA's Earth Science Division committed to an open data policy for all civilian Earth satellite data. NASA's Earth Observing System Data and Information System (EOSDIS) became the first large scale data system to facilitate public access to global Earth system data and information. This presentation reviews key elements of EOSDIS data policy and data management activities that support the OPEN Government Data Act.

Moses, John F.↗

Extravehicular Activity Mission System Software (EMSS) - Enabling Human Planetary Exploration Data Within The Broader Planetary Data Ecosystem

The planetary science community is once again on the verge of generating, capturing and analyzing human planetary exploration data, this time via the Artemis program. Artemis missions will involve robotic missions in addition to human extravehicular activity (EVA) where crew will be generating scientific data [1]. Present-day robotic mission data expectations for data archiving involves ingesting data into the Planetary Data System (PDS), but how might PDS be leveraged/adapted/ready (or not) for human spaceflight mission data, particularly EVA data that includes non-scientific data that provides important context to the scientific data gathered on the lunar surface? This question has broader implications than what this abstract can answer, but we wanted to pose the question to 1) get conversations started and 2) highlight how operations software data handling could play a role in overall data curation.

M J Miller↗

Automated Data Accountability for Missions in Mars Rover Data

As the Mars Curiosity Rover transmits data to the JPL Ground Data System (GDS), it frequently observes data loss and corruption, requiring re-transmits from the rover and Ground Data System Analysts (GDSA) to monitor the downlink process. As new missions are launched, the GDSA team redistributes analysts to these new missions, causing shortages in previous missions. The GDSA team can significantly benefit from the automation and optimization of the downlink process of telemetry data. In fact, there is a need for a better understanding of why the data is corrupted, so that the GDSA team can best determine the root cause of the issues in the GDS. This paper presents machine learning and deep learning based approaches to automate and optimize the detection of data loss. We first created a pipeline to automatically accumulate data from the telemetry databases (MAROS, Telemetry Data Storage, and GDS Elastic Search Database) in the downlink process. With our newly created datasets, we perform feature selection to supplement the GDSA understanding of the downlink process and provide supplemental analysis on the importance of different features. We implement various machine learning and deep learning based models, including support vector machines, ensemble methods, and deep neural networks and evaluate their accuracies in identifying whether a downlink process is complete or incomplete. We utilize fast hyperparameter optimization methods that allow our models to quickly be re-trained, allowing them to quickly be tuned and optimized on daily incoming data in real time. This hyperparameter optimization also allows our methods to be quickly integrated into other JPL missions. Our results show that our best-performing machine learning and deep learning based models outperform the existing GDSA detection software by 6 accuracy points and can aid analysts by providing insights into the data accountability problem. Since these various machine learning and deep learning approaches vary significantly in interpretability, we provide a discussion on the tradeoffs between their performance and trustworthiness in helping detect issues in data transmission.

Divsalar, Dariush↗

NASA’s Human Data Repositories: An In Depth Look at the New Data Request Process

As NASA transitions its focus to travel back to the moon and on to new destinations, the need to ensure the capture, analysis, and application of research and medical data is of greater urgency than at any other previous time. In this era of limited resources and challenging schedules, the Human Research Program (HRP), based at NASA’s Johnson Space Center (JSC), recognizes the need to extract the greatest possible amount of information from the data already captured. To this end, the HRP Chief Scientist Office (CSO), HRP Program Planning and Control (PP&C) Office, and the Space Medicine Operations Division have been working together to make reuse of both research data and medical monitoring data more accessible to the user community through the Life Science Data Archive (LSDA) and the Lifetime Surveillance of Astronaut Health (LSAH) Repositories. The task of both LSDA and LSAH repositories is to acquire, preserve, and distribute retrospective research (LSDA) and medical (LSAH) data and information both within the NASA community and to the science community at large, for knowledge discovery, retrospective analysis, and planning of future research studies. An additional goal is to encourage collaboration with non-NASA institutions also faced with enhancing human performance in extreme environments. In September 2022, the LSDA website and its contents transitioned to a new NASA Life Sciences Portal (https://nlsp.nasa.gov/explore/lsdahome). This site continues to feature publicly releasable information such as non-attributable datasets, experiment descriptions (from Project Mercury to ISS, as well as from multiple flight analog missions), descriptions of medical monitoring data, and LSAH newsletters (1992 - 2022). The website also provides an updated portal to request additional research and medical data not accessible from the public website. This presentation will provide an in-depth look at the new system as it relates to finding and requesting retrospective data. We will also detail processes from making a request to delivering data for different types of data requests (i.e., attributable, or non-attributable). This includes descriptions of various approval boards, what information and actions the requestor is responsible for, and key milestones in making data available for reuse.

D. M. Thomas↗