SEARCH · Search NASA

Results for “Data aggregation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

All sources NASA NTRS DOE OSTI

DOE OSTI2023

Secure Data Logging and Processing with Blockchain and Machine Learning (Final Report)

Secure Data Logging and Processing with Blockchain and Machine Learning (ML) research is focused on the development of a platform to securely log and process sensor data in fossil power plants. The platform integrates two emerging technologies, blockchain and ML, and incorporates several innovative mechanisms to ensure the integrity, reliability, and resiliency of power systems. The goal is to protect the power plant from various cyberattacks such as false data injection and denial of service attacks using these technologies. The research goal was enabled by the following Research Project Objectives: 1) Secure authentication and identity verification of sensor nodes, actuators, and other equipment within a network. 2) Development of mechanisms that ensure only data sent by legitimate sensors are accepted and stored in the data repository. 3) Development of data aggregation methodologies using ML / Deep Learning (DL) algorithms to minimize noise / faulty data. 4) Implementation of the blockchain technologies to provide data security using secured IOTA framework & nodes.

20 FOSSIL-FUELED POWER PLANTS↗

NASA NTRS2017

Re-Organizing Earth Observation Data Storage to Support Temporal Analysis of Big Data

The Earth Observing System Data and Information System archives many datasets that are critical to understanding long-term variations in Earth science properties. Thus, some of these are large, multi-decadal datasets. Yet the challenge in long time series analysis comes less from the sheer volume than the data organization, which is typically one (or a small number of) time steps per file. The overhead of opening and inventorying complex, API-driven data formats such as Hierarchical Data Format introduces a small latency at each time step, which nonetheless adds up for datasets with O(10^6) single-timestep files. Several approaches to reorganizing the data can mitigate this overhead by an order of magnitude: pre-aggregating data along the time axis (time-chunking); storing the data in a highly distributed file system; or storing data in distributed columnar databases. Storing a second copy of the data incurs extra costs, so some selection criteria must be employed, which would be driven by expected or actual usage by the end user community, balanced against the extra cost.

data storage↗

DOE OSTI2026

Investigating the Determinants of Household Capabilities Burden During Power Outages: The Case of Winter Storm Uri

Existing research primarily uses census data to identify the vulnerability of communities to hazards. These vulnerability indices provide aggregated data and are not hazard-specific nor well-validated with post-event data. In contrast, our study uses household survey data (n=1065) to understand which Texan households suffered the greatest loss of their capabilities due to power outages and other utility service disruptions during Winter Storm Uri. Inspired by the Capabilities Approach, our measures of burden include the number of household capability types disrupted during the outages (e.g., cooking, heating, refrigeration), the severity of impact for each disrupted capability, and the additional time and financial costs of coping with these disruptions. We perform a clustering analysis, and find two distinct groups in our data, consisting of ‘lesser burden' and ‘heavier burden' households. Results indicate that the households experiencing the heaviest capabilities burden were most likely to experience longer power outages and the loss of water services. They were also more likely to have a Hispanic-Latino household member, lack access to a generator, live in a rented home, have larger households with more young children, fewer adults over 65, lower household incomes, been impacted by the COVID-19 pandemic, and more family characteristics that made life harder. We also fit a logistic regression model to assess the role of outage, household, and community characteristics in predicting differences in capabilities burden. Our results offer insights into enumerating the consequences of utility service disruptions on households, which can inform more targeted and equitable resilience strategies.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2024

Modifying the Asynchronous Jacobi Method for Data Corruption Resilience

Moving scientific computation from high-performance computing (HPC) and cloud computing (CC) environments to devices on the edge, i.e., physically near instruments of interest, has received tremendous interest in recent years. Such edge computing environments can operate on data in situ, offering enticing benefits over data aggregation to HPC and CC facilities that include avoiding costs of transmission, increased data privacy, and real-time data analysis. Because of the inherent unreliability of edge computing environments, new fault-tolerant approaches must be developed before the benefits of edge computing can be realized. Motivated by algorithm-based fault tolerance, a variant of the asynchronous Jacobi (ASJ) method is developed that achieves resilience to data corruption by rejecting solution approximations from neighbor devices according to a bound derived from convergence theory. Numerical results on a two-dimensional Poisson problem show that the new rejection criterion, along with a novel approximation to the shortest path length on which the criterion depends, restores convergence for the ASJ variant in the presence of certain types data corruption. Numerical results are obtained for when the singular values in the analytic bound are approximated. Additional linear systems are also explored, one with a more dense sparsity pattern and one that includes advection. All results indicate that successful resilience to data corruption depends on whether the bound tightens fast enough to reject corrupted data before the iteration evolution deviates significantly from that predicted by the convergence theory defining the bound. This observation generalizes to future work on algorithm-based fault tolerance for other asynchronous algorithms, including upcoming approaches that leverage Krylov subspaces.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Impact Mapping for Geospatial Reasoning and Decision Making

Objective The reported study evaluated a novel approach to aiding geospatial reasoning and decision making. Background Impact mapping aims to alleviate the cognitive demands of geospatial tasks in part by externalizing data in the form of an integrated decision surface. This is achieved by aggregating data across multiple sources of information and visualizing their combined utility rather than objective measurements or individual utility. Previous research has shown that geospatial decisions improve when aided in this manner, but it remains unknown if dynamic decision making, often plagued by fatigue and anchoring bias, would benefit similarly. Method The experiment implemented a systematic manipulation of the presence of a composite impact map and the number of attributes present in a two-stage disaster relief, resource allocation task to investigate when and how impact mapping is beneficial or deleterious to decision makers. Results The presence of the composite impact map increased the utility of selected sites, increased re-planning decisions, reduced information display views, and reduced workload. Generally, the effect of the composite impact map was greater when participants were asked to evaluate more attributes. Conclusion Composite impact maps appear to improve repeated geospatial reasoning and minimize anchoring bias because they alleviate the cognitive demands otherwise necessary to interpret and maintain information from multiple attributes. Application Data visualization techniques, such as impact mapping, can improve repeated geospatial decision making in environments that include high cognitive demand.

Illingworth, David A.↗

DOE OSTI2021

ShopperWorkerRatio

These data characterize spatial and temporal variation in the ratio between shoppers and workers in public places - points of interest - in the United States between 2019 and 2020. The underlying data on foot traffic to public places is collected by SafeGraph, a commercial data aggregator. We estimate the ratio of shoppers to workers based on recorded visit duration.These data may be useful for understanding how use of public spaces have changed during the Coronavirus pandemic, and more generally for understanding activity patterns in public.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

NASA NTRS1993

Mobile and replicated alignment of arrays in data-parallel programs

When a data-parallel language like FORTRAN 90 is compiled for a distributed-memory machine, aggregate data objects (such as arrays) are distributed across the processor memories. The mapping determines the amount of residual communication needed to bring operands of parallel operations into alignment with each other. A common approach is to break the mapping into two stages: first, an alignment that maps all the objects to an abstract template, and then a distribution that maps the template to the processors. We solve two facets of the problem of finding alignments that reduce residual communication: we determine alignments that vary in loops, and objects that should have replicated alignments. We show that loop-dependent mobile alignment is sometimes necessary for optimum performance, and we provide algorithms with which a compiler can determine good mobile alignments for objects within do loops. We also identify situations in which replicated alignment is either required by the program itself (via spread operations) or can be used to improve performance. We propose an algorithm based on network flow that determines which objects to replicate so as to minimize the total amount of broadcast communication in replication. This work on mobile and replicated alignment extends our earlier work on determining static alignment.

Chatterjee, Siddhartha↗

NASA NTRS1993

The alignment-distribution graph

Implementing a data-parallel language such as Fortran 90 on a distributed-memory parallel computer requires distributing aggregate data objects (such as arrays) among the memory modules attached to the processors. The mapping of objects to the machine determines the amount of residual communication needed to bring operands of parallel operations into alignment with each other. We present a program representation called the alignment distribution graph that makes these communication requirements explicit. We describe the details of the representation, show how to model communication cost in this framework, and outline several algorithms for determining object mappings that approximately minimize residual communication.

Chatterjee, Siddhartha↗

NASA NTRS1993

The alignment-distribution graph

Implementing a data-parallel language such as Fortran 90 on a distributed-memory parallel computer requires distributing aggregate data objects (such as arrays) among the memory modules attached to the processors. The mapping of objects to the machine determines the amount of residual communication needed to bring operands of parallel operations into alignment with each other. We present a program representation called the alignment-distribution graph that makes these communication requirements explicit. We describe the details of the representation, show how to model communication cost in this framework, and outline several algorithms for determining object mappings that approximately minimize residual communication.

Chatterjee, Siddhartha↗

DOE OSTI2022

The ABCflux database: Arctic–boreal CO 2 flux observations and ancillary information aggregated to monthly time steps across terrestrial ecosystems

Past efforts to synthesize and quantify the magnitude and change in carbon dioxide (CO 2 ) fluxes in terrestrial ecosystems across the rapidly warming Arctic–boreal zone (ABZ) have provided valuable information but were limited in their geographical and temporal coverage. Furthermore, these efforts have been based on data aggregated over varying time periods, often with only minimal site ancillary data, thus limiting their potential to be used in large-scale carbon budget assessments. To bridge these gaps, we developed a standardized monthly database of Arctic–boreal CO 2 fluxes (ABCflux) that aggregates in situ measurements of terrestrial net ecosystem CO 2 exchange and its derived partitioned component fluxes: gross primary productivity and ecosystem respiration. The data span from 1989 to 2020 with over 70 supporting variables that describe key site conditions (e.g., vegetation and disturbance type), micrometeorological and environmental measurements (e.g., air and soil temperatures), and flux measurement techniques. Here, we describe these variables, the spatial and temporal distribution of observations, the main strengths and limitations of the database, and the potential research opportunities it enables. In total, ABCflux includes 244 sites and 6309 monthly observations; 136 sites and 2217 monthly observations represent tundra, and 108 sites and 4092 observations represent the boreal biome. The database includes fluxes estimated with chamber (19 % of the monthly observations), snow diffusion (3 %) and eddy covariance (78 %) techniques. The largest number of observations were collected during the climatological summer (June–August; 32 %), and fewer observations were available for autumn (September–October; 25 %), winter (December–February; 18 %), and spring (March–May; 25 %). ABCflux can be used in a wide array of empirical, remote sensing and modeling studies to improve understanding of the regional and temporal variability in CO 2 fluxes and to better estimate the terrestrial ABZ CO 2 budget.

59 BASIC BIOLOGICAL SCIENCES↗

NASA NTRS1984

Selection of a map grid for data analysis and archival

Arguments for selection of a map grid are reiterated by illustrating the quantitative effects on data quality caused by using different grids. It is shown that the use of the rectangular latitude-longitude grid actually degrades data quality, increases stored data volume, and increases the complexity of data manipulation. Use of an equal-area grid is shown to be a proper way to aggregate data, but such grids are thought to be inconvenient. A solution to this dilemma is proposed by showing that the analysis and archival map grids need not be the same. The results of a proper analysis of the data on an equal-area grid can be remapped to the more 'convenient' rectangular latitude-longitude grid without loss of quality.

Rossow, W. B.↗

NASA NTRS2005

NASA Earth Sciences Data Support System and Services for the Northern Eurasia Earth Science Partnership Initiative

The presentation describes data management of NASA remote sensing data for Northern Eurasia Earth Science Partnership Initiative (NEESPI). Many types of ground and integrative (e.g., satellite, GIs) data will be needed and many models must be applied, adapted or developed for properly understanding the functioning of Northern Eurasia cold and diverse regional system. Mechanisms for obtaining the requisite data sets and models and sharing them among the participating scientists are essential. The proposed project targets integration of remote sensing data from AVHRR, MODIS, and other NASA instruments on board US- satellites (with potential expansion to data from non-US satellites), customized data products from climatology data sets (e.g., ISCCP, ISLSCP) and model data (e.g., NCEPNCAR) into a single, well-architected data management system. It will utilize two existing components developed by the Goddard Earth Sciences Data & Information Services Center (GES DISC) at the NASA Goddard Space Flight Center: (1) online archiving and distribution system, that allows collection, processing and ingest of data from various sources into the online archive, and (2) user-friendly intelligent web-based online visualization and analysis system, also known as Giovanni. The former includes various kinds of data preparation for seamless interoperability between measurements by different instruments. The latter provides convenient access to various geophysical parameters measured in the Northern Eurasia region without any need to learn complicated remote sensing data formats, or retrieve and process large volumes of NASA data. Initial implementation of this data management system will concentrate on atmospheric data and surface data aggregated to coarse resolution to support collaborative environment and climate change studies and modeling, while at later stages, data from NASA and non-NASA satellites at higher resolution will be integrated into the system.

Leptoukh, Gregory↗

NASA NTRS2006

NASA Earth Sciences Data Support System and Services for the Northern Eurasia Earth Science Partnership Initiative

The presentation describes the recently awarded ACCESS project to provide data management of NASA remote sensing data for the Northern Eurasia Earth Science Partnership Initiative (NEESPI). The project targets integration of remote sensing data from MODIS, and other NASA instruments on board US-satellites (with potential expansion to data from non-US satellites), customized data products from climatology data sets (e.g., ISCCP, ISLSCP) and model data (e.g., NCEP/NCAR) into a single, well-architected data management system. It will utilize two existing components developed by the Goddard Earth Sciences Data & Information Services Center (GES DISC) at the NASA Goddard Space Flight Center: (1) online archiving and distribution system, that allows collection, processing and ingest of data from various sources into the online archive, and (2) user-friendly intelligent web-based online visualization and analysis system, also known as Giovanni. The former includes various kinds of data preparation for seamless interoperability between measurements by different instruments. The latter provides convenient access to various geophysical parameters measured in the Northern Eurasia region without any need to learn complicated remote sensing data formats, or retrieve and process large volumes of NASA data. Initial implementation of this data management system will concentrate on atmospheric data and surface data aggregated to coarse resolution to support collaborative environment and climate change studies and modeling, while at later stages, data from NASA and non-NASA satellites at higher resolution will be integrated into the system.

Leptoukh, Gregory↗

NASA NTRS2017

NASA Wrangler: Automated Cloud-Based Data Assembly in the RECOVER Wildfire Decision Support System

NASA Wrangler is a loosely-coupled, event driven, highly parallel data aggregation service designed to take advantageof the elastic resource capabilities of cloud computing. Wrangler automatically collects Earth observational data, climate model outputs, derived remote sensing data products, and historic biophysical data for pre-, active-, and post-wildfire decision making. It is a core service of the RECOVER decision support system, which is providing rapid-response GIS analytic capabilities to state and local government agencies. Wrangler reduces to minutes the time needed to assemble and deliver crucial wildfire-related data.

decision support↗

DOE OSTI2021

A Multi-criteria CCUS Screening Evaluation of the Gulf of Mexico, USA - Supplementary Data

The Wendt et al. study aims to incorporate multiple and disparate carbon capture, utilization, and storage (CCUS) decision-making criteria into a systematic, quantitative analytical approach to help identify areas with potentially high suitability to serve as offshore CO2 storage or EOR regions. Spatially-distributed data from publicly-available sources within the Gulf of Mexico (GOM) study area (limited to federal waters; state waters were not evaluated) was compiled using the U.S. Department of Energy's (DOE) National Energy Technology Laboratory (NETL) Cumulative Spatial Impact Layers™ (CSIL) tool to easily aggregate data based on evenly-distributed grids across the study region set at a resolution of approximately 25 square miles (65 square kilometers). The data included in this Microsoft Excel™ workbook provide the aggregated scores for each grid point across the study domain as well the weighting for each criterion under the four scenarios evaluated.

CCUS↗

DOE OSTI2021

Accelerating Multigrid-based Hierarchical Scientific Data Refactoring on GPUs

Rapid growth in scientific data and a widening gap between computational speed and I/O bandwidth make it increasingly infeasible to store and share all data produced by scientific simulations. Instead, we need methods for reducing data volumes: ideally, methods that can scale data volumes adaptively so as to enable negotiation of performance and fidelity tradeoffs in different situations. Multigrid-based hierarchical data representations hold promise as a solution to this problem, allowing for flexible conversion between different fidelities so that, for example, data can be created at high fidelity and then transferred or stored at lower fidelity via logically simple and mathematically sound operations. However, the effective use of such representations has been hindered until now by the relatively high costs of creating, accessing, reducing, and otherwise operating on such representations. We describe here highly optimized data refactoring kernels for GPU accelerators that enable efficient creation and manipulation of data in multigrid-based hierarchical forms. We demonstrate that our optimized design can achieve up to 250 TB/s aggregated data refactoring throughput—83% of theoretical peak—on 1024 nodes of the Summit supercomputer. We showcase our optimized design by applying it to a large-scale scientific visualization workflow and the MGARD lossy compression software.

Data refactoring↗

DOE OSTI2021

Accelerating Multigrid-based Hierarchical Scientific Data Refactoring on GPUs

Chen, Jieyang↗

DOE OSTI2024

Federated Machine Learning-Based Anomaly Detection System for Synchrophasor Network Using Heterogeneous Data Sets: Preprint

Synchrophasor technology is widely deployed in the energy management system to monitor the grid health at micro level and perform necessary corrective actions in real time; however, integrated phasor devices and data aggregators are exposed to several cybersecurity threats. This paper proposes a federated ML(FML)-based ADS to detect several data integrity attacks in the synchrophasor network. The proposed approach integrates the horizontal FML technique and consists of substation-based local models and a control center-based global model. The proposed methodology includes training local models using heterogeneous data sets that include network and grid information and updating the global model through multiple iterations by sharing model gradients. Finally, the trained global model is applied to identify cyberattacks, normal operation, and physical events. To validate the proof of concept, we used synthetic data sets generated by Mississippi State University and Oak Ridge National Laboratory for training and testing the classification models using the National Renewable Energy Laboratory's high performance computing resources. Our experimental results, computed through several performance measures, reveal that the proposed approach shows consistent performance during the binary, three-class, and multiclass classifications while ensuring privacy of synchrophasor data.

anomaly detection system↗