Search NASA⌕ Search

SEARCH · Search NASA

Results for “big data and analytics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Development of Explainable, Knowledge-Guided AI Models to Enhance the E3SM Land Model Development and Uncertainty Quantification

Focal Area(s): (2)Predictive modeling using AI techniques and AI-derived model components; use of AI and other tools to design a prediction system comprising of a hierarchy of models. (3) Insight gleaned from complex data (both observed and simulated) using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge- guided AI. Science Challenge: The Energy Exascale Earth System Model (E3SM) is a fully coupled, state-of-the-science Earth system model that uses code optimized for DOE's advanced computers to address the most critical scientific questions facing our nation and society (Golaz et al., 2019). The E3SM Land model (ELM) is designed to understand how the changes in terrestrial land surfaces will interact with other Earth system components and has been used to understand hydrologic cycles, biogeophysics, and ecosystem dynamics. In spite of great successes, the ELM has several known issues that restrain rapid improvements. For example, the ELM uses equilibrium models to simulate dynamic land-climate interactions and it requires long model spin-up time to identify suitable initial conditions for transient simulations. The ELM lacks built-in uncertainty mechanisms that can improve the robustness of model predictions. The ELM is a holistic, deterministic model system with a rigid design, and in many situations, it is hard to modify the ELM system to incorporate new theory/hypothesis and new data across scales to address emerging science problems (such as predicting the impacts of water cycle extremes). In addition, The ELM is technically optimized for traditional CPU-centric computers and it cannot fully utilize the current and incoming leadership computers for model simulations and uncertainty quantification (UQ). The success of artificial intelligence (AI) has inspired scientists to use AI models to discover intrinsic features from simulation data (Chattopadhyay et al., 2020) and observational data (Reichstein et al., 2019) to gain further process understanding of Earth science problems. However, autonomous AI model training through deep learning usually requires a huge amount of annotated data. To overcome the limitations from the data and computing resources, knowledge-guided AI models are necessary where human-knowledge is ingested in model construction (Banino et al., 2018) and training process (Silver et al., 2016) for efficient learning. Herein, we present a new way that leverages the process understanding from the ELM to guide AI model development for the ELM enhancement and UQ. We hope this study can inspire further Earth and environmental system model developments and transformations.

54 ENVIRONMENTAL SCIENCES↗

Facilitating better and faster simulations of aerosol-cloud interactions in Earth system models

Focal Area(s): 1. Predictive modeling through the use of AI techniques and AI-derived model components; the use of AI and other tools to design a prediction system comprising a hierarchy of models. 2. Insight gleaned from complex data (both observed and simulated) using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI. Science Challenge: One major challenge that Earth system models (ESMs) face in providing credible prediction of the Earth system and its water cycle characteristics (e.g., mean state, variability, and extreme events) is to accurately simulate aerosol-cloud interactions (ACI). The physical, chemical, and dynamical processes affecting ACI are extremely complex and they range from nanoscale to planetary scale. In each model development cycle, scientists spend significant efforts investigating model deficiencies and uncertainties associated with aerosols (e.g., emissions, chemical processes, aerosol microphysics, and transport) and clouds (e.g., macrophysics, microphysics, turbulence, and large-scale circulation) in order to develop improved treatments. However, despite decades of active research, ACI is still a major source of uncertainty in climate projections, even though great progress has been made. Specific scientific challenges include: (i) Parameterizations are developed based on limited data; (ii) The complexity of a parameterization required for accurate predictions is not understood; (iii) Incomplete and unknown physics leads to errors in the fully coupled Earth system; and (iv) Complex physics is computationally too expensive to employ in ESMs.

54 ENVIRONMENTAL SCIENCES↗

Framework for an adaptive integrated observation system using a hierarchy of machine learning approaches

Focal Area(s): 1. Data acquisition enabled by machine learning, AI, and advanced methods including experimental/network design/optimization, and hardware-related efforts involving AI. 2. Insight gleaned from complex measurements using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI Science Challenge and Rationale: Atmospheric processes are stochastic, occur at scales from the micrometer to many kilometers, and are constantly changing over time. Characterizing these interactions and associated environmental conditions using traditional measurement techniques is difficult and can take years to build statistics on atmospheric phenomena that occurs episodically. Developing new innovative approaches to modify sampling strategies in real-time would enable the routine collection of targeted measurements focused on a specific set of science questions.

54 ENVIRONMENTAL SCIENCES↗

Geophysical Retrievals in an Artificial Intelligence (AI) Framework for Illuminating Processes Controlling Water Cycle

Focal Area(s): This white paper responds to Focal Area #3: Insight gleaned from complex data (both observed and simulated) using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI. Science Challenge: This white paper addresses the water-cycle and data-model integration grand challenge. It leverages data from the Atmospheric Radiation Measurement (ARM) Climate Research Facility, and Next-Generation Ecosystem Experiment (NGEE), and Science Focus Area (SFA). The white paper focuses on controlling cloud, precipitation, and radiative properties as observed and simulated by the Earth System Models (ESM). The described framework can be readily applied to any other ensemble of instruments, including satellites and other ground-based networks.

54 ENVIRONMENTAL SCIENCES↗

Hybridizing Machine Learning and Physically-based Earth System Models to Improve Prediction of Multivariate Extreme Events (AI Exploration of Wildland Fire Prediction)

Focal Areas: This project responds to two focal areas identified in the DOE Call for AI4ESP White Papers: 1) Predictive modeling through the use of artificial intelligence (AI) techniques, and 2) insights gleaned from complex data using explainable AI and big data analytics. Science Challenge: Large wildland fires (hereafter wildfires) appearing as high-impact compound climate extreme events are closely related to hydroclimate and water cycle extremes that modulate surface fuel supply and combustibility. These compound events have multivariate climatic features (e.g., temperature, precipitation, relative humidity, wind, lightning) and societal drivers (e.g., forest management, land use change, human caused ignitions). Meanwhile, they induce strong feedbacks to the coupled atmosphere, biosphere, and hydrosphere by perturbing regional and global radiation budget as well as ecological, biogeochemical, and water cycles across multiple spatiotemporal scales. The nonlinear interactions between these natural and anthropogenic components of the Earth system are too complex to be completely and adequately represented in today’s Earth system models (ESMs). The inherent stochastic nature of fire activity at all scales further increases the difficulty of its prediction using ESMs that are usually developed from deterministic equations and parameterizations. Besides, concurrence of long-term (decadal to interdecadal) global climate change and fire regime shifts overlapping with short-term (intraseasonal to interannual) variations of regional fire weather and burning activity confound predictability of these compound extreme events. We propose to address the above scientific challenges by using machine learning (ML)-based data-driven modeling techniques to integrate observations and physically-based ESMs’ simulations in a computationally efficient hybrid prediction system. This prediction system is supposed to characterize the wildfire’s sensitivity to climate and exogenous drivers at high resolution (~ 0.25°) on subseasonal to seasonal (S2S) timescales providing improved predictability and explainability. We will use the system to help identify: (1) What are the computational elements of a hybrid system needed to predict compound climate extreme events such as global wildfires? (2) What are the key drivers (either natural or anthropogenic) that modulate short-term variations of multivariate fire weather and burning activity over different regions? How can one take advantage of those driver-response relationships to improve the predictability of large wildfires on S2S time scales? (3) What are the underlying physical mechanisms and sources of improved predictability? Which ML techniques are optimal in revealing and adapting these mechanisms?

54 ENVIRONMENTAL SCIENCES↗

Improve wildfire predictability driven by extreme water cycle with interpretable physically-guided ML/AI

Focal Area(s): Predictive modeling using AI techniques and AI-derived model components; use of AI and other tools to design a prediction system comprising of a hierarchy of models (Primary); insight gleaned from complex data (both observed and simulated) using AI, big data analytics, and other advanced methods, including explainable AI and physics or knowledge-guided AI (Secondary). Science Challenge: Wildfires modify land surface characteristics, such as vegetation composition, soil and litter carbon stocks, and surface albedo, with significant consequences for the regional carbon cycle. For example, tropical regions (i.e., African and South America) are particularly vulnerable to wildfire, account for more than 80% of the global burned area, and emit ~1.4 PgC y -1 into the atmosphere together with other dust and aerosols that strongly affect regional climate.

54 ENVIRONMENTAL SCIENCES↗

In Situ Inference for Earth System Predictability

Focal Area: Focal Area 3: Insight gleaned from complex simulated data using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI. Science Challenge: An understanding of future evolution in precipitation extremes is critical to numerous DOE mission questions. Extreme events are by nature short time-scale events that are difficult to diagnose in available model data. Accurate modeling of extreme events necessarily requires high spatial resolution at the storm scale locally. However, the environment in which storms grow is dependent on global, remote, processes. These complex spatiotemporal relationships are impossible to diagnose at resolutions required to accurately model storms responsible for extreme precipitation. At exascale, climate simulations will produce results at fine enough resolution to investigate these relationships. However, the resulting data from these simulations will be far too large to save for post-simulation analysis. We advocate for fitting statistical models inside the simulations as they run, a context known as in situ, which will facilitate scientific investigations using the full fine-scale data stream. Figure 1 shows an example of the type of model we could consider, a Bayesian hierarchical spatial regression model. Precipitation extremes at each grid cell are modeled using extreme value distributions. Since extremes are rare, fitting models to individual grid cells can result in high variance and poor estimates. Instead, the model can be made more robust by smoothing the parameters of the extreme value model across space. Additionally, the parameters themselves can be functionally linked to other variables elsewhere in the simulation. Thus, we can use the fine-scale data to build more robust models for extremes that link extreme behavior to other climate patterns.

54 ENVIRONMENTAL SCIENCES↗

Integrating Applied Energy and BER Smart Data Capabilities to Develop a DOE Data Fabric for Energy-Water R&D

Focal Area(s): 1) Data acquisition and assimilation enabled by machine learning, AI, and advanced methods including experimental/network design/optimization, unsupervised learning (including deep learning), and hardware-related efforts involving AI (e.g., edge computing). Science Challenge: DOE R&D, including DOE’s Basic Energy Research (BER)’s Environmental Systems Science Division (EESSD) program and DOE’s applied energy research (AER) programs (EERE, FE, and NE) are producers and consumers of Earth systems datasets. This white paper focuses on the first topic area from the call in relation to how crosscutting resources and innovations from DOE’s EESSD and AER can be brought to bear to mutual benefit and more efficient energy-water, Earth system data resources through improved. The overarching challenge posed by this call focuses on how DOE can directly leverage artificial intelligence (AI) to engineer a substantial (paradigm-changing) improvement in Earth System Predictability? While stemming from DOE BER’s EESSD program, this is a challenge that is faced and also being addressed by DOE’s AER programs. Over the past decade plus, FE, EERE, and NE programs have made important strides towards addressing this need. These strides are in many ways highly complementary to EESSD’s MODEX efforts. Energy water systems spanning metocean to groundwater to surface water systems all are data driven whether for basic energy or applied energy. These are remote, multi-variate, complex natural, and in many cases engineered, systems. Key needs and challenges of both EESSD and AER include developing data-focused tools to enhance data search and discovery to fill in knowledge gaps (address sparse data challenge), and rapidly transform datasets, including disparate and multi-source data. Leveraging DOE on-premise computing (HPC, exascale) infrastructure supports the computing-intensive algorithms required to execute these data acquisition and transformation processes to derive enriched knowledge and data, driving AI/ML and big data analytics for these systems. The opportunity lies in combining BER and AER efforts to provide a more robust, advanced, efficient and complete computing data fabric to address energy-water data acquisition and assimilation needs which currently pose significant impediments to AI/ML predictions and research.

54 ENVIRONMENTAL SCIENCES↗

Past and Future Trends of Severe Storms

The objective of this research is to use machine learning methods to establish past trends in the severe storm record and build on this information to provide severe storms projections in a future climate. This objective aligns with Focal Area 3, which emphasizes big data analytics in combination with knowledge-guided AI.

58 GEOSCIENCES↗

Representing the Unrepresented Impact of River Ice on Hydrology, Biogeochemistry, Vegetation, and Geomorphology: A Hybrid Physics-Machine Learning Approach

Focal areas include: Predictive modeling through the use of AI techniques and AI-derived model components; the use of AI and other tools to design a prediction system composed of a hierarchy of models (e.g., AI driven model/component/parameterization selection). Insight gleaned from complex data (both observed and simulated) using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI.

54 ENVIRONMENTAL SCIENCES↗

Evaluating Offshore Infrastructure Integrity

Drilling in the offshore environment involves a complex network of infrastructure including pipelines, platforms, rigs, subsea installations, ports, and terminals. Government and industry partners have developed this network over many decades and it remains a critical part of the United States (U.S.) energy portfolio. Many of the major components of this system have been designed with a 20- to 30-year lifespan, yet consistent and growing energy demands support the need to extend the design life of existing infrastructure or repurpose it for secondary needs (i.e. enhanced oil recovery, carbon storage, and new wells). As a result, a growing portion of the offshore infrastructure in the U.S. is approaching or has exceeded its original design life. A critical step in ensuring the continued safe and effective operation of offshore infrastructure is developing a comprehensive understanding of the state of offshore infrastructure and the factors that effect it. The purpose of this project is to assess the current state of existing infrastructure and identify the factors involved in infrastructure degradation through the development and application of big data analytics, machine learning, and advanced spatio-temporal analysis. The project leverages existing data at NETL and combines it with new information on offshore oil and gas structures and the ambient offshore environment in an effort to identify patterns associated with infrastructure integrity. Building on the identified trends and patterns, this project incorporates exploratory analytics and spatial analysis tools in conjunction with machine learning and statistical models to characterize the condition of existing platforms in the offshore environment and predict their risk of failure.

02 PETROLEUM↗

Evolving Architectures and Considerations to address Distributed Energy Resources and Non-Wired Alternatives

The electric grid is in the beginning stage of a transformation, driven by a combination of shutdowns of coal-fired plants, commissioning of new natural-gas plants, and tremendous growth in energy supply from renewables such as wind, and solar. As utilities navigate this transformation, their progress is supported by advances in Operational Technologies (OT), and Informational Technologies (IT), such as automation, smart inverters, cloud computing, mobile computing, machine learning, big data analytics, which have the potential to enable advanced capabilities more efficiently and at a lower cost. This white paper focuses on the architectural considerations that will allow the industry to transition in a planned manner. It introduces and formalizes two architectural constructs –the data bus and the control bus. The data bus is responsible for carrying all non-operational models and information necessary to drive utility decisions. In contrast, the control bus is responsible for carrying all operational data and control actions taken at the local level, centralized level, or other levels in-between. The paper reviews those architectural considerations, their requirements and how they will evolve. This paper intends to provide a context for vendors, utilities, and their service providers to review and understand the changes that are coming and get ready for them.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Big Data Analysis of Synchrophasor Data: Outcomes of Research Activities Supported by DOE FOA 1861

This report describes the key outcomes of research activities sponsored by the Department of Energy’s Funding Opportunity Announcement (FOA) number 1861 that was aimed at advancing the state-of-the-art in big data analytics applied to transmission-level synchrophasor measurements. The FOA resulted in eight research grants where the awardees developed machine learning and artificial intelligence tools and approaches. The commonalities in tools and approaches used by the awardees are explored, and insights gained from how the project outcomes might be operationalized are discussed. This report does not seek to comprehensively summarize all research supported by the FOA, rather it focuses on enabling the fast dissemination of major findings to the broader power systems community.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

EDX ClaiMM: Digital Resources for the Critical Minerals and Materials Community

Securing critical mineral supply chains is essential for transitioning to a clean energy economy and for maintaining national security. Big-data analytics can serve as a cost-effective means of identifying new domestic critical mineral resources but only if data can be easily located and digested. Using ArcGIS Enterprise Sites, EDX ClaiMM was developed to increase the accessibility of critical minerals data, reducing time spent on data collection and integration. Hosted tools provide rapid visualization and exploration of key datasets, unlocking insights to support resource assessments.

Yesenchak, Rachel↗

Emerging Materials Technologies That Matter to Manufacturers

A brief overview of emerging materials technologies. Exploring the weight reduction benefit of replacing Carbon Fiber with Carbon Nanotube (CNT) in Polymer Composites. Review of the benign purification method developed for CNT sheets. The future of manufacturing will include the integration of computational material design and big data analytics, along with Nanomaterials as building blocks.

nanomaterials↗

Utilizing HDF4 File Content Maps for the Cloud

We demonstrate a prototype study that HDF4 file content map can be used for efficiently organizing data in cloud object storage system to facilitate cloud computing. This approach can be extended to any binary data formats and to any existing big data analytics solution powered by cloud computing because HDF4 file content map project started as long term preservation of NASA data that doesn't require HDF4 APIs to access data.

Elastic Search↗