Search NASA⌕ Search

SEARCH · Search NASA

Results for “data lakes”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

Tsdat: An Open-Source Data Standardization Framework for Marine Energy and Beyond

Many organizations are tasked with the collection and processing of large quantities of data from various measurement devices. Data reported from these sources are often not interoperable with datasets and software used by analysts and other organizations in the same domain, introducing barriers for collaboration on large-scale projects. This poses a particular problem for cross-device comparisons and machine learning applications, which rely on large quantities of data from multiple sources. To address these challenges, the open-source Time-Series Data Pipelines (Tsdat) Python framework was developed by Pacific Northwest National Laboratory, with strategic guidance and direction provided by the National Renewable Energy Laboratory and Sandia National Laboratories to facilitate collaboration and accelerate advancements in the marine energy domain through the development of an open-source ecosystem of tools. This paper will describe the Tsdat framework and the data standards within which it operates. A beta version of Tsdat has been released and is being used by several projects in marine energy, wind energy, and building energy systems.

big data↗

2021 Annual Technology Baseline (ATB) Cost and Performance Data for Electricity Generation Technologies

Starting in 2015 NREL has presented the Annual Technology Baseline (ATB) in an Excel workbook that contains detailed cost and performance data, both current and projected, for renewable and conventional technologies. The workbook includes a spreadsheet for each technology. This version of the workbook provides the final updates to data for the 2021 ATB. In 2019 and 2020, NREL has also provided selected data in Tableau workbooks and structured summary csv files. The data for 2015 - 2020 is located on https://data.nrel.gov. In 2021 and going forward, the data is cloud optimized and provided in the OEDI data lake. A website documents this and future data at https://atb.nrel.gov.

Array↗

Data-Centric AI and the Open Energy Data Initiative (OEDI)

This presentation emphasizes the critical importance of data-centric AI. The limitations of model-centric AI when dealing with poor or insufficient data are highlighted, and it is illustrated how training models on inaccurate or noisy data leads to suboptimal results. This talk advocates for a hybrid approach that combines a focus on data quality and model parameters to achieve optimal results. The Open Energy Data Initiative (OEDI) is introduced as a valuable resource for obtaining high-quality energy-related datasets, hosting nearly 2,000 publicly accessible datasets, including 99 solar-related datasets, totaling over 2.7 petabytes of data. OEDI's data lakes enable users to query and work with data without extensive transfers. In conclusion, the significance of data-centric AI and adherence to data curation best practices is emphasized, positioning OEDI as a prime source of high-quality data for AI and machine learning in the renewable energy sector.

AI↗

Forecasting Lake-Effect Precipitation in the Great Lakes Region Using NASA Enhanced-Satellite Data

Lake-effect precipitation is common in the Great Lakes region, particularly during the late fall and winter. The synoptic processes of lake-effect precipitation are well understood by operational forecasters, but individual forecast events still present a challenge. Locally run, high resolution models can assist the forecaster in identifying the onset and duration of precipitation, but model results are sensitive to initial conditions, particularly the assumed surface temperature of the Great Lakes. The NASA Short-term Prediction Research and Transition (SPoRT) Center has created a Great Lakes Surface Temperature (GLST) composite, which uses infrared estimates of water temperatures obtained from the MODIS instrument aboard the Aqua and Terra satellites, other coarser resolution infrared data when MODIS is not available, and ice cover maps produced by the NOAA Great Lakes Environmental Research Lab (GLERL). This product has been implemented into the Weather Research and Forecast (WRF) model Environmental Modeling System (WRF-EMS), used within forecast offices to run local, high resolution forecasts. The sensitivity of the model forecast to the GLST product was analyzed with a case study of the Lake Effect Storm Echinacea, which produced 10 to 12 inches of snowfall downwind of Lake Erie, and 8 to 18 inches downwind of Lake Ontario from 27-29 January 2010. This research compares a forecast using the default Great Lakes surface temperatures from the Real Time Global sea surface temperature (RTG SST), in the WRF-EMS model to the enhanced NASA SPoRT GLST product to study forecast impacts. Results from this case study show that the SPoRT GLST contained less ice cover over Lake Erie and generally cooler water temperatures over Lakes Erie and Ontario. Latent and sensible heat fluxes over Lake Ontario were decreased in the GLST product. The GLST product decreased the quantitative precipitation forecast (QPF), which can be correlated to the decrease in temperatures and heat fluxes. A slight increase in precipitation coverage was noted over Lake Erie due to a decrease in ice cover. Both the RTG SST and the GLST products predicted the precipitation south of the actual location of precipitation. This single case study is the first part of an examination to determine how MODIS data can be applied to improve model forecasts in the Great Lakes region.

Cipullo, Michelle↗

Trophic classification of Colorado lakes utilizing contact data, Landsat and aircraft-acquired multispectral scanner data

Multispectral scanner data, acquired over several Colorado lakes using Landsat-1 and aircraft, were used in conjunction with National Eutrophication Survey contact-sensed data to determine the feasibility of assessing lacustrine trophic levels. A trophic state index was developed using contact-sensed data for several trophic indicators (chlorophyll a, inverse of Secchi disk transparency, conductivity, total phosphorous, total organic nitrogen, algal assay yield). Relationships between the digitally processed multispectral scanner data, several trophic indicators, and the trophic index were examined using a supervised multispectral classification technique and regression techniques. Statistically significant correlations exist between spectral bands, several of the trophic indicators (chlorophyll a, Secchi disk transparency, total organic nitrogen), and the trophic state index. Color-coded photomaps were generated which depict the spectral aspects of trophic state. Multispectral scanner data acquired from satellite and aircraft platforms can be used to advantage in lake monitoring and survey programs.

Boland, D. H. P.↗

Quantitative interpretation of Great Lakes remote sensing data

The paper discusses the quantitative interpretation of Great Lakes remote sensing water quality data. Remote sensing using color information must take into account (1) the existence of many different organic and inorganic species throughout the Great Lakes, (2) the occurrence of a mixture of species in most locations, and (3) spatial variations in types and concentration of species. The radiative transfer model provides a potential method for an orderly analysis of remote sensing data and a physical basis for developing quantitative algorithms. Predictions and field measurements of volume reflectances are presented which show the advantage of using a radiative transfer model. Spectral absorptance and backscattering coefficients for two inorganic sediments are reported.

Shook, D. F.↗

NASA’s Safety, Reliability, and Mission Assurance Digital Future

The evolution from “document-centric” to “data-centric” and “model-centric” information leveraging structured data and model-based approaches is at the heart of digital engineering transformational efforts underway across industry and government. It is these approaches that pave the way for data lakes, Authoritative Sources of Truth (ASOTs), and systems- of-systems interoperability and the corresponding transformational benefits thereof. Such benefits include increased data availability, data access equity, data traceability, real-time analytics, batch analytics, and (most importantly) acceleration of the time-to-value and time-to-insights associated with engineering products and analyses. The longer-term benefits of reusability, customization and traceability are even more promising. For Safety and Mission Assurance (SMA), and Mission Success (SMS) activities; realization of such benefits is essential to provide engineers and analysts alike vital information when needed to support critical decision making throughout the entire life cycle. The SMA community often operate in parallel with engineering activities, for which information exchange with relevant context is paramount. Far too often, such information lags key decision points and/or is absent of the robust, integrated, knowledge needed, given inherent barriers associated with traditional document-centric means to data sharing, analysis, and reporting. This paper provides an overview of how NASA’s Office of Safety and Mission Assurance (OSMA) is evolving its policies, standards, guidance, and training to transform to eliminate such barriers, thus realizing the benefits emerging in this new digital era. A roadmap for achieving this digital future is presented along with key building blocks involving use and implementation of concepts such as: Objectives-Hierarchies, Objective-Driven Requirements, Accepted Standards, Safety and Assurance Cases, data digitization (i.e., ontologies, structured data, and model-centric data), FAIR (Findable, Accessible, Interoperable, & Reusable) and/or FAIRUST (Findable, Accessible, Interoperable, Reusable, Understandable, Secure, and Trusted) principles [1]. This paper also describes how OSMA, leveraging the Agency’s overall commitment to Digital Transformation (DT), is using the power of Policy, “Digital” Domain representation, Product Evolution, and Community Outreach and Engagement as part of a strategic vision and roadmap to evolve and transform its SMA organizations to become better able to serve its stakeholders and customers. Future publications will elaborate on these building blocks and deeper concepts.

Authoritative Source of Truth (ASOT),↗

District wide water resources investigation and management using LANDSAT data. Phase 1: Lake volume

A technique for estimating available water storage volume using LANDSAT data was developed and applied to Lake Washington and Lake Harris in central Florida. The technique can be applied two ways. First, where the historical stage records are available, the historical LANDSAT data can be used to establish the relationship between lake volume and lake stage. In the second case, where the historical stage records are not available, the historical LANDSAT data can be used to estimate the historical lake stage after the lake volume and stage information become available in the future.

Shih, S. F.↗

A Case Study of Multimodal, Multi-institutional Data Management for the Combinatorial Materials Science Community

Although the convergence of high-performance computing, automation, and machine learning has significantly altered the materials design timeline, transformative advances in functional materials and acceleration of their design will require addressing the deficiencies that currently exist in materials informatics, particularly a lack of standardized experimental data management. The challenges associated with experimental data management are especially true for combinatorial materials science, where advancements in automation of experimental workflows have produced datasets that are often too large and too complex for human reasoning. The data management challenge is further compounded by the multimodal and multi-institutional nature of these datasets, as they tend to be distributed across multiple institutions and can vary substantially in format, size, and content. Furthermore, modern materials engineering requires the tuning of not only composition but also of phase and microstructure to elucidate processing–structure–property–performance relationships. To adequately map a materials design space from such datasets, an ideal materials data infrastructure would contain data and metadata describing (i) synthesis and processing conditions, (ii) characterization results, and (iii) property and performance measurements. In this work, we present a case study for the low-barrier development of such a dashboard that enables standardized organization, analysis, and visualization of a large data lake consisting of combinatorial datasets of synthesis and processing conditions, X-ray diffraction patterns, and materials property measurements generated at several different institutions. While this dashboard was developed specifically for data-driven thermoelectric materials discovery, we envision the adaptation of this prototype to other materials applications, and, more ambitiously, future integration into an all-encompassing materials data management infrastructure.

36 MATERIALS SCIENCE↗

The Mars 2020 Ground Data System Architecture

The Mars 2020 Mission’s primary objective is to collect 20 geographically unique samples during its prime mission of one and a quarter Martian years, or just over 2 Earth years. Mission planners determined the project needed to develop a system that would enable the operations team to analyze engineering and science data, make science decisions, select viable rover targets at a millimeter resolution and validate an uplink bundle for a car sized rover with more complex science instruments than any previous Mars surface mission. All this had to be done within a five hour time frame. Doing this with a small team would be a challenge, but this had to be accomplished by a large team of engineers and scientists located across North America and Europe. Achieving this level of operational efficiency was unheard of in the prime mission. In addition, the mission had another set of requirements that had nothing to do with surface operations; the Mars 2020 Ground Data System (GDS) was also expected to comply with a new set of security requirements to keep up with the ever changing cybersecurity landscape. The Mars 2020 Ground Data System (GDS) is a re-architected version of the Mars Science Laboratory GDS. The primary goal was to integrate the lessons learned from previous Mars surface missions, accommodate a set of new requirements and capabilities required to ensure mission success, and comply with a new set of cybersecurity controls. The new architecture includes several unique qualities including a data lake, language-agnostic system-wide event-based operations, containerization, automated deployment, network segmentation, infrastructure-as-code, API-driven interfaces, and the first Mars surface GDS to operate primarily in the cloud. The new architecture enabled greater access to the system’s data, tighter integration with the operations team, and a higher level of traceability. The availability of the data also enabled a new set of capabilities previously not possible on surface missions. These new capabilities include an autonomous data to information, pipeline for downlink analysis, horizontal scaling of science data processing capabilities, autonomous round trip data tracking of science and engineering data, integration of flight system state into the tactical planning cycle, high fidelity targeting utilizing kinematic data, and hierarchical image and 3d meshes data representations. This paper will introduce the requirements for the Mars 2020 Mission, the heritage architecture, and the rationale for the changes to achieve the new architecture. The paper will continue to describe the fundamental changes made to the GDS architecture, how these changes enabled a more tightly integrated GDS, and the new capabilities that were enabled by the new architecture. The paper will conclude with the lessons learned from the process of rearchitecting a heritage GDS system and from the first 200 days of operations supporting over 800 users from around the world.

Lopez-Roig, Reynaldo↗

DeepLynx Ecosystem 2025

Poor data integration and governance continue to plague complex engineering projects, resulting in missed cost, schedule, and performance targets. Departments operate in isolated systems with manual data exchange, creating fragmented information that compounds errors and leads to significant delays and cost overruns. The DeepLynx ecosystem addresses these challenges through an open-source, modular data management platform that transforms fragmented project data into an integrated digital thread. Built on a federated microservice architecture, the ecosystem comprises seven specialized tools centered around DeepLynx Nexus, a unified data catalog with hierarchical organization and graph-based navigation capabilities. The ecosystem includes: DeepLynx Stream for real-time timeseries data ingestion from industrial sources; DeepLynx Ingest for governed data uploads with formal review workflows; DeepLynx Lattice for ontology-based entity and relationship extraction; DeepLynx Run for workflow orchestration and secure AI/ML compute; DeepLynx Visualize for 3D digital twin visualization; and DeepLynx Insight for AI-assisted document analysis with traceable, grounded responses. Deployable in cloud, on-premise, or hybrid environments using containerized Docker applications and Helm charts, the DeepLynx ecosystem provides flexible infrastructure that adapts to organizational requirements. By consolidating project data into a unified data lake with role-based access controls and OAuth2 authentication, DeepLynx enables digital thread and digital twin capabilities that improve decision-making, reduce risk, and support complex engineering workflows throughout the project lifecycle.

42 - ENGINEERING↗

The Role of Declining Snow Cover in the Desiccation of the Great Salt Lake, Utah, using MODIS Data

The Great Salt Lake (GSL) in Utah has been shrinking since the middle of the 19th Century, leading to decreased area and volume, and increased salinity. We use satellite data products from the Terra and Aqua MODerate-resolution Imaging Spectroradiometer (MODIS) and the Landsat-7 and -8 satellites, along with meteorological and streamflow data, and modeled data products to study the relationship between changing snow-cover conditions and the decline of the GSL since 2000 in the context of the historical record of lake levels. The GSL basin includes much of the snow-dominated Wasatch and Uinta mountain ranges to the east of the lake. Snowmelt feeds the Bear, Jordan, and Weber rivers which are the three main rivers that flow into the lake. Snowmelt-timing maps, derived from a new MODIS standard snow-cover product, MOD10A1F, show that snow melted ~9.5 days earlier in the GSL basin during the study period, extending from 2000 – 2018. Air temperatures derived from 26 meteorological stations and surface temperatures measured by the Aqua MODIS land-surface temperature (LST) products, MYD21A1D and MYD21A1N, show trends of increasing temperature of ~0.94°C (a=0.05), and ~2.18°C, respectively, with most of the LST trends in the GSL basin being statistically significant (a=0.05). Increasing air temperatures in the basin have led to less precipitation falling as snow, lower snow depth (by ~34.5 mm (=0.01)) and snow-water equivalent (0.02 mm (a=0.01)), and earlier snowmelt. Also during the study period, Global Land surface Evaporation Amsterdam Model data show evaporation increasing by ~3.2 mm/yr, with trends in much of the basin being statistically significant (a=0.05). Trends calculated from the various products are generally in agreement indicating higher temperatures, greater evaporation, less snowfall and snow-on-the ground, and earlier snowmelt. Earlier snowmelt contributes to increasing evaporative loss from water flowing toward the lake. Furthermore, a lower mountain snowpack and less precipitation falling as snow (versus rain) is associated with lower stream discharge even if overall precipitation stays the same. The surface-water temperature of the GSL also increased over the study period by ~ 0.69°C, according to the MODIS LST data products, and the surface-water elevation of the lake dropped by ~1.7 m between 2000 and 2018 based on United States Geological Survey measurements, and the areal extent of the lake decreased by ~901 km2 as measured using Landsat imagery. Desiccation of the lake is associated with deleterious effects on wildlife, recreational activities, and some local industries. And, importantly, an expanding lake bed can also fuel dust storms that promote dangerous air quality along the Wasatch Front. This work elucidates the key role that satellite remote sensing can play in documenting earlier snowmelt and other changes in the GSL basin that influence the ongoing decline of the Great Salt Lake.

Dorothy K Hall↗

Time Variable Gravity from Local Mascon Analysis of GRACE Data

We have analyzed GRACE Level 1-B data in 2003 and assessed a new approach for extracting time variable gravity that isolates the gravity signal in both time and space. The Level-1B satellite-to-satellite range rate (KBRR) data and accelerometry are processed in daily arcs using the precise orbit products produced by the GRACE team from GPS to calibrate both the accelerometer and KBRR data. We then adjusted select components of the intersatellite baseline vector for each data segment isolated to the region of interest. Herein, we solved for mass anomalies in 45 deg x 45 deg blocks over the Amazon and the nearby Atlantic Ocean and estimate mass flux in units of cm of water over each block. We show with this approach that we can recover mass anomalies on a submonthly basis with 10 to 15 day temporal resolution. We discuss the important issues related to this solution, including the size of the mascon blocks, the weight given to the temporal and spatial constraint used to stabalize the solutions, as well as the optimal correlation in time and distance. We compare the the mascon results with solutions obtained from the more standard approach using spherical harmonics and with independent hydrology models and lake data. This technique demonstrates that sub-monthly medium wavelength mass flux phenomena are well sensed by the hyper-precise line of sight velocity data produced from GRACE.

Lemoine, Frank G.↗

NASA Earth Systems Digital Twins (ESDT)

"Similarly to artificial intelligence, which is now revolutionizing many aspects of our daily lives, Earth system digital twin technologies have the potential to revolutionize the way Earth Science research will be conducted in the future, and how results and knowledge from this research will provide information to support decision making and yield impactful societal benefits. An Earth System Digital Twin or ESDT is a dynamic and interactive information system that first provides a digital replica of the past and current states of the Earth or Earth system as accurately and timely as possible; second, allows for computing forecasts of future states under nominal assumptions and based on the current replica; and third, offers the capability to investigate many hypothetical scenarios under varying impact assumptions. In other words, an ESDT provides the integrated What-Now, What-Next, and What-If pictures of the Earth or Earth system, by continuously ingesting newly observed data and by leveraging multiple interconnected models, machine learning as well advanced computing and visualization capabilities. Digital twins have been developed in engineering since 2002, but the interest in digital twins for the Earth domain is more recent and stems from the convergence of several developments: - The huge amount of diverse data that has now been collected continuously for more than 50 years, and that is becoming more and more difficult to access, understand, and utilize. - At the same time, because of climate change and its impacts the information produced by all of this data is becoming of interest to many new non-traditional users for analyzing and predicting various phenomena. - Because of advances in computational and visualization capabilities and the parallel unprecedented development of machine learning (ML), extracting relevant information from these large amounts of data and running complex models faster has become possible. As a result, it is becoming necessary and possible to build intuitive and interactive frameworks that will enable users with various skill levels and/or organizational hierarchy levels to easily access large amounts of targeted information along with the relevant tools and models (Earth system and human activity models), to support them in analyzing and visualizing this information, to help them understand interactions among models, to visualize the potential outcomes of various impacts, and to support decision or policy making. The full power of digital twins is that, through an integrated representation and standardized tools and software technologies, the same digital replica can address the needs of multiple users at various resolutions (spatial and temporal) and for various applications (science, economic, policy, etc.) – “from farmer to scientist”. With all these interests at stake, the challenges of building optimal digital twins are many and complex. The first challenge is to determine if a Digital Twin should be global or local, and multi-domain or thematic. For example, some domains such as Climate or Weather will require a global Digital Twin or Digital Twin capabilities while science areas such as Biodiversity might be more local. We can also envision that multiple thematic ESDTs, e.g., Air Quality, Wildfires, Hydrology could be federated or provide input to other ESDTs, either on a regional level or to a more global ESDT. Overall, we can imagine a future “web” of Digital Twins co-existing in a hierarchy or in a network, and capable of being connected or federated depending on the needs. This last point brings up the very important challenge of interoperability, including standards and protocols that will need to be built into these systems from the beginning. Each individual digital twin would have full flexibility in internal construction but would need standards-based interfaces (input and output) or hooks to make it compatible with others. Another challenge when building digital twins will be to decide how to organize each digital replica. Based on the applications targeted by the DT under implementation, various amounts and types of raw data, Analysis Ready Data (ARD) and information will need to be incorporated. Depending on the required latencies and needs of the users, various solutions can be considered, including Data Cubes, Data Lakes, pointers, or computing information on demand. We envision that each ESDT will choose a solution adapted to its specific objectives. Another important challenge is the type(s) of visualization that will be used, as well as the level of interactivity and refresh rate that will be required. Again, this will depend on the objectives of the ESDT, but also on the various users’ needs. In most cases, several types of visualizations and human interfaces will need to be offered depending on the projected users of that system. In parallel to the challenges highlighted above, there are also many tools and technologies that will need to be developed or improved for all types of digital twins. Among those are improved machine learning technologies, for example providing explainability, but also ML techniques for causality and providing a better integration of physics models. Additionally, reliable uncertainty quantification methods will be needed for all ESDT components, from validating data fusion and assimilation to assessing the accuracy of ML models and weighing the values of decisions supported by those systems. This presentation introduces the ESDT concept, presents several ESDT use cases, and a proposed ESDT architecture framework, as well as various technologies being developed by the Advanced Information Systems Technology (AIST) Program."

Earth Science Remote Sensing; Information Systems↗

Propagation of electrons emitting weak type III bursts in coronal streamers

The observations of weak type III bursts at 73.8, 57.5, 50.0, and 38.5 MHz from Clark Lake Radio Observatory on four days are reported and their characteristics are discussed. In addition to Clark Lake data, the magnetogram and sunspot/active region data and the coronal streamer data obtained by HAO's Coronagraph/Polarimeter aboard SMM satellite are used to study the location of the burst sources with respect to the coronal streamers emanating from active regions. It is shown that the bursts occur within or close to the edge of dense coronal streamers implying that the coronal streamers contain open magnetic field lines along which the electrons generating the bursts propagate. The positional analysis of the bursts is used to estimate the variation of coronal electron density with radial distance.

Gopalswamy, N.↗

Advanced Information Systems Technology for NASA Earth System Digital Twins (ESDT)

ESA and NASA have both started programs to design and develop Digital Twins of the Earth and/or Earth systems. Organized around interconnected, multi-domain, high-scale modeling capabilities, the three major components of an Earth System Digital Twin are a continuously updated Digital Replica of the Earth System of interest, dynamic Forecasting models, and Impact Assessment capabilities. Earth System Digital Twins integrate diverse Earth and human activity models, continuous observations, and information system capabilities to provide unified, comprehensive representations and predictions that can be utilized for monitoring the health of the Planet, as well as for developing actionable information to support decision making. More generally, Digital Twins will help researchers better understand the fundamental Earth systems that impact everything from wildfires to climate change. This Townhall will first provide a short description of ESA’s, NASA’s and CNES’s current efforts in Digital Twins: • Destination Earth (DestinE) is the first initiative of the European Union to create a Digital Twin of the Earth, particularly focused on weather and climate, and as a coordinated effort between the European Centre for Medium-Range Weather Forecasts (ECMWF), ESA and the European Organization for the Exploitation of Meteorological Satellites (EUMETSAT). • Additionally, ESA is developing several Digital Twin Earth Precursor Activities to study some of the key scientific and technical challenges associated with building a “Digital Twin Earth”. These include forests, hydrology, Antarctica, food systems, oceans, and climate hot spots Digital Twin prototypes. • NASA Earth Science Technology Office (ESTO)’s Advanced Information Systems Technology (AIST) Program has started an initiative in Earth System Digital Twins (ESDT), with 14 current projects in this area, developing various information systems technologies and prototypes that will help prepare the development of future NASA Digital Twins. • CNES is currently developing the concept of a Digital Twin Factory (DTF), which relies on a data lake, a high computing capability using clouds and/or HPC and has thematic algorithms and methodologies able to generate registered and coherent layers of information in order to enrich a datacube from which physical indicators can be computed spatially. NASA, ESA and CNES are also currently defining science use cases that will be presented during the townhall; those and a few short discussions of current projects will serve as a starting point to engage a dialogue about Digital Twins of the Earth with the IGARSS community.

earth science remote sensing; Information systems↗