Search NASA⌕ Search

SEARCH · Search NASA

Results for “data challenges”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 703 records · Page 39
All sourcesNASA NTRSDOE OSTI
NASA NTRS1992

Development of an expert data reduction assistant

We propose the development of an expert system tool for the management and reduction of complex data sets. The proposed work is an extension of a successful prototype system for the calibration of CCD images developed by Dr. Johnston in 1987. The reduction of complex multi-parameter data sets presents severe challenges to a scientist. Not only must a particular data analysis system be mastered, (e.g. IRAF/SDAS/MIDAS), large amounts of data can require many days of tedious work and supervision by the scientist for even the most straightforward reductions. The proposed Expert Data Reduction Assistant will help the scientist overcome these obstacles by developing a reduction plan based on the data at hand and producing a script for the reduction of the data in a target common language.

Miller, Glenn E.↗
NASA NTRS1993

Development of an expert data reduction assistant

We propose the development of an expert system tool for the management and reduction of complex datasets. the proposed work is an extension of a successful prototype system for the calibration of CCD (charge coupled device) images developed by Dr. Johnston in 1987. (ref.: Proceedings of the Goddard Conference on Space Applications of Artificial Intelligence). The reduction of complex multi-parameter data sets presents severe challenges to a scientist. Not only must a particular data analysis system be mastered, (e.g. IRAF/SDAS/MIDAS), large amounts of data can require many days of tedious work and supervision by the scientist for even the most straightforward reductions. The proposed Expert Data Reduction Assistant will help the scientist overcome these obstacles by developing a reduction plan based on the data at hand and producing a script for the reduction of the data in a target common language.

Miller, Glenn E.↗
NASA NTRS2018

Climatespark: an In-Memory Distributed Computing Framework for Big Climate Data Analytics

The unprecedented growth of climate data creates new opportunities for climate studies, and yet big climate data pose a grand challenge to climatologists to efficiently manage and analyze big data. The complexity of climate data content and analytical algorithms increases the difficulty of implementing algorithms on high performance computing systems. This paper proposes an in-memory, distributed computing framework, ClimateSpark, to facilitate complex big data analytics and time-consuming computational tasks. Chunking data structure improves parallel I/O efficiency, while a spatiotemporal index is built for the chunks to avoid unnecessary data reading and preprocessing. An integrated, multi-dimensional, array-based data model (ClimateRDD) and ETL operations are developed to address big climate data variety by integrating the processing components of the climate data lifecycle. ClimateSpark utilizes Spark SQL and Apache Zeppelin to develop a web portal to facilitate the interaction among climatologists, climate data, analytic operations and computing resources (e.g., using SQL query and Scala/Python notebook). Experimental results show that ClimateSpark conducts different spatiotemporal data queries/analytics with high efficiency and data locality. ClimateSpark is easily adaptable to other big multiple- dimensional, array-based datasets in various geoscience domains.

Hu, Fei↗
DOE OSTI2020

CoREC: Scalable and Resilient In-memory Data Staging for In-situ Workflows

The dramatic increase in the scale of current and planned high-end HPC systems is leading new challenges, such as the growing costs of data movement and IO, and the reduced mean time between failures (MTBF) of system components. In-situ workflows, i.e., executing the entire application workflows on the HPC system, have emerged as an attractive approach to address data-related challenges by moving computations closer to the data, and staging-based frameworks have been effectively used to support in-situ workflows at scale. However, the resilience of these staging-based solutions has not been addressed, and they remain susceptible to expensive data failures. Furthermore, naive use of data resilience techniques such as n-way replication and erasure codes can impact latency and/or result in significant storage overheads. In this article, we present CoREC, a scalable and resilient in-memory data staging runtime for large-scale in-situ workflows. CoREC uses a novel hybrid approach that combines dynamic replication with erasure coding based on data access patterns. It also leverages multiple levels of replications and erasure coding to support diverse data resiliency requirements. Furthermore, the article presents optimizations for load balancing and conflict-avoiding encoding, and a low overhead, lazy data recovery scheme. We have implemented the CoREC runtime and have deployed with the DataSpaces staging service on leadership class computing machines and present an experimental evaluation in the article. Here, the experiments demonstrate that CoREC can tolerate in-memory data failures while maintaining low latency and sustaining high overall storage efficiency at large scales.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Deep Learning-based Parameterization of Complex 3D CO2 Saturation Data in Large-scale Geological Carbon Storage

In deep learning (DL), dimension reduction plays a pivotal role in improving training efficiency and minimizing overfitting, especially when working with complex datasets like three-dimensional (3D) saturation data. In the context of geological carbon storage (GCS), 3D saturation data introduces unique challenges due to its sparse nature and sharp transitions at plume boundaries, known as shock fronts. To tackle these challenges, we developed a novel DL framework that combines dimension reduction with advanced 3D reconstruction techniques. Our approach utilizes latent variables derived from 2D average saturation fields to efficiently capture the essential features of high-dimensional data while reducing the number of variables. This enhances both the robustness and accuracy of DL models, making the framework more practical for real-world applications. By offering a tailored solution for modeling complex 3D saturation dynamics, this framework holds significant potential for environmental monitoring, energy storage, and other geological applications.

Wang, Hongsheng [University of Texas at Austin]↗
NASA NTRS2016

The Last Orbit: Planning Cassini's Plummet into Saturn

Cassini’s final orbit around Saturn will culminate in a dramatic ending as the spacecraft plunges into the ringed planet’s atmosphere, never to escape or be heard from again. The last hours of the mission prior to the final loss of signal have some of the most unique and valuable science to date. Cassini will take a unique trajectory to dive deep into the atmosphere on its approach to final disposal and no spacecraft, Cassini included, has entered these depths of Saturn’s atmosphere. The science community has placed heavy emphasis onthis once-in-a-lifetime opportunity to inspect these deeper regions of Saturn’s atmosphere. The Cassini project specifically aims to collect the very last bits of data during the final plunge to get samples of the deepest regions before the spacecraft is lost forever. The desire to collect the final bits of data presents several challenges. Cassini’s Mission Planning (MP) team has developed an End of Mission (EOM) scenario to tackle these demands. The EOM scenario outlines the framework for the entire last orbit of the mission and details thestrategy for data collection and transmission. Attaining near real-time transmission is key for the acquisition of the very last bits of data. The Cassini spacecraft will use a new mode of operations to successfully achieve this real-time transmission. In addition to this primary investigation and planning for telecommunications, key risks have been studied within the realm of the last orbit. Ultimately, this paper shows how the Cassini Project plans to ensurethe return of every last bit of data before the spacecraft is consumed by Saturn forever.

Bittner, Molly E.↗
DOE OSTI2022

Case Study: NREL Campus Chilled Water Storage Potential: Benchmark Datasets Development and Applications, Task 4 - Use Case Demonstration

The Benchmark Datasets Development and Applications project is a three-year collaboration between the National Renewable Energy Laboratory (NREL), Oak Ridge National Laboratory, Pacific Northwest National Laboratory, and Lawrence Berkeley National Laboratory. The project seeks to collect and curate high-resolution, well-calibrated time series of building operational and indoor/outdoor environmental data, which are crucial to understanding and optimizing building energy efficiency performance and demand flexibility capabilities as well as benchmarking energy algorithms. Project outcomes include approximately twelve high-fidelity building datasets, enhanced data representation tools, and four case studies to illustrate example applications. The goal of these case studies is to define and execute analyses that demonstrate how one or more datasets collected through this project can address a data gap or challenge historically faced by building stakeholders. This technical paper summarizes the findings of one of these case studies, in which we studied the operational efficiencies of the central cooling system at NREL. We looked at three years of data from the three chillers in the Field Test Laboratory Building (FTLB), from 2019 to 2021, to compare equipment operation and demand throughout the time period. Our analysis indicates that all three chillers are operating at or below the optimal loading conditions for most of the operation time, and thus there was no efficiency drop due to loading of the chillers at full capacity. Our recommendation is that no chiller capacity increase is needed; instead, the central plant could benefit from adopting advanced control logics for optimal sequencing of chillers during part load operations. Analysis of adding chilled water thermal storage to the central plant indicated 34% savings in demand cost and 24.5% savings in total cost (energy consumption and demand charge cost). The payback period is estimated to be 11-22 years with an assumed TES cost of $\$$100-$200 per ton. This case study shows how a selected dataset is used to solve a practical building problem - learning the operational status of its components, analyzing the effectiveness of a proposed new technique, and aiding decision-making for the building operations and maintenance team.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

Synchrotron X-ray diffraction analysis of constituent phases in transition joint between nickel alloy 738LC and a MnFeCoNiCu alloy

Synchrotron x-ray diffraction (XRD) analysis was performed on transition joints between a single-phase MnFeCoNiCu alloy and Ni-base Alloy 738LC to efficiently identify the constituent phases across the interface, with different levels of material intermixing generated by laser-welding with variable power. Here, the intermixing extent was quantified by postmortem energy dispersive x-ray spectroscopy mapping. Diffraction-based analyses on complex composition spaces with limited prior data present significant challenges because elemental substitution in both disordered and ordered phases is more extensive than in conventional alloy spaces, which may effectuate relatively large shifts in the observed lattice parameters that convolute the analysis. Therefore, thermodynamic simulations and crystallographic literature data were employed to construct a system-specific diffraction library of twelve prospective phases for the composition space investigated. Subsequently, for predicted disordered cubic phases, statistical hard-sphere models were established to estimate the lattice parameters and predict diffraction peak positions for inclusion in the library. The library was then employed to analyze diffraction profiles measured from the variably intermixed transition joints, with focus on accounting for both high and low-intensity peaks. 99.0 % of diffraction peaks with relative intensity greater than 0.001 were assigned to phases from the system-specific library, exemplifying rigorous peak accounting and indicating that no unexpected phases were present. Up to six of the twelve library phases were experimentally found in the transition joints. The lattice parameters predicted by the statistical hard-sphere model based upon thermodynamic simulations agree reasonably well with the measured values for the disordered FCC matrix phase.

36 MATERIALS SCIENCE↗
NASA NTRS2008

Wavenumber Domain Focusing of Squinted SAR Data with a Curved Orbit Geometry

Synthetic Aperture Radar systems provide raw data that need focusing to achieve full-resolution imaging. Current SAR applications, including interferometry, require accurate, phase-preserving, and precisely co-registered coherent images over large ground swaths with the highest achievable resolution. In addition to these challenges, stripmap SAR data may be acquired with an off-broadside (squinted) geometry, either by design or through platform motion. The precise batch focusing of these large aperture and wide bandwidth data sets is known to require a 2D frequency processing approach. The standard wave domain focusing algorithm, however, is only exact for data acquired on a rectilinear trajectory. We investigate a generalization of the standard omega-k focusing formulation that allows curved data acquisition tracks. The new formulation can be used in conjunction with a known extension for conical, squinted imaging grids. The approximations necessary to allow the generalized geometry are analyzed to determine the range of applicability of the proposed algorithm. The theory is validated using data simulated with parameters similar to the UAVSAR L-band SAR system.

curved orbit geometry↗
DOE OSTI2025

Toward a Generalizable Prediction Model of Molten Salt Mixture Density with Chemistry‐Informed Transfer Learning

Optimally designing applications of molten salts requires knowledge of their thermophysical properties over a wide range of temperatures and compositions. There exist significant gaps in existing databases and this data can be challenging to experimentally measure due to high temperatures, salt corrosivity, and salt hygroscopicity. Existing databases have been used to create Redlich–Kister (RK) models for mixture density showing improved accuracy with respect to ideal mixing assumptions, but these models require subcomponent data measurements for each new system, therefore lacking generality. In order to address generalizability and data sparsity, a transfer learning procedure is proposed to train deep neural networks (DNNs) using a combination of semi‐empirical relationships (RK), data from the thermophysical arm of the molten salt thermal properties database and universal ab initio properties of component mixtures taken from the joint automated repository for various integrated simulations (JARVIS) classical force‐field inspired descriptors database to predict density in molten salts. Herein, it is shown that DNNs predict molten salt density with an r 2 over 0.99 and a mean absolute percentage error under 1%, outperforming alternative methods.

inorganic materials↗
NASA NTRS2017

Coupled Data Assimilation for Integrated Earth System Analysis and Prediction: Goals, Challenges, and Recommendations

The purpose of this report is to identify fundamental issues for coupled data assimilation (CDA), such as gaps in science and limitations in forecasting systems, in order to provide guidance to the World Meteorological Organization (WMO) on how to facilitate more rapid progress internationally. Coupled Earth system modeling provides the opportunity to extend skillful atmospheric forecasts beyond the traditional two-week barrier by extracting skill from low-frequency state components such as the land, ocean, and sea ice. More generally, coupled models are needed to support seamless prediction systems that span timescales from weather, subseasonal to seasonal (S2S), multiyear, and decadal. Therefore, initialization methods are needed for coupled Earth system models, either applied to each individual component (called Weakly Coupled Data Assimilation - WCDA) or applied the coupled Earth system model as a whole (called Strongly Coupled Data Assimilation - SCDA). Using CDA, in which model forecasts and potentially the state estimation are performed jointly, each model domain benefits from observations in other domains either directly using error covariance information known at the time of the analysis (SCDA), or indirectly through flux interactions at the model boundaries (WCDA). Because the non-atmospheric domains are generally under-observed compared to the atmosphere, CDA provides a significant advantage over single-domain analyses. Next, we provide a synopsis of goals, challenges, and recommendations to advance CDA: Goals: (a) Extend predictive skill beyond the current capability of NWP (e.g. as demonstrated by improving forecast skill scores), (b) produce physically consistent initial conditions for coupled numerical prediction systems and reanalyses (including consistent fluxes at the domain interfaces), (c) make best use of existing observations by allowing observations from each domain to influence and improve the full earth system analysis, (d) develop a robust observation-based identification and understanding of mechanisms that determine the variability of weather and climate, (e) identify critical weaknesses in coupled models and the earth observing system, (f) generate full-field estimates of unobserved or sparsely observed variables, (g) improve the estimation of the external forcings causing changes to climate, (h) transition successes from idealized CDA experiments to real-world applications. Challenges: (a) Modeling at the interfaces between interacting components of coupled Earth system models may be inadequate for estimating uncertainty or error covariances between domains, (b) current data assimilation methods may be insufficient to simultaneously analyze domains containing multiple spatiotemporal scales of interest, (c) there is no standardization of observation data or their delivery systems across domains, (d) the size and complexity of many large-scale coupled Earth system models makes it is difficult to accurately represent uncertainty due to model parameters and coupling parameters, (e) model errors lead to local biases that can transfer between the different Earth system components and lead to coupled model biases and long-term model drift, (e) information propagation across model components with different spatiotemporal scales is extremely complicated, and must be improved in current coupled modeling frameworks, (h) there is insufficient knowledge on how to represent evolving errors in non-atmospheric model components (e.g. as sea ice, land and ocean) on the timescales of NWP.

CDA↗
DOE OSTI2023

Derivative-based SINDy (DSINDy): Addressing the challenge of discovering governing equations from noisy data

Recent advances in the field of data-driven dynamics allow for the discovery of ODE systems using state measurements. One approach, known as Sparse Identification of Nonlinear Dynamics (SINDy), assumes the dynamics are sparse within a predetermined basis in the states and finds the expansion coefficients through linear regression with sparsity constraints. This approach requires an accurate estimation of the state time derivatives, which is not necessarily possible in the high-noise regime without additional constraints. We present an approach called Derivative-based SINDy (DSINDy) that combines two novel methods to improve ODE recovery at high-noise levels. First, we denoise the state variables by applying a projection operator that leverages the assumed basis for the system dynamics. Second, we use a second order cone program (SOCP) to find the derivative and governing equations simultaneously. We derive theoretical results for the projection-based denoising step, which allow us to estimate the values of hyperparameters used in the SOCP formulation. This underlying theory helps limit the number of required user-specified parameters. Finally, we present results demonstrating that our approach leads to improved system recovery for the Van der Pol oscillator, the Duffing oscillator, the Rössler attractor, and the Lorenz 96 model.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2007

Build-Up Approach to Updating the Mock Quiet Spike Beam Model

When a new aircraft is designed or a modification is done to an existing aircraft, the aeroelastic properties of the aircraft should be examined to ensure the aircraft is flight worthy. Evaluating the aeroelastic properties of a new or modified aircraft can include performing a variety of analyses, such as modal and flutter analyses. In order to produce accurate results from these analyses, it is imperative to work with finite element models (FEM) that have been validated by or correlated to ground vibration test (GVT) data, Updating an analytical model using measured data is a challenge in the area of structural dynamics. The analytical model update process encompasses a series of optimizations that match analytical frequencies and mode shapes to the measured modal characteristics of structure. In the past, the method used to update a model to test data was "trial and error." This is an inefficient method - running a modal analysis, comparing the analytical results to the GVT data, manually modifying one or more structural parameters (mass, CG, inertia, area, etc.), rerunning the analysis, and comparing the new analytical modal characteristics to the GVT modal data. If the match is close enough (close enough defined by analyst's updating requirements), then the updating process is completed. If the match does not meet updating-requirements, then the parameters are changed again and the process is repeated. Clearly, this manual optimization process is highly inefficient for large FEM's and/or a large number of structural parameters. NASA Dryden Flight Research Center (DFRC) has developed, in-house, a Mode Matching Code that automates the above-mentioned optimization process, DFRC's in-house Mode Matching Code reads mode shapes and frequencies acquired from GVT to create the target model. It also reads the current analytical model, as we11 as the design variables and their upper and lower limits. It performs a modal analysis on this model and modifies it to create an updated model that has similar mode shapes and frequencies as those of the target model. The Mode Matching Code output frequencies and modal assurance criteria (MAC) values that allow for the quantified comparison of the updated model versus the target model. A recent application of this code is the F453 supersonic flight testing platform, NASA DFRC possesses a modified F-15B that is used as a test bed aircraft for supersonic flight experiments. Traditionally, the finite element model of the test article is generated. A GVT is done on the test article ta validate and update its FEM. This FEM is then mated to the F-15B model, which was correlated to GVT data in fall of 2004, A GVT is conducted with the test article mated to the aircraft, and this mated F-15B/ test article FEM is correlated to this final GVT.

Herrera, Claudia Y.↗
NASA NTRS1985

Challenger STS-17 (41-G) post-flight best estimate trajectory products: Development and summary results

Results from the STS-17 (41-G) post-flight products are presented. Operational Instrumentation recorder gaps, coupled with the limited tracking coverage available for this high inclination entry profile, necessitated selection of an anchor epoch for reconstruction corresponding to an unusually low altitude of h approx. 297 kft. The final inertial trajectory obtained, BT17N26/UN=169750N, is discussed in Section I, i.e., relative to the problems encountered with the OI and ACIP recorded data on this Challenger flight. Atmospheric selection, again in view of the ground track displacement from the remote meteorological sites, constituted a major problem area as discussed in Section II. The LAIRS file provided by Langley was adopted, with NOAA data utilized over the lowermost approx. 7 kft. As discussed in Section II, the Extended BET, ST17BET/UN=274885C, suggests a limited upper altitude (H approx. 230 kft) for which meaningful flight extraction can be expected. This is further demonstrated, though not considered a limitation, in Section III wherein summary results from the AEROBET (NJ0333 with NJ0346 as duplicate) are presented. GTFILEs were generated only for the selected IMU (IMU2) and the Rate Gyro Assembly/Accelerometer Assembly data due to the loss of ACIP data. Appendices attached present inputs for the generation of the post-flight products (Appendix A), final residual plots (Appendix B), a two second spaced listing of the relevant parameters from the Extended BET (Appendix C), and an archival section (Appendix D) devoting input (source) and output files and/or physical reels.

Kelly, G. M.↗
DOE OSTI2020

Leveraging Real-World Driving Data for Design and Impact Evaluation of Energy Efficient Control Strategies

Modeling and simulation are crucial in the development of advanced energy efficient control strategies. Utilizing real-world driving data as the underlying basis for control design and simulation lends veracity to projected real-world energy savings. Standardized drive cycles are limited in their utility for evaluating advanced driving strategies that utilize connectivity and on-vehicle sensing, primarily because they are typically intended for evaluating emissions and fuel economy under controlled conditions. Real-world driving data, because of its scale, is a useful representation of various road types, driving styles, and driving environments. The scale of real-world data also presents challenges in effectively using it in simulations. A fast and efficient simulation methodology is necessary to handle the large number of simulations performed for design analysis and impact evaluation of control strategies. In this study, two methods are presented of leveraging real-world data in both design optimization of energy efficient control strategies and in evaluating the real-world impact of those control strategies upon large-scale deployment. Through these methodologies, strategies with highest impact on energy savings were selected to be implemented as control algorithms. The developed algorithms were incorporated into a vehicle dynamics and powertrain control architecture implemented on a Cadillac CT6 demonstration vehicle. The control algorithms were then exercised on real-world driving scenarios to determine their impact on collective energy savings. The methodology utilizes the large-scale driving data sets maintained by the National Renewable Energy Laboratory to extract real-world driving scenarios and efficient simulation software tools. The insights obtained through this research help in guiding technology selection for energy efficient driving controls.

ARPA-E↗
NASA NTRS2019

Surface Biology & Geology Pathfinder Data Analysis Pipeline

NASA's future global orbital mission, currently in development as the Surface Biology and Geology (SBG) Designated Observable study, will acquire relatively high resolution solar-reflected spectroscopy and thermal infrared observations. Innovative processes must be utilized for handling the high volume of data anticipated to be collected, which is anticipated to exceed 100 terabytes/day, greater than NASA's total extant airborne hyperspectral data collection. Collecting, processing/re-processing, disseminating, and exploiting this volume of data presents new challenges. To begin addressing them, NASA is drawing upon the expertise developed from its astrophysics programs to address Earth science and applications. Specifically, NASA is adapting the science processing operations technology developed for the Kepler and TESS planet-hunting missions for imaging spectroscopy data processing. This technology development has been the foundation for the remarkable scientific successes of Kepler and TESS. The Kepler/TESS data processing technology provides a scalable architecture for robust, repeatable, and replicable science and application products while enabling the Earth science community to develop, test, and implement new algorithms. Our effort to leverage this existing capability has begun by ingesting data and applying workflows from the EO-1/Hyperion 17-year mission archive that provides globally sampled visible through shortwave infrared spectra that are representative of SBG data types and volumes. This pathfinding data processing system will help define the solutions to processing SBG data volumes and will enable the scientific community to interact with the data and processing pipeline to create new science products.

Jenkins, Jon↗
DOE OSTI2026

The Art of Automation: Translating Electron Microscopy Workflows Into Automated Processes

Acquiring data using a scanning transmission electron microscope (STEM) is a complex, multi-step process. The intricacy of the process depends on the type of sample, composition of the material, desired results of the experiment, resolution requirement and other experimental factors. Each experiment presents unique complications, such as sample drift and contamination, that the microscopist must consider when acquiring data. All these challenges are handled fluidly and expertly by experienced microscopists, but to reach new levels of innovation in material development, including greater reproducibility, throughput, and precision, the automation of these workflows is essential. The initial phase of this work involved translating intuition-based workflows into discrete, programmable steps. Some common key stages in STEM workflows are the initial tuning, scanning the sample for areas of interest, and then acquiring the data. Each stage can be broken further into specific parameter adjustments, such as aberration correction and dwell time optimization, depending on the experiment. When deconstructing various experiments each step was assessed for automation feasibility based on the amount of real time operator decisions. There are steps that lend themselves to automation more readily than others, such as course focusing and sample screening, but there is potential for full automation of all stages with time. As an initial step, an automated montage routine was developed, allowing for the efficient acquisition of large portions of the sample without requiring continuous intervention from the operator. The automation of this small process of the procedure demonstrates the value of this capability. A major challenge in automation arises from discrepancies between commanded, reported and actual stage movements. Using systematic tests, stage movement was quantified. This error can be corrected algorithmically for more accurate workflows in the future. Expanding automation capabilities would result in larger, more efficient data acquisition which allows for more robust statistical analysis. Additionally, this work lays the groundwork for a closed loop system where machine learning algorithms would intake automatically acquired data and make real time decisions. By progressively automating this instrument, this work establishes the foundation for fully automated experimentation in transmission electron microscopy.

97 MATHEMATICS AND COMPUTING↗