Search NASA⌕ Search

SEARCH · Search NASA

Results for “data challenges”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21
All sourcesNASA NTRSDOE OSTI
NASA NTRS1999

Earth Science Data Archive and Access at the NASA/Goddard Space Flight Center Distributed Active Archive Center (DAAC)

The Goddard Distributed Active Archive Center (DAAC), as an integral part of the Earth Observing System Data and Information System (EOSDIS), is the official source of data for several important earth remote sensing missions. These include the Sea-viewing Wide-Field-of-view Sensor (SeaWiFS) launched in August 1997, the Tropical Rainfall Measuring Mission (TRMM) launched in November 1997, and the Moderate Resolution Imaging Spectroradiometer (MODIS) scheduled for launch in mid 1999 as part of the EOS AM-1 instrumentation package. The data generated from these missions supports a host of users in the hydrological, land biosphere and oceanographic research and applications communities. The volume and nature of the data present unique challenges to an Earth science data archive and distribution system such as the DAAC. The DAAC system receives, archives and distributes a large number of standard data products on a daily basis, including data files that have been reprocessed with updated calibration data or improved analytical algorithms. A World Wide Web interface is provided allowing interactive data selection and automatic data subscriptions as distribution options. The DAAC also creates customized and value-added data products, which allow additional user flexibility and reduced data volume. Another significant part of our overall mission is to provide ancillary data support services and archive support for worldwide field campaigns designed to validate the results from the various satellite-derived measurements. In addition to direct data services, accompanying documentation, WWW links to related resources, support for EOSDIS data formats, and informed response to inquiries are routinely provided to users. The current GDAAC WWW search and order system is being restructured to provide users with a simplified, hierarchical access to data. Data Browsers have been developed for several data sets to aid users in ordering data. These Browsers allow users to specify spatial, temporal, and other parameter criteria in searching for and previewing data.

Leptoukh, Gregory↗
NASA NTRS2012

Innovative Video Diagnostic Equipment for Material Science

Materials science experiments under microgravity increasingly rely on advanced optical systems to determine the physical properties of the samples under investigation. This includes video systems with high spatial and temporal resolution. The acquisition, handling, storage and transmission to ground of the resulting video data are very challenging. Since the available downlink data rate is limited, the capability to compress the video data significantly without compromising the data quality is essential. We report on the development of a Digital Video System (DVS) for EML (Electro Magnetic Levitator) which provides real-time video acquisition, high compression using advanced Wavelet algorithms, storage and transmission of a continuous flow of video with different characteristics in terms of image dimensions and frame rates. The DVS is able to operate with the latest generation of high-performance cameras acquiring high resolution video images up to 4Mpixels@60 fps or high frame rate video images up to about 1000 fps@512x512pixels.

Capuano, G.↗
DOE OSTI2021

Machine Learning Based Software Package for Reproducible Data Analysis for Electron Microscopy

Advances in hardware and computation technologies are enabling large hyperspectral imaging during Electron Microscopy (EM), resulting in datasets being produced at rapidly increasing spatial and temporal resolutions. This presents exciting opportunities for microstructure analysis, but also poses great challenges in data analysis. Under this STTR program, QuesTek Innovations LLC, a leader in the field of integrated computational materials engineering (ICME), collaborated with Lawrence Livermore National Laboratory and Argonne National Laboratory. The team developed an open-source machine learning (ML) powered tool for EM data analysis of multiple systems and data types, and collected EM data for ML model development using the cutting-edge PicoProbe equipment at Argonne National Laboratory.

Lin, Yu↗
NASA NTRS2000

KDD Services at the Goddard Earth Sciences Distributed Active Archive Center

NASA's Goddard Earth Sciences Distributed Active Archive Center (GES DAAC) processes, stores and distributes earth science data from a variety of remote sensing satellites. End users of the data range from instrument scientists to global change and climate researchers to federal agencies and foreign governments. Many of these users apply data mining techniques to large volumes of data (up to 1 TB) received from the GES DAAC. However, rapid advances in processing power are enabling increases in data processing that are outpacing tape drive performance and network capacity. As a result, the proportion of data that can be distributed to users continues to decrease. As mitigation, we are migrating more data mining and mining preparation activities into the data center in order to reduce the data volume that needs to be distributed and to offer the users a more useful and manageable product. This migration of activities faces a number of technical and human-factor challenges. As data reduction and mining algorithms are normally quite specific to the user's research needs, the user's algorithm must be integrated virtually unchanged into the archive environment. Also, the archive itself is busy with everyday data archive and distribution activities and cannot be dedicated to, or even impacted by, the mining activities. Therefore, we schedule KDD 'campaigns' (similar to reprocessing campaigns), during which we schedule a wholesale retrieval of specific data products, offering users the opportunity to extract information from the data being retrieved during the campaign.

Lynnes, Christopher↗
DOE OSTI2026

Machine Learning Digital Twin for Lithium Ion Battery State of Health Predictions

A digital twin system has been established to model the long term degradation of the state of health of lithium ion batteries. Two data streams result from the computational model of the system and from the physical experiment measurements. A machine learning pipeline has been developed at the nexus of these data streams. Leveraging the unique data sources in multiple transfer learning approaches has lead to the development of multiple cell specific machine learning digital twins. Our discussion on this first of its kind technology will cover challenges in data ingestion, scalability, architecture and orchestration, and research findings.

25 - ENERGY STORAGE↗
NASA NTRS2006

Engineering the LISA Project: Systems Engineering Challenges

The Laser Interferometer Space Antenna (LISA) is a joint NASA/ESA mission to detect and measure gravitational waves with periods from 1 s to 10000 s. The systems engineering challenges of developing a giant interferometer, 5 million kilometers on a side, an: numerous. Some of the key challenges are presented in this paper. The organizational challenges imposed by sharing the engineering function between three centers (ESA ESTEC, NASA GSFC, and JPL) across nine time zones are addressed. The issues and approaches to allocation of the acceleration noise and measurement sensitivity budget terms across a traditionally decomposed system are discussed. Additionally, using LISA to detect gravitational waves for the first time presents significant data analysis challenges, many of which drive the project system design. The approach to understanding the implications of science data analysis on the system is also addressed.

systems engineering↗
DOE OSTI2022

MAPPRAISER: A massively parallel map-making framework for multi-kilo pixel CMB experiments

Forthcoming cosmic microwave background (CMB) polarized anisotropy experiments have the potential to revolutionize our understanding of the Universe and fundamental physics. The sought-after, tale-telling signatures will be however distributed over voluminous data sets which these experiments will collect. These data sets will need to be efficiently processed and unwanted contributions due to astrophysical, environmental, and instrumental effects characterized and efficiently mitigated in order to uncover the signatures. This poses a significant challenge to data analysis methods, techniques, and software tools which will not only have to be able to cope with huge volumes of data but to do so with unprecedented precision driven by the demanding science goals posed for the new experiments. A keystone of efficient CMB data analysis is solvers of very large linear systems of equations. Such systems appear in very diverse contexts throughout CMB data analysis pipelines, however they typically display similar algebraic structures and can therefore be solved using similar numerical techniques. Linear systems arising in the so-called map-making problem are one of the most prominent and common ones. In this work we present a massively parallel, flexible and extensible framework, comprised of a numerical library, MIDAPACK, and a high level code, MAPPRAISER, which provide tools for solving efficiently such systems. Here, the framework implements iterative solvers based on conjugate gradient techniques: enlarged and preconditioned using different preconditioners. We demonstrate the framework on simulated examples reflecting basic characteristics of the forthcoming data sets issued by ground-based and satellite-borne instruments, executing it on as many as 16,384 compute cores. The software is developed as an open source project freely available to the community at: https://github.com/B3Dcmb/midapack.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

Evaluating the state-of-the-art in remote volcanic eruption characterization Part II: Ulawun volcano, Papua New Guinea

Retrospective eruption characterization is valuable for advancing our understanding of volcanic systems and evaluating our observational capabilities, especially with remote technologies (defined here as a space-borne system or non-local, ground-based instrumentation which include regional and remote infrasound sensors). In June 2019, the open-system Ulawun volcano, Papua New Guinea, produced a VEI 4 eruption. We combined data from satellites (including Sentinel-2, TROPOMI, MODIS, Himawari-8), the International Monitoring System infrasound network, and GLD360 globally detected lightning with information from the local authorities and social media to characterize the pre-, syn- and post-eruptive behaviour. The Rabaul Volcano Observatory recorded ~24 h of seismicity and detected SO 2 emissions ~16 h before the visually-documented start of the Plinian phase on 26 June at 04:20 UTC. Infrasound and SO2 detections suggest the eruption started during the night on 24 June 2019 at 10:39 UTC ~38 h before ash detections with a gas-dominated jetting phase. Local reports and infrasound detections show that the second phase of the eruption started on 25 June 19:28 UTC with ~6 h of jetting. The first detected lightning occurred on 26 June 00:14 UTC, and ash emissions were first detected by Himawari-8 at 01:00 UTC. Post-eruptive satellite imagery indicates new flow deposits to the south and north of the edifice and ash fall to the west and southwest. In particular, regional infrasound data provided novel insight into eruption onset and syn-eruptive changes in intensity. We conclude that, while remote observations are sufficient for detection and tracking of syn-eruptive changes, key challenges in data latency, acquisition, and synthesis must be addressed to improve future near-real-time characterization of eruptions at minimally-monitored or unmonitored volcanoes.

58 GEOSCIENCES↗
DOE OSTI2021

Gaming the beamlines—employing reinforcement learning to maximize scientific outcomes at large-scale user facilities

Abstract Beamline experiments at central facilities are increasingly demanding of remote, high-throughput, and adaptive operation conditions. To accommodate such needs, new approaches must be developed that enable on-the-fly decision making for data intensive challenges. Reinforcement learning (RL) is a domain of AI that holds the potential to enable autonomous operations in a feedback loop between beamline experiments and trained agents. Here, we outline the advanced data acquisition and control software of the Bluesky suite, and demonstrate its functionality with a canonical RL problem: cartpole. We then extend these methods to efficient use of beamline resources by using RL to develop an optimal measurement strategy for samples with different scattering characteristics. The RL agents converge on the empirically optimal policy when under-constrained with time. When resource limited, the agents outperform a naive or sequential measurement strategy, often by a factor of 100%. We interface these methods directly with the data storage and provenance technologies at the National Synchrotron Light Source II, thus demonstrating the potential for RL to increase the scientific output of beamlines, and layout the framework for how to achieve this impact.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Polarized and unpolarized gluon PDFs: Generative machine learning applications for lattice QCD matrix elements at short distance and large momentum

Lattice quantum chromodynamics (QCD) calculations share a defining challenge by requiring a small finite range of spatial separation z between quark/gluon bilinears for controllable power corrections in the perturbative QCD factorization, and a large hadron boost p z for a successful determination of collinear parton distribution functions (PDFs). However, these two requirements make the determination of PDFs from lattice data very challenging. We present the application of generative machine learning algorithms to estimate the polarized and unpolarized gluon correlation functions utilizing short-distance data and extending the correlation up to z p z ≲ 14 , surpassing the current capabilities of lattice QCD calculations. We train physics-informed machine learning algorithms to learn from the short-distance correlation at z ≲ 0.36 fm and take the limit, p z → ∞ , thereby minimizing possible contamination from the higher-twist effects for a successful reconstruction of the polarized gluon PDF. We also expose the bias and problems with underestimating uncertainties associated with the use of model-dependent and overly constrained functional forms, such as x α ( 1 − x ) β and its variants to extract PDFs from the lattice data. We propose the use of generative machine learning algorithms to mitigate these issues and present our determination of the polarized and unpolarized gluon PDFs in the nucleon. Published by the American Physical Society 2025

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRSDate not supplied

Modeling Stellar Jitter and its Effects on the Stellar Spectrum for More Precise Radial Velocity Measurements

Detection of Earth-size exoplanets is a technological and data analysis challenge. Future progress in Earth-mass exoplanet detection is expected from the development of extreme precision radial velocity (EPRV) measurements. The required precision of these measurements necessitates developing a new physics-based data analysis methodology to discriminate planetary signals from host-star-related effects, taking stellar variability and instrumental uncertainties into account. In this work, we investigate and quantify stellar disturbances of the planet-hosting solar-type star HD121504 from 3D radiative modeling obtained with the StellarBox code. The model was used to determine the statistical properties of the turbulent plasma and to obtain synthetic spectroscopic observations for several Fe I lines at different locations on the stellar disk to mimic high-resolution spectroscopic observations. Presented are the initial results of our effort to characterize stellar jitter properties as well as future objectives.

SMD↗
DOE OSTI2021

Error-controlled, progressive, and adaptable retrieval of scientific data with multilevel decomposition

Extreme-scale simulations and high-resolution instruments have been generating an increasing amount of data, which poses significant challenges to not only data storage during the run, but also post-processing where data will be repeatedly retrieved and analyzed for a long period of time. The challenges in satisfying a wide range of post-hoc analysis needs while minimizing the I/O overhead caused by inappropriate and/or excessive data retrieval should never be left unmanaged. In this paper, we propose a data refactoring, compressing, and retrieval framework capable of 1) fine-grained data refactoring with regard to precision; 2) incrementally retrieving and recomposing the data in terms of various error bounds; and 3) adaptively retrieving data in multi-precision and multi-resolution with respect to different analysis. With the progressive data re-composition and the adaptable retrieval algorithms, our framework significantly reduces the amount of data retrieved when multiple incremental precision are requested and/or the downstream analysis time when coarse resolution is used. Experiments show that the amount of data retrieved under the same progressively requested error bound using our framework is 64% less than that using state-of-the-art single-error-bounded approaches. Parallel experiments with up to 1, 024 cores and ~ 600 GB data in total show that our approach yields 1.36× and 2.52× performance over existing approaches in writing to and reading from persistent storage systems, respectively.

Liang, Xin↗
DOE OSTI2022

Toward CO 2 Multimeasurement Geophysical Monitoring in the North Dakota CarbonSAFE Project

Paper presented at 16th International Conference on Greenhouse Gas Control Technologies (GHGT-16), October 23–27, 2022, Lyon, France. Overcoming data acquisition challenges associated with a lake in the study area, electrical infrastructure around the plant, noise from mine activities, complex near-surface conditions associated with reclaimed mine land, high wind speed, and extremely cold temperatures, data processing, modeling, and inversion of the baseline geophysical data demonstrated the importance of high-quality data for a CO 2 geophysical monitoring program. Implementing more advanced data acquisition and processing techniques in future monitoring surveys can further improve confidence in interpretations of geophysical survey data.

20 FOSSIL-FUELED POWER PLANTS↗
NASA NTRS2013

Geologic Interpretation of Data Sets Collected by Planetary Analog Geology Traverses and by Standard Geologic Field Mapping: A Comparison Study - Part 1

Geologic maps integrate the distributions, contacts, and compositions of rock and sediment bodies as a means to interpret local to regional formative histories. Applying terrestrial mapping techniques to other planets is challenging because data is collected primarily by orbiting instruments, with infrequent, spatiallylimited in situ human and robotic exploration. Although geologic maps developed using remote data sets and limited "Apollo-style" field access likely contain inaccuracies, the magnitude, type, and occurrence of these are only marginally understood. This project evaluates the interpretative and cartographic accuracy of both field- and remote-based mapping approaches by comparing two 1:24,000 scale geologic maps of the San Francisco Volcanic Field (SFVF), north-central Arizona. The first map is based on traditional field mapping techniques, while the second is based on remote data sets, augmented with limited field observations collected during NASA Desert Research & Technology Studies (RATS) 2010 exercises. The RATS mission used Apollo-style methods not only for pre-mission traverse planning but also to conduct geologic sampling as part of science operation tests. Cross-comparison demonstrates that the Apollo-style map identifies many of the same rock units and determines a similar broad history as the field-based map. However, field mapping techniques allow markedly improved discrimination of map units, particularly unconsolidated surficial deposits, and recognize a more complex eruptive history than was possible using Apollo-style data. Further, the distribution of unconsolidated surface units was more obvious in the remote sensing data to the field team after conducting the fieldwork. The study raises questions about the most effective approach to balancing mission costs with the rate of knowledge capture, suggesting that there is an inflection point in the "knowledge capture curve" beyond which additional resource investment yields progressively smaller gains in geologic knowledge.

Eppler, Dean B.↗
DOE OSTI2023

Generating Synthetic Time Series Photovoltaic Data with Real-World Physical Challenges and Noise for Use in Algorithm Test and Validation

The PV Fleet Data Initiative and other projects seek the develop algorithms for automated analysis of PV time series data for extraction of statistical information and other parameters of the data such as degradation rates, soiling loss information, tracker performance, clipping or curtailment, system availability and other valuable information. While there is a vast body of PV data available for application of said extraction algorithms it is difficult to validate these algorithms because the true parameters to be extracted are not known. There has been a wide use of synthetic data in the literature for algorithm validation but this synthetic data is typically very bounded by the problem or topic at hand. The PV Fleet Data Initiative project has demonstrated that real time series PV data almost always includes a host of data quality and physical problems that, in reality, any automated PV abstraction algorithm must handle appropriately. For this reason, this work describes the development of a complex synthetic PV times series data set that includes data quality and physical problems that have been experienced in real world PV data. The various quality and physical problems are documented in the synthetic data so that users can test the validity of various PV extraction algorithms as well as develop new algorithms to solve problems this data set can support.

14 SOLAR ENERGY↗
NASA NTRS2003

RGS Spectroscopy of the Cygnus Loop XA Knot

The observations were performed at the end of April 2002, and the data were received in July 2002. Unfortunately, the observations were badly compromised by high levels of background radiation; one the three observations lost entirely. Two replacement observations were scheduled for November 2002, and were only made available in January of 2003. Consequently, we have had little time to grapple with the unusual data analysis challenges. The search for a postdoctoral fellow has been successfully concluded, and Manami Sasaki began working for us in January 2003. She will be supported in part by these funds, and will be working to help understand these data. Examination of the RGS 'Orders' images indicate the presence of broad emission lines (as expected for the diffuse XA knot). However, examination of the 'Spatial' dispersion/cross-dispersion images indicate that the emission is also broad in the cross-dispersion direction. (As a crosscheck, some of the 'Lockman Hole' datasets were also examined as representative 'sky background' datasets; in these, both types of images are relatively flat (outside the calibration source regions). The quicklook plots of the spectra show the expected O VII and O VIII lines, in addition to a complex around 35 Angstroms; the approx. 35 Angstrom line is likely the C V He-beta line at 34.97 Angstrom, but identifying the additional line(s) will require a more careful reduction of the data. Consequently, there is valuable information to be extracted from these data, but it is complicated by diffuse nature of the emission. Because the angular scale is large, we will have to make use of sky background datasets in order to do the background fitting. A color composite image of OM data in the three UV bands was presented at the 'How does the Galaxy Work?' meeting, and compared to optical and X-ray imaging data. Quantitative analysis will require obtaining the effective bandpasses of the UV filters so that the predominant line and continuum contributions can be identified using plasma shock emission models. In view of the facts that the data were only obtained relatively recently, and the complexity of the data analysis, we request a one year no-cost extension on the grant.

Gaetz, Terrance J.↗
NASA NTRS2018

Steps Toward Improved Integration, Search, and Analysis of Heterogeneous Data in the Astrobiology Habitable Environments Database

The Astrobiology Habitable Environments Database (AHED) is a new data system being developed as a long-term, open-access repository for astrobiology data. AHED is intended to store user-contributed results from NASA or externally-funded research in astrobiology, and to encourage sharing and synergy within the astrobiology community. However, the interdisciplinary nature of astrobiology presents some specific challenges to data management, integration, and analysis within AHED. In some disciplines (e.g., genomics), open databases thrive because the contributed products are fairly uniform and standardized (e.g., sequence data). In astrobiology, each investigation produces a unique set of data products; this makes it difficult to search across different datasets to find similar data, or to combine results from separate investigations. With AHED, we are taking steps to ensure there is adequate metadata - both at the dataset and record levels - to facilitate search, integration, and analysis. At the dataset level, we are developing a new metadata standard for describing astrobiology datasets, with detailed information about content, funding source, and scientific relevance, along with a set of topical keywords for characterizing datasets. At the record level, we are encouraging users to provide more structured content and finer-grained metadata. In many user-contributed science data repositories, few restrictions are placed on the uploaded data format, and minimal or no record-level metadata is required; thus users are unburdened when it comes to data preparation. The tradeoff is that deep integration and search across datasets is almost impossible without standardized structures and metadata. Although AHED users are free to upload minimally-described datasets, they will be encouraged to use database authoring tools (supplied by the underlying platform - Open Data Repository's Data Publisher) plus a set of customizable astrobiology-specific templates to help structure their data and provide standardized metadata. In reward for their extra effort, AHED will be able to deliver enhanced search, discovery, and analysis capabilities.

astrobiology↗
NASA NTRS2019

Developing a Community of Practice for Applied Uses of Future PACE Data to Address Marine Food Security Challenges

External interaction:The Plankton, Aerosol, Cloud, ocean Ecosystem (PACE) mission will include a hyperspectral imaging radiometer to advance ecosystem monitoring beyond heritage retrievals of the concentration of surface chlorophyll and other traditional ocean color variables, offering potential for novel science and applications. PACE is the first NASA ocean color mission to occur under the agency's new and evolving effort to directly engage practical end users prior to satellite launch to increase adoption of this freely available data toward societal challenges. Here we describe early efforts to engage a community of practice around marine food-related resource management, business decisions, and policy analysis. Obviously one satellite cannot meet diverse end user needs at all scales and locations, but understanding downstream needs helps in the assessment of information gaps and planning how to optimize the unique strengths of PACE data in combination with the strengths of other satellite retrievals, in situ measurements, and models. Higher spectral resolution data from PACE can be fused with information from satellites with higher spatial or temporal resolution, plus other information, to enable identification and tracking of new marine biological indicators to guide sustainable management. Accounting for the needs of applied researchers as well as non-traditional users of satellite data early in the PACE mission process will ultimately serve to broaden the base of informed users and facilitate faster adoption of the most advanced science and technology toward the challenge of mitigating food insecurity.

Schollaert Uz, Stephanie↗