Search NASA⌕ Search

SEARCH · Search NASA

Results for “data challenges”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

LCLS Big Data Handling – How I Learned to Stop Worrying and Love the Data Deluge

Advanced data and computing systems are vital to Linac Coherent Light Source (LCLS) operations, data interpretation and overall scientific productivity. The transition to MHz-era operation marks a fundamental change in scale that requires new infrastructure and architectures to link LCLS to the required scale of computing needed for scientific interpretation. The LCLS-II Data System meets big data challenges by implementing configurable data reduction that can adapt to multiple science areas, real-time analysis frameworks to provide visualization and fast feedback, and the ability to transfer data to local and remote computational facilities for near real time analysis at the appropriate scale. Feature extracted information generated in the data analysis pipeline - at the edge, local compute, or remote High-Performance Computing (HPC) resources - can be used to steer experiments and inform user decisions during beam time. Artificial Intelligence and Machine Learning (AI/ML) techniques present new opportunities to rapidly analyse large datasets and direct experiments, but create new challenges in scaling, adaptability, complexity, and trustworthiness. We describe how the LCLS-II Data System architecture addresses its data-driven challenges in the areas of data acquisition, data processing, data management, and workflow orchestration to decrease the overall time-to-science and provide a vision for future developments.

artificial intelligence↗
NASA NTRS1988

Lateral stability and control derivatives extracted from space shuttle Challenger flight data

Flight data taken from six flights of the Space Transportation System shuttle Challenger (STS-6, 7, 8, 11, 13 and 17) during atmospheric entry are analyzed to determine the shuttle lateral aerodynamic characteristics. Maximum likelihood estimation is applied to data derived from accelerometer and rate gyro measurements and trajectory, meteorological and control surface data to estimate lateral-directional stability and control derivatives. The vehicle stability and control surface effectiveness are compared across the flights and to preflight predicted values.

Schiess, James R.↗
NASA NTRS2001

Constituent Data Assimilation: Challenges and Limitations

The assimilation of observations of atmospheric constituents naturally divides into two major pieces. The first is the assimilation of trace gases whose variability is related to atmospheric motions. The second is the assimilation of trace gases which are sharply influenced by chemical exchange between different constituents. In order to advance beyond the initial successes of explorative investigation of assimilation techniques, tremendous challenges must be met to improve the geophysical integrity of assimilated data products. A subject of special interest is ozone near the tropopause. At the tropopause the information from both the observations and the model simulation becomes most uncertain. However a number of important geophysical parameters, e.g. stratosphere-troposphere exchange and tropospheric ozone, require the assimilation to have high accuracy at the tropopause. This talk will review the current status of the quality of assimilated data products near the tropopause, what must be done to improve the assimilation near the tropopause, and the intrinsic limitations that will require additional sources of information in order for the field to advance.

Rood, Richard B.↗
NASA NTRS2006

Scaling Equations for Ballistic Modeling of Solid Rocket Motor Case Breach

This paper explores the development of a series of scaling equations that can take a known nominal motor performance and scale it for small and growing case failures. This model was developed for the Malfunction-Turn Study as part of Return to Flight activities for the Space Shuttle program. To verify the model, data from the Challenger accident (STS- 51L) were used. The model is able to predict the motor performance beyond the last recorded Challenger data and show how the failed right hand booster would have performed if the vehicle had remained intact.

McMillin, Joshua E.↗
DOE OSTI2024

Measurement of the Neutron Electromagnetic Form Factor Ratio at High Momentum Transfer

The inner structure of the nucleon (proton and neutron) remains a topic of great interest in nuclear and particle physics, after many decades of study. For example, understanding the quark-gluon dynamics inside the nucleon would shed light on how 99% of the nucleon mass is created. The neutron electromagnetic form factors, Gn E and Gn M , give important insights into the neutron structure. The Super BigBite Spectrometer (SBS) program at Jefferson Lab (JLab) seeks to extend the form factor measurements for both the proton and the neutron. The neutron electric form actor, Gn E , has been historically difficult to measure due to the short lifetime of the free neutron and the small value of Gn E . The GEn-II experiment is part of the SBS program and seeks to measure Gn E , significantly increasing the high momentum transfer coverage. A newly designed polarized 3He target increased the figure of merit by three times compared to previous measurements. The analysis of this data is especially challenging due to the unprecedented high-rate environment caused by the open nature of the spectrometer with a direct line of sight to the target. This required developing new Gas Electron Multiplier (GEM) particle trackers which can cover large areas demanded by this setup and handle particle rates up to 500 kHz/cm2. Rates this high over a large area is unprecedented in particle tracking systems and came with a number of challenges. Data taken in the SBS program was critical to understanding hardware and software solutions that improved the track reconstruction efficiency to be >97% with a position resolution of 70 ?m. In previous experiments the proton electromagnetic form factors, Gp E and Gp M were measured up to Q2 = 8.5 GeV2 and Q2 = 30 GeV2, respectively, while Gn E has only been measured up to Q2 = 3.4 GeV2. The GEn-II experiment has measured the neutron form factor ratio, Gn E/Gn M, at Q2 values of 2.90, 6.50, and 9.47 GeV2 by scattering a polarized electron beam with a polarized 3He target, used here as an effective polarized neutron target, and measuring the double spin asymmetry of the cross section. Previous Gn E measurements do not extend above Q2 = 3.4 GeV2, and therefore this analysis has extended the world data by almost three times. The background correction is especially difficult at the higher Q2 settings leading to large systematic errors. As very exploratory results from this early analysis of the data, we find for Q2 = 2.90 GeV2, Gn E = 0.0157 ±stat 0.0016 ±sys 0.0011, for Q2 = 6.50 GeV2, Gn E = 0.0067 ±stat 0.0019 ±sys 0.0005, and for Q2 = 9.46 GeV2, Gn E = 0.0046 ±stat 0.0023 ±sys 0.0005. These results are compared to predictions from the Dyson-Schwinger Equations (DSE) model and a Relativistic Constituent Quark Model (RCQM).

Jeffas, Sean↗
NASA NTRS2019

A Web of Data Analytics Services

Cloud Computing has become the ubiquitous approach to our Big Data challenge. However, one will quickly discover that moving (a.k.a. forklifting) existing on-premise data analytics solutions to the Cloud doesn’t always translate to costing saving and performance boost. The Cloud’s elasticity, its availability, and its wide selection of computing options and selections of costing models making Cloud an attractive environment to tackle our Big Data challenge. The fact is Cloud, on its own, is not the silver bullet to our daunting challenge need for analyze and derive scientific inferences through vast collections of multi-sensor measurements. We would like to have all scientific data in one easy to access environment, but getting the world of scientific data in one analytic system is immensely difficult to achieve. This paper describes the data analytics web architecture NASA is developing by infusing instances of Integrated Data Analytics systems next to the data. The goal is to minimize unnecessary data movement through collection of data access and analytics webservices for researchers to interact with and analyze measurements without have to download data to their local computer. These services are RESTful and provisioned by the data centers with the help from subject matter and science experts. These services encapsulate the physical computing infrastructure, which could local computing cluster, on-premise or public Cloud environment.

Huang, Thomas↗
DOE OSTI2021

COVID-19 Knowledge Graph -- Dataset for SMCDC 2021 Challenge 2

This repository contains the data for the 2021 Smoky Mountains Computational Sciences Data Challenge (SMCDC21) Challenge 2 -- Finding Novel Links in COVID-19 Knowledge Graph. The total size of all files in this repository is 285MB. Challenge website: https://smc-datachallenge.ornl.gov/2021-challenge-2/ More information about the challenge and the data: https://github.com/ORNL/smcdc-2021-covid-kg

60 APPLIED LIFE SCIENCES↗
DOE OSTI2024

Exascale Computing and Data Handling: Challenges and Opportunities for Weather and Climate Prediction

The emergence of exascale computing and artificial intelligence offer tremendous potential to significantly advance Earth system prediction capabilities. However, enormous challenges must be overcome to adapt models and prediction systems to use these new technologies effectively. A 2022 WMO report on exascale computing recommends “urgency in dedicating efforts and attention to disruptions associated with evolving computing technologies that will be increasingly difficult to overcome, threatening continued advancements in weather and climate prediction capabilities.” Further, the explosive growth in data from observations, model and ensemble output, and postprocessing threatens to overwhelm the ability to deliver timely, accurate, and precise information needed for decision-making. Artificial intelligence (AI) offers untapped opportunities to alter how models are developed, observations are processed, and predictions are analyzed and extracted for decision-making. Given the extraordinarily high cost of computing, growing complexity of prediction systems, and increasingly unmanageable amount of data being produced and consumed, these challenges are rapidly becoming too large for any single institution or country to handle. This paper describes key technical and budgetary challenges, identifies gaps and ways to address them, and makes a number of recommendations.

Atmosphere↗
DOE OSTI2022

Method for automatic correction of offset drift in online sensors

Abstract Successful operation and optimization of water treatment systems hinge on the availability of high-quality online sensor measurements. Ideally, the available measurements should be simultaneously accurate (i.e., unbiased and precise), representative, voluminous, and timely. This remains a pain-point in current water infrastructures, forming a barrier to a wider adoption of advanced and autonomous control systems. While short-lived symptoms, such as outliers and spikes, can be detected or corrected with state-of-the-art tools for fault detection and identification, it is much more difficult to detect, diagnose, and correct the symptoms of slow faults, such as changes in offset or sensitivity due to drift. The time scale of drift is often longer than the time scales of the system dynamics of interest. Moreover, sensor drift has been shown to occur at the same time and with similar rates when sensors are exposed to the same conditions. This challenges data quality management strategies based on redundancy. In this contribution, we develop a new method, including both a hands-off sensor calibration mechanism and an information-seeking control architecture that can handle the unique challenge of simultaneous and similar drift in online sensors.

Chowdhury, Dhrubajit↗
DOE OSTI2022

Challenges in Bioinformatics Workflows for Processing Microbiome Omics Data at Scale

The nascent field of microbiome science is transitioning from a descriptive approach of cataloging taxa and functions present in an environment to applying multi-omics methods to investigate microbiome dynamics and function. A large number of new tools and algorithms have been designed and used for very specific purposes on samples collected by individual investigators or groups. While these developments have been quite instructive, the ability to compare microbiome data generated by many groups of researchers is impeded by the lack of standardized application of bioinformatics methods. Additionally, there are few examples of broad bioinformatics workflows that can process metagenome, metatranscriptome, metaproteome and metabolomic data at scale, and no central hub that allows processing, or provides varied omics data that are findable, accessible, interoperable and reusable (FAIR). Here, we review some of the challenges that exist in analyzing omics data within the microbiome research sphere, and provide context on how the National Microbiome Data Collaborative has adopted a standardized and open access approach to address such challenges.

NMDC, Microbiome↗
DOE OSTI2024

Modern chemical graph theory

Abstract Graph theory has a long history in chemistry. Yet as the breadth and variety of chemical data is rapidly changing, so too do graph encoding methods and analyses that yield qualitative and quantitative insights. Using illustrative cases within a basic mathematical framework, we showcase modern chemical graph theory's utility in Chemists' analysis and model development toolkit. The encoding of both experimental and simulation data is discussed at various levels of granularity of information. This is followed by a discussion of the two major classes of graph theoretical analyses: identifying connectivity patterns and partitioning methods. Measures, metrics, descriptors, and topological indices are then introduced with an emphasis upon enhancing interpretability and incorporation into physical models. Challenging data cases are described that include strategies for studying time dependence. Throughout, we incorporate recent advancements in computer science and applied mathematics that are propelling chemical graph theory into new domains of chemical study. This article is categorized under: Molecular and Statistical Mechanics > Molecular Dynamics and Monte‐Carlo Methods Structure and Mechanism > Computational Materials Science Structure and Mechanism > Molecular Structures

Leite, Leonardo S. G.↗
NASA NTRS2009

Porting Gravitational Wave Signal Extraction to Parallel Virtual Machine (PVM)

Laser Interferometer Space Antenna (LISA) is a planned NASA-ESA mission to be launched around 2012. The Gravitational Wave detection is fundamentally the determination of frequency, source parameters, and waveform amplitude derived in a specific order from the interferometric time-series of the rotating LISA spacecrafts. The LISA Science Team has developed a Mock LISA Data Challenge intended to promote the testing of complicated nested search algorithms to detect the 100-1 millihertz frequency signals at amplitudes of 10E-21. However, it has become clear that, sequential search of the parameters is very time consuming and ultra-sensitive; hence, a new strategy has been developed. Parallelization of existing sequential search algorithms of Gravitational Wave signal identification consists of decomposing sequential search loops, beginning with outermost loops and working inward. In this process, the main challenge is to detect interdependencies among loops and partitioning the loops so as to preserve concurrency. Existing parallel programs are based upon either shared memory or distributed memory paradigms. In PVM, master and node programs are used to execute parallelization and process spawning. The PVM can handle process management and process addressing schemes using a virtual machine configuration. The task scheduling and the messaging and signaling can be implemented efficiently for the LISA Gravitational Wave search process using a master and 6 nodes. This approach is accomplished using a server that is available at NASA Ames Research Center, and has been dedicated to the LISA Data Challenge Competition. Historically, gravitational wave and source identification parameters have taken around 7 days in this dedicated single thread Linux based server. Using PVM approach, the parameter extraction problem can be reduced to within a day. The low frequency computation and a proxy signal-to-noise ratio are calculated in separate nodes that are controlled by the master using message and vector of data passing. The message passing among nodes follows a pattern of synchronous and asynchronous send-and-receive protocols. The communication model and the message buffers are allocated dynamically to address rapid search of gravitational wave source information in the Mock LISA data sets.

Thirumalainambi, Rajkumar↗
DOE OSTI2023

A Joint Roman Space Telescope and Rubin Observatory Synthetic Wide-Field Imaging Survey

We present and validate 20 deg(2) of overlapping synthetic imaging surveys representing the full depth of the Nancy Grace Roman Space Telescope High-Latitude Imaging Survey (HLIS) and 5 yr of observations of the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST). The two synthetic surveys are summarized, with reference to the existing 300 deg(2) of LSST simulated imaging produced as part of Dark Energy Science Collaboration (DESC) Data Challenge 2 (DC2). Both synthetic surveys observe the same simulated DESC DC2 universe. For the synthetic Roman survey, we simulate for the first time fully chromatic images along with the detailed physics of the Sensor Chip Assemblies derived from lab measurements using the flight detectors. The simulated imaging and resulting pixel-level measurements of photometric properties of objects span a wavelength range of similar to 0.3-2.0 mu m. We also describe updates to the Roman simulation pipeline, changes in how astrophysical objects are simulated relative to the original DC2 simulations, and the resulting simulated Roman data products. We use these simulations to explore the relative fraction of unrecognized blends in LSST images, finding that 20-30 per cent of objects identified in LSST images with i-band magnitudes brighter than 25 can be identified as multiple objects in Roman images. These simulations provide a unique testing ground for the development and validation of joint pixel-level analysis techniques of ground- and space-based imaging data sets in the second half of the 2020s - in particular the case of joint Roman-LSST analyses.

gravitational lensing↗
NASA NTRS2019

Overcoming the Challenges of Data Integration and Automation

Artificial Intelligence (AI) and machine learning (ML) are gaining increased attention as a way to leverage the world's data to solve problems. Although AI and ML offer much potential, there are often misconceptions about the application of such techniques. This panel discussion includes speakers from airlines and the research community who will present machine learning approaches they have developed on a variety of aviation data including digital flight data, safety reporting data, and traffic flow data. They will explain the purpose of the application, the data used, and the lessons learned in the development and deployment of their solutions. The panel discussion will focus on common pitfalls in developing and AI solution, the dangers of the current hype around AI, tips for gaining value from a machine learning solution, how to determine whether a machine learning approach is appropriate for a problem, and more.

Matthews, Bryan L.↗