Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Challenge”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Exascale Computing and Data Handling: Challenges and Opportunities for Weather and Climate Prediction

The emergence of exascale computing and artificial intelligence offer tremendous potential to significantly advance Earth system prediction capabilities. However, enormous challenges must be overcome to adapt models and prediction systems to use these new technologies effectively. A 2022 WMO report on exascale computing recommends “urgency in dedicating efforts and attention to disruptions associated with evolving computing technologies that will be increasingly difficult to overcome, threatening continued advancements in weather and climate prediction capabilities.” Further, the explosive growth in data from observations, model and ensemble output, and postprocessing threatens to overwhelm the ability to deliver timely, accurate, and precise information needed for decision-making. Artificial intelligence (AI) offers untapped opportunities to alter how models are developed, observations are processed, and predictions are analyzed and extracted for decision-making. Given the extraordinarily high cost of computing, growing complexity of prediction systems, and increasingly unmanageable amount of data being produced and consumed, these challenges are rapidly becoming too large for any single institution or country to handle. This paper describes key technical and budgetary challenges, identifies gaps and ways to address them, and makes a number of recommendations.

Atmosphere↗
DOE OSTI2019

The future of legume genetic data resources: Challenges, opportunities, and priorities

Abstract Legumes, comprising one of the largest, most diverse, and most economically important plant families, are the subject of vibrant research and development worldwide. Continued improvement of legume crops will benefit from the recent proliferation of genetic (including genomic) resources; but the diversity, scale, and complexity of these resources presents challenges to those managing and using them. A workshop held in March of 2019 addressed questions of data resources and priorities for the legumes. The workshop identified various needs and recommendations: (a) Develop strategies to effectively store, integrate, and relate genetic resources collected in different projects. (b) Leverage information collected across many legume species by standardizing data formats and ontologies, improving the state of metadata about datasets, and increasing use of the FAIR data principles. (c) Advocate for the critical role that curators exercise in integrating complex datasets into databases and adding high value metadata that enable downstream analytics and facilitate practical applications. (d) Implement standardized software and database development practices to best leverage limited developer time and expertise gained from the various legume (and other) species. (e) Develop tools and databases that can manage genetic information for the world's plant genetic resources, enabling efficient incorporation of important traits into breeding programs. (f) Centralize information on databases, tools, and training materials and establish funding streams to support training and outreach.

Bauchet, Guillaume J.↗
DOE OSTI2015

Challenge Online Time Series Clustering For Demand Response A Theory to Break the ‘Curse of Dimensionality'

The advent of smart meters and advanced communication infrastructures catalyzes numerous smart grid applications such as dynamic demand response, and paves the way to solve challenging research problems in sustainable energy consumption. The space of solution possibilities are restricted primarily by the huge amount of generated data requiring considerable computational resources and efficient algorithms. To overcome this Big Data challenge, data clustering techniques have been proposed. Current approaches however do not scale in the face of the “increasing dimensionality” problem where a cluster point is represented by the entire customer consumption time series. To overcome this aspect we first rethink the way cluster points are created and designed, and then design an efficient online clustering technique for demand response (DR) in order to analyze high volume, high dimensional energy consumption time series data at scale, and on the fly. Our online algorithm is randomized in nature, and provides optimal performance guarantees in a computationally efficient manner. Unlike prior work we (i) study the consumption properties of the whole population simultaneously rather than developing individual models for each customer separately, claiming it to be a ‘killer’ approach that breaks the “curse of dimensionality” in online time series clustering, and (ii) provide tight performance guarantees in theory to validate our approach. Our insights are driven by the field of sociology, where collective behavior often emerges as the result of individual patterns and lifestyles.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2022

Method for automatic correction of offset drift in online sensors

Abstract Successful operation and optimization of water treatment systems hinge on the availability of high-quality online sensor measurements. Ideally, the available measurements should be simultaneously accurate (i.e., unbiased and precise), representative, voluminous, and timely. This remains a pain-point in current water infrastructures, forming a barrier to a wider adoption of advanced and autonomous control systems. While short-lived symptoms, such as outliers and spikes, can be detected or corrected with state-of-the-art tools for fault detection and identification, it is much more difficult to detect, diagnose, and correct the symptoms of slow faults, such as changes in offset or sensitivity due to drift. The time scale of drift is often longer than the time scales of the system dynamics of interest. Moreover, sensor drift has been shown to occur at the same time and with similar rates when sensors are exposed to the same conditions. This challenges data quality management strategies based on redundancy. In this contribution, we develop a new method, including both a hands-off sensor calibration mechanism and an information-seeking control architecture that can handle the unique challenge of simultaneous and similar drift in online sensors.

Chowdhury, Dhrubajit↗
DOE OSTI2022

Challenges in Bioinformatics Workflows for Processing Microbiome Omics Data at Scale

The nascent field of microbiome science is transitioning from a descriptive approach of cataloging taxa and functions present in an environment to applying multi-omics methods to investigate microbiome dynamics and function. A large number of new tools and algorithms have been designed and used for very specific purposes on samples collected by individual investigators or groups. While these developments have been quite instructive, the ability to compare microbiome data generated by many groups of researchers is impeded by the lack of standardized application of bioinformatics methods. Additionally, there are few examples of broad bioinformatics workflows that can process metagenome, metatranscriptome, metaproteome and metabolomic data at scale, and no central hub that allows processing, or provides varied omics data that are findable, accessible, interoperable and reusable (FAIR). Here, we review some of the challenges that exist in analyzing omics data within the microbiome research sphere, and provide context on how the National Microbiome Data Collaborative has adopted a standardized and open access approach to address such challenges.

NMDC, Microbiome↗
DOE OSTI2019

Cosmology data analysis challenges and opportunities in the LSST sky survey

Fueled by advances in software, microelectronics, and large optics fabrication, a new type of sky survey will soon begin. In a relentless campaign of 15 second exposures with a 3 gigapixel camera, the Large Synoptic Survey Telescope will cover the sky deeply every week for ten years. LSST will chart billions of remote galaxies, providing multiple probes of the mysterious Dark Matter and Dark Energy. Multiple probes of the effects of dark energy over an unprecedented volume of the universe will allow us to measure how dark energy behaves over time to high precision. Hundreds of petabytes of high dimensional complex data will be mined and compared with Exascale simulations. After reviewing the LSST project, I will describe some of the computational challenges and opportunities.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Modern chemical graph theory

Abstract Graph theory has a long history in chemistry. Yet as the breadth and variety of chemical data is rapidly changing, so too do graph encoding methods and analyses that yield qualitative and quantitative insights. Using illustrative cases within a basic mathematical framework, we showcase modern chemical graph theory's utility in Chemists' analysis and model development toolkit. The encoding of both experimental and simulation data is discussed at various levels of granularity of information. This is followed by a discussion of the two major classes of graph theoretical analyses: identifying connectivity patterns and partitioning methods. Measures, metrics, descriptors, and topological indices are then introduced with an emphasis upon enhancing interpretability and incorporation into physical models. Challenging data cases are described that include strategies for studying time dependence. Throughout, we incorporate recent advancements in computer science and applied mathematics that are propelling chemical graph theory into new domains of chemical study. This article is categorized under: Molecular and Statistical Mechanics > Molecular Dynamics and Monte‐Carlo Methods Structure and Mechanism > Computational Materials Science Structure and Mechanism > Molecular Structures

Leite, Leonardo S. G.↗
NASA NTRS2009

Porting Gravitational Wave Signal Extraction to Parallel Virtual Machine (PVM)

Laser Interferometer Space Antenna (LISA) is a planned NASA-ESA mission to be launched around 2012. The Gravitational Wave detection is fundamentally the determination of frequency, source parameters, and waveform amplitude derived in a specific order from the interferometric time-series of the rotating LISA spacecrafts. The LISA Science Team has developed a Mock LISA Data Challenge intended to promote the testing of complicated nested search algorithms to detect the 100-1 millihertz frequency signals at amplitudes of 10E-21. However, it has become clear that, sequential search of the parameters is very time consuming and ultra-sensitive; hence, a new strategy has been developed. Parallelization of existing sequential search algorithms of Gravitational Wave signal identification consists of decomposing sequential search loops, beginning with outermost loops and working inward. In this process, the main challenge is to detect interdependencies among loops and partitioning the loops so as to preserve concurrency. Existing parallel programs are based upon either shared memory or distributed memory paradigms. In PVM, master and node programs are used to execute parallelization and process spawning. The PVM can handle process management and process addressing schemes using a virtual machine configuration. The task scheduling and the messaging and signaling can be implemented efficiently for the LISA Gravitational Wave search process using a master and 6 nodes. This approach is accomplished using a server that is available at NASA Ames Research Center, and has been dedicated to the LISA Data Challenge Competition. Historically, gravitational wave and source identification parameters have taken around 7 days in this dedicated single thread Linux based server. Using PVM approach, the parameter extraction problem can be reduced to within a day. The low frequency computation and a proxy signal-to-noise ratio are calculated in separate nodes that are controlled by the master using message and vector of data passing. The message passing among nodes follows a pattern of synchronous and asynchronous send-and-receive protocols. The communication model and the message buffers are allocated dynamically to address rapid search of gravitational wave source information in the Mock LISA data sets.

Thirumalainambi, Rajkumar↗
DOE OSTI2023

A Joint Roman Space Telescope and Rubin Observatory Synthetic Wide-Field Imaging Survey

We present and validate 20 deg(2) of overlapping synthetic imaging surveys representing the full depth of the Nancy Grace Roman Space Telescope High-Latitude Imaging Survey (HLIS) and 5 yr of observations of the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST). The two synthetic surveys are summarized, with reference to the existing 300 deg(2) of LSST simulated imaging produced as part of Dark Energy Science Collaboration (DESC) Data Challenge 2 (DC2). Both synthetic surveys observe the same simulated DESC DC2 universe. For the synthetic Roman survey, we simulate for the first time fully chromatic images along with the detailed physics of the Sensor Chip Assemblies derived from lab measurements using the flight detectors. The simulated imaging and resulting pixel-level measurements of photometric properties of objects span a wavelength range of similar to 0.3-2.0 mu m. We also describe updates to the Roman simulation pipeline, changes in how astrophysical objects are simulated relative to the original DC2 simulations, and the resulting simulated Roman data products. We use these simulations to explore the relative fraction of unrecognized blends in LSST images, finding that 20-30 per cent of objects identified in LSST images with i-band magnitudes brighter than 25 can be identified as multiple objects in Roman images. These simulations provide a unique testing ground for the development and validation of joint pixel-level analysis techniques of ground- and space-based imaging data sets in the second half of the 2020s - in particular the case of joint Roman-LSST analyses.

gravitational lensing↗
DOE OSTI2018

Global carbon intensity of crude oil production

Producing, transporting, and refining crude oil into fuels such as gasoline and diesel accounts for ~15 to 40% of the “well-to-wheels” life-cycle greenhouse gas (GHG) emissions of transport fuels. Reducing emissions from petroleum production is of particular importance, as current transport fleets are almost entirely dependent on liquid petroleum products, and many uses of petroleum have limited prospects for near-term substitution (e.g., air travel). Better understanding of crude oil GHG emissions can help to quantify the benefits of alternative fuels and identify the most cost-effective opportunities for oil-sector emissions reductions. Yet, while regulations are beginning to address petroleum sector GHG emissions, and private investors are beginning to consider climate-related risk in oil investments, such efforts have generally struggled with methodological and data challenges. First, no single method exists for measuring the carbon intensity (CI) of oils. Second, there is a lack of comprehensive geographically rich datasets that would allow evaluation and monitoring of life-cycle emissions from oils. We have previously worked to address the first challenge by developing open-source oil-sector CI modeling tools [OPGEE, supplementary materials (SM) 1.1]. Furthermore, we address the second challenge by using these tools to model well-to-refinery CI of all major active oil fields globally—and to identify major drivers of these emissions.

02 PETROLEUM↗
DOE OSTI2018

Review of Data Landscape: Challenges and Opportunities

Traditional (macro-) economic impact analysis has a role in the long-run analysis of the effects of changes in resilience, since in the long-run the economy adjusts to the microeconomic impacts through various mechanisms (Kunreuther & Roth, 1998). However, measures of economic health, growth, or expansion are not sufficient for measuring resilience. The majority of infrastructure in the United States is privately owned and operated or is managed through some private-public arrangement. Most effects from changes in resilience should be assessed through short-run microeconomic analysis since the actions of firms will be spurred by internal economic decisionmaking that will have immediate impacts on local economies. Any forthcoming efforts to include resilience in the economic impacts of long duration EP outages must accommodate the private and simultaneously public nature of the EP infrastructure as well as its role as lifeline infrastructure. Resilience metrics and methodologies will only be helpful to stakeholders if these metrics help them understand the value of improvements to the resilience of communities, infrastructures, or industries. In their paper, Schellenberg et al. (2018) have provided a well thought out "lay of the lane of the common methods of estimating the economic costs of EP disruptions, which includes discussion of the difference between costs of outage and regional economic modeling, strengths and weaknesses of methods, the difficulty in incorporating resilience, data collection and availability issues, and recommendations for future research.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2013

Data Crosscutting Requirements Review

In April 2013, a diverse group of researchers from the U.S. Department of Energy (DOE) scientific community assembled to assess data requirements associated with DOE-sponsored scientific facilities and large-scale experiments. Participants in the review included facilities staff, program managers, and scientific experts from the offices of Basic Energy Sciences, Biological and Environmental Research, High Energy Physics, and Advanced Scientific Computing Research. As part of the meeting, review participants discussed key issues associated with three distinct aspects of the data challenge: 1) processing, 2) management, and 3) analysis. These discussions identified commonalities and differences among the needs of varied scientific communities. They also helped to articulate gaps between current approaches and future needs, as well as the research advances that will be required to close these gaps. Moreover, the review provided a rare opportunity for experts from across the Office of Science to learn about their collective expertise, challenges, and opportunities. The "Data Crosscutting Requirements Review" generated specific findings and recommendations for addressing large-scale data crosscutting requirements.

ASCR↗
NASA NTRS2019

Overcoming the Challenges of Data Integration and Automation

Artificial Intelligence (AI) and machine learning (ML) are gaining increased attention as a way to leverage the world's data to solve problems. Although AI and ML offer much potential, there are often misconceptions about the application of such techniques. This panel discussion includes speakers from airlines and the research community who will present machine learning approaches they have developed on a variety of aviation data including digital flight data, safety reporting data, and traffic flow data. They will explain the purpose of the application, the data used, and the lessons learned in the development and deployment of their solutions. The panel discussion will focus on common pitfalls in developing and AI solution, the dangers of the current hype around AI, tips for gaining value from a machine learning solution, how to determine whether a machine learning approach is appropriate for a problem, and more.

Matthews, Bryan L.↗