Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Challenge”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Smoky Mountain Data Challenge 2021: An Open Call to Solve Scientific Data Challenges Using Advanced Data Analytics and Edge Computing

The 2021 Smoky Mountains Computational Sciences and Engineering Conference enlists scientists from across Oak Ridge National Laboratory (ORNL) and industry to be data sponsors and help create data analytics and edge computing challenges for eminent datasets in a variety of scientific domains. This work describes the significance of each of the eight datasets and their associated challenge questions. The challenge questions for each dataset were required to cover multiple difficulty levels. An international call for participation was sent to students, asking them to form teams of up to six people and apply novel data analytics and edge computing methods to solve these challenges.

Devineni, Pravallika↗
DOE OSTI2017

Data Challenges in Estimating the Capacity Value of Solar Photovoltaics

We examine the robustness of solar capacity-value estimates to three important data issues. The first is the sensitivity to using hourly averaged as opposed to subhourly solar-insolation data. The second is the sensitivity to errors in recording and interpreting load data. The third is the sensitivity to using modeled as opposed to measured solar-insolation data. We demonstrate that capacity-value estimates of solar are sensitive to all three of these factors, with potentially large errors in the capacity-value estimate in a particular year. If multiple years of data are available, the biases introduced by using hourly averaged solar-insolation can be smoothed out. Multiple years of data will not necessarily address the other data-related issues that we examine. Our analysis calls into question the accuracy of a number of solar capacity-value estimates relying exclusively on modeled solar-insolation data that are reported in the literature (including our own previous works). Our analysis also suggests that multiple years' historical data should be used for remunerating solar generators for their capacity value in organized wholesale electricity markets.

14 SOLAR ENERGY↗
DOE OSTI2023

SNAP diffraction dataset for 2023 SMC data challenge

The data provided this challenge is ice under high pressure measured using the Spallation Neutrons and Pressure Diffractometer (SNAP) at the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory. The data is stored in a hdf5 file following the NeXus standard and can be read with tools built for either. While the NeXus format is self-describing, there is benefit to explaining some details. The data is stored in a single NXdata entry within a single NXentry. The NXdata has several fields denoting the 3-dimensional data (signal), the axes (D0 is the Qx axis, D1 is the Qy axis, and D2 is the Qz axis), and fields for the uncertainties and masking information. The data can be quickly viewed using the LoadMD algorithm and slice viewer in the Mantid workbench https://www.mantidproject.org.

36 MATERIALS SCIENCE↗
DOE OSTI2022

NOMAD total scattering dataset for SMC data challenge

The data provided for this challenge was measured using the Nanoscale-Ordered Materials Diffractometer (NOMAD) at the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory. The data is stored in a hdf5 file following the NeXus standard and can be read with tools built for either. While the NeXus format is self-describing, there is benefit to explaining some details. The data is stored in 4 NXentries in the file. The NXentries that begin with 'amorphous_SiO2' are for the amorphous data, and the NXentries that begin with 'crystalbolite_SiO2' are for the crystalline material. Solutions that were produced by the scientist are in the entries that end with '_byhand'. Each of the NXdata groups are the plottable data with the 'signal', axes, and (in the case of by-hand components) 'auxiliary_signals' describing which fields should be used. The by-hand component ranges are listed in a 'component' attribute of the various signals. The filtered Sr data is the Fourier transform of the combined components. The data can be quickly viewed using tools such as Nexpy or HDFview. Most languages have libraries that can work with HDF5 (eg. H5py for python) a partial list is provided at https://manual.nexusformat.org/utilities.html The data can be quickly viewed using tools such as Nexpy or HDFview. https://neutrons.ornl.gov/nomad https://www.hdfgroup.org/solutions/hdf5 https://www.nexusformat.org/

36 MATERIALS SCIENCE↗
DOE OSTI2025

A Parameter-masked Mock Data Challenge for Beyond-two-point Galaxy Clustering Statistics

The past few years have seen the emergence of a wide array of novel techniques for analyzing high-precision data from upcoming galaxy surveys, which aim to extend the statistical analysis of galaxy clustering data beyond the linear regime and the canonical two-point (2pt) statistics. We test and benchmark some of these new techniques in a community data challenge named “Beyond-2pt,” initiated during the Aspen 2022 Summer Program “Large-Scale Structure Cosmology beyond 2-Point Statistics,” whose first round of results we present here. The challenge data set consists of high-precision mock galaxy catalogs for clustering in real space, in redshift space, and on a light cone. Participants in the challenge have developed end-to-end pipelines to analyze mock catalogs and extract unknown (“masked”) cosmological parameters of the underlying ΛCDM models with their methods. The methods represented are density-split clustering, nearest neighbor statistics, BACCO power spectrum emulator, void statistics, LEFTfield field-level inference using effective field theory (EFT), and joint power spectrum and bispectrum analyses using both EFT and simulation-based inference. In this work, we review the results of the challenge, focusing on problems solved, lessons learned, and future research needed to perfect the emerging beyond-2pt approaches. The unbiased parameter recovery demonstrated in this challenge by multiple statistics and the associated modeling and inference frameworks supports the credibility of cosmology constraints from these methods. The challenge data set is publicly available, and we welcome future submissions from methods that are not yet represented.

Krause, Elisabeth [Univ. of Arizona, Tucson, AZ (U↗
DOE OSTI2020

Smoky Mountain Data Challenge 2020: An Open Call to Solve Data Problems in the Areas of Neutron Science, Material Science, Urban Modeling and Dynamics, Geophysics, and Biomedical Informatics

The 2020 Smoky Mountains Computational Sciences and Engineering Conference enlists research scientists from across Oak Ridge National Laboratory (ORNL) to be data sponsors and help create data analytics challenges for eminent data sets at the laboratory. This work describes the significance of each of the seven data sets and their as- sociated challenge questions. The challenge questions for each data set were required to cover multiple difficulty levels. An international call for participation was sent to students, and researchers asking them to form teams of up to four people to apply novel data analytics techniques to these data sets.

Parete-Koon, Suzanne↗
DOE OSTI2023

Data Challenges in Multi-Sensor Data Science System for Monitoring a Solvent Extraction Process

Idaho National Laboratory (INL) is maintaining and gaining knowledge into the nuclear fuel cycle by building a test bed to allow researchers the opportunity to study nuclear fuel processing operations. This includes studying solvent extraction processes that use centrifugal contactors. As part of INL’s mission, the goal of this project is to develop a system that utilizes non-traditional measurement sources such as vibration, acoustics, current, light, flow, and temperature in conjunction with data-based, machine learning techniques that will allow for signal discovery. This multisensory data can support the development of safeguards by design, provide operator process awareness, and discover process anomalies. This poster will highlight some of the data collection and analytics challenges for the multi-sensor system as well as the mitigation strategies to build a robust system. Additionally, some preliminary data from the first testing campaign will be shown to help illustrate the data needs of the system.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗
DOE OSTI2023

The LSST AGN Data Challenge: Selection Methods

Abstract Development of the Rubin Observatory Legacy Survey of Space and Time (LSST) includes a series of Data Challenges (DCs) arranged by various LSST Scientific Collaborations that are taking place during the project's preoperational phase. The AGN Science Collaboration Data Challenge (AGNSC-DC) is a partial prototype of the expected LSST data on active galactic nuclei (AGNs), aimed at validating machine learning approaches for AGN selection and characterization in large surveys like LSST. The AGNSC-DC took place in 2021, focusing on accuracy, robustness, and scalability. The training and the blinded data sets were constructed to mimic the future LSST release catalogs using the data from the Sloan Digital Sky Survey Stripe 82 region and the XMM-Newton Large Scale Structure Survey region. Data features were divided into astrometry, photometry, color, morphology, redshift, and class label with the addition of variability features and images. We present the results of four submitted solutions to DCs using both classical and machine learning methods. We systematically test the performance of supervised models (support vector machine, random forest, extreme gradient boosting, artificial neural network, convolutional neural network) and unsupervised ones (deep embedding clustering) when applied to the problem of classifying/clustering sources as stars, galaxies, or AGNs. We obtained classification accuracy of 97.5% for supervised models and clustering accuracy of 96.0% for unsupervised ones and 95.0% with a classic approach for a blinded data set. We find that variability features significantly improve the accuracy of the trained models, and correlation analysis among different bands enables a fast and inexpensive first-order selection of quasar candidates.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2020

Square Kilometre Array Science Data Challenge 1: analysis and results

ABSTRACT As the largest radio telescope in the world, the Square Kilometre Array (SKA) will lead the next generation of radio astronomy. The feats of engineering required to construct the telescope array will be matched only by the techniques developed to exploit the rich scientific value of the data. To drive forward the development of efficient and accurate analysis methods, we are designing a series of data challenges that will provide the scientific community with high-quality data sets for testing and evaluating new techniques. In this paper, we present a description and results from the first such Science Data Challenge 1 (SDC1). Based on SKA MID continuum simulated observations and covering three frequencies (560, 1400, and 9200 MHz) at three depths (8, 100, and 1000 h), SDC1 asked participants to apply source detection, characterization, and classification methods to simulated data. The challenge opened in 2018 November, with nine teams submitting results by the deadline of 2019 April. In this work, we analyse the results for eight of those teams, showcasing the variety of approaches that can be successfully used to find, characterize, and classify sources in a deep, crowded field. The results also demonstrate the importance of building domain knowledge and expertise on this kind of analysis to obtain the best performance. As high-resolution observations begin revealing the true complexity of the sky, one of the outstanding challenges emerging from this analysis is the ability to deal with highly resolved and complex sources as effectively as the unresolved source population.

Bonaldi, A.↗
DOE OSTI2020

The LSST DESC data challenge 1: generation and analysis of synthetic images for next-generation surveys

Data Challenge 1 (DC1) is the first synthetic data set produced by the Rubin Observatory Legacy Survey of Space and Time (LSST) Dark Energy Science Collaboration (DESC). DC1 is designed to develop and validate data reduction and analysis and to study the impact of systematic effects that will affect the LSST data set. DC1 is comprised of r -band observations of 40 deg 2 to 10 yr LSST depth. In this paper, we present each stage of the simulation and analysis process: (a) generation, by synthesizing sources from cosmological N -body simulations in individual sensor-visit images with different observing conditions; (b) reduction using a development version of the LSST Science Pipelines; and (c) matching to the input cosmological catalogue for validation and testing. We verify that testable LSST requirements pass within the fidelity of DC1. We establish a selection procedure that produces a sufficiently clean extragalactic sample for clustering analyses and we discuss residual sample contamination, including contributions from inefficiency in star–galaxy separation and imperfect deblending. We compute the galaxy power spectrum on the simulated field and conclude that: (i) survey properties have an impact of 50 per cent of the statistical uncertainty for the scales and models used in DC1; (ii) a selection to eliminate artefacts in the catalogues is necessary to avoid biases in the measured clustering; and (iii) the presence of bright objects has a significant impact (2σ–6σ) in the estimated power spectra at small scales (ℓ > 1200), highlighting the impact of blending in studies at small angular scales in LSST.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2019

Getting Data Out of the Ground: Modern Data Challenges Facing EGS Collab, the DOE Geothermal Data Repository, and the Geothermal Industry

The amount of data being generated by the geothermal industry is increasing exponentially. Advanced tools, higher resolution sensors, and digitally integrated equipment are providing valuable insight into geothermal exploration and operational efficiencies, but the volumes of data they generate and the unique properties of Enhanced Geothermal System (EGS) projects are creating challenges of their own. This paper explores those challenges and the steps the U.S. Department of Energy (DOE) is taking to address them through comprehensive data management strategies, improvements to DOE's Geothermal Data Repository (GDR), and the development of a secure data collaboration tool for DOE's Frontier Observatory for Research in Geothermal Energy (FORGE) and EGS Collab projects.

15 GEOTHERMAL ENERGY↗
DOE OSTI2018

Epidemiological Data Challenges: Planning for a More Robust Future Through Data Standards

Accessible epidemiological data are of great value for emergency preparedness and response, understanding disease progression through a population, and building statistical and mechanistic disease models that enable forecasting. The status quo, however, renders acquiring and using such data difficult in practice. In many cases, a primary way of obtaining epidemiological data is through the internet, but the methods by which the data are presented to the public often differ drastically among institutions. As a result, there is a strong need for better data sharing practices. This paper identifies, in detail and with examples, the three key challenges one encounters when attempting to acquire and use epidemiological data: (1) interfaces, (2) data formatting, and (3) reporting. These challenges are used to provide suggestions and guidance for improvement as these systems evolve in the future. If these suggested data and interface recommendations were adhered to, epidemiological and public health analysis, modeling, and informatics work would be significantly streamlined, which can in turn yield better public health decision-making capabilities.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2025

RTN-107: The Rubin Observatory Target-of-Opportunity Mock Data Challenge

We describe the activities of the Target-of-Opportunity mock data challenge, taking place from Sep 22 2025 - Oct 18 2025. We center this activity in four questions that are critical for maximizing the scientific output of the ToO system: (1) How quickly can Rubin Observatory start observing after a ToO alert is received? (2) How efficient is Rubin Observatory at recovering the host of a ToO event? (3) How accurate are the observing strategies that the community has created for the ToO program? (4) How can expert ToO scientists interact effectively with the ToO system, where many processes are fully automated? In this challenge, and the report summarized herein, we aim to answer the aforementioned questions to better the Rubin ToO program.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2015

Challenges and solutions for the analysis of in situ, in crystallo micro-spectrophotometric data

The particular challenge of the analysis of optical absorption and Raman spectroscopic data measured from protein crystals and how the SLS-APE software toolbox supports scientists in dealing with such data is described. Combining macromolecular crystallography with in crystallo micro-spectrophotometry yields valuable complementary information on the sample, including the redox states of metal cofactors, the identification of bound ligands and the onset and strength of undesired photochemistry, also known as radiation damage. However, the analysis and processing of the resulting data differs significantly from the approaches used for solution spectrophotometric data. The varying size and shape of the sample, together with the suboptimal sample environment, the lack of proper reference signals and the general influence of the X-ray beam on the sample have to be considered and carefully corrected for. In the present article, how to characterize and treat these sample-dependent artefacts in a reproducible manner is discussed and the SLS-APEin situ, in crystallo optical spectroscopy data-analysis toolbox is demonstrated.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2024

MSD CoP Webinar: AI and Extreme Events - Overcoming Data Challenges for Improved Characterization of Climate Extremes

Context: This webinar was hosted by the MultiSector Dynamics Community of Practice (MSD CoP; https://multisectordynamics.org). Abstract: Artificial Intelligence (AI) models require large volumes of data for training and testing. Data requirements present challenges for using AI to explore extreme events with limited observational data. This webinar will showcase two innovative methods developed by part of the European Climate Intelligence (CLINT) project to overcome data challenges and harness AI to improve our understanding of climate extremes. Dr. Ascenso will present his research on data augmentation methods to improve estimates of tropical cyclones using satellite data. His presentation will review established methods for data augmentation and explore opportunities and challenges for using generative AI to generate images of extreme, life-threatening tropical cyclones. Next, Dr. Plesiat will present his research on deep learning techniques to overcome limited observational data sets. His presentation will illustrate deep learning methods to develop AI reconstructions of four climate indices across Europe. Presenters : Dr. Guido Ascenso (post-doctoral researcher, Politecnico di Milano); Dr. Étienne Plésiat (German Climate Computing Centre - DKRZ) Moderator(s): Stefano Galelli (MSD CoP WG Co-Lead), David Gold (MSD CoP WG Co-Lead), Jillian Sturtevant (MSD CoP WG Communications Officer), Matteo Giuliani (Politecnico di Milano, MSD CoP WG Member, Moderator and Organizer) This webinar was held on: October 11, 2024 from 11AM - 1PM ET

AI↗
DOE OSTI2019

Challenge data set for macromolecular multi-microcrystallography

A synthetic data set demonstrating a particularly challenging case of indexing ambiguity in the context of radiation damage was generated. This set shall serve as a standard benchmark and reference point for the ongoing development of new methods and new approaches to robust structure solution when single-crystal methods are insufficient. Of the 100 short wedges of data, only the first 36 are currently necessary to solve the structure by `cheating', or using the correct reference structure as a guide. The total wall-clock time and number of crystals required to solve the structure without cheating is proposed as a metric for the efficacy and efficiency of a given multi-crystal automation pipeline.

36 MATERIALS SCIENCE↗
DOE OSTI2013

Large-Scale Data Challenges in Future Power Grids

This paper describes technical challenges in supporting large-scale real-time data analysis for future power grid systems and discusses various design options to address these challenges. Even though the existing U.S. power grid has served the nation remarkably well over the last 120 years, big changes are in the horizon. The widespread deployment of renewable generation, smart grid controls, energy storage, plug-in hybrids, and new conducting materials will require fundamental changes in the operational concepts and principal components. The whole system becomes highly dynamic and needs constant adjustments based on real time data. Even though millions of sensors such as phase measurement units (PMUs) and smart meters are being widely deployed, a data layer that can support this amount of data in real time is needed. Unlike the data fabric in cloud services, the data layer for smart grids must address some unique challenges. This layer must be scalable to support millions of sensors and a large number of diverse applications and still provide real time guarantees. Moreover, the system needs to be highly reliable and highly secure because the power grid is a critical piece of infrastructure. No existing systems can satisfy all the requirements at the same time. We examine various design options. In particular, we explore the special characteristics of power grid data to meet both scalability and quality of service requirements. Our initial prototype can improve performance by orders of magnitude over existing general-purpose systems. The prototype was demonstrated with several use cases from PNNL’s FPGI and was shown to be able to integrate huge amount of data from a large number of sensors and a diverse set of applications.

Smart Grid, Real Time, Commercial Products, Big Da↗