SEARCH · Search NASA

Results for “Data Challenge”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2025

RTN-107: The Rubin Observatory Target-of-Opportunity Mock Data Challenge

We describe the activities of the Target-of-Opportunity mock data challenge, taking place from Sep 22 2025 - Oct 18 2025. We center this activity in four questions that are critical for maximizing the scientific output of the ToO system: (1) How quickly can Rubin Observatory start observing after a ToO alert is received? (2) How efficient is Rubin Observatory at recovering the host of a ToO event? (3) How accurate are the observing strategies that the community has created for the ToO program? (4) How can expert ToO scientists interact effectively with the ToO system, where many processes are fully automated? In this challenge, and the report summarized herein, we aim to answer the aforementioned questions to better the Rubin ToO program.

79 ASTRONOMY AND ASTROPHYSICS↗

NASA NTRSDate not supplied

Big Data Challenges at CCMC

Like other research centers, the Community Coordinated Modeling Center (CCMC, https://ccmc.gsfc.nasa.gov) at NASA Goddard Space Flight Center (GSFC) is also experiencing the big data challenges. CCMC hosts over 80 space weather models for Runs On Request (ROR), Continuous Runs and Instant Runs simulation services for the research community. In addition, CCMC has started to support simulation output onboarding in response to the Open Science initiative. Overall, we have accumulated over petabytes of simulation output data and are rapidly growing. In this presentation, we will discuss our data and storage challenges. We will present our attempts to address our challenges and any associated lessons learned. CCMC uses Apache Airflow to ensure data transfer is consistent. We will give a brief overview on how we leverage Apache Airflow to enhance our environment.

space weather↗

NASA NTRSDate not supplied

Big Data Challenges at the Community Coordinated Modeling Center (CCMC)

space weather↗

NASA NTRS2009

The Status of the Mock LISA Data Challenges

For the last four years, many gravitational-wave researchers around the world have participated in the Mock LISA Data Challenges (MLDCs), a program to demonstrate and encourage the development of LISA data-analysis capabilities, tools and techniques. In this poster, we present a summary of the results of MLDC 3, which was completed in 2009. During MLDC 3, 27 participants from 15 institutions successfully analyzed data sets that included Galactic binaries, coalescing spinning massive black holes, extreme-mass-ratio inspirals, cosmic-string cusp bursts and a stochastic gravitational-wave background. We also describe the technical and scientific challenges that will be addressed by future MLI)Cs, starting with MLDC 4, which is currently in progress.

Baker, John↗

NASA NTRS2021

GLOBE Observer and the GO on a Trail Data Challenge: A Citizen Science Approach to Generating a Global Land Cover Land Use Reference Dataset

Land cover and land use are highly visible indicators of climate change and human disruption to natural processes. While land cover is frequently monitored over a large area using satellite data, ground-based reference data is valuable as a comparison point. The NASA-funded GLOBE Observer (GO) program provides volunteer-collected land cover photos tagged with location, date and time, and, in some cases, land cover type. When making a full land cover observation, volunteers take six photos of the site, one facing north, south, east, and west (N-S-E-W) respectively, one pointing straight up to capture canopy and sky, and one pointing down to document ground cover. Together, the photos document a 100 meter square of land. Volunteers may then optionally tag each N-S-E-W photo with the land cover types present. Volunteers collect the data through a smartphone app, also called GLOBE Observer, resulting in consistent data. While land cover data collected through GLOBE Observer is ongoing, this paper presents the results of a data challenge held between June 1 and October 15, 2019. Called “GO on a Trail,” the challenge focused on collecting land cover observations at designated points along the Lewis and Clark National Historic Trail (LCNHT), which transects approximately 7,900 kilometers of the United States from east to west. The challenge also included an international component in which participants were recognized for the most data collected during the period. The challenge resulted in more than 2800 land cover data points from around the world with just over 900 land cover data points along the LCNHT. The data collected along the LCNHT is largely within a 1,000 meter corridor along the historic route. Internationally, the challenge had strong participation in Australia, where a Scouts Australia team-based competition was conducted in conjunction with GO on a Trail. GLOBE Observer collections can serve as reference data, ground truthing satellite imagery for the improvement and verification of broad land cover maps. Continued collection using this protocol will build a database documenting climate-related land cover and land use change into the future.

land cover↗

DOE OSTI2015

Challenges and solutions for the analysis of in situ, in crystallo micro-spectrophotometric data

The particular challenge of the analysis of optical absorption and Raman spectroscopic data measured from protein crystals and how the SLS-APE software toolbox supports scientists in dealing with such data is described. Combining macromolecular crystallography with in crystallo micro-spectrophotometry yields valuable complementary information on the sample, including the redox states of metal cofactors, the identification of bound ligands and the onset and strength of undesired photochemistry, also known as radiation damage. However, the analysis and processing of the resulting data differs significantly from the approaches used for solution spectrophotometric data. The varying size and shape of the sample, together with the suboptimal sample environment, the lack of proper reference signals and the general influence of the X-ray beam on the sample have to be considered and carefully corrected for. In the present article, how to characterize and treat these sample-dependent artefacts in a reproducible manner is discussed and the SLS-APEin situ, in crystallo optical spectroscopy data-analysis toolbox is demonstrated.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

DOE OSTI2024

MSD CoP Webinar: AI and Extreme Events - Overcoming Data Challenges for Improved Characterization of Climate Extremes

Context: This webinar was hosted by the MultiSector Dynamics Community of Practice (MSD CoP; https://multisectordynamics.org). Abstract: Artificial Intelligence (AI) models require large volumes of data for training and testing. Data requirements present challenges for using AI to explore extreme events with limited observational data. This webinar will showcase two innovative methods developed by part of the European Climate Intelligence (CLINT) project to overcome data challenges and harness AI to improve our understanding of climate extremes. Dr. Ascenso will present his research on data augmentation methods to improve estimates of tropical cyclones using satellite data. His presentation will review established methods for data augmentation and explore opportunities and challenges for using generative AI to generate images of extreme, life-threatening tropical cyclones. Next, Dr. Plesiat will present his research on deep learning techniques to overcome limited observational data sets. His presentation will illustrate deep learning methods to develop AI reconstructions of four climate indices across Europe. Presenters : Dr. Guido Ascenso (post-doctoral researcher, Politecnico di Milano); Dr. Étienne Plésiat (German Climate Computing Centre - DKRZ) Moderator(s): Stefano Galelli (MSD CoP WG Co-Lead), David Gold (MSD CoP WG Co-Lead), Jillian Sturtevant (MSD CoP WG Communications Officer), Matteo Giuliani (Politecnico di Milano, MSD CoP WG Member, Moderator and Organizer) This webinar was held on: October 11, 2024 from 11AM - 1PM ET

AI↗

DOE OSTI2019

Challenge data set for macromolecular multi-microcrystallography

A synthetic data set demonstrating a particularly challenging case of indexing ambiguity in the context of radiation damage was generated. This set shall serve as a standard benchmark and reference point for the ongoing development of new methods and new approaches to robust structure solution when single-crystal methods are insufficient. Of the 100 short wedges of data, only the first 36 are currently necessary to solve the structure by `cheating', or using the correct reference structure as a guide. The total wall-clock time and number of crystals required to solve the structure without cheating is proposed as a metric for the efficacy and efficiency of a given multi-crystal automation pipeline.

36 MATERIALS SCIENCE↗

NASA NTRS2015

High Resolution Nature Runs and the Big Data Challenge

NASA's Global Modeling and Assimilation Office at Goddard Space Flight Center is undertaking a series of very computationally intensive Nature Runs and a downscaled reanalysis. The nature runs use the GEOS-5 as an Atmospheric General Circulation Model (AGCM) while the reanalysis uses the GEOS-5 in Data Assimilation mode. This paper will present computational challenges from three runs, two of which are AGCM and one is downscaled reanalysis using the full DAS. The nature runs will be completed at two surface grid resolutions, 7 and 3 kilometers and 72 vertical levels. The 7 km run spanned 2 years (2005-2006) and produced 4 PB of data while the 3 km run will span one year and generate 4 BP of data. The downscaled reanalysis (MERRA-II Modern-Era Reanalysis for Research and Applications) will cover 15 years and generate 1 PB of data. Our efforts to address the big data challenges of climate science, we are moving toward a notion of Climate Analytics-as-a-Service (CAaaS), a specialization of the concept of business process-as-a-service that is an evolving extension of IaaS, PaaS, and SaaS enabled by cloud computing. In this presentation, we will describe two projects that demonstrate this shift. MERRA Analytic Services (MERRA/AS) is an example of cloud-enabled CAaaS. MERRA/AS enables MapReduce analytics over MERRA reanalysis data collection by bringing together the high-performance computing, scalable data management, and a domain-specific climate data services API. NASA's High-Performance Science Cloud (HPSC) is an example of the type of compute-storage fabric required to support CAaaS. The HPSC comprises a high speed Infinib and network, high performance file systems and object storage, and a virtual system environments specific for data intensive, science applications. These technologies are providing a new tier in the data and analytic services stack that helps connect earthbound, enterprise-level data and computational resources to new customers and new mobility-driven applications and modes of work. In our experience, CAaaS lowers the barriers and risk to organizational change, fosters innovation and experimentation, and provides the agility required to meet our customers' increasing and changing needs

big data analysis↗

DOE OSTI2013

Large-Scale Data Challenges in Future Power Grids

This paper describes technical challenges in supporting large-scale real-time data analysis for future power grid systems and discusses various design options to address these challenges. Even though the existing U.S. power grid has served the nation remarkably well over the last 120 years, big changes are in the horizon. The widespread deployment of renewable generation, smart grid controls, energy storage, plug-in hybrids, and new conducting materials will require fundamental changes in the operational concepts and principal components. The whole system becomes highly dynamic and needs constant adjustments based on real time data. Even though millions of sensors such as phase measurement units (PMUs) and smart meters are being widely deployed, a data layer that can support this amount of data in real time is needed. Unlike the data fabric in cloud services, the data layer for smart grids must address some unique challenges. This layer must be scalable to support millions of sensors and a large number of diverse applications and still provide real time guarantees. Moreover, the system needs to be highly reliable and highly secure because the power grid is a critical piece of infrastructure. No existing systems can satisfy all the requirements at the same time. We examine various design options. In particular, we explore the special characteristics of power grid data to meet both scalability and quality of service requirements. Our initial prototype can improve performance by orders of magnitude over existing general-purpose systems. The prototype was demonstrated with several use cases from PNNL’s FPGI and was shown to be able to integrate huge amount of data from a large number of sensors and a diverse set of applications.

Smart Grid, Real Time, Commercial Products, Big Da↗

NASA NTRS2012

Big Data Challenges for Large Radio Arrays

Future large radio astronomy arrays, particularly the Square Kilometre Array (SKA), will be able to generate data at rates far higher than can be analyzed or stored affordably with current practices. This is, by definition, a "big data" problem, and requires an end-to-end solution if future radio arrays are to reach their full scientific potential. Similar data processing, transport, storage, and management challenges face next-generation facilities in many other fields.

Combining↗

NASA NTRS2020

Clouds Around the World: How a Simple Citizen Science Data Challenge Became a Worldwide Success

Citizen science is often recognized for its potential to directly engage the public in science, and is uniquely positioned to support and extend participants’ learning in science. In March 2018, the Global Learning and Observations to Benefit the Environment (GLOBE) Program, NASA’s largest and longest-lasting citizen science program about Earth, organized a month-long event that asked people around the world to contribute daily cloud observations and photographs of the sky (15 March–15 April 2018). What was considered a simple engagement activity turned into an unprecedented worldwide event that garnered major public interest and media recognition, collecting over 55,000 observations from 99 different countries, in more than 15,000 locations, on every continent including Antarctica. The event was called the “Spring Cloud Challenge” and was created to 1) engage the general public in the scientific process and promote the use of the GLOBE Observer app, 2) collect ground-based visual observations of varying cloud types during boreal spring, and 3) increase the number and locations of ground-based visual cloud observations collocated with cloud-observing satellites. The event resulted in roughly 3 times more observations than during the historic and highly publicized 2017 North American total solar eclipse. The dataset also includes observations over the Drake Passage in Antarctica and reports from intense Saharan dust events. This article describes how the challenge was crafted, outreach to volunteer scientists around the world, details of the data collected, and impact of the data.

GLOBE↗

DOE OSTI2020

VULCAN temperature dataset for SMC data challenge

The VULCAN Beamline dataset provides the sample measurement, where temperatures is recorded in two physically different places on the sample. These are held in two different hdf5 groups in the data file.

42 ENGINEERING↗

DOE OSTI2013

Simulation and big data challenges in tuning building energy models

EnergyPlus is the flagship building energy simulation software used to model whole building energy consumption for residential and commercial establishments. A typical input to the program often has hundreds, sometimes thousands of parameters which are typically tweaked by a buildings expert to “get it right”. This process can sometimes take months. “Autotune” is an ongoing research effort employing machine learning techniques to automate the tuning of the input parameters for an EnergyPlus input description of a building. Even with automation, the computational challenge faced to run the tuning simulation ensemble is daunting and requires the use of supercomputers to make it tractable in time. In this paper, we describe the scope of the problem, particularly the technical challenges faced and overcome, and the software infrastructure developed/in development when taking the EnergyPlus engine, which was primarily designed to run on desktops, and scaling it to run on shared memory supercomputers (Nautilus) and distributed memory supercomputers (Frost and Titan). The parametric simulations produce data in the order of tens to a couple of hundred terabytes. We describe the approaches employed to streamline and reduce bottlenecks in the workflow for this data, which is subsequently being made available for the tuning effort as well as made available publicly for open-science.

Sanyal, Jibonananda↗

DOE OSTI2013

Simulation and Big Data Challenges in Tuning Building Energy Models

EnergyPlus is the flagship building energy simulation software used to model whole building energy consumption for residential and commercial establishments. A typical input to the program often has hundreds, sometimes thousands of parameters which are typically tweaked by a buildings expert to get it right . This process can sometimes take months. Autotune is an ongoing research effort employing machine learning techniques to automate the tuning of the input parameters for an EnergyPlus input description of a building. Even with automation, the computational challenge faced to run the tuning simulation ensemble is daunting and requires the use of supercomputers to make it tractable in time. In this proposal, we describe the scope of the problem, the technical challenges faced and overcome, the machine learning techniques developed and employed, and the software infrastructure developed/in development when taking the EnergyPlus engine, which was primarily designed to run on desktops, and scaling it to run on shared memory supercomputers (Nautilus) and distributed memory supercomputers (Frost and Titan). The parametric simulations produce data in the order of tens to a couple of hundred terabytes.We describe the approaches employed to streamline and reduce bottlenecks in the workflow for this data, which is subsequently being made available for the tuning effort as well as made available publicly for open-science.

Sanyal, Jibonananda↗

DOE OSTI2014

The big data challenges of connectomics

The structure of the nervous system is extraordinarily complicated because individual neurons are interconnected to hundreds or even thousands of other cells in networks that can extend over large volumes. Mapping such networks at the level of synaptic connections, a field called connectomics, began in the 1970s with a the study of the small nervous system of a worm and has recently garnered general interest thanks to technical and computational advances that automate the collection of electron-microscopy data and offer the possibility of mapping even large mammalian brains. However, modern connectomics produces ‘big data’, unprecedented quantities of digital information at unprecedented rates, and will require, as with genomics at the time, breakthrough algorithmic and computational solutions. Here in this paper we describe some of the key difficulties that may arise and provide suggestions for managing them.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2021

2021 Smoky Mountains Conference Data Challenge Synthetic-to-Real Domain Adaptation for Autonomous Driving Dataset

The dataset is comprised of both real and synthetic images from a vehicle's forward-facing camera. Each camera image is accompanied by a corresponding pixel-level semantic segmentation image (all files are .png files). In total, the dataset contains 5600 images in the training/validation set and 1400 images in the testing set. The training dataset contains mostly synthetic RGB images collected with a wide range of weather and lighting conditions using the CARLA simulator [1]. In addition, the training data also includes a small pre-selected subset of data from the Cityscapes training dataset â€“ which is comprised of RGB-segmentation image pairs from driving scenarios in various European cities [2]. The testing data is split into three sets. The first set contains synthetic CARLA images with weather/lighting conditions that were not present in the training set. The second set is a subset of the Cityscapes testing dataset. Finally, the third set is an unknown testing set which will not be revealed to the participants until after the submission deadline. [1] Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. (2017, October). CARLA: An open urban driving simulator. In Conference on robot learning (pp. 1-16). PMLR. [2] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., ... and Schiele, B. (2016). The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3213-3223).

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2021

Deep glassy state dynamic data challenge glass models: Elastic models

The idea of an “ideal” glass transition temperature has persisted at least since the work from Kauzmann when it was observed that the entropy of glass-forming liquids extrapolated to below that of the crystal, thus suggesting the need for a phase transition at a finite or non-zero absolute temperature. This thermodynamic paradox was also found to be related to the observations of a diverging of the extrapolated viscosity or relaxation times at a temperature near to this ideal glass transition. Recently, however, we have carried out experiments using both an ancient amber material and an ultra-stable amorphous fluoropolymer that challenge the ideas of the divergence of the viscosity or relaxation times at this ideal glass transition. In the present manuscript we have evaluated two theories of the glass transition that are based on ideas related to elasticity of the amorphous glass-forming material. We find that the models from both J. Dyre and his group and of K.S. Schweizer and his group not only show non-diverging behavior but are also in some agreement with the new data, though still showing somewhat slower dynamics than those observed in the experiments. The work shows that the data are good enough to distinguish between theories and it is suggested that other mechanisms may be needed to fully describe the non-diverging responses of the ultra-stable glasses.

36 MATERIALS SCIENCE↗