Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Challenge”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2015

Data Confidentiality Challenges in Big Data Applications

In this paper, we address the problem of data confidentiality in big data analytics. In many fields, much useful patterns can be extracted by applying machine learning techniques to big data. However, data confidentiality must be protected. In many scenarios, data confidentiality could well be a prerequisite for data to be shared. We present a scheme to provide provable secure data confidentiality and discuss various techniques to optimize performance of such a system.

data confidentiality↗
NASA NTRSDate not supplied

Data Quality Challenges for Analysis Ready Data (ARD)

Data quality plays a critical role in research and applications. The Earth Science Information Partners (ESIP) Information Quality Cluster (IQC) defines four aspects of information quality: Science, Product, Stewardship, and Services. The ESIP IQC has become internationally recognized as an authoritative and responsive resource of information and guidance to data producers and distributors on how to implement data quality standards and best practices for their science data systems, datasets, and data/metadata dissemination services. In recent years, cloud computing environments have provided scale-up capabilities such as data archives and services, enabling interdisciplinary science and applications. More value-added products are expected from data service providers, including Analysis Ready Data (ARD). ARD refers to data that has been preprocessed into a form that allows immediate analysis by the end user, processed to a minimum set of requirements and provides interoperability over time and across multiple datasets. Once a dataset has been developed from its original form to produce ARD, what quality characteristics should the derived dataset or ARD possess? Also, is it safe to assume that the quality of the ARD is consistent with the quality of the source data, or are there special attributes to an ARD that would warrant a secondary, independent quality assessment? What provenance (also called “data lineage”) information needs to be included in ARD? It is important to answer these questions, especially given the ease of use of ARD, and the consequent temptation by users to trust ARD without understanding the limitations or possible variations in quality compared to the source data. In this presentation, we will discuss data quality challenges for ARD products and services and introduce IQC for participation.

data quality↗
DOE OSTI2024

Challenges of open data in aquatic sciences: issues faced by data users and data providers

Free use and redistribution of data (i.e., Open Data) increases the reproducibility, transparency, and pace of aquatic sciences research. However, barriers to both data users and data providers may limit the adoption of Open Data practices. Here, we describe common Open Data challenges faced by data users and data providers within the aquatic sciences community (i.e., oceanography, limnology, hydrology, and others). These challenges were synthesized from literature, authors’ experiences, and a broad survey of 174 data users and data providers across academia, government agencies, industry, and other sectors. Through this work, we identified seven main challenges: 1) metadata shortcomings, 2) variable data quality and reusability, 3) open data inaccessibility, 4) lack of standardization, 5) authorship and acknowledgement issues 6) lack of funding, and 7) unequal barriers around the globe. Our key recommendation is to improve resources to advance Open Data practices. This includes dedicated funds for capacity building, hiring and maintaining of skilled personnel, and robust digital infrastructures for preparation, storage, and long-term maintenance of Open Data. Further, to incentivize data sharing we reinforce the need for standardized best practices to handle data acknowledgement and citations for both data users and data providers. We also highlight and discuss regional disparities in resources and research practices within a global perspective.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

Human Factors in Streaming Data Analysis: Challenges and Opportunities for Information Visualization: Human Factors in Streaming Data Analysis

State-of-the-art visual analytics models and frameworks mostly assume a static snapshot of the data, while in many cases it is a stream with constant updates and changes. Exploration of streaming data poses unique challenges as machine-level computations and abstractions need to be synchronized with the visual representation of the data and the temporally evolving human insights. In the visual analytics literature, we lack a thorough characterization of streaming data and analysis of the challenges associated with task abstraction, visualization design, and adaptation of the role of human-in-the-loop for exploration of data streams. We aim to fill this gap by conducting a survey of the state-of-the-art in visual analytics of streaming data for systematically describing the contributions and shortcomings of current techniques and analyzing the research gaps that need to be addressed in the future. Our contributions are: i) problem characterization for identifying challenges that are unique to streaming data analysis tasks, ii) a survey and analysis of the state-of-the-art in streaming data visualization research with a focus on the visualization design space for dynamic data and the role of the human-in-the-loop, and iii) reflections on the design-trade-offs for streaming visual analytics techniques and their practical applicability in real-world application scenarios.

streaming data↗
DOE OSTI2023

2023 Data Science Challenge Notebooks

THE DATA SCIENCE INSTITUTE WILL BE INVITING STUDENTS FROM UC MERCED AND UC RIVERSIDE TO PARTICIPATE IN A DATA SCIENCE CHALLENGE AT LLNL. IN THE CHALLENGE, THE STUDENTS WILL EXPLORE A DATA-DRIVEN APPROACH TO RECONSTRUCTING ELECTROANATOMICAL MAPS OF THE HEART AT CLINICALLY RELEVANT RESOLUTIONS, COMBINING INPUT FROM THE STANDARD 12-LEAD ECG WITH ADVANCED MACHINE LEARNING TECHNIQUES. THE DATA THE STUDENTS WILL HAVE ACCESS TO HAS BEEN IM REVIEWED FOR EXTERNAL RELEASE. (LLNL-MI-835833) THESE NOTEBOOKS WILL PROVIDE THE STUDENTS WITH INSTRUCTIONS OF HOW TO LOAD THE DATA AND WORK WITH THE DATA, ALONG WITH TUTORIALS TO DO BASIC MACHINE LEARNING TASKS LIKE LOGISTIC REGRESSION AND IMAGE CLASSIFICATION.

Landajuela Larma, Mikel↗
DOE OSTI2017

Scaling up to address data science challenges

Statistics and Data Science provide a variety of perspectives and technical approaches for exploring and understanding Big Data. Partnerships between scientists from different fields such as statistics, machine learning, computer science, and applied mathematics can lead to innovative approaches for addressing problems involving increasingly large amounts of data in a rigorous and effective manner that takes advantage of advances in computing. Here, this article will explore various challenges in Data Science and will highlight statistical approaches that can facilitate analysis of large-scale data including sampling and data reduction methods, techniques for effective analysis and visualization of large-scale simulations, and algorithms and procedures for efficient processing.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2006

Using global aerosol models and satellite data for air quality studies: Challenges and data needs

Aerosol particles, also known as PM2.5 (particle diameter less than 2.5 pm) and PM10 (particle diameter less than 10 pm), are one of the key atmospheric components that determines air quality. Yet, air quality forecasts for PM are still in their infancy and remain a challenging task. It is difficult to simply relate PM levels to local meteorological conditions, and large uncertainties exist in regional air quality model emission inventories and initial and boundary conditions. Especially challenging are periods when a significant amount of aerosol comes from outside the regional modeling domain through long-range transport. In the past few years, NASA has launched several satellites with global aerosol measurement capabilities, providing large-scale chemical weather pictures. NASA has also supported development of global models which simulate atmospheric transport and transformation processes of important atmospheric gas and aerosol species. I will present the current modeling and satellite capabilities for PM2.5 studies, the possibilities and challenges in using satellite data for PM2.5 forecasts, and the needs of future remote sensing data for improving air quality monitoring and modeling.

Chin, Mian↗
DOE OSTI2021

Methods for a blind analysis of isobar data collected by the STAR collaboration

In 2018, the STAR collaboration collected data from $_{44}^{96}{\mathrm{Ru}}+_{44}^{96}{\mathrm{Ru}}$ and $_{40}^{96}{\mathrm{Zr}}+_{40}^{96}{\mathrm{Zr}}$ at $\sqrt{s_\text {NN}}=200$ GeV to search for the presence of the chiral magnetic effect in collisions of nuclei. The isobar collision species alternated frequently between $_{44}^{96}{\mathrm{Ru}}+_{44}^{96}{\mathrm{Ru}}$ and $_{40}^{96}{\mathrm{Zr}}+_{40}^{96}{\mathrm{Zr}}$ . In order to conduct blind analyses of studies related to the chiral magnetic effect in these isobar data, STAR developed a three-step blind analysis procedure. Analysts are initially provided a “reference sample” of data, comprised of a mix of events from the two species, the order of which respects time-dependent changes in run conditions. After tuning analysis codes and performing time-dependent quality assurance on the reference sample, analysts are provided a species-blind sample suitable for calculating efficiencies and corrections for individual $\approx 30$-min data-taking runs. For this sample, species-specific information is disguised, but individual output files contain data from a single isobar species. Only run-by-run corrections and code alteration subsequent to these corrections are allowed at this stage. Following these modifications, the “frozen” code is passed over the fully un-blind data, completing the blind analysis. As a check of the feasibility of the blind analysis procedure, analysts completed a “mock data challenge,” analyzing data from Au + Au collisions at $\sqrt{s_\text {NN}}=27$ GeV, collected in 2018. The Au + Au data were prepared in the same manner intended for the isobar blind data. Finally, the details of the blind analysis procedure and results from the mock data challenge are presented.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
NASA NTRSDate not supplied

Data Democratization: Challenges and Opportunities

Democratizing Earth data is one of the challenges many organizations around the world face in order to maximize the use of their Earth data for research, applications, education, and societal benefits. For example, at the NASA Goddard Earth Sciences (GES) Data and Information Services Center (DISC), over 1600 global and regional datasets in several NASA Earth science focus areas, including atmospheric composition, water and energy cycles, and climate variability, are archived and distributed to the public. Giovanni, the Geospatial Interactive Online Visualization and Analysis Infrastructure, was developed by GES DISC to facilitate data access and exploration, especially for novice users of Earth science. With Giovanni, users can analyze and visualize over 2000 Earth science variables (e.g., precipitation, aerosol, surface wind) without downloading data, software, the expert understanding of data formats and structures, and coding skills, lowering the barrier to data analysis/comparison by preprocessing and accessing to the data. Results of data analysis and visualization can be accessed in several popular formats (e.g., NetCDF, CSV). As a result of Giovanni's efforts, more than 3000 referral papers have been published in various fields. In spite of this, Giovanni is still difficult to use for some users. For instance, if one searches for "precipitation," it will return over 150 related variables. The question is, which one to use? Furthermore, variables from different data providers (e.g., satellites and models) are named differently with different units, further confusing users, especially those outside the communities. Data democratization is complex and multifaceted. Challenges include service and data discovery, user experiences, visualization, data quality, trustworthiness, and more. In this presentation, we will examine Giovanni as an example of challenges and opportunities in developing data democratization services.

data democratization↗
DOE OSTI2020

Data Federation Challenges in Remote Near-Real-Time Fusion Experiment Data Processing

Fusion energy experiments and simulations provide critical information needed to plan future fusion reactors. As next-generation devices like ITER move toward long-pulse experiments, analyses, including AI and ML, should be performed in a wide range of time and computing constraints, from near-real-time constraints, between-shot analysis, and to campaign-wide long-term analysis. However, the data volume, velocity, and variety make it extremely challenging for analyses using only local computational resources. Researchers need the ability to compose and execute workflows spanning edge resources to large-scale high-performance computing facilities.We present Delta, a system to address data analysis challenges, including AI/ML, in fusion science, by leveraging the ADIOS I/O library and middleware, to support executing science workflows over the wide area network for near-real-time streaming. We discuss the data federation challenges in performing remote workflows, focusing on on-going research work in (1) managing, reducing, and streaming data to minimize I/O and data movement overheads, (2) decompressing and reorganizing data for analysis, and (3) executing workflows for automated data analysis. We introduce examples for deep-learning based data analysis for the fusion domain and demonstrate how we use Delta to construct end-to-end workflows for a fusion device in Korea, connecting a remote DOE facility in the USA. The capability demonstrated by this project is the basis for improving the state of the art for near-real-time data federation amongst remote facilities.

Choi, Jong Youl↗
NASA NTRSDate not supplied

Open Source Application of Fusing Aerosol Products from GEO and LEO Satellites

Retrieving aerosol optical depths (AODs) from sun-synchronous polar orbiting (aka low earth orbit, LEO) satellites, such as MODISs, and VIIRSs, OMI, TROPOMI, etc, has become well-established as a tool for extracting information on particulate matter (PM) and related processes in the atmosphere. However, with recently launched geostationary satellites (GEO), such as GOES-16/17/18, and Himawari-8/9, and Meteosat Third Generation (MTG) they provide a much higher temporal resolution (order of 10 minutes), typically an image once or more per hour during daylight compared to LEO once per day. By combining these observations, we may be able to characterize the diurnal cycle of global AOD at the local, regional and global scale. While the science community is still exploring the new data from GEO observations, we have been thinking about how to properly combine/merge/fuse those data considering differences in their spatial and temporal resolutions. However, this poses a “Big Data” challenge. The big data challenge is not just about data storage, but also about data discoverability, and accessibility, and even more, about data migration/mirroring in the cloud-computing environment. This paper is merely showing some of the efforts and approaches we have attempted in fusing six satellites’ Level 2 aerosol data (three are from GEO (GOES-16/17 and Himawari-8), and the other three are from LEO (TERRA/MODIS, AQUA/MODIS, SNPP-VIIRS) from Dark Target (DT) aerosol retrieval algorithm. Having the on-demand capability of fusing remote sensing products onto the desired temporal and spatial domain enables researchers and application practitioners to better manipulate and work with satellite and sensor data. It is our hopeWe hope that by making such an open-source package, and the accompanying functionality, the scientific community will be granted easier access to aerosol data processing resources. The MEaSUREs Program (Making Earth System Data Records for Use in Research Environments) expands our understanding of the Earth's current system through atmospheric and surface measurements. In an effort to aid the scientific research component and improve open source methods, this project developed Python code for fusing six satellite Level 2 aerosol data (three are from geostationary satellites (GEO), and the other three are from low earth orbital satellites (LEO)) from Dark Target Aerosol Retrieval Algorithm.

Jennifer Wei↗
DOE OSTI2026

Revolutionizing thermal Management in Next-Generation AI data centers: Challenges and breakthrough innovations

Data centers (DCs) serve as critical infrastructure for powering the growth and evolution of AI. Next-generation AI DCs present unique challenges in thermal management driven by unprecedented computational demands. This paper provides a comprehensive summary of key stakeholder perspectives on technology gaps, infrastructure requirements, test bed needs, emerging opportunities, and preliminary solutions related to thermal management for AI DCs. It establishes six strategic pillars of thermal management for next generation AI DC: reliability, deployability, efficiency, resilience, measurability, and valorization. The discussion spans a range of critical topics, including advanced cooling technologies, thermal strategies for emerging modular and edge DCs, system-level optimization and control frameworks, infrastructure planning and grid integration designs, benchmarking approaches, and pathways for waste heat recovery and reuse. The proposed research, development, and demonstration efforts are aimed at accelerating the deployment of AI DCs while ensuring energy efficiency, reliability, safety, and regulatory compliance.

Wang, Pengtao [ORNL] (ORCID:0000000214713429)↗
DOE OSTI2015

Enhancing disease surveillance with novel data streams: challenges and opportunities

Novel data streams (NDS), such as web search data or social media updates, hold promise for enhancing the capabilities of public health surveillance. In this paper, we outline a conceptual framework for integrating NDS into current public health surveillance. Our approach focuses on two key questions: What are the opportunities for using NDS and what are the minimal tests of validity and utility that must be applied when using NDS? Identifying these opportunities will necessitate the involvement of public health authorities and an appreciation of the diversity of objectives and scales across agencies at different levels (local, state, national, international). We present the case that clearly articulating surveillance objectives and systematically evaluating NDS and comparing the performance of NDS to existing surveillance data and alternative NDS data is critical and has not sufficiently been addressed in many applications of NDS currently in the literature.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2018

De-starring Challenging Cygnus Data: A Comparison of Three Methods

This report addresses recent questions concerning the RadPro de-starring process. Ultimately, the issues that gave rise to the questions proved to be unrelated to the de-starring process. However, the nature of the questions exposed some misunderstanding on how to evaluate de-starring results. A fundamental of the de-starring process is that not every data point that is extracted as a star is actually a star. This work applies and compares three different methods to very challenging data to illustrate how to evaluate de-starring results. These three methods include (1) the procedure in RadPro, (2) a version of StarStats8, and (3) a method found in the literature by Abel Diaz (L.A. Cosmic), that was developed for astrophysical applications by Pieter G. van Dokkum. In L.A. Cosmic the radiation stars are generated by cosmic-ray hits and the data of interest are true physical stars and galaxies. Dokkum’s work is heavily referenced and appears in over 1,000 citations; a scholarly article on the topic describes it. The term de-starring will include removing cosmic-ray hits from such data. Another de-starring application, developed by Archambault et al. at the M. D. Anderson Cancer Center at the University of Texas in Houston is reviewed in this work but no comparison is made, as the algorithm is very similar to that of RadPro. This process was not named in their paper, and we will refer to it as MDACC. The focus of this report is on identifying stars. Computing replacement values, a different topic, is not covered in this report.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2024

Application of a Prize Mechanism to Address Data Utilization Challenges at Utilities

The electric industry sector is facing an “explosion” of data from a variety of sources. Electric sector stakeholders need to define how to capitalize on large datasets, both those they create and those from other sources (like data on weather, buildings, electric vehicles, etc.), to improve reliability and resilience and meet the changing system dynamics from renewable integration. For the electricity sector to fully utilize these vast new datasets, it must undergo a transformation in how it manages data quality, storage, and processing. The U.S. Department of Energy (DOE) Office of Electricity (OE) is committed to accelerating research, development, and demonstration of new technologies and tools within the electricity sector to advance reliability, resilience, and affordable operation of the power system. Through the prize mechanism, OE identified two widespread data-related challenges for utilities—load modeling and data analysis automation—and offered an opportunity for utilities and teams of software engineers to identify additional challenges faced by utilities. After completing one round of the American-Made Digitizing Utilities Prize, OE, the National Renewable Energy Laboratory (NREL) as the prize administrator, and Pacific Northwest National Laboratory (PNNL) as the domain experts have compiled the results and lessons learned to feed into the second round of the prize.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2014

Accelerating Data Acquisition, Reduction, and Analysis at the Spallation Neutron Source

ORNL operates the world's brightest neutron source, the Spallation Neutron Source (SNS). Funded by the US DOE Office of Basic Energy Science, this national user facility hosts hundreds of scientists from around the world, providing a platform to enable break-through research in materials science, sustainable energy, and basic science. While the SNS provides scientists with advanced experimental instruments, the deluge of data generated from these instruments represents both a big data challenge and a big data opportunity. For example, instruments at the SNS can now generate multiple millions of neutron events per second providing unprecedented experiment fidelity but leaving the user with a dataset that cannot be processed and analyzed in a timely fashion using legacy techniques. To address this big data challenge, ORNL has developed a near real-time streaming data reduction and analysis infrastructure. The Accelerating Data Acquisition, Reduction, and Analysis (ADARA) system provides a live streaming data infrastructure based on a high-performance publish subscribe system, in situ data reduction, visualization, and analysis tools, and integration with a high-performance computing and data storage infrastructure. ADARA allows users of the SNS instruments to analyze their experiment as it is run and make changes to the experiment in real-time and visualize the results of these changes. In this paper we describe ADARA, provide a high-level architectural overview of the system, and present a set of use-cases and real-world demonstrations of the technology.

Campbell, Stuart I↗
DOE OSTI2025

ECON-T and ECON-D: Endcap Concentrator ASICs for the CMS HGCAL

With over 6 million channels, the High Granularity Calorimeter (HGCAL) for the CMS HL-LHC Upgrade presents a unique data challenge. The ECON ASICs provide critical on-detector data reduction for the 40 MHz trigger path (ECON-T) and 750 kHz data acquisition path (ECON-D) of the HGCAL. The ASICs, fabricated in 65 nm CMOS, are rad-tolerant (600 Mrad) with low power consumption (<2.5 mW/channel). This presentation is the first comprehensive description of the ECON designs, first functionality and radiation tests for the ECON-T ASIC, and first results from the full production of 75k ECON-D and ECON-T ASICs.

Bergamin, G. [CERN] (ORCID:0000000285758704)↗