Search NASA⌕ Search

SEARCH · Search NASA

Results for “data challenges”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 559 records · Page 31
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Bringing heterogeneity to the CMS software framework

The advent of computing resources with co-processors, for example Graphics Processing Units (GPU) or Field-Programmable Gate Arrays (FPGA), for use cases like the CMS High-Level Trigger (HLT) or data processing at leadership-class supercomputers imposes challenges for the current data processing frameworks. These challenges include developing a model for algorithms to offload their computations on the co-processors as well as keeping the traditional CPU busy doing other work. The CMS data processing framework, CMSSW, implements multithreading using the Intel Threading Building Blocks (TBB) library, that utilizes tasks as concurrent units of work. In this paper we will discuss a generic mechanism to interact effectively with non-CPU resources that has been implemented in CMSSW. In addition, configuring such a heterogeneous system is challenging. In CMSSW an application is configured with a configuration file written in the Python language. The algorithm types are part of the configuration. The challenge therefore is to unify the CPU and co-processor settings while allowing their implementations to be separate. We will explain how we solved these challenges while minimizing the necessary changes to the CMSSW framework. We will also discuss on a concrete example how algorithms would offload work to NVIDIA GPUs using directly the CUDA API.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRS1992

Ocean climate data for user community in West and Central Africa: Needs, opportunities, and challenges

The urgent need to improve data delivery systems needed by scientists studying ocean role in climate and climate characteristics has been manifested in recent years because of the unprecedented climatic events experienced in many parts of the world. Indeed, there has been a striking and growing realization by governments and the general public indicating that national economies and human welfare depend on climate and its variability. In West and Central Africa, for instance climatic events, which have resulted in floods and droughts, have caused a lot of concern to both governments and people of the region. In particular, the droughts have been so widespread that greater awareness and concern have become generated for the need to find solutions to the problems created by the consequences of the climatic events. Particularly in the southern border regions of the Sahara Desert as well as in the Sahel region, the drought episodes considerably reduced food production and led to series of socioeconomic problems, not only in the areas affected by the droughts, but also in the other parts of West Africa. The various climatic variabilities which have caused the climatic events are no doubt related to the ocean-atmosphere interactions. Unfortunately, not much has been done on the understanding of these interactions, particularly as they affect developing countries. Indeed, not much has been done to develop programs which will reflect the general concerns and needs for researching into the ocean-atmosphere systems and their implications on man-environmental systems in many developing countries. This is for example, true of West and Central Africa, where compared with the middle latitude countries, much less is known about the characteristics of the ocean-atmosphere systems and their significance on man-environmental systems of the area.

Ojo, S. O.↗
NASA NTRS2017

VIIP 2017 Clinical Update

INTRODUCTION: NASA's Space Medicine community knowledge regarding the "Vision Impairment Intracranial Pressure", or VIIP.has been evolving over time.. Various measures of occupational health related to this condition had to be determined and then plans/processes put into place. The most robust of these processes were inititated in 2010. This presentation will provide a clinic update of the astronaut occupational health data related to VIIP. METHODS: NASA and its international partners require its astronauts to undergo routine health measures deemed important to monitoring VIIP. The concern is that the spaceflight environment aboard ISS could cause some astronauts to have physiologic changes detrimental to either ongoing mission operations or long-term health related to the ocular system and possibly the CNS. Specific medical tests include but are not limited to brain/orbit MRI (NASA unique protocol), OCT, fundoscopy and ocular ultrasound. Measures are taken prior to spaceflight, in-flight and post-flight. Measures to be reported include incidence of disc edema, globe flattening, choroidal folds, ONSD and change in refractive error. RESULTS: 73 ISS astronauts have been evaluated at least partially for VIIP related measures. Of these individuals, approximately 1 in 7 have experienced disc edema. The prevalence of the other findings is more complicated as the medical testing has changed over time. Overall, 26 separate individuals have experienced at least one of the findings NASA has associated with VIIP Another confounding factor is most of the astronauts have prior spaceflight experience at the time of the "pre-flight" testing. DISCUSSION: In 2010 NASA and its US operating segment (USOS) partners (CSA, ESA and JAXA) began routine occupational monitoring and data collection for most VIIP related changes. Interpretation of that data is extremely challenging for several reasons. For example, the determination of disc edema is the most complete finding as we have had highly qualified optometrists routinely and competently performing post-flight funduscopic exams for the entirety of the ISS program. Yet in 2013 NASA added OCT to our in-flight suite of eye exams. Shortly after routine screening with the OCT, a new term appeared within VIIP vernacular - "subclinical disc edema". OCT has much greater ability to measure change within the retina and provides significantly more data to analyze, understand and communicate out. Communicating VIIP data clearly adds even more challenge. Historically we've reported data per eye and not necessarily per person. This has led to difficulty in understanding how many individuals have experienced "VIIP" within the aerospace medicine community. The presenter will attempt to provide clear and concise communication of VIIP findings.

Tarver, William↗
NASA NTRS1992

Daily quality assurance software for a satellite radiometer system

Six Differential Microwave Radiometers (DMR) on COBE (Cosmic Background Explorer) measure the large-angular-scale isotropy of the cosmic microwave background (CMB) at 31.5, 53, and 90 GHz. Quality assurance software analyzes the daily telemetry from the spacecraft to ensure that the instrument is operating correctly and that the data are not corrupted. Quality assurance for DMR poses challenging requirements. The data are differential, so a single bad point can affect a large region of the sky, yet the CMB isotropy requires lengthy integration times (greater than 1 year) to limit potential CMB anisotropies. Celestial sources (with the exception of the moon) are not, in general, visible in the raw differential data. A 'quicklook' software system was developed that, in addition to basic plotting and limit-checking, implements a collection of data tests as well as long-term trending. Some of the key capabilities include the following: (1) stability analysis showing how well the data RMS averages down with increased data; (2) a Fourier analysis and autocorrelation routine to plot the power spectrum and confirm the presence of the 3 mK 'cosmic' dipole signal; (3) binning of the data against basic spacecraft quantities such as orbit angle; (4) long-term trending; and (5) dipole fits to confirm the spacecraft attitude azimuth angle.

Keegstra, P. B.↗
NASA NTRS1993

Analyzing Martian winds and tracer concentrations using Mars Observer data

During the courses of a day, the Mars Observer spacecraft will acquire globally distributed profiles of the martian atmosphere. It is highly desirable that this data be assembled into synoptic weather maps (complete specifications of the atmospheric pressure, temperature, and winds at a given time), which can in turn be used as starting points in the study of many meteorological phenomena. Unfortunately, the special nature of the Mars Observer data presents several challenges above and beyond the usual difficult problem of data initialization. Mars Observer atmospheric data will consist almost exclusively of asynoptic vertical profiles of temperatures (or radiances) and pressures, whereas winds are generally in balance with horizontal gradients of these quantities (which will not be observed). It will therefore be necessary to resort to dynamical models to analyze the wind fields. As a rule, data assimilation into atmospheric models can result in the generation of spurious gravity waves, so special steps must be taken to suppress these. In addition, the asynoptic nature of the data will require a four-dimensional (space and time) data assimilation scheme. The problem is to find a full set of meteorological fields (winds and temperatures) such that, when marched forward in time in the model, they achieve a best fit (in the weighted least-squares sense) to the data. The proposed solution is to develop a model especially for the Mars Observer data assimilation problem. Gravity waves are filtered from the model by eliminating all divergence terms from the prognostic divergence equation. This leaves a diagnostic gradient wind relation between the rotational wind and the temperature field. The divergent wind is diagnosed as the wind required to maintain the gradient wind balance in the presence of the diabatic heating. The primitive equations of atmospheric dynamics (with three principal dependent variables) are thus reduced to a simpler system with a single prognostic equation for temperature - the variable that will be best observed. (This balance system was apparently first derived by Charney as a first-order Rossby number expansion of the equations of motion). Experience with a full primitive equation model of the martian atmosphere indicates that a further simplification is possible: at least for short-term integrations, the model can be linearized about the zonally symmetric basic state.

Houben, Howard C.↗
DOE OSTI2021

Revisiting Huffman Coding: Toward Extreme Performance on Modern GPU Architectures

Today's high-performance computing (HPC) applications are producing vast volumes of data, which are challenging to store and transfer efficiently during the execution, such that data compression is becoming a critical technique to mitigate the storage burden and data movement cost. Huffman coding is arguably the most efficient Entropy coding algorithm in information theory, such that it could be found as a fundamental step in many modern compression algorithms such as DEFLATE. On the other hand, today's HPC applications are more and more relying on the accelerators such as GPU on supercomputers, while Huffman encoding suffers from low throughput on GPUs, resulting in a significant bottleneck in the entire data processing. In this paper, we propose and implement an efficient Huffman encoding approach based on modern GPU architectures, which addresses two key challenges: (1) how to parallelize the entire Huffman encoding algorithm, including codebook construction, and (2) how to fully utilize the high memory-bandwidth feature of modern GPU architectures. The detailed contribution is fourfold. (1) We develop an efficient parallel codebook construction on GPUs that scales effectively with the number of input symbols. (2) We propose a novel reduction based encoding scheme that can efficiently merge the codewords on GPUs. (3) We optimize the overall GPU performance by leveraging the state-of-the-art CUDA APIs such as Cooperative Groups. (4) We evaluate our Huffman encoder thoroughly using six real-world application datasets on two advanced GPUs and compare with our implemented multi-threaded Huffman encoder. Experiments show that our solution can improve the encoding throughput by up to 5.0x and 6.8x on NVIDIA RTX 5000 and V100, respectively, over the state-of-the-art GPU Huffman encoder, and by up to 3.3x over the multi-thread encoder on two 28-core Xeon Platinum 8280 CPUs.

Tian, Jiannan↗
DOE OSTI2021

Revisiting Huffman Coding: Toward Extreme Performance on Modern GPU Architectures

Today's high-performance computing (HPC) applications are producing vast volumes of data, which are challenging to store and transfer efficiently during the execution, such that data compression is becoming a critical technique to mitigate the storage burden and data movement cost. Huffman coding is arguably the most efficient Entropy coding algorithm in information theory, such that it could be found as a fundamental step in many modern compression algorithms such as DEFLATE. On the other hand, today's HPC applications are more and more relying on the accelerators such as GPU on supercomputers, while Huffman encoding suffers from low throughput on GPUs, resulting in a significant bottleneck in the entire data processing. In this paper, we propose and implement an efficient Huffman encoding approach based on modern GPU architectures, which addresses two key challenges: (1) how to parallelize the entire Huffman encoding algorithm, including codebook construction, and (2) how to fully utilize the high memory-bandwidth feature of modern GPU architectures. The detailed contribution is fourfold. (1) We develop an efficient parallel codebook construction on GPUs that scales effectively with the number of input symbols. (2) We propose a novel reduction based encoding scheme that can efficiently merge the codewords on GPUs. (3) We optimize the overall GPU performance by leveraging the state-of-the-art CUDA APIs such as Cooperative Groups. (4) We evaluate our Huffman encoder thoroughly using six real-world application datasets on two advanced GPUs and compare with our implemented multithreaded Huffman encoder. Experiments show that our solution can improve the encoding throughput by up to 5.0× and 6.8× on NVIDIA RTX 5000 and V100, respectively, over the state-of-the-art GPU Huffman encoder, and by up to 3.3× over the multithread encoder on two 28-core Xeon Platinum 8280 CPUs.

Tian, Jiannan↗
DOE OSTI2023

Conflation of Geospatial POI Data and Ground-level Imagery

The code solves the problem of conflating POI (Points of Interest) geospatial data and geo-tagged ground-level imagery data. The main challenges with fusion or conflation of these two types of data has been that these two data entities are represented not only in several ways in current state-of-the-art, but also there is mismatch of representation format of these two data types. This source code/software brings POI datapoints and ground-level imagery datapoints into same representation format, and then generates valuable Knowledge Graph combining POI and images data (so that it can be used for multitude of applications).

De, Debraj↗
NASA NTRS1996

BIOME: A browser-aware search and order system

The Oak Ridge National Laboratory (ORNL) Distributed Active Archive Center (DAAC), which is associated with NASA's Earth Observing System Data and Information System (EOSDIS), provides access to a large number of tabular and imagery datasets used in ecological and environmental research. Because of its large and diverse data holdings, the challenge for the ORNL DAAC is to help users find data of interest from the hundreds of thousands of files available at the DAAC without overwhelming them. Therefore, the ORNL DAAC developed the Biogeochemical Information Ordering Management Environment (BIOME), a search and order system for the World Wide Web (WWW). The WWW provides a new vehicle that allows a wide range of users access to the data. This paper describes the specialized attributes incorporated into BIOME that allow researchers easy access to an otherwise bewildering array of data products.

Grubb, Jon W.↗
NASA NTRS1997

BIOME: A scientific data archive search-and-order system using browser-aware, dynamic pages

The Oak Ridge National Laboratory's (ORNL) Distributed Active Archive Center (DAAC) is a data archive and distribution center for the National Air and Space Administration's (NASA) Earth Observing System Data and Information System (EOSDIS). Both the Earth Observing System (EOS) and EOSDIS are components of NASA's contribution to the US Global Change Research Program through its Mission to Planet Earth Program. The ORNL DAAC provides access to data used in ecological and environmental research such as global change, global warming, and terrestrial ecology. Because of its large and diverse data holdings, the challenge for the ORNL DAAC is to help users find data of interest from the hundreds of thousands of files available at the DAAC without overwhelming them. Therefore, the ORNL DAAC has developed the Biogeochemical Information Ordering Management Environment (BIOME), a customized search and order system for the World Wide Web (WWW). BIOME is a public system located at http://www-eosdis. ornl.gov/BIOME/biome.html.

CLIMATIC CHANGE↗
NASA NTRS2017

Collaborative Data Publication Utilizing the Open Data Repository's Data Publisher

For small communities in multidisciplinary fields such as astrobiology, publishing and sharing data can be challenging. While large, homogenous fields often have repositories and existing data standards, small groups of independent researchers have few options for publishing data that can be utilized within their community. In conjunction with teams at NASA Ames and the University of Arizona, a number of pilot studies are being conducted to assess the needs of these research groups and to guide the software development so that it allows them to publish and share their data collaboratively.

Human-readable interfaces↗
NASA NTRS2017

Complexities in Subsetting Level 2 Data

Satellite Level 2 data presents unique challenges for tools and services. From nonlinear spatial geometry to inhomogeneous file data structure to inconsistent temporal variables to complex data variable dimensionality to multiple file formats, there are many difficulties in creating general tools for Level 2 data support. At NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), we are implementing a general Level 2 Subsetting service for Level 2 data to a user-specified spatio-temporal region of interest (ROI). In this presentation, we will unravel some of the challenges faced in creating this service and the strategies we used to surmount them.

Data acces↗
DOE OSTI2025

High-Fidelity, Large-Scale, Realistic Dataset Development

The final report summarizes the work performed for supporting the ARPA-E Grid Optimization Competition (Challenge 2 and Challenge 3) within the stated period. Challenge 2 For the challenge period, the main responsibility of the team is to investigate, gen- erate, and deliver parts of the data sets for the competition, based on the competition model for Challenge 2, existing data sets from Challenge 1, and data source supplied by other data set teams. Challenge 3 For the challenge period, the main responsibility of the team is to propose, create, deliver, and maintain the data format during the competition period. The data format will specify how the benchmark data will be represented and communicated to competitors. It will also specify how competitors should report back the solutions. The data format will be closely aligned with the problem formulation (maintained by the formulation team) and the solution validation process (maintained by the validation team). Our team is also responsible in investigating, generating, and delivering parts of the data sets for the competition. The data sets will be created based on the competition model for Challenge 3, existing data sets from Challenge 1 and Challenge 2, and data source supplied by other data set teams.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
NASA NTRS2014

Challenges in Evaluating Relationships Between Quantitative Data (Carbon Dioxide) and Qualitative Data (Self-Reported Visual Changes)

Understanding the nuances in clinical data is critical in developing a successful data analysis plan. Carbon dioxide (CO2) data are collected on board the International Space Station (ISS) in a continuous stream. Clinical data on ISS are primarily collected via conversations between individual crewmembers and NASA Flight Surgeons during weekly Private Medical Conferences (PMC). Law, et.al, 20141 demonstrated a statistically significant association between weekly average CO2 levels on ISS and self-reported headaches over the reporting period from March 14, 2001 to May 31, 2012. The purpose of this analysis is to describe the evaluation of a possible association between visual changes and CO2 levels on ISS and to discuss challenges in developing an appropriate analysis plan. METHODS & PRELIMINARY RESULTS: A first analysis was conducted following the same study design as the published work on CO2 and self-reported headaches1; substituting self-reported changes in visual acuity in place of self-reported headaches. The analysis demonstrated no statistically significant association between visual impairment characterized by vision symptoms self-reported during PMCs and ISS average CO2 levels over ISS missions. Closer review of the PMC records showed that vision outcomes are not well-documented in terms of clinical severity, timing of onset, or timing of resolution, perhaps due to the incipient nature of vision changes. Vision has been monitored in ISS crewmembers, pre- and post-flight, using standard optometry evaluations. In-flight visual assessments were limited early in the ISS program, primarily consisting of self-perceived changes reported by crewmembers. Recently, on-orbit capabilities have greatly improved. Vision data ranges from self-reported post-flight changes in visual acuity, pre- to postflight changes identified during fundoscopic examination, and in-flight progression measured by advanced on-orbit clinical imaging capabilities at predetermined testing intervals. In contrast, CO2 data are recorded in a continuous stream over time; however, for the initial analysis this data was categorized into weekly averages.

Mendez, C. M.↗
DOE OSTI2024

Simulating image coaddition with the Nancy Grace Roman Space Telescope – I. Simulation methodology and general results

The upcoming Nancy Grace Roman Space Telescope will carry out a wide-area survey in the near-infrared. A key science objective is the measurement of cosmic structure via weak gravitational lensing. Roman data will be undersampled, which introduces new challenges in the measurement of source galaxy shapes; a potential solution is to use linear algebra-based coaddition techniques such as imcom that combine multiple undersampled images to produce a single oversampled output mosaic with a desired ‘target’ point spread function (PSF). We present here an initial application of imcom to 0.64 square degrees of simulated Roman data, based on the Roman branch of the Legacy Survey of Space and Time (LSST) Dark Energy Science Collaboration (DESC) Data Challenge 2 (DC2) simulation. We show that imcom runs successfully on simulated data that includes features such as plate scale distortions, chip gaps, detector defects, and cosmic ray masks. We simultaneously propagate grids of injected sources and simulated noise fields as well as the full simulation. We quantify the residual deviations of the PSF from the target (the ‘leakage’), as well as noise properties of the output images; we discuss how the overall tiling pattern as well as Moiré patterns appear in the final leakage and noise maps. We include appendices on interpolation algorithms and the interaction of undersampling with image processing operations that may be of broader applicability. The companion paper (‘Paper II’) explores the implications for weak lensing analyses.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

The Challenges of Using Real-Time Detection Systems: From Data Gathering to Actionable Information [Slides]

This session will discuss challenges and real life examples associated with Real-Time Detection Systems (RTDS) used in exposure assessment strategies. This first presentation will provide an overview of RTDS as used as part of an overall exposure assessment strategy and to document regulatory compliance. Managing OH consequences has been problematic when it comes to actionable information gleaned from sensor data. The judgement of regulatory entities, can be at odds with IH practitioners implementing an exposure strategy that creates exposure profiles and judges workplace exposures.

61 RADIATION PROTECTION AND DOSIMETRY↗
DOE OSTI2025

From Machine Learning to Machine Reasoning: A Model-based Approach to Analyze Equipment Reliability Data

In current nuclear power plants (NPPs) a large amount of condition-based data which can be used to assess and monitor component health and performance. Assessing component health from such data can be performed with a large variety of methods. While the analysis of numeric data can be performed with several methods, the extraction of information from textual data remains a challenge. Currently employed natural language processing (NLP) methods do not really provide quantitative information that might be contained in IRs. In addition, the integration of numeric and textual data to identify possible causal relationships between data elements is still an unresolved challenge. This paper presents an approach to extract information from textual (e.g., incident or maintenance reports) and numeric data that relies on model based system engineer (MBSE) models. MBSE are diagrams designed to represent system and component dependencies (from both a form and functional point of view). In our approach, MBSE models emulate system engineer knowledge about component/system architecture. NLP methods are employed to perform syntactic and semantic analyses. Syntactic analysis analyzes the grammatical structure of a sentence while semantic analysis is designed to analyze the logic structure of a sentence. An innovative element of our approach is that semantic analysis uses MBSE models to identify links between textual elements. Similarly, numeric data is directly linked to elements of the MBSE models in order to map which functions are being monitored.

97 - MATHEMATICS AND COMPUTING↗