Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Challenge”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26

Maintaining Trust in Reduction: Preserving the Accuracy of Quantities of Interest for Lossy Compression

As the growth of data sizes continues to outpace computational resources, there is a pressing need for data reduction techniques that can significantly reduce the amount of data and quantify the error incurred in compression. Compressing scientific data presents many challenges for reduction techniques since it is often on non-uniform or unstructured meshes, is from a high-dimensional space, and has many Quantities of Interests (QoIs) that need to be preserved. To illustrate these challenges, we focus on data from a large scale fusion code, XGC. XGC uses a Particle-In-Cell (PIC) technique which generates hundreds of PetaBytes (PBs) of data a day, from thousands of timesteps. XGC uses an unstructured mesh, and needs to compute many QoIs from the raw data, f.One critical aspect of the reduction is that we need to ensure that QoIs derived from the data (density, temperature, flux surface averaged momentums, etc.) maintain a relative high accuracy. We show that by compressing XGC data on the high-dimensional, nonuniform grid on which the data is defined, and adaptively quantizing the decomposed coefficients based on the characteristics of the QoIs, the compression ratios at various error tolerances obtained using a multilevel compressor (MGARD) increases more than ten times. We then present how to mathematically guarantee that the accuracy of the QoIs computed from the reduced f is preserved during the compression. We show that the error in the XGC density can be kept under a user-specified tolerance over 1000 timesteps of simulation using the mathematical QoI error control theory of MGARD, whereas traditional error control on the data to be reduced does not guarantee the accuracy of the QoIs.

Gong, Qian↗

Fourier-informed knot placement schemes for B-spline approximation

Fitting B-splines to scientific data is especially challenging when the given data contain noise, jumps, or corners. Here, we describe how periodic data sets with these features can be efficiently approximated with B-splines by analyzing the Fourier spectrum of the data. Our method uses a collection of spectral filters to compute high-order derivatives, smoothed versions of noisy data, and the locations of jump discontinuities. Further, these quantities are then combined to choose knots that capture the qualitative features of the data, leading to accurate B-spline approximations with few knots. The method we introduce is direct and does not require any intermediate B-spline fitting before choosing the final knot distribution. Aside from fast Fourier transforms to transfer to and from Fourier space, the method runs in linear time with very little communication. We assess performance on several test cases in one and two dimensions, including data sets with jump discontinuities and noise. These tests show the method fits discontinuous data without spurious oscillations and remains effective in the presence of noise.

97 MATHEMATICS AND COMPUTING↗

Multi-omic data integration enables discovery of hidden biological regularities

Rapid growth in size and complexity of biological data sets has led to the ‘Big Data to Knowledge’ challenge. We develop advanced data integration methods for multi-level analysis of genomic, transcriptomic, ribosomal profiling, proteomic and fluxomic data. First, we show that pairwise integration of primary omics data reveals regularities that tie cellular processes together in Escherichia coli: the number of protein molecules made per mRNA transcript and the number of ribosomes required per translated protein molecule. Second, we show that genome-scale models, based on genomic and bibliomic data, enable quantitative synchronization of disparate data types. Integrating omics data with models enabled the discovery of two novel regularities: condition invariant in vivo turnover rates of enzymes and the correlation of protein structural motifs and translational pausing. These regularities can be formally represented in a computable format allowing for coherent interpretation and prediction of fitness and selection that underlies cellular physiology.

59 BASIC BIOLOGICAL SCIENCES↗

First test results of the HGCAL concentrator ASICs: ECON-T and ECON-D

With over 6 million channels, the High Granularity Calorimeter for the CMS HL-LHC upgrade presents a unique data transmission challenge. The ECON ASICs provide a critical stage of on-detector data compression and selection for the trigger path (ECON-T) and data acquisition path (ECON-D) of the HGCAL. The ASICs, fabricated in 65 nm CMOS, are radiation tolerant up to 200 Mrad and require low power consumption: < 2.5 mW/sensor-channel per chip. Here, we report on the first functionality and radiation tests for the ECON-D-P1 full-functionality prototype. We present a comparison of single event effect (SEE) cross sections measured for different methods of triple modular redundancy using test results from the ECON-T-P1 full-functionality prototype.

47 OTHER INSTRUMENTATION↗

HP-MDR: High-performance and Portable Data Refactoring and Progressive Retrieval with Advanced GPUs

Scientific applications produce vast amounts of data, posing grand challenges in the underlying data management and analytic tasks. Progressive compression is a promising way to address this problem, as it allows for on-demand data retrieval with significantly reduced data movement cost. However, most existing progressive methods are designed for CPUs, leaving a gap for them to unleash the power of today’s heterogeneous computing systems with GPUs.In this work, we propose HP-MDR, a high-performance and portable data refactoring and progressive retrieval framework for GPUs. Our contributions are four-fold: (1) We carefully optimize the bitplane encoding and lossless encoding, two key stages in progressive methods, to achieve high performance on GPUs; (2) We propose pipeline optimization and incorporate it with data refactoring and progressive retrieval workflows to further enhance the performance for large data process; (3) We leverage our framework to enable high-performance data retrieval with guaranteed error control for common Quantities of Interest; (4) We evaluate HP-MDR and compare it with state of the arts using five real-world datasets. Experimental results demonstrate that HP-MDR delivers an average 13.68 × and 6.31 × throughput in data refactoring and progressive retrieval tasks, respectively. It also leads to 11.22 × throughput for recomposing required data representations under Quantity-of-Interest error control and 6.04 × performance for the corresponding end-to-end data retrieval, when compared with state-of-the-art solutions.

Li, Yanliang [University of Oregon]↗

Pilot-Scale CFP Commissioning: Creative Problem Solving and Lessons Learned

We have commissioned a pilot-scale, ex-situ catalytic fast pyrolysis (CFP) reactor system in the Thermal and Catalytic Process Development Unit (TCPDU) at the National Renewable Energy Laboratory. The FCC-style CFP system, including a riser and continuous regeneration, was added to the existing half-ton-per-day biomass fast pyrolysis unit and was designed to run a zeolite-type catalyst. The system is unique due to its scale and catalyst flow control scheme. As the design of this system was informed by early data from a bench-scale reactor, several technical scale-up challenges had to be overcome. Additionally, physical constraints and the complexities of biomass vapors required and inspired creative solutions. This talk will discuss the specific challenges we found and addressed, including an oxygen deficiency in the regenerator and plugging in the exit streams. We explored several options for increasing oxygen availability and selected a cyclone-catalyst return design on the exit line so we could increase air flow through the regenerator. We found that hot gas filtration is essential in any exit streams containing water vapor and trace amounts of catalyst; any combination of catalyst and condensate quickly makes mud that plugs equipment and severely limits run time. Duplicate filters are also required to extend run time. Most of the equipment and solutions were designed and installed by our team of engineers and technicians, working closely together in an iterative process to design, install, operate, then redesign. Collaborations with other national laboratories with computational fluid dynamics expertise helped inform designs and identify creative solutions to these scale-up challenges. Ultimately, data from the reactor system will be used to validate a comprehensive kinetic model of CFP that can be used to better design and inform new systems at various scales.

09 BIOMASS FUELS↗

Development and Preliminary Analysis of a U.S. Geothermal Heat Pump Installation Database

This paper seeks to addresses the significant gap in the literature regarding the installation and adoption of geothermal heat pump (GHP) systems in the United States. While the "2021 U.S. Geothermal Power Production and District Heating Market Report" published by the National Renewable Energy Laboratory (NREL) focused on direct-use geothermal district heating systems, it did not include an analysis of GHP installations (Robins et al. 2021). To bridge this gap, NREL has compiled a novel database currently containing 70,470 records of GHP installations, primarily sourced from state well permits and small-scale studies. Our methodology emphasizes the collection, cleaning, and standardization of data, addressing challenges such as inconsistent reporting formats and privacy concerns. Despite limitations in data on capacity, costs, and performance, our preliminary geospatial analysis reveals insights into the distribution of GHP systems across urban and rural areas and climate zones. The paper highlights the importance of publicly accessible data for advancing GHP technology adoption with a discussion of existing data sources and their limitations, advocating for improved collaboration between NREL and industry stakeholders.

data collection↗

An open-access simulated earthquake ground-motion database for an M7 Hayward Fault earthquake in the San Francisco Bay Region

Comprehensive understanding of earthquake ground motions, particularly in the near-fault region of large-magnitude events, is limited by gaps in strong-motion data. This challenge is prominent in areas with high seismic hazard but infrequent large earthquakes where data is sparse and difficult to interpret. These data limitations lead to uncertainties in the development of site-specific ground motions, which are crucial for engineering risk assessments. To address these challenges, physics-based regional-scale ground-motion simulations have been developed. With the emergence of exaflop-scale computing ecosystems, it is now possible to simulate regional earthquake processes at unprecedented fidelity and generate the large number of fault rupture realizations necessary to characterize both intra- and inter-event ground-motion variability. This article introduces a new database of simulated earthquake ground motions, created for applications in earthquake engineering, earthquake planning, and emergency response. The inaugural version of the database features simulated ground motions for a magnitude 7 Hayward Fault earthquake in the San Francisco Bay Region (SFBR), using the EarthQuake SIMulation (EQSIM) simulation framework and the Graves–Pitarka kinematic rupture model. The aim is to provide high-fidelity, spatially dense, three-component motions generated on the Department of Energy’s (DOE) newest generation of graphics processing unit (GPU)-accelerated supercomputers. These motions are being made openly available to the engineering, scientific, and disaster planning communities. In addition, this work develops protocols for the efficient dissemination of these large data sets and emphasizes community engagement to build confidence in their application. This article discusses the methodology behind the data, underlying software verification and validation, scalable data management, and a user interface for data access. The goal is to facilitate widespread use and elicit expert feedback to maximize the utility and exploitation of simulated motions. While the initial focus is on the San Francisco Region, simulations for additional regions will be added as the DOE program progresses.

Simulated ground-motion database↗

Data Science and Urban Air Mobility: Challenges and Opportunities

Aviation is broadly a combination of aircraft, airspace and airports. The data science life cycle comprises of five steps - capture, maintain, process, analyze and communicate. The presentation introduces the legacy of conventional aviation research in the context of the data science life cycle to motivate the challenges with Urban Air Mobility, a field that is quite nascent. A summary of recent research will be presented to highlight the innovative ways to address the challenges. Examples provided will include the generation of synthetic data, encounter models from simulations, and leveraging novel and diverse data sets from traditional transportation and non-aviation sources, to analyze problems of operation in urban airspace. Finally, opportunities will be identified for further exploration, niche development and filling the gaps in the field of data science for UAM.

Urban Air Mobility↗

Data Science and Urban Air Mobility: Challenges and Opportunities

Aviation is broadly a combination of aircraft, airspace and airports. The data science life cycle comprises of five steps - capture, maintain, process, analyze and communicate. The presentation introduces the legacy of conventional aviation research in the context of the data science life cycle to motivate the challenges with Urban Air Mobility, a field that is quite nascent. A summary of recent research will be presented to highlight the innovative ways to address the challenges. Examples provided will include the generation of synthetic data, encounter models from simulations, and leveraging novel and diverse data sets from traditional transportation and non-aviation sources, to analyze problems of operation in urban airspace. Finally, opportunities will be identified for further exploration, niche development and filling the gaps in the field of data science for UAM.

Urban Air Mobility↗

Automatic building information model query generation

Energy efficient building design and construction calls for extensive collaboration between different subfields of the Architecture, Engineering and Construction (AEC) community. Performing building design and construction engineering raises challenges on data integration and software interoperability. Using Building Information Modeling (BIM) data hub to host and integrate building models is a promising solution to address those challenges, which can ease building design information management. However, the partial model query mechanism of current BIM data hub collaboration model has several limitations, which prevents designers and engineers to take advantage of BIM. To address this problem, we propose a general and effective approach to generate query code based on a Model View Definition (MVD). This approach is demonstrated through a software prototype called QueryGenerator. In conclusion, by demonstrating a case study using multi-zone air flow analysis, we show how our approach and tool can help domain experts to use BIM to drive building design with less labour and lower overhead cost.

97 MATHEMATICS AND COMPUTING↗

Data Automata in Scala

The field of runtime verification has during the last decade seen a multitude of systems for monitoring event sequences (traces) emitted by a running system. The objective is to ensure correctness of a system by checking its execution traces against formal specifications representing requirements. A special challenge is data parameterized events, where monitors have to keep track of the combination of control states as well as data constraints, relating events and the data they carry across time points. This poses a challenge wrt. efficiency of monitors, as well as expressiveness of logics. Data automata is a form of automata where states are parameterized with data, supporting monitoring of data parameterized events. We describe the full details of a very simple API in the Scala programming language, an internal DSL (Domain-Specific Language), implementing data automata. The small implementation suggests a design pattern. Data automata allow transition conditions to refer to other states than the source state, and allow target states of transitions to be inlined, offering a temporal logic flavored notation. An embedding of a logic in a high-level language like Scala in addition allows monitors to be programmed using all of Scala's language constructs, offering the full flexibility of a programming language. The framework is demonstrated on an XML processing scenario previously addressed in related work.

runtime verification↗

Big Data Platforms as a Service: Challenges and Approach

Infrastructure-as-a-Service has revolutionized the manner in which users commission computing infrastructure. Coupled with Big Data platforms (Hadoop, Cassandra), IaaS has democratized the ability to store and process massive datasets. For users that need to customize or create new Big Data stacks, however, readily available solutions do not yet exist. Users must first acquire the necessary cloud computing infrastructure, and manually install the prerequisite software. For complex distributed services this can be a daunting challenge. To address this issue, we argue that distributed services should be viewed as a single application consisting of virtual machines. Users should no longer be concerned about individual machines or their internal organization. To illustrate this concept, we introduce Cloud-Get, a distributed package manager that enables the simple installation of distributed services in a cloud computing environment. Cloud-Get enables users to instantiate and modify distributed services, including Big Data services, using simple commands. Cloud-Get also simplifies creating new distributed services via standardized package definitions.

99 GENERAL AND MISCELLANEOUS↗

ALACRITY: Analytics-Driven Lossless Data Compression for Rapid In-Situ Indexing, Storing, and Querying. In: Transactions on Large-Scale Data- and Knowledge-Centered Systems X. Lecture Notes in Computer Science, vol 8220

High-performance computing architectures face nontrivial data processing challenges, as computational and I/O components further diverge in performance trajectories. For scientific data analysis in particular, methods based on generating heavyweight access acceleration structures, e.g. indexes, are becoming less feasible for ever-increasing dataset sizes. We present ALACRITY, demonstrating the effectiveness of a fused data and index encoding of scientific, floating-point data in generating lightweight data structures amenable to common types of queries used in scientific data analysis. We exploit the representation of floating-point values by extracting significant bytes, using the resulting unique values to bin the remaining data along fixed-precision boundaries. To optimize query processing, we use an inverted index, mapping each generated bin to a list of records contained within, allowing us to optimize query processing with attribute range constraints. Overall, the storage footprint for both index and data is shown to be below numerous configurations of bitmap indexing, while matching or outperforming query performance.

Compression Ratio↗

Insights into the year-round vertical distribution of chlorophyll concentration in high-latitude Arctic Ocean: implications for primary production

Climate-induced rapid changes in the Arctic Ocean, such as decreasing sea ice extent and increasing water temperature, are altering nutrient and light availability, profoundly impacting primary producer growth. However, access to the high-latitude Arctic Ocean is limited, and satellite data are primarily available only during summer, making continuous in-situ data collection challenging. We collected year-round chlorophyll-a (Chl-a) concentration data in high-latitude regions using a mooring system and performed a comparative analysis with reanalysis data. Unlike previous satellite-based studies, which typically rely on surface measurements, we used the annual vertical distribution of Chl-a. These data were applied to the vertically generalized production model to accurately estimate annual primary production. The moored Chl-a concentration data showed that phytoplankton exhibited a typical subsurface chlorophyll maximum (SCM) layer as sea ice retreated in June. Contrary to the gradually deepening SCM distribution predicted by model-based reanalysis data, the SCM layer persisted for approximately 4 months. This indicates that light and nutrient conditions within the SCM layer remained stable, sustaining continuous phytoplankton growth. Annual primary production, reflecting this vertical distribution of Chl-a concentration, was 6.85 gC m −2 yr −1 . This exceeded satellite-based estimates by at least two-fold, highlighting the significant underestimation of primary production by satellite approaches. Estimating primary production while accounting for the vertical distribution of phytoplankton and light is essential for improving ecological models to better understand carbon cycle and food web changes in the Arctic Ocean, with important implications for climate change predictions.

Arctic Ocean↗

Multimodal X-ray nano-spectromicroscopy analysis of chemically heterogeneous systems

Abstract Understanding the nanoscale chemical speciation of heterogeneous systems in their native environment is critical for several disciplines such as life and environmental sciences, biogeochemistry, and materials science. Synchrotron-based X-ray spectromicroscopy tools are widely used to understand the chemistry and morphology of complex material systems owing to their high penetration depth and sensitivity. The multidimensional (4D+) structure of spectromicroscopy data poses visualization and data-reduction challenges. This paper reports the strategies for the visualization and analysis of spectromicroscopy data. We created a new graphical user interface and data analysis platform named XMIDAS (X-ray multimodal image data analysis software) to visualize spectromicroscopy data from both image and spectrum representations. The interactive data analysis toolkit combined conventional analysis methods with well-established machine learning classification algorithms (e.g. nonnegative matrix factorization) for data reduction. The data visualization and analysis methodologies were then defined and optimized using a model particle aggregate with known chemical composition. Nanoprobe-based X-ray fluorescence (nano-XRF) and X-ray absorption near edge structure (nano-XANES) spectromicroscopy techniques were used to probe elemental and chemical state information of the aggregate sample. We illustrated the complete chemical speciation methodology of the model particle by using XMIDAS. Next, we demonstrated the application of this approach in detecting and characterizing nanoparticles associated with alveolar macrophages. Our multimodal approach combining nano-XRF, nano-XANES, and differential phase-contrast imaging efficiently visualizes the chemistry of localized nanostructure with the morphology. We believe that the optimized data-reduction strategies and tool development will facilitate the analysis of complex biological and environmental samples using X-ray spectromicroscopy techniques.

36 MATERIALS SCIENCE↗

Avian Activity Classification Using Recurrent Networks to Fuse Videos with Metadata on Imbalanced Datasets

Activity classification plays a crucial role in various real-life scenarios involving both humans and animals. There is an increasing need for precise activity classification focused on avian-solar interactions, as the usage of solar energy facilities, such as photovoltaic array power stations, has been observed to impact bird species richness, behavior, and activity. However, there has been no work to develop an automated system to monitor and classify these avian-solar interactions. All current methods rely on human observers, which is time and human resources costly and subject to errors related to searcher efficiency. With the recent success of Deep Learning models in activity classification problems, this paper develops a recurrent neural network-based model to automatically classify six avian activities around solar energy facilities. Our proposed model integrates critical feature engineering metadata with video frame data, enabling improved learning and more accurate activity classification. Furthermore, we address the challenge of data imbalance during training and demonstrate the efficacy of our model in detecting and classifying different activities within video tracks. Additionally, we analyze the saliency/backpropagation map of the trained proposed model and validate its decision-making rationale.

Avian activity classification; bidirectional LSTM;↗

Status on Iterative Transform Phase Retrieval Applied to the GBT Data

This slide presentation reviews the use of iterative transform phase retrieval in the analysis of the Green Bank Radio Telescope (GBT) Data. It reviews the NASA projects that have used phase retrieval, and the testbed for the algorithm to be used for the James Webb Space Telescope. It shows the comparison of phase retrieval with an interferometer, and reviews the two approaches used for phase retrieval, iterative transform (ITA) or parametric (non-linear least squares model fitting). The concept of ITA Phase Retrieval is reviewed, and the application to Radio Antennas is reviewed. The presentation also examines the National Radio Astronomy Observatory (NRAO) data from the GBT, and the Fourier model that NRAO uses to analyze the data. The challenge for ITA phase retrieval is reviewed, and the coherent approximation for incoherent data is shown. The validity of the approximation is good for a large tilt. There is a review of the proof of concept of the Phase Review simulation using the input wavefront, and the initial sampling parameters estimate from the focused GBT data.

Dean, Bruce↗