SEARCH · Search NASA

Results for “Large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17

All sources NASA NTRS DOE OSTI

DOE OSTI2020

Generation and evaluation of synthetic patient data

Background: Machine learning (ML) has made a significant impact in medicine and cancer research; however, its impact in these areas has been undeniably slower and more limited than in other application domains. A major reason for this has been the lack of availability of patient data to the broader ML research community, in large part due to patient privacy protection concerns. High-quality, realistic, synthetic datasets can be leveraged to accelerate methodological developments in medicine. By and large, medical data is high dimensional and often categorical. These characteristics pose multiple modeling challenges. Methods: In this paper, we evaluate three classes of synthetic data generation approaches; probabilistic models, classification-based imputation models, and generative adversarial neural networks. Metrics for evaluating the quality of the generated synthetic datasets are presented and discussed. Results: While the results and discussions are broadly applicable to medical data, for demonstration purposes we generate synthetic datasets for cancer based on the publicly available cancer registry data from the Surveillance Epidemiology and End Results (SEER) program. Specifically, our cohort consists of breast, respiratory, and non-solid cancer cases diagnosed between 2010 and 2015, which includes over 360,000 individual cases. Conclusions: We discuss the trade-offs of the different methods and metrics, providing guidance on considerations for the generation and usage of medical synthetic data.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2026

Leveraging large language models to address data scarcity in machine learning for graphene synthesis

Machine learning in experimental materials science faces significant challenges due to the scarcity of data, which are costly and time-consuming to generate, particularly when relying on in-house experiments. Literature data mining offers a potential solution but introduces issues like mixed data quality, inconsistent formats, and non-uniform reporting of synthesis parameters, resulting in partially missing and heterogeneous features across the dataset. Here, we propose data imputation and feature engineering methods that employ pre-trained large language models (LLMs) to enhance machine learning performance on scarce, heterogeneous datasets, demonstrated on graphene CVD synthesis data and the ML-HydPARK hydrogen storage dataset. GPT models perform data imputation via tailored prompting and semantic normalization of inconsistently reported features through embeddings, for example, to harmonize the complex nomenclature of CVD substrates. Beyond yielding more diverse and richer feature representations than traditional methods such as K-nearest neighbors (KNN) and Multivariate Imputation by Chained Equations (MICE), LLM-based data imputation is evaluated against dataset characteristics and prompting strategies. We vary the level of autonomy granted to the LLM, from generic prompting that leverages pre-trained knowledge for autonomous data generation to data-informed prompting that constrains outputs using target-specific information, and demonstrate which level of autonomy yields superior imputation performance across datasets and feature types. The proposed data engineering methods markedly improve downstream performance; for example, in graphene layer number classification using a support vector machine (SVM), binary accuracy increases from 39% to 65% and ternary accuracy from 52% to 72%. Fine-tuning experiments on both datasets show that combining our proposed LLM-based data imputation and feature encoding methods with numerical machine learning predictors outperforms standalone fine-tuned LLM predictors in data-scarce settings. The proposed strategies emphasize data enhancement techniques rather than refining learning architectures or regularizing loss functions, offering a broadly applicable framework for improving machine learning performance on scarce, inhomogeneous datasets.

Chemical vapor deposition↗

DOE OSTI2020

Seismology in the Cloud: A New Streaming Workflow

Data-intensive research in seismology is experiencing a recent boom, driven in part by large volumes of available data and advances in the growing field of data science. However, there are significant barriers to processing large data volumes, such as long retrieval times from data repositories, complex data management, and limited computational resources. New tools and platforms have reduced the barriers to entry for scientific cluster computing, including the maturation of the commercial cloud as an accessible instrument for research. Here, we build a customized research cluster in the cloud to test a new workflow for large-scale seismic analysis, in which data are processed as a stream (retrieved on-the-fly and acted upon without storing), with data from the Incorporated Research Institutions for Seismology Data Management Center. We use this workflow to deploy a spectral peak detection algorithm over 5.6 TB of compressed continuous seismic data from 2074 stations of the USArray Transportable Array EarthScope network. Using a 50-node cluster in the cloud, we completed the noise survey in 80 hr, with an average data throughput of 1.7 GB per minute. By varying cluster sizes, we find the scaling of our analysis to be sublinear, due to a combination of algorithmic limitations and data center response times. The cloud-based streaming workflow represents an order-of-magnitude increase in acquisition and processing speed compared to a traditional download-store-process workflow, and offers the additional benefits of employing a flexible, accessible, and widely used computing architecture. It is limited, however, due to its reliance on Internet transfer speeds and data center service capacity, and may not work well for repeated analyses or those for which even higher data throughputs are needed. These research applications will require a new class of cloud-native approaches in which both data and analysis are in the cloud.

58 GEOSCIENCES↗

DOE OSTI2020

Disentangling Sources of Gene Tree Discordance in Phylogenomic Data Sets: Testing Ancient Hybridizations in Amaranthaceae s.l

Gene tree discordance in large genomic data sets can be caused by evolutionary processes such as incomplete lineage sorting and hybridization, as well as model violation, and errors in data processing, orthology inference, and gene tree estimation. Species tree methods that identify and accommodate all sources of conflict are not available, but a combination of multiple approaches can help tease apart alternative sources of conflict. Here, using a phylotranscriptomic analysis in combination with reference genomes, we test a hypothesis of ancient hybridization events within the plant family Amaranthaceae s.l. that was previously supported by morphological, ecological, and Sanger-based molecular data. The data set included seven genomes and 88 transcriptomes, 17 generated for this study. We examined gene-tree discordance using coalescent-based species trees and network inference, gene tree discordance analyses, site pattern tests of introgression, topology tests, synteny analyses, and simulations. We found that a combination of processes might have generated the high levels of gene tree discordance in the backbone of Amaranthaceae s.l. Furthermore, we found evidence that three consecutive short internal branches produce anomalous trees contributing to the discordance. Overall, our results suggest that Amaranthaceae s.l. might be a product of an ancient and rapid lineage diversification, and remains, and probably will remain, unresolved. This work highlights the potential problems of identifiability associated with the sources of gene tree discordance including, in particular, phylogenetic network methods. Our results also demonstrate the importance of thoroughly testing for multiple sources of conflict in phylogenomic analyses, especially in the context of ancient, rapid radiations. We provide several recommendations for exploring conflicting signals in such situations.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2023

WiP Abstract: Edge-Based Privacy of Naturalistic Driving Data Collection

Collecting large driving datasets is important for data-driven transportation research and studied in naturalistic driving [3]. Due to the standard implementation of a Controller Area Network (CAN) bus for a vehicle’s inter-module communication, many off-the-shelf devices can easily transform a vehicle into a rich data collection utility [2]. Vehicles with Adaptive Cruise Control (ACC) are an example of a feature resulting in emergent traffic behavior when scaled [4]. While these utilities were designed with particular data use cases, data may be publicly shared to benefit other researchers through online tools like CyVerse [1]. However, such data should only be shared when any private information is removed. This private information may exist as a set of GPS coordinates, since the start and end points of a trip may designate a driver’s place of residence or work. When considering larger continuously-collected data sets with a focus on naturalistic driving, many drivers are needed to make data collection feasible. Removal of private information becomes much more of a challenge since every driver may uniquely define their geographic privacy. This project aims to build upon the foundation of libpanda [2] by adding features of edge-based data privacy enforcement. In it’s current form, libpanda uses a GPS in conjunction with a CAN interface to record data. Libpanda feature s a set of startup and shutdown scripts to perform automatic data collection and upload. With additional support hardware on a Raspberry Pi, the Pi can maintain power on vehicle shutdown to automatically upload data before shutdown.

Bunting, Matt↗

DOE OSTI2022

DeepZipper: A Novel Deep-learning Architecture for Lensed Supernovae Identification

Large-scale astronomical surveys have the potential to capture data on large numbers of strongly gravitationally lensed supernovae (LSNe). To facilitate timely analysis and spectroscopic follow-up before the supernova fades, an LSN needs to be identified soon after it begins. To quickly identify LSNe in optical survey data sets, we designed ZipperNet, a multibranch deep neural network that combines convolutional layers (traditionally used for images) with long short-term memory layers (traditionally used for time series). We tested ZipperNet on the task of classifying objects from four categories—no lens, galaxy-galaxy lens, lensed Type-Ia supernova, lensed core-collapse supernova—within high-fidelity simulations of three cosmic survey data sets: the Dark Energy Survey, Rubin Observatory’s Legacy Survey of Space and Time (LSST), and a Dark Energy Spectroscopic Instrument (DESI) imaging survey. Among our results, we find that for the LSST-like data set, ZipperNet classifies LSNe with a receiver operating characteristic area under the curve of 0.97, predicts the spectroscopic type of the lensed supernovae with 79% accuracy, and demonstrates similarly high performance for LSNe 1–2 epochs after first detection. We anticipate that a model like ZipperNet, which simultaneously incorporates spatial and temporal information, can play a significant role in the rapid identification of lensed transient systems in cosmic survey experiments.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2023

Transferring a Molecular Foundation Model for Polymer Property Predictions

Transformer-based large language models have remarkable potential to accelerate design optimization for applications such as drug development and material discovery. Self-supervised pretraining of transformer models requires large-scale data sets, which are often sparsely populated in topical areas such as polymer science. Further, state-of-the-art approaches for polymers conduct data augmentation to generate additional samples but unavoidably incur extra computational costs. In contrast, large-scale open-source data sets are available for small molecules and provide a potential solution to data scarcity through transfer learning. In this work, we show that using transformers pretrained on small molecules and fine-tuned on polymer properties achieves comparable accuracy to those trained on augmented polymer data sets for a series of benchmark prediction tasks.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2021

Synchrophasor Data Compression Under Disturbance Conditions via Cross-Entropy-Based Singular Value Decomposition

The increasing deployment of phasor measurement units and the advances of their reporting rates are challenging the present data centers in terms of storing and analyzing large-volume data. Under power system disturbance conditions, it is difficult to retain critical information while compressing the synchrophasor data effectively. This article combines the cross entropy and the singular value decomposition, proposing a novel model to compress the synchrophasor data to an extremely small size yet keep superior accuracy. The proposed model is extensively tested and compared with the state-of-the-art algorithms using the simulated and the FNET/GridEye field-collected data. The result indicates that the proposed algorithm has superior performance in compressing the data while retaining critical information under disturbance conditions.

42 ENGINEERING↗

DOE OSTI2026

Techno-Economic Assessment of Data Center Load Demand Powered by Small Modular Reactors and Distributed Energy Resources

The rapid increase in data center energy demand, driven by AI and large-scale data processing, poses significant challenges to global energy infrastructure. Data centers require substantial and reliable energy for continuous operations and high-performance computing. Current electrical grids face issues such as transmission bottlenecks and aging infrastructure, making it difficult to meet these demands. Integrating inverter-based-resources (IBRs) like solar and wind presents both opportunities and challenges due to their intermittent nature. Small Modular Reactors (SMRs) offer a promising solution with their enhanced safety, modularity, reliability, and scalability, providing consistent base load power ideal for data center operations. This study presents a comprehensive techno-economic assessment of powering data center load demand using a combination of SMRs and IBRs with grid-connected and islanded mode. This study utilized Idaho National Laboratory’s (INL) HPC data center hourly load profiles and Xendee microgrid optimization platform to conduct the analysis. In this configuration, SMRs serves as the primary base load power source, consistently providing a steady supply of electricity necessary to meet the minimum load demand of the data center with support from the IBRs. Key performance indicators such as Levelized Cost of Electricity (LCOE), Net Present Value (NPV) has been calculated to assess the economic feasibility. The findings from this research will underscore the strategic benefits of integrating SMR plant with DERs – particularly for critical infrastructure load such as data centers.

14 - SOLAR ENERGY↗

DOE OSTI2020

Using SPT polarization, Planck 2015, and non-CMB data to constrain tilted spatially-flat and untilted nonflat ΛCDM, XCDM, and ΦCDM dark energy inflation cosmologies

We use six tilted spatially-flat and untilted non-flat dark energy cosmological models in analyses of South Pole Telescope polarization (SPTpol) cosmic microwave background (CMB) data, alone and in combination with Planck 2015 CMB data and non-CMB data, namely, the Pantheon Type Ia supernovae apparent magnitudes, a collection of baryon acoustic oscillation data points, Hubble parameter measurements, and growth rates. Although the cosmological models that best-fit the Planck CMB and non-CMB data do not provide good fits to the SPTpol data, with the χ2's exceeding the expected value, given the uncertainties, in each model the cosmological parameter constraints from the SPTpol data and from the Planck CMB and non-CMB data are largely mutually consistent. Furthermore, when the smaller angular scale SPTpol data are used jointly with either the Planck data alone or with the Planck CMB and the non-CMB data to constrain untilted non-flat models, spatially-closed models remain favored over their corresponding flat limits. When used in conjunction with Planck data, non-CMB data (baryon acoustic oscillation measurements in particular, from six experiments) have significantly more constraining power than the SPTpol data.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2025

High-performance data format for scientific data storage and analysis

Here, in this article, we present the High-Performance Output (HiPO) data format developed at Jefferson Laboratory for storing and analyzing data from Nuclear Physics experiments. The format was designed to efficiently store large amounts of experimental data, utilizing modern fast compression algorithms. The purpose of this development was to provide organized data in the output, facilitating access to relevant information within the large data files. The HiPO data format has features that are suited for storing raw detector data, reconstruction data, and the final physics analysis data efficiently, eliminating the need to do data conversions through the lifecycle of experimental data. The HiPO data format is implemented in C++ and JAVA, and provides bindings to FORTRAN, Python, and Julia, providing users with the choice of data analysis frameworks to use. In this paper, we will present the general design and functionalities of the HiPO library and compare the performance of the library with more established data formats used in data analysis in High Energy and Nuclear Physics (such as ROOT and Parquete). In columnar data analysis, HiPO surpasses established data formats in performance and can be effectively applied to data analysis in other scientific fields.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2021

Clinical knowledge extraction via sparse embedding regression (KESER) with multi-center large scale electronic health record data

The increasing availability of electronic health record (EHR) systems has created enormous potential for translational research. However, it is difficult to know all the relevant codes related to a phenotype due to the large number of codes available. Traditional data mining approaches often require the use of patient-level data, which hinders the ability to share data across institutions. In this project, we demonstrate that multi-center large-scale code embeddings can be used to efficiently identify relevant features related to a disease of interest. We constructed large-scale code embeddings for a wide range of codified concepts from EHRs from two large medical centers. We developed knowledge extraction via sparse embedding regression (KESER) for feature selection and integrative network analysis. We evaluated the quality of the code embeddings and assessed the performance of KESER in feature selection for eight diseases. Besides, we developed an integrated clinical knowledge map combining embedding data from both institutions. The features selected by KESER were comprehensive compared to lists of codified data generated by domain experts. Features identified via KESER resulted in comparable performance to those built upon features selected manually or with patient-level data. The knowledge map created using an integrative analysis identified disease-disease and disease-drug pairs more accurately compared to those identified using single institution data. Analysis of code embeddings via KESER can effectively reveal clinical knowledge and infer relatedness among codified concepts. KESER bypasses the need for patient-level data in individual analyses providing a significant advance in enabling multi-center studies using EHR data.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2025

Using Temporal Information from Human Mobility Data to Detect Anchor Points

Spatiotemporal mobility data are available in massive quantities, but large quantities of data typically include fewer variables or data fields. Often, the only available fields are User ID, Longitude, Latitude, Timestamp (ULLT). This raises an important question: how much can we infer about human mobility patterns using only these four fields? With ULLT data, we do not know individuals' socioeconomic status information or when they are visiting their anchor points (AP) or locations (such as homes, places of employment, or schools), and it is a modern challenge to use this data to infer these characteristics. When detecting anchor locations with limited input information, verification and validation (VV) are significant challenges. This paper addresses the problem of identifying individuals' anchor locations using only temporal information from spatiotemporal datasets with limited attributes. Our approach does not explicitly use latitude and longitude during analysis. Locationbased information is only employed in the preprocessing stage to identify periods of movement (trips) and stops (dwelling). Beyond this step, all analysis is based on temporal patterns. In theory, if stops and dwell times could be detected through alternative means, our method could function entirely without location-based input. We demonstrate this methodology on the 2017 National Household Travel Survey (NHTS) data, because it includes a carefully designed and collected time use survey with representative sampling and labeled ground truth. The high-quality survey data allows us to test the accuracy of our methods because NHTS contains intended place labels and agent/user characteristics. We have also applied our validated AP identification algorithm on very large-scale GPS based trajectory data for Patterns-of-Life (PoL) assessment and other applications, but due to space limit that could not be presented here.

McBride, Liz [ORNL] (ORCID:0000000286925869)↗

DOE OSTI2022

Big Data Seismology

The discipline of seismology is based on observations of ground motion that are inherently undersampled in space and time. Our basic understanding of earthquake processes and our ability to resolve 4D Earth structure are fundamentally limited by data volume. Currently, Big Data Seismology is an emergent revolution involving the use of large, data-dense inquiries that is providing new opportunities to make fundamental advances in these areas. This article reviews recent scientific advances enabled by Big Data Seismology through the context of three major drivers: the development of new data-dense sensor systems, improvements in computing, and the development of new types of techniques and algorithms. Each driver is explored in the context of both global and exploration seismology, alongside collaborative opportunities that combine the features of long-duration data collections (common to global seismology) with dense networks of sensors (common to exploration seismology). The review explores some of the unique challenges and opportunities that Big Data Seismology presents, drawing on parallels from other fields facing similar issues. Finally, recent scientific findings enabled by dense seismic data sets are discussed, and we assess the opportunities for significant advances made possible with Big Data Seismology. This review is designed to be a primer for seismologists who are interested in getting up-to-speed with how the Big Data revolution is advancing the field of seismology.

58 GEOSCIENCES↗

DOE OSTI2025

Streaming Compression of Scientific Data via Weak-SINDy

Here, in this paper, a streaming weak-SINDy algorithm is developed specifically for compressing streaming scientific data. The production of scientific data, either via simulation or experiments, is undergoing a stage of exponential growth, which makes data compression important and often necessary for storing and utilizing large scientific data sets. As opposed to classical “offline” compression algorithms that perform compression on a readily available data set, streaming compression algorithms compress data “online” while the data generated from simulation or experiments is still flowing through the system. This feature makes streaming compression algorithms well suited for scientific data compression, where storing the full data set offline is often infeasible. This work proposes a new streaming compression algorithm, streaming weak-SINDy, which takes advantage of the underlying data characteristics during compression. The streaming weak-SINDy algorithm constructs feature matrices and target vectors in the online stage via a streaming integration method in a memory efficient manner. The feature matrices and target vectors are then used in the offline stage to build a model through a regression process that aims to recover equations that govern the evolution of the data. For compressing high-dimensional streaming data, we adopt a streaming proper orthogonal decomposition (POD) process to reduce the data dimension and then use the streaming weak-SINDy algorithm to compress the temporal data of the POD expansion. We propose modifications to the streaming weak-SINDy algorithm to accommodate the dynamically updated POD basis. By combining the built model from the streaming weak-SINDy algorithm and a small amount of data samples, the full data flow could be reconstructed accurately at a low memory cost, as shown in the numerical tests.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Analyzing Mass Spectrometry Imaging Data of 13C-Labeled Phospholipids in Camelina sativa and Thlaspi arvense (Pennycress) Embryos

The combination of 13C-isotopic labeling and mass spectrometry imaging (MSI) offers an approach to analyze metabolic flux in situ. However, combining isotopic labeling and MSI presents technical challenges ranging from sample preparation, label incorporation, data collection, and analysis. Isotopic labeling and MSI individually create large, complex data sets, and this is compounded when both methods are combined. Therefore, analyzing isotopically labeled MSI data requires streamlined procedures to support biologically meaningful interpretations. Using currently available software and techniques, here we describe a workflow to analyze 13C-labeled isotopologues of the membrane lipid and storage oil lipid intermediate―phosphatidylcholine (PC). Our results with embryos of the oilseed crops, Camelina sativa and Thlaspi arvense (pennycress), demonstrated greater 13C-isotopic labeling in the cotyledons of developing embryos compared with the embryonic axis. Greater isotopic enrichment in PC molecular species with more saturated and longer chain fatty acids suggest different flux patterns related to fatty acid desaturation and elongation pathways. The ability to evaluate MSI data of isotopically labeled plant embryos will facilitate the potential to investigate spatial aspects of metabolic flux in situ.

metabolomics↗

DOE OSTI2024

Globus service enhancements for exascale applications and facilities

Many extreme-scale applications require the movement of large quantities of data to, from, and among leadership computing facilities, as well as other scientific facilities and the home institutions of facility users. These applications, particularly when leadership computing facilities are involved, can touch upon edge cases (e.g., terabyte files) that had not been a focus of previous Globus optimization work, which had emphasized rather the movement of many smaller (megabyte to gigabyte) files. We report here on how automated client-driven chunking can be used to accelerate both the movement of large files and the integrity checking operations that have proven to be essential for large data transfers. In conclusion, we present detailed performance studies that provide insights into the benefits of these modifications in a range of file transfer scenarios.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2019

Large-Scale Grid Models: Synthetic Data and Co-Simulation

Power grid modernization and wide-spread distributed energy resource (DER) integration are driving an explosion of interest in advanced algorithms and new optimization schemes. However, there has historically been a lack of open-access realistically sized and detailed datasets-particularly for the distribution system-and simulation tools able to model these systems. This talk will present recent advances with very large-scale (>>1M nodes) distribution data sets and the use of co-simulation to tractably model large integrated transmission-distribution grids and their interactions with new controls/optimization.

24 POWER TRANSMISSION AND DISTRIBUTION↗