Search NASA⌕ Search

SEARCH · Search NASA

Results for “massive datasets”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Picasso: Memory-Efficient Graph Coloring Using Palettes With Applications in Quantum Computing

A coloring of a graph is an assignment of colors to vertices such that no two neighboring vertices have the same color. The need for memory-efficient coloring algorithms is motivated by their application in computing clique partitions of graphs arising in quantum computations where the objective is to map a large set of Pauli strings into a compact set of unitaries. We present Picasso, a randomized memory-efficient iterative parallel graph coloring algorithm with theoretical sublinear space guarantees under practical assumptions. The parameters of our algorithm provide a trade-off between coloring quality and resource consumption. To assist the user, we also propose a machine learning model to predict the coloring algorithm’s parameters considering these trade-offs. We provide a sequential and a parallel implementation of the proposed algorithm. We perform an experimental evaluation on a 64-core AMD CPU equipped with 512 GB of memory and an Nvidia A100 GPU with 40GB of memory. For a small dataset where existing coloring algorithms can be executed within the 512 GB memory budget, we show up to 68× memory savings. On massive datasets we demonstrate that GPU-accelerated Picasso can process inputs with 49.5× more Pauli strings (vertex set in our graph) and 2,478× more edges than state-of-the-art parallel approaches.

artificial intelligence, quantum computing↗

Segmentation of Unstructured Datasets

Datasets generated by computer simulations and experiments in Computational Fluid Dynamics tend to be extremely large and complex. It is difficult to visualize these datasets using standard techniques like Volume Rendering and Ray Casting. Object Segmentation provides a technique to extract and quantify regions of interest within these massive datasets. This thesis explores basic algorithms to extract coherent amorphous regions from two-dimensional and three-dimensional scalar unstructured grids. The techniques are applied to datasets from Computational Fluid Dynamics and from Finite Element Analysis.

Bhat, Smitha↗

Bulk Data Movement for Climate Dataset: Efficient Data Transfer Management with Dynamic Transfer Adjustment

Many scientific applications and experiments, such as high energy and nuclear physics, astrophysics, climate observation and modeling, combustion, nano-scale material sciences, and computational biology, generate extreme volumes of data with a large number of files. These data sources are distributed among national and international data repositories, and are shared by large numbers of geographically distributed scientists. A large portion of data is frequently accessed, and a large volume of data is moved from one place to another for analysis and storage. One challenging issue in such efforts is the limited network capacity for moving large datasets to explore and manage. The Bulk Data Mover (BDM), a data transfer management tool in the Earth System Grid (ESG) community, has been managing the massive dataset transfers efficiently with the pre-configured transfer properties in the environment where the network bandwidth is limited. Dynamic transfer adjustment was studied to enhance the BDM to handle significant end-to-end performance changes in the dynamic network environment as well as to control the data transfers for the desired transfer performance. We describe the results from the BDM transfer management for the climate datasets. We also describe the transfer estimation model and results from the dynamic transfer adjustment.

97 MATHEMATICS AND COMPUTING↗

Breaking the Bottleneck of Genomes: Understanding Gene Function Across Taxa Workshop Report

In the last few decades, high-throughput technologies using various “omics” have enabled unprecedented views of biological systems at the molecular level. In parallel, the integration of omic datasets using computational modeling has provided new understanding of biological processes in organisms relevant to the U.S. Department of Energy’s (DOE) missions in energy and the environment. Collectively, these developments have spearheaded the advancement of systems biology, which can be viewed as a holistic approach for deciphering the complexity of biological systems. However, as high-throughput omic technologies and integrative systems biology efforts have improved our understanding of some biological systems, analyzing and finding meaningful answers within these massive datasets remain extremely challenging, in large part due to the lack of fundamental knowledge of gene function. Indeed, all sequenced genomes, both microbes and plants, contain large numbers of genes of “unknown function” that significantly limit scientists’ ability to model, predict, and engineer organisms with enhanced functions relevant to DOE. Current methodologies can be employed to decipher gene function, but they are typically slow, laborious, inefficient, and not scalable. This “bottleneck” in genome understanding could be broken with new, innovative, and transformative experimental tools, datasets, and computation that can define gene function on a massive and high-throughput scale compatible with the pace of DNA sequencing. In light of this grand challenge, DOE’s Office of Biological and Environmental Research (BER) convened the Breaking the Bottleneck of Genomes: Understanding Gene Function Across Taxa workshop on November 1–2, 2018. This workshop brought together leaders in microbiology, plant sciences, technology, and computation, who collectively identified the experimental and data analysis gaps preventing large-scale gene function determination as well as opportunities for overcoming these gaps.

59 BASIC BIOLOGICAL SCIENCES↗

Dark Field Proton Radiography (Los Alamos LDRD Report)

Proton radiography at LANL (pRad) takes advantage of the high flux, high proton energy, and flexible timing structure of LANSCE's 800-MeV proton beam to visualize dense, dynamic systems under extreme strain. With recent needs identified for the development of the W93, the pRad team has been actively searching for ways to increase the facility's radiographic capabilities. Recently, we discovered that utilizing two magnetic lenses (one upstream and one downstream of the object), and specific combinations of collimator choices within the two lenses, that the signal-to-noise ratio was improved by more than a factor of two. However, we did not fully understand the mechanism, and found that results did not match forward modeling predictions. This work specifically sought to implement an engineering solution to solve this problem, through the development of an actuated collimator that reduced the time required to swap collimator settings from a typical 4-8 hours, to less than 30 seconds. This allowed, in combination with a unique multi-material, multi-resolution stepwedge, to acquire an unprecedently enormous dataset, with in total 24 collimation combinations, and multiple diffuser settings and pulse patterns, equivalent to almost a full run cycle's worth of data in a few days. This massive dataset is currently being harnessed to make a fully-predictive, ultra-fast Python-based model of the system, to dial in specific settings for any future experiments that fully optimize that radiographic sensitivity to the material of interest.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Scientific data analysis on data-parallel platforms.

As scientific computing users migrate to petaflop platforms that promise to generate multi-terabyte datasets, there is a growing need in the community to be able to embed sophisticated analysis algorithms in the computing platforms' storage systems. Data Warehouse Appliances (DWAs) are attractive for this work, due to their ability to store and process massive datasets efficiently. While DWAs have been utilized effectively in data-mining and informatics applications, they remain largely unproven in scientific workloads. In this paper we present our experiences in adapting two mesh analysis algorithms to function on five different DWA architectures: two Netezza database appliances, an XtremeData dbX database, a LexisNexis DAS, and multiple Hadoop MapReduce clusters. The main contribution of this work is insight into the differences between these DWAs from a user's perspective. In addition, we present performance measurements for ten DWA systems to help understand the impact of different architectural trade-offs in these systems.

99 GENERAL AND MISCELLANEOUS↗

Mass Spectrometry Imaging Quick View

MSI QuickView is a software designed to provide a graphical user interface (GUI) for greatly speeding up experimental feedback (visualization and analysis) of mass spectrometry imaging (MSI or IMS) data during data acquisition. Often different software loads the entire data set, i.e., all lines of data into computer memory (RAM). This causes out of memory errors for larger datasets. We solved this in MSI QuickView by reading in the data one line at a time. Only the required information (e.g. the final pixel values for that line of heat map) is maintained in RAM. Interim analysis options include the mean intensity vs. m/z spectrum, intensity vs. time spectrums for up to 6 different m/z values or ranges chosen by the user and heat maps for each line. This assists in validating the usefulness of the particular experiment after scanning the first few lines. In addition, the tool facilitates further processing and analysis of the massive datasets. The user can manually pick different m/z values, time ranges, scroll through the spectra for any line in the data without having to load it in manually, save multiple images, change aspect ratios for the heat maps, and process the heat maps in multiple ways including overlaying images at different m/z values, displaying up to 9 different heat maps, alignment of scans along each line etc. There is no manipulation of the data required by the user to visualize the data.

Carson, JamesP.↗

Estimating basis functions in massive fields under the spatial mixed effects model

Abstract Spatial prediction is commonly achieved under the assumption of a Gaussian random field by obtaining maximum likelihood estimates of parameters, and then using the kriging equations to arrive at predicted values. For massive datasets, fixed rank kriging using the expectation–maximization algorithm for estimation has been proposed as an alternative to the usual but computationally prohibitive kriging method. The method reduces computation cost of estimation by redefining the spatial process as a linear combination of basis functions and spatial random effects. A disadvantage of this method is that it imposes constraints on the relationship between the observed locations and the knots. We develop an alternative method that utilizes the spatial mixed effects model, but allows for additional flexibility by estimating the range of the spatial dependence between the observations and the knots via an alternating expectation conditional maximization algorithm. Experiments show that our methodology improves estimation without sacrificing prediction accuracy while also minimizing the additional computational burden of extra parameter estimation. The methodology is applied to a temperature dataset archived by the United States National Climate Data Center, with improved results over previous methodology.

Pazdernik, Karl↗

Frictionless knowledge injection for few-shot learning

Cutting-edge machine learning methods often require large volumes of curated training data, precluding their use in national security problems with rare events in massive datasets. We present a method for incorporating abstract knowledge into models tailored for sparse data. A subject matter expert defines salient concepts using data examples, which are encoded in the model’s embedding space. Models are then trained to respect these concepts. This method enables knowledge injection, yielding effective models with limited labeled data and the ability to assess model sensitivity for subject matter expertise across the nonproliferation mission space, as demonstrated with Raman spectra analysis.

Stomps, Jordan [ORNL] (ORCID:0000000178114479)↗

Scalable 3D reconstruction for X-ray single particle imaging with online machine learning

X-ray free-electron lasers offer unique capabilities for measuring the structure and dynamics of biomolecules, helping us understand the basic building blocks of life. Notably, high-repetition-rate free-electron lasers enable single particle imaging, where individual, weakly scattering biomolecules are imaged under near-physiological conditions with the opportunity to access fleeting states that cannot be captured in cryogenic or crystallized conditions. Existing X-ray single particle reconstruction algorithms, which estimate the particle orientation for each image independently, are slow and memory-intensive when handling the massive datasets generated by emerging free-electron lasers. Here, we introduce X-RAI (X-Ray single particle imaging with Amortized Inference), an online reconstruction framework that estimates the structure of 3D macromolecules from large X-ray single particle datasets. X-RAI consists of a convolutional encoder, which amortizes pose estimation over large datasets, as well as a physics-based decoder, which employs an implicit neural representation to enable high-quality 3D reconstruction in an end-to-end, self-supervised manner. We demonstrate that X-RAI achieves state-of-the-art performance for small-scale datasets in simulation and challenging experimental settings and demonstrate its unprecedented ability to process large datasets containing millions of diffraction images in an online fashion. These abilities signify a paradigm shift in X-ray single particle imaging towards real-time reconstruction.

Computer science↗

Foundation models for atomistic simulation of chemistry and materials

Conventional computational methods for modeling chemical and materials systems are limited by system size and timescale, forcing a trade-off between quantum-mechanical accuracy and the sampling needed for realistic observables. Large language and vision foundation models — pre-trained on massive datasets using transformer architectures — have revolutionized many fields. It is thus interesting to ask whether a foundation model — subject to suitable data, parameter scaling and training — could enable learned simulations of chemistry and materials. Here, in this study, we review the field of machine-learned interatomic potentials (MLIPs) and posit that scaling up large and diverse chemical and materials datasets and highly expressive architectures using advanced training strategies should result in models that are: more efficient, transferable, robust to out-of-distribution scenarios, and easier to fine-tune to a variety of downstream physical observables than models trained from scratch on small datasets corresponding to specific, targeted atomistic simulation tasks. We provide specific criteria for creating such large-scale MLIP foundation models, coordinated strategies for their development, evaluation and deployment, and highlight potential emergent capabilities that could transform predictive simulations in chemistry and materials science and accelerate discovery across multiple technological domains.

Yuan, Eric C.-Y. [University of California, Berkel↗

An AI-accelerated pathway for reproducible and stable halide perovskites

Halide perovskites (HPs) have remarkable optoelectronic properties, and in the last decade their photovoltaic power conversion efficiency and light-emitting diode efficiency have skyrocketed. Despite the surge in research on these burgeoning materials, two key challenges in the field remain: material irreproducibility and instability. Their behavior is especially dynamic in response to environmental stressors, due to complex interactions with the perovskite crystal lattice. Here, in this review, we survey the latest achievements in HP materials research accomplished with the assistance of artificial intelligence (AI), through the implementation of automated experimentation and machine learning (ML) data analysis. Automated synthesis and characterization tackle problems with material irreproducibility by systematically controlling parameters with very high precision, creating massive datasets, and allowing methodical comparisons from which unbiased conclusions can be drawn. AI can reveal otherwise unnoticed trends, inform future experiments with the highest potential information gain, and forecast future performance. The review concludes with a forward viewpoint of how human-assisted closed-loop laboratories and shared databases allow halide perovskite materials’ processing, properties, and performance to be potentially optimized with AI, accelerating the development of highly reproducible and stable optoelectronic devices.

Hering, Abigail R. [Univ. of California, Davis, CA↗

Using the optimal combined index weight ratio to improve the probability of anomaly detection in big area additive manufacturing

Big Area Additive Manufacturing (BAAM) of composites requires significant time, energy, and material, so it is critical to reduce production inefficiencies to make functional parts without multiple iterations. Statistical process control coupled with Principal Component Analysis (PCA) is a powerful technique that provides a quick, computationally inexpensive, and intuitive way for operators to detect defects that form in a manufacturing process without massive datasets. Recently, a combined index that is a weighted sum of the Hotelling's T 2 and squared residual error statistics has been proposed that can be monitored in one chart, improving interpretation accuracy and simplicity. However, the literature does not offer a formal method to optimise the weights. Here, we introduce two new approaches to the traditional weight selection approach using simulated and BAAM image data. Approach 1 uses a theoretically motivated optimum inspired by probabilistic principal component analysis. Approach 2 systematically varies the ratio of the weights to find the optimum. We show that approach 1 delivers optimal anomaly detection performance in select cases while approach 2 fares better in practice. Surprisingly, we also show that choosing a more complex PCA model has a minimal negative impact on anomaly detection performance compared to a more simplistic model.

3-dimensional printing↗

Elastic Resource Management for Deep Learning Applications in a Container Cluster

The increasing demand for learning from massive datasets is restructuring our economy. Effective learning, however, involves nontrivial computing resources. Most businesses utilize commercial infrastructure providers (e.g., AWS) to host their computing clusters in the cloud, where various jobs compete for available resources. While cloud resource management is a fruitful research field that has made many advances in production, such as Kubernetes and YARN, few efforts have been invested to further optimize the system performance, especially for deep learning (DL) training jobs in a container cluster. This work introduces FlowCon, a system that is able to monitor the individual evaluation functions of DL jobs at runtime, and thus to make placement decisions on resource allocations elastically. Here, we present a detailed design and implementation of FlowCon and conduct intensive experiments over various DL models. The results demonstrate that FlowCon significantly improves DL job completion time and resource utilization efficiency, compared to default systems. According to the results, FlowCon is able to improve the completion time by up to 68.8% and meanwhile, reduce the makespan by 18.0%, in the presence of various DL job workloads.

97 MATHEMATICS AND COMPUTING↗

Design Principles for Effective Knowledge Discovery from Big Data

Big data phenomenon refers to the practice of collection and processing of very large data sets and associated systems and algorithms used to analyze these massive datasets. Architectures for big data usually range across multiple machines and clusters, and they commonly consist of multiple special purpose sub-systems. Coupled with the knowledge discovery process, big data movement offers many unique opportunities for organizations to benefit (with respect to new insights, business optimizations, etc.). However, due to the difficulty of analyzing such large datasets, big data presents unique systems engineering and architectural challenges. In this paper, we present three sys- tem design principles that can inform organizations on effective analytic and data collection processes, system organization, and data dissemination practices. The principles presented derive from our own research and development experiences with big data problems from various federal agencies, and we illustrate each principle with our own experiences and recommendations.

Architecture↗

Computationally Efficient Training of Deep Neural Networks via Transfer Learning

Transfer learning is a highly successful approach to train deep neural networks for customized image recognition tasks. Transfer learning leverages deep learning models that are previously trained on massive datasets and re-trains the network for a novel image recognition dataset. Typically, the advantage of transfer learning has been measured in sample efficiency, but instead, we investigate the computational efficiency of transfer learning. A good pre-trained model provides features that can be used as input to a new classifier (usually the top layers of a neural network). We show that if a good pre-trained model is selected, that training a new classifier can be much more computationally efficient than training a deep neural network without transfer learning. The first step in transfer learning is to select a pre-trained model to use as a feed-forward network for generating features. Here this selection process either uses human intuition/convenience or a methodical but computationally intensive validation loop. Instead, we would prefer a method to select the pre-trained model that will produce the best transfer results for the least amount of computation. To this end, we provide a computationally efficient metric for the fit between a pre-trained model and a novel image recognition task. The better the fit, the less computation will be needed to re-train the pre-trained model for the novel task. As machine learning becomes ubiquitous, highly-accurate trained models will proliferate and computationally efficient transfer learning methods will enable rapid development of new image recognition models.

97 MATHEMATICS AND COMPUTING↗

Scalable deep text comprehension for Cancer surveillance on high-performance computing

Background: Deep Learning (DL) has advanced the state-of-the-art capabilities in bioinformatics applications which has resulted in trends of increasingly sophisticated and computationally demanding models trained by larger and larger data sets. This vastly increased computational demand challenges the feasibility of conducting cutting-edge research. One solution is to distribute the vast computational workload across multiple computing cluster nodes with data parallelism algorithms. In this study, we used a High-Performance Computing environment and implemented the Downpour Stochastic Gradient Descent algorithm for data parallelism to train a Convolutional Neural Network (CNN) for the natural language processing task of information extraction from a massive dataset of cancer pathology reports. We evaluated the scalability improvements using data parallelism training and the Titan supercomputer at Oak Ridge Leadership Computing Facility. To evaluate scalability, we used different numbers of worker nodes and performed a set of experiments comparing the effects of different training batch sizes and optimizer functions.Results: We found that Adadelta would consistently converge at a lower validation loss, though requiring over twice as many training epochs as the fastest converging optimizer, RMSProp. The Adam optimizer consistently achieved a close 2nd place minimum validation loss significantly faster; using a batch size of 16 and 32 allowed the network to converge in only 4.5 training epochs.Conclusions: We demonstrated that the networked training process is scalable across multiple compute nodes communicating with message passing interface while achieving higher classification accuracy compared to a traditional machine learning algorithm.

Qiu, John X.↗