Search NASA⌕ Search

SEARCH · Search NASA

Results for “large data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

On-demand Overlay Networks for Large Scientific Data Transfers

Large scale scientific data transfers are central to scientific processes. Data from large experimental facilities have to be moved to local institutions for analysis or often data needs to be moved between local clusters and large supercomputing centers. In this paper, we propose and evaluate a network overlay architecture to enable highthroughput, on-demand, coordinated data transfers over wide-area networks. Our work leverages Phoebus and On-demand Secure Circuits and AdvanceReservation System (OSCARS) to provide high performance wide-area network connections. OSCARS enables dynamic provisioning of network paths with guaranteed bandwidth and Phoebus enables the coordination and effective utilization of the OSCARS network paths. Our evaluation shows that this approach leads to improved end-to-end data transfer throughput with minimal overheads. The achievedthroughput using our overlay was limited only by the ability of the end hosts to sink the data.

97 MATHEMATICS AND COMPUTING↗

Extreme Compression for Large Scale Data Store

For the last 5 years Accelogic pioneered and perfected a radically new theory of numerical computing codenamed “Compressive Computing”, which has an extremely profound impact on real-world computer science [1]. At the core of this new theory is the discovery of one of its fundamental theorems which states that, under very general conditions, the vast majority (typically between 70% and 80%) of the bits used in modern large-scale numerical computations are absolutely irrelevant for the accuracy of the end result. This theory of Compressive Computing provides mechanisms able to identify (with high intelligence and surgical accuracy) the number of bits (i.e., the precision) that can be used to represent numbers without affecting the substance of the end results, as they are computed and vary in real time. The bottom line outcome would be to provide a state-of-the-art compression algorithm that surpasses those currently available in the ROOT framework, with the purpose of enabling substantial economic and operational gains (including speedup) for High Energy and Nuclear Physics data storage/analysis. In our initial studies, a factor of nearly x4 (3.9) compression was achieved with RHIC/STAR data where ROOT compression managed only x1.4.In this contribution, we will present our concepts of “functionally lossless compression”, have a glance at examples and achievements in other communities, present the results and outcome of our current, ongoing R&D, as well as present a high-level view of our plan to move forward with a ROOT implementation that would deliver a basic solution readily integrated into HENP applications. As a collaboration of experimental scientists, private industry, and the ROOT Team, our aim is to capitalize on the substantial success delivered by the initial effort and produce a robust technology properly packaged as an open-source tool that could be used by virtually every experiment around the world as means for improving data management and accessibility.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Using Large Language Models to help customers monitor global threat data

Large Language Models have proven adept at answering general knowledge questions. To make these generative AI tools useful to our mission customers for monitoring global threats, the data sciences team at Sandia is utilizing retrieval augmented generation (RAG) techniques to customize these models with local data. The local data we use consists of data such as research articles and patent abstracts that we've collected over the last several years using automated pipelines.

Herzer, John Andrew [Sandia National Laboratories ↗

Putting Our Industry's Data to Work: A Case Study of Large-Scale Data Aggregation: Preprint

With increasing deployment of Advanced Metering Infrastructure (AMI), Building Automation System (BAS) controls, Internet of Things (IoT) network devices, and data-driven evaluation, measurement, and verification studies, the building sector is currently generating a staggering amount of energy-related data. In the right hands, these data sets can contribute to increased comfort and energy savings for building occupants and a more reliable electrical grid; however, due to a combination of factors, including significant privacy concerns, much of the data that are presently generated and stored are not used outside of basic operational applications. In the past year, our team has dedicated over 2,000 person-hours to accessing building energy data for a project funded by the U.S. Department of Energy (DOE) Building Technologies Office. We sought whole-building or end-use (e.g., lighting) timeseries data at the individual-building or equipment level where possible or aggregated information, such as timeseries averages and quartiles by building type (e.g., office, retail, hospital), where sharing individual building information was not an option. We are additionally working with IoT and BAS data sets to derive information important to the project. We have assembled an extensive data set that will enable the development of publicly available end-use load profiles to benefit the U.S. building and electricity industries. Here we present an overview of the data set that we have assembled to date, the motivators and approaches that got us here, and the lessons we learned through our efforts. We also discuss work underway that presents additional options for future data access.

building energy data↗

Efficient Feature-Driven Visualization of Large-Scale Scientific Data

Very large, complex scientific data acquired in many research areas creates critical challenges for scientists to understand, analyze, and organize their data. The objective of this project is to expand the feature extraction and analysis capabilities to develop powerful and accurate visualization tools that can assist domain scientists with their requirements in multiple phases of scientific discovery. We have recently developed several feature-driven visualization methods for extracting different data characteristics of volumetric datasets. Our results verify the hypothesis in the proposal and will be used to develop additional prototype systems.

97 MATHEMATICS AND COMPUTING↗

Kepler + MeDICi

applications are often structured as workflows that execute a series of interdependent, distributed software modules to analyze large data sets. Passing very large data sets between workflow steps can incur unacceptable overheads with workflow engines that are not designed to efficiently handle large data transfers. In this paper, we describe how we have enhanced a Kepler workflow application with the MeDICi Integration Framework, which has been specifically designed to support the execution of data intensive workflows. The MeDICi technology provides a scalable, component-based architecture that efficiently handles integration with heterogeneous, distributed software systems. This paper describes the MeDICi Integration Framework and the mechanisms we used to integrate MeDICi components with Kepler workflow actors. We illustrate this solution with a workflow application for an atmospheric sciences application, which demonstrates the advantages of this approach.

97 MATHEMATICS AND COMPUTING↗

A Framework for International Collaboration on ITER Using Large-Scale Data Transfer to Enable Near-Real-Time Analysis

The global nature of the ITER project along with its projected ~ petabyte per day data generation presents a unique challenge, but also an opportunity for the fusion community to rethink, optimize, and enhance our scientific discovery process. Recognizing this, collaborative research with computational scientists was undertaken over the past several years to create a framework for large-scale data movement across wide-area networks (WANs), to enable global near-real time analysis of fusion data. This would broaden the available computational resources for analysis/simulation, and increase the number of researchers actively participating in experiments. An official demonstration of this framework for fast, large data transfer and real-time analysis was carried out between the KSTAR tokamak in Daejeon, Korea and PPPL in Princeton, USA. Streaming large data transfer, with near real-time movie creation and analysis of the KSTAR Electron Cyclotron Emission imaging (ECEI) data, was performed using the I/O framework ADIOS, and comparisons made at PPPL with simulation results from the XGC1 code. These demonstrations were made possible utilizing an optimized network configuration at PPPL, which achieved over 8.8 Gbps (88% utilization) in throughput tests from NFRI to PPPL. This demonstration showed the feasibility for large-scale data analysis of KSTAR data, and provides a nascent framework to enable use of globally distributed computational and personnel resources in pursuit of scientific knowledge from the ITER experiment.

43 PARTICLE ACCELERATORS↗

Large-Scale Data Analytics and Its Relationship to Simulation

Large-Scale Data Analytics (LSDA) problems require finding meaningful patterns in data sets that are so large as to require leading-edge processing and storage capability. LSDA problems are increasingly important for government mission work, industrial application, and scientific discovery. Effective solution of some important LSDA problems requires a computational workload that is substantially different from that associated with traditional High Performance Computing (HPC) simulations intended to help understand physical phenomena or to conduct engineering. While traditional HPC application codes exploit structural regularity and data locality to improve performance, many analytics problems lead more naturally to very fine-grained communication between unpredictable sets of processors, resulting in less regular communication patterns that do not map efficiently on to typical HPC systems. In both simulation and analytics domains, however, data movement increasingly dominates the performance, energy usage, and price of computing systems. It is therefore plausible that we could find a more synergistic technology path forward. Even though future machines may continue to be configured differently for the two domains, a more common technological roadmap between them in the form of a degree of convergence in the underlying componentry and design principles to address these common technical challenges could have substantial technical and economic benefits.

97 MATHEMATICS AND COMPUTING↗

Standard Energy Efficiency Data Platform

The SEED platform is expected to be a building energy performance data management tool that provides federal, state and local governments, building owners and operators with an easy, flexible and cost-effective method to collect information about groups of buildings, oversee compliance with energy disclosure laws and demonstrate the economic and environmental benefits of energy efficiency. It will allow users to leverage a local application to manage data disclosure and large data sets without the IT investment of developing custom applications. The first users of SEED will be agencies that need to collect, store, and report/share large data sets generated by benchmarking, energy auditing, retro-commissioning or retrofitting of many buildings. Similarly, building owners and operators will use SEED to manage their own energy data in a common format and centralized location. SEED users will also control the disclosure of their information for compliance requirements, recognition programs such as ENERGY STAR, or data sharing with the Buildings Performance Database and/or other third parties at their discretion.

Cheifetz, D.Magnus↗

Standard Energy Efficiency Data Platform

The SEED platform is expected to be a building energy performance data management tool that provides federal, state and local governments, building owners and operators with an easy, flexible and cost-effective method to collect information about groups of buildings, oversee compliance with energy disclosure laws and demonstrate the economic and environmental benefits of energy efficiency. It will allow users to leverage a local application to manage data disclosure and large data sets without the IT investment of developing custom applications. The first users of SEED will be agencies that need to collect, store, and report/share large data sets generated by benchmarking, energy auditing, retro-commissioning or retrofitting of many buildings. Similarly, building owners and operators will use SEED to manage their own energy data in a common format and centralized location. SEED users will also control the disclosure of their information for compliance requirements, recognition programs such as ENERGY STAR, or data sharing with the Buildings Performance Database and/or other third parties at their discretion.

Cheifetz, D. Magnus↗

Efficient Checkpointing of Virtual Machines using Virtual Machine Introspection

Cloud Computing environments rely heavily on system-level virtualization. This is due to the inherent benefits of virtualization including fault tolerance through checkpoint/restart (C/R) mechanisms. Because clouds are the abstraction of large data centers and large data centers have a higher potential for failure, it is imperative that a C/R mechanism for such an environment provide minimal latency as well as a small checkpoint file size. Recently, there has been much research into C/R with respect to virtual machines (VM) providing excellent solutions to reduce either checkpoint latency or checkpoint file size. However, these approaches do not provide both. This paper presents a method of checkpointing VMs by utilizing virtual machine introspection (VMI). Through the usage of VMI, we are able to determine which pages of memory within the guest are used or free and are better able to reduce the amount of pages written to disk during a checkpoint. We have validated this work by using various benchmarks to measure the latency along with the checkpoint size. With respect to checkpoint file size, our approach results in file sizes within 24% or less of the actual used memory within the guest. Additionally, the checkpoint latency of our approach is up to 52% faster than KVM s default method.

Aderholdt, Ferrol↗