Search NASA⌕ Search

SEARCH · Search NASA

Results for “DISTRIBUTED DATA PROCESSING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17
All sourcesNASA NTRSDOE OSTI
NASA NTRS1996

An MPI-IO Interface to HPSS

This paper describes an implementation of the proposed MPI-IO standard for parallel I/O. Our system uses third-party transfer to move data over an external network between the processors where it is used and the I/O devices where it resides. Data travels directly from source to destination, without the need for shuffling it among processors or funneling it through a central node. Our distributed server model lets multiple compute nodes share the burden of coordinating data transfers. The system is built on the High Performance Storage System (HPSS), and a prototype version runs on a Meiko CS-2 parallel computer.

Parallel Processing↗
DOE OSTI2015

Deactivation and decommissioning web log analysis using big data technology - 15710

The D and D Knowledge Management Information Tool (D and D KM- IT) is a web-based knowledge management information tool built for the D and D user community. Big data is a massive volume of structured and unstructured datasets which is so large that it's difficult to process using traditional database techniques. Web logs are the repository of files which are generated automatically for any operation on the web site. Web log files generated from the D and D KM-IT will be processed using the Apache Hadoop Framework to extract meaningful data. Hadoop is an open-source software framework for storing and processing big data in a distributed environment on large clusters of commodity hardware. Hadoop framework consists of two main layers: Hadoop distributed file system (HDFS) and the Execution engine (MapReduce). Automatic parallelization and distribution; clean and simple programming abstraction. Implementation: Web log files from the D and D KM-IT server are fetched using Apache Flume and loaded into the Hadoop distributed file system. Using the Apache Hcatalog tool, the web log data is parsed and converted into a structured table format. A Pig program is developed to query the tables to retrieve and store the significant information onto the HDFS. The extracted data is fed to business visualization tools such as Microsoft Excel for analysis and reporting. Results The reports generated through the web log data processing will be analyzed to improve the usability and performance of the D and D KM-IT. Conclusion/Future Work: User keyword information can be explored to add new trending topics to the D and D KM-IT application. Statistics generated can be compared to analysis reports from existing analytical tools.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
NASA NTRS2012

Content Platforms Meet Data Storage, Retrieval Needs

Earth is under a constant barrage of information from space. Whether from satellites orbiting our planet, spacecraft circling Mars, or probes streaking toward the far reaches of the Solar System, NASA collects massive amounts of data from its spacefaring missions each day. NASA s Earth Observing System (EOS) satellites, for example, provide daily imagery and measurements of Earth s atmosphere, oceans, vegetation, and more. The Earth Observing System Data and Information System (EOSDIS) collects all of that science data and processes, archives, and distributes it to researchers around the globe; EOSDIS recently reached a total archive volume of 4.5 petabytes. Try to store that amount of information in your standard, four-drawer file cabinet, and you would need 90 million to get the job done. To manage the flood of information, NASA has explored technologies to efficiently collect, archive, and provide access to EOS data for scientists today and for years to come. One such technology is now providing similar capabilities to businesses and organizations worldwide.

Source record↗
NASA NTRS1983

Upper Atmosphere Research Satellite (UARS) trade analysis

The Upper Atmosphere Research Satellite (UARS) which will collect data pertinent to the Earth's upper atmosphere is described. The collected data will be sent to the central data handling facility (CDHF) via the UARS ground system and the data will be processed and distributed to the remote analysis computer systems (RACS). An overview of the UARS ground system is presented. Three configurations were developed for the CDHF-RACS system. The CDHF configurations are discussed. The IBM CDHF configuration, the UNIVAC CDHF configuration and the vax cluster CDHF configuration are presented. The RACS configurations, the IBM RACS configurations, UNIVAC RACS and VAX RACS are detailed. Due to the large on-line data estimate to approximately 100 GB, a mass storage system is considered essential to the UARS CDHF. Mass storage systems were analyzed and the Braegan ATL, the RCA optical disk, the IBM 3850 and the MASSTOR M860 are discussed. It is determined that the type of mass storage system most suitable to UARS is the automated tape/cartridge device. Two devices of this type, the IBM 3850 and the MASSTOR MSS are analyzed and the applicable tape/cartridge device is incorporated into the three CDHF-RACS configurations.

Fox, M. M.↗
DOE OSTI2018

Keyless Infrastructure Security Solution (KISS): VOLTTRON™ KSI ® Blockchain Design and Specification

This document provides the technical design specifications that are required for Keyless Infrastructure Security Solution (KISS) development, including, definitions and roles of the new KISS-related products—GridAware and LinkLite—an overview of the DNP3 communications the VOLTTRON™ DNP3 agent is using for distribution control, data flow, historian processing, setup, and installation. These technical specifications provide step-by-step instructions for configuration, implementation, and deployment of the KISS subsystem, docket creation, and verification using the GridAware dashboard. Based on this document and the KISS specification requirements and device risk assessment documents, KISS will be developed by integrating keyless signature infrastructure (KSI) blockchain to increase the cybersecurity, integrity, and trustworthiness of critical energy delivery systems (EDSs). The KISS architecture consists of a VOLTTRON™ agent-based distribution control system platform and GridAware (KSI blockchain capability) for secure energy exchanges in a decentralized electrical network.

42 ENGINEERING↗
DOE OSTI2025

Preparation of the Multi-Site Data Processing at the Vera C. Rubin Observatory

The Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST) Camera is scheduled to start taking data in the summer of 2025. The Data Release Production will run the LSST Science Pipe software at data facilities in the US, France and the UK. The LSST Science Pipeline consists of complex directed acyclic graphs (DAGs) of tasks. Rubin will use the Production and Distributed Analysis (PanDA) workflow and workload management system to orchestrate this complex workflow and the distribution of workloads to the data facilities. When run end-to-end by a team of data production staff, this processing (the Science Pipelines, distributed by the workflow and workload management system) is referred to as a 'campaign'. This paper describes the central services and data facility specific services that support this multi-site data process model, including the service deployment infrastructure, the workload and workflow system, the Campaign Management tools, and connection to Rubin Data Management. This paper will also mention the experience of processing the Rubin Commissioning Camera data. All these are part of the effort to scale up the processing capabilities for the expected very large data volume from the LSST Camera.

Yang, Wei [SLAC]↗
NASA NTRS1977

Instrumentation complex for Langley Research Center's National Transonic Facility

The instrumentation discussed in the present paper was developed to ensure reliable operation for a 2.5-meter cryogenic high-Reynolds-number fan-driven transonic wind tunnel. It will incorporate four CPU's and associated analog and digital input/output equipment, necessary for acquiring research data, controlling the tunnel parameters, and monitoring the process conditions. Connected in a multipoint distributed network, the CPU's will support data base management and processing; research measurement data acquisition and display; process monitoring; and communication control. The design will allow essential processes to continue, in the case of major hardware failures, by switching input/output equipment to alternate CPU's and by eliminating nonessential functions. It will also permit software modularization by CPU activity and thereby reduce complexity and development time.

Russell, C. H.↗
DOE OSTI2012

Extra Dimensions: 3D and Time in PDF Documentation

Experimental science is replete with multi-dimensional information which is often poorly represented by the two dimensions of presentation slides and print media. Past efforts to disseminate such information to a wider audience have failed for a number of reasons, including a lack of standards which are easy to implement and have broad support. Adobe's Portable Document Format (PDF) has in recent years become the de facto standard for secure, dependable electronic information exchange. It has done so by creating an open format, providing support for multiple platforms and being reliable and extensible. By providing support for the ECMA standard Universal 3D (U3D) file format in its free Adobe Reader software, Adobe has made it easy to distribute and interact with 3D content. By providing support for scripting and animation, temporal data can also be easily distributed to a wide, non-technical audience. We discuss how the field of radiation imaging could benefit from incorporating full 3D information about not only the detectors, but also the results of the experimental analyses, in its electronic publications. In this article, we present examples drawn from high-energy physics, mathematics and molecular biology which take advantage of this functionality. We demonstrate how 3D detector elements can be documented, using either CAD drawings or other sources such as GEANT visualizations as input.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
NASA NTRS1989

Plate motions and deformations from geologic and geodetic data

The very long baseline interferometry (VLBI) measurements made in the western U.S. since 1979 provide discrete samples of the temporal and spatial deformation field. The interpretation of the VLBI derived rates of deformation requires an examination of geologic information and more densely sampled ground based geodetic data. Triangulation and trilateration data measured on two regional networks, one in the central Mojave Desert and one in the Coast Ranges east of the San Andreas fault, were processed. At the spatial scales spanned by these local geodetic networks, auxiliary geologic and geophysical data were utilized to examine the relation between measured incremental strain and the accommodation of strain seen in local geologic structures, strain release in earthquakes, and principal stress directions inferred from in situ measurements. VLBI data was also processed from stations distributed across the Pacific-North America plate boundary zone in the western U.S. The VLBI data were used to constrain the integrated rate of deformation across portions of the continental plate boundary in California and to provide a tectonic framework to interpret regional geodetic and geologic studies.

Jordan, Thomas H.↗
DOE OSTI2015

Distributed computing strategies for processing of FT-ICR MS imaging datasets for continuous mode data visualization

High-resolution Fourier transform ion cyclotron resonance (FT-ICR) mass spectrometry imaging enables the spatial mapping and identification of biomolecules from complex surfaces. The need for long time-domain transients, and thus large raw file sizes, results in a large amount of raw data (“big data”) that must be processed efficiently and rapidly. This can be compounded by largearea imaging and/or high spatial resolution imaging. For FT-ICR, data processing and data reduction must not compromise the high mass resolution afforded by the mass spectrometer. The continuous mode “Mosaic Datacube” approach allows high mass resolution visualization (0.001 Da) of mass spectrometry imaging data, but requires additional processing as compared to featurebased processing. We describe the use of distributed computing for processing of FT-ICR MS imaging datasets with generation of continuous mode Mosaic Datacubes for high mass resolution visualization. An eight-fold improvement in processing time is demonstrated using a Dutch nationally available cloud service.

Environmental Molecular Sciences Laboratory↗
DOE OSTI2024

Deep kernel methods learn better: from cards to process optimization

Abstract The ability of deep learning methods to perform classification and regression tasks relies heavily on their capacity to uncover manifolds in high-dimensional data spaces and project them into low-dimensional representation spaces. In this study, we investigate the structure and character of the manifolds generated by classical variational autoencoder (VAE) approaches and deep kernel learning (DKL). In the former case, the structure of the latent space is determined by the properties of the input data alone, while in the latter, the latent manifold forms as a result of an active learning process that balances the data distribution and target functionalities. We show that DKL with active learning can produce a more compact and smooth latent space which is more conducive to optimization compared to previously reported methods, such as the VAE. We demonstrate this behavior using a simple cards dataset and extend it to the optimization of domain-generated trajectories in physical systems. Our findings suggest that latent manifolds constructed through active learning have a more beneficial structure for optimization problems, especially in feature-rich target-poor scenarios that are common in domain sciences, such as materials synthesis, energy storage, and molecular discovery. The Jupyter Notebooks that encapsulate the complete analysis accompany the article.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1979

Information adaptive system of NEEDS

The NASA End-to-End Data System (NEEDS) program was initiated by NASA to improve significantly the state of the art in acquisition, processing, and distribution of space-acquired data for the mid-1980s and beyond. The information adaptive system (IAS) is a program element under NEEDS Phase II which addresses sensor specific processing on board the spacecraft. The IAS program is a logical first step toward smart sensors, and IAS developments - particularly the system components and key technology improvements - are applicable to future smart efforts. The paper describes the design goals and functional elements of the IAS. In addition, the schedule for IAS development and demonstration is discussed.

Howle, W. M., Jr.↗
NASA NTRS2017

GES DISC Data Recipes in Jupyter Notebooks

The Earth Science Data and Information System (ESDIS) Project manages twelve Distributed Active Archive Centers (DAACs) which are geographically dispersed across the United States. The DAACs are responsible for ingesting, processing, archiving, and distributing Earth science data produced from various sources (satellites, aircraft, field measurements, etc.). In response to projections of an exponential increase in data production, there has been a recent effort to prototype various DAAC activities in the cloud computing environment. This, in turn, led to the creation of an initiative, called the Cloud Analysis Toolkit to Enable Earth Science (CATEES), to develop a Python software package in order to transition Earth science data processing to the cloud. This project, in particular, supports CATEES and has two primary goals. One, to transition data recipes created by the Goddard Earth Science Data and Information Service Center (GES DISC) into an interactive and educational environment using JupyterNotebooks. Two, to acclimate Earth scientists to cloud computing. To accomplish these goals, we create JupyterNotebooks to compartmentalize the different steps of data analysis and help users obtain and parse data from the command line. We also develop a Docker container, comprised of Jupyter Notebooks, Python dependencies, and command line tools, and configure it into an easy-to-deploy package. The end result is an end-to-end product that simulates the use case of end users working in the cloud computing environment.

discoverability↗
DOE OSTI2024

Density estimation via measure transport: Outlook for applications in the biological sciences

Abstract One among several advantages of measure transport methods is that they allow or a unified framework for processing and analysis of data distributed according to a wide class of probability measures. Within this context, we present results from computational studies aimed at assessing the potential of measure transport techniques, specifically, the use of triangular transport maps, as part of a workflow intended to support research in the biological sciences. Scenarios characterized by the availability of limited amount of sample data, which are common in domains such as radiation biology, are of particular interest. We find that when estimating a distribution density function given limited amount of sample data, adaptive transport maps are advantageous. In particular, statistics gathered from computing series of adaptive transport maps, trained on a series of randomly chosen subsets of the set of available data samples, leads to uncovering information hidden in the data. As a result, in the radiation biology application considered here, this approach provides a tool for generating hypotheses about gene relationships and their dynamics under radiation exposure.

gene expression data↗
DOE OSTI2015

The creation of a comprehensive metocean data set for offshore wind turbine simulations: Comprehensive metocean data set

A database of meteorological and ocean conditions is presented for use in offshore wind energy research and design. The original data are from 23 ocean sites around the USA and were obtained from the National Data Buoy Center run by the National Oceanic and Atmospheric Administration. The data are presented in a processed form that includes the variables of interest for offshore wind energy design: wind speed, significant wave height, wave peak-spectral period, wind direction and wave direction. For each site, a binning process is conducted to create conditional probability functions for each of these variables. The sites are then grouped according to geographic location and combined to create three representative sites, including a West Coast site, an East Coast site and a Gulf of Mexico site. Both the processed data and the probability distribution parameters for the individual and representative sites are being hosted on a publicly available domain by the National Renewable Energy Laboratory, with the intent of providing a standard basis of comparison for meteorological and ocean conditions for offshore wind energy research worldwide.

17 WIND ENERGY↗
DOE OSTI2020

Distributed non-negative matrix factorization with determination of the number of latent features

The holistic analysis and understanding of the latent (that is, not directly observable) variables and patterns buried in large datasets is crucial for data-driven science, decision making and emergency response. Such exploratory analyses require devising unsupervised learning methods for data mining and extraction of the latent features, and non-negative matrix factorization (NMF) is one of the prominent such methods. NMF is based on compute-intense non-convex constrained minimization, which, for large datasets requires fast and distributed algorithms. However, current parallel implementations of NMF fail to estimate the number of latent features. In practice, identifying these features is both difficult and significant for pattern recognition and latent feature analysis, especially for large dense matrices. Here, we introduce a distributed NMF algorithm coupled with distributed custom clustering followed by a stability analysis on dense data, which we call DnMFk, to determine the number of latent variables. The results on synthetic data and the classical Swimmer data set demonstrate the accuracy of model determination while scaling nearly linearly across multiple processors for large data. Further, we employ DnMFk to determine the number of hidden features from a terabyte matrix.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Life Cycle Analysis of Natural Gas Supply Chain and End Use Applications in the United States

Natural gas (NG) plays a crucial role in current and future energy systems in the United States due to its abundance and affordability. In this study a life cycle analysis of the NG supply chain in the United States was conducted using Argonne's R&D GREET model, examining stages from recovery to distribution using reported field data processed and documented by National Energy Technology Laboratory. Supply chain emissions were evaluated across multiple spatial scales, including national average, overall regional production, region-to-region, and basin-to-region scenarios. The GHG intensity of the U.S. average NG supply chain was estimated at 10.3 kg CO 2 e/MMBtu (lower heating value), with a range across regions from 7.8 kg CO 2 e/MMBtu (Northeast) to 15.1 kg CO 2 e/MMBtu (Pacific). The analysis further assessed how upstream NG emissions influence the life cycle GHG emissions of key end-use applications, including electricity generation (0.044–0.086 kg CO 2 e/kWh from upstream NG in combined cycle facilities), hydrogen production (1.04–2.20 kg CO 2 e/kg H 2 for steam methane reforming [SMR] and 1.06–2.23 kg CO 2 e/kg for autothermal reforming [ATR]), and transit bus operation utilizing compressed natural gas fuel (0.19–0.37 kg CO 2 e/mile) and hydrogen fuel (0.12–0.25 kg CO 2 e/mile for hydrogen produced in SMR and ATR).

compression↗