Search NASA⌕ Search

SEARCH · Search NASA

Results for “DISTRIBUTED DATA PROCESSING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Distributed non-negative matrix factorization with determination of the number of latent features

The holistic analysis and understanding of the latent (that is, not directly observable) variables and patterns buried in large datasets is crucial for data-driven science, decision making and emergency response. Such exploratory analyses require devising unsupervised learning methods for data mining and extraction of the latent features, and non-negative matrix factorization (NMF) is one of the prominent such methods. NMF is based on compute-intense non-convex constrained minimization, which, for large datasets requires fast and distributed algorithms. However, current parallel implementations of NMF fail to estimate the number of latent features. In practice, identifying these features is both difficult and significant for pattern recognition and latent feature analysis, especially for large dense matrices. Here, we introduce a distributed NMF algorithm coupled with distributed custom clustering followed by a stability analysis on dense data, which we call DnMFk, to determine the number of latent variables. The results on synthetic data and the classical Swimmer data set demonstrate the accuracy of model determination while scaling nearly linearly across multiple processors for large data. Further, we employ DnMFk to determine the number of hidden features from a terabyte matrix.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Life Cycle Analysis of Natural Gas Supply Chain and End Use Applications in the United States

Natural gas (NG) plays a crucial role in current and future energy systems in the United States due to its abundance and affordability. In this study a life cycle analysis of the NG supply chain in the United States was conducted using Argonne's R&D GREET model, examining stages from recovery to distribution using reported field data processed and documented by National Energy Technology Laboratory. Supply chain emissions were evaluated across multiple spatial scales, including national average, overall regional production, region-to-region, and basin-to-region scenarios. The GHG intensity of the U.S. average NG supply chain was estimated at 10.3 kg CO 2 e/MMBtu (lower heating value), with a range across regions from 7.8 kg CO 2 e/MMBtu (Northeast) to 15.1 kg CO 2 e/MMBtu (Pacific). The analysis further assessed how upstream NG emissions influence the life cycle GHG emissions of key end-use applications, including electricity generation (0.044–0.086 kg CO 2 e/kWh from upstream NG in combined cycle facilities), hydrogen production (1.04–2.20 kg CO 2 e/kg H 2 for steam methane reforming [SMR] and 1.06–2.23 kg CO 2 e/kg for autothermal reforming [ATR]), and transit bus operation utilizing compressed natural gas fuel (0.19–0.37 kg CO 2 e/mile) and hydrogen fuel (0.12–0.25 kg CO 2 e/mile for hydrogen produced in SMR and ATR).

compression↗
DOE OSTI2019

PNNL-Comp-Mass-Spec/LIQUID

The OMICS group at PNNL has developed a wide range of software utilities over the last 15+ years. These tools are used to manage data files, transform data, visualize data, and automate data processing. The software is distributed on GitHub as open source, typically under the Apache 2.0 license agreement. See https://github.com/PNNL-Comp-Mass-Spec Lipid analysis tool developed by PNNL

Monroe, Matthew↗
DOE OSTI2017

A Dynamic Landsat Derived Normalized Difference Vegetation Index (NDVI) Product for the Conterminous United States

Satellite derived vegetation indices (VIs) are broadly used in ecological research, ecosystem modeling, and land surface monitoring. The Normalized Difference Vegetation Index (NDVI), perhaps the most utilized VI, has countless applications across ecology, forestry, agriculture, wildlife, biodiversity, and other disciplines. Calculating satellite derived NDVI is not always straight-forward, however, as satellite remote sensing datasets are inherently noisy due to cloud and atmospheric contamination, data processing failures, and instrument malfunction. Readily available NDVI products that account for these complexities are generally at coarse resolution; high resolution NDVI datasets are not conveniently accessible and developing them often presents numerous technical and methodological challenges. Here, we address this deficiency by producing a Landsat derived, high resolution (30 m), long-term (30+ years) NDVI dataset for the conterminous United States. We use Google Earth Engine, a planetary-scale cloud-based geospatial analysis platform, for processing the Landsat data and distributing the final dataset. We use a climatology driven approach to fill missing data and validate the dataset with established remote sensing products at multiple scales. We provide access to the composites through a simple web application, allowing users to customize key parameters appropriate for their application, question, and region of interest.

42 ENGINEERING↗
NASA NTRS1985

International Ultraviolet Explorer Observatory operations

This volume contains the final report for the International Ultraviolet Explorer IUE Observatory Operations contract. The fundamental operational objective of the International Ultraviolet Explorer (IUE) program is to translate competitively selected observing programs into IUE observations, to reduce these observations into meaningful scientific data, and then to present these data to the Guest Observer in a form amenable to the pursuit of scientific research. The IUE Observatory is the key to this objective since it is the central control and support facility for all science operations functions within the IUE Project. In carrying out the operation of this facility, a number of complex functions were provided beginning with telescope scheduling and operation, proceeding to data processing, and ending with data distribution and scientific data analysis. In support of these critical-path functions, a number of other significant activities were also provided, including scientific instrument calibration, systems analysis, and software support. Routine activities have been summarized briefly whenever possible.

Source record↗
NASA NTRS1995

Development and analysis of SCR requirements tables for system scenarios

We describe the use of scenarios to develop and refine requirement tables for parts of the Earth Observing System Data and Information System (EOSDIS). The National Aeronautics and Space Administration (NASA) is developing EOSDIS as part of its Mission-To-Planet-Earth (MTPE) project to accept instrument/platform observation requests from end-user scientists, schedule and perform requested observations of the Earth from space, collect and process the observed data, and distribute data to scientists and archives. Current requirements for the system are managed with tools that allow developers to trace the relationships between requirements and other development artifacts, including other requirements. In addition, the user community (e.g., earth and atmospheric scientists), in conjunction with NASA, has generated scenarios describing the actions of EOSDIS subsystems in response to user requests and other system activities. As part of a research effort in verification and validation techniques, this paper describes our efforts to develop requirements tables from these scenarios for the EOSDIS Core System (ECS). The tables specify event-driven mode transitions based on techniques developed by the Naval Research Lab's (NRL) Software Cost Reduction (SCR) project. The SCR approach has proven effective in specifying requirements for large systems in an unambiguous, terse format that enhance identification of incomplete and inconsistent requirements. We describe development of SCR tables from user scenarios and identify the strengths and weaknesses of our approach in contrast to the requirements tracing approach. We also evaluate the capabilities of both approach to respond to the volatility of requirements in large, complex systems.

Callahan, John R.↗
NASA NTRS2004

Research and Development in Very Long Baseline Interferometry (VLBI)

Contents include the following: 1.Observation coordination. 2. Data acquisition system control software. 3. Station support. 4. Correlation, data processing, and analysis. 5. Data distribution and archiving. 6. Technique improvement and research. 7. Computer support.

Himwich, William E.↗
NASA NTRS1997

ASF RADARSAT Processor System Performance Summary

The Alaska SAR Facility (ASF) located at the University of Alaska Fairbanks (UAF) has been in operations since 1991 serving as a key data acquisition, processing, archive and distribution center for a number of polar orbiting SAR (synthetic aperture radar) satelittes including ERS 1/2 and JERS-1.

synthetic aperture radar RADARSAT Mapping↗
NASA NTRS2019

Small Satellites for NASA Earth Science

NASA’s Earth Science Division (ESD) seeks to develop a scientific understanding of Earth and its response to natural and human-induced changes. Earth is a system comprised of diverse components interacting in complex ways. Understanding Earth’s atmosphere, surface and interior, oceans and surface water, ice and snow, and life as a single connected system is necessary in order to improve our predictions of climate, weather, and natural hazards. The ESD’s Flight Program consists of a coordinated series of satellite and airborne systems providing long and short-term, global and regional observations. In addition, the Flight Program provides infrastructure for operating these missions, processing their scientific data, and distributing them on a free and open basis to researchers, operational users, and the public. The Flight Program currently has 24 operating Earth observing space missions and instruments. There are 18 more missions and instruments planned for launch over the next five years. These comprise missions recommended by the National Academies 2017 Earth Science Decadal Survey, missions and selected instruments to ensure availability of key climate data sets, operational missions to sustain the land imaging provided by the Landsat system, and small-sized competitively selected orbital and instrument missions of opportunity belonging to the Earth Venture (EV) program. The Earth Science Decadal Survey, released in early 2018, recommended four new Flight Program elements in addition to the above activities that comprise the Program of Record (POR). Small satellites (~500 kg or less) are essential components of these activities. Presently, there is an increasing use of micro and nanosatellites (or CubeSats) in constellations to support NASA ESD’s scientific objectives. These include the Cyclone Global Navigation Satellite System (CYGNSS) for observing tropical cyclone intensification and genesis factors, the Timed-Resolved Observations of Precipitation structure and storm Intensity with a Constellation of Smallsats (TROPICS) mission, and the Polar Radiant Energy in the Far InfraRed Experiment (PREFIRE) CubeSat mission. ESD small satellite initiatives like the Small Satellite Constellation Data Buy and Venture Class Launch Services (VCLS) are also underway. The Earth Science Technology Office’s (ESTO) In-Space Validation of Earth Science Technologies (InVEST) and the Venture Technology program elements have launched seven 3U and 6U CubeSat missions to validate advanced instruments and related technologies. An equivalent number of InVEST and other technology demonstration CubeSats are being prepared for launch in the next year. An overview of plans and current status including topics related to small satellite enabling activities will be presented.

Neeck, Steven P.↗
DOE OSTI2025

SWARM: Reimagining scientific workflow management systems in a distributed world

Modern scientific workflows process massive amounts of data from diverse instruments and sensors, leveraging geographically distributed, heterogeneous compute and storage resources—from leadership-class systems to edge devices—connected by high-performance networks. The diversity of resources introduces challenges in harnessing their full potential, with resilience issues arising across applications, system software, networks, storage, and hardware. Today, workflow management systems (WMS) coordinate the execution of computation and data management tasks across target resources. However, WMS’s centralized nature makes them vulnerable to faults and scalability issues that may result in failures of entire computational campaigns. In conclusion, this paper introduces a novel agentic framework for workflow management, fully distributing and decentralizing the WMS functions and modeling them as swarm intelligence agents infused with advanced artificial intelligence solutions and traditional distributed computing algorithms that can make coordinated decisions in the presence of failures of the underlying cyberinfrastructure.

Swarm intelligence↗
NASA NTRS2001

The Role of Advanced Information System Technology in Remote Sensing for NASA's Earth Science Enterprise in the 21st Century

Future NASA Earth observing satellites will carry high-precision instruments capable of producing large amounts of scientific data. The strategy will be to network these instrument-laden satellites into a web-like array of sensors to facilitate the collection, processing, transmission, storage, and distribution of data and data products - the essential elements of what we refer to as "Information Technology." Many of these Information Technologies will enable the satellite and ground information systems to function effectively in real-time, providing scientists with the capability of customizing data collection activities on a satellite or group of satellites directly from the ground. In future systems, extremely large quantities of data collected by scientific instruments will require the fastest processors, the highest communication channel transfer rates, and the largest data storage capacity to insure that data flows smoothly from the satellite-based instrument to the ground-based archive. Autonomous systems will control all essential processes and play a key role in coordinating the data flow through space-based communication networks. In this paper, we will discuss those critical information technologies for Earth observing satellites that will support the next generation of space-based scientific measurements of planet Earth, and insure that data and data products provided by these systems will be accessible to scientists and the user community in general.

Prescott, Glenn↗
NASA NTRS2013

The GLAS Science Algorithm Software (GSAS) User's Guide Version 7

The Geoscience Laser Altimeter System (GLAS) is the primary instrument for the ICESat (Ice, Cloud and Land Elevation Satellite) laser altimetry mission. ICESat was the benchmark Earth Observing System (EOS) mission for measuring ice sheet mass balance, cloud and aerosol heights, as well as land topography and vegetation characteristics. From 2003 to 2009, the ICESat mission provided multi-year elevation data needed to determine ice sheet mass balance as well as cloud property information, especially for stratospheric clouds common over polar areas. It also provided topography and vegetation data around the globe, in addition to the polar-specific coverage over the Greenland and Antarctic ice sheets.This document is the final version of the GLAS Science Algorithm Software Users Guide document. It contains the instructions to install the GLAS Science Algorithm Software (GSAS) in the production environment that was used to create the standard data products. It also describes the usage of each GSAS program in that environment with their required inputs and outputs. Included are a number of utility programs that are used to create ancillary data files that are used in the processing but generally are not distributed to the public as data products. Of importance is the values for the large number of constants used in the GSAS algorithm during processing are provided in an appendix.

Lee, Jeffrey E.↗
NASA NTRS1977

UH cosmic rays: Possible origin in massive stars

The origin of the Z greater than 28, ultraheavy, cosmic rays in supernova explosions of massive stars is considered. For Z greater than 70, the UH data is dominated by an r-process source distribution, but for the elements just beyond iron, 29 or = Z less than 36, the data cannot be explained by any single process of nucleosynthesis. This problem is solved naturally in a massive star model by secondary neutron capture reactions occuring during core helium burning and during explosive carbon burning. Interstellar propagation calculations were performed with these episodes of synthesis as source distributions, and the results offer an explanation for the current UH cosmic-ray data. The heavy element synthesis during explosive carbon burning is reexamined using more realistic initial conditions given by the post-helium-burning configuration of the star. Effects of preferential acceleration are considered, and experimental tests are discussed.

Wefel, J. P.↗
NASA NTRS2010

Reducing the Volume of NASA Earth-Science Data

A computer program reduces data generated by NASA Earth-science missions into representative clusters characterized by centroids and membership information, thereby reducing the large volume of data to a level more amenable to analysis. The program effects an autonomous data-reduction/clustering process to produce a representative distribution and joint relationships of the data, without assuming a specific type of distribution and relationship and without resorting to domain-specific knowledge about the data. The program implements a combination of a data-reduction algorithm known as the entropy-constrained vector quantization (ECVQ) and an optimization algorithm known as the differential evolution (DE). The combination of algorithms generates the Pareto front of clustering solutions that presents the compromise between the quality of the reduced data and the degree of reduction. Similar prior data-reduction computer programs utilize only a clustering algorithm, the parameters of which are tuned manually by users. In the present program, autonomous optimization of the parameters by means of the DE supplants the manual tuning of the parameters. Thus, the program determines the best set of clustering solutions without human intervention.

Lee, Seungwon↗
NASA NTRS2017

Lessons Learned While Exploring Cloud-Native Architectures for NASA EOSDIS Applications and Systems

NASA's Earth Observing System (EOS) is a coordinated series of satellites for long term global observations. NASA's Earth Observing System Data and Information System (EOSDIS) is a multi-petabyte-scale archive of environmental data that supports global climate change research by providing end-to-end services from EOS instrument data collection to science data processing to full access to EOS and other earth science data. On a daily basis, the EOSDIS ingests, processes, archives and distributes over 3 terabytes of data from NASA's Earth Science missions representing over 6000 data products ranging from various types of science disciplines. EOSDIS has continually evolved to improve the discoverability, accessibility, and usability of high-impact NASA data spanning the multi-petabyte-scale archive of Earth science data products. Reviewed and approved by Chris Lynnes.

Cumulus↗
NASA NTRS2012

IT Data Mining Tool Uses in Aerospace

Data mining has a broad spectrum of uses throughout the realms of aerospace and information technology. Each of these areas has useful methods for processing, distributing, and storing its corresponding data. This paper focuses on ways to leverage the data mining tools and resources used in NASA's information technology area to meet the similar data mining needs of aviation and aerospace domains. This paper details the searching, alerting, reporting, and application functionalities of the Splunk system, used by NASA's Security Operations Center (SOC), and their potential shared solutions to address aircraft and spacecraft flight and ground systems data mining requirements. This paper also touches on capacity and security requirements when addressing sizeable amounts of data across a large data infrastructure.

Monroe, Gilena A.↗