Search NASA⌕ Search

SEARCH · Search NASA

Results for “cloud computing, data cloud”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

iDDS: intelligent distributed dispatch and scheduling for workflow orchestration

The intelligent distributed dispatch and scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the large hadron collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration. This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the electron–ion collider. By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2012

Enabling Earth Science Through Cloud Computing

Cloud Computing holds tremendous potential for missions across the National Aeronautics and Space Administration. Several flight missions are already benefiting from an investment in cloud computing for mission critical pipelines and services through faster processing time, higher availability, and drastically lower costs available on cloud systems. However, these processes do not currently extend to general scientific algorithms relevant to earth science missions. The members of the Airborne Cloud Computing Environment task at the Jet Propulsion Laboratory have worked closely with the Carbon in Arctic Reservoirs Vulnerability Experiment (CARVE) mission to integrate cloud computing into their science data processing pipeline. This paper details the efforts involved in deploying a science data system for the CARVE mission, evaluating and integrating cloud computing solutions with the system and porting their science algorithms for execution in a cloud environment.

science data system↗
DOE OSTI2021

Rahasak—Scalable blockchain architecture for enterprise applications

Blockchain-based decentralized infrastructure has been adapted in various industries to handle the sensitive data in a privacy-preserving manner without trusting third parties. However, integrating state-of-the-art blockchain platforms with the scalable, enterprise-level applications result in several challenges. Current blockchain platforms do not support high transaction throughput, lack high scalability, and cannot provide real-time transaction processing and back-pressure operation handling in high transaction throughput applications(e.g Big data, IoT). In this paper, we propose a novel permissioned blockchain platform “Rahasak” for highly scalable, enterprise applications. Rahasak blockchain adopts the Apache Kafka-based consensus on top of a “Validate-Execute-Group” blockchain architecture to handle realtime transaction execution on the blockchain. The architecture is equipped with a functional programming and actor-based smart contract platform that enables concurrent execution of transactions in the blockchain. Rahasak supports high transaction throughput, high scalability, concurrent transaction execution, data analytics features. Finally, with Rahasak, we make blockchain more scalable, secure, structured and meaningful for further data analytics.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2018

NASA's Earth Observing Data and Information System - Near-Term Challenges

NASA's Earth Observing System Data and Information System (EOSDIS) has been a central component of the NASA Earth observation program since the 1990's. EOSDIS manages data covering a wide range of Earth science disciplines including cryosphere, land cover change, polar processes, field campaigns, ocean surface, digital elevation, atmosphere dynamics and composition, and inter-disciplinary research, and many others. One of the key components of EOSDIS is a set of twelve discipline-based Distributed Active Archive Centers (DAACs) distributed across the United States. Managed by NASA's Earth Science Data and Information System (ESDIS) Project at Goddard Space Flight Center, these DAACs serve over 3 million users globally. The ESDIS Project provides the infrastructure support for EOSDIS, which includes other components such as the Science Investigator-led Processing systems (SIPS), common metadata and metrics management systems, specialized network systems, standards management, and centralized support for use of commercial cloud capabilities. Given the long-term requirements, and the rapid pace of information technology and changing expectations of the user community, EOSDIS has evolved continually over the past three decades. However, many challenges remain. Challenges addressed in this paper include: growing volume and variety, achieving consistency across a diverse set of data producers, managing information about a large number of datasets, migration to a cloud computing environment, optimizing data discovery and access, incorporating user feedback from a diverse community, keeping metadata updated as data collections grow and age, and ensuring that all the content needed for understanding datasets by future users is identified and preserved.

Remote Sensing↗
NASA NTRSDate not supplied

The Future of NASA Earth Science in the Commercial Cloud: Challenges and Opportunities

NASA produces a large volume and variety of data products that are used every day to support research, decision making, and education. The widespread use of NASA’s Earth Science data is enabled by NASA’s Earth Science Data System (ESDS) program, which oversees the archiving and distribution of these data and invests in the development of new data systems and tools. However, NASA’s current approach to Earth Science data distribution — based on distributed institutional archives with individual on-premises high-performance computing capabilities — faces some significant challenges, including massive increases in data volume from upcoming missions, a greater need for transdisciplinary science that synthesizes many different kinds of observations, and a push to make science more open, inclusive, and accessible. To address these challenges, NASA is aggressively migrating its Earth Science data and related tools and services into the commercial cloud. Migration of data into the commercial cloud can significantly improve NASA’s existing data system capabilities by (1) providing more flexible options for storage and compute (including rapid, as-needed access to state-of-the-art capabilities); (2) by centralizing and standardizing data access, which gives all of NASA’s institutional data centers access to all of each other’s datasets; and (3) by facilitating “analysis-in-place”, whereby users can bring their own computational workflows and tools to the data rather than having to maintain their own copies of NASA datasets. However, migration to the commercial cloud also poses some significant challenges, including (1) managing costs under a “pay-as-you-go” model; (2) incompatibility with existing tools and data formats with object-based storage and network access; (3) vendor lock-in; (4) challenges with data access for workflows that mix on-premise and cloud computing; and (5) standardization for highly diverse data as is present in NASA’s data archive. I conclude with two examples of recent NASA activities showcasing capabilities enabled by the commercial cloud: An interactive analysis and development platform for analyzing airborne imaging spectroscopy data, and a new collection of tools and services for data discovery, analysis, publication, and data-driven storytelling (Visualization, Exploration, and Data Analysis, VEDA).

Alexey N Shiklomanov↗
DOE OSTI2016

Impact of Data Placement on Resilience in Large-Scale Object Storage Systems

Distributed object storage architectures have become the de facto standard for high-performance storage in big data, cloud, and HPC computing. Object storage deployments using commodity hardware to reduce costs often employ object replication as a method to achieve data resilience. Repairing object replicas after failure is a daunting task for systems with thousands of servers and billions of objects, however, and it is increasingly difficult to evaluate such scenarios at scale on realworld systems. Resilience and availability are both compromised if objects are not repaired in a timely manner. In this work we leverage a high-fidelity discrete-event simulation model to investigate replica reconstruction on large-scale object storage systems with thousands of servers, billions of objects, and petabytes of data. We evaluate the behavior of CRUSH, a well-known object placement algorithm, and identify configuration scenarios in which aggregate rebuild performance is constrained by object placement policies. After determining the root cause of this bottleneck, we then propose enhancements to CRUSH and the usage policies atop it to enable scalable replica reconstruction. We use these methods to demonstrate a simulated aggregate rebuild rate of 410 GiB/s (within 5% of projected ideal linear scaling) on a 1,024-node commodity storage system. We also uncover an unexpected phenomenon in rebuild performance based on the characteristics of the data stored on the system.

Carns, Philip↗
NASA NTRS2011

Leveraging the Cloud for Robust and Efficient Lunar Image Processing

The Lunar Mapping and Modeling Project (LMMP) is tasked to aggregate lunar data, from the Apollo era to the latest instruments on the LRO spacecraft, into a central repository accessible by scientists and the general public. A critical function of this task is to provide users with the best solution for browsing the vast amounts of imagery available. The image files LMMP manages range from a few gigabytes to hundreds of gigabytes in size with new data arriving every day. Despite this ever-increasing amount of data, LMMP must make the data readily available in a timely manner for users to view and analyze. This is accomplished by tiling large images into smaller images using Hadoop, a distributed computing software platform implementation of the MapReduce framework, running on a small cluster of machines locally. Additionally, the software is implemented to use Amazon's Elastic Compute Cloud (EC2) facility. We also developed a hybrid solution to serve images to users by leveraging cloud storage using Amazon's Simple Storage Service (S3) for public data while keeping private information on our own data servers. By using Cloud Computing, we improve upon our local solution by reducing the need to manage our own hardware and computing infrastructure, thereby reducing costs. Further, by using a hybrid of local and cloud storage, we are able to provide data to our users more efficiently and securely. 12 This paper examines the use of a distributed approach with Hadoop to tile images, an approach that provides significant improvements in image processing time, from hours to minutes. This paper describes the constraints imposed on the solution and the resulting techniques developed for the hybrid solution of a customized Hadoop infrastructure over local and cloud resources in managing this ever-growing data set. It examines the performance trade-offs of using the more plentiful resources of the cloud, such as those provided by S3, against the bandwidth limitations such use encounters with remote resources. As part of this discussion this paper will outline some of the technologies employed, the reasons for their selection, the resulting performance metrics and the direction the project is headed based upon the demonstrated capabilities thus far.

Cloud Computing↗
DOE OSTI2011

Towards Composing Data Aware Systems Biology Workflows on Cloud Platforms: A MeDICi-based Approach

Cloud computing is being increasingly adopted for deploying systems biology scientific workflows. Scientists developing these workflows use a wide variety of fragmented and competing data sets and computational tools of all scales to support their research. To this end, the synergy of client side workflow tools with cloud platforms is a promising approach to share and reuse data and workflows. In such systems, the location of data and computation is essential consideration in terms of quality of service for composing a scientific workflow across remote cloud platforms. In this paper, we describe a cloud-based workflow for genome annotation processing that is underpinned by MeDICi - a middleware designed for data intensive scientific applications. The workflow implementation incorporates an execution layer for exploiting data locality that routes the workflow requests to the processing steps that are colocated with the data. We demonstrate our approach by composing two workflowswith the MeDICi pipelines.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2019

Data automation at light sources

Rapidly growing data volumes at light sources demand increasingly automated data collection, distribution, and analysis processes, in order to enable new scientific discoveries while not overwhelming finite human capabilities. We present here the case for automating and outsourcing light source science using cloud-hosted data automation and enrichment services, institutional computing resources, and high- performance computing facilities to provide cost-effective, scalable, and reliable implementations of such processes. We discuss three specific services that accomplish these goals for data distribution, automation, and transformation. In the first, Globus cloud-hosted data automation services are used to implement data capture, distribution, and analysis workflows for Advanced Photon Source and Advanced Light Source beamlines, leveraging institutional storage and computing. In the second, such services are combined with cloud-hosted data indexing and institutional storage to create a collaborative data publication, indexing, and discovery service, the Materials Data Facility (MDF), built to support a host of informatics applications in materials science. The third integrates components of the previous two projects with machine learning capabilities provided by the Data and Learning Hub for science (DLHub) to enable on-demand access to machine learning models from light source data capture and analysis workflows, and provides simplified interfaces to train new models on data from sources such as MDF on leadership scale computing resources. We draw conclusions about best practices for building next-generation data automation systems for future light sources.

Blaiszik, Ben↗
NASA NTRSDate not supplied

Hydrological Data at the NASA GES DISC: Current Capabilities and New Opportunities

The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is one of twelve NASA Earth science data centers that document, process, archive and distribute data from Earth observation missions and projects. GES DISC maintains an archive of several hydrology datasets, including the Land Data Assimilation Systems (LDAS) and the Gravity Recovery and Climate Experiment (GRACE) Data Assimilation for Drought Monitoring (GRACE-DA-DM) data products. These datasets include model output of heat fluxes, rain, snow, soil temperature, soil moisture, and runoff; and observational forcing data, including surface pressure, temperature, precipitation, downward shortwave and longwave radiation, humidity, and wind. The temporal resolution of the hydrology data at GES DISC ranges from hourly to monthly, and spatial resolutions range from 0.1° to 1.0°. The GES DISC provides services which enable users to aggregate, temporally and spatially subset, regrid, and visualize archived data including the GES DISC Subsetter, Hydrology Data Rods, and the Geospatial Interactive Online Visualization and Analysis Infrastructure (GIOVANNI). The Hydrology Data Rods service optimally reorganizes large hydrological data sets as extended time series, providing more efficient access for the hydrological community. The time series data (aka “data rods”) were integrated into hydrology community tools, such as the Data Rods Explorer on HydroShare. Furthermore, the GES DISC is in the process of migrating its data and services to the cloud. Hydrological data available at the GES DISC are now available in the Amazon Web Services (AWS) cloud (us-west-2 region) providing users Direct S3 data access and the capability for cloud computing operations. In this presentation, the hydrology data products and services currently available at the GES DISC will be summarized. Also discussed are the migration to the cloud, user support through this transition, and the status of migrating the data rods service to the cloud.

Ashley Heath↗
DOE OSTI2024

Assessment of Cloud-based Applications for Enabling a Scalable Riskinformed Predictive Maintenance Strategy

The current light-water reactor fleet uses time-based maintenance strategies to achieve high-capacity factors. But to make nuclear more competitive in the energy market, these reactors could utilize emerging artificial intelligence (AI) and cloud computing technologies to achieve a cost-effective, predictive-maintenance strategy. This paper presents discussion and results on the application of cloud computing in the nuclear industry. The technical viability of cloud computing was analyzed using data from a boiling-water reactor’s safety relief valve. The models were hosted on three different systems: a local personal computer, Idaho National Laboratory’s high-performance computer system, and Microsoft Azure. The data were loaded and processed, and two types of models were trained in an A/B fashion. Based on the speed at which these actions were completed, it was determined that cloud computing affords adequate computing resources. Additionally, the computing power can scale with the demanded load. To enable cloud computing in the existing fleet, additional sensors, networks, and other requirements must be implemented to ensure a smooth transition from current maintenance strategies. However, the benefit is that the plants no longer need to manage their own servers, software, cybersecurity, and information technology support staff for in-house data analytics purpose. Many of these features can be offloaded to the cloud provider for a potential cost savings. Demonstrating how AI can improve the maintenance and operation of non-safety-related systems seems the likely path forward for implementing AI and cloud computing resources inside nuclear power plants.

azure↗
DOE OSTI2015

Intermolecular Network Theory:A General Approach for Understanding the Structural and Dynamic Properties of Liquids and Solutions

Graph theory applications within the physical sciences have a long history. However, it has never moved to the forefront of analytical techniques employed by the computational chemistry community. Though much of the earliest work leveraged percolation theory as a backdrop for understanding critical phenomena and phase transitions within statistical mechanical simulations, modern data science has much to contribute. Here in this paper, we focus on the realm of intermolecular networks, where vertices represent molecules or particles, and an edge represents an intermolecular or interparticle interaction. These interactions are widely understood to dictate many of the physical properties of soft matter, and for liquids in particular, understanding this network is essential. Consider the sheer volume of ~70 years of literature associated with the hydrogen bond network of water, a topic that remains an active area of research even today. The aim of this work is twofold: first, to put into context prior algorithms and analyses that employed “connectedness theory” or lattice-based models to understand the intermolecular networks of chemical systems; and second to discuss a more general strategy for analyzing the multiscale structural and dynamic properties of chemical systems—intermolecular network theory. This approach encompasses foundational methods such as percolation theory, but also utilizes contemporary graph theoretical analyses that have evolved alongside the development of the World Wide Web, cloud computing, and big data. It is the realm of intermolecular network theory to determine the relationships between the essential physics of a system and the topological properties of the network and to derive new techniques in graph theory that can be related to the underlying physico-chemical or reactive properties of a system. Examples are used throughout this chapter that demonstrate applications to a wide variety of complex chemical systems, including ion association within electrolytes, mass transport and the properties of liquid–liquid phase boundaries, and multicomponent solutions.

Complex networks↗
NASA NTRS2019

Federated Cloud Challenges in NASA's Earth Science Data Systems (Why So Difficult?)

NASA is presented with a number of opportunities and challenges in federating its Earth Science Data Systems in the burgeoning world of cloud computing. Cloud hosting of Earth Science data provides a new way of bringing data together, at least from a virtual location sense, and is one of the main motives for NASA to host data there. However, NASA is also faced with a Big Data Variety challenge, brought on by the variety of the EO datasets in its archives. This diversity requires many diverse science archives to service the different science communities. As a result, nearly every major function in the Earth Observing System Data and Information System (EOSDIS) must also be federated across its data centers. This pattern is repeated with many of the outside agencies and organizations that EOSDIS federates with, such as the Committee for Earth Observing Satellites, leading to pioneering work on "deep federation" in a joint project with the European Space Agency to develop a Multi-Mission Algorithm and Analysis Platform.

Lynnes, Christopher↗
NASA NTRS2012

Investigation into Cloud Computing for More Robust Automated Bulk Image Geoprocessing

Geospatial resource assessments frequently require timely geospatial data processing that involves large multivariate remote sensing data sets. In particular, for disasters, response requires rapid access to large data volumes, substantial storage space and high performance processing capability. The processing and distribution of this data into usable information products requires a processing pipeline that can efficiently manage the required storage, computing utilities, and data handling requirements. In recent years, with the availability of cloud computing technology, cloud processing platforms have made available a powerful new computing infrastructure resource that can meet this need. To assess the utility of this resource, this project investigates cloud computing platforms for bulk, automated geoprocessing capabilities with respect to data handling and application development requirements. This presentation is of work being conducted by Applied Sciences Program Office at NASA-Stennis Space Center. A prototypical set of image manipulation and transformation processes that incorporate sample Unmanned Airborne System data were developed to create value-added products and tested for implementation on the "cloud". This project outlines the steps involved in creating and testing of open source software developed process code on a local prototype platform, and then transitioning this code with associated environment requirements into an analogous, but memory and processor enhanced cloud platform. A data processing cloud was used to store both standard digital camera panchromatic and multi-band image data, which were subsequently subjected to standard image processing functions such as NDVI (Normalized Difference Vegetation Index), NDMI (Normalized Difference Moisture Index), band stacking, reprojection, and other similar type data processes. Cloud infrastructure service providers were evaluated by taking these locally tested processing functions, and then applying them to a given cloud-enabled infrastructure to assesses and compare environment setup options and enabled technologies. This project reviews findings that were observed when cloud platforms were evaluated for bulk geoprocessing capabilities based on data handling and application development requirements.

Brown, Richard B.↗
DOE OSTI2013

Final Technical Report for "High Energy Physics at The University of Iowa"

Particle Physics explores the very fundamental building blocks of our universe: the nature of forces, of space and time. By exploring very energetic collisions of sub-nuclear particles with sophisticated detectors at the colliding beam accelerators (as well as others), experimental particle physicists have established the current theory known as the Standard Model (SM), one of the several theoretical postulates to explain our everyday world. It explains all phenomena known up to a very small fraction of a second after the Big Bang to a high precision; the Higgs boson, discovered recently, was the last of the particle predicted by the SM. However, many other phenomena, like existence of dark energy, dark matter, absence of anti-matter, the parameters in the SM, neutrino masses etc. are not explained by the SM. So, in order to find out what lies beyond the SM, i.e., what conditions at the earliest fractions of the first second of the universe gave rise to the SM, we constructed the Large Hadron Collider (LHC) at CERN after the Tevatron collider at Fermi National Accelerator Laboratory. Each of these projects helped us push the boundary further with new insights as we explore a yet higher energy regime. The experiments are extremely complex, and as we push the boundaries of our existing knowledge, it also requires pushing the boundaries of our technical knowhow. So, not only do we pursue humankind’s most basic intellectual pursuit of knowledge, we help develop technology that benefits today’s highly technical society. Our trained Ph.D. students become experts at fast computing, manipulation of large data volumes and databases, developing cloud computing, fast electronics, advanced detector developments, and complex interfaces in several of these areas. Many of the Particle physics Ph.D.s build their careers at various technology and computing facilities, even financial institutions use some of their skills of simulation and statistical prowess. Additionally, last but not least, today’s discoveries make for tomorrow’s practical uses of an improved life style, case in point, internet technology, fiber optics, and many such things. At The University of Iowa we are involved in the LHC experiments, ATLAS and CMS, building equipment, with calibration and maintenance, supporting the infrastructure in hardware, software and analysis as well as participating in various aspects of data analyses. Our theory group works on fundamentals of field theories and on exploration of non-accelerator high energy neutrinos and possible dark matter searches.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗