Search NASA⌕ Search

SEARCH · Search NASA

Results for “cloud computing, data cloud”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Workflows Community Summit 2022: A Roadmap Revolution

Scientific workflows have become integral tools in broad scientific computing use cases. Science discovery is increasingly dependent on workflows to orchestrate large and complex scientific experiments that range from the execution of a cloud-based data preprocessing pipeline to multi-facility instrument-to-edge-to-HPC computational workflows. Given the changing landscape of scientific computing (often referred to as a computing continuum) and the evolving needs of emerging scientific applications, it is paramount that the development of novel scientific workflows and system functionalities seek to increase the efficiency, resilience, and pervasiveness of existing systems and applications. Specifically, the proliferation of machine learning/artificial intelligence (ML/AI) workflows, need for processing large-scale datasets produced by instruments at the edge, intensification of near real-time data processing, support for long-term experiment campaigns, and emergence of quantum computing as an adjunct to HPC, have significantly changed the functional and operational requirements of workflow systems. Workflow systems now need to, for example, support data streams from the edge-to-cloud-to-HPC, enable the management of many small-sized files, allow data reduction while ensuring high accuracy, orchestrate distributed services (workflows, instruments, data movement, provenance, publication, etc.) across computing and user facilities, among others. Further, to accelerate science, it is also necessary that these systems implement specifications/standards and APIs for seamless (horizontal and vertical) integration between systems and applications, as well as enable the publication of workflows and their associated products according to the FAIR principles.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2013

Integration and Commissioning of a Prototype Federated Cloud for Scientific Workflows

The Fermilab Grid and Cloud Computing Department and the KISTI Global Science experimental Data hub Center propose a joint project. The goals are to enable scientific workflows of stakeholders to run on multiple cloud resources by use of (a) Virtual Infrastructure Automation and Provisioning, (b) Interoperability and Federat ion of Cloud Resources , and (c) High-Throughput Fabric Virtualization. This is a matching fund project in which Fermilab and KISTI will contribute equal resources .

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Confidentiality-preserving machine learning algorithms for soft-failure detection in optical communication networks

Automated fault management is at the forefront of next-generation optical communication networks. The increase in complexity of modern networks has triggered the need for programmable and software-driven architectures to support the operation of agile and self-managed systems. In these scenarios, the European Telecommunications Standards Institute zero-touch network and service management approach is imperative. The need for machine learning algorithms to process the large volume of telemetry data brings safety concerns as distributed cloud-computing solutions become the preferred approach for deploying reliable communication network automation. This paper’s contribution is twofold. First, we propose a simple yet effective method to guarantee the confidentiality of the telemetry data based on feature scrambling. The method allows the operation of third-party computational services without direct access to the full content of the collected data. Additionally, the effectiveness of four unsupervised machine learning algorithms for soft-failure detection is evaluated when applied to the scrambled telemetry data. The methods are based on factor analysis, principal component analysis, nonlinear principal component analysis, and singular value decomposition. Most dimensionality reduction algorithms have the common property that they can maintain similar levels of fault classification performance while hiding the data structure from unauthorized access. Evaluations of the proposed algorithms demonstrate this capability.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

MSD CoP Webinar: "Advances in MSD-LIVE to Support the MSD Community of Practice"

Context: This webinar was hosted by the MultiSector Dynamics Community of Practice (MSD CoP; https://multisectordynamics.org). Advances in MSD-LIVE to Support the MSD Community of Practice Presenters: Casey Burleyson and Zoe Guillen (Pacific Northwest National Laboratory) Abstract: The MultiSector Dynamics Living, Intuitive, Value-adding, Environment (MSD-LIVE; msdlive.org) is a cloud-based data management system and advanced computing platform that enables MSD researchers to document and archive their data, run their models and analysis tools, and share their data, software, and workflows within the MSD Community of Practice. Recently, several high-profile datasets have attracted many new users to MSD-LIVE. This webinar has two goals: 1) To refamiliarize the MSD community and new users with the components of the platform (e.g., the data repository, model training notebooks, and data dashboards) and to highlight examples of how these components are advancing MSD science and 2) To demonstrate new features in v3 of the platform, released in late 2025. The main new feature in v3 is the ability to interactively explore data in MSD-LIVE without downloading it. MSD-LIVE users can now click a button in our data repository and launch a blank Jupyter notebook with access to the underlying data on AWS. Users can use the notebook to write analysis, visualization, or subsetting routines that process the data directly on the AWS cloud. We also added a GitHub integration feature that allows users to share analysis or visualization code they develop with the community of MSD-LIVE users. The webinar will wrap up with a look at what's coming next for MSD-LIVE in 2026. Moderator: Patrick M. Reed (MSD CoP Facilitation Team) This webinar was held on: May 12th, 2026 from 1-2 PM EST.

Open Science↗
DOE OSTI2020

Platform for efficient large-scale storage and analysis of multi-omics data in plant and microbial systems (Final Technical Report)

Genomic variation at the sequence level fundamentally affects the phenotypic state of all organisms at all stages of development, while dynamic processes such as changes in the epigenome (e.g. DNA methylation state) and transcriptome regulate the specific phenotype expressed at any given state of development based upon that genomic variation. In plants, DNA methylation is a particularly important mechanism for both regulating transcriptomic expression and for management of genomic variations that could be deleterious to the organism due to the presence of active retrotransposons in plant genomes. While DNA methylation is heritable, it is also dynamic through a given plant’s development and life cycle, particularly during the development from seed to mature specimen suggesting variations in DNA methylation could be critical regulators of biologically and commercially important phenotypes such as time to flowering; in addition, plant DNA methylation is more complex than that of animals, with methylation of CHG and CHH trinucleotides evident in addition to the better-known CG methylation. The complexity of plant DNA methylation and its interplay with genomic sequence variation, transcriptomics and other epigenomic factors demand a storage and analysis framework that can cope with the complexity both within a single specimen and with analyses that span many individuals and even many species, such as attempts to extend models from model organisms to commercially relevant species. In addition to complexity, the rapid development and proliferation of sequencing technology has led to an explosion of data volume that conventional storage and analysis solutions will likely be unable to cope with in the long run. We proposed to study these with suitable distributed storage and computation and therefore for the application of cloud computing to biological analyses; integrate with existing data sources and compatible with virtually any interface use case, from fully automated shell scripts to notebooks and do all these at scale in this STTR grant.

60 APPLIED LIFE SCIENCES↗
NASA NTRSDate not supplied

NASA Cloud Data Access

Explore the source record for details and available documents.

Cloud Computing↗
DOE OSTI2015

Next Generation Workload Management System For Big Data on Heterogeneous Distributed Computing

The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. Experiments at the LHC explore the fundamental nature of matter and the basic forces that shape our universe, and were recently credited for the discovery of a Higgs boson. ATLAS and ALICE are the largest collaborations ever assembled in the sciences and are at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, both experiments rely on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Management System (WMS) for managing the workflow for all data processing on hundreds of data centers. Through PanDA, ATLAS physicists see a single computing facility that enables rapid scientific breakthroughs for the experiment, even though the data centers are physically scattered all over the world. The scale is demonstrated by the following numbers: PanDA manages O(10 2 ) sites, O(10 5 ) cores, O(10 8 ) jobs per year, O(10 3 ) users, and ATLAS data volume is O(10 17 ) bytes. In 2013 we started an ambitious program to expand PanDA to all available computing resources, including opportunistic use of commercial and academic clouds and Leadership Computing Facilities (LCF). The project titled 'Next Generation Workload Management and Analysis System for Big Data' (BigPanDA) is funded by DOE ASCR and HEP. Extending PanDA to clouds and LCF presents new challenges in managing heterogeneity and supporting workflow. The BigPanDA project is underway to setup and tailor PanDA at the Oak Ridge Leadership Computing Facility (OLCF) and at the National Research Center "Kurchatov Institute" together with ALICE distributed computing and ORNL computing professionals. Our approach to integration of HPC platforms at the OLCF and elsewhere is to reuse, as much as possible, existing components of the PanDA system. Finally, we will present our current accomplishments with running the PanDA WMS at OLCF and other supercomputers and demonstrate our ability to use PanDA as a portal independent of the computing facilities infrastructure for High Energy and Nuclear Physics as well as other data-intensive science applications.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1969

A cloud-cover simulation procedure.

Worldwide cloud cover simulation procedure providing data for computer simulation of earth oriented space missions, describing computer program using Monte Carlo mission simulation

Brown, S. C.↗
NASA NTRS1994

ERS-1 scatterometer estimates of annual variations of Atlantic ITCZ and Pacific NECC

A 10 m height wind vector data set, named CMODFD, from ERS-1 Active Microwave Instrument (AMI) measurements, was compared with moored buoy wind observations. The CMODFD wind vectors compared favorably with moored buoy wind observations at about 60 sites. Satisfactory results were obtained from intercomparisons between CMODFD and moored buoy estimates of horizontal wind divergence and wind stress curl. The CMODFD positions of the Intertropical Convergence Zone (ITCZ) at 28 deg west were nearly equal to those determined from satellite images of clouds. The CMODFD data were employed to compute the Ekman vertical velocity in the region of the Pacific North Equatorial Countercurrent (NECC), and a correlative relation was found between the annual cycles and longitudinal variations of the NECC and Ekman vertical motion.

Halpern, D.↗
NASA NTRSDate not supplied

Advanced Information Systems Technology for NASA Earth System Digital Twins (ESDT)

ESA and NASA have both started programs to design and develop Digital Twins of the Earth and/or Earth systems. Organized around interconnected, multi-domain, high-scale modeling capabilities, the three major components of an Earth System Digital Twin are a continuously updated Digital Replica of the Earth System of interest, dynamic Forecasting models, and Impact Assessment capabilities. Earth System Digital Twins integrate diverse Earth and human activity models, continuous observations, and information system capabilities to provide unified, comprehensive representations and predictions that can be utilized for monitoring the health of the Planet, as well as for developing actionable information to support decision making. More generally, Digital Twins will help researchers better understand the fundamental Earth systems that impact everything from wildfires to climate change. This Townhall will first provide a short description of ESA’s, NASA’s and CNES’s current efforts in Digital Twins: • Destination Earth (DestinE) is the first initiative of the European Union to create a Digital Twin of the Earth, particularly focused on weather and climate, and as a coordinated effort between the European Centre for Medium-Range Weather Forecasts (ECMWF), ESA and the European Organization for the Exploitation of Meteorological Satellites (EUMETSAT). • Additionally, ESA is developing several Digital Twin Earth Precursor Activities to study some of the key scientific and technical challenges associated with building a “Digital Twin Earth”. These include forests, hydrology, Antarctica, food systems, oceans, and climate hot spots Digital Twin prototypes. • NASA Earth Science Technology Office (ESTO)’s Advanced Information Systems Technology (AIST) Program has started an initiative in Earth System Digital Twins (ESDT), with 14 current projects in this area, developing various information systems technologies and prototypes that will help prepare the development of future NASA Digital Twins. • CNES is currently developing the concept of a Digital Twin Factory (DTF), which relies on a data lake, a high computing capability using clouds and/or HPC and has thematic algorithms and methodologies able to generate registered and coherent layers of information in order to enrich a datacube from which physical indicators can be computed spatially. NASA, ESA and CNES are also currently defining science use cases that will be presented during the townhall; those and a few short discussions of current projects will serve as a starting point to engage a dialogue about Digital Twins of the Earth with the IGARSS community.

earth science remote sensing; Information systems↗
NASA NTRSDate not supplied

A Machine Learning Approach to Determine Surface Radiative Fluxes based on CERES Observations

The Clouds and Earth’s Radiant Energy System (CERES) projects provides satellite-based observations of the radiative fluxes and clouds systems. CERES climate quality data products typically take several months of calibration and validation before release to the public. An alternative data product, Fast Longwave and Shortwave radiative Flux (FLASHFlux), was created to provide data to the applied sciences and educational users. FLASHFlux provides Top-of-Atmosphere radiative fluxes, Clouds properties, and parameterized surface radiative fluxes within four days for footprint (Level 2) data. We investigate the use of Artificial Neural Network (ANN) using MODerate resolution Imaging Spectroradiometer (MODIS) derived clouds properties and meteorology from the Global Assimilation and Meteorology Office (GMAO) scaled to the CERES footprint from the CERES Clouds Radiative Swath (CRS) data product to compute surface radiative fluxes. We test ANN produce fluxes against surface fluxes produced from the Fu-Liou model used in CRS and the Langley Parameterized Shortwave Algorithm (LPSA) and Langley Parameterized Longwave Algorithm (LPLA) used in FLASHFlux. We also validated each model with ground-based observations. Furthermore, we investigate Leave-One-Feature-Out Importance (LOFO) to evaluate the significance of each feature in our training and provide insight for future models. Advances in machine learning, along with increases in computational capabilities and available data allow us to estimate effects of unresolved processes in our climate without direct modeling. This work evaluates the ability to create accurate data-driven models to supplement or replace current models that estimate surface radiative fluxes.

Climatology↗
DOE OSTI2026

Bridging Cloud and Edge Computing at NREL Using CONNECT: Cloud Optimized Networking for Next-Gen Edge Computing Technologies [Slides]

CONNECT is an innovative on-premise hardware and software solution that integrates edge and cloud computing infrastructure at NREL. Built on the AWS Greengrass middleware and leveraging the MQTT protocol, CONNECT enables real-time data streaming from IoT devices and gateways to both cloud and local services, empowering researchers to rapidly capture, analyze, and act upon edge-generated data while leveraging cloud capabilities. The platform addresses research infrastructure challenges by providing a pre-approved platform which is already configured with the correct networking and cybersecurity baselines thus eliminating procurement delays and enabling on-demand availability. CONNECT's hybrid architecture efficiently manages burstable workloads, allowing research teams to dynamically scale computational capacity, handle peak data loads, and reduce operational bottlenecks. Advanced capabilities include built-in GPU support for executing machine learning models which enables low-latency inference at the edge from models trained in the cloud. This architecture supports real-time analytics and filtering, providing a mechanism to allow only transmitting and processing high-value data. Cloud-based configuration management permits engineers to manage on-premise systems remotely, optimizing operational efficiency. By bridging edge and cloud computing, CONNECT provides NREL researchers with a flexible, scalable platform that accelerates scientific discovery while maintaining robust security and performance standards.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Accelerating science: The usage of commercial clouds in ATLAS Distributed Computing

The ATLAS experiment at CERN is one of the largest scientific machines built to date and will have ever growing computing needs as the Large Hadron Collider collects an increasingly larger volume of data over the next 20 years. ATLAS is conducting R&D projects on Amazon Web Services and Google Cloud as complementary resources for distributed computing, focusing on some of the key features of commercial clouds: lightweight operation, elasticity and availability of multiple chip architectures. The proof of concept phases have concluded with the cloud-native, vendoragnostic integration with the experiment’s data and workload management frameworks. Google Cloud has been used to evaluate elastic batch computing, ramping up ephemeral clusters of up to O(100k) cores to process tasks requiring quick turnaround. Amazon Web Services has been exploited for the successful physics validation of the Athena simulation software on ARM processors. We have also set up an interactive facility for physics analysis allowing endusers to spin up private, on-demand clusters for parallel computing with up to 4 000 cores, or run GPU enabled notebooks and jobs for machine learning applications. The success of the proof of concept phases has led to the extension of the Google Cloud project, where ATLAS will study the total cost of ownership of a production cloud site during 15 months with 10k cores on average, fully integrated with distributed grid computing resources and continue the R&D projects.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRSDate not supplied

TPSAS-NF1676L-11678-DND

Cloud vertical profiles retrieved from CALIPSO and CloudSat data provide additional information that greatly improves the estimate of atmospheric radiative heating rates and potentially improves the estimate of the global surface radiation budget. To utilize CALIPSO and CloudSat cloud vertical profiles, we combined their profiles and generated merged profiles. Weused the merged cloud profiles and MODIS derived cloud profiles by the CERES cloud algorithm in irradiance computations. In this study, the improvement of the irradiance by CALIPSO and CloudSat derived merged cloud vertical profiles is evaluated by comparing modeled irradiances with and without merged cloud profiles. Because of lower cloud base heights provided by the merged profiles compared with cloud base heights derived from an empirical formula using the MODIS derived cloud top height and optical thickness, the global annual mean surface longwave downward irradiance increases approximately by 7 Wm-2. While CALIPSO version 3 data have not been incorporated in the irradiance computation and the effect of the new CALIPSO data needs to be assessed, a comparison with surface observations suggests an improvement of surface downward longwave irradiance computations. The surface downward longwave irradiance computed with CALIPSO and CloudSat merged clouds agrees better with surface observations than the irradiance computed with MODIS only does. The bias difference over ARM SGP sites with 2 years of data decreases from -5 W m-2to -2 Wm-2, and it decreases from -9 Wm-2to no bias over polar validation sites. The result suggests that CALIPSO CloudSat derived merged clouds significantly improve surface longwave irradiance estimates once they are combined with MODIS derived cloud properties and with temperature and humidity profiles from reanalysis.

Seiji Kato↗
DOE OSTI2015

GRDC. A Collaborative Framework for Radiological Background and Contextual Data Analysis

The Radiation Mobile Analysis Platform (RadMAP) is unique in its capability to collect both high quality radiological data from both gamma-ray detectors and fast neutron detectors and a broad array of contextual data that includes positioning and stance data, high-resolution 3D radiological data from weather sensors, LiDAR, and visual and hyperspectral cameras. The datasets obtained from RadMAP are both voluminous and complex and require analyses from highly diverse communities within both the national laboratory and academic communities. Maintaining a high level of transparency will enable analysis products to further enrich the RadMAP dataset. It is in this spirit of open and collaborative data that the RadMAP team proposed to collect, calibrate, and make available online data from the RadMAP system. The Berkeley Data Cloud (BDC) is a cloud-based data management framework that enables web-based data browsing visualization, and connects curated datasets to custom workflows such that analysis products can be managed and disseminated while maintaining user access rights. BDC enables cloud-based analyses of large datasets in a manner that simulates real-time data collection, such that BDC can be used to test algorithm performance on real and source-injected datasets. Using the BDC framework, a subset of the RadMAP datasets have been disseminated via the Gamma Ray Data Cloud (GRDC) that is hosted through the National Energy Research Science Computing (NERSC) Center, enabling data access to over 40 users at 10 institutions.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2012

Big Data Platforms as a Service: Challenges and Approach

Infrastructure-as-a-Service has revolutionized the manner in which users commission computing infrastructure. Coupled with Big Data platforms (Hadoop, Cassandra), IaaS has democratized the ability to store and process massive datasets. For users that need to customize or create new Big Data stacks, however, readily available solutions do not yet exist. Users must first acquire the necessary cloud computing infrastructure, and manually install the prerequisite software. For complex distributed services this can be a daunting challenge. To address this issue, we argue that distributed services should be viewed as a single application consisting of virtual machines. Users should no longer be concerned about individual machines or their internal organization. To illustrate this concept, we introduce Cloud-Get, a distributed package manager that enables the simple installation of distributed services in a cloud computing environment. Cloud-Get enables users to instantiate and modify distributed services, including Big Data services, using simple commands. Cloud-Get also simplifies creating new distributed services via standardized package definitions.

99 GENERAL AND MISCELLANEOUS↗
NASA NTRSDate not supplied

Towards A Flexible Data Fusion Tool Incorporating Model, Satellite, Regulatory Monitor and Low-Cost Sensor Data for Air Quality Estimation and Forecasting

Air quality managers, researchers, and concerned community scientists around the world have a variety of sources for air quality information, ranging from traditional regulatory monitoring networks and atmospheric chemistry models to remote sensing data products and low-cost sensor networks. However, the ability to incorporate data from these disparate sources and synthesize a comprehensive overview of the local air quality situation remains a considerable barrier for many end-users. This presentation will outline a tool, currently in development, which will address this need using a flexible data fusion approach. The tool will make use of air quality forecast model outputs (primarily from the NASA GEOS-CF composition forecast modeling system), satellite remote sensing data (from instruments including MODIS, VIIRS, TROPOMI, plus TEMPO for the US when available), and in-situ data from official regulatory and/or low-cost networks where these are available. The ability to incorporate data from low-cost sensor networks will be a key feature of the tool; it will make use of other available data sources to calibrate the low-cost sensor data on a regional scale, then use these calibrated low-cost sensor data for localized updating to resolve finer-scale air quality patterns. Development of this tool is taking place with the help of national and international partners and end-user groups, coordinated through the US EPA and the United Nations Environment Programme (UNEP). The tool is being developed on the Google Earth Engine cloud computing platform to facilitate integration of diverse data sources and free access by a broad community of end-users. Stewardship of the tool will be passed to US EPA and UNEP to support future activities with end-users in the US and around the world, and the tool itself will remain freely accessible. We hope that this tool will lower the barrier to entry for various user groups worldwide, including community scientists, who struggle to integrate disparate data sources to gain insight into their local air quality situations. This presentation will cover the early stages of the development of the tool, including the underlying methods and some pilot case studies in integrating low-cost sensor data.

global models↗