SEARCH · Search NASA

Results for “cloud computing, data cloud”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

All sources NASA NTRS DOE OSTI

NASA NTRS2012

Cloud Computing for Mission Design and Operations

The space mission design and operations community already recognizes the value of cloud computing and virtualization. However, natural and valid concerns, like security, privacy, up-time, and vendor lock-in, have prevented a more widespread and expedited adoption into official workflows. In the interest of alleviating these concerns, we propose a series of guidelines for internally deploying a resource-oriented hub of data and algorithms. These guidelines provide a roadmap for implementing an architecture inspired in the cloud computing model: associative, elastic, semantical, interconnected, and adaptive. The architecture can be summarized as exposing data and algorithms as resource-oriented Web services, coordinated via messaging, and running on virtual machines; it is simple, and based on widely adopted standards, protocols, and tools. The architecture may help reduce common sources of complexity intrinsic to data-driven, collaborative interactions and, most importantly, it may provide the means for teams and agencies to evaluate the cloud computing model in their specific context, with minimal infrastructure changes, and before committing to a specific cloud services provider.

cloud computing↗

DOE OSTI2024

Modifying the Asynchronous Jacobi Method for Data Corruption Resilience

Moving scientific computation from high-performance computing (HPC) and cloud computing (CC) environments to devices on the edge, i.e., physically near instruments of interest, has received tremendous interest in recent years. Such edge computing environments can operate on data in situ, offering enticing benefits over data aggregation to HPC and CC facilities that include avoiding costs of transmission, increased data privacy, and real-time data analysis. Because of the inherent unreliability of edge computing environments, new fault-tolerant approaches must be developed before the benefits of edge computing can be realized. Motivated by algorithm-based fault tolerance, a variant of the asynchronous Jacobi (ASJ) method is developed that achieves resilience to data corruption by rejecting solution approximations from neighbor devices according to a bound derived from convergence theory. Numerical results on a two-dimensional Poisson problem show that the new rejection criterion, along with a novel approximation to the shortest path length on which the criterion depends, restores convergence for the ASJ variant in the presence of certain types data corruption. Numerical results are obtained for when the singular values in the analytic bound are approximated. Additional linear systems are also explored, one with a more dense sparsity pattern and one that includes advection. All results indicate that successful resilience to data corruption depends on whether the bound tightens fast enough to reject corrupted data before the iteration evolution deviates significantly from that predicted by the convergence theory defining the bound. This observation generalizes to future work on algorithm-based fault tolerance for other asynchronous algorithms, including upcoming approaches that leverage Krylov subspaces.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

The disCO2ver Platform: Curating Data and Tools for Geologic Carbon Sequestration and Deep Subsurface Research Systems

The U.S. DOE National Energy Technology Laboratory has invested 12+ years of development into the data repository and digital laboratory, the Energy Data eXchange (EDX, edx.netl.doe.gov). Supporting a variety of research areas across the DOE Office of Fossil Energy and Carbon Management, the platform has successfully curated and preserved thousands of data products from DOE research. The Carbon Storage Program has successfully supported data curation, upload, and publishing of data products on EDX for many years, demonstrating a success story of how resources like EDX can effectively help with long term preservation and publishing of DOE data products. EDX continues to shift towards cloud-supported infrastructure, taking a hybrid approach combining on-premises compute and storage integrated with cloud-hosted services. The integration of cloud compute and hybrid architecture enables the development of EDX-hosted platforms that tailor the data and tools hosted on them to a specific community, enables implementation of machine learning tools for data discovery and filtering, and enables the hosting of virtual (online user interface) tools. Geologic carbon sequestration (GCS) research continues to scale up in response to the current administration goals to reduce greenhouse gas emissions and transition the energy economy. Over the last year, EDX’s disCO2ver platform has been developed in response to the need for access to data products and tools to support the scaling up of GCS research. disCO2ver provides access to data resources and tools, produced by DOE and outside authoritative external resources. The platform also provides a user-access control component for the virtualization and cloud hosting of tools. Tools that need to be virtualized, to eliminate the need for users to download the tool and use local compute resources, is essential to supporting big-data analysis and machine learning that is becoming common place in carbon storage modeling, risk analysis, and data publishing practices. This talk will review the EDX’s disCO2ver platform and the current work ongoing to curate data and tools to support GCS and deep subsurface systems research.

Morkner, Paige↗

DOE OSTI2017

Virtual machine provisioning, code management, and data movement design for the Fermilab HEPCloud Facility

The Fermilab HEPCloud Facility Project has as its goal to extend the current Fermilab facility interface to provide transparent access to disparate resources including commercial and community clouds, grid federations, and HPC centers. This facility enables experiments to perform the full spectrum of computing tasks, including data-intensive simulation and reconstruction. We have evaluated the use of the commercial cloud to provide elasticity to respond to peaks of demand without overprovisioning local resources. Full scale data-intensive workflows have been successfully completed on Amazon Web Services for two High Energy Physics Experiments, CMS and NOνA, at the scale of 58000 simultaneous cores. This paper describes the significant improvements that were made to the virtual machine provisioning system, code caching system, and data movement system to accomplish this work. The virtual image provisioning and contextualization service was extended to multiple AWS regions, and to support experiment-specific data configurations. A prototype Decision Engine was written to determine the optimal availability zone and instance type to run on, minimizing cost and job interruptions. We have deployed a scalable on-demand caching service to deliver code and database information to jobs running on the commercial cloud. It uses the frontiersquid server and CERN VM File System (CVMFS) clients on EC2 instances and utilizes various services provided by AWS to build the infrastructure (stack). We discuss the architecture and load testing benchmarks on the squid servers. We also describe various approaches that were evaluated to transport experimental data to and from the cloud, and the optimal solutions that were used for the bulk of the data transport. Finally, we summarize lessons learned from this scale test, and our future plans to expand and improve the Fermilab HEP Cloud Facility.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

NASA NTRS2002

Software for Simulation of Hyperspectral Images

A package of software generates simulated hyperspectral images for use in validating algorithms that generate estimates of Earth-surface spectral reflectance from hyperspectral images acquired by airborne and spaceborne instruments. This software is based on a direct simulation Monte Carlo approach for modeling three-dimensional atmospheric radiative transport as well as surfaces characterized by spatially inhomogeneous bidirectional reflectance distribution functions. In this approach, 'ground truth' is accurately known through input specification of surface and atmospheric properties, and it is practical to consider wide variations of these properties. The software can treat both land and ocean surfaces and the effects of finite clouds with surface shadowing. The spectral/spatial data cubes computed by use of this software can serve both as a substitute for and a supplement to field validation data.

Richtsmeier, Steven C.↗

NASA NTRS2004

Simulation of Hyperspectral Images

A software package generates simulated hyperspectral imagery for use in validating algorithms that generate estimates of Earth-surface spectral reflectance from hyperspectral images acquired by airborne and spaceborne instruments. This software is based on a direct simulation Monte Carlo approach for modeling three-dimensional atmospheric radiative transport, as well as reflections from surfaces characterized by spatially inhomogeneous bidirectional reflectance distribution functions. In this approach, "ground truth" is accurately known through input specification of surface and atmospheric properties, and it is practical to consider wide variations of these properties. The software can treat both land and ocean surfaces, as well as the effects of finite clouds with surface shadowing. The spectral/spatial data cubes computed by use of this software can serve both as a substitute for, and a supplement to, field validation data.

Richsmeier, Steven C.↗

DOE OSTI2021

Cyber-Physical System Implementation for Manufacturing With Analytics in the Cloud Layer

Effective and efficient modern manufacturing operations require the acceptance and incorporation of the fourth industrial revolution, also known as Industry 4.0. Traditional shop floors are evolving their production into smart factories. To continue this trend, a specific architecture for the cyber-physical system is required, as well as a systematic approach to automate the application of algorithms and transform the acquired data into useful information. This work makes use of an approach that distinguishes three layers that are part of the existing Industry 4.0 paradigm: edge, fog, and cloud. Each of the layers performs computational operations, transforming the data produced in the smart factory into useful information. Trained or untrained methods for data analytics can be incorporated into the architecture. A case study is presented in which a real-time statistical control process algorithm based on control charts was implemented. The algorithm automatically detects changes in the material being processed in a computerized numerical control (CNC) machine. The algorithm implemented in the proposed architecture yielded short response times. The performance was effective since it automatically adapted to the machining of aluminum and then detected when the material was switched to steel. The data were backed up in a database that would allow traceability to the line of g-code that performed the machining.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2019

Description of the ARM Large-Scale Forcing Data from the Constrained Variational Analysis (VARANAL) Version 2

This technical report represents an update of the previous technical report written by Zhang et al. (2001a) (available at http://www.arm.gov/publications/tech_reports/arm-tr-005.pdf), which described the U.S. Department of Energy Atmospheric Radiation Measurement (ARM) user facility constrained variational analysis (VARANAL) that is used to develop the large-scale forcing data for driving single-column models (SCMs), cloud-resolving models (CRMs), and large-eddy simulation models (LES). The VARANAL algorithm was originally developed by Zhang and Lin (1997) and Zhang et al. (2001b) at Stony Brook University and was migrated to the Lawrence Livermore National Laboratory (LLNL) as the ARM operational objective analysis system in May 1999. Since then, the algorithm has been evolved with time along with the availability of new observations and techniques to meet various modeling needs. Major updates include: (1) The method used to develop multi-year continuous forcing data (Xie et al. 2004), (2) The incorporation of eddy correlation flux measurement system (ECOR) turbulent fluxes into the analysis (Tang et al. 2019), and (3) Improvements to the workflow (e.g., implementing part of the code into the ARM Data Integrator [ADI]) to increase efficiency. The ARM large-scale forcing data have been widely used for SCM/CRM/LES to understand and improve physical processes in models. Zhang et al. (2016) has provided a comprehensive review of the SCM concept, early efforts to derive forcing data for SCM studies, efforts of the ARM constrained variational analysis, and previous SCM studies using ARM cases. This technical report focuses on the constrained variational analysis algorithm and the introduction of the ARM VARANAL products.

58 GEOSCIENCES↗

NASA NTRS1990

Cloud parameters from GOES visible and infrared radiances during the FIRE Cirrus IFO, October 1986

Visible (VIS, 0.65 micron) and infrared (IR, 10.5 microns) channels on geostationary satellites are the key elements of the International Satellite Cloud Climatology Project (ISCCP). All daytime ISCCP cloud parameters are derived from a combination of VIS and IR data. Validation and improvement of the ISCCP and other cloud retrieval algorithms are important components of the First ISCCP Regional Experiment (FIRE) Intensive Field Observations (IFO). Data from the Cirrus IFO (October 19 to November 2, 1986) over Wisconsin are available for validating cirrus cloud retrievals from satellites. The Geostationary Operational Environmental Satellite (GOES) located over the Equator at approximately 100 deg W provided nearly continuous measurements of VIS and IR radiances over the IFO areas. The preliminary results of cloud parameters derived from the IFO GOES data are presented. Cloud attitudes are first derived using an algorithms without corrections for cloud emissivity. These same parameters will then be computed from the same data relying on an emissivity correction algorithm based on correlative data taken during the Cirrus IFO.

Heck, Patrick W.↗

NASA NTRS1974

Aircraft observations of Venus' near-infrared reflection spectrum - Implications for cloud composition

A comparison of aircraft-based measurement data on Venus' near-infrared (1.2- to 4.1-micron) reflection spectrum with computer generated spectra of a number of cloud candidates shows a 75-% or more concentrated water solution of sulfuric acid to give the only acceptable match to the profile of Venus' strong 3-micron absorption feature. However, the measurement data obtained also show a modest decline in reflectivity from 2.3-micron to 1.2-micron wavelength, which is inconsistent with the flat spectrum of sulfuric acid in this spectral region. It is hypothesized that this decline is due to impurities in the sulfuric acid droplets.

Pollack, J. B.↗

DOE OSTI2024

Operational experience and R&D results using the Google Cloud for High-Energy Physics in the ATLAS experiment

The ATLAS experiment at CERN relies on a Worldwide Distributed Computing Grid infrastructure to support its physics program at the Large Hadron Collider. ATLAS has integrated cloud computing resources to complement its Grid infrastructure and conducted an R&D program on Google Cloud Platform. These initiatives leverage key features of commercial cloud providers: lightweight configuration and operation, elasticity and availability of diverse infrastructures. Here this paper examines the seamless integration of cloud computing services as a conventional Grid site within the ATLAS workflow management and data management systems, while also offering new setups for interactive, parallel analysis. It underscores pivotal results that enhance the on-site computing model and outlines several R&D projects that have benefited from large-scale, elastic resource provisioning models. Furthermore, this study discusses the impact of cloud-enabled R&D projects in three domains: accelerators and AI/ML, ARM CPUs and columnar data analysis techniques.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

NASA NTRS2018

User Metrics in NASA Earth Science Data Systems

This presentation the collection and use of user metrics in NASA's Earth Science data systems. A variety of collection methods is discussed, with particular emphasis given to the American Customer Satisfaction Index (ASCI). User sentiment on potential use of cloud computing is presented, with generally positive responses. The presentation also discusses various forms of automatically collected metrics, including an example of the relative usage of different functions within the Giovanni analysis system.

user metrics↗

DOE OSTI2016

An Assessment of New Satellite Data Products for the Development of a Long-Term Global Solar Resource at 10-100 km

A project representing an effort to reprocess the NASA based solar resource data sets is reviewed. The effort represented a collaboration between NASA, NOAA, NREL and the SUNY-Albany and aimed to deliver a 10 km resolution, 3-hourly data set spanning from 1983 through near-present. Part of the project was to transition project capability to NREL for annual processing to extend data set. Due to delays in the key input project called ISCCP, we evaluate only Beta versions of this data set and also introduce the potential use of another NASA Langley based cloud data set for the CERES project. The CERES project uses these cloud properties to compute global top-of-atmosphere and surface fluxes at the 1x1 degree resolution. Here, we also briefly discuss these data sets in potential usage for solar resource benchmarking.

14 SOLAR ENERGY↗

DOE OSTI2025

Navigating Integration: Key Challenges for Data Centers, Nuclear Stakeholders, and Utility Operators

The rapid expansion of data centers, driven by the exponential growth in data-processing and storage needs, presents significant challenges and opportunities for various stakeholders, including data center developers, nuclear energy providers, and utility companies. Data centers are projected to consume 6.7–12% of United States (U.S.) electricity by 2028, driven by artificial intelligence (AI) and cloud-computing demands. Nuclear energy offers reliability and dispatchable baseload power, but data centers need power now while nuclear still needs time to address siting, fast power ramping, and regulatory hurdles. Utilities must keep pace with the unprecedented acceleration of large load interconnection requests and urgently adapt to high-density loads while maintaining grid stability, reliability, and accelerating interconnection timelines. This report dives into these challenges and proposes key collaboration strategies to streamline data center integration that aligns with recent federal initiatives like America’s AI Action Plan and related executive orders that emphasize the importance of data center growth, nuclear energy expansion, and maintaining a competitive edge in the global AI race.

21 - SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLAN↗

NASA NTRS1984

Determination of cloud parameters from infrared sounder data

The World Climate Research Programme (WCRP) plan is concerned with the need to develop a uniform global cloud climatology as part of a broad research program on climate processes. The International Satellite Cloud Climatology Project (ISCCP) has been approved as the first project of the WCRP. The ISCCP has the basic objective to collect and analyze satellite radiance data to infer the global distribution of cloud radiative properties in order to improve the modeling of cloud effects on climate. Research is conducted to explore an algorithm for retrieving cloud properties by utilizing the available infrared sounder data from polar-orbiting satellites. A numerical method is developed for computing cloud top heights, amount, and emissivity on the basis of a parameterized infrared radiative transfer equation for cloudy atmospheres. Theoretical studies were carried out by considering a synthetic atmosphere.

Yeh, H.-Y. M.↗

NASA NTRSDate not supplied

Modern Scientific Data Governance Framework

Science has entered the era of Big Data with new challenges related to data governance, stewardship, and management. The existing data governance practices must catch up to ensure proper data management. Existing data governance policies and stewardship best practices tend to be disconnected from operational data management practices and enforcement and mainly exist in well-meaning documents or reports. These governance policies are, at best, partially implemented and rarely monitored or audited. In addition, existing governance policies keep adding additional data management steps that require a human, ‘a data steward’, in the loop, and the cost of data management can no longer scale proportionately with the current and future increased data volume and complexity. The goal for developing an updated data governance framework is to modernize scientific data governance to the reality of Big data and align it with the current technology trends such as cloud computing and AI. The goals of this framework are two folds. One is to ensure thoroughness that the governance adequately covers the entire data life cycle. Two, provide a practical approach that offers a consistent and repeatable process for different projects. Three core principles ground this framework. First, focus on just enough governance and prevent data governance from becoming a roadblock toward the scientific process. Remove any unnecessary processes and steps. Second, automate data management steps where possible. Actively remove steps that require ‘human in the loop’ within the management process to be efficient and scale with increasing data. Third, all the processes should continually be optimized using quantified metrics to streamline the monitoring and auditing workflows.

Rahul Ramachandran↗

DOE OSTI2016

Editorial [Special issue on software defined networks and infrastructures, network function virtualisation, autonomous systems and network management]

For last few years, there has been a tremendous growth in data traffic due to high adoption rate of mobile devices and cloud computing. Internet of things (IoT) will stimulate even further growth. This is increasing scale and complexity of telecom/internet service provider (SP) and enterprise data centre (DC) compute and network infrastructures. As a result, managing these large network-compute converged infrastructures is becoming complex and cumbersome. To cope up, network and DC operators are trying to automate network and system operations, administrations and management (OAM) functions. OAM includes all non-functional mechanisms which keep the network running.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2015

Intelligent Observation Strategies for Geosynchronous Remote Sensing for Natural Hazards

Geosynchronous satellites offer a unique perspective for monitoring environmental factors important to understanding natural hazards and supporting the disasters management life cycle, namely forecast, detection, response, recovery and mitigation. In the NASA decadal survey for Earth science, the GEO-CAPE mission was proposed to address coastal and air pollution events in geosynchronous orbit, complementing similar initiatives in Asia by the South Koreans and by ESA in Europe, thereby covering the northern hemisphere. In addition to analyzing the challenges of identifying instrument capabilities to meet the science requirements, and the implications of hosting the instrument payloads on commercial geosynchronous satellites, the GEO-CAPE mission design team conducted a short study to explore strategies to optimize the science return for the coastal imaging instrument. The study focused on intelligent scheduling strategies that took into account cloud avoidance techniques as well as onboard processing methods to reduce the data storage and transmission loads. This paper expands the findings of that study to address the use of intelligent scheduling techniques and near-real time data product acquisition of both the coastal water and air pollution events. The topics include the use of onboard processing to refine and execute schedules, to detect cloud contamination in observations, and to reduce data handling operations. Analysis of state of the art flight computing capabilities will be presented, along with an assessment of cloud detection algorithms and their performance characteristics. Tools developed to illustrate operational concepts will be described, including their applicability to environmental monitoring domains with an eye to the future. In the geostationary configuration, the payload becomes a networked thing with enough connectivity to exchange data seamlessly with users. This allows the full field of view to be sensed at very high rate under the control of ground infrastructure, resulting in improved efficiencies, accuracy and science benefits. Hence a remote sensing payload and its data may become one of millions of connected objects in the emerging Internet of Things (IoT), and be as easily accessible by a users smart phone as any other smart appliance.

0↗