Search NASA⌕ Search

SEARCH · Search NASA

Results for “CLOUD COMPUTING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 505 records · Page 28
All sourcesNASA NTRSDOE OSTI
NASA NTRS2012

The Virtual Climate Data Server (vCDS): An iRODS-Based Data Management Software Appliance Supporting Climate Data Services and Virtualization-as-a-Service in the NASA Center for Climate Simulation

Scientific data services are becoming an important part of the NASA Center for Climate Simulation's mission. Our technological response to this expanding role is built around the concept of a Virtual Climate Data Server (vCDS), repetitive provisioning, image-based deployment and distribution, and virtualization-as-a-service. The vCDS is an iRODS-based data server specialized to the needs of a particular data-centric application. We use RPM scripts to build vCDS images in our local computing environment, our local Virtual Machine Environment, NASA s Nebula Cloud Services, and Amazon's Elastic Compute Cloud. Once provisioned into one or more of these virtualized resource classes, vCDSs can use iRODS s federation capabilities to create an integrated ecosystem of managed collections that is scalable and adaptable to changing resource requirements. This approach enables platform- or software-asa- service deployment of vCDS and allows the NCCS to offer virtualization-as-a-service: a capacity to respond in an agile way to new customer requests for data services.

Schnase, John L.↗
DOE OSTI2018

Intelligently-Automated Facilities Expansion with the HEPCloud Decision Engine

The next generation of High Energy Physics experiments are expected to generate exabytes of data---two orders of magnitude greater than the current generation. In order to reliably meet peak demands, facilities must either plan to provision enough resources to cover the forecasted need, or find ways to elastically expand their computational capabilities. Commercial cloud and allocation-based High Performance Computing (HPC) resources both have explicit and implicit costs that must be considered when deciding when to provision these resources, and to choose an appropriate scale. In order to support such provisioning in a manner consistent with organizational business rules and budget constraints, we have developed a modular intelligent decision support system (IDSS) to aid in the automatic provisioning of resources---spanning multiple cloud providers, multiple HPC centers, and grid computing federations.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2023

The disCO2ver Platform: Curating Data and Tools for Geologic Carbon Sequestration and Deep Subsurface Research Systems

The U.S. DOE National Energy Technology Laboratory has invested 12+ years of development into the data repository and digital laboratory, the Energy Data eXchange (EDX, edx.netl.doe.gov). Supporting a variety of research areas across the DOE Office of Fossil Energy and Carbon Management, the platform has successfully curated and preserved thousands of data products from DOE research. The Carbon Storage Program has successfully supported data curation, upload, and publishing of data products on EDX for many years, demonstrating a success story of how resources like EDX can effectively help with long term preservation and publishing of DOE data products. EDX continues to shift towards cloud-supported infrastructure, taking a hybrid approach combining on-premises compute and storage integrated with cloud-hosted services. The integration of cloud compute and hybrid architecture enables the development of EDX-hosted platforms that tailor the data and tools hosted on them to a specific community, enables implementation of machine learning tools for data discovery and filtering, and enables the hosting of virtual (online user interface) tools. Geologic carbon sequestration (GCS) research continues to scale up in response to the current administration goals to reduce greenhouse gas emissions and transition the energy economy. Over the last year, EDX’s disCO2ver platform has been developed in response to the need for access to data products and tools to support the scaling up of GCS research. disCO2ver provides access to data resources and tools, produced by DOE and outside authoritative external resources. The platform also provides a user-access control component for the virtualization and cloud hosting of tools. Tools that need to be virtualized, to eliminate the need for users to download the tool and use local compute resources, is essential to supporting big-data analysis and machine learning that is becoming common place in carbon storage modeling, risk analysis, and data publishing practices. This talk will review the EDX’s disCO2ver platform and the current work ongoing to curate data and tools to support GCS and deep subsurface systems research.

Morkner, Paige↗
NASA NTRS2001

Combined Infrared Stereo and Laser Ranging Cloud Measurements from Shuttle Mission STS-85

Multi-angle remote sensing provides a wealth of information for earth and climate monitoring. And, as technology advances so do the options for developing instrumentation versatile enough to meet the demands associated with these types of measurements. In the current work, the multiangle measurement capability of the Infrared Spectral Imaging Radiometer is demonstrated. This instrument flew as part of mission STS-85 of the space shuttle Columbia in 1997 and was the first earth-observing radiometer to incorporate an uncooled microbolometer array detector as its image sensor. Specifically, a method for computing cloud-top height from the multi-spectral stereo measurements acquired during this flight has been developed and the results demonstrate that a vertical precision of 10.6 km was achieved. Further, the accuracy of these measurements is confirmed by comparison with coincident direct laser ranging measurements from the Shuttle Laser Altimeter. Mission STS-85 was the first space flight to combine laser ranging and thermal IR camera systems for cloud remote sensing.

Lancaster, Redgie S.↗
NASA NTRS1982

Solar reflection from a regular array of horizontally finite clouds

The reflected flux from a regular array of 2- and 3-D clouds has been computed to estimate the effect of fractional cloud cover on albedos and the solar flux available to heat the earth's surface. The broken clouds are represented by a regular array of identical cuboids for the 3-D problem and equally spaced, infinitely long, bars for the 2-D problem. A diffusion approximation to the radiative transfer equation is used to compute the fluxes leaving each face of the cloud. Interaction between clouds is simulated by assuming diffuse exitance from the cloud faces and applying angle factors to obtain modified boundary conditions on each cloud face.

Weinman, J. A.↗
NASA NTRS2004

Combined Infrared Stereo and Laser Ranging Cloud Measurements from Shuttle Mission STS-85

Multiangle remote sensing provides a wealth of information for earth and climate monitoring, such as the ability to measure the height of cloud tops through stereoscopic imaging. As technology advances so do the options for developing spacecraft instrumentation versatile enough to meet the demands associated with multiangle measurements. One such instrument is the infrared spectral imaging radiometer, which flew as part of mission STS-85 of the space shuttle in 1997 and was the first earth- observing radiometer to incorporate an uncooled microbolometer array detector as its image sensor. Specifically, a method for computing cloud-top height with a precision of +/- 620 m from the multispectral stereo measurements acquired during this flight has been developed, and the results are compared with coincident direct laser ranging measurements from the shuttle laser altimeter. Mission STS-85 was the first space flight to combine laser ranging and thermal IR camera systems for cloud remote sensing.

Lancaster, R. S.↗
NASA NTRS2009

Controlling Infrastructure Costs: Right-Sizing the Mission Control Facility

Johnson Space Center's Mission Control Center is a space vehicle, space program agnostic facility. The current operational design is essentially identical to the original facility architecture that was developed and deployed in the mid-90's. In an effort to streamline the support costs of the mission critical facility, the Mission Operations Division (MOD) of Johnson Space Center (JSC) has sponsored an exploratory project to evaluate and inject current state-of-the-practice Information Technology (IT) tools, processes and technology into legacy operations. The general push in the IT industry has been trending towards a data-centric computer infrastructure for the past several years. Organizations facing challenges with facility operations costs are turning to creative solutions combining hardware consolidation, virtualization and remote access to meet and exceed performance, security, and availability requirements. The Operations Technology Facility (OTF) organization at the Johnson Space Center has been chartered to build and evaluate a parallel Mission Control infrastructure, replacing the existing, thick-client distributed computing model and network architecture with a data center model utilizing virtualization to provide the MCC Infrastructure as a Service. The OTF will design a replacement architecture for the Mission Control Facility, leveraging hardware consolidation through the use of blade servers, increasing utilization rates for compute platforms through virtualization while expanding connectivity options through the deployment of secure remote access. The architecture demonstrates the maturity of the technologies generally available in industry today and the ability to successfully abstract the tightly coupled relationship between thick-client software and legacy hardware into a hardware agnostic "Infrastructure as a Service" capability that can scale to meet future requirements of new space programs and spacecraft. This paper discusses the benefits and difficulties that a migration to cloud-based computing philosophies has uncovered when compared to the legacy Mission Control Center architecture. The team consists of system and software engineers with extensive experience with the MCC infrastructure and software currently used to support the International Space Station (ISS) and Space Shuttle program (SSP).

Martin, Keith↗
DOE OSTI2024

Utilizing Distributed Heterogeneous Computing with PanDA in ATLAS

In recent years, advanced and complex analysis workflows have gained increasing importance in the ATLAS experiment at CERN, one of the large scientific experiments at LHC. Support for such workflows has allowed users to exploit remote computing resources and service providers distributed worldwide, overcoming limitations on local resources and services. The spectrum of computing options keeps increasing across the Worldwide LHC Computing Grid (WLCG), volunteer computing, high-performance computing, commercial clouds, and emerging service levels like Platform-as-a-Service (PaaS), Container-as-a-Service (CaaS) and Function-as-a-Service (FaaS), each one providing new advantages and constraints. Users can significantly benefit from these providers, but at the same time, it is cumbersome to deal with multiple providers, even in a single analysis workflow with fine-grained requirements coming from their applications’ nature and characteristics. In this paper, we will first highlight issues in geographically-distributed heterogeneous computing, such as the insulation of users from the complexities of dealing with remote providers, smart workload routing, complex resource provisioning, seamless execution of advanced workflows, workflow description, pseudointeractive analysis, and integration of PaaS, CaaS, and FaaS providers. We will also outline solutions developed in ATLAS with the Production and Distributed Analysis (PanDA) system and future challenges for LHC Run4.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2020

Radiative Heating Rate Profiles over the Southeast Atlantic Ocean during the 2016 and 2017 Biomass Burning Seasons

Marine boundary layer clouds, including the transition from stratocumulus to cumulus, are poorly represented in numerical weather prediction and general circulation models. Further uncertainties in the cloud structure arise in the presence of biomass burning carbonaceous aerosol, as is the case over the southeast Atlantic Ocean where biomass burning aerosol is transported from the African continent. As the aerosol plume progresses across the southeast Atlantic Ocean, radiative heating within the aerosol layer has the potential to alter the thermodynamic environment and therefore the cloud structure; however, limited work has been done to quantify this along the trajectory of the aerosol plume in the region. The deployment of the First Atmospheric Radiation Measurement Mobile Facility in support of the Layered Atlantic Smoke Interactions with Clouds field campaign provided a unique opportunity to collect observations of cloud and aerosol properties during two consecutive biomass burning seasons during July through October of 2016 and 2017 over Ascension Island (7.96 °S, 14.35 °W). Using observed profiles of temperature, humidity, and clouds from the field campaign, alongside aerosol optical properties from MERRA-2 as input for the Rapid Radiation Transfer Model, profiles of the radiative heating rate due to aerosols and clouds were computed. Radiative heating is also assessed across the southeast Atlantic Ocean using an ensemble of back trajectories from the Hybrid Single Particle Lagrangian Integrated Trajectory Model. Idealized experiments using the Rapid Radiation Transfer Model with and without aerosols and a range of values for the single scattering albedo demonstrate that shortwave heating within the aerosol layer above Ascension Island can locally range between 2 and 8 K per day depending on the aerosol optical properties, though impacts of the aerosol can be felt elsewhere in the atmospheric column. When considered under clear conditions, the aerosol has a cooling effect at the TOA and based on the observed cloud properties at Ascension Island, the cloud albedo is not large enough to overcome this. Shortwave radiative heating due to biomass burning aerosol is not balanced by additional longwave cooling, and the net radiative impact results in a stabilization of the lower troposphere. However, these results are extremely sensitive to the single scattering albedo assumptions in models.

Allison B. Marquardt Collow↗
DOE OSTI2020

Radiative heating rate profiles over the southeast Atlantic Ocean during the 2016 and 2017 biomass burning seasons

Marine boundary layer clouds, including the transition from stratocumulus to cumulus, are poorly represented in numerical weather prediction and general circulation models. Further uncertainties in the cloud structure arise in the presence of biomass burning carbonaceous aerosol, as is the case over the southeast Atlantic Ocean, where biomass burning aerosol is transported from the African continent. As the aerosol plume progresses across the southeast Atlantic Ocean, radiative heating within the aerosol layer has the potential to alter the thermodynamic environment and therefore the cloud structure; however, limited work has been done to quantify this along the trajectory of the aerosol plume in the region. The deployment of the first Atmospheric Radiation Measurement (ARM) Mobile Facility (AMF1) in support of the Layered Atlantic Smoke Interactions with Clouds field campaign provided a unique opportunity to collect observations of cloud and aerosol properties during two consecutive biomass burning seasons during July through October of 2016 and 2017 over Ascension Island (7.96° S, 14.35° W). Using observed profiles of temperature, humidity, and clouds from the field campaign alongside aerosol optical properties from Modern-Era Retrospective analysis for Research and Applications, Version 2 (MERRA-2), as input for the Rapid Radiation Transfer Model (RRTM), profiles of the radiative heating rate due to aerosols and clouds were computed. Radiative heating is also assessed across the southeast Atlantic Ocean using an ensemble of back trajectories from the Hybrid Single Particle Lagrangian Integrated Trajectory (HYSPLIT) model. Idealized experiments using the RRTM with and without aerosols and a range of values for the single-scattering albedo (SSA) demonstrate that shortwave (SW) heating within the aerosol layer above Ascension Island can locally range between 2 and 8 K d -1 depending on the aerosol optical properties, though impacts of the aerosol can be felt elsewhere in the atmospheric column. When considered under clear conditions, the aerosol has a cooling effect at the TOA, and based on the observed cloud properties at Ascension Island, the cloud albedo is not large enough to overcome this. Shortwave radiative heating due to biomass burning aerosol is not balanced by additional longwave (LW) cooling, and the net radiative impact results in a stabilization of the lower troposphere. However, these results are extremely sensitive to the single-scattering albedo assumptions in models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2015

3D cloud detection and tracking system for solar forecast using multiple sky imagers

We propose a system for forecasting short-term solar irradiance based on multiple total sky imagers (TSIs). The system utilizes a novel method of identifying and tracking clouds in three-dimensional space and an innovative pipeline for forecasting surface solar irradiance based on the image features of clouds. First, we develop a supervised classifier to detect clouds at the pixel level and output cloud mask. In the next step, we design intelligent algorithms to estimate the block-wise base height and motion of each cloud layer based on images from multiple TSIs. Thus, this information is then applied to stitch images together into larger views, which are then used for solar forecasting. We examine the system’s ability to track clouds under various cloud conditions and investigate different irradiance forecast models at various sites. We confirm that this system can 1) robustly detect clouds and track layers, and 2) extract the significant global and local features for obtaining stable irradiance forecasts with short forecast horizons from the obtained images. Finally, we vet our forecasting system at the 32-megawatt Long Island Solar Farm (LISF). Compared with the persistent model, our system achieves at least a 26% improvement for all irradiance forecasts between one and fifteen minutes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Collaborative Project: High-resolution Global Modeling of the Effects of Subgrid-Scale Clouds and Turbulence on Precipitating Cloud Systems

We proposed to implement, test, and evaluate recently developed turbulence parameterizations, using a wide variety of methods and modeling frameworks together with observations including ARM data. We have successfully tested three different turbulence parameterizations in versions of the Community Atmosphere Model: CLUBB, SHOC, and IPHOC. All three produce significant improvements in the simulated climate. CLUBB will be used in CAM6, and also in ACME. SHOC is being tested in the NCEP forecast model. In addition, we have achieved a better understanding of the strengths and limitations of the PDF-based parameterizations of turbulence and convection.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2016

Co-Operative Resource Allocation: Building an Open Cloud Market Using Shared Infrastructure

In this paper we present DRIVE, a distributed service-based system designed to facilitate an open economic market for federating Cloud providers. Additionally, to address the challenges associated with market ownership and operation we propose the use of a cooperative (co-op) infrastructure in which the services that make up DRIVE are hosted across participants' resources. To prevent malicious behavior we use cryptographic, secure and privacy preserving allocation protocols as a means of establishing trust in the allocation infrastructure. Finally, we investigate through simulation the effect of different strategies, pricing functions, and penalty models on allocation performance and revenue, and show that the overhead of running DRIVE's services on commodity infrastructure is modest.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Interpreting and Stabilizing Machine-Learning Parametrizations of Convection

Neural networks are a promising technique for parameterizing subgrid-scale physics (e.g., moist atmospheric convection) in coarse-resolution climate models, but their lack of interpretability and reliability prevents widespread adoption. For instance, it is not fully understood why neural network parameterizations often cause dramatic instability when coupled to atmospheric fluid dynamics. This paper introduces tools for interpreting their behavior that are customized to the parameterization task. First, we assess the nonlinear sensitivity of a neural network to lower-tropospheric stability and the midtropospheric moisture, two widely studied controls of moist convection. Second, we couple the linearized response functions of these neural networks to simplified gravity wave dynamics, and analytically diagnose the corresponding phase speeds, growth rates, wavelengths, and spatial structures. To demonstrate their versatility, these techniques are tested on two sets of neural networks, one trained with a super-parameterized version of the Community Atmosphere Model (SPCAM) and the second with a near-global cloud-resolving model (GCRM). Additionally, even though the SPCAM simulation has a warmer climate than the cloud-resolving model, both neural networks predict stronger heating/drying in moist and unstable environments, which is consistent with observations. Moreover, the spectral analysis can predict that instability occurs when GCMs are coupled to networks that support gravity waves that are unstable and have phase speeds larger than 5 m s -1 . In contrast, standing unstable modes do not cause catastrophic instability. Using these tools, differences between the SPCAM-trained versus GCRM-trained neural networks are analyzed, and strategies to incrementally improve both of their coupled online performance unveiled.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

INTEGRATED WORKFLOW MANAGEMENT FOR PARTICLE ACCELERATOR SIMULATION

Supercomputing systems are used for a wide range of computationally demanding tasks in many fields of science and engineering. They play a key role in numerical simulation, in which mathematical models are computed in order to simulate the behavior of physical systems. Scientists and engineers that use supercomputers for numerical simulation often have their productivity limited by the need to manually organize and manage extremely large amounts of data that are often produced and consumed by the software programs run on these systems. Recognizing these limitations, Kitware Inc. (Clifton Park, NY) and SLAC National Accelerator Laboratory (Menlo Park, CA) are developing an advanced software platform that can reduce the cognitive overhead required by knowledge workers when using supercomputers for numerical simulation. Phase I of the project is complete and includes the development of new capabilities for organizing simulation project files, improvements to the user interface and overall usability, and deployment of a “middle tier” server to sit between user desktop machines and supercomputers to offload much of the data management workload. The project also developed prototype software for executing sequences of numerical simulations, and a prototype for migrating supercomputing software to cloud-based computing systems to provide a potential alternative to supercomputers with different logistical and price-to-performance tradeoffs.

Tourtellott, John↗
DOE OSTI2022

Elastic Resource Management for Deep Learning Applications in a Container Cluster

The increasing demand for learning from massive datasets is restructuring our economy. Effective learning, however, involves nontrivial computing resources. Most businesses utilize commercial infrastructure providers (e.g., AWS) to host their computing clusters in the cloud, where various jobs compete for available resources. While cloud resource management is a fruitful research field that has made many advances in production, such as Kubernetes and YARN, few efforts have been invested to further optimize the system performance, especially for deep learning (DL) training jobs in a container cluster. This work introduces FlowCon, a system that is able to monitor the individual evaluation functions of DL jobs at runtime, and thus to make placement decisions on resource allocations elastically. Here, we present a detailed design and implementation of FlowCon and conduct intensive experiments over various DL models. The results demonstrate that FlowCon significantly improves DL job completion time and resource utilization efficiency, compared to default systems. According to the results, FlowCon is able to improve the completion time by up to 68.8% and meanwhile, reduce the makespan by 18.0%, in the presence of various DL job workloads.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Datum: A Scientific Metadata Catalog

The data catalog market is currently flooded with a myriad of different products, but none serve the scientific community well. There are cloud-native tools like Databricks, Snowflake,to on-premise solutions like Collibra and Datahub. The common failing of all these tools however, is their inability to serve the scientific data community directly. Most catalogs are targeted towards financial, health, or user data - not sensor or scientific domain data. They also prioritize integrations that often don’t exist or are just starting to be used in the scientific realm - all while ignoring common scientific tools and file types. Datum is a catalog which targets the scientific data directly, including the tools and networks in which those tools are used. We work with the producers and consumers of the data where they are, targeting cloud and on-premise with a focus on classified networks. Datum is an Erlang/Elixir application. Technical Features Note: The features listed below are still under development and may change, slightly, upon final delivery of the product. File Formats - Datum has the ability to read additional metadata and provides processing pipelines for the following file formats: Plain Text, PDF, LaTeX, HTML, Open Document Format (.odt), XML, CSV/TSV (and other standard delimiters), OpenDocument Database and Spreadsheets, Geo-Referenced TIFF, Common Data Format, HDF/HDF5, LabView TDMS, Excel, DeltaTables, Parquet, Apache Iceberg, Apache Hudi and many others. Metadata Collection - Scanners for the local and networked file systems and cloud storage providers. Network integration with common databases such as MSSQL and MySQL. User Plugin System - Users are able to provide either file processing, metadata extraction, or sampling plugins in the programming language of their choice. Authentication/Authorization -: OIDC integration, SCIM provisioning and EntraID integration out of the box. Full user and group management system with a “least privilege” operating mode. Governance - Customizable data governance platform; dictate and enforce required metadata, enforce data embargos, and enforce user agreements and NDAs before data access. Ability to create health checks on data, rejecting abandoned or poorly curated data and automatically removing it from the search index. Ability for users to submit corrections. Search - Semantic search is a first class citizen. No licenses to expensive, external software required. Integrated use of vectors and vector-based search allows for AI agent integration at all levels of operation. Metadata Model - Display and control data’s lineage and connections to other data and data directories. Data is modeled after a filesystem - an organization instantly recognizable and navigable by most any user. CLI and SDK - Ships with a Command Line Interface (CLI) tool and with a fully-featured Python SDK. This allows for rapid and programmatic use of Datum by every level of user. Minimal Infrastructure - Datum ships as a single executable file and can be run on any operating system and most CPU architectures. Datum has no reliance on external databases, search indexing tools, or other outside services - and it runs equally well on edge computing devices, cloud services, or in a clustered HPC environment.

darrington, john↗