Search NASA⌕ Search

SEARCH · Search NASA

Results for “hybrid cloud”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting

Hybrid cloud bursting (i.e., leasing temporary off-premise cloud resources to boost the overall capacity during peak utilization) can be a cost-effective way to deal with the increasing complexity of big data analytics, especially for iterative applications. However, the low throughput, high latency network link between the on-premise and off-premise resources (“weak link”) makes maintaining scalability difficult. While several data locality techniques have been designed for big data bursting on hybrid clouds, their effectiveness is difficult to estimate in advance. Yet such estimations are critical, because they help users decide whether the extra pay-as-you-go cost incurred by using the off-premise resources justifies the runtime speed-up. To this end, the current paper presents a performance model and methodology to estimate the runtime of iterative MapReduce applications in a hybrid cloud-bursting scenario. The paper focuses on the overhead incurred by the weak link at fine granularity, for both the map and the reduce phases. This approach enables high estimation accuracy, as demonstrated by extensive experiments at scale using a mix of real-world iterative MapReduce applications from standard big data benchmarking suites that cover a broad spectrum of data patterns. As a result, not only are the produced estimations accurate in absolute terms compared with experimental results, but they are also up to an order of magnitude more accurate than applying state-of-art estimation approaches originally designed for single-site MapReduce deployments.

97 MATHEMATICS AND COMPUTING↗

SatCORPS Hybridized Cloud Product Data Storage: The Design of a Hybrid Data Repository That Leverages the Strengths of the Cloud and the Data Center

There is a strong demand for the near real time NASA Langley Satellite ClOud and Radiation Property retrieval System (SatCORPS) products. As important as real-time information, archived copies of the products form the basis for targeted research focusing on specific events or conditions. To make these SatCORPS products available for downloading, the SatCORPS group has developed a number of tools and technologies to create a hybrid data storage system that leverages the strengths of both cloud and on-premises resources. In this work, we describe the technologies the group uses to marshal disparate data repositories and materialize them into a single searchable overview and give a broad description of the organization of the dataset. As with any implementation, the strengths, weaknesses and constraints surrounding the components establish priorities and provide insight where trade-offs are necessary. We further describe the design and architecture underpinning our hybrid data repository and delivery system.

AWS↗

Cloud-Precipitation Hybrid Regimes and their Projection onto IMERG Precipitation Data

We extend and enhance the concept of the Cloud Regimes (CRs) developed from two-dimensional joint histograms of cloud optical thickness and cloud top pressure from the Moderate Resolution Imaging Spectroradiometer (MODIS), by adding precipitation information in order to better understand cloud-precipitation relationships. Taking advantage of the high-resolution Integrated Multi-satellitE Retrievals for GPM (IMERG) precipitation dataset, cloud-precipitation “hybrid” regimes are derived by implementing the k-means clustering algorithm with advanced initialization and objective measures to determine the most optimal clusters. By expressing precipitation rates within 1-degree grid cell as histograms and making choices on the relative weight of cloud and precipitation, we could obtain several editions of hybrid cloud-precipitation regimes (CPRs), and examine their characteristics. In the deep tropics, when precipitation is weighted weakly, the cloud part of the hybrid entroids resembles the centroid of cloud-only regimes, but still tightens the cloud-precipitation relationship by decreasing the precipitation variability of each regime. As precipitation weight progressively increases, the shape of the cloudy part of the hybrid centroids becomes blunter, while the precipitation part of the centroids sharpens. In the case where cloud and precipitation are weighted equally, the CPRs representing high clouds with intermediate to heavy precipitation exhibit distinct features in the precipitation parts of the centroids, which allows us to project them onto the 30-minly IMERG domain. Such a projection can be used to overcome the temporal sparseness of MODIS cloud observations, which leads to great application potential for various convection-focused studies, including diurnal cycle analysis.

cloud-precipitation↗

Understanding the Performance and Potential of Cloud Computing for Scientific Applications

In this paper, commercial clouds bring a great opportunity to the scientific computing area. Scientific applications usually require significant resources, however not all scientists have access to sufficient high-end computing systems, may of which can be found in the Top500 list. Cloud Computing has gained the attention of scientists as a competitive resource to run HPC applications at a potentially lower cost. But as a different infrastructure, it is unclear whether clouds are capable of running scientific applications with a reasonable performance per money spent. This work studies the performance of public clouds and places this performance in context to price. We evaluate the raw performance of different services of AWS cloud in terms of the basic resources, such as compute, memory, network and I/O. We also evaluate the performance of the scientific applications running in the cloud. This paper aims to assess the ability of the cloud to perform well, as well as to evaluate the cost of the cloud running scientific applications. We developed a full set of metrics and conducted a comprehensive performance evlauation over the Amazon cloud. We evaluated EC2, S3, EBS and DynamoDB among the many Amazon AWS services. We evaluated the memory sub-system performance with CacheBench, the network performance with iperf, processor and network performance with the HPL benchmark application, and shared storage with NFS and PVFS in addition to S3. We also evaluated a real scientific computing application through the Swift parallel scripting system at scale. Armed with both detailed benchmarks to gauge expected performance and a detailed monetary cost analysis, we expect this paper will be a recipe cookbook for scientists to help them decide where to deploy and run their scientific applications between public clouds, private clouds, or hybrid clouds.

97 MATHEMATICS AND COMPUTING↗

Jay : A software framework for prototyping and evaluating offloading applications in hybrid edge clouds

Abstract We present Jay , a software framework for offloading applications in hybrid edge clouds. Jay provides an API, services, and tools that enable mobile application developers to implement, instrument, and evaluate offloading applications using configurable cloud topologies, offloading strategies, and job types. We start by presenting Jay 's job model and the concrete architecture of the framework. We then present the programming API with several examples of customization. Then, we turn to the description of the internal implementation of Jay instances and their components. Finally, we describe the Jay Workbench, a tool that allows the setup, execution, and reproduction of experiments with networks of hosts with different resource capabilities organized with specific topologies. The complete source code for the framework and workbench is provided in a GitHub repository.

Silva, Joaquim↗

EDX Spatial: Leveraging cloud and hybrid data management resources for spatial data

In the last few years, the National Energy Technology Laboratory has started leveraging cloud-hosted services for hosting spatial data collections published on the Energy Data eXchange (EDX). Using cloud-hosted storage and compute options offers multiple benefits for visualization and tool development through utilization of spatial data resources. The transition from on-premises services to cloud-hosted services has enabled a few key features: increased accessibility of large derivative datasets, dynamic integration of external authoritative data resources directly from outside entities through representational state transfer application programming interfaces (REST API), and the ability to produce complex mapping applications, dashboards, and online maps. As a result of the shift, NETL has launched EDX Spatial, a platform that leverages on-premises resources combined with cloud compute capabilities to enable enhanced online mapping interfaces and optimize data access. In addition, a unified workflow for handling the public release of spatial data products through EDX has been developed, including standardization of data hosting practices, metadata, symbology, and application elements. This poster reviews the opportunity of leveraging cloud-hosted and hybrid data management solutions for spatial data, and discusses the benefits and lessons learned while leveraging these services through the data repository EDX.

Morkner, Paige↗

NREL's Journey with HPC in the Cloud and Hybrid Computing

This is a planned lightning talk at the NLIT Summit 2025 conference. This would serve as somewhat of a progress update to the presentation I gave at re:Invent 2024 back in November which can be seen here: https://www.youtube.com/watch?t=2133&v=NMq3kL9qObU&feature=youtu.be (my section begins at the included timestamp value). This presentation discusses our usage of Cloud-hosted HPC systems, and in what circumstances they benefit our researchers strategically. We have been making incremental progress in this area since that recording, so for this presentation I would include our latest experiences and observations as we are beginning to implement a hybrid HPC solution. We're in the midst of a cross-team effort of implementing a prototype hybridization solution which would allow users to strategically burst jobs to the cloud. In this talk for NLIT, I would detail lessons-learned, non-starters, architecture diagrams, and other implementation details that may benefit those interested as we continue our experimentation. Our prototype may not be complete by the time of this presentation, but even in the discovery phase of our anticipated design we've discovered a lot of information from others who have worked on hybrid solutions that are worth sharing.

97 MATHEMATICS AND COMPUTING↗

Exploration of cloud computing late start LDRD #149630 : Raincoat. v. 2.1.

This report contains documentation from an interoperability study conducted under the Late Start LDRD 149630, Exploration of Cloud Computing. A small late-start LDRD from last year resulted in a study (Raincoat) on using Virtual Private Networks (VPNs) to enhance security in a hybrid cloud environment. Raincoat initially explored the use of OpenVPN on IPv4 and demonstrates that it is possible to secure the communication channel between two small 'test' clouds (a few nodes each) at New Mexico Tech and Sandia. We extended the Raincoat study to add IPSec support via Vyatta routers, to interface with a public cloud (Amazon Elastic Compute Cloud (EC2)), and to be significantly more scalable than the previous iteration. The study contributed to our understanding of interoperability in a hybrid cloud.

99 GENERAL AND MISCELLANEOUS↗

Combined Optical and Radio-Frequency Perspectives on a Hybrid Cloud-To-Ground Lightning Flash Observed by the FORTE Satellite

In this work, we use the coincident optical and radio-frequency (RF) measurements taken by the Fast On-orbit Recording of Transient Events (FORTE) satellite to shed light on common optical signatures recorded by NASA and NOAA lightning imagers during Cloud-to-Ground (CG) lightning. We build flash cluster data for FORTE using the same techniques as the NASA/NOAA instruments to document the optical/RF evolution of an oceanic hybrid -CG flash. The flash began with strong Very High Frequency (VHF)-band emission from a Narrow Bipolar Event (NBE) that initiated a period of normal bilevel intracloud (IC) activity in two vertical layers (8 and 12 km) that lasted for 490 ms. VHF waveforms show step leader activity ahead of the return stroke. All impulsive VHF sources after the stroke come from the lower (8 km) layer. K-changes are noted following the return stroke, but no subsequent strokes are detected. The optical flash began 136 ms after the NBE RF pulse. Twenty-two of the 33 optical groups were dim and occurred during the in-cloud phase of the flash. This activity included both isolated pulses and sustained periods of illumination over tens of milliseconds. Initial cloud pulses accounted for 23% of the total optical radiance from the flash. Illumination during the return stroke contributed 58% of the total radiance, and the K-changes and cloud pulses after the stroke supplied the remaining 19%. These results highlight the benefit of having RF alongside optical lightning measurements for clarifying signatures in the optical data and providing information on their physical origins.

54 ENVIRONMENTAL SCIENCES↗

Quasi-Langrangian models of nascent thermals

The motions in and around an isolated thermal were studied and rising motion in the core, and sinking motion on the outside were found; while the circulation resembled that of a vortex ring. In an entity cloud model, cloudy thermal is tracked, in a Lagrangian fashion, as a discrete entity; the field of motion in and around the thermal is not explicitly simulated. Field of motion cloud models, in which the equations of motion are numerically integrated on an Eulerian grid were developed. It is shown that the great potential of a hybrid cloud model can combine the simplicity of the entity models with the generality and flexibility of the field-of-motion models. A key problem to be overcome in the development of a hybrid model is the formulation of a mathematical framework within which the cloud dynamics can be represented.

Rambaldi, S.↗

Creating Satellite Data Products in the Cloud: SatCORPS Global Cloud Composites

Real time satellite observations and real time derived cloud products are becoming an important tool for both science as well as business ventures. The SatCORPS group leverages public and private cloud-based sources of satellite observations to create its Global Cloud Composite product in near real-time. This dataset allows others access to cloud information that can be accessed directly from the cloud. In this work, we describe the software algorithms and software infrastructure that we have created to create and distribute this product through our hybrid cloud and on-premises system that leverages the strengths and weaknesses of each platform. We will also describe the GCC product itself in terms of the scientific parameters available, resolution and temporal availability. Finally, we will also describe our web-based tools including our Geographic Information System that can be used for visualization and analysis. The products from the processing can be geospatially-service enabled as ArcGIS Image Services and Open Geospatial Consortium (OGC) Web Mapping/Coverage Services for visualization and analysis via a web mapping application and common geospatial software. The SatCORPS Global Composite Cloud product provides sophisticated global cloud products with very low latency which we see that as filling a rapidly growing need in the research, modelling and business community. We present a detailed description of the SatCORPS GCC product in terms of the capabilities and research benefits as well as the delivery architecture in AWS. We take the discussion further and describe the web tools that the SatCORPS group has developed that allow users to access and use the GIS data that it creates.

Global Cloud Composite SatCORPS SMCE AWS↗

Utilization of Streamtubes to Analyze the Physical Interaction of a Dispersed Cloud with the CRM65 Hybrid Midspan Model

Recently there have been numerous efforts to identify the relevant icing physics related to the formation of complex three dimensional features, sometimes referred to as ’scallops’, on swept wings. However, much of the physics is still not well understood. This paper computationally investigates the interaction of the icing cloud with the 65 percent Common Research Model (CRM65). Both the interaction with an uniced model, and the interaction with a representative simulated three dimensional ice accretion are analyzed. Preliminary results suggest that the liquid water content increases near the aerodynamic body. For small droplets on the uniced geometry, the particle velocity vector becomes nearly parallel with the aerodynamic model resulting in a low value of collection efficiency. When a three dimensional feature with a length scale much smaller than the leading edge of the airfoil is introduced into the flow, the impingement of small particles can become significantly more perpendicular to the particle velocity vector in the region of these features. Since the liquid water content near the body has increased due to the interaction with the larger scale features of the aerodynamic model, i.e. the leading edge of the swept wing, extremely high collection efficiency is observed. These results suggest that three dimensional features are likely a significant physical driver that should be modeled in some capacity when simulating the impingement of a cloud on an aerodynamic model.

Icing↗

Utilization of Streamtubes to Analyze the Physical Interaction of a Dispersed Cloud with the CRM65 Hybrid Midspan Model

Recently there have been numerous efforts to identify the relevant icing physics related to the formation of complex three dimensional features, sometimes referred to as ’scallops’, on swept wings. However, much of the physics is still not well understood. This paper computationally investigates the interaction of the icing cloud with the 65 percent Common Research Model (CRM65). Both the interaction with an uniced model, and the interaction with a representative simulated three dimensional ice accretion are analyzed. Preliminary results suggest that the liquid water content increases near the aerodynamic body. For small droplets on the uniced geometry, the particle velocity vector becomes nearly parallel with the aerodynamic model resulting in a low value of collection efficiency. When a three dimensional feature with a length scale much smaller than the leading edge of the airfoil is introduced into the flow, the impingement of small particles can become significantly more perpendicular to the particle velocity vector in the region of these features. Since the liquid water content near the body has increased due to the interaction with the larger scale features of the aerodynamic model, i.e. the leading edge of the swept wing, extremely high collection efficiency is observed. These results suggest that three dimensional features are likely a significant physical driver that should be modeled in some capacity when simulating the impingement of a cloud on an aerodynamic model.

Icing↗