Search NASA⌕ Search

SEARCH · Search NASA

Results for “hybrid cloud”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting

Hybrid cloud bursting (i.e., leasing temporary off-premise cloud resources to boost the overall capacity during peak utilization) can be a cost-effective way to deal with the increasing complexity of big data analytics, especially for iterative applications. However, the low throughput, high latency network link between the on-premise and off-premise resources (“weak link”) makes maintaining scalability difficult. While several data locality techniques have been designed for big data bursting on hybrid clouds, their effectiveness is difficult to estimate in advance. Yet such estimations are critical, because they help users decide whether the extra pay-as-you-go cost incurred by using the off-premise resources justifies the runtime speed-up. To this end, the current paper presents a performance model and methodology to estimate the runtime of iterative MapReduce applications in a hybrid cloud-bursting scenario. The paper focuses on the overhead incurred by the weak link at fine granularity, for both the map and the reduce phases. This approach enables high estimation accuracy, as demonstrated by extensive experiments at scale using a mix of real-world iterative MapReduce applications from standard big data benchmarking suites that cover a broad spectrum of data patterns. As a result, not only are the produced estimations accurate in absolute terms compared with experimental results, but they are also up to an order of magnitude more accurate than applying state-of-art estimation approaches originally designed for single-site MapReduce deployments.

97 MATHEMATICS AND COMPUTING↗

Understanding the Performance and Potential of Cloud Computing for Scientific Applications

In this paper, commercial clouds bring a great opportunity to the scientific computing area. Scientific applications usually require significant resources, however not all scientists have access to sufficient high-end computing systems, may of which can be found in the Top500 list. Cloud Computing has gained the attention of scientists as a competitive resource to run HPC applications at a potentially lower cost. But as a different infrastructure, it is unclear whether clouds are capable of running scientific applications with a reasonable performance per money spent. This work studies the performance of public clouds and places this performance in context to price. We evaluate the raw performance of different services of AWS cloud in terms of the basic resources, such as compute, memory, network and I/O. We also evaluate the performance of the scientific applications running in the cloud. This paper aims to assess the ability of the cloud to perform well, as well as to evaluate the cost of the cloud running scientific applications. We developed a full set of metrics and conducted a comprehensive performance evlauation over the Amazon cloud. We evaluated EC2, S3, EBS and DynamoDB among the many Amazon AWS services. We evaluated the memory sub-system performance with CacheBench, the network performance with iperf, processor and network performance with the HPL benchmark application, and shared storage with NFS and PVFS in addition to S3. We also evaluated a real scientific computing application through the Swift parallel scripting system at scale. Armed with both detailed benchmarks to gauge expected performance and a detailed monetary cost analysis, we expect this paper will be a recipe cookbook for scientists to help them decide where to deploy and run their scientific applications between public clouds, private clouds, or hybrid clouds.

97 MATHEMATICS AND COMPUTING↗

Jay : A software framework for prototyping and evaluating offloading applications in hybrid edge clouds

Abstract We present Jay , a software framework for offloading applications in hybrid edge clouds. Jay provides an API, services, and tools that enable mobile application developers to implement, instrument, and evaluate offloading applications using configurable cloud topologies, offloading strategies, and job types. We start by presenting Jay 's job model and the concrete architecture of the framework. We then present the programming API with several examples of customization. Then, we turn to the description of the internal implementation of Jay instances and their components. Finally, we describe the Jay Workbench, a tool that allows the setup, execution, and reproduction of experiments with networks of hosts with different resource capabilities organized with specific topologies. The complete source code for the framework and workbench is provided in a GitHub repository.

Silva, Joaquim↗

EDX Spatial: Leveraging cloud and hybrid data management resources for spatial data

In the last few years, the National Energy Technology Laboratory has started leveraging cloud-hosted services for hosting spatial data collections published on the Energy Data eXchange (EDX). Using cloud-hosted storage and compute options offers multiple benefits for visualization and tool development through utilization of spatial data resources. The transition from on-premises services to cloud-hosted services has enabled a few key features: increased accessibility of large derivative datasets, dynamic integration of external authoritative data resources directly from outside entities through representational state transfer application programming interfaces (REST API), and the ability to produce complex mapping applications, dashboards, and online maps. As a result of the shift, NETL has launched EDX Spatial, a platform that leverages on-premises resources combined with cloud compute capabilities to enable enhanced online mapping interfaces and optimize data access. In addition, a unified workflow for handling the public release of spatial data products through EDX has been developed, including standardization of data hosting practices, metadata, symbology, and application elements. This poster reviews the opportunity of leveraging cloud-hosted and hybrid data management solutions for spatial data, and discusses the benefits and lessons learned while leveraging these services through the data repository EDX.

Morkner, Paige↗

NREL's Journey with HPC in the Cloud and Hybrid Computing

This is a planned lightning talk at the NLIT Summit 2025 conference. This would serve as somewhat of a progress update to the presentation I gave at re:Invent 2024 back in November which can be seen here: https://www.youtube.com/watch?t=2133&v=NMq3kL9qObU&feature=youtu.be (my section begins at the included timestamp value). This presentation discusses our usage of Cloud-hosted HPC systems, and in what circumstances they benefit our researchers strategically. We have been making incremental progress in this area since that recording, so for this presentation I would include our latest experiences and observations as we are beginning to implement a hybrid HPC solution. We're in the midst of a cross-team effort of implementing a prototype hybridization solution which would allow users to strategically burst jobs to the cloud. In this talk for NLIT, I would detail lessons-learned, non-starters, architecture diagrams, and other implementation details that may benefit those interested as we continue our experimentation. Our prototype may not be complete by the time of this presentation, but even in the discovery phase of our anticipated design we've discovered a lot of information from others who have worked on hybrid solutions that are worth sharing.

97 MATHEMATICS AND COMPUTING↗

Exploration of cloud computing late start LDRD #149630 : Raincoat. v. 2.1.

This report contains documentation from an interoperability study conducted under the Late Start LDRD 149630, Exploration of Cloud Computing. A small late-start LDRD from last year resulted in a study (Raincoat) on using Virtual Private Networks (VPNs) to enhance security in a hybrid cloud environment. Raincoat initially explored the use of OpenVPN on IPv4 and demonstrates that it is possible to secure the communication channel between two small 'test' clouds (a few nodes each) at New Mexico Tech and Sandia. We extended the Raincoat study to add IPSec support via Vyatta routers, to interface with a public cloud (Amazon Elastic Compute Cloud (EC2)), and to be significantly more scalable than the previous iteration. The study contributed to our understanding of interoperability in a hybrid cloud.

99 GENERAL AND MISCELLANEOUS↗

Combined Optical and Radio-Frequency Perspectives on a Hybrid Cloud-To-Ground Lightning Flash Observed by the FORTE Satellite

In this work, we use the coincident optical and radio-frequency (RF) measurements taken by the Fast On-orbit Recording of Transient Events (FORTE) satellite to shed light on common optical signatures recorded by NASA and NOAA lightning imagers during Cloud-to-Ground (CG) lightning. We build flash cluster data for FORTE using the same techniques as the NASA/NOAA instruments to document the optical/RF evolution of an oceanic hybrid -CG flash. The flash began with strong Very High Frequency (VHF)-band emission from a Narrow Bipolar Event (NBE) that initiated a period of normal bilevel intracloud (IC) activity in two vertical layers (8 and 12 km) that lasted for 490 ms. VHF waveforms show step leader activity ahead of the return stroke. All impulsive VHF sources after the stroke come from the lower (8 km) layer. K-changes are noted following the return stroke, but no subsequent strokes are detected. The optical flash began 136 ms after the NBE RF pulse. Twenty-two of the 33 optical groups were dim and occurred during the in-cloud phase of the flash. This activity included both isolated pulses and sustained periods of illumination over tens of milliseconds. Initial cloud pulses accounted for 23% of the total optical radiance from the flash. Illumination during the return stroke contributed 58% of the total radiance, and the K-changes and cloud pulses after the stroke supplied the remaining 19%. These results highlight the benefit of having RF alongside optical lightning measurements for clarifying signatures in the optical data and providing information on their physical origins.

54 ENVIRONMENTAL SCIENCES↗

A Cybersecurity Threat Profile for a Connected Lighting System

In anticipation of improved energy performance and cost savings, cities and building owners are increasingly considering “smart lighting initiatives” that aim to convert their collection of simple luminaires (i.e., lighting fixtures) into an intelligent connected lighting system (CLS) capable of remotely monitoring energy consumption and fault conditions, and possibly implementing adaptive lighting schemes. The U.S. Department of Energy (DOE) has set an national goal of tripling the energy efficiency and demand flexibility of the buildings sector by 2030, relative to 2020 levels 1. It is forecast that connected lighting systems can contribute to that goal by delivering 125 TWh of annual energy savings by 2035 2, equivalent to the annual output of 50 typical (500 MW) power plants. However, these energy savings and the DOE goal are put at significant risk if connected technologies are not adopted due to real or perceived cybersecurity concerns. Connected IoT devices such as these have historically been rife with vulnerabilities which sometimes put security considerations secondary to functionality and operability. What are the cybersecurity threats that will impact these systems, as formerly banal luminaires transition into intelligent connected devices that collect information about themselves, their surrounding environment, and possibly us? In this paper we analyze a threat profile performed on a fault-detection use case for streetlights. A threat profile establishes security requirements, justifies security measures, yields actionable controls, and effectively communicates risk to stakeholders. This effort provides critical information for making threat-based decisions to increase security at a reasonable cost, and can effectively be used by development teams, software architects, and managers to make cybersecurity a part of their ongoing culture of awareness, training, and prevention. This leads to more secure systems and better-understood security. On-premise, cloud, and hybrid architectures with different authentication mechanisms were modeled and later categorized using the Microsoft STRIDE framework. An analysis of the recommended controls for each threat was performed to determine which controls could and should be put in place by manufacturers or third-party suppliers, and which controls need to be left up the end-user to implement. Fifty-seven threats were identified. Among our key findings: (1) 65% (37/57) of the threats did not involve the luminaires, but rather the other components needed to communicate with and manage them; (2) 63% (36/57) of the threats could have been mitigated through manufacturer-implemented defensive techniques or “controls”; and (3) 23% (13/57) of the threats were dependent on the network configuration. Recommendations based on the results of this work are made to key stakeholder groups. Notably, lighting technology developers are advised to address all threats that can be reasonably controlled with baked-in technology solutions (e.g., encryption or authentication controls), and employ some form of secure supply chain management and tracking where other parts (e.g., sensors, microprocessors) of a luminaire must also be built and manufactured with the proper security controls in place. Developers should also review threats involving assets not developed in-house to understand how connectivity with other devices will affect their product during system operation and determine if a compensating control for a defense-in-depth strategy will be needed. Finally, those interested in deploying CLS should compare the differences between cloud and on-premise models to determine which is more suitable for their needs and the abilities of their security team.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

A Cybersecurity Threat Profile for a Connected Lighting System

In this paper we analyze a threat profile performed on a fault-detection use case for streetlights. A threat profile establishes security requirements, justifies security measures, yields actionable controls, and effectively communicates risk to stakeholders. This effort provides critical information for making threat-based decisions to increase security at a reasonable cost, and can effectively be used by development teams, software architects, and managers to make cybersecurity a part of their ongoing culture of awareness, training, and prevention. This leads to more secure systems and better-understood security. On-premise, cloud, and hybrid architectures with different authentication mechanisms were modeled and later categorized using the Microsoft STRIDE framework. An analysis of the recommended controls for each threat was performed to determine which controls could and should be put in place by manufacturers or third-party suppliers, and which controls need to be left up the end-user to implement.

97 MATHEMATICS AND COMPUTING↗

Image-Driven Hybrid Structural Analysis Based on Continuum Point Cloud Method with Boundary Capturing Technique

Conventional approaches for the structural health monitoring of infrastructures often rely on physical sensors or targets attached to structural members, which require considerable preparation, maintenance, and operational effort, including continuous on-site adjustments. This paper presents an image-driven hybrid structural analysis technique that combines digital image processing (DIP) and regression analysis with a continuum point cloud method (CPCM) built on a particle-based strong formulation. Polynomial regressions capture the boundary shape change due to the structural loading and precisely identify the edge and corner coordinates of the deformed structure. The captured edge profiles are transformed into essential boundary conditions. This allows the construction of a strongly formulated boundary value problem (BVP), classified as the Dirichlet problem. Capturing boundary conditions from the digital image is novel, although a similar approach was applied to the point cloud data. It was shown that the CPCM is more efficient in this hybrid simulation framework than the weak-form-based numerical schemes. Unlike the finite element method (FEM), it can avoid aligning boundary nodes with regression points. A three-point bending test of a rubber beam was simulated to validate the developed technique. The simulation results were benchmarked against numerical results by ANSYS and various relevant numerical schemes. The technique can effectively solve the Dirichlet-type BVP, yielding accurate deformation, stress, and strain values across the entire problem domain when employing a linear strain model and increasing the number of CPCM nodes. In addition, comparative analysis with conventional displacement tracking techniques verifies the developed technique’s robustness. The proposed technique effectively circumvents the inherent limitations of traditional monitoring methods resulting from the reliance on physical gauges or target markers so that a robust and non-contact solution for remote structural health monitoring in real-scale infrastructures can be provided, even in unfavorable experimental environments.

Chemistry↗

A hybrid approach to estimate the complex motions of clouds in sky images

Tracking the motion of clouds is essential to forecasting the weather and to predicting the short-term solar energy generation. Existing techniques mainly fall into two categories: variational optical flow, and block matching. In this article, we summarize recent advances in estimating cloud motion using ground-based sky imagers and quantitatively evaluate state-of-the-art approaches. Then we propose a hybrid tracking framework to incorporate the strength of both block matching and optical flow models. To validate the accuracy of the proposed approach, we introduce a series of synthetic images to simulate the cloud movement and deformation, and thereafter comprehensively compare our hybrid approach with several representative tracking algorithms over both simulated and real images collected from various sites/imagers. The results show that our hybrid approach outperforms state-of-the-art models by reducing at least 30% motion estimation errors compared with the ground-truth motions in most of simulated image sequences. Furthermore, our hybrid model demonstrates its superior efficiency in several real cloud image datasets by lowering at least 15% Mean Absolute Error (MAE) between predicted images and ground-truth images.

54 ENVIRONMENTAL SCIENCES↗

The disCO2ver Platform: Curating Data and Tools for Geologic Carbon Sequestration and Deep Subsurface Research Systems

The U.S. DOE National Energy Technology Laboratory has invested 12+ years of development into the data repository and digital laboratory, the Energy Data eXchange (EDX, edx.netl.doe.gov). Supporting a variety of research areas across the DOE Office of Fossil Energy and Carbon Management, the platform has successfully curated and preserved thousands of data products from DOE research. The Carbon Storage Program has successfully supported data curation, upload, and publishing of data products on EDX for many years, demonstrating a success story of how resources like EDX can effectively help with long term preservation and publishing of DOE data products. EDX continues to shift towards cloud-supported infrastructure, taking a hybrid approach combining on-premises compute and storage integrated with cloud-hosted services. The integration of cloud compute and hybrid architecture enables the development of EDX-hosted platforms that tailor the data and tools hosted on them to a specific community, enables implementation of machine learning tools for data discovery and filtering, and enables the hosting of virtual (online user interface) tools. Geologic carbon sequestration (GCS) research continues to scale up in response to the current administration goals to reduce greenhouse gas emissions and transition the energy economy. Over the last year, EDX’s disCO2ver platform has been developed in response to the need for access to data products and tools to support the scaling up of GCS research. disCO2ver provides access to data resources and tools, produced by DOE and outside authoritative external resources. The platform also provides a user-access control component for the virtualization and cloud hosting of tools. Tools that need to be virtualized, to eliminate the need for users to download the tool and use local compute resources, is essential to supporting big-data analysis and machine learning that is becoming common place in carbon storage modeling, risk analysis, and data publishing practices. This talk will review the EDX’s disCO2ver platform and the current work ongoing to curate data and tools to support GCS and deep subsurface systems research.

Morkner, Paige↗

A Decadal Hybrid GCM Simulation Using Deep‐Learning‐Based Cloud and Convection Parameterization Generalized to a Warm Climate

A critical challenge for machine‐learning (ML) parameterization in global climate models (GCMs) is to achieve stable, accurate simulations under climates not seen during training. Previous studies have demonstrated promising offline performance and year‐long online stability in aquaplanet simulations but have encountered difficulties in real geography and under climate warming. Here we report that a GCM with real geography configuration using neural‐network‐based cloud and convection parameterization, trained exclusively with present‐day climate data, successfully performs a stable, decade‐long simulation of a warm climate with +4 K sea surface temperature (SST). The neural network (NN) is based on Han et al. (2023, https://doi.org/10.1029/2022ms003508 ) with additional inputs. The simulation captures the global precipitation distribution, surface temperatures, vertical atmospheric structures, and extreme precipitation very well, closely matching simulations from both the superparameterized CAM (SPCAM) and the conventional CAM5 in the warm climate without accuracy degradation compared to those in the baseline climate. Moreover, it produces a climate response to +4 K SST in atmospheric thermodynamic states and circulations similar to those from SPCAM and CAM5. Prognostic ablation tests on NN input variables show that the NN without convective memory as input suffers from numerical instability, and the NN without considering radiative variables and land fraction as input, or with reduced training samples produce less accurate results. To our knowledge, this is the first time an ML parameterization successfully achieves online extrapolation to a warm climate without using additional warm‐climate data for training. It demonstrates the potential of ML‐driven parameterizations for credible long‐term climate projections.

Atmosphere model↗

NREL Stratus - Enabling Workflows to Fuse Data Streams, Modeling, Simulation, and Machine Learning

Integrating cloud services into advanced computing facilities provides significant new capabilities over focusing solely on traditional high performance computing (HPC) workloads. This brings complementary capabilities as well as enabling new focused roles for HPC. They are especially potent for workflows that fuse data streams, modeling and simulation ('modsim') and machine learning. A key challenge to adopting a hybrid edge-cloud-HPC model is to align optimal capability, data, and user intent on the right resources for each step in a workflow.?The NREL Stratus service provides a basis for this: Stratus layers capabilities needed to make?cloud services accessible to a lab-based scientific community on commercial offerings, and; currently supports upwards of 200 projects ranging from IOT integration to traditional modeling and simulation. This provides a real-world inventory of scientific workflow elements. A growing knowledge base enables placing these elements appropriately between the edge, cloud, and traditional HPC. This paper outlines a vision via reference architecture and the application of that architecture in a typical workflow highlighting multiple components: sensor data intake, cleaning and transforming (edge/cloud suitable); generation of synthetic data through modsim, computationally heavy ML training and hyperparameter optimization (HPC suitable), and; inference and deployment (cloud ideal). Every step in such a workflow involves a cost-benefit analysis regarding the data movement, computational efficiency, availability, latency, and resource capabilities. The reference architecture and examples outlined allow for understanding new opportunities in the context of emerging workflows that combine IOT, cloud, and HPC to bolster scientific productivity.

AI↗

28 NREL Stratus - Enabling Workflows to Fuse Data Streams, Modeling, Simulation, and Machine Learning: Preprint

Integrating cloud services into advanced computing facilities provides significant new capabilities over focusing solely on traditional high performance computing (HPC) workloads. This brings complementary capabilities as well as enabling new focused roles for HPC. They are especially potent for workflows that fuse data streams, modeling and simulation ('modsim') and machine learning. A key challenge to adopting a hybrid edge-cloud-HPC model is to align optimal capability, data, and user intent on the right resources for each step in a workflow.?The NREL Stratus service provides a basis for this: Stratus layers capabilities needed to make?cloud services accessible to a lab-based scientific community on commercial offerings, and; currently supports upwards of 200 projects ranging from IOT integration to traditional modeling and simulation. This provides a real-world inventory of scientific workflow elements. A growing knowledge base enables placing these elements appropriately between the edge, cloud, and traditional HPC. This paper outlines a vision via reference architecture and the application of that architecture in a typical workflow highlighting multiple components: sensor data intake, cleaning and transforming (edge/cloud suitable); generation of synthetic data through modsim, computationally heavy ML training and hyperparameter optimization (HPC suitable), and; inference and deployment (cloud ideal). Every step in such a workflow involves a cost-benefit analysis regarding the data movement, computational efficiency, availability, latency, and resource capabilities. The reference architecture and examples outlined allow for understanding new opportunities in the context of emerging workflows that combine IOT, cloud, and HPC to bolster scientific productivity.

AI↗