Search NASA⌕ Search

SEARCH · Search NASA

Results for “cloud computing, data cloud”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

The integration of heterogeneous resources in the CMS Submission Infrastructure for the LHC Run 3 and beyond

While the computing landscape supporting LHC experiments is currently dominated by x86 processors at WLCG sites, this configuration will evolve in the coming years. LHC collaborations will be increasingly employing HPC and Cloud facilities to process the vast amounts of data expected during the LHC Run 3 and the future HL-LHC phase. These facilities often feature diverse compute resources, including alternative CPU architectures like ARM and IBM Power, as well as a variety of GPU specifications. Using these heterogeneous resources efficiently is thus essential for the LHC collaborations reaching their future scientific goals. The Submission Infrastructure (SI) is a central element in CMS Computing, enabling resource acquisition and exploitation by CMS data processing, simulation and analysis tasks. The SI must therefore be adapted to ensure access and optimal utilization of this heterogeneous compute capacity. Some steps in this evolution have been already taken, as CMS is currently using opportunistically a small pool of GPU slots provided mainly at the CMS WLCG sites. Additionally, Power9 processors have been validated for CMS production at the Marconi-100 cluster at CINECA. This note will describe the updated capabilities of the SI to continue ensuring the efficient allocation and use of computing resources by CMS, despite their increasing diversity. The next steps towards a full integration and support of heterogeneous resources according to CMS needs will also be reported.

Pérez-Calero Yzquierdo, Antonio↗
DOE OSTI2015

The CMS Tier0 goes cloud and grid for LHC Run 2

In 2015, CMS will embark on a new era of collecting LHC collisions at unprecedented rates and complexity. This will put a tremendous stress on our computing systems. Prompt Processing of the raw data by the Tier-0 infrastructure will no longer be constrained to CERN alone due to the significantly increased resource requirements. In LHC Run 2, we will need to operate it as a distributed system utilizing both the CERN Cloud-based Agile Infrastructure and a significant fraction of the CMS Tier-1 Grid resources. In another big change for LHC Run 2, we will process all data using the multi-threaded framework to deal with the increased event complexity and to ensure efficient use of the resources. Furthermore, this contribution will cover the evolution of the Tier-0 infrastructure and present scale testing results and experiences from the first data taking in 2015.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

A geospatial risk analysis graphical user interface for identifying hazardous chemical emission sources

Background: Performing back trajectory and forward trajectory using the Hybrid Single-Particle Lagrangian Integrated Trajectory Model (HYSPLIT) is a reliable approach for assessing particle transport after release among mid-field atmospheric models. HYSPLIT has an externally facing online interface that allows non-expert users to run the model trajectories without requiring extensive training or programming. However, the existing HYSPLIT interface is limited if simulations have a large amount of meteorological data and timesteps that are not coincident. The objective of this study is to design and develop a more robust tool to rapidly evaluate hazard transport conditions and to perform risk analysis, while still maintaining an intuitive and user-friendly interface. Methods: HYSPLIT calculates forward and backward trajectories of particles based on wind speed, wind direction, and the corresponding location, timestamp, and Pasquill stability classes of the regions of the atmosphere in terms of the wind speed, the amount of solar radiation, and the fractional cloud cover. The computed particle transport trajectories, combined with the online Proton Transfer Reaction-Mass Spectrometry (PTR-MS) data (https://figshare.com/articles/dataset/ARL_Data_from_PROS_station_at_Hanford_site/19993964), can be used to identify and quantify the sources and affected area of the hazardous chemicals’ emission using the potential source distribution function (PSDF). PSDF is an improved statistical function based on the well-known potential source contribution function (PSCF) in establishing the air pollutant source and receptor relationship. Performing this analysis requires a range of meteorological and pollutant concentration measurements to be statistically meaningful. The existing HYSPLIT graphical user interface (GUI) does not easily permit computations of trajectories of a dataset of meteorological data in high temporal frequency. To improve the performance of HYSPLIT computations from a large dataset and enhance risk analysis of the accidental release of material at risk, a geospatial risk analysis tool (GRAT-GUI) is created to allow large data sets to be processed instantaneously and to provide ease of visualization. Results: The GRAT-GUI is a native desktop-based application and can be run in any Windows 10 system without any internet access requirements, thus providing a secure way to process large meteorological datasets even on a standalone computer. GRAT-GUI has features to import, integrate, and convert meteorological data with various formats for hazardous chemical emission source identification and risk analysis as a self-explanatory user interface. The tool is available at https://figshare.com/articles/software/GRAT/19426742.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

FedADMP: A Joint Anomaly Detection and Mobility Prediction Framework via Federated Learning

With the proliferation of mobile devices and smart cameras, detecting anomalies and predicting their mobility are critical for enhancing safety in ubiquitous computing systems. Due to data privacy regulations and limited communication bandwidth, it is infeasible to collect, transmit, and store all data from mobile devices at a central location. To overcome this challenge, we propose FedADMP, a federated learning based joint Anomaly Detection and Mobility Prediction framework. FedADMP adaptively splits the training process between the server and clients to reduce computation loads on clients. To protect the privacy of user data, clients in FedADMP upload only intermediate model parameters to the cloud server. We also develop a differential privacy method to prevent the cloud server and external attackers from inferring private information during the model upload procedure. Extensive experiments using real-world datasets show that FedADMP consistently outperforms existing methods.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Are Marine Low Cloud Droplet Concentrations Buffered by Entrained Aitken‐Mode Aerosol (Final Technical Report)

During the summertime, the high-latitude oceans come to life with green phytoplankton, which gain their energy from sunlight and are food for sea creatures small and large. When the phytoplankton are eaten or die, sulfur-rich gases are released under the ocean surface and mix into the air. Observations suggest that, over the Southern Ocean, frequent storms lift this air high into the atmosphere while raining out particulates like salt. As a result, the sulfur-rich air then spawns high concentrations of small ‘Aitken-mode’ aerosol particles. We hypothesize that these particles work their way down into the marine boundary layer, where they can replenish the supply of cloud-condensation nuclei scavenged by frequent precipitation. Further, this process maintains high concentrations of liquid cloud droplets in austral summer, promoting more sunlight to be reflected to space. We call this ‘Aitken buffering’. The primary objective of this project has been to document and test what role Aitken-mode aerosols play in clouds over the Southern Ocean and elsewhere. This effort has included three main components: 1) developing a computer model that realistically simulates the aerosol processes and the small-scale turbulent air motions that move aerosols around and create the clouds, 2) using that model to interpret and extend these observations for process understanding, by allowing different factors that contribute to the aerosol budget, such as surface wind speed, precipitation, surface gas exchange, etc. to be separated, and 3) studying Aitken-mode aerosol and its variability with a focus over the Southern Ocean and Antarctica, using data from Atmospheric Radiation Measurement (ARM) sites and other available observations. Initial computer studies in more idealized conditions found that elevated concentrations of Aitken-mode aerosols above the clouds could help prevent the breakup of those clouds by acting as cloud-condensation nuclei after they were entrained into the cloudy boundary layer. Simulations of a day during the ACE-ENA field campaign showed that Aitken-mode aerosols could also prevent cloud breakup under more realistic conditions. A new method that extracts information about Aitken-mode aerosols from measurements of aerosols onto which cloud droplets can form finds that Aitken-mode aerosols do vary seasonally over the Southern Ocean, with a peak in summertime, as described above. Other work during this project has focused on understanding how patterns of water vapor, clouds and precipitation are coupled within low-lying clouds over the oceans, and also on how cloud droplets cluster within clouds and how the distribution of cloud droplet sizes change as dry air is mixed into clouds, with the latter studies also using observations from ACE-ENA.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2013

The W40 region in the gould belt: An embedded cluster and H II region at the junction of filaments

We present a multiwavelength study of the W40 star-forming region using infrared (IR) observations in the UKIRT JHK bands, Spitzer Infrared Array Camera bands, and Herschel PACS bands, 2.12 μm H{sub 2} narrowband imaging, and radio continuum observations from GMRT (610 and 1280 MHz), in a field of view (FoV) of ∼34' × 40'. Archival Spitzer observations in conjunction with near-IR observations are used to identify 1162 Class II/III and 40 Class I sources in the FoV. The nearest-neighbor stellar surface density analysis shows that the majority of these young stellar objects (YSOs) constitute the embedded cluster centered on the high-mass source IRS 1A South. Some YSOs, predominantly the younger population, are distributed along and trace the filamentary structures at lower stellar surface density. The cluster radius is measured to be 0.44 pc—matching well with the extent of radio emission—with a peak density of 650 pc{sup –2}. The JHK data are used to map the extinction in the region, which is subsequently used to compute the cloud mass—126 M {sub ☉} and 71 M {sub ☉} for the central cluster and the northern IRS 5 region, respectively. H{sub 2} narrowband imaging shows significant emission, which prominently resembles fluorescent emission arising at the borders of dense regions. Radio continuum analysis shows that this region has a blister morphology, with the radio peak coinciding with a protostellar source. Free-free emission spectral energy distribution analysis is used to obtain physical parameters of the overall photoionized region and the IRS 5 sub-region. This multiwavelength scenario is suggestive of star formation having resulted from the merging of multiple filaments to form a hub. Star formation seems to have taken place in two successive epochs, with the first epoch traced by the central cluster and the high-mass star(s)—followed by a second epoch that is spreading into the filaments as uncovered by the Class I sources and even younger protostellar sources along the filaments. The IRS 5 H II region displays indications of swept-up material that has possibly led to the formation of protostars.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2018

Direct 400 Vdc to 1 Vdc power conversion with input series output parallel connection for data center power supplies

Data centers consume an ever-increasing amount of electricity because of the rapid growth of cloud computing and digital information storage. A high voltage point of load (HV POL) converter is proposed to convert the 400-VDC distribution voltage to 1-VDC within a single stage to increase the power conversion efficiency. A six-phase input series output parallel (ISOP) connected structure is implemented for the HV POL. The symmetrical controlled half bridge current doubler is selected as the converter topology in the ISOP structure. The full load efficiency is improved by 4% points compared with state of the art products. A voltage compensator has been designed in order to meet the strict dynamic voltage regulation requirement. A laboratory prototype has been built, and experimental results have been provided to verify the proposed HV POL with a single power conversion stage can meet the dynamic voltage regulation requirement for an on-board power supply with higher efficiency compared to the conventional architecture.

Cui, Yutian↗
DOE OSTI2014

Cloud computing method for dynamically scaling a process across physical machine boundaries

A cloud computing platform includes first device having a graph or tree structure with a node which receives data. The data is processed by the node or communicated to a child node for processing. A first node in the graph or tree structure determines the reconfiguration of a portion of the graph or tree structure on a second device. The reconfiguration may include moving a second node and some or all of its descendant nodes. The second and descendant nodes may be copied to the second device.

Gillen, Robert E.↗
DOE OSTI2019

A Real-Time Vehicle-Specific Eco-Routing Model for On-Board Navigation Applications Capturing Transient Vehicle Behavior

The paper develops a real-time vehicle-specific eco-routing model for use in on-board navigation systems for both conventional fuel- and battery-powered electric vehicles. The model uses a novel link cost function that retains all microscopic transient behavior along a link by capturing within-link speed, acceleration, and road grade variations. Specifically, eight vehicle-agnostic link-specific variables are transmitted to the cloud and fused with information received from other connected vehicles. The fused data are then sent back to the connected vehicles and used to compute vehicle-specific eco-routes (edge computing). The solution produces an energy-efficient dynamic user-equilibrium traffic assignment that is solved incrementally. A numerical experiment is first designed to test the model, demonstrating the benefits associated with vehicle-specific eco-routes compared to those produced by traditional models. The model is then implemented in a micro-simulation framework to evaluate the network-wide impacts of the proposed energy-efficient user-optimum incremental traffic assignment for different traffic demand levels. Results demonstrate that the proposed model produces lower network-wide energy consumption levels compared to traditional energy-efficient and travel time user-optimum traffic assignment methods for both conventional fuel and electric vehicles. Conventional fuel savings relative to travel time routing decrease with increasing traffic demand levels, while electric power savings do not vary monotonically with congestion levels. Energy savings relative to traditional eco-routing are also demonstrated to not vary monotonically with demand levels. Results of the study also demonstrate that the transportation network layout significantly affects the eco-routing system performance. Finally, the feasibility of the proposed model in real-world applications is discussed and presented.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2020

Resource Management for Processing Wide Area Data Streams on Supercomputers

Modern scientific instruments generate enormous amount of data. Typically, the data collected from the instruments are stored in one or more files that are then moved to a distant supercomputer for processing. The final results are sent back to the user. In order to make effective use of the time on expensive instruments, experimenters want to process the data as they are generated. They want to stream the data from instruments' memory directly to a supercomputer's memory for analysis. Since the compute nodes in a supercomputer are not connected directly to the wide area network, the data streams need to be passed through intermediate gateway nodes. As opposed to the best effort file transfers, data streaming applications require resources at a specific time for a specific period. In this paper, we present a system model for enabling data streaming through gateway nodes and an algorithm to efficiently allocate gateway node resources along with compute nodes. We evaluate the algorithm using real-world traces on the Chameleon Cloud. The results show that our system can schedule compute and gateway resources efficiently for streaming analysis.

data streaming↗
DOE OSTI2012

Collaborative cloud-enabled tools allow rapid, reproducible biological insights

Microbial ecologists today face critical computational barriers. The rapid increase in the quantity of data acquired by modern sequencing instruments makes analysis by hand infeasible, and even software developed just a few years ago cannot scale to modern data sets. As a result, making advanced, scalable algorithms and large-scale computational resources available to end-users is necessary to advancing our understanding of microbial ecology.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2014

Karhunen-Loeve expansion analysis of uncertainties in cloud microphysical property retrievals

Here, this study proposes a methodology of quantifying uncertainties for cloud retrievals on model resolution to facilitate the comparison with model outputs. Primary component analysis is applied to reduce the dimension of random variables (up to a factor of 50) and reveal the cross correlations in the input data, making large sampling computationally feasible and uncertainty quantification accurate and reliable. Our approach has the capability of parameterizing input uncertainties and attributing the uncertainties in the retrieval output to each individual source, which allows sensitivity analysis of cloud retrieval algorithms and provides directions for improving observation instruments as well as strategies. We applied the method to characterize uncertainties in cloud ice water content (IWC) retrieved from the DOE Atmospheric Radiation Measurement (ARM) programs baseline cloud microphysical retrieval algorithm (MICROBASE). We test it with a selected ice cloud case observed on 9 March 2000 at the ARM Southern Great Plains site during its 2000 cloud intensive observing period. The test results indicate that (1) uncertainties in the output retrieved by MICROBASE are comparable amongst different retrievals; (2) The mean values obtained by our UQ method are closer to the aircraft data with less errors compared to the direct ensemble average; (3) Ice water path (IWP) generally incurred larger uncertainty in optically thin ice clouds and there was more variability in vertical in the retrieved IWC; and (4) Uncertainties in the output are mainly due to the interactions among different modes of ARM radar pro files.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2015

RACORO continental boundary layer cloud investigations. Part I: Case study development and ensemble large-scale forcings

Observation-based modeling case studies of continental boundary layer clouds have been developed to study cloudy boundary layers, aerosol influences upon them, and their representation in cloud- and global-scale models. Three 60-hour case study periods span the temporal evolution of cumulus, stratiform, and drizzling boundary layer cloud systems, representing mixed and transitional states rather than idealized or canonical cases. Based on in-situ measurements from the RACORO field campaign and remote-sensing observations, the cases are designed with a modular configuration to simplify use in large-eddy simulations (LES) and single-column models. Aircraft measurements of aerosol number size distribution are fit to lognormal functions for concise representation in models. Values of the aerosol hygroscopicity parameter, κ, are derived from observations to be ~0.10, which are lower than the 0.3 typical over continents and suggestive of a large aerosol organic fraction. Ensemble large-scale forcing datasets are derived from the ARM variational analysis, ECMWF forecasts, and a multi-scale data assimilation system. The forcings are assessed through comparison of measured bulk atmospheric and cloud properties to those computed in 'trial' large-eddy simulations, where more efficient run times are enabled through modest reductions in grid resolution and domain size compared to the full-sized LES grid. Simulations capture many of the general features observed, but the state-of-the-art forcings were limited at representing details of cloud onset, and tight gradients and high-resolution transients of importance. Methods for improving the initial conditions and forcings are discussed. The cases developed are available to the general modeling community for studying continental boundary clouds.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2015

Mesh-free data transfer algorithms for partitioned multiphysics problems: Conservation, accuracy, and parallelism

In this study we analyze and extend mesh-free algorithms for three-dimensional data transfer problems in partitioned multiphysics simulations. We first provide a direct comparison between a mesh-based weighted residual method using the common-refinement scheme and two mesh-free algorithms leveraging compactly supported radial basis functions: one using a spline interpolation and one using a moving least square reconstruction. Through the comparison we assess both the conservation and accuracy of the data transfer obtained from each of the methods. We do so for a varying set of geometries with and without curvature and sharp features and for functions with and without smoothness and with varying gradients. Our results show that the mesh-based and mesh-free algorithms are complementary with cases where each was demonstrated to perform better than the other. We then focus on the mesh-free methods by developing a set of algorithms to parallelize them based on sparse linear algebra techniques. This includes a discussion of fast parallel radius searching in point clouds and restructuring the interpolation algorithms to leverage data structures and linear algebra services designed for large distributed computing environments. The scalability of our new algorithms is demonstrated on a leadership class computing facility using a set of basic scaling studies. Finally, these scaling studies show that for problems with reasonable load balance, our new algorithms for both spline interpolation and moving least square reconstruction demonstrate both strong and weak scalability using more than 100,000 MPI processes with billions of degrees of freedom in the data transfer operation.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Framework for Analysis and Visualization of Simulation Data (VizAly)

As supercomputing resources increase, cosmological scientists are able to run more detailed and larger simulations generating massive amounts of data. Analyzing these simulations with an available open-source toolkit is important for collaborative Department of Energy scientific discovery across labs, universities, and other partners. Developed software as a part of this collection include: comparing data with other existing simulations, verifying and validating results with observation databases, new halo finder algorithms, and using analytical tools to get insights into the physics of the cosmological universe. The goal of this software project is to provide a set of open-source libraries, tools, and packages for large-scale cosmology that allows scientists to visualize, analyze, and compare large-scale simulation and observational data sets. Developed software will provide a variety of methods for processing, visualization, and analysis of astronomical observation and cosmological simulation data. These tools are intended for deployment on multiple scientific computing platforms, including but not limited to personal computers, cloud computing, experimental sites (telescopes) and high performance supercomputers.

Wendelberger, Joanne↗
DOE OSTI2022

The Astronomy Commons Platform: A Deployable Cloud-based Analysis Platform for Astronomy

Abstract We present a scalable, cloud-based science platform solution designed to enable next-to-the-data analyses of terabyte-scale astronomical tabular data sets. The presented platform is built on Amazon Web Services (over Kubernetes and S3 abstraction layers), utilizes Apache Spark and the Astronomy eXtensions for Spark for parallel data analysis and manipulation, and provides the familiar JupyterHub web-accessible front end for user access. We outline the architecture of the analysis platform, provide implementation details and rationale for (and against) technology choices, verify scalability through strong and weak scaling tests, and demonstrate usability through an example science analysis of data from the Zwicky Transient Facility’s 1Bn+ light-curve catalog. Furthermore, we show how this system enables an end user to iteratively build analyses (in Python) that transparently scale processing with no need for end-user interaction. The system is designed to be deployable by astronomers with moderate cloud engineering knowledge, or (ideally) IT groups. Over the past 3 yr, it has been utilized to build science platforms for the DiRAC Institute, the ZTF partnership, the LSST Solar System Science Collaboration, and the LSST Interdisciplinary Network for Collaboration and Computing, as well as for numerous short-term events (with over 100 simultaneous users). In a live demo instance, the deployment scripts, source code, and cost calculators are accessible. 4 4 http://hub.astronomycommons.org/

79 ASTRONOMY AND ASTROPHYSICS↗