Search NASA⌕ Search

SEARCH · Search NASA

Results for “Amazon Web Services”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

HEPCloud, a New Paradigm for HEP Facilities: CMS Amazon Web Services Investigation

Historically, high energy physics computing has been performed on large purpose-built computing systems. These began as single-site compute facilities, but have evolved into the distributed computing grids used today. Recently, there has been an exponential increase in the capacity and capability of commercial clouds. Cloud resources are highly virtualized and intended to be able to be flexibly deployed for a variety of computing tasks. There is a growing interest among the cloud providers to demonstrate the capability to perform large-scale scientific computing. In this paper, we discuss results from the CMS experiment using the Fermilab HEPCloud facility, which utilized both local Fermilab resources and virtual machines in the Amazon Web Services Elastic Compute Cloud. We discuss the planning, technical challenges, and lessons learned involved in performing physics workflows on a large-scale set of virtualized resources. Additionally, we will discuss the economics and operational efficiencies when executing workflows both in the cloud and on dedicated resources.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2018

A comparison of Amazon Web Services and Microsoft Azure cloud platforms for high performance computing

Advances in commercial cloud computing necessitate continual evaluation of the cloud's performance on a variety of applications. This work looks at compute oriented instances from Amazon Web Services and Microsoft Azure cloud platforms and evaluates them with several high-performance computing benchmarks, including HPCC and HPCG. These benchmarks illustrate that the most cost competitive solution depends on the application to be run.

Kotas, Charlotte↗
DOE OSTI2022

Development of an end state vision to implement digital monitoring in nuclear plants

Transitioning from an onsite Maintenance & Diagnostics Center to cloud-based services offers many new opportunities with computing power and storage, but also new challenges in terms of networking and security. This report will cover everything required for that transition including data processing and uploading to cloud services, feature selection, model creation, and result visualization for decision making. Although there are several other cloud-based services (e.g. Amazon Web Services and Google Cloud), this report explores Microsoft Azure to simplify nomenclature and maintain a consistent focus. Many of the services offered by Microsoft Azure are also available in the other cloud-based services, and their differences have been recorded in other literature. The Azure services most important to a nuclear power plant including networking & security, storage & databases, and Artificial Intelligence (AI) are reviewed here. Networking covers all aspects related to communication to Azure resources including security, privacy, and redundancy. Storage & databases includes data storage, upgrading, patching, backups, and monitoring. The AI services allows the user access to the machine learning (ML) techniques developed with Azure including automated ML, anomaly detection, computer vision, and natural language processing. This report summaries the features, capabilities, and challenges when using cloud-based services in a user-friendly manner.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2024

Secure Federated Learning Across Heterogeneous Cloud and High-Performance Computing Resources: A Case Study on Federated Fine-Tuning of LLaMA 2

Federated learning enables multiple data owners to collaboratively train robust machine learning models without transferring large or sensitive local datasets by only sharing the parameters of the locally trained models. Here, in this article, we elaborate on the design of our Advanced Privacy-Preserving Federated Learning (APPFL) framework, which streamlines end-to-end secure and reliable federated learning experiments across cloud computing facilities and high-performance computing resources by leveraging Globus Compute, a distributed function as a service platform, and Amazon Web Services. We further demonstrate the use case of APPFL in fine-tuning an LLaMA 2 7B model using several cloud resources and supercomputers.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

Inferring Netflow Data

Netflow data is a specific format for looking at network traffic. With this format, many security tests are easy to run. Cloud Service Providers produce logs that do not contain all of the content that Netflow data contains. There is a possibility that the fields that are in Netflow data but absent in Cloud Service Provider logs can be inferred from the fields that are present in the Cloud Service Provider logs. By inferring the missing data fields, the same test that can be run with Netflow data would be possible to run on the logs produced by Cloud Service Providers. Cloud Service Providers include Amazon Web Services and Box.com. There are multiple different types of Cloud Services, and each provider handles them differently and produces different logs. There are IaaS (meaning infrastructure as a service), Saas (meaning software as a service), and Paas (meaning platform as a service). Each provides a different use for the user.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

User Access to Scientific Facilities via 5G: A Cyber Security Thought Experiment

5G is more than an over-the-air radio technology upgrade. It is a strategy to extend Mobile Network Operator service offerings beyond traditional voice, instant messaging and Internet access. 5G Mobile Network Operators will offer new telecommunication services that include enhanced guarantees of confidentiality, integrity and availability. How could such services change the way Science collaborations connect scientists to supercomputers and other scientific facilities? Current scientific collaborations implicitly trust cloud service providers to securely store and process data. The perceived risks of outsourcing Science data security are counterbalanced by assurances that cloud providers operate at a scale that allows them to implement security measures impractical for Science collaborations (e.g. continuous system administrator behavioral monitoring and strict individual separation of duties). If that is true for a cloud service provider like Amazon Web Services (2018 revenue: $25.7 billion), could it also be true for Mobile Network Operators like Verizon Wireless (2018 revenue: $91.7 billion) or AT&T Mobility (2018 revenue: $71.3 billion)? DOE Leadership Class supercomputer facility users currently access them from the public Internet via Secure Shell. The sponsors and operators of the supercomputer facilities have determined that the public Internet path between the Scientist’s Device and the Login Node does not natively provide enough confidentiality or integrity to protect those communications. Therefore, the facilities achieve additional confidentiality and integrity by requiring Secure Shell encryption across those untrusted network paths. Using 5G Network Slice technology, a Mobile Network Operator may offer communication services between supercomputer users and facilities that natively provide confidentiality and integrity guarantees. Sponsors and operators of supercomputer facilities may determine that these guarantees provide enough confidentiality and integrity to protect those communications. If so, a 5G Network Slice could replace an SSH session running over the public Internet. Finally, this use case could be extended to other Office of Science user facility access requirements. Consider microscopy instruments at (e.g.) the Center for Nanoscale Materials or the Environmental Molecular Sciences Laboratory. The embedded systems controlling such instruments may not always support encrypted network access technologies like SSH. 5G Network Slices may offer an alternative to current VPN or SSH tunneling techniques, with additional benefits like guaranteed minimum bandwidth.

5G↗
DOE OSTI2024

Accelerating science: The usage of commercial clouds in ATLAS Distributed Computing

The ATLAS experiment at CERN is one of the largest scientific machines built to date and will have ever growing computing needs as the Large Hadron Collider collects an increasingly larger volume of data over the next 20 years. ATLAS is conducting R&D projects on Amazon Web Services and Google Cloud as complementary resources for distributed computing, focusing on some of the key features of commercial clouds: lightweight operation, elasticity and availability of multiple chip architectures. The proof of concept phases have concluded with the cloud-native, vendoragnostic integration with the experiment’s data and workload management frameworks. Google Cloud has been used to evaluate elastic batch computing, ramping up ephemeral clusters of up to O(100k) cores to process tasks requiring quick turnaround. Amazon Web Services has been exploited for the successful physics validation of the Athena simulation software on ARM processors. We have also set up an interactive facility for physics analysis allowing endusers to spin up private, on-demand clusters for parallel computing with up to 4 000 cores, or run GPU enabled notebooks and jobs for machine learning applications. The success of the proof of concept phases has led to the extension of the Google Cloud project, where ATLAS will study the total cost of ownership of a production cloud site during 15 months with 10k cores on average, fully integrated with distributed grid computing resources and continue the R&D projects.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2022

Scalability Testing Approach for Internet of Things for Manufacturing SQL and NoSQL Database Latency and Throughput

The proliferation of low-cost sensors and industrial data solutions has continued to push the frontier of manufacturing technology. Machine learning and other advanced statistical techniques stand to provide tremendous advantages in production capabilities, optimization, monitoring, and efficiency. The tremendous volume of data gathered continues to grow, and the methods for storing the data are critical underpinnings for advancing manufacturing technology. This work aims to investigate the ramifications and design tradeoffs within a decoupled architecture of two prominent database management systems (DBMS): sql and NoSQL. A representative comparison is carried out with Amazon Web Services (AWS) DynamoDB and AWS Aurora MySQL. The technologies and accompanying design constraints are investigated, and a side-by-side comparison is carried out through high-fidelity industrial data simulated load tests using metrics from a major US manufacturer. The results support the use of simulated client load testing for comparing the latency of database management systems as a system scales up from the prototype stage into production. As a result of complex query support, MySQL is favored for higher-order insights, while NoSQL can reduce system latency for known access patterns at the expense of integrated query flexibility. Here, by reviewing this work, a manufacturer can observe that the use of high-fidelity load testing can reveal tradeoffs in IoTfM write/ingestion performance in terms of latency that are not observable through prototype-scale testing of commercially available cloud DB solutions.

AWS↗
DOE OSTI2023

Multi-decade high-resolution regional hindcasts for wave energy resource characterization in U.S. coastal waters

Long-term, high-resolution, regional wave hindcast datasets were generated using unstructured-grid Simulating WAves Nearshore (SWAN) models for the U.S. coastal waters to support nearshore wave energy development in the U.S. including those bordering U.S. territorial islands. The model domains resolved the entire U.S. exclusive economic zones, with a spatial resolution of approximately 200 m nearshore. The regional SWAN models were driven by the global WAVEWATCH III® model outputs and run for a 42-year period from 1979 to 2020. Extensive model validations were performed using buoy observations and altimeter data. Regional resource characterization was performed based on hindcast data points at 2 km from shore and along the 100 m isobath. Aggregations of wave resource parameters were produced, and spatial and seasonal variations were analyzed for all the regions. Wave resource metrics recommended by international standards, including a 3-hour time series of six resource parameters, hourly frequency- and directionally resolved wave spectra at selected “virtual buoy” locations, and average-annual values of omni-directional wave power, significant wave height, and energy period are publicly disseminated through an Amazon Web Service and a Marine Energy Atlas web application tool to facilitate wave energy research and a wide range of coastal ocean applications.

16 TIDAL AND WAVE POWER↗
DOE OSTI2018

Leveraging Technology Services for Public Good

As the technological world expands, vulnerabilities of our critical infrastructure are becoming clear. Fortunately, emerging services provide an opportunity to improve the efficiency and security of current practices. In particular, serverless computing (such as Amazon Web Services and REDFISHs Acequia) provide opportunities to improve current practices. However, the critical infrastructure needs to evolve and that will require due diligence to ensure that transferring aspects of its practices onto the internet is done in a secure manner.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2018

Alfalfa [SWR 18-17]

Alfalfa is a web based software service that implements a virtual building control system using the EnergyPlus simulation as a backend. Conventional building energy simulation is non interactive, meaning that a simulation is started and run to completion without any user input during the simulation. Alfalfa allows a client to start a building simulation, then interactively read sensor locations and dynamically command actuators in realtime. The Alfalfa web service conforms to the Project Haystack standard, which is a specification adopted by many building control system vendors. The service enables Haystack compliant control system vendors to test their control modules and algorithms without the need for a physical building or modifying their hardware and software, enabling faster and cheaper development cycles. Alfalfa has use cases beyond control system testing. One example is building fault detection, where physical building performance can be compared to simulated values in realtime, indicating where there may be a problem in the building. Another use case is informing predictive control systems, where near future events are simulated to premptively guide control decisions in the real building. This software is web based and designed to be self service. Users of Alfalfa can upload an OpenStudio building model and begin communicating with the simulated building without any advanced setup or configuration, as if they were communicating a real building control system. Behind the scenes, Alfalfa makes extensive use of Amazon Web Services, in order to scale from a very small number of users and buildings, to extremely large numbers in a cost effective way.

Benne, Kyle↗
DOE OSTI2018

Scalable pCT Image Reconstruction Delivered as a Cloud Service

We describe a cloud-based medical image reconstruction service designed to meet a real-time and daily demand to reconstruct thousands of images from proton cancer treatment facilities worldwide. Rapid reconstruction of a three-dimensional Proton Computed Tomography (pCT) image can require the transfer of 100 GB of data and use of approximately 120 GPU-enabled compute nodes. The nature of proton therapy means that demand for such a service is sporadic and comes from potentially hundreds of clients worldwide. We thus explore the use of a commercial cloud as a scalable and cost-efficient platform for pCT reconstruction. To address the high performance requirements of this application we leverage Amazon Web Services' GPU-enabled cluster resources that are provisioned with high performance networks between nodes. To support episodic demand, we develop an on-demand multi-user provisioning service that can dynamically provision and resize clusters based on image reconstruction requirements, priorities, and wait times. We compare the performance of our pCT reconstruction service running on commercial cloud resources with that of the same application on dedicated local high performance computing resources. We show that we can achieve scalable and on-demand reconstruction of large scale pCT images for simultaneous multi-client requests, processing images in less than 10 minutes for less than $10 per image.

Cloud computing↗
DOE OSTI2022

Application of Quantum Machine Learning to High Energy Physics Analysis at LHC Using Quantum Computer Simulators and Quantum Computer Hardware

Machine learning enjoys widespread success in High Energy Physics (HEP) analyses at LHC. However the ambitious HL-LHC program will require much more computing resources in the next two decades. Quantum computing may offer speed-up for HEP physics analyses at HL-LHC, and can be a new computational paradigm for big data analyses in High Energy Physics.We have successfully employed three methods (1) Variational Quantum Classifier (VQC) method, (2) Quantum Support Vector Machine Kernel (QSVM-kernel) method and (3) Quantum Neural Network (QNN) method for two LHC flagship analyses: ttH (Higgs production in association with two top quarks) and H->mumu (Higgs decay to two muons, the second generation fermions). We shall address the progressive improvements in performance from method (1) to method (3).We will present our experiences and results of a study on LHC High Energy Physics data analyses with IBM Quantum Simulator and Quantum Hardware (using IBM Qiskit framework), Google Quantum Simulator (using Google Cirq framework), and Amazon Quantum Simulator (using Amazon Braket cloud service). The work is in the context of a Qubit platform (a gate-model quantum computer). Taking into account the present limitation of hardware access, different quantum machine learning methods are studied on simulators and the results are compared with classical machine learning methods (BDT, classical Support Vector Machine and classical Neural Network). Furthermore, we do apply quantum machine learning on IBM quantum hardware to compare performance between quantum simulator and quantum hardware. The work is performed by an international and interdisciplinary collaboration with the Department of Physics and Department of Computer Sciences of University of Wisconsin, CERN Quantum Technology Initiative, IBM Research Zurich, IBM T.J. Watson Research Center, Fermilab Quantum Institute, BNL Computational Science Initiative, State University of New York at Stony Brook, and Quantum Computing and AI Research of Amazon Web Services. This work pioneers a close collaboration of academic institutions with industrial corporations in the High Energy Physics analyses effort. Though the size of event samples in future HL-LHC physics and the limited number of qubits pose some challenges to the Quantum Machine learning studies for High Energy Physics, more advanced quantum computers with larger number of qubits, reduced noise and improved running time (as envisioned by IBM and Google) may outperform classical machine learning in both classification power and in speed.Although the era of efficient quantum computing may still be years away, we have made promising progress and obtained preliminary results in applying quantum machine learning to High Energy Physics. A PROOF OF PRINCIPLE.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2017

The HEPCloud Facility: elastic computing for High Energy Physics – The NOvA Use Case

The need for computing in the HEP community follows cycles of peaks and valleys mainly driven by conference dates, accelerator shutdown, holiday schedules, and other factors. Because of this, the classical method of provisioning these resources at providing facilities has drawbacks such as potential overprovisioning. As the appetite for computing increases, however, so does the need to maximize cost efficiency by developing a model for dynamically provisioning resources only when needed. To address this issue, the HEPCloud project was launched by the Fermilab Scientific Computing Division in June 2015. Its goal is to develop a facility that provides a common interface to a variety of resources, including local clusters, grids, high performance computers, and community and commercial Clouds. Initially targeted experiments include CMS and NOvA, as well as other Fermilab stakeholders. In its first phase, the project has demonstrated the use of the “elastic” provisioning model offered by commercial clouds, such as Amazon Web Services. In this model, resources are rented and provisioned automatically over the Internet upon request. In January 2016, the project demonstrated the ability to increase the total amount of global CMS resources by 58,000 cores from 150,000 cores - a 25 percent increase - in preparation for the Recontres de Moriond. In March 2016, the NOvA experiment has also demonstrated resource burst capabilities with an additional 7,300 cores, achieving a scale almost four times as large as the local allocated resources and utilizing the local AWS s3 storage to optimize data handling operations and costs. NOvA was using the same familiar services used for local computations, such as data handling and job submission, in preparation for the Neutrino 2016 conference. In both cases, the cost was contained by the use of the Amazon Spot Instance Market and the Decision Engine, a HEPCloud component that aims at minimizing cost and job interruption. This paper describes the Fermilab HEPCloud Facility and the challenges overcome for the CMS and NOvA communities.

Fuess, S.↗
DOE OSTI2024

The ATLAS experiment software on ARM

With an increased dataset obtained during the Run 3 of the LHC at CERN and the even larger expected increase of the dataset by more than one order of magnitude for the HL-LHC, the ATLAS experiment is reaching the limits of the current data processing model in terms of traditional CPU resources based on x86_64 architectures and an extensive program for software upgrades towards the HL-LHC has been set up. The ARM architecture is becoming a competitive and energy efficient alternative. Some surveys indicate its increased presence in HPCs and commercial clouds, and some WLCG sites have expressed their interest. Chip makers are also developing their next generation solutions on ARM architectures, sometimes combining ARM and GPU processors in the same chip. Consequently it is important that the ATLAS software embraces the change and is able to successfully exploit this architecture. We report on the successful porting to ARM of the Athena software framework, which is used by ATLAS for both online and offline computing operations. Furthermore we report on the successful validation of simulation workflows running on ARM resources. For this we have set up an ATLAS Grid site using ARM compatible middleware and containers on Amazon Web Services (AWS) ARM resources. The ARM version of Athena is fully integrated in the regular software build system and distributed in the same way as other software releases. In addition, the workflows have been integrated into the HEPscore benchmark suite which is the planned WLCG wide replacement of the HepSpec06 benchmark used for Grid site pledges. In the overall porting process we have used resources on AWS, Google Cloud Platform (GCP) and CERN. A performance comparison of different architectures and resources will be discussed.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2017

Experience in using commercial clouds in CMS

Historically high energy physics computing has been performed on large purpose-built computing systems. In the beginning there were single site computing facilities, which evolved into the Worldwide LHC Computing Grid (WLCG) used today. The vast majority of the WLCG resources are used for LHC computing and the resources are scheduled to be continuously used throughout the year. In the last several years there has been an explosion in capacity and capability of commercial and academic computing clouds. Cloud resources are highly virtualized and intended to be able to be flexibly deployed for a variety of computing tasks. There is a growing interest amongst the cloud providers to demonstrate the capability to perform large scale scientific computing. In this presentation we will discuss results from the CMS experiment using the Fermilab HEPCloud Facility, which utilized both local Fermilab resources and Amazon Web Services (AWS). The goal was to work with AWS through a matching grant to demonstrate a sustained scale approximately equal to half of the worldwide processing resources available to CMS. We will discuss the planning and technical challenges involved in organizing the most IO intensive CMS workflows on a large-scale set of virtualized resource provisioned by the Fermilab HEPCloud. We will describe the data handling and data management challenges. Also, we will discuss the economic issues and cost and operational efficiency comparison to our dedicated resources. At the end we will consider the changes in the working model of HEP computing in a domain with the availability of large scale resources scheduled at peak times.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2017

Virtual machine provisioning, code management, and data movement design for the Fermilab HEPCloud Facility

The Fermilab HEPCloud Facility Project has as its goal to extend the current Fermilab facility interface to provide transparent access to disparate resources including commercial and community clouds, grid federations, and HPC centers. This facility enables experiments to perform the full spectrum of computing tasks, including data-intensive simulation and reconstruction. We have evaluated the use of the commercial cloud to provide elasticity to respond to peaks of demand without overprovisioning local resources. Full scale data-intensive workflows have been successfully completed on Amazon Web Services for two High Energy Physics Experiments, CMS and NOνA, at the scale of 58000 simultaneous cores. This paper describes the significant improvements that were made to the virtual machine provisioning system, code caching system, and data movement system to accomplish this work. The virtual image provisioning and contextualization service was extended to multiple AWS regions, and to support experiment-specific data configurations. A prototype Decision Engine was written to determine the optimal availability zone and instance type to run on, minimizing cost and job interruptions. We have deployed a scalable on-demand caching service to deliver code and database information to jobs running on the commercial cloud. It uses the frontiersquid server and CERN VM File System (CVMFS) clients on EC2 instances and utilizes various services provided by AWS to build the infrastructure (stack). We discuss the architecture and load testing benchmarks on the squid servers. We also describe various approaches that were evaluated to transport experimental data to and from the cloud, and the optimal solutions that were used for the bulk of the data transport. Finally, we summarize lessons learned from this scale test, and our future plans to expand and improve the Fermilab HEP Cloud Facility.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2022

Semantic Property Graph for Scalable Knowledge Graph Analytics

Graphs are a natural and fundamental representation to describe entities, relationships, activities, and evolution of complex systems. Many domains such as communication, citation, procurement, biology, social media, and transportation can be modeled as a set of entities and their relationships. Resource Description Framework (RDF) and Labeled Property Graph (LPG) are two of the most used data models to encode information in a graph. Both models are similar in terms of using basic graph elements such as nodes and edges but differ in terms of the modeling approach, expressibility, serialization, and target applications. RDF is a flexible data exchange model for expressing information about entities but it tends to a have high memory footprint and inefficient storage, which does not make it a natural choice to perform scalable graph analytics. In contrast, LPG has gained traction as a reliable model to perform scalable graph analytic tasks such as sub-graph matching, network alignment, and real-time knowledge graph query. It provides efficient storage, fast traversal, and flexibility to model various real-world domains. At the same time, the LPG lacks the support of a formal knowledge representation such as an ontology to provide automated knowledge inference. We propose Semantic Property Graph (SPG) as a logical projection of reified RDF into the LPG model. SPG continues to use RDF ontology to define the type hierarchy of the projected graph and validate it against a given ontology. We present a framework to convert reified RDF graphs into SPG using two different computing environments. We also present cloud-based graph migration capabilities using Amazon Web Services.

Purohit, Sumit↗