GloudSim: Google trace based cloud simulator with virtual machines: GloudSim: Google trace based cloud simulator with virtual machines
Explore the source record for details and available documents.
97 MATHEMATICS AND COMPUTING↗
SEARCH · Search NASA
Results for “Google Cluster”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Flexible Pilot Jobs Framework for Distributed High Throughput Computing
Experimental particle physics has been at the forefront of analyzing the world’s largest datasets for decades. The high-energy physics (HEP) community was among the first to develop suitable software and computing tools for this purpose. GlideinWMS is a Glidein-based workload management system whose purpose is to provide experiments like CMS at CERN, DUNE at Fermilab, and others, a way to access and efficiently use vast amounts of computing resources. This system wants to provide a simple way to submit jobs to a set of computing resources, that will be provided to users behind the scenes. Glideins are the pilot jobs executed on the worker nodes at the grid sites, performing operations such as hardware detection, environment setup, and error handling. After all these operations, they will launch the actual user job. Many grid sites are supported, such as shared clusters, Google CE, and AWS. My internship aimed to design and code a flexible pilot jobs framework that will replace the one used by GlideinWMS, developing a modular and flexible skeleton of the Glidein and adding further functionalities. My project also focused on the application of machine learning techniques as support to this management system.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
Accelerating science: The usage of commercial clouds in ATLAS Distributed Computing
The ATLAS experiment at CERN is one of the largest scientific machines built to date and will have ever growing computing needs as the Large Hadron Collider collects an increasingly larger volume of data over the next 20 years. ATLAS is conducting R&D projects on Amazon Web Services and Google Cloud as complementary resources for distributed computing, focusing on some of the key features of commercial clouds: lightweight operation, elasticity and availability of multiple chip architectures. The proof of concept phases have concluded with the cloud-native, vendoragnostic integration with the experiment’s data and workload management frameworks. Google Cloud has been used to evaluate elastic batch computing, ramping up ephemeral clusters of up to O(100k) cores to process tasks requiring quick turnaround. Amazon Web Services has been exploited for the successful physics validation of the Athena simulation software on ARM processors. We have also set up an interactive facility for physics analysis allowing endusers to spin up private, on-demand clusters for parallel computing with up to 4 000 cores, or run GPU enabled notebooks and jobs for machine learning applications. The success of the proof of concept phases has led to the extension of the Google Cloud project, where ATLAS will study the total cost of ownership of a production cloud site during 15 months with 10k cores on average, fully integrated with distributed grid computing resources and continue the R&D projects.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
Utilizing Airborne and Space-Based Remote Sensing Imagery to Implement the Unvegetated-Vegetated Ratio to Assess Salt Marsh Vulnerability in South Carolina
Among the most productive ecosystems on earth, salt marshes provide crucial ecosystem services including water filtration, shoreline protection, storm surge buffering, and flood mitigation. Marshes are largely dependent on their sediment budget which can significantly vary across a region and can be used to determine the life span of the marsh. Upstream land use change near Charleston, South Carolina, along with rising sea levels, are expected to alter sediment budgets and threaten marsh stability and long-term health. The unvegetated-vegetated ratio (UVVR), developed by researchers at USGS, is a scalable and efficient method to assess vulnerability. The NASA DEVELOP National Program collaborated with the South Carolina Department of Natural Resources, the South Carolina Department of Health and Environmental Control, and the United States Geological Survey Woods Hole Coastal and Marine Science Center to apply the UVVR method within Google Earth Engine. Marsh vulnerability was analyzed using UVVR derived from clustering and manual interpretation of National Agriculture Imagery Program (NAIP) high-resolution aerial imagery. NAIP derived UVVR was aggregated to Landsat 8 Operational Land Imager (OLI) and Landsat 7 Enhanced Thematic Mapper (ETM+) resolution and projection. A Random Forest Regression between Landsat derived data and UVVR was modeled to estimate a potential relationship. The estimation of this relationship was used to produce temporal change analysis maps of salt marsh vulnerability back to 1984. The NAIP imagery processed through Google Earth Engine allowed us to make detailed UVVR maps for 2009, 2015, 2017, and 2019 for decision making within South Carolina. Google Earth Engine scripting provided a novel approach to UVVR methodology that will allow decision makers to input new marsh regions and easily calculate marsh vulnerability without external data downloading. These results were used to understand what areas of the marsh need most resource allocation in the future.
NASA DEVELOP↗
Execute BEE workflows on private cloud infrastructure-2.3.6.01 - LANL ATDM ST / STNS01-22 Milestone Completion Documentation (BEE-FY21 P6-2) [Slides]
This work involves the creation of the Cloud Launcher, a new subcomponent of BEE, and the extension of the BEETaskManager to run on Cloud systems. BEE will be able to interact with the Google Compute Engine and OpenStack cloud APIs to set up simple Cloud clusters for launching HPC job scripts. BEE will use existing functionality to launch jobs that previously could only be launched on HPC systems. The BEETaskManager will handle launching tasks on the Cloud cluster.
97 MATHEMATICS AND COMPUTING↗
NASA Taxonomies for Searching Problem Reports and FMEAs
Many types of hazard and risk analyses are used during the life cycle of complex systems, including Failure Modes and Effects Analysis (FMEA), Hazard Analysis, Fault Tree and Event Tree Analysis, Probabilistic Risk Assessment, Reliability Analysis and analysis of Problem Reporting and Corrective Action (PRACA) databases. The success of these methods depends on the availability of input data and the analysts knowledge. Standard nomenclature can increase the reusability of hazard, risk and problem data. When nomenclature in the source texts is not standard, taxonomies with mapping words (sets of rough synonyms) can be combined with semantic search to identify items and tag them with metadata based on a rich standard nomenclature. Semantic search uses word meanings in the context of parsed phrases to find matches. The NASA taxonomies provide the word meanings. Spacecraft taxonomies and ontologies (generalization hierarchies with attributes and relationships, based on terms meanings) are being developed for types of subsystems, functions, entities, hazards and failures. The ontologies are broad and general, covering hardware, software and human systems. Semantic search of Space Station texts was used to validate and extend the taxonomies. The taxonomies have also been used to extract system connectivity (interaction) models and functions from requirements text. Now the Reconciler semantic search tool and the taxonomies are being applied to improve search in the Space Shuttle PRACA database, to discover recurring patterns of failure. Usual methods of string search and keyword search fall short because the entries are terse and have numerous shortcuts (irregular abbreviations, nonstandard acronyms, cryptic codes) and modifier words cannot be used in sentence context to refine the search. The limited and fixed FMEA categories associated with the entries do not make the fine distinctions needed in the search. The approach assigns PRACA report titles to problem classes in the taxonomy. Each ontology class includes mapping words - near-synonyms naming different manifestations of that problem class. The mapping words for Problems, Entities and Functions are converted to a canonical form plus any of a small set of modifier words (e.g. non-uniformity NOT + UNIFORM.) The report titles are parsed as sentences if possible, or treated as a flat sequence of word tokens if parsing fails. When canonical forms in the title match mapping words, the PRACA entry is associated with the corresponding Problem, Entity or Function in the ontology. The user can search for types of failures associated with types of equipment, clustering by type of problem (e.g., all bearings found with problems of being uneven: rough, irregular, gritty ). The results could also be used for tagging PRACA report entries with rich metadata. This approach could also be applied to searching and tagging failure modes, failure effects and mitigations in FMEAs. In the pilot work, parsing 52K+ truncated titles (the test cases that were available), has resulted in identification of both a type of equipment and type of problem in about 75% of the cases. The results are displayed in a manner analogous to Google search results. The effort has also led to the enrichment of the taxonomy, adding some new categories and many new mapping words. Further work would make enhancements that have been identified for improving the clustering and further reducing the false alarm rate. (In searching for recurring problems, good clustering is more important than reducing false alarms). Searching complete PRACA reports should lead to immediate improvement.
Malin, Jane T.↗
NASA’s Prototype Spectral Water Inversion Processor and Emulator (SWIPE): Towards Global Coastal and Inland Water Quality and Algal Biodiversity Monitoring
Degradation of Earth’s inland water resources due to anthropogenic perturbations and climate anomalies at both local and global scales continues to place human health at substantial risk. There is now a growing necessity to develop pragmatic approaches that allow timely and effective extrapolation of local processes, to spatially resolved global products, and to promote operational and sustainable resource policy management. This presentation will provide updates on NASA’s prototype open-source aquatic modeling platform, Spectral Water Inversion Processor and Emulator (SWIPE), which is a comprehensive, multi-faceted modeling platform for both forward and inverse modeling of diverse aquatic ecosystems from the benthos to top-of-atmosphere (TOA). SWIPE provides a cohesive application which leverages recent advancements in particle modeling, Big Data analytics, and machine learning to develop a high-fidelity synthetic training ground for sensitivity studies and algorithm development for multispectral or upcoming hyperspectral missions. Some of the prominent features of SWIPE to be discussed include: 1. Advanced hyperspectral modeling of globally diverse algal and non-algal particles using a novel two-layer coated sphere scattering model and radiative transfer modeling, 2. Massive, highly detailed synthetic spectral libraries of Analysis-Ready-Data (ARD) which include spectral libraries of particle microphysics, water biogeophysical and optical properties, as well as surface and TOA reflectances at 1 nm resolution, 3. An ensemble of pre-built analytic, machine learning, and deep learning inversion algorithms for various water quality and biodiversity related retrieval parameters and uncertainty quantification, 4. Sensor-agnostic water quality inversion at wide ranging spatial and spectral resolutions including a codebase for seamless application in the Google Earth Engine and NASA Earth Exchange (NEX) for planetary scale analysis. SWIPE will be a fully open-source platform based in python with comprehensive documentation, tutorials, and options for distributed computing on high performance computing clusters or on single, local machines. Further, we will discuss how we envision SWIPE contributing towards a global analysis of coastal and inland water quality dynamics.
top-of-atmosphere (TOA)↗
Knowledge Discovery for Smart Grid Operation, Control, and Situation Awareness -- A Big Data Visualization Platform
In this paper, a big data visualization platform is designed to discover the hidden useful knowledge for smart grid (SG) operation, control and situation awareness. The spawn of smart sensors at both grid side and customer side can provide large volume of heterogeneous data that collect information in all time spectrums. Extracting useful knowledge from this big-data poll is still challenging. In this paper, the Apache Spark, an open source cluster computing framework, is used to process the big-data to effectively discover the hidden knowledge. A high-speed communication architecture utilizing the Open System Interconnection (OSI) model is designed to transmit the data to a visualization platform. This visualization platform uses Google Earth, a global geographic information system (GIS) to link the geological information with the SG knowledge and visualize the information in user defined fashion. The University of Denver's campus grid is used as a SG test bench and several demonstrations are presented for the proposed platform.
24 POWER TRANSMISSION AND DISTRIBUTION↗
Panel-Segmentation [SWR-21-18]
Panel-Segmentation contains the scripts for automated metadata extraction of solar PV installations, using satellite imagery coupled with computer vision techniques. In this package, the user can perform the following actions: *Automatically generate a satellite image using a set of lat-long coordinates, and a Google Maps API key. Users would need to set up a Google Cloud account and get a Maps Static API key. Please refer to Setting Up Google Maps Static API Key section for this process. *Perform image segmentation on the satellite image, to locate the solar array(s) in the image on a pixel-by-pixel basis, using an image segmentation model (panel_detection_model.pth). Get classification of the installation (rooftop, ground mounted fixed-tilt or tracking, carport, etc). *Perform azimuth estimation on each solar array cluster in the masked image. *Detect solar panels and get its latitude, longitude, and address within a geographic bounding box through the SOL-Searcher Pipeline. *Detect and calculate hurricane damage on solar installations given pre-hurricane and post-hurricane satellite imagery through the Hurricane Detection Pipeline. *Detect and calculate hail damage on solar installations given satellite imagery through the Hail Detection pipeline. *Convert NOAA MESH (Maximum Estimated Size of Hail) grib2 files into kml or geojson files. *Estimate tilt and azimuth of a solar array by processing USGS LiDAR data for the array’s location.
Edun, Ayobami↗
SV-Sim: Scalable PGAS-based State Vector Simulation of Quantum Circuits
High-performance quantum circuit simulation in a classic HPC is still imperative in the NISQ era. Observing that the major obstacle of scalable state-vector quantum simulation arises from the massively fine-grained irregular data-exchange with remote nodes, in this paper we present SV-Sim to apply the emerging PGAS-based communication models (i.e., direct peer access for intra-node CPUs/GPUs and SHMEM for inter-node CPU/GPU clusters) for efficient scalable quantum circuit simulation. Through an orchestrated device functional pointer design, SV-Sim is able to abstract the quantum gate sets across various heterogeneous backends, including IBM/Intel/AMD CPUs, NVIDIA /AMD GPUs, and Intel MIC, in a unified framework, but still asserting outstanding performance and tractable interface to higher-level quantum programming environments, such as IBM Qiskit, Microsoft Q\# and Google Cirq. Circumventing the disability of polymorphism in GPUs and leveraging the device-initiated one-sided communication, SV-Sim can process dynamically synthesized quantum circuit in a single GPU/CPU kernel without the need of expensive JIT or runtime branching, significantly improving the performance and simplifying the programming complexity for the emerging variational quantum algorithms. Evaluations on NVIDIA A100-DGX-1, V100-DGX-2, AMD MI100, ALCF Theta, and OLCF Summit HPCs show that SV-Sim can delivery scalable performance on various state-of-the-art HPC platforms, offering a useful tool for quantum algorithm validation and verification.
Li, Ang↗
Utah FORGE: Site Earthquake Animation
This is a .kml earthquake animation covering the period of 1991 - 2011 for the Utah Milford FORGE site. It displays seismic events using different sized bubbles according to magnitude. It covers the general Utah FORGE area (large shaded rectangle) with the final site displayed as a smaller polygon along the northwestern margin. Earthquakes are subdivided into clusters and the time, date, and magnitude of each event is included. Nearby seismic stations are symbolized with triangles. This was created by the University of Utah Seismograph Stations (UUSS).
15 GEOTHERMAL ENERGY↗
Tensile behavior of dual-phase titanium alloys under high-intensity proton beam exposure: radiation-induced omega phase transformation in Ti-6Al-4V
A high-intensity proton beam exposure with 181 MeV energy has been conducted at Brookhaven Linac Isotope Producer facility on various material specimens for accelerator targetry applications, including titanium alloys as a beam window material. The radiation damage level of the analyzed capsule was 0.25 dpa at the beam center region with an irradiation temperature around 120 oC. Tensile tests showed increased hardness and a large decrease in ductility for the dual a+ß-phase Ti-6Al-4V Grade-5 and Grade-23 extra low interstitial alloys, with the near a-phase Ti-3Al-2.5V Grade-9 alloy still exhibiting uniform elongation of a few % after irradiation. Transmission Electron Microscope analyses on Ti-6Al-4V indicated clear evidence of a high-density of defect clusters with size less than 2 nm in each a-phase grain. The ß-phase grains did not contain any visible defects such as loops or black dots, while the diffraction patterns clearly indicated ?-phase precipitation in an advanced formation stage. The radiation-induced ?-phase transformation in the ß-phase could lead to greater loss of ductility in Ti-6Al-4V alloys in comparison with Ti-3Al-2.5V alloy with less ß-phase.google scho
Tensile Test, titanium, acclerator, proton beam↗
High Altitude Platform System (HAPS) Communication Support for Wildland Firefighting
High Altitude Platform Systems (HAPS) are emerging aircraft and balloon-type technology that can host payloads and provide services from the stratosphere. One potential HAPS use case is to provide wireless communication services for mobile devices, such as LTE, to wildland firefighters who often operate in locations without terrestrial wireless communications coverage. In this research we analyze historical wildland fire data to provide estimates of the annual number of HAPS required to support a fire season. We apply agglomerative clustering to group historical daily satellite-based fire observations where each cluster is analogous to a required HAPS vehicle. Our lower and upper bound estimates span a range of years, communication payload footprints, the minimum days of clusters prior to launch, and categories of fires. Additionally, we consider a case where HAPS vehicles can be transferred between fires after the initial fire has dissipated. In our specific case study from 2022 “Significant” fires (greater than 40,000 acres), we approximate that either 8 balloon HAPS vehicles without considering overprovisioning for station-keeping limitations or 23 fixed-wing aircraft would be required. Overprovisioning can scale the estimate for balloon vehicles based on reader preference, and for reference, Google Loon overprovisioned by 5-10x. Furthermore, in the case where budgets are constrained and not all of the estimated HAPS vehicles can be acquired, we provide operational insight on where to deploy HAPS vehicles. Generally, in the Spring months we see that HAPS vehicles are needed in the south and southeast of the US which transitions to the north and west as the fire season progresses.
Aaron J. Burns↗
Developing Natural Language Processing and Supervised Learning Techniques to Classify Mars Tasks
As NASA's Human Research Program (HRP) prepares for long-duration Mars missions, understanding astronaut tasks is crucial. This study, conducted at NASA Glenn Research Center (GRC), employed Natural Language Processing (NLP) and machine learning techniques to analyze and classify Mars tasks. A list of 1,058 Mars tasks was provided by HRP experts including binary labeling of 18 Human System Task Categories (HSTCs). We developed an NLP model using Google's BERT language model to capture the semantic and syntactic nuances of these tasks. Supervised training was initially applied to a subset of the NLP-analyzed tasks to assess the model's effectiveness in classifying the remaining tasks. Incorporating HSTC descriptions significantly enhanced the classification accuracy for 9 out of the 18 HSTCs and reduced training time. To address the issue of severe class imbalance in the HSTC data, we introduced innovative weighting and sampling techniques for data augmentation. We then fine-tune BERT to implement a pairwise relatedness scoring method, allowing us to cluster tasks based on their relatedness and similarity, getting a step closer to labeling the tasks without supervision. In this presentation we guide you through data preprocessing, deciphering key syntax components using BERT, and performing supervised classification of the Mars tasks. This work showcases the potential use of advanced NLP techniques to analyze Mars missions to be incorporated into various crew health and performance analyses.
GenAI↗
Future SDN-HPON Control Plane Architecture and Protocol for On-Demand Terabit End-to-End Extreme-Scale Science Applications
The UC Davis team conducted architecture, algorithm and experimental studies for next generation ultra-high-bandwidth optical networks in support of extreme-scale science applications. In particular, over the course of this 3-year project, the UC Davis team achieved the following main accomplishments: Designed a network architecture for automated multi-domain service provisioning: Designed information exchange schemes between autonomous systems (domains) Designed application programming interfaces to orchestrate resource reservation in data centers, high performance computation facilities, and communication networks belonging to different autonomous systems Experimentally assessed the multi-domain network architecture in a distributed field trial set-up connecting premises in three continents Designed a broker-based orchestrated multi-domain optical networking framework supporting observe-analyze-act cycle based cognitive inter-domain service provisioning: Designed an inter-domain operation mechanism, e.g., topology abstraction, for the broker plane and domain managers to collaboratively accomplish inter-domain service provisioning while respecting the autonomy of each administrative domain Designed an alien wavelength monitoring scheme and an artificial neural network based optical signal-to-noise ratio estimator for quality-of-transmission aware inter-domain service provisioning Developed a deep neural network based multi-domain traffic predictor and a cognitive routing and spectrum assignment algorithm to realize knowledge-based autonomous traffic engineering in multi-domain networks Implemented an optical signal-to-noise ratio estimator and traffic predictor with Google’s Tensor flow and assessed their performance using experimental and synthetic data Demonstrated the benefits of the proposed cognitive schemes with experiments ona two-domain elastic optical network testbed and evaluations on a six-domain simulation framework Built a two-domain software-defined networking control plane testbed using the ONOS platform and demonstrated the workflow of broker and domain manager cooperation, including hierarchical performance monitoring, topology abstraction and cognitive resource allocation. Designed self-learning cognitive service provisioning schemes using advanced machine learning technologies: Designed a self-taught anomaly detection framework for optical networks, including an unsupervised data clustering module for pattern recognition, a supervised data regression and classification module for online anomaly detection, and the system operation principle Experimentally validated the effectiveness of the self-taught anomaly detection framework with various failure/anomaly scenarios. Designed a self-learning autonomic routing, modulation and spectrum assignment (RMSA) provisioning framework for elastic optical networks with deep reinforcement learning. Designed effective modeling and training mechanisms for the self-learning RMSA framework Conducted extensive numerical simulations with various traffic patterns and topologies to verify the advantage of the self-learning RMSA framework.
42 ENGINEERING↗
Landslides in West Coast Metropolitan Areas: The Role of Extreme Weather Events
Rainfall-induced landslides represent a pervasive issue in areas where extreme rainfall intersects complex terrain. A farsighted management of landslide risk requires assessing how landslide hazard will change in coming decades and thus requires, inter alia, that we understand what rainfall events are most likely to trigger landslides and how global warming will affect the frequency of such weather events. We take advantage of 9 years of landslide occurrence data compiled by collating Google news reports and of a high-resolution satellite-based daily rainfall data to investigate what weather triggers landslide along the West Coast US. We show that, while this landslide compilation cannot provide consistent and widespread monitoring everywhere, it captures enough of the events in the major urban areas that it can be used to identify the relevant relationships between landslides and rainfall events in Puget Sound, the Bay Area, and greater Los Angeles. In all these regions, days that recorded landslides have rainfall distributions that are skewed away from dry and low-rainfall accumulations and towards heavy intensities. However, large daily accumulation is the main driver of enhanced hazard of landslides only in Puget Sound. There, landslide are often clustered in space and time and major events are primarily driven by synoptic scale variability, namely "atmospheric rivers" of high humidity air hitting anywhere along the West Coast, and the interaction of frontal system with the coastal orography. The relationship between landslide occurrences and daily rainfall is less robust in California, where antecedent precipitation (in the case of the Bay area) and the peak intensity of localized downpours at sub-daily time scales (in the case of Los Angeles) are key factors not captured by the same-day accumulations. Accordingly, we suggest that the assessment of future changes in landslide hazard for the entire the West Coast requires consideration of future changes in the occurrence and intensity of atmospheric rivers, in their duration and clustering, and in the occurrence of short-duration (sub-daily) extreme rainfall as well. Major regional landslide events, in which multiple occurrences are recorded in the catalog for the same day, are too rare to allow a statistical characterization of their triggering events, but a case study analysis indicates that a variety of synoptic-scale events can be involved, including not only atmospheric rivers but also broader cold- and warm-front precipitation. That a news-based catalog of landslides is accurate enough to allow the identification of different landslide/ rainfall relationships in the major urban areas along the US West Coast suggests that this technology can potentially be used for other English-language cities and could become an even more powerful tool if expanded to other languages and non-traditional news sources, such as social media.
landslides↗
SciSpot: Scientific Computing On Temporally Constrained Cloud Preemptible VMs
Not provided.
Computer Science↗
A Visual Analytics Paradigm Enabling Trillion-Edge Graph Exploration
We present a visual analytics paradigm and a system prototype for exploring web-scale graphs. A web-scale graph is described as a graph with ~one trillion edges and ~50 billion vertices. While there is an aggressive R&D effort in processing and exploring web-scale graphs among internet vendors such as Facebook and Google, visualizing a graph of that scale still remains an underexplored R&D area. The paper describes a nontraditional peek-and-filter strategy that facilitates the exploration of a graph database of unprecedented size for visualization and analytics. We demonstrate that our system prototype can 1) preprocess a graph with ~25 billion edges in less than two hours and 2) support database query and visualization on the processed graph database afterward. Based on our computational performance results, we argue that we most likely will achieve the one trillion edge mark (a computational performance improvement of 40 times) for graph visual analytics in the near future.
graph visualization↗