Search NASA⌕ Search

SEARCH · Search NASA

Results for “cloud computing, data cloud”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

X-composer: enabling cross-environments in-situ workflows between HPC and cloud

As large-scale scientific simulations and big data analyses become more popular, it is increasingly more expensive to store huge amounts of raw simulation results to perform post-analysis. To minimize the expensive data I/O, "in-situ" analysis is a promising approach, where data analysis applications analyze the simulation generated data on the fly without storing it first. However, it is challenging to organize, transform, and transport data at scales between two semantically different ecosystems due to the distinct software and hardware difference. To tackle these challenges, we design and implement the X-Composer framework. X-Composer connects cross-ecosystem applications to form an "in-situ" scientific workflow, and provides a unified approach and recipe for supporting such hybrid in-situ workflows on distributed heterogeneous resources. X-Composer reorganizes simulation data as continuous data streams and feeds them seamlessly into the Cloud-based stream processing services to minimize I/O overheads. For evaluation, we use X-Composer to set up and execute a cross-ecosystem workflow, which consists of a parallel Computational Fluid Dynamics simulation running on HPC, and a distributed Dynamic Mode Decomposition analysis application running on Cloud. Our experimental results show that X-Composer can seamlessly couple HPC and Big Data jobs in their own native environments, achieve good scalability, and provide high-fidelity analytics for ongoing simulations in real-time.

Wang, Dali↗
DOE OSTI2015

Validation of the National Solar Radiation Database (NSRDB) (2005-2012): Preprint

Publicly accessible, high-quality, long-term, satellite-based solar resource data is foundational and critical to solar technologies to quantify system output predictions and deploy solar energy technologies in grid-tied systems. Solar radiation models have been in development for more than three decades. For many years, the National Renewable Energy Laboratory (NREL) developed and/or updated such models through the National Solar Radiation Data Base (NSRDB). There are two widely used approaches to derive solar resource data from models: (a) an empirical approach that relates ground-based observations to satellite measurements and (b) a physics-based approach that considers the radiation received at the satellite and creates retrievals to estimate clouds and surface radiation. Although empirical methods have been traditionally used for computing surface radiation, the advent of faster computing has made operational physical models viable. The Global Solar Insolation Project (GSIP) is an operational physical model from the National Oceanic and Atmospheric Administration (NOAA) that computes global horizontal irradiance (GHI) using the visible and infrared channel measurements from the Geostationary Operational Environmental Satellites (GOES) system. GSIP uses a two-stage scheme that first retrieves cloud properties and then uses those properties in the Satellite Algorithm for Surface Radiation Budget (SASRAB) model to calculate surface radiation. NREL, the University of Wisconsin, and NOAA have recently collaborated to adapt GSIP to create a high temporal and spatial resolution data set. The product initially generates the cloud properties using the AVHRR Pathfinder Atmospheres-Extended (PATMOS-x) algorithms [3], whereas the GHI is calculated using SASRAB. Then NREL implements accurate and high-resolution input parameters such as aerosol optical depth (AOD) and precipitable water vapor (PWV) to compute direct normal irradiance (DNI) using the DISC model. The AOD and PWV, temperature, and pressure data are also combined with the MMAC model to simulate solar radiation under clear-sky conditions. The current NSRDB update is based on a 4-km x 4-km resolution at a 30-minute time interval, which has a higher temporal and spatial resolution. This paper demonstrates the evaluation of the data set using ground-measured data and detailed evaluation statistics. The result of the comparison shows a good correlation to the NSRDB data set. Further, an outline of the new version of the NSRDB and future plans for enhancement and improvement are provided.

14 SOLAR ENERGY↗
DOE OSTI2016

The Medical Science DMZ

Objective We describe use cases and an institutional reference architecture for maintaining high-capacity, data-intensive network flows (e.g., 10, 40, 100 Gbpsþ) in a scientific, medical context while still adhering to security and privacy laws and regulations. Materials and Methods High-end networking, packet filter firewalls, network intrusion detection systems. Results We describe a “Medical Science DMZ” concept as an option for secure, high-volume transport of large, sensitive data sets between research institutions over national research networks. Discussion The exponentially increasing amounts of “omics” data, the rapid increase of high-quality imaging, and other rapidly growing clinical data sets have resulted in the rise of biomedical research “big data.” The storage, analysis, and network resources required to process these data and integrate them into patient diagnoses and treatments have grown to scales that strain the capabilities of academic health centers. Some data are not generated locally and cannot be sustained locally, and shared data repositories such as those provided by the National Library of Medicine, the National Cancer Institute, and international partners such as the European Bioinformatics Institute are rapidly growing. The ability to store and compute using these data must therefore be addressed by a combination of local, national, and industry resources that exchange large data sets. Maintaining data-intensive flows that comply with HIPAA and other regulations presents a new challenge for biomedical research. Recognizing this, we describe a strategy that marries performance and security by borrowing from and redefining the concept of a “Science DMZ”—a framework that is used in physical sciences and engineering research to manage high-capacity data flows. Conclusion By implementing a Medical Science DMZ architecture, biomedical researchers can leverage the scale provided by high-performance computer and cloud storage facilities and national high-speed research networks while preserving privacy and meeting regulatory requirements.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

The Concept of a Quantum Edge Simulator: Edge Computing and Sensing in the Quantum Era

Sensors, enabling observations across vast spatial, spectral, and temporal scales, are major data generators for information technology (IT). Processing, storing, and communicating this ever-growing amount of data pose challenges for the current IT infrastructure. Edge computing—an emerging paradigm to overcome the shortcomings of cloud-based computing—could address these challenges. Furthermore, emerging technologies such as quantum computing, quantum sensing, and quantum communications have the potential to fill the performance gaps left by their classical counterparts. Here, we present the concept of an edge quantum computing (EQC) simulator—a platform for designing the next generation of edge computing applications. An EQC simulator is envisioned to integrate elements from both quantum technologies and edge computing to allow studies of quantum edge applications. The presented concept is motivated by the increasing demand for more sensitive and precise sensors that can operate faster at lower power consumption, generating both larger and denser datasets. These demands may be fulfilled with edge quantum sensor networks. Envisioning the EQC era, we present our view on how such a scenario may be amenable to quantification and design. Given the cost and complexity of quantum systems, constructing physical prototypes to explore design and optimization spaces is not sustainable, necessitating EQC infrastructure and component simulators to aid in co-design. We discuss what such a simulator may entail and possible use cases that invoke quantum computing at the edge integrated with new sensor infrastructures.

47 OTHER INSTRUMENTATION↗
DOE OSTI2024

AEPF: Attention-Enabled Point Fusion for 3D Object Detection

Current state-of-the-art (SOTA) LiDAR-only detectors perform well for 3D object detection tasks, but point cloud data are typically sparse and lacks semantic information. Detailed semantic information obtained from camera images can be added with existing LiDAR-based detectors to create a robust 3D detection pipeline. With two different data types, a major challenge in developing multi-modal sensor fusion networks is to achieve effective data fusion while managing computational resources. With separate 2D and 3D feature extraction backbones, feature fusion can become more challenging as these modes generate different gradients, leading to gradient conflicts and suboptimal convergence during network optimization. To this end, we propose a 3D object detection method, Attention-Enabled Point Fusion (AEPF). AEPF uses images and voxelized point cloud data as inputs and estimates the 3D bounding boxes of object locations as outputs. An attention mechanism is introduced to an existing feature fusion strategy to improve 3D detection accuracy and two variants are proposed. These two variants, AEPF-Small and AEPF-Large, address different needs. AEPF-Small, with a lightweight attention module and fewer parameters, offers fast inference. AEPF-Large, with a more complex attention module and increased parameters, provides higher accuracy than baseline models. Experimental results on the KITTI validation set show that AEPF-Small maintains SOTA 3D detection accuracy while inferencing at higher speeds. AEPF-Large achieves mean average precision scores of 91.13, 79.06, and 76.15 for the car class’s easy, medium, and hard targets, respectively, in the KITTI validation set. Results from ablation experiments are also presented to support the choice of model architecture.

Chemistry↗
DOE OSTI2025

OLCF’s Advanced Computing Ecosystem (ACE): FY25 Update for Ongoing Efforts

The advent of widespread use of artificial intelligence (AI) and machine learning (ML) models in science, coupled with fast data production rates of scientific instruments strain the traditional batch-oriented high-performance computing (HPC) environment. As scientific exploration continues to require more data and faster processing and analysis, new emerging technologies and capabilities to enable cross-facility and time-sensitive workflows are required for seamless integration of HPC and experimental facilities. The Advanced Computing Ecosystem (ACE) is a strategic initiative within the Oak Ridge Leadership Computing Facility (OLCF) established in 2024 to support the development of cutting-edge technologies to advance computational research and infrastructure at OLCF and across the Department of Energy (DOE). Several DOE initiatives are spearheading the evolution of the scientific landscape by blurring facility boundaries and connecting the user facilities to advance scientific capabilities and ensure energy dominance. The DOE Integrated Research Infrastructure (IRI) program is one example that is laying a foundation to support complex cross-facility workflows. The IRI program aims to integrate diverse computational resources, data infrastructures, and scientific instruments to facilitate collaboration and accelerate scientific discovery. The Interconnected Science Ecosystem (INTERSECT) initiative at Oak Ridge National Laboratory (ORNL) is another example that aims to revolutionize scientific research through AI-driven, interconnected autonomous laboratories and research facilities. Finally, the American Science Cloud (AmSC), recently announced in the “One Big Beautiful Bill”, aims to leverage prior infrastructure efforts of the IRI and automation and AI efforts of INTERSECT (and others) to build a federated, AI-augmented AmSC platform to unify the DOE’s computing, experimental, and data resources to catalyze scientific innovation.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Preparing an on-Demand Cloud Processing Workflow for NISAR Ecosystems Science Products

In preparation for the NISAR launch and data collection in 2024, the NISAR Project Science Team is building workflows for each Science Team discipline (Ecosystems, Cryosphere, and Solid Earth). This abstract focuses on the Ecosystem disciplines and the development of on-demand cloud-processing workflows for wetlands inundation, forest biomass, agricultural active crop area, and forest disturbance. The workflow simulates NISAR data using UAVSAR or ALOS-2 Single Look Complex data, which are processed to Level 2 geocoded polarimetric covariance matrix products using InSAR Scientific Computing Environment 3.0 software and to Level 3 science products using the Algorithm Theoretical Basis Documents. In this presentation, we describe these workflows and efforts to improve efficiency and data accessibility by using a cloud processing system. We present preliminary sample products from each Ecosystem discipline: inundation, forest biomass, crop area, and forest disturbance.

Christensen, Alexandra↗
DOE OSTI2014

SU-D-BRD-02: A Web-Based Image Processing and Plan Evaluation Platform (WIPPEP) for Future Cloud-Based Radiotherapy

Purpose: Visualization and processing of medical images and radiation treatment plan evaluation have traditionally been constrained to local workstations with limited computation power and ability of data sharing and software update. We present a web-based image processing and planning evaluation platform (WIPPEP) for radiotherapy applications with high efficiency, ubiquitous web access, and real-time data sharing. Methods: This software platform consists of three parts: web server, image server and computation server. Each independent server communicates with each other through HTTP requests. The web server is the key component that provides visualizations and user interface through front-end web browsers and relay information to the backend to process user requests. The image server serves as a PACS system. The computation server performs the actual image processing and dose calculation. The web server backend is developed using Java Servlets and the frontend is developed using HTML5, Javascript, and jQuery. The image server is based on open source DCME4CHEE PACS system. The computation server can be written in any programming language as long as it can send/receive HTTP requests. Our computation server was implemented in Delphi, Python and PHP, which can process data directly or via a C++ program DLL. Results: This software platform is running on a 32-core CPU server virtually hosting the web server, image server, and computation servers separately. Users can visit our internal website with Chrome browser, select a specific patient, visualize image and RT structures belonging to this patient and perform image segmentation running Delphi computation server and Monte Carlo dose calculation on Python or PHP computation server. Conclusion: We have developed a webbased image processing and plan evaluation platform prototype for radiotherapy. This system has clearly demonstrated the feasibility of performing image processing and plan evaluation platform through a web browser and exhibited potential for future cloud based radiotherapy.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2018

High Resolution Measurements and Modeling in the Arctic

This report details the activity of the project, "High Resolution Modeling and Measurements in the Arctic" spanning Fiscal Years 2016 - 2018 supported by the Sandia National Laboratories Laboratory Directed Research and Development (LDRD) program. The project's primary goal was to test the hypothesis that global climate model bias of low boundary layer clouds lacking liquid water in the Arctic could be improved by increasing horizontal resolution in the model. As model resolution is constrained by computational resources, four different model types were explored and compared to test the project's primary theory. Given the Arctic is a data-sparse region lacking robust data sets of liquid water in clouds, this project also obtained in situ measurements of low clouds with sensors on a tethered balloon system to constrain and compare with the models. Although other model biases remained, the liquid water path generally increased with resolution, supporting the original hypothesis.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2018

A Cloud-Based Decision Support System Framework for Hydropower Biological Evaluation

Hydropower has become one of the most important energy sources: it accounts for more than 80% of the world’s renewable energy and about 16% of all the energy in the world. Significantly more hydropower capacity is planned to be developed. However, hydro-structures, including powered dams, may have adverse biological effects on fishes, especially on migrating fish. For instance, fish can be injured or even killed when they pass through turbines. This is why biological evaluations on hydro-structures are greatly needed to estimate fish injury and mortality rates. The Hydropower Biological Evaluation Toolset (HBET) is an integrated suite of science-based desktop tools designed to evaluate whether the hydraulic conditions of hydropower structures are fish friendly by analyzing collected data and providing estimated injury and mortality rates. Sensor Fish, a small, autonomous sensor package, is used by HBET to record data describing the conditions that live fish passing through a hydropower structure will face. In this paper, we incorporated cloud computing into HBET, and migrated it into a cloud-based decision support system framework for hydropower biological evaluation. This will make the evaluation systems more scalable and flexible; however, this also introduces a significant challenge: how to maintain the security while retaining scalability and flexibility. We discuss the technical methodologies and algorithms implemented in the proposed framework, and analyze the countermeasures implemented therein.

Decision Support System, Hydropower, Dam, Fish Inj↗
DOE OSTI2019

A Scalable Online Monitoring System Based on Elasticsearch for Distributed Data Acquisition in Cms

The part of the CMS Data Acquisition (DAQ) system responsible for data readout and event building is a complex network of interdependent distributed applications. To ensure successful data taking, these programs have to be constantly monitored in order to facilitate the timeliness of necessary corrections in case of any deviation from specified behaviour. A large number of diverse monitoring data samples are periodically collected from multiple sources across the network. Monitoring data are kept in memory for online operations and optionally stored on disk for post-mortem analysis. We present a generic, reusable solution based on an open source NoSQL database, Elasticsearch, which is fully compatible and non-intrusive with respect to the existing system. The motivation is to benefit from an off the-shelf software to facilitate the development, maintenance and support efforts. Elasticsearch provides failover and data redundancy capabilities as well as a programming language independent JSON-over-HTTP interface. The possibility of horizontal scaling matches the requirements of a DAQ monitoring system. The data load from all sources is balanced by redistribution over an Elasticsearch cluster that can be hosted on a computer cloud. In order to achieve the necessary robustness and to validate the scalability of the approach the above monitoring solution currently runs in parallel with an existing in-house developed DAQ monitoring system.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2014

Integration of Photovoltaics into Building Energy Usage through Advanced Control of Rooftop Unit

This paper presents a computational approach to forecast photovoltaic (PV) power in kW based on a neural network linkage of publicly available cloud cover data and on-site solar irradiance sensor data. We also describe a control approach to utilize rooftop air conditioning units (RTUs) to support renewable integration. The PV forecasting method is validated using data from a rooftop PV panel installed on the Distributed Energy, Communications, and Controls (DECC) laboratory at Oak Ridge National Laboratory. The validation occurs in multiple phases to ensure that each component of the approach is the best representation of the actual expected output. The control of the RTU is based on model predictive methods.

Starke, Michael R↗
DOE OSTI2021

Progress on the National Solar Radiation Data Base (NSRDB): A New DNI Computation

This study introduces a new technique to compute direct normal irradiance (DNI) for improving the National Solar Radiation Data Base (NSRDB). A finite-surface integration algorithm is developed to compute solar radiation in differential solid angles and efficiently infer its contribution to a surface perpendicular to the solar direction. A lookup table of cloud bi-directional transmittance distribution function (BTDF) is developed by use of the discrete ordinates radiative transfer (DISORT) model for possible solar and observing directions and various cloud optical and microphysical properties. In each solar incident direction, DNI is given by the cloud BTDFs from approximately 200 differential solid angles. The simulated DNI is calibrated and evaluated using surface observations at the National Renewable Energy Laboratory's (NREL's) Solar Energy Research Laboratory (SRRL) and the Atmospheric Radiation Measurement (ARM) Southern Great Plains (SGP) facility.

41 EE - Solar Energy Technologies Office (EE-4S)↗
DOE OSTI2020

ARM Data-Oriented Metrics and Diagnostics Package for Climate Model Evaluation

A Python-based metrics and diagnostics package is currently being developed by the U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) user facility Infrastructure Team at Lawrence Livermore National Laboratory to facilitate the use of long-term, high-frequency measurements from ARM in evaluating the regional climate simulation of clouds, radiation, and precipitation. This metrics and diagnostics package computes climatological means of targeted climate model simulation and generates tables and plots for comparing the model simulation with ARM observational data.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2014

Compiler Optimization for Extreme-Scale Scripting

The data-driven task parallelism execution model can support parallel programming models that are well suited for large-scale distributed-memory parallel computing, for example, simulations and analysis pipelines running on clusters and clouds. We describe a novel compiler intermediate representation and optimizations for this execution model, including adaptions of standard techniques alongside novel techniques. These techniques are applied to Swift/T, a high-level scripting language for flexible data flow composition of functions, which may be serial or use lower-level parallel programming models such as MPI and OpenMP. This paper presents preliminary results, indicating that our compiler optimizations reduce communication overhead by 70% to 93% on distributed-memory systems.

Armstrong, Timothy G.↗