Predictive Analytics-based U.S. Inland Waterways Voyage Planning Analysis Tool (VPAT)
Explore the source record for details and available documents.
54 ENVIRONMENTAL SCIENCES↗
SEARCH · Search NASA
Results for “cloud computing; data analysis”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Analytical Functions for 200 West Pump-and-Treat SCADA Sensor Data
Historical operations at the U.S. Department of Energy’s Hanford Site included disposal of waste fluids to the subsurface in the 200 West Area on the Hanford Central Plateau. Subsequent infiltration of fluids has resulted in groundwater contamination with carbon tetrachloride, nitrate, uranium, technetium-99, and other contaminants. A pump-and-treat (P&T) system, with an extraction/injection well network and an aboveground treatment plant, was implemented as part of interim and final remedies in the 200 West area. The HYPATIA single-page web application (part of the SOCRATES suite) is being developed to provide access to and analysis of chemistry and treatment facility sensor data for this 200 West P&T system. For the web application, analytical algorithms were developed to perform summing, differencing, smoothing, outlier detection, change-point detection, mass flow rate, and injectivity calculations on the data. Candidate algorithms were identified and tested, with the best-performing algorithms then assembled for implementation in HYPATIA. Because HYPATIA is hosted on the Amazon Web Services (AWS) cloud computing platform, algorithms were implemented in a back-end AWS Lambda function that can be called by the HYPATIA front end. The Lambda function applies the requested data processing to specified data via functions written in R, Python, and JavaScript. Development, testing, and review of the data analysis algorithms was completed under an NQA-1 quality program. This new HYPATIA functionality will provide information to support site decisions regarding P&T system performance and optimization.
12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
rabpro: global watershed boundaries, river elevation profiles, and catchment statistics
River and Basin Profiler (rabpro) is a Python package to delineate watersheds, extract river flowlines and elevation profiles, and compute watershed statistics for any location on the Earth’s surface. As fundamental hydrologically-relevant units of surface area, watersheds are areas of land that drain via aboveground pathways to the same location, or outlet. Delineations of watershed boundaries are typically performed on digital elevation models (DEMs) that represent surface elevations as gridded rasters. Depending on the resolution of the DEM and the size of the watershed, delineation may be very computationally expensive. With this in mind, we designed rabpro to provide user-friendly workflows to manage the complexity and computational expense of watershed calculations given an arbitrary coordinate pair. In addition to basic watershed delineation, rabpro will extract the elevation profile for a watershed’s mainchannel flowline. This enables the computation of river slope, which is a critical parameter in many hydrologic and geomorphologic models. Finally, rabpro provides a user-friendly wrapper around Google Earth Engine’s (GEE) Python API to enable cloud-computing of zonal watershed statistics and/or time-varying forcing data from hundreds of available datasets. Altogether, rabpro provides the ability to automate or semi-automate complex watershed analysis workflows across broad spatial extents.
54 ENVIRONMENTAL SCIENCES↗
The disCO2ver Platform: Curating Data and Tools for Geologic Carbon Sequestration and Deep Subsurface Research Systems
The U.S. DOE National Energy Technology Laboratory has invested 12+ years of development into the data repository and digital laboratory, the Energy Data eXchange (EDX, edx.netl.doe.gov). Supporting a variety of research areas across the DOE Office of Fossil Energy and Carbon Management, the platform has successfully curated and preserved thousands of data products from DOE research. The Carbon Storage Program has successfully supported data curation, upload, and publishing of data products on EDX for many years, demonstrating a success story of how resources like EDX can effectively help with long term preservation and publishing of DOE data products. EDX continues to shift towards cloud-supported infrastructure, taking a hybrid approach combining on-premises compute and storage integrated with cloud-hosted services. The integration of cloud compute and hybrid architecture enables the development of EDX-hosted platforms that tailor the data and tools hosted on them to a specific community, enables implementation of machine learning tools for data discovery and filtering, and enables the hosting of virtual (online user interface) tools. Geologic carbon sequestration (GCS) research continues to scale up in response to the current administration goals to reduce greenhouse gas emissions and transition the energy economy. Over the last year, EDX’s disCO2ver platform has been developed in response to the need for access to data products and tools to support the scaling up of GCS research. disCO2ver provides access to data resources and tools, produced by DOE and outside authoritative external resources. The platform also provides a user-access control component for the virtualization and cloud hosting of tools. Tools that need to be virtualized, to eliminate the need for users to download the tool and use local compute resources, is essential to supporting big-data analysis and machine learning that is becoming common place in carbon storage modeling, risk analysis, and data publishing practices. This talk will review the EDX’s disCO2ver platform and the current work ongoing to curate data and tools to support GCS and deep subsurface systems research.
Morkner, Paige↗
Joint Genome Institute Analysis Workflow Service (JAWS) v2.7
The U.S. Department of Energy Joint Genome Institute (JGI) has developed the JGI Analysis Workflow Service (JAWS) as a distributed framework to run computational workflows across diverse high-performance computing (HPC) and cloud environments. JAWS enhances the reusability, scalability, and robustness of scientific workflows by orchestrating data movement, code execution, and results retrieval across multiple DOE facilities. At its core, JAWS integrates the Cromwell workflow engine to run workflows expressed in the Workflow Description Language (WDL), ensuring portability and interoperability. To provide consistent runtime environments, JAWS employs container technologies such as Shifter, Apptainer, and Docker. Workflow tasks are managed via HTCondor on HPC backends, while Globus ensures secure and efficient data transfer between sites. JAWS is deployed as a multi-site workflow manager across national laboratory computing facilities, with dedicated instances supporting community projects such as the National Microbiome Data Collaborative (NMDC) and KBase. This distributed, service-oriented architecture enables users to "write once, run anywhere," providing scalable, production-quality workflow execution.
Kirton, Edward↗
Simulated sulfur K-edge X-ray absorption spectroscopy database of lithium thiophosphate solid electrolytes
X-ray absorption spectroscopy (XAS) is a premier technique for materials characterization, providing key information about the local chemical environment of the absorber atom. In this work, we develop a database of sulfur K-edge XAS spectra of crystalline and amorphous lithium thiophosphate materials based on the atomic structures reported in Chem. Mater., 34, 6702 (2022). The XAS database is based on simulations using the excited electron and core-hole pseudopotential approach implemented in the Vienna Ab initio Simulation Package. Our database contains 2681 S K-edge XAS spectra for 66 crystalline and glassy structure models, making it the largest collection of first-principles computational XAS spectra for glass/ceramic lithium thiophosphates to date. This database can be used to correlate S spectral features with distinct S species based on their local coordination and short-range ordering in sulfide-based solid electrolytes. The data is openly distributed via the Materials Cloud, allowing researchers to access it for free and use it for further analysis, such as spectral fingerprinting, matching with experiments, and developing machine learning models.
36 MATERIALS SCIENCE↗
Optical Particle Measurements during EPCAPE Field Campaign Report
This campaign requested the deployment of the U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) User Facility optical particle counter (OPC) at the first ARM Mobile Facility (AMF1) located at the Scripps Pier in La Jolla, California during the Eastern Pacific Cloud Aerosol Precipitation Experiment (EPCAPE). The addition of the OPC was requested for two reasons. (1) Close the gap between the scanning mobility particle sizer (SMPS) and aerodynamic particle sizer (APS) size distribution from the Aerosol Observing System (AOS) measurements. (2) Principal investigator Petters has been working with Tracking Aerosol Convection Interaction Experiment (TRACER) data to compute particle fluxes from Doppler lidar (Petters et al. 2024). Briefly, backscatter flux is obtained using the eddy covariance technique using the Doppler vertical velocity and attenuated backscatter. Building upon prior studies, we were able to relate backscatter to particle number concentration by calibrating the lidar retrievals against optical particle counter-measured ground-based aerosol size distribution and radiosonde-interpolated relative humidity at lidar sample height. Performing similar analysis was of interest to EPCAPE to better understand the emissions and vertical transport of large particles into the overlying stratus clouds. However, as stated above, this analysis requires an optical size distribution that covers the 0.3-30-μm-diameter size range. The OPC was deployed between 2023-04-14 and 2024-02-14. The deployment, data quality analysis, and data archiving was handled by the DOE ARM instrument mentor team without additional involvement by the principal investigator. Data quality was marked as “routine” for the majority of the campaign.
54 ENVIRONMENTAL SCIENCES↗
The influence of cloud cover on the reliability of satellite-based solar resource data
Satellite-based solar resource data are often developed and validated by using binary cloudiness categories: clear sky or overcast cloudy sky. To investigate the reliability of solar resource data in partially cloudy conditions, we estimate cloud fraction using two distinct algorithms: a physical retrieval model using surface observed global horizontal irradiance (GHI) and direct normal irradiance (DNI) and a temporal average of cloud mask data estimated by the observed DNI. Our analysis reveals a significant presence of scattered clouds, broken clouds, and mismatches between satellite- and surface-based cloud data at 17 surface sites across the contiguous United States, though confidently clear and cloudy conditions collectively account for more than 70 % of the data. Solar radiation is computed using the National Solar Radiation Database (NSRDB) algorithm and validated using surface observations. Here, our findings suggest that, in the presence of scattered clouds, NSRDB data for clear-sky conditions can be subject to significant overestimation. In cloudy-sky conditions classified by satellite data, DNI computed by the Fast All-sky Radiation Model for Solar applications with DNI (FARMS-DNI) can be underestimated when limited clouds are detected by surface observations. The bias observed in several cloudiness categories indicates that the NSRDB is exceptionally accurate in confidently clear conditions. However, clear-sky conditions with scattered clouds and mismatched cloud data contribute significantly to the overall uncertainties in the NSRDB. Therefore, future improvements in solar resource data should involve development and implementation of satellite-derived cloud fraction and should consider a novel radiative transfer model accounting for amplified cloud reflection. The evaluation within cloudiness categories also provides a physical rationale for the superior performance of FARMS-DNI compared to the Direct Insolation Simulation Code (DISC) in both cloudy-sky and all-sky conditions.
14 SOLAR ENERGY↗
Caerbannog Point Clouds
The Caerbannog Point Clouds provide point-sampled 3D models occluded in clouds of points. We synthesized the 3D point clouds from polygonal models, point-sampling the models and surrounding them in a point cloud such that the shape of the model is occluded in any 2D projection. We obscure our model-of-interest by repeatedly surrounding it with an amorphous cloud of points, giving the overall point cloud a structure of organic nature, like that of shrubbery. In a user study, participants were significantly better at identifying the models when visualized as 3D scatterplots under rotation than in axis-aligned 2D scatterplots. We provide three point-clouds, both occluded and unoccluded: the Stanford bunny, Utah Teapot, and OSG Cow.
97 MATHEMATICS AND COMPUTING↗
Quantifying the Solar Energy Resource for Puerto Rico
After Hurricane Maria, multiple U.S. Department of Energy laboratories studied the state of the electric grid in Puerto Rico and analyzed grid resilience and grid integration of renewable energy. As part of the work done at the National Renewable Energy Laboratory, researchers created new solar resource data, conducted a technical potential and supply curve analysis, and studied the interannual variability of the solar resource. A new methodology was developed to downscale solar resource data from the National Solar Radiation Data Base (NSRDB) from a 4-km x 4-km spatial and 30-minute temporal resolution to a 2 km x 2 km and 5-minute resolution. This methodology primarily used simple physical principles to develop high-resolution cloud properties which were then used to compute solar radiation. The high-resolution datasets were validated against ground measurements and the error metrics were found to be similar to the original lower resolution dataset. Using 20 years of downscaled data from the NSRDB multi-year capacity factors for photovoltaics (PV) were developed for both single-axis tracking and fixed latitude-tilt configurations. Use of the multi-year data provides the ability to understand variability in capacity factors due to variability in weather over a long period of time. For Puerto Rico the coastal regions were found to have significant higher capacity factors than inland. Using land-use and terrain information a technical potential analysis was conducted for Puerto Rico. This analysis restricted single PV plant development to a maximum of 100 MW nameplate capacity. The nameplate capacity for each municipality were then determined. Based on our assumptions, 56 of the 78 total municipalities of Puerto Rico contain some level of solar capacity. Most of the interior municipalities did not have any capacity because of the geographic exclusions used in this study. The lowest capacity for a PV plant observed in a municipality was 10 MW. The maximum capacity within a county was 2,000 MW. Further a supply curve analysis was conducted by taking the results of the technical potential and quantifying system and transmission costs. The levelized cost of energy (LCOE) was calculated for each theoretical PV plant site, and the levelized cost of transmission was added to the LCOE to produce a total cost estimate for each site. The results of the supply curve analysis allow for a relative comparison of the cost for integrating new PV capacity into the grid. This analysis indicates that cheaper total LCOE sites tend to be larger in capacity. The total capacity in this study was found to be far beyond the maximum peak load for the island. However, this study does not consider the economic and market potential for development. The cumulative capacity presented in this study assumes that the best locations are developed first and ignores the complex decision paths for new power plant development. Therefore, this analysis can only be treated as illustrative. Finally this study investigates the impact of inter-annual variability of resource using a variety of metrices including probability of exceedance and variation in capacity factor and LCOE. This study demonstrates that the capacity factor or LCOE could vary by over 10% year to year. This clearly indicates the risks involved in using any particular year of data and clearly points to the use of multi-year data to reduce some of the risks related to variability in weather.
14 SOLAR ENERGY↗
Evaluation of Historic Nuclear Cloud Measurement for Entrainment Parameter Determination
We evaluated the uncertainties associated with the entrainment parameters computed from nuclear cloud measurements taken during historic weapon testing campaigns. This evaluation found that variations in the way cloud heights and diameters were defined during the analysis of film records result in very large uncertainties in the entrainment parameters, severely limiting the utility of these data. For the measurements of atmospheric conditions at the time of the tests, the main sources of error are associated with the differences in the location and timing of the soundings.
98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗
IRIS-MEMFLOW: Data Flow-Enabled Portable Memory Orchestration in IRIS Runtime for Diverse Heterogeneity
Task-based programming models and execution paradigms provide a means to decompose a computation by expressing it as a graph in which each node represents a specific computation operating on memory objects and the edges define the dependencies in the execution flow. In this execution model, independent nodes in the graph can be executed concurrently in different computing devices, making it suitable for heterogeneous systems in which computing devices with different architectures coexist. However, careful memory orchestration across heterogeneous devices is needed because copies of the same memory object may reside in multiple devices during execution. Manually ensuring such an orchestration is quite challenging. Not only must an application developer guard against race conditions, but they must also optimize data movement between the host and devices because unnecessary data movement significantly impacts performance. To mitigate these challenges, we enhance the IRIS heterogeneous runtime and introduce IRIS-MEMFLOW–a data flow–enabled portable memory abstraction for seamlessly orchestrating memory in diverse heterogeneous computing environments. By using data-flow analysis, IRIS-MEMFLOW guards against race conditions while multiple heterogeneous devices access memory objects. IRIS-MEMFLOW also optimizes data movement between the host and devices without manual intervention. As a result, IRIS provides improved programming productivity, performance, and portability for multidevice heterogeneous executions in high-performance computing and cloud systems that run diverse architectures from different vendors. The efficacy of IRIS-MEMFLOW is evaluated through experiments that show its capability in terms of programming productivity, multidevice heterogeneity, portability, and low overhead versus the state of the art.
Monil, M. A. H. [ORNL] (ORCID:0000000334194037)↗
Modifying the Asynchronous Jacobi Method for Data Corruption Resilience
Moving scientific computation from high-performance computing (HPC) and cloud computing (CC) environments to devices on the edge, i.e., physically near instruments of interest, has received tremendous interest in recent years. Such edge computing environments can operate on data in situ, offering enticing benefits over data aggregation to HPC and CC facilities that include avoiding costs of transmission, increased data privacy, and real-time data analysis. Because of the inherent unreliability of edge computing environments, new fault-tolerant approaches must be developed before the benefits of edge computing can be realized. Motivated by algorithm-based fault tolerance, a variant of the asynchronous Jacobi (ASJ) method is developed that achieves resilience to data corruption by rejecting solution approximations from neighbor devices according to a bound derived from convergence theory. Numerical results on a two-dimensional Poisson problem show that the new rejection criterion, along with a novel approximation to the shortest path length on which the criterion depends, restores convergence for the ASJ variant in the presence of certain types data corruption. Numerical results are obtained for when the singular values in the analytic bound are approximated. Additional linear systems are also explored, one with a more dense sparsity pattern and one that includes advection. All results indicate that successful resilience to data corruption depends on whether the bound tightens fast enough to reject corrupted data before the iteration evolution deviates significantly from that predicted by the convergence theory defining the bound. This observation generalizes to future work on algorithm-based fault tolerance for other asynchronous algorithms, including upcoming approaches that leverage Krylov subspaces.
97 MATHEMATICS AND COMPUTING↗
Workflows Community Summit: Tightening the Integration between Computing Facilities and Scientific Workflows
Scientific workflows are used almost universally across science domains for solving complex and largescale computing and data analysis problems. The importance of workflows is highlighted by the fact that they have underpinned some of the most significant discoveries of the past decades. Many of these workflows have significant computational, storage, and communication demands, and thus must execute on a range of large-scale computer systems, from local clusters to public clouds and upcoming exascale HPC platforms. Managing these executions is often a significant undertaking, requiring a sophisticated and versatile software infrastructure. Historically, infrastructures for workflow execution consisted of complex, integrated systems, developed in-house by workflow practitioners with strong dependencies on a range of legacy technologies—even including sets of ad hoc scripts. Due to the increasing need to support workflows, dedicated workflow systems were developed to provide abstractions for creating, executing, and adapting workflows conveniently and efficiently while ensuring portability. While these efforts are all worthwhile individually, there are now hundreds of independent workflow systems. These workflow systems are created and used by thousands of researchers and developers, leading to a rapidly growing corpus of workflows research publications. The resulting workflow system technology landscape is fragmented, which may present significant barriers for future workflow users due to many seemingly comparable, yet usually mutually incompatible, systems that exist. In order to tackle some of the challenges described above, the DOE-funded ExaWorks and NSF-funded WorkflowsRI projects have organized in 2021 a series of events entitled the “Workflows Community Summit”. The third edition of the “Workflows Community Summit” explored workflows challenges and opportunities from the perspective of computing centers and facilities. This third summit builds on two prior summits (https://workflowsri.org/summits) that (i) established a high level vision for workflows research; and (ii) explored technical approaches for realizing that vision. The third summit brought together a small group of facilities representatives with the aim to understand how workflows are currently being used at each facility, how facilities would like to interact with workflow developers and users, how workflows fit with facility roadmaps, and what opportunities there are for tighter integration between facilities and workflows. This report documents and organizes the wealth of information provided by the participants before, during, and after the summit.
97 MATHEMATICS AND COMPUTING↗
Watermarks in stream processing systems: semantics and comparative analysis of Apache Flink and Google cloud dataflow
Streaming data processing is an exercise in taming disorder: from oftentimes huge torrents of information, we hope to extract powerful and timely analyses. But when dealing with streaming data, the unbounded and temporally disordered nature of real-world streams introduces a critical challenge: how does one reason about the completeness of a stream that never ends? In this paper, we present a comprehensive definition and analysis of watermarks, a key tool for reasoning about temporal completeness in infinite streams.First, we describe what watermarks are and why they are important, highlighting how they address a suite of stream processing needs that are poorly served by eventually-consistent approaches:• Computing a single correct answer, as in notifications.• Reasoning about a lack of data, as in dip detection.• Performing non-incremental processing over temporal subsets of an infinite stream, as in statistical anomaly detection with cubic spline models.• Safely and punctually garbage collecting obsolete inputs and intermediate state.• Surfacing a reliable signal of overall pipeline health.Second, we describe, evaluate, and compare the semantically equivalent, but starkly different, watermark implementations in two modern stream processing engines: Apache Flink and Google Cloud Dataflow.
Akidau, Tyler↗
Cloud based tool for analysis of chemical kinetic mechanisms
The size and complexity of chemical kinetic models used to simulate combustion of hydrocarbon fuels continue to increase with our improved understanding of the underlying physical mechanisms, along with the improved ability to utilize such models as the result of increased computational power and efficiency. As mechanisms grow beyond thousands of species and tens of thousands of reactions, it becomes increasingly difficult to manually check for errors and physical inconsistencies. In this work, we present several automated methods to check for such issues in the specification of chemical reaction models. First, we demonstrate how discontinuities in thermodynamic data can cause simulation difficulties manifested as long wall-clock times and convergence failures. To correct this type of problem, we describe an automated method for refitting thermodynamic parameters. We also outline several methods to check the timescales of reaction rate coefficients to ensure physical consistency. All the methods are made available through a web-based tool (https://combustiontools.llnl.gov) intended to aid mechanism developers and users to improve the accuracy and performance of fuel models used by the combustion community.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
Accelerating science: The usage of commercial clouds in ATLAS Distributed Computing
The ATLAS experiment at CERN is one of the largest scientific machines built to date and will have ever growing computing needs as the Large Hadron Collider collects an increasingly larger volume of data over the next 20 years. ATLAS is conducting R&D projects on Amazon Web Services and Google Cloud as complementary resources for distributed computing, focusing on some of the key features of commercial clouds: lightweight operation, elasticity and availability of multiple chip architectures. The proof of concept phases have concluded with the cloud-native, vendoragnostic integration with the experiment’s data and workload management frameworks. Google Cloud has been used to evaluate elastic batch computing, ramping up ephemeral clusters of up to O(100k) cores to process tasks requiring quick turnaround. Amazon Web Services has been exploited for the successful physics validation of the Athena simulation software on ARM processors. We have also set up an interactive facility for physics analysis allowing endusers to spin up private, on-demand clusters for parallel computing with up to 4 000 cores, or run GPU enabled notebooks and jobs for machine learning applications. The success of the proof of concept phases has led to the extension of the Google Cloud project, where ATLAS will study the total cost of ownership of a production cloud site during 15 months with 10k cores on average, fully integrated with distributed grid computing resources and continue the R&D projects.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
Physics-Informed Machine Learning from Observations for Clouds, Convection, and Precipitation Parameterizations and Analysis
Fusing learning from Earth observations from space and from the ground (e.g., ARM) with newly developed interpretable and generalizable physics-informed model structures to improve the parameterization of clouds and convection and advance the simulation, understanding and analysis of hydrological extreme events. This covers the prescribed foci of (2) predictive modeling using a hierarchy of models and (3) insight gleaned from complex data.
54 ENVIRONMENTAL SCIENCES↗