Results for “Github” — Page 23

DOE OSTI2024

Gupta-et-al_2024_EarthsFuture

Results from Gupta et al. submitted to Earth's Future. All code to reproduce the experiment and make the figures can be found here: https://github.com/rg727/Gupta-etal_2024_EarthsFuture The data provided in this repository are (1) Weather Regime Data , (2) Hydroclimate Data, and (3) CALFEWS output. In (1), there are Markov chains of daily weather regimes generated over the 600-year paleo-period. In (2), there are three sets of data: Historical daily CDEC data for 12 input locations into CALFEWS, 600-year long daily paleo data (streamflow and snow) for each input location, and (3) 600-year long daily climate-change data (4 degree temperature increase + 7% precipitation scaling applied to (2)) which serves as the "climate change scenario" in the study. Please reference the GitHub repository on how to use these data to reproduce the results. The CALFEWS output for the Paleo and Climate Change scenarios is stored in (3). More information can be found in the Gupta-et-al_2024_EarthsFuture-README file.

Gupta, Rohini↗

DOE OSTI2025

TRAILS Output Files

Overview This data repository contains ZIP files that store compressed versions of the output of running the WaterPaths utility planning and management tool in the DU Re-Evaluation mode (to download the tool, please see this GitHub repository). The tool was used to simulate the six-utility North Carolina Research Triangle problem. Details on the contents of each ZIP file can be seen below. Data details Temporal range: Weekly data for 2,344 weeks from 2015 to 2060 (45 years). Spatial range: Six water utilities in the North Carolina Research Triangle region (0: Chapel Hil/OWASA, 1: Durham, 2: Cary, 3: Raleigh, 4: Pittsboro, and 5: Chatham) File types: CSV and OUT Different solutions available The solution numbers correspond to the different pathway strategies (henceforth referred to as "solutions") discussed in paper's main and supporting text (abstract and link to the paper here). They are as follows: Sol92: The Durham-focused pathway strategy Sol132: The Raleigh-focused pathway strategy Sol140: The regionally-robust pathway strategy Objectives files These files can be accessed by unzipping solXX_objectives_pathways.zip that contains 1,000 Objectives_RDMXX_solsXX_to_XX.csv files. Each CSV file will consist of a row representing all the objective values for that specific solution, while every six columns represents the reliability, restriction frequency, infrastructure net present value ($ mil), peak financial cost, worst-case cost, and unit cost ($ per MG; in that order) for each of the six utilities. There will be 1,000 such files, denoting the performance of the six utilities across the 1,000 deeply uncertain states of the world (DU SOWs). Pathway files These files can be accessed by unzipping solXX_objectives_pathways.zip that contains 1,000 Pathways_sXX_RDMXX.out file. Each OUT corresponds to the set of infrastructure being triggered in a specific DU SOW, and each file will have the name file will consist of four tab-delimited columns that are described as follows: Realization: The realization in which an infrastructure options being triggered utility: The utility currently triggering infrastructure week: The week in which a specific infrastructure option is being triggered infra.: The infrastructure option being triggered If the OUT file contains only the header line, no infrastructure was triggered for that specific DU SOW. Policies files These files can be obtained by unzipping Policies.zip. Each of the 1,000 CSV files within the unzipped folder will contain weekly water use restriction policies for all 1,000 hydroclimatic realizations within a specific DU SOW. The column structure is as follows: 0rest_m: restriction multiplier for utility 0 (values between 0 and 1) 1rest_m: restriction multiplier for utility 1 (values between 0 and 1) 2rest_m: restriction multiplier for utility 2 (values between 0 and 1) 3rest_m: restriction multiplier for utility 3 (values between 0 and 1) 4rest_m: restriction multiplier for utility 4 (values between 0 and 1) 5rest_m: restriction multiplier for utility 5 (values between 0 and 1) 0transf: transfer volume for utility 0 (in MGD) 1transf: transfer volume for utility 1 (in MGD) 2transf: transfer volume for utility 2 (in MGD) 3transf: transfer volume for utility 3 (in MGD) 4transf: transfer volume for utility 4 (in MGD) 5transf: transfer volume for utility 5 (in MGD) Water Sources files These files can be obtained by unzipping WaterSources_subset.zip. Each of the 100 CSV files within the unzipped folder will contain weekly state variables at each water source for all 1,000 hydroclimatic realizations within a specific DU SOW. The column structure is as follows: Xvolume: available water volume from source X (in MGD) Xs_area: surface area of source X (in ACF) Xdemand: demand drawn from a water source from source X (in MGD) Xup_spill: upstream spillage from source X (in MGD) Xww_inflow: wastewater inflow from source X (in MGD) Xcatch_inflow: upstream catchment inflow to source X (in MGD) Xevap: evaporation multiplier for source X (values between 0 and 1) Xds_spill: downstream spillage from source X (in MGD) X_Y_alloc_cap: the allocated capacity from source X to utility Y (values between 0 and 1) X_Y_alloc_dem: the allocated demand from source X to utility Y (values between 0 and 1) Xtrmt_alloc_Y: the allocated treatment capacity from source X to utility Y (values between 0 and 1) Utilities files These files can be obtained by unzipping Utilities_subset.zip. Each of the 100 CSV files within the unzipped folder will contain weekly state variables at each utility for all 1,000 hydroclimatic realizations within a specific DU SOW. The column structure is as follows: Xst_vol: total available storage volume of utility X (in MG) Xcapacity: total storage capacity of utility X (in MG) Xnet_inf: : net inflow for all storage infrastructure for utility X (in MGD) Xst_rof: short term ROF for utility X (values between 0 and 1) Xst_stor_rof: short-term storage ROF for utility X (values between 0 and 1) Xst_trmt_rof: short-term treatment ROF for utility X (values between 0 and 1) Xlt_rof: long-term ROF for utility X (values between 0 and 1) Xlt_stor_rof: long-term storage ROF for utility X (values between 0 and 1) Xlt_trmt_rof: long-term treatment ROF for utility X (values between 0 and 1) Xrest_demand: restricted demand for utility X (in MGD) Xunrest_demand: unrestricted demand for utility X (in MGD) Xunfulf_demand: unfulfilled demand for utility X (in MGD) Xwastewater: wastewater return for utility X (in MGD) Xtreat_capacity: total treatment capacity for utility X (in MG) Xcont_fund: reserve (contingency) fund balance for utility X Xins_pout: insurance payout for utility X (% annual volumetric revenue) Xins_price: insurance price for utility X (% annual volumetric revenue) Xinfra_npv: infrastructure net present value for utility ($mil) Xst_vol: total available storage volume of utility X (in MG) Xdebt_serv: debt service for utility X (usually once per year if the infrastructure is triggered; % annual volumetric revenue) Xstor_vol: total stored volume (in MGD) Xobs_ann_dem: observed annual demand for utility X (in MGD) Xproj_dem: projected annual demand for utility X (in MGD) Xpv_debt_serv: present value of debt service payments for utility X (% annual volumetric revenue) Xgross_rev: gross revenue for utility X ($mil) Acknowledgment IM3 is a multi-institutional effort led by Pacific Northwest National Laboratory and supported by the U.S. Department of Energy's Office of Science as part of research in MultiSector Dynamics, Earth and Environmental Systems Modeling Program.

Artificial Intelligence↗

DOE OSTI2025

Supporting Information - How Can Crop Production Adapt to growing groundwater restrictions in the West?

Abstract This data plan outlines the structure and content of datasets generated and utilized in the research presented in the manuscript abstract. Groundwater overdraft has led to serious water supply issues in the US West. Most western states manage groundwater use through a permitting system, but California has only recently begun to restrict groundwater use statewide with the 2014 passage of groundwater restrictions which target the elimination of groundwater overdraft practices by 2042. With groundwater extraction curtailed, crop production in the US West (a $95 billion industry annually) will be affected, and appropriate response strategies will be needed to ensure minimal disruption to food production and the regional economy. In this paper, we explore the adoption of alternative adaptive responses: (a) deficit irrigation; (b) switching to less water-intensive crops; (c) changing the extent of irrigated land (including fallowing); and (d) geographically shifting crop production. Employing an integrated modeling approach, we explicitly capture the interactions and feedbacks between local hydrology, changes in crop yields, crop and land use decision-making, changes in crop prices, and regional shifts in crop production. We find that the optimal adaptive response is spatially heterogeneous and comprises a portfolio of strategies. Southwestern states and California will be the most impacted by groundwater restrictions. The optimal responses in these states are to both adopt deficit irrigation strategies and reduce a portion of their irrigated croplands, resulting in a shift in crop production to northwestern states with a larger supply of water. The datasets detailed in this repository represent the output from these integrated models, specifically designed to support the analysis and visualization presented in the manuscript and supplemental information. These datasets, used in conjunction with the scripts available at our associated GitHub repository (https://github.com/pches/Femeena_etal_How_can_crop_production_adapt), enable the reproduction of figures and facilitate a comprehensive understanding of the optimal adaptive strategies in agriculture for mitigating water stress under varying groundwater extraction scenarios, as outlined in the abstract.

DNDCe↗

DOE OSTI2025

Supporting Information - How Can Crop Production Adapt to growing groundwater restrictions in the West?

Abstract This data plan outlines the structure and content of datasets generated and utilized in the research presented in the manuscript abstract. Groundwater overdraft has led to serious water supply issues in the US West. Most western states manage groundwater use through a permitting system, but California has only recently begun to restrict groundwater use statewide with the 2014 passage of groundwater restrictions which target the elimination of groundwater overdraft practices by 2042. With groundwater extraction curtailed, crop production in the US West (a $95 billion industry annually) will be affected, and appropriate response strategies will be needed to ensure minimal disruption to food production and the regional economy. In this paper, we explore the adoption of alternative adaptive responses: (a) deficit irrigation; (b) switching to less water-intensive crops; (c) changing the extent of irrigated land (including fallowing); and (d) geographically shifting crop production. Employing an integrated modeling approach, we explicitly capture the interactions and feedbacks between local hydrology, changes in crop yields, crop and land use decision-making, changes in crop prices, and regional shifts in crop production. We find that the optimal adaptive response is spatially heterogeneous and comprises a portfolio of strategies. Southwestern states and California will be the most impacted by groundwater restrictions. The optimal responses in these states are to both adopt deficit irrigation strategies and reduce a portion of their irrigated croplands, resulting in a shift in crop production to northwestern states with a larger supply of water. The datasets detailed in this repository represent the output from these integrated models, specifically designed to support the analysis and visualization presented in the manuscript and supplemental information. These datasets, used in conjunction with the scripts available at our associated GitHub repository (https://github.com/pches/Femeena_etal_How_can_crop_production_adapt), enable the reproduction of figures and facilitate a comprehensive understanding of the optimal adaptive strategies in agriculture for mitigating water stress under varying groundwater extraction scenarios, as outlined in the abstract.

DNDCe↗

DOE OSTI2026

Inputs to GCAM-USA: IM3 Phase 2 Experiments

Overview This dataset contains XML input files for the IM3 Phase 2 version of GCAM-USA. The files are organized into two categories: Scenario-specific inputs represent hydroclimate and socioeconomic effects on water availability, heating and cooling degree-hours, and agricultural productivity. They support eight IM3 canonical scenarios: rcp45cooler_ssp3 rcp45cooler_ssp5 rcp45hotter_ssp3 rcp45hotter_ssp5 rcp85cooler_ssp3 rcp85cooler_ssp5 rcp85hotter_ssp3 rcp85hotter_ssp5 Model-improvement inputs extend GCAM-USA v5.3 with updated representations of coal and nuclear power plant retirements, electricity trade among U.S. interconnections, offshore carbon storage costs, and groundwater depletion constraints. Data structure Scenario-specific inputs rcp45_runoff/ and rcp85_runoff/XML files describing water availability by HUC2 basin under the RCP 4.5 and RCP 8.5 scenarios. rcp45_hdcd/ and rcp85_hdcd/XML files containing monthly-day and monthly-night heating and cooling degree-hours at the U.S. state level for different RCP-SSP combinations. rcp45_agyields/ and rcp85_agyields/XML files describing changes in agricultural productivity at the intersection of GCAM regions and HUC2 water basins for different RCP-SSP combinations. rcp45_emissions_pathway/The emissions-constraint XML file used to represent the RCP 4.5 pathway. Model-improvement inputs core_retire/Updates coal-fired power plant retirement schedules based on New England ISO. GCAMUSA_IM3_elec_trade_interconnect.xmlRestricts electricity trade to occur within the ERCOT, WECC, and IE interconnections. nuclear_USA.xmlUpdates the retirement schedules of the Diablo Canyon and Palisades nuclear power plants. high_cost_offshore_carbon.xmlUpdates the assumed cost of offshore carbon storage. water_supply_constrained_gleeson_5pct.xmlReplaces WaterGAP historical groundwater-depletion estimates with data from the Gleeson dataset and limits groundwater extraction to 5% of the available groundwater in each Superwell grid cell. How to use the data This dataset is designed for use with the IM3 version of GCAM-USA. Download or clone GCAM-USA from the IM3 GCAM GitHub repository at https://github.com/IMMM-SFA/gcam-core and check out the gcam-usa-im3 branch. Place the downloaded folder im3scenarios in the gcam-core/input directory while preserving the provided folder structure.

Energy↗

DOE OSTI2025

Supporting Information - How Can Crop Production Adapt to growing groundwater restrictions in the West?

Abstract This data plan outlines the structure and content of datasets generated and utilized in the research presented in the manuscript abstract. Groundwater overdraft has led to serious water supply issues in the US West. Most western states manage groundwater use through a permitting system, but California has only recently begun to restrict groundwater use statewide with the 2014 passage of groundwater restrictions which target the elimination of groundwater overdraft practices by 2042. With groundwater extraction curtailed, crop production in the US West (a $95 billion industry annually) will be affected, and appropriate response strategies will be needed to ensure minimal disruption to food production and the regional economy. In this paper, we explore the adoption of alternative adaptive responses: (a) deficit irrigation; (b) switching to less water-intensive crops; (c) changing the extent of irrigated land (including fallowing); and (d) geographically shifting crop production. Employing an integrated modeling approach, we explicitly capture the interactions and feedbacks between local hydrology, changes in crop yields, crop and land use decision-making, changes in crop prices, and regional shifts in crop production. We find that the optimal adaptive response is spatially heterogeneous and comprises a portfolio of strategies. Southwestern states and California will be the most impacted by groundwater restrictions. The optimal responses in these states are to both adopt deficit irrigation strategies and reduce a portion of their irrigated croplands, resulting in a shift in crop production to northwestern states with a larger supply of water. The datasets detailed in this repository represent the output from these integrated models, specifically designed to support the analysis and visualization presented in the manuscript and supplemental information. These datasets, used in conjunction with the scripts available at our associated GitHub repository (https://github.com/pches/Femeena_etal_How_can_crop_production_adapt), enable the reproduction of figures and facilitate a comprehensive understanding of the optimal adaptive strategies in agriculture for mitigating water stress under varying groundwater extraction scenarios, as outlined in the abstract.

DNDCe↗

DOE OSTI2026

Repository of HydroSMADE: Hydropower Site-level Monthly Availability Data Ensemble for 1950-2100 at Existing and Potential Global Sites

This repository presents HydroSMADE—Hydropower Site-level Monthly Availability Data Ensemble, a new open dataset that provides monthly hydropower availability for 1,593 existing and 124,333 potential sites worldwide over the period 1950–2100. The dataset is generated by using a global hydrologic model (Xanthos) with explicit representation of hydropower operation. Specifically, HydroSMADE distinguishes between storage and diversion sites, applies optimized operating rules, and incorporates site-specific characteristics such as generation capacity, maximum turbine flow, and reservoir storage. Driven by bias-corrected meteorological inputs, the data is provided for 30 alternative future scenarios. The scenarios consist of the full factorial combination of three standard CMIP6 atmospheric forcing pathways (SSP1-2.6, SSP3-7.0, and SSP5-8.5) and ten CMIP6 General Circulation Models (GCMs): GFDL-ESM4, IPSL-CM6A-LR, MPI-ESM1-2-HR, MRI-ESM2-0, EC-Earth3, CanESM5, MIROC6, CNRM-ESM2-1, UKESM1-0-LL, and CNRM-CM6-1. The repository contains a total of 122 files: a text file (readme.txt) containing a brief description of the included data, a CSV file containing site attributes, and the remaining 120 files (in CSV) containing site-level monthly hydropower availability. Example Jupyter Notebooks to explore the HydroSMADE dataset are available on GitHub at https://github.com/kamal0013/HydroSMADE More details on the methods and technical validation of HydroSMADE are available in the following paper by the same authors: Chowdhury, A. K., Abeshu, G. W., Zhao, M., Wild, T. B., Hassan, N., Ying, Z., Kim, G. J., Matthew, B., Jonathan, L., & Li, H.-Y. (Submitted). Hydropower Site-level Monthly Availability Data Ensemble for 1950-2100 at Existing and Potential Global Sites.

Existing and Potential Sites↗

DOE OSTI2026

Sensitivity Analysis of Drivers Water Shortage in the Los Angeles Region During Drought

The code and detailed step-by-step instructions for generating the model output data, processing results, and analysis and plotting are provided at https://github.com/IMMM-SFA/Ferencz_et_al_2026_ER_Water. The PyArtes model is a python adaptation of the Artes model. PyArtes uses many of the same input data and optimization model architecture as Artes. Documentation for the PyArtes model is provided in the Supplement to the paper. The primary data product are simulated monthly water shortages for indoor and outdoor demand under a large ensemble of drought scenarios (>13,000). The droughts are hypothetical and are not based on historical time series data of supply sources - though historical data did help inform ranges explored for supply parameters. Demands are informed by recent 2017-2021 water supply data. Demands used for the model can be accessed at https://github.com/IMMM-SFA/Ferencz_et_al_2026_ER_Water. Simulations resolve demand for over 90 water providers in the study region. The results report 36 months of water shortage data for each indoor and outdoor demand node. The study also developed a multilayer perceptron (MLP) neural network trained on a subset of the simulated shortage ensemble to emulate worst annual water shortage for a given set of parameter multipliers -- provided the parameter values fall within the ranges sampled in the ensemble. Emulated water shortages for synthetic ensembles are in the MLP-generated shortages folder. The MLP model was used to generate larger ensembles to support Sobol analysis that would have been extremely computationally expensive to simulate. Datasets provided in this repository*: Simulated shortages. These results are used for the analysis for Figures 5, 8, and 9 in the paper, and also to train the MLP emulator. .zip file containing outputs for the 13,312 scenario ensemble. Separate .csv files for indoor and outdoor shortage for each scenario. Rows = demand ids (~100), Columns = months (36) Units = acre-feet/month of shortage (shortage = monthly demand - supply). 1 acft = 1233.48 m^3 .csv files of aggregated shortages derived from the 13,312 ensemble Rows = scenarios (13,312), Columns = demand ids (~100) Units = acre-feet/year (either worst annual shortage or total shortage over the 3-year drought) .csv file of the parameter multipliers scenarios for the ensemble .csv file of the parameter ranges and baseline values the multipliers were applied to MLP-generated shortages. These results are used for Figures 4, 6, and 7 in the paper. mwd higher folder: scenario ensembles, emulated worst year total shortages (acft), and Sobol results Emulated shortages. Rows = scenarios, columns = demand ids, units acft Sobol results. Rows = demand ids, columns Sobol (S1, ST, or 95% confidence interval) value for each parameter mwd lower folder: scenario ensembles, emulated worst year total shortages (acft), and Sobol results same organization as mwd higher MLP performance: performance metrics (R^2, RMSE, BIAS, MAPE) for the testing subset (20% or 2,662 scenarios) and simulated vs emulated worst year shortage (acre-feet/year) for every demand node, MWD wholesale regions, and the entire study region (LAC). Supporting data for figures. Figure plotting scripts in the associated GitHub repo. These files support analysis and visualization. Geospatial Data used for plotting simulated water shortages and Sobol results. Dictionary of full names for demand nodes in the model and estimates of water supply by source type informed by Artes input files and California Urban Water Management Planning data: https://water.ca.gov/Programs/Water-Use-And-Efficiency/Urban-Water-Use-Efficiency/Urban-Water-Management-Plans *Readme files provided for each folder.

drought↗

DOE OSTI2026

MSD CoP Webinar: "Advances in MSD-LIVE to Support the MSD Community of Practice"

Context: This webinar was hosted by the MultiSector Dynamics Community of Practice (MSD CoP; https://multisectordynamics.org). Advances in MSD-LIVE to Support the MSD Community of Practice Presenters: Casey Burleyson and Zoe Guillen (Pacific Northwest National Laboratory) Abstract: The MultiSector Dynamics Living, Intuitive, Value-adding, Environment (MSD-LIVE; msdlive.org) is a cloud-based data management system and advanced computing platform that enables MSD researchers to document and archive their data, run their models and analysis tools, and share their data, software, and workflows within the MSD Community of Practice. Recently, several high-profile datasets have attracted many new users to MSD-LIVE. This webinar has two goals: 1) To refamiliarize the MSD community and new users with the components of the platform (e.g., the data repository, model training notebooks, and data dashboards) and to highlight examples of how these components are advancing MSD science and 2) To demonstrate new features in v3 of the platform, released in late 2025. The main new feature in v3 is the ability to interactively explore data in MSD-LIVE without downloading it. MSD-LIVE users can now click a button in our data repository and launch a blank Jupyter notebook with access to the underlying data on AWS. Users can use the notebook to write analysis, visualization, or subsetting routines that process the data directly on the AWS cloud. We also added a GitHub integration feature that allows users to share analysis or visualization code they develop with the community of MSD-LIVE users. The webinar will wrap up with a look at what's coming next for MSD-LIVE in 2026. Moderator: Patrick M. Reed (MSD CoP Facilitation Team) This webinar was held on: May 12th, 2026 from 1-2 PM EST.

Open Science↗

DOE OSTI2017

ATLAS (Automatic Tool for Local Assembly Structures) - A Comprehensive Infrastructure for Assembly, Annotation, and Genomic Binning of Metagenomic and Metaranscripomic Data

ATLAS (Automatic Tool for Local Assembly Structures) is a comprehensive multiomics data analysis pipeline that is massively parallel and scalable. ATLAS contains a modular analysis pipeline for assembly, annotation, quantification and genome binning of metagenomics and metatranscriptomics data and a framework for reference metaproteomic database construction. ATLAS transforms raw sequence data into functional and taxonomic data at the microbial population level and provides genome-centric resolution through genome binning. ATLAS provides robust taxonomy based on majority voting of protein coding open reading frames rolled-up at the contig level using modified lowest common ancestor (LCA) analysis. ATLAS provides robust taxonomy based on majority voting of protein coding open reading frames rolled-up at the contig level using modified lowest common ancestor (LCA) analysis. ATLAS is user-friendly, easy install through bioconda maintained as open-source on GitHub, and is implemented in Snakemake for modular customizable workflows.

genome binning↗

DOE OSTI2022

WaterTAP3 Model Results for NAWI's Baseline Analyses

Description: This folder contains the results for the WaterTAP3 model that was used for the eight NAWI (National Alliance for Water Innovation) baseline studies published in the Environmental Science and Technology special issue: Technology Baselines and Innovation Priorities for Water Treatment and Supply. The data structure and content are described in a README.txt file. For more details on how to use the data and interpret the results please refer to the model documentation and GitHub site linked in the submission.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2020

DISCo-microbe: design of an identifiable synthetic community of microbes

Microbiomes are extremely important for their host organisms, providing many vital functions and extending their hosts’ phenotypes. Natural studies of host-associated microbiomes can be difficult to interpret due to the high complexity of microbial communities, which hinders our ability to track and identify individual members along with the many factors that structure or perturb those communities. For this reason, researchers have turned to synthetic or constructed communities in which the identities of all members are known. However, due to the lack of tracking methods and the difficulty of creating a more diverse and identifiable community that can be distinguished through next-generation sequencing, most such in vivo studies have used only a few strains. To address this issue, we developed DISCo-microbe, a program for the design of an identifiable synthetic community of microbes for use in in vivo experimentation. The program is composed of two modules; (1) create, which allows the user to generate a highly diverse community list from an input DNA sequence alignment using a custom nucleotide distance algorithm, and (2) subsample, which subsamples the community list to either represent a number of grouping variables, including taxonomic proportions, or to reach a user-specified maximum number of community members. As an example, we demonstrate the generation of a synthetic microbial community that can be distinguished through amplicon sequencing. The synthetic microbial community in this example consisted of 2,122 members from a starting DNA sequence alignment of 10,000 16S rRNA sequences from the Ribosomal Database Project. We generated simulated Illumina sequencing data from the constructed community and demonstrate that DISCo-microbe is capable of designing diverse communities with members distinguishable by amplicon sequencing. Using the simulated data we were able to recover sequences from between 97–100% of community members using two different post-processing workflows. Furthermore, 97–99% of sequences were assigned to a community member with zero sequences being misidentified. We then subsampled the community list using taxonomic proportions to mimic a natural plant host–associated microbiome, ultimately yielding a diverse community of 784 members. DISCo-microbe can create a highly diverse community list of microbes that can be distinguished through 16S rRNA gene sequencing, and has the ability to subsample (i.e., design) the community for the desired number of members and taxonomic proportions. Although developed for bacteria, the program allows for any alignment input from any taxonomic group, making it broadly applicable. The software and data are freely available from GitHub and Python Package Index (PYPI).

16S rRNA↗

DOE OSTI2022

Exploring MDSplus data-acquisition software and custom devices

MDSplus is a software tool designed for data acquisition, storage, and analysis of complex scientific experiments. Over the years, MDSplus has primarily been used for data management for fusion experiments. This paper demonstrates that MDSplus can be used for a much wider variety of systems and experiments. We present a step-by-step tutorial describing how to create a simple experiment, manage the data, and analyze it using MDSplus and Python. To this end, a custom example device was developed to be used as the data source. This device was built on an opensource electronic hardware platform, and it consists of a microcontroller and two sensors. We read data from these sensors, store it in MDSplus, and use JupyterLab to visualize and process it. This project and code demo are available on the GitHub site at this URL: https://github.com/santorofer/MDSplusAndCustomeDevices

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2024

Y1000 gene family

All relative analysis data and codes used in this study are available on the Figshare data repository. You can also find the codes on the Github repository at https://github.com/vnuii/Y1000_gene_family.

budding yeast↗

DOE OSTI2024

Generic and ML Workloads in an HPC Datacenter: Node Energy, Job Failures, and Node-Job Analysis

HPC datacenters offer a backbone to the modern digital society. Increasingly, they run Machine Learning (ML) jobs next to generic, compute-intensive workloads, supporting science, business, and other decision-making processes. However, understanding how ML jobs impact the operation of HPC datacenters, relative to generic jobs, remains desirable but understudied. In this work, we leverage long-term operational data, collected from a national-scale production HPC datacenter, and statistically compare how ML and generic jobs can impact the performance, failures, resource utilization, and energy consumption of HPC datacenters. Our study provides key insights, e.g., ML-related power usage causes GPU nodes to run into temperature limitations, median/mean runtime and failure rates are higher for ML jobs than for generic jobs, both ML and generic jobs exhibit highly variable arrival processes and resource demands, significant amounts of energy are spent on unsuccessfully terminating jobs, and concurrent jobs tend to terminate in the same state. We open-source our cleaned-up data traces on Zenodo (https://doi. org/10.5281/zenodo.13685426), and provide our analysis toolkit as software hosted on GitHub (https://github.com/atlarge-research/2024-icpads-hpc-workload-characterization). This study offers multiple benefits for data center administrators, who can improve operational efficiency, and for researchers, who can further improve system designs, scheduling techniques, etc.

crossanalysis↗

NASA NTRS2016

The Trick Simulation Toolkit: A NASA/Open source Framework for Running Time Based Physics Models

This paper describes the design and use at of the Trick Simulation Toolkit, a simulation development environment for creating high fidelity training and engineering simulations at the NASA Johnson Space Center and many other NASA facilities. It describes Trick's design goals and how the development environment attempts to achieve those goals. It describes how Trick is used in some of the many training and engineering simulations at NASA. Finally it describes the Trick NASA/Open source project on Github.

Penn, John M.↗

NASA NTRS2015

The Open Data Repositorys Data Publisher

Data management and data publication are becoming increasingly important components of researcher's workflows. The complexity of managing data, publishing data online, and archiving data has not decreased significantly even as computing access and power has greatly increased. The Open Data Repository's Data Publisher software strives to make data archiving, management, and publication a standard part of a researcher's workflow using simple, web-based tools and commodity server hardware. The publication engine allows for uploading, searching, and display of data with graphing capabilities and downloadable files. Access is controlled through a robust permissions system that can control publication at the field level and can be granted to the general public or protected so that only registered users at various permission levels receive access. Data Publisher also allows researchers to subscribe to meta-data standards through a plugin system, embargo data publication at their discretion, and collaborate with other researchers through various levels of data sharing. As the software matures, semantic data standards will be implemented to facilitate machine reading of data and each database will provide a REST application programming interface for programmatic access. Additionally, a citation system will allow snapshots of any data set to be archived and cited for publication while the data itself can remain living and continuously evolve beyond the snapshot date. The software runs on a traditional LAMP (Linux, Apache, MySQL, PHP) server and is available on GitHub (http://github.com/opendatarepository) under a GPLv2 open source license. The goal of the Open Data Repository is to lower the cost and training barrier to entry so that any researcher can easily publish their data and ensure it is archived for posterity.

Astrobiology data↗

NASA NTRS2016

Holodeck Testbed Project

The main objective of the Holodeck Testbed is to create a cost effective, realistic, and highly immersive environment that can be used to train astronauts, carry out engineering analysis, develop procedures, and support various operations tasks. Currently, the Holodeck testbed allows to step into a simulated ISS (International Space Station) and interact with objects; as well as, perform Extra Vehicular Activities (EVA) on the surface of the Moon or Mars. The Holodeck Testbed is using the products being developed in the Hybrid Reality Lab (HRL). The HRL is combining technologies related to merging physical models with photo-realistic visuals to create a realistic and highly immersive environment. The lab also investigates technologies and concepts that are needed to allow it to be integrated with other testbeds; such as, the gravity offload capability provided by the Active Response Gravity Offload System (ARGOS). My main two duties were to develop and animate models for use in the HRL environments and work on a new way to interface with computers using Brain Computer Interface (BCI) technology. On my first task, I was able to create precise computer virtual tool models (accurate down to the thousandths or hundredths of an inch). To make these tools even more realistic, I produced animations for these tools so they would have the same mechanical features as the tools in real life. The computer models were also used to create 3D printed replicas that will be outfitted with tracking sensors. The sensor will allow the 3D printed models to align precisely with the computer models in the physical world and provide people with haptic/tactile feedback while wearing a VR (Virtual Reality) headset and interacting with the tools. Getting close to the end of my internship the lab bought a professional grade 3D Scanner. With this, I was able to replicate more intricate tools at a much more time-effective rate. The second task was to investigate the use of BCI to control objects inside the hybrid reality ISS environment. This task looked at using an Electroencephalogram (EEG) headset to collect brain state data that could be mapped to commands that a computer could execute. On this Task, I had a setback with the hardware, which stopped working and was returned to the vendor for repair. However, I was still able to collect some data, was able to process it, and started to create correlation algorithms between the electrical patterns in the brain and the commands we wanted the computer to carry out. I also carried out a test to investigate the comfort of the headset if it is worn for a long time. The knowledge gained will benefit me in my future career. I learned how to use various modeling and programming tools that included Blender, Maya, Substance Painter, Artec Studio, Github, and Unreal Engine 4. I learned how to use a professional grade 3D scanner and 3D printer. On the BCI Project I learned about data mining and how to create correlation algorithms. I also supported various demos including a live demo of the hybrid reality lab capabilities at ComicPalooza. This internship has given me a good look into engineering at NASA. I developed a more thorough understanding of engineering and my overall confidence has grown. I have also realized that any problem can be fixed, if you try hard enough, and as an engineer it is your job to not only fix problems but to embrace coming up with solutions to those problems.

Arias, Adriel↗