SEARCH · Search NASA

Results for “h5py”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

8 records

All sources NASA NTRS DOE OSTI

DOE OSTI2025

Hydrologic Model Data for the East Fork Poplar Creek Watershed Simulated with the Advanced Terrestrial Simulator (ATS): Streamflow and Network Expansion–Contraction Dynamics

This dataset supports hydrologic modeling and stream network expansion–contraction analysis for the East Fork Poplar Creek (EFPC) Watershed in Tennessee. It includes a Jupyter notebook for model setup, model configuration files, simulation outputs, and derived products used to evaluate model performance and investigate stream dynamics under varying hydrologic conditions. The dataset was generated using the Watershed Workflow Python package and the Advanced Terrestrial Simulator (ATS), enabling integrated surface–subsurface hydrologic simulations using a stream-aligned mesh. Outputs include high-resolution time series of streamflow, active network length, water table depth, and related hydrologic variables. Also included are spatially explicit stream persistency indices and classifications of reaches as perennial or non-perennial. These data facilitate reproducibility and support further research on stream intermittency and variability in network extent.The model data archive is organized in following directories:1) model_setup_inputsContains the Watershed Workflow Jupyter notebooks (accessed through any open source code editor), selected input datasets, and resulting ATS input files, including XML files (access through any open source code editor), computational mesh (.exo files can be viewed using Paraview), and meteorological forcing files (.h5 files can be accessed through h5py python package and HDFView open source software). 2) model_outputsIncludes ATS simulation outputs relevant to this study. Time series of spatially integrated or averaged variables (e.g., streamflow, water table depth) are provided as CSV files. Select spatial fields (e.g., ponded depth and water table depth) are saved as pickled Python objects to reduce file size, and can be accessed through pickle package in Python. Key geometry objects from Watershed Workflow—such as the surface mesh and river tree—are also included to support analysis of streamflow persistency and expansion–contraction dynamics. These files can also be accessed through Watershed Workflow Python package.3) model_evaluationProvides observed streamflow time series and field survey-based flow regime classifications used to evaluate model performance. Jupyter notebooks for processing ATS outputs and comparing model predictions with observations to build confidence in the model prior to scientific analysis are also included.4) Q_L_relationshipsContains workflows for generating time series of discharge, active network length, and related hydrologic variables used in the stream network expansion–contraction analysis. Includes routines for delineating baseflow-dominated periods. For each catchment, notebooks and processed data (as pickled DataFrames accessed through Pandas Python package) are provided. 5) figure_scriptsProvides the Jupyter notebooks used to generate the figures presented in the paper.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2026

Permafrost Thaw, Uneven Subsidence and Projected Drying of Ice-wedge Polygon Tundra: Modeling Archive

This dataset is a model archive of the paper Permafrost Thaw, Uneven Subsidence and Projected Drying of Ice-wedge Polygon Tundra (in prep) to support a modeling study investigating how projected increases in Arctic temperature and precipitation will jointly influence hydrologic conditions in ice-rich tundra landscapes. With this dataset, this study is to address the research question: Will Arctic tundra landscapes become wetter or drier with increasing precipitation and temperature in the future when thaw-induced ground subsidence and associated microtopographic evolution are represented? The simulations focus on ice-wedge polygon tundra, a widespread form of ice-rich permafrost terrain that is highly sensitive to thaw-driven landscape change. This dataset contains model input and output data for four study watersheds in Alaska: Anaktuvuk, Utqiagvik (formerly Barrow), Brooks Foothills, and Prudhoe Bay. Simulations were performed using the Advanced Terrestrial Simulator (ATS, v1.5), a physics-rich integrated surface–subsurface hydrologic model. For each watershed, ten modeling cases were performed representing two landscape evolution conditions (with subsidence and without subsidence) combined with five climate forcing scenarios derived from Shared Socioeconomic Pathways (SSP5, SSP5 with precipitation trend, SSP2, SSP2 with precipitation trend, and SSP2 with double precipitation trend). Particularly, for each watershed under the forcing SSP2 with precipitation trend, there are two additional simulations considering spatially heterogeneous subsidence distributions: one assumes randomly distributed scaling and the other includes elevation dependent distribution scaling. These simulations span 1980–2099 and include spin-up runs (1980–2009) followed by transient projections (2010–2099). To facilitate reproducibility of simulations, all datasets are organized by watershed. For each study watershed, the dataset contains: (1) Pre-partitioned mesh files for 32-core modeling (.par.32.XX), located in EACH_WATERSHED/mesh/basin; and also a non-partitioned mesh file (.exo) located in EACH_WATERSHED/mesh; (2) Climate forcings corresponding to the five SSP scenarios (.h5), located in EACH_WATERSHED/data; (3) Final states (.h5) from column spin-up modeling used to initialize historical watershed-scale spin-up runs from 1980 to 2009, located in EACH_WATERSHED/PreSpinupHistorical; (4) Final states (.h5) of historical watershed-scale spin-up runs from 1980 to 2009 used to initialize projection runs, located in EACH_WATERSHED/Spinup_daymetERA5; (5) ATS modeling input files (.xml), located in EACH_WATERSHED/EACH_SIMULATION_SCENARIO/inputfiles; (6) ATS modeling output files (.dat), located in in EACH_WATERSHED/EACH_SIMULATION_SCENARIO/combined_obs; (7) For the Brooks Foothills watershed, additional spatial model outputs are provided (.h5) for selected years (2033 and 2093) used to generate spatial figures in this study, located in Brooksfoothills/EACH_SIMULATION_SCENARIO/results-WITH/WITHOUT_SUBSIDENCE-year2033/2093. All data files with suffix .h5 can be accessible through Python h5py, and all data files with suffix of .dat can be imported by Python pandas. Mesh file with .exo can be visualized through Paraview or read by Python netCDF. The Next-Generation Ecosystem Experiments in the Arctic (NGEE Arctic) project is a research effort to reduce uncertainty in the Department of Energy’s Energy Exascale Earth System Model (E3SM) by developing a predictive understanding of Arctic tundra ecosystems underlain by permafrost and to quantify feedbacks from the Arctic tundra to the Earth system. NGEE Arctic is supported by the Department of Energy's Office of Biological and Environmental Research. Over Phases 1–3, observations made by the NGEE Arctic team across a gradient of permafrost landscapes in Arctic Alaska improved the representation of tundra processes in the land surface component of E3SM (the E3SM Land Model, ELM). Model improvements emphasized unique aspects of permafrost environments and explored reductions in model complexity while retaining predictive power. The Arctic-informed ELM developed by NGEE Arctic has been used to make novel predictions on processes ranging from permafrost thaw to soil biogeochemical cycling to Earth system feedbacks associated with the unique characteristics of tundra plants. In Phase 4, the NGEE Arctic team is evaluating our new predictive understanding under novel conditions across the Arctic domain. In collaboration with partners at long-term pan-Arctic research sites we are examining whether an Arctic-informed ELM can faithfully simulate interactions among surface and subsurface processes at site, regional, and pan-Arctic scales. In turn, we are using variety of tools to dynamically extend and evaluate ELM inference, with an emphasis on data synthesis and pan-Arctic model evaluation, reintegration of code with an evolving E3SM, scaling across heterogeneous Arctic landscapes, and the appropriate representation of the impacts of increasingly frequent Arctic disturbances.

EARTH SCIENCE > ATMOSPHERE > PRECIPITATION↗

DOE OSTI2026

A bespoke model of Arctic river basins based on hillslope delineation: Model Archive

This dataset is a model archive of the paper A bespoke model of Arctic river basins based on hillslope delineation (in prep), which introduces a watershed decomposition and parameterization method for large scale permafrost hydrology simulation. With this dataset, this study aims to address the research question: whether a computationally efficient hillslope-based modeling framework can reliably simulate discharge at Arctic river-basin scales. This dataset contains model input and output data for five modeling scenarios at a study site located in the Sagavanirktok River basin. The five modeling scenarios include three modeling cases under temperate conditions using full 3D, decomposed 3D, and decomposed 2D modeling strategies; and two modeling cases under actual Arctic conditions with permafrost using full 3D and decomposed 2D modeling strategies. Simulations were performed using the Advanced Terrestrial Simulator (ATS, v1.6 for three temperate scenarios and v1.5 for two Arctic scenarios), a physics-rich integrated surface–subsurface hydrologic model with cryo-hydrology features. For the three temperate models, simulations were conducted for the period of 10/01/1993 - 09/30/2002; and for the two Arctic models, simulations were conducted for the period of 01/01/1994 - 12/31/2002. To facilitate reproducibility of simulations, all datasets are organized hierarchically. The dataset contains: (1) Mesh files (.exo) for full 3D model, decomposed 3D models, and decomposed 2D models, located in huc/190604020802_gauge15906000/mesh/. Mesh files can be visualized through Paraview or read by Python. (2) Climate forcings (.h5) for full 3D model and decomposed 3D/2D models are located in huc/190604020802_gauge15906000/daymet_onePiece/, and huc/190604020802_gauge15906000/vp_pr_revised_daymet_1980_2006_with_wind/ separately. Accessible by Python. (3) Raw measured gage discharge (.csv) from USGS, located in huc/190604020802_gauge15906000/gaged_basin15906000_discharge_usgs/. Accessible by Python. (4) Delineated subdomain raster (.tif) and shape files (.shp), and the final parameterized results (.npy) for decomposed models, located in huc/190604020802_gauge15906000/data_preprocessed-meshing. Accessible by Python. (5) Temperate models are located in nonpermaf_huc190604020802_gauge15906000/, which includes three cases: decomposed 2D models (inside model_0*-hillslope_*), decomposed 3D models (inside model_1*-subcatchment_*), and full 3D model (inside model_2*-onepiece_*). Two step spin-up results (checkpoint_final.h5) are located in model_*1-*_spinup_steadystate and model_*2-*_spinup_cycle, separately, which are used to initialize real transient models. The input files (.xml) and output results (.dat) of the real transient models are located in model_*3-*_transient/. Especially, for two example hillslope models (ID=-11 and 11), additional h5py files are included in model_03-hillslope_transient/hillslope-11/, model_03-hillslope_transient/hillslope11, model_13-subcatchment_transient/subcatchment-11/, model_13-subcatchment_transient/subcatchment/11, respectively, which are used to plot the saturation figure (Figure 5) in the manuscript. Accessible by Python. (6) Arctic models are located in huc190604020802_gauge15906000/, which includes two cases: decomposed 2D models (inside model_04-hillslope_transient), and full 3D model (inside model_05-onepiece_transient_mannp1_ra). Three step spin-up results (checkpoint_final.h5) are located in model_01-column_freezeup/, model_02-column_spinup/, model_03-hillslope_spinup/, respectively, which are used to initialize real 2D transient hillslope models. The input files (.xml) and output results (.dat) of transient 2D hillslope models are located in model_04-hillslope_transient/. The input files (.xml) and output results (.dat) of the full 3D transient model is located in model_05-onepiece_transient_mannp1_ra/. The full 3D transient model is initialized by model_02-column_spinup/. Accessible by Python. (7) The MOSART routed discharge results (.csv) under Arctic conditions is located in huc190604020802_gauge15906000/MOSART/. Accessible by Python. (8) All Python codes (.py) used to parameterize full 3D model to decomposed 2D models are located in script/. These codes fit with watershed workflow (a watershed delineation tool) v1.4 under the branch gaob/v1.4 from https://github.com/gaobhub/watershed-workflow.git.

EARTH SCIENCE > CRYOSPHERE↗

DOE OSTI2026

Field and Model Data Associated with the Manuscript “Drivers of Streamflow Intermittency in Humid Regions: 2. Evaluating Controls on Flow Persistence in an Urbanized Catchment”

This package contains field data, modeling files, and scripts supporting the investigation of the drivers of streamflow intermittency in an urbanized catchment. It includes the field data collected from electrical resistivity tomography (ERT) surveys, distributed temperature sensing (DTS), continuous self-potential (SP) monitoring, groundwater and stilling well. In addition, it contains the data and results of the coupled water- and electrical-flow model developed using the COMSOL Multiphysics and Advanced Terrestrial Simulator (ATS), as well as software files and Jupyter notebooks used to process the data and generate figures in the manuscript submitted for peer review. The data archive is organized in the following directories: 1) Climate Includes hourly precipitation and daily evapotranspiration time series (2024 – 2025) provided as CSV files, alongside a text file detailing dataset units. 2) Coupled_model Field_Application subfolder contains the ATS XML input scripts, data files, output data for the SP site. It also contains the Jupyter notebook (Plot_final_calib.ipynb) to visualize the results of the modeled SP, stream-groundwater exchange and moisture content. The flow model simulation is executed using the ATS XML scripts and the included Python script (generate_data_set.py) to convert ATS output to COMSOL-ready input. COMSOL Multiphysics template (.m can only be used with COMSOL with MATLAB) is executed using the ATS output data to simulate the potential field. 3) Discharge Includes the electrical conductivity (EC) time series (provided as CSV files) from salt slug injections. It also includes the Jupyter notebook (Discharge_process.ipynyb) used to estimate discharge. All discharge measurements collated into rating_curve_processed.csv 4) DTS Contains collated DTS data including raw Stokes and anti-Stokes measurement (provided as .h5 file). It also includes DTS processing.ipynb, a Jupyter notebook for calibrating the DTS data using dts_calibration Python package. cooler_calibration.csv is the DTS calibration CSV used in the calibration sequence. 5) ERT Contains raw resistivity data (provided as CSV files), spatial location of each of the electrodes (provided as CSV files), and files used for the resistivity inversion. 6) Slug_test Includes the slug test data at all the groundwater wells provided as CSV files, as well as the Jupyter notebook (Slug_test.ipynb) for calculating hydraulic conductivity. 7) SP Contains the SP data collected in field at the SP sites (provided as CSV files). 8) Well_data Contains two subfolders: 1) Raw, which provides unprocessed pressure, electrical conductivity and temperature timeseries downloaded from the loggers in all the groundwater and stilling wells, and 2) Processed, which contains sorted, QA/QC timeseries data for each well. The data archive also contains data_process.ipynb, a Jupyter notebook used for field data analysis and generating figures (plotting well, SP, climate, and discharge data, as well as calculating head gradient at sites with nested groundwater wells). Note: Code files (.ipynb, .py, .xml) can be opened in any standard code editor, .exo file can be viewed using Paraview, .h5 files can be opened using HDFView software and h5py Python package, and .resipy file can be opened with the open-source ResIPy software.

ATS↗

DOE OSTI2026

Field and Model Data Associated with the Manuscript “Drivers of Streamflow Intermittency in Humid Regions: 1. Evaluating Above- and Below-ground Controls of Flow Persistence in a Forested Catchment”

This package contains field data, modeling files, and scripts supporting the investigation of the drivers of streamflow intermittency in a forested catchment. It includes the field data collected from electrical resistivity tomography (ERT) surveys, ground penetrating radar (GPR), continuous self-potential (SP) monitoring, electromagnetic (EM) imaging, groundwater and stilling well. In addition, it contains the data and results of the coupled water- and electrical-flow model developed using the COMSOL Multiphysics and Advanced Terrestrial Simulator (ATS), as well as software files and Jupyter notebooks used to process the data and generate figures in the manuscript submitted for peer review. The data archive is organized in the following directories: 1) Climate Includes hourly precipitation and daily evapotranspiration time series (2024 – 2025) provided as CSV files, alongside a text file detailing dataset units. 2) Coupled_model Contains two subfolders: Synthetic and Field_Application subfolder. Synthetic subfolder contains the ATS XML input script (can be opened using any code editor) for the four synthetic hydrological cases tested (Connected and gaining, Connected and losing, Disconnected and losing, and dry stream). It also includes other experimental cases to test the influence of precipitation and concentration gradient. For each synthetic case, the flow model simulation is executed using the ATS XML scripts and the included Python script (generate_data_set.py) to convert ATS output to COMSOL-ready input. COMSOL Multiphysics template (.mph can be opened with the commercial software COMSOL and requires a license) is executed using the ATS output data to simulate the potential field. It also includes the Synthetic_model_plot.ipynb (can be opened using any code editor) to visualize the SP result and generate manuscript figures. The data subfolder contains mesh files to run both the ATS (.exo and .stl files can be viewed using Paraview; .h5 files can be opened using HDFView software and h5py Python package) and COMSOL models. Field_Application subfolder contains two subfolders: ES_MDA_inversion and Final_Model. ES_MDA_inversion contains the Python script (.py can be opened using any code editor) and SP observation data used to run the Ensemble Smoother with Multiple Data Assimilation (ES-MDA) inversion sequence to get the optimal model parameters. The Final_model subfolder contains the ATS XML input scripts, data files, output data for the two SP sites. The same workflow steps outlined for the Synthetic subfolder apply here. It also contains the Jupyter notebook (Plot_final_calib.ipynb) to visualize the results of the modeled SP, stream-groundwater exchange and moisture content. 3) Discharge Includes the electrical conductivity (EC) time series (provided as CSV files) from salt slug injections. It also includes the Jupyter notebook (Discharge_process.ipynyb) used to estimate discharge. All discharge measurements collated into rating_curve_processed.csv 4) EM Contains the CSV file of the EM data from the DUALEM-42, including spatial coordinates (x, y, z), apparent conductivity, and in-phase measurements at 2 m coil separations for horizontal coplanar (HCP) and perpendicular (PRP) geometries. 5) ERT Contains raw resistivity data (provided as CSV files), spatial location of each of the electrodes (provided as CSV files), and files used for the resistivity inversion (.resipy can be opened with the open-source ResIPy software). 6) GPR Includes GPR field datasets collected at 100 MHz and 250 MHz antenna frequencies, along with the processing/interpretation project file (GPR_process.gpz can be viewed using EKKO_Project 6, a commercial software by Sensors & Software that requires a license). 7) Slug_test Includes the slug test data at all the groundwater wells provided as CSV files, as well as the Jupyter notebook (Slug_test.ipynb) for calculating hydraulic conductivity. 8) SP Contains the SP data collected in field at the two SP sites (one in the perennial reach and the other in the intermittent reach), provided as DAT files. 9) Well_data Contains two subfolders: 1) Raw, which provides unprocessed pressure, electrical conductivity and temperature timeseries downloaded from the loggers in all the groundwater and stilling wells, and 2) Processed, which contains sorted, QA/QC timeseries data for each well. The data archive also contains data_process.ipynb, a Jupyter notebook used for field data analysis and generating figures (plotting well, SP, climate, and discharge data, as well as calculating head gradient at sites with nested groundwater wells). It also includes DTW.ipynb, a Jupyter notebook containing the code for the dynamic time warping (DTW) with sliding window to evaluate SP signal synchronicity.

ATS↗

DOE OSTI2026

ML-based Micro-CT SOFC Microstructure Models (from Kent 2026 Microstructural Augmentation paper)

Overview -------------------------- This repository contains datasets from the manuscript **"Enhanced Generalizability to Deep-Learning Quantification of 3D Microstructural Characteristics through Microstructurally Aware Augmentation of Scarce Data"** (*William F. Kent, Rochan Bajpai, Rachel C. Kurchin, William K. Epting, Harry W. Abernathy, Paul A. Salvador. Submitted 2026*). The methods are also described in the dissertation **Data Intensive Analysis of Solid Oxide Cell Microstructures** (*Doctoral dissertation, Carnegie Mellon University, 2025*). The datasets here are trained convolutional neural network (CNN) models for predicting key microstructural properties of solid oxide cell (SOC) electrodes from low-res, 2-channel 3D images, as well as some helpful code. The parameters for input images are provided in the paper. Sample data is provided in the file `Combined_anode_aug_dual_1k_examples` - that particular data was used to train `anode_all_aug.pth` and will work most accurately with that model. Please familiarize yourself with all caveats on accuracy and applicability, as detailed in the associated paper. Usage -------------------------- The basic usage is as follows, assuming `model_fn` is the path to the .pth file, and `X` is 2-channel input image(s) of the proper dimensions (either one image of shape `[2,12,24,24]`, or a batch of N input images of shape `[N,2,12,24,24]`): from CNN_inferencer import load_model_for_inference model = load_model_for_inference(model_fn) y_predicted = model(X) The model object automatically handles input scaling and output de-scaling based on the way the models were trained - in other words, pass in a 2-channel micro-CT image, and it will output microstructural property values in real units. ## Other model object attributes Note that model has useful attributes other than its forward pass model(X). * `model.output_descaler` - returns the output descaler object. Model does the de-scaling when generating inferences, but you may want to re-use this de-scaler on other values to e.g. compare predictions to ground truth from already-scaled training data. * `model.prop_names` - Gives the property names of the predicted y values, in order. Only exists if there's an output scaler as part of the model object, which there will be in the models provided here. ## Usage with sample data Here is a short script to use with the included sample data. from CNN_inferencer import display_predictions, load_model_for_inference, calculate_mape, parity_plot import h5py import numpy as np model_fn = 'anode_all_aug.pth' data_fn = 'Combined_anode_aug_dual_1k_examples.h5' N_samples = 200 figure_outdir = '.' model = load_model_for_inference(model_fn) with h5py.File(data_fn,'r') as f: XX = f['X'] #These are the 2-channel 3D images yy = f['y'] #These are the ground-truth microstructural properties, but they have been scaled for training - need to de-scale below N = XX.shape[0] #How many images total in the input data file #Run inferences on N_samples random samples from XX. #Run in a batch, much more efficient than one at a time. ii = np.random.choice(N,N_samples,replace=False) ii.sort() y_pred = model(XX[ii]) #Get the original/true (but normalized/scaled) values from the training dataset... #Because they were normalized, they are not in real units yet. So let's also de-scale them using model.output_scaler. y_true = model.output_scaler.transform(yy[ii]) #Let's display actual values for just 5 random ones for i in np.random.choice(N_samples,5,replace=False): display_predictions(y_true[i], y_pred[i], model.prop_names) #Make parity plots for each property (ground truth vs predicted values) #Also label each plot with the mean abs. percent error (MAPE) of the predicted values for i,key in enumerate(model.prop_names): mape = calculate_mape(y_true[:,i], y_pred[:,i]) parity_plot(y_true[:,i], y_pred[:,i], figure_outdir, key, extra_title=f' ({mape:.2f}% MAPE)')

3D microstructure↗

NASA NTRS2017

Hierarchical Data Formats (HDF) Update

In this presentation, we will talk about the latest releases of HDF4 and HDF5 software and tools, new features available in HDF5, and roadmap for the HDF software. We will also solicit feedback from the users of HDF data and HDF application developers on new features and new tools. The talk will cover: Difference between 1.8 and 1.10 releases and how and when to move to the latest release Features of the recent HDF5 1.8.19, 1.10.1 and HDF 4.2.13 Overview of HDF View 3.0 and other enhancements to tools Supported compilers and systems Open discussion of new requirements and wish list of the HDF features Compression library for interoperability with h5py and Pandas and better floating-point data compression.

HDFView↗

DOE OSTI2024

Delta-Rice: A HDF5 Compression Plugin optimized for Digitized Detector Data

Delta-Rice is an HDF5 (The HDF Group et al., 2020) filter plugin that was developed to compress digitized detector signals recorded by the Nab experiment (Fry et al., 2019), a fundamental neutron physics experiment. This is a two-step process where incoming data is passed through a pre-processing filter and then compressed with Rice coding. A routine for determining the optimal pre-processing filter for a dataset is provided along with an example GPU deployment. When applied to data collected by the Nab data acquisition system, this method produced output files 29% their initial size, and was able to do so with an average read/write throughput in excess of 2 GB/s on a single CPU. Compared to the widely used Gzip compression routine, Delta-Rice reduces the file size by 33% more with over an order of magnitude increase in read/write throughput. Delta-Rice is available on CPU to users through the HDF5 library.

97 MATHEMATICS AND COMPUTING↗