Search NASA⌕ Search

SEARCH · Search NASA

Results for “File System”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Controller area network decoder (CAN-D)

A system and method for decoding an unknown automotive controller area network (“CAN”) message definitions. CAN data vehicle signal mappings are typically held in secret and varied by automotive model and year. Without knowledge of the mappings, the wealth of real-time vehicle data hidden in the automotive CAN packets is uninterpretable—impeding research, after-market tuning, efficiency and performance monitoring, fault diagnosis, and privacy-related technologies. This technology can ascertain the CAN signals' boundaries (start bit and length), endianness (byte ordering), signedness (binary-to-integer encoding) from raw CAN data. This allows conversion of CAN data to time series. Interpreting the translated CAN data's physical meaning and finding a linear mapping to standard units (e.g., knowing the signal is speed and scaling values to represent units of miles per hour) can be achieved for many signals by leveraging diagnostic standards to obtain real-time measurements of in-vehicle systems. The system and method can be integrated into lightweight hardware enabling an OBD-II plugin for real-time in-vehicle CAN decoding or run on standard computers. The system can output a standard DBC file with the signal definition information.

Verma, Kiren E.↗
DOE OSTI2024

System for controller area network payload decoding

A system for decoding an unknown automotive controller area network (“CAN”) message definitions. CAN data vehicle signal mappings are typically held in secret and varied by automotive model and year. Without knowledge of the mappings, the wealth of real-time vehicle data hidden in the automotive CAN packets is uninterpretable—impeding research, after-market tuning, efficiency and performance monitoring, fault diagnosis, and privacy-related technologies. This system can ascertain the CAN signals' boundaries (start bit and length), endianness (byte ordering), signedness (binary-to-integer encoding) from raw CAN data. This allows conversion of CAN data to time series. Interpreting the translated CAN data's physical meaning and finding a linear mapping to standard units (e.g., knowing the signal is speed and scaling values to represent units of miles per hour) can be achieved for many signals by leveraging diagnostic standards to obtain real-time measurements of in-vehicle systems. The system can be integrated into lightweight hardware enabling an OBD-II plugin for real-time in-vehicle CAN decoding or run on standard computers. The system can output a standard DBC file with the signal definition information.

Verma, Kiren E.↗
DOE OSTI2025

Online and Offline Data Quality Monitoring for the Mu2e Calorimeter

This thesis presents the design, implementation, and validation of a calorimeter Data Quality Monitoring (DQM) toolchain for the Mu2e experiment at Fermilab. Mu2e searches for charged lepton flavor violation via coherent muon-to-electron conversion in the field of an aluminum nucleus, $\mu^- Al \rightarrow e^-Al$, a process whose observation would constitute clear evidence of physics beyond the Standard Model. Achieving target sensitivity requires stringent control of detector performance and data integrity during acquisition, as subtle issues in readout configuration, data formatting, or electronics behavior can compromise reconstruction and bias downstream analyzes. To address these challenges, this work develops a multi-layer DQM approach spanning both raw data validation and reconstructed digi-level diagnostics. At the low level, a fragment analysis component performs word- and bit-field decoding of calorimeter readout blocks, enabling sanity checks of the expected structure and producing detailed error and integrity statistics useful for commissioning and troubleshooting. At the digi level, the CaloDigiDQM analyzer is implemented within the art framework and transforms each CaloDigiCollection into a structured hierarchy of ROOT histograms designed for fast drill-down diagnostics. The module generates coherent monitoring views at global, disk, board, and channel granularity, including occupancy, waveform-derived features (baseline, RMS, peak amplitude and position), and left-right sensor consistency metrics. Detector-aware channel-to-electronics mapping is performed through the conditions system (CaloDAQMap), ensuring that diagnostics remain aligned with hardware identifiers used in operations. For end-to-end testing without reliance on live DAQ data, a synthetic CaloDigi producer is developed to generate realistic waveforms with controlled noise and pulse shapes. The resulting system supports both offline ROOT-file production and online operation, including optional histogram streaming through otsdaq via ots::HistoSender. This toolchain provides a practical and scalable foundation for calorimeter commissioning and stable data collection, enabling early detection of anomalies and reducing operational risk for Mu2e.

Vakulenko, Mark [Drew U.] (ORCID:0009000276197818)↗
DOE OSTI2019

Frio 2 : Raw CASSM Datasets

This submission contains the raw SEG-2 seismic datasets acquired in the CASSM experiment, part of the Frio-2 GCS pilot. The directory also includes the log files generated by the Geode seismograph (geometrics) used during acquisition. All SEG-2 files have a ".dat" suffix and log files have a ".log" suffix. Various supplementary files are also included (see below) including a set of flow and seismic models used in interpretation. The earlyGathers.tar.gz resource includes raw data acquired during system testing and pre-injection periods. All files correspond to the single source continuous active source seismic monitoring (CASSM) dataset described in the following papers. Daley, T.M., Solbau, R.D., Ajo-Franklin, J.B., and Benson, S.M. 2007, “Continuous Active-Source Seismic Monitoring of CO2 Injection in a Brine Aquifer”, Geophysics, Vol. 72, No. 5, p. A57-A61 Daley, T., Ajo-Franklin, J.B., and C. Doughty, 2011, “Constraining the reservoir model of an injected CO2 plume with crosswell CASSM at the Frio-II Brine Pilot”, International Journal of Greenhouse Gas Control (IJGGC), Vol. 5, No. 2, pp. 1022-1030. Zhu, T., Ajo-Franklin, J., and T. Daley, 2017, “Spatio-temporal changes in seismic attenuation caused by injected CO2 at the Frio-II pilot site, Dayton TX, USA”, Journal of Geophysical Research – Solid Earth, Vol. 122, No. 9, pp. 7156-7171. cassmDataRaw.tar : The main tar file containing all of the SEG-2 field records and the Geode log files. Stacking, time sampling (dt), and date information should be in the .log file. Each file corresponds to a single stacked gather for the single source deployed. Data was recorded on a 24 channel downhole hydrophone array. fieldNotes.tar : contains some of the scanned observer logs from the experiments as well as some planning images fieldPhotos.tar : contains a small set of photographs taken during the experiment frio_flow_seismic_models.mat : a matlab structure containing a set of flow and seismic models used in interpreting the CASSM data (see IJGGC paper).

GCS↗
DOE OSTI2018

Sandia Cloud Forensics and Incident Response Platform v 1.0

As systems and devices become virtualized and deployed in the cloud, the hypervisor becomes an increasingly appropriate place to collect performance data, system state, system landscape, function calls, transaction traces, and other characteristics. We developed a method by which an introspection application may be coupled with a hypervisor to “reach into” the VM with minimal intrusiveness to collect data critical to the reconstruction of events, files, and operations. Such a capability is required to take advantage of the hypervisor as an instrumentation platform and to integrate that data with more traditional collection mechanisms. The concept of a VM serviced by a lightweight hypervisor is a relatively new paradigm for forensic practitioners. Traditional forensic techniques, based on assumptions that the filesystem was directly interacting with the hardware through an abstraction, afforded the forensic practitioner the assumption that there was nothing controlling the application below the filesystem. This is not the case when using virtualized technologies. Hypervisors can covertly monitor, introspect and interact with the guest in a transparent fashion. The problems of storage and collection of actionable data are exhausting. The current challenge is most hypervisors do not expose a useful application programming interface (API) at a sufficient level to do transparent, fine-grained and customizable introspection. Scalable VM instrumentation and introspection at an in-depth level requires fast handling of events, as well as direct access to VM state. Furthermore, deep introspection benefits greatly from the ability to gather data from the hardware during the VM’s exit to the hypervisor. All of this requires identical access to the system as the hypervisor itself; improper use of this ability could easily cause system instability. It is for this reason we believe that the hypervisor developers have been hesitant to grant this much control through their APIs. However, our approach leverages other means to collect and monitor the guests in a targeted fashion. Virtual machine introspection (VMI) is a technique used to monitor the runtime state of a system-level virtual machine. The runtime state can be may include processor registers, memory, disk, network, and any other hardware-level events A review of research literature and current VMI technologies exposed several limitations and tradeoffs in VMI approaches, including: the use of in-guest agents; kernel to user space transitions (dramatically slowing down processing); VMI tool pre-configuration requirements; hypervisor version lock-in or source code patching; reliance on operating system symbols; limited processor features due to hypervisor (even if the hardware could do more). To address these constraints, a VMI tool was envisioned to provide the cloud forensic capabilities while having as few of these limitations as possible. The Kernel-based Virtual Machine Introspection (KVMi) tool was developed for hypervisors on Intel’s x86-64 architecture. To meet performance, scoping and use-case demands, the follow criteria were applied to KVMi to support live, forensic data collection: • Shall not require in-guest agents. • Shall work with any recent version of modern hypervisors. • Shall introspect upon any guest (VM) running modern versions of Windows, Linux and OS X. • Shall have the ability to find and track operating system artifacts • Shall not require operating system symbol files. • Shall be able to fully handle VM-exits, bypassing execution of the hypervisor if necessary, to facilitate new features the hypervisor may not support. • Shall be compilable/loadable on a running system with standard build tools.

Stout, WilliamM.S.↗
DOE OSTI2026

Frontier Job-Centric Telemetry Dataset

Comprehensive analysis of high-performance computing (HPC) systems requires linking workload execution to system behavior. This kind of analysis is vital for diagnosing performance issues, managing capacity, detecting anomalous workloads, and understanding how applications interact with system hardware. This job-centric telemetry dataset unifies scheduler job records with node-level measurements, enabling direct association between workloads and their corresponding power, thermal, and performance characteristics. It contains sanitized, scheduler related metadata for 152,400 individual jobs that ran on the Frontier supercomputer and ended on selected days throughout 2024 and 2025, a subpopulation of ~6.8% of the total number of allocated jobs with non-zero run time on the system over that same period. Each is linked with files that contain telemetry time series records of the power utilization and temperature behavior of its allocated nodes and their processors during the run time of the job. Where available, a portion of the job files also contain network performance time series. Jobs are sampled from select days that reflect normal levels of user activity and possess job size distributions with large numbers of leadership class jobs (>20% of Frontier nodes). Jobs in this dataset attempt to best represent successful user workflows.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2019

eZeeKonfigurator v1.0

eZeeKonfigurator is a web application and supporting tools for configuring Zeek (formerly Bro) systems. Zeek is an open-source tool developed at LBNL that's used to secure computer networks. A recent version of Zeek introduced the configuration framework, where changes can be made while the system is running. However, the provided method is difficult to operationalize, as it requires a set of text files to be deployed and updated across all the Zeek systems. eZeeKonfigurator provides an easy-to-use web interface to this. Users can view and modify options in a web application, and then those changes are deployed to the desired Zeek systems. Additionally, users benefit from additional features, such as: * Logging and auditing, to determine who made what change and when * Automated expiration of changes (e.g. an exception is added for 2 weeks, after which time it will be removed) * Validation of changes and verification that the change has been applied * Fine-grained role-based access

Grigorescu, Vlad↗
DOE OSTI2024

Generation and validation of comprehensive synthetic weather histories using auto-regressive moving-average models

As energy system design moves to more complex methods of optimization including machine learning there is a significant need for more weather data than is available. One method to solve this is using synthetic data models such as the auto-regressive moving-average (ARMA) model which has been frequently utilized to create such data. This paper looks at extending the ARMA algorithm to generate solar components through the use of clearsky detrending, maintaining vector relationships and by leveraging physical relationships. The method for the creation of entirely synthetic weather data files including key weather variables for energy system analysis is presented. Furthermore, a detailed comparison of energy system simulations utilizing both real and synthetic data is made using NREL’s System Advisor Model. Whilst good agreement is made for the solar variables, and other weather variables, ARMA methods often fail to capture the standard deviation and skew of annual weather distributions. Vector-ARMA is shown to maintain correlations between variables and thus generate data sets that perform similarly in energy system design. Here, it is finally shown that the ARMA method fails to preserve day-today correlations in weather variables and thus over-predicts optimal energy storage by 21% for a residential solar application.

42 ENGINEERING↗
DOE OSTI2024

Pan-Arctic daily streamflow at 0.5-degree resolution

Data was derived using daily runoff from models participating in the Coupled Model Intercomparison Project, version 6 (CMIP6) that have been routed through the Model for Scale Adaptive River Transport (MOSART) routing model to represent streamflow at individual streamflow gage locations. Data from 750 streamflow gage locations are represented, which are indexed by gage id according to the metadata.csv file. Streamflow gage location sites were based on gage records downloaded from the United States Geological Survey (USGS), National Water Data (HYDAT) Archive from Canada, and the State Hydrological Institute of Russia. Data are available from 11 Earth System Models (ESMs) in total at a daily timestep with a duration of 1920-2099 (1850-2014 for E3SM only). The spatial domain of the modeled data is 0.5 degree resolution. Earth System Models and Units of mean daily streamflow are presented in cubic meters per second (cms). Data files consist of 11 Earth System Model mean daily streamflow output per gage location in .csv format, 1 metadata file in .csv format and 1 Readme file in .docx format (13 files in total). These data were used to benchmark CMIP6 modeled representations of streamflow against gage records.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Characterization and identification of HPC applications at leadership computing facility

High Performance Computing (HPC) is an important method for scientific discovery via large-scale simulation, data analysis, or artificial intelligence. Leadership-class supercomputers are expensive, but essential to run large HPC applications. The Petascale era of supercomputers began in 2008, with the first machines achieving performance in excess of one petaflops, and with the advent of new supercomputers in 2021 (e.g., Aurora, Frontier), the Exascale era will soon begin. However, the high theoretical computing capability (i.e., peak FLOPS) of a machine is not the only meaningful target when designing a supercomputer, as the resources demand of applications varies. A deep understanding of the characterization of applications that run on a leadership supercomputer is one of the most important ways for planning its design, development and operation. In order to improve our understanding of HPC applications, user demands and resource usage characteristics, we perform correlative analysis of various logs for different subsystems of a leadership supercomputer. This analysis reveals surprising, sometimes counter-intuitive patterns, which, in some cases, conflicts with existing assumptions, and have important implications for future system designs as well as supercomputer operations. For example, our analysis shows that while the applications spend significant time on MPI, most applications spend very little time on file I/O. Combined analysis of hardware event logs and task failure logs show that the probability of a hardware FATAL event causing task failure is low. Combined analysis of control system logs and file I/O logs reveals that pure POSIX I/O is used more widely than higher level parallel I/O. Based on holistic insights of the application gained through combined and co-analysis of multiple logs from different perspectives and general intuition, we engineer features to "fingerprint" HPC applications. We use t-SNE (a machine learning technique for dimensionality reduction) to validate the explainability of our features and finally train machine learning models to identify HPC applications or group those with similar characteristic. To the best of our knowledge, this is the first work that combines logs on file I/O, computing, and inter-node communication for insightful analysis of HPC applications in production.

Liu, Zhengchun↗
DOE OSTI2020

Jefferson Laboratory C100 Superconducting Radio-Frequency Cavity Fault Data, 2020

The dataset was created to train machine learning models for the task of identifying the (1) cavity and (2) fault type from C100-type cryomodules at the Thomas Jefferson National Accelerator Facility (Jefferson Lab), thereby replacing the time-consuming efforts of a subject matter expert. Superconducting radio-frequency (SRF) cavity trips represent a significant source of accelerator downtime. Real-time – rather than post-mortem – identification of the offending cavity and classification of the fault type would give control room operators valuable feedback for corrective action planning. The anticipated benefit is increased beam-on-target time for users and provides performance metrics that can be used to improve future cavity designs. A series of 17 RF signals are recorded for each of the 8 cavities in a C100 cryomodule every time a cavity trips. These time-series signals are written to file using a specially designed data acquisition system. The dataset represents fault events recorded during Continuous Electron Beam Accelerator Facility (CEBAF) beam operations between January 18, 2019 and March 9, 2020. The following filtering steps were applied to collected data; (1) only 4 of the 17 signals per cavity are retained (GMES, GASK, CRFP, DETA2) (2) only events with data from each of the eight cavities in the cryomodule are kept, (3) only events that were sampled at 5 kHz were kept, (4) events from cryomodule 0L04 were neglected, (5) events occurring between February 4, 3PM and February 5, 12PM were neglected. As a result of preprocessing, the dataset is comprised of 2,375 unique events. The full dataset is comprised of three files: features.csv, cavity_labels.csv, fault_labels.csv. Each instance in faults.csv includes a timestamp (“date_time”), a label for the cryomodule which experienced the trip (“zone_label”), and 192 features (“feature_1”, “feature_2”... “feature_192”). The features correspond to 6 autoregressive features for each of 4 signals per cavity for each of the 8 cavities (6 × 4 signals/cavity × 8 cavities/cryomodule = 192). To deal with the large variation of signal amplitudes, time-series standardization via the z-score (standard score) function was applied prior to computing the features. For each instance, there is an associated label for the (1) cavity which faulted first (cavity_labels.csv) and (2) the type of fault that caused the trip (fault_labels.csv). The cavity identification can take values of [0, 1, 2, 3, 4, 5, 6, 7, 8] and the fault type can take values of [‘Microphonics’, ‘Quench_100ms’, ‘Controls_Fault’, ‘E_Quench’, ‘Quench_3ms’, ‘Single_Cav_Turn_Off’ , ‘Heat_Riser_Choke’, ‘Multi_Cav_Turn_Off’].

43 PARTICLE ACCELERATORS↗
DOE OSTI2024

Documentation for Python Automation in CYME for DER Hosting Capacity Analysis of Different Feeder Configurations

This file provides the documentation for several Python scripts that were developed by Sandia National Laboratories to enhance the automation and customization capabilities for performing various distribution system planning and analysis tasks in CYME. Specifically, these scripts (.py files detailed in Figure 1) enable the user to evaluate different distribution system configurations and the resulting impacts on hosting capacity results and other metrics. In general, these scripts—and the accompanying documentation—provide the foundation upon which future customized tools can be created. For example, the scripts show how to extract and modify parameters of various circuit components, set up and run analyses using built-in CYME tools (iteratively), and export reports for further evaluations and comparisons. Thus, the capabilities and syntaxes used in the scripts can be adapted and leveraged for countless other objectives.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

Cross-calibration of Fuji TR image plate and RAR 2492 x-ray film to determine the response of a DITABIS Super Micron image plate scanner

Fuji TR image plate is frequently used as a replacement detector medium for x-ray imaging and spectroscopy diagnostics at NIF, Omega, and Z facilities. However, the familiar Fuji BAS line of image plate scanners is no longer supported by the industry, and so a replacement scanning system is needed. While the General Electric Typhoon line of scanners could replace the Fuji systems, the shift away from photo stimulated luminescence units to 16-bit grayscale Tag Image File Format (TIFF) leaves a discontinuity when comparing data collected from both systems. For the purposes of quantitative spectroscopy, a known unit of intensity applied to the grayscale values of the TIFF is needed. The DITABIS Super Micron image plate scanning system was tested and shown to potentially rival the resolution and dynamic range of Kodak RAR 2492 x-ray film. However, the absolute sensitivity of the scanner is unknown. In this work, a methodology to cross calibrate Fuji TR image plate and the absolutely calibrated Kodak RAR 2492 x-ray film is presented. Details of the experimental configurations used are included. An energy dependent scale factor to convert Fuji TR IP scanned on a DITABIS Super Micron scanner from 16-bit grayscale TIFF to intensity units (i.e., photons per square micron) is discussed.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2016

A Data Quality Filter for PMU Measurements: Description, Experience, and Examples

Networks of phasor measurement units (PMUs) continue to grow, and along with them, the amount of data available for analysis. With so much data, it is impractical to identify and remove poor quality data manually. The data quality filter described in this paper was developed for use with the Data Integrity and Situation Awareness Tool (DISAT), which analyzes PMU data to identify anomalous system behavior. The filter operates based only on the information included in the data files, without supervisory control and data acquisition (SCADA) data, state estimator values, or system topology information. Measurements are compared to preselected thresholds to determine if they are reliable. Along with the filter's description, examples of data quality issues from application of the filter to nine months of archived PMU data are provided. The paper is intended to aid the reader in recognizing and properly addressing data quality issues in PMU data.

Phasor Measurement Unit↗
DOE OSTI2022

FY22 Progress Report: SRNL Analysis of ICCWR LCM and WAMS Data for Corrosion and Cracking

Algorithms for machine learning and data analysis for the 3013 Surveillance Program are being developed in an ongoing collaborative effort by the Savannah River National Laboratory (SRNL) and the University of South Carolina (USC). To detect the presence of corrosion and cracking, data is collected from large binary files generated by a Laser Confocal Microscope (LCM) or Wide Area 3D Measurement System (WAMS). Software is being developed to use the physical attributes in the data files (e.g., height, color, and grayscale values; all as functions of a location in a plane projection) to detect the presence of surface corrosion and cracking. A user-friendly Matlab Graphical User Interface (GUI) that reads data from either LCM or WAMS files was developed to integrate input data with software developed for processing and evaluation. The GUI can selectively download binary data, interrogate data attributes, label data for training ML algorithms, flag significant features, execute Machine Learning (ML) algorithms, output parameters from trained ML algorithms, report ML model accuracy with respect to labeled data, and generate graphical representations for various analyses. Surface defects can be called out by setting user-specified thresholds, feature based analysis or machine learning algorithms. Enhancements to data labeling capability have been developed to address this essential precursor to application of ML routines. Efficient labeling is particularly important in view of the very large volume of data required to train ML algorithms.

3013 Corrosion↗
DOE OSTI2024

HERO WEC 2024 Hydraulic Configuration Deployment Data

The following submission includes raw and processed data from the in water deployment of NREL's Hydraulic and Electric Reverse Osmosis Wave Energy Converter (HERO WEC), in the form of parquet files, TDMS files, CSV files, bag files and MATLAB workspaces. This dataset was collected in March 2024 at the Jennette's pier test site in North Carolina. This submission includes the following: - Data description document (HERO WEC FY24 Hydraulic Deployment Data Descriptions.doc) - This document includes detailed descriptions of the type of data and how it was processed and/or calculated. - Processed MATLAB workspace - The processed data is provided in the form of a single MATLAB workspace containing data from the full deployment. This workspace contains data from all sensors down sampled to 10 Hz along with all array Value Added Products (VAPs). - MATLAB visualization scripts - The MATLAB workspaces can be visualized using the file "HERO_WEC_2024_Hydraulic_Config_Data_Viewer.m/mlx". The user simply needs to download the processed MATLAB workspaces, specify the desired start and end times and run this file. Both the .m and .mlx file format has been provided depending on the user's preference. - Summary Data - The fully processed data was used to create a summary data set with averages and important calculations performed on 30-minute intervals to align with the intervals of wave resource data reported from nearby CDIP ocean observing buoys located 20km East of Jennette's pier and 40km Northeast of Jennette's pier. The wave resource data provided in this data set is to be used for reference only due the difference in water depth and proximity to shore between the Jennette's pier test site and the locations of the ocean observing buoys. This data is provided in the Summary Data zip folder, which includes this data set in the form of a MATLAB workspace, parquet file, and excel spreadsheet. - Processed Parquet File - The processed data is provided in the form of a single parquet file containing data from all HERO WEC sensors collected during the full deployment. Data in these files has been down sampled to 10 Hz and all array VAPs are included. - Interim Filtered Data - Raw data from each sensor group partitioned into 30-minute parquet files. These files are outputs from an intermediate stage of data processing and contain the raw data with no Quality Control (QC) or calculations performed in a format that is easier to use than the raw data. - Raw Data - Raw, unprocessed data from this deployment can be found in the Raw Data zip folder. This data is provided in the form of TDMS, CSV, and bag files in the original format output by the MODAQ system. - Python Data Processing Script - This links to an NREL public github repository containing the python script used to go from raw data to fully processed parquet files. Additional documentation on how to use this script is included in the github repository. This data set has been developed by the National Renewable Energy Laboratory, operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. Funding provided by the U.S. Department of Energy Office of Energy Efficiency and Renewable Energy Water Power Technologies Office.

16 TIDAL AND WAVE POWER↗
DOE OSTI2019

gcamdata: An R Package for Preparation, Synthesis, and Tracking of Input Data for the GCAM Integrated Human-Earth Systems Model

The increasing data requirements of complex models demand robust, reproducible, and transparent systems to track and prepare models’ inputs. Here we describe version 1.0 of the gcamdata R package that processes raw inputs to produce the hundreds of XML files needed by the GCAM integrated human-earth systems model. It features extensive functional and unit testing, data tracing and visualization, and enforces metadata, documentation, and flexibility in its component data-processing subunits. Although this package is specific to GCAM, many of its structural pieces and approaches should be broadly applicable to, and reusable by, other complex model/data systems aiming to improve transparency, reproducibility, and flexibility.

54 ENVIRONMENTAL SCIENCES↗