Search NASA⌕ Search

SEARCH · Search NASA

Results for “data shifts”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

The Zwicky Transient Facility Bright Transient Survey. III. BTSbot: Automated Identification and Follow-up of Bright Transients with Deep Learning

Abstract The Bright Transient Survey (BTS) aims to obtain a classification spectrum for all bright ( m peak ≤ 18.5 mag) extragalactic transients found in the Zwicky Transient Facility (ZTF) public survey. BTS critically relies on visual inspection (“scanning”) to select targets for spectroscopic follow-up, which, while effective, has required a significant time investment over the past ∼5 yr of ZTF operations. We present BTSbot , a multimodal convolutional neural network, which provides a bright transient score to individual ZTF detections using their image data and 25 extracted features. BTSbot is able to eliminate the need for daily human scanning by automatically identifying and requesting spectroscopic follow-up observations of new bright transient candidates. BTSbot recovers all bright transients in our test split and performs on par with scanners in terms of identification speed (on average, ∼1 hr quicker than scanners). We also find that BTSbot is not significantly impacted by any data shift by comparing performance across a concealed test split and a sample of very recent BTS candidates. BTSbot has been integrated into Fritz and Kowalski , ZTF’s first-party marshal and alert broker, and now sends automatic spectroscopic follow-up requests for the new transients it identifies. Between 2023 December and 2024 May, BTSbot selected 609 sources in real time, 96% of which were real extragalactic transients. With BTSbot and other automation tools, the BTS workflow has produced the first fully automatic end-to-end discovery and classification of a transient, representing a significant reduction in the human time needed to scan.

Rehemtulla, Nabeel (ORCID:0000000256832389)↗

Storing Affordability: Battery Storage as an Asset to Reduce Data Center Cost Shifts

This report examines how battery energy storage systems (BESS) can help utilities accommodate large load growth while protecting affordability for existing ratepayers. Rapid growth in electricity demand from artificial intelligence (AI) data centers is straining the U.S. grid. Furthermore, many new data centers are entering rural markets, which could offer economic benefits but may also pose implementation challenges for smaller utilities. At the same time, retail electricity prices are increasing faster than inflation, elevating customer affordability as a key challenge. While data centers have not been the primary driver of increases in residential prices to date, they have pushed wholesale energy and capacity prices higher in several markets. Fundamental utility cost-allocation principles show that data center growth can be rate-positive for existing customers only if new peak demand grows faster than the costs a utility must incur to serve it. Several factors, including a utility’s degree of wholesale market exposure, forecast uncertainty and stranded-asset risk, and tariff design can determine the outcome of load growth on retail rates. Energy storage can make several affordability contributions in the face of this landscape of uncertainty and market volatility, including deferral of higher-cost grid investments through improved utilization of existing assets and flexibility of new large loads, insulation from volatile wholesale prices through peak shaving, and reliability support to address grid risks stemming from the behavior of AI data center loads. Different potential BESS deployment pathways—utility-scale front-of-the-meter systems, aggregated small-scale storage installations, and data center-sited behind-the-meter storage—are compared against each other and against conventional capacity alternatives. This framework is intended as a conceptual resource to utilities, particularly smaller public utilities with rural service territories, who may be considering the role that energy storage can play in insulating existing ratepayers from data center cost shifts.

25 ENERGY STORAGE↗

Improved Annotation of Untargeted Metabolomics Data through Buffer Modifications That Shift Adduct Mass and Intensity

Annotation of untargeted high-resolution full-scan LC-MS metabolomics data remains challenging due to individual metabolites generating multiple LC-MS peaks arising from isotopes, adducts and fragments. Adduct annotation is a particular challenge, as the same mass difference between peaks can arise from adduct formation, fragmentation, or different biological species. To address this, here we describe a Buffer Modification Workflow (BMW), in which the same sample is run by LC-MS in both liquid chromatography solvent with 14 NH 3 -acetate buffer, and in solvent with the buffer modified with 15 NH 3 -formate. Buffer switching results in characteristic mass and signal intensity changes for adduct peaks, facilitating their annotation. This relatively simple and convenient chromatography modification annotated yeast metabolomics data with similar effectiveness to growing the yeast in isotope-labeled media. Application to mouse liver data annotated both known metabolite and known adduct peaks with 95% accuracy. Altogether, it identified 26% of ~ 27,000 liver LC-MS features as putative metabolites, of which ~ 2600 showed HMDB or KEGG database formula match. This workflow is well-suited to biological samples that cannot be readily isotope labeled, including plants, mammalian tissues, and tumors.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Methodological challenges and new perspectives of shifting vegetation phenology in eddy covariance data

While numerous studies report shifts in vegetation phenology, in this regard eddy covariance (EC) data, despite its continuous high-frequency observations, still requires further exploration. Furthermore, there is no general consensus on optimal methodologies for data smoothing and extracting phenological transition dates (PTDs). Here, we revisit existing methodologies and present new prospects to investigate phenological changes in gross primary productivity (GPP) from EC measurements. First, we present a smoothing technique of GPP time series through the derivative of its smoothed annual cumulative sum. Second, we calculate PTDs and their trends from a commonly used threshold method that identifies days with a fixed percentage of the annual maximum GPP. A systematic analysis is performed for various thresholds ranging from 0.1 to 0.7. Lastly, we examine the relation of PTDs trends to trends in GPP across the years on a weekly basis. Results from 47 EC sites with long time series (> 10 years) show that advancing trends in start of season (SOS) are strongest at lower thresholds but for the end of season (EOS) at higher thresholds. Moreover, the trends are variable at different thresholds for individual vegetation types and individual sites, outlining reasonable concerns on using a single threshold value. Relationship of trends in PTDs and weekly GPP reveal association of advanced SOS and delayed EOS to increase in immediate primary productivity, but not to the trends in overall seasonal productivity. Drawing on these analyses, we emphasise on abstaining from subjective choices and investigating relationship of PTDs trend to finer temporal trends of GPP. Our study examines existing methodological challenges and presents approaches that optimize the use of EC data in identifying vegetation phenological changes and their relation to carbon uptake.

54 ENVIRONMENTAL SCIENCES↗

Survey of Time Shift Detection Algorithms for Measured PV Data

In this research, three variations of time shift detection algorithms were tested for their ability to detect time shift issues (including daylight savings time and random time shifts) in measured PV data sets. Two algorithms from the Python PVAnalytics package were assessed, and one algorithm from the Solar-Data-Tools package was assessed. Each algorithm's ability to accurately detect and measure time shifts was assessed.

automated preprocessing↗

Ripening of Rh Nanoparticle Catalysts in Reverse Water–Gas Shift via a Data-Driven Model Combining Physics, Theory, and Experiment

Degradation via sintering is an ongoing challenge that impedes the broad commercial success of supported metallic nanoparticle catalysts. To mitigate degradation via informed catalyst design and process operations, here we aim to disambiguate the underlying mechanisms of sintering by combining theory and experiment in a quantitative framework. While mechanistic sintering models exist, they only model a single sintering pathway, even though multiple sintering mechanisms can occur simultaneously or dominate at different stages of the process. Data-driven machine learning models have emerged as a means to represent complex processes through data regression. However, machine learning models have very large data needs and lack mechanistic insights due to their black-box encoding. To develop an interpretive model of catalyst degradation via sintering, we constructed a hybrid model combining mechanistic “physics-based” models and data-driven methods to obtain both reliable predictions and mechanistic insights regarding experimentally observed sintering phenomena. Focusing on nanoparticle sintering in the Rh–TiO 2 catalyst for the reverse water–gas shift (RWGS) reaction, the hybrid model couples a mechanistic term for Ostwald ripening with energy values calculated via density functional theory (DFT) with a parametric, data-driven discrepancy function term for unmodeled mechanisms. The hybrid model is trained using Bayesian inference with data collected from small-angle X-ray scattering (SAXS) in situ experiments wherein average nanoparticle diameter versus time was measured at three relevant operating temperatures. The calibrated hybrid model results show that an Ostwald ripening-only model parameterized with fixed DFT energies does not fully capture the time and temperature dependence of the SAXS-observed sintering kinetics, and that an additional functional contribution, or DFT energy calibration, is required to reconcile simulation and experiment. Analysis of the hybrid-model error confirms that the hybrid model outperforms both the purely mechanistic and purely data-driven alternatives in terms of expected predictive accuracy for time-evolving average particle sizes. Furthermore, the results support the hypothesis that the Ostwald ripening mechanism is less important for explaining the sintering phenomena as operating temperature increases under an assumed fixed DFT parameterization. This could be explained in one of two ways: either latent, unmodeled sintering mechanisms dominate at higher temperatures, or the DFT uncertainty increases with temperature. The proposed modeling approach directly links theory to experiments and simulations via a statistical hybrid modeling framework and can be extended to other catalytic systems to improve predictive models and mechanistic understanding.

Bayesian hybrid modeling↗

Machine learning for domain transfer between simulated and experimental 2D X-ray diffraction patterns using generative adversarial networks

X-ray diffraction (XRD) is a well-established technique for analyzing materials at an atomic level. Dynamic compression experiments (DCE), in which materials are subject to extreme pressures, can provide fundamental understanding to pressure-induced phase transitions and compression of the crystal lattice. The analysis of XRD patterns from highly compressed samples is non-trivial given the sparsity of data, high experimental costs, and the fact that the data is often marred with X-ray background and other artifacts. While accurate computational frameworks exist, they solve the forward problem—from structures and orientations to XRD patterns. Solving the inverse problem for 2D experimental diffraction patterns is currently a complex manual process of matching and comparing experimentally observed patterns to computationally generated ones. Machine learning is a promising tool for automating the matching process but often requires data-intensive architectures. Here, in this study, we use a CycleGAN to translate the domain of limited experimental data to a domain in which there is readily available simulated data. This domain shift allows data-intensive machine learning models that have only been trained on simulated XRD patterns to be used in the analysis of experiments.

Brozak, Samantha Jean [Sandia National Laboratorie↗

Dynamic Testing of a Commercial FRAM Device Under Gamma Ray Dose and Neutron Beam

Here, this work presents dose rate and neutron testing on a ferroelectric random-access memory (FRAM) device in dynamic operation during radiation pulses. Radiation failure modes are shown to be unique. Dose rate failure is shown to be based primarily on integrated dose in the radiation pulse and terminates the FRAM operation and can alter up to 8 bytes of data in storage. Neutron failures are attributed to a shifting of data like a clocking error.

Harris, Nathan↗

Data and script associated with “Shifts in Rain-Snow Partitioning Drive Faster Water Transit Times in the US Pacific Northwest”

This data package contains the data and code to use and run the Water Tracer enabled version of the Weather Research and Forecasting Hydrologic model (WT-WRF-Hydro) with the Sequential Precipitation Input Tagging (SPIT) framework. It is associated with the publication “Shifts in Rain-Snow Partitioning Drive Faster Water Transit Times in the US Pacific Northwest” published in Scientific Reports (Butler et al., 2026; https://doi.org/10.1038/s41598-026-46539-1). We use the Continental U.S. (CONUSII; Rasmussen et al., 2021) dataset to force the model with an historical climate (2006–2013) and a future climate (2086–2093) with a representative carbon pathway (RCP) 8.5 scenario. We use the model to calculate water transit times in five headwater catchments within the U.S. Pacific Northwest. We also show key hydrologic and environmental variables that affect water transit times and changes in the future. Finally, we use observed data to validate the model such as stream water isotopes, snowpack characteristics, and stream discharge. For details on how to navigate data packages generated by this project, see https://data.ess-dive.lbl.gov/portals/PNNLRiverCorridorSFA/About. In addition to a readme, this data package also includes a file-level metadata (FLMD) file that describes each file and a data dictionary (DD) that describes all column/row headers and variable definitions. The data package consists of 11 folders: (1) "Figures" contains the exported figures used in the manuscript; (2) "Model_Isotope_Date" contains the WT-WRF-Hydro isotope date used in model validation; (3) “Model_Outputs_Future” contains the WT-WRF-Hydro future climate outputs; (4) “Model_Outputs_Historical” contains the WT-WRF-Hydro historical climate outputs; (5) “Model_Outputs_Weights_Areas” contains the WT-WRF-Hydro weights per catchment used to calculate water transit times and isotopes in stream water; (6) “MODIS_data_scripts” contains data used to validate snow conditions in the study area; (7) “Observed_Flow_Data” contains the observed streamflow data used in model validation; (8) “Observed_Isotope_Data” contains the observed stream water isotope data used in model validation; (9) “Scripts” contains the Python scripts used to general results and the figures; (10) “Statistic_Outputs” contains the water transit time statistical outputs reported in this manuscript; (11) “Validation_SNOTEL” contains the SNOTEL data used in model validation. The files in this data package have the following file extensions: .tif, .txt, .csv, .pdf, .py, .jpg, and .png.

American River↗

Data from Improved Annotation of Untargeted Metabolomics Data through Buffer Modifications That Shift Adduct Mass and Intensity

Annotation of untargeted high-resolution full-scan LC-MS metabolomics data remains challenging due to individual metabolites generating multiple LC-MS peaks arising from isotopes, adducts, and fragments. Adduct annotation is a particular challenge, as the same mass difference between peaks can arise from adduct formation, fragmentation, or different biological species. To address this, here we describe a buffer modification workflow (BMW) in which the same sample is run by LC-MS in both liquid chromatography solvent with 14NH3–acetate buffer and in solvent with the buffer modified with 15NH3–formate. Buffer switching results in characteristic mass and signal intensity changes for adduct peaks, facilitating their annotation. This relatively simple and convenient chromatography modification annotated yeast metabolomics data with similar effectiveness to growing the yeast in isotope-labeled media. Application to mouse liver data annotated both known metabolite and known adduct peaks with 95% accuracy. Overall, it identified 26% of ∼27 000 liver LC-MS features as putative metabolites, of which ∼2600 showed HMDB or KEGG database formula match. This workflow is well suited to biological samples that cannot be readily isotope labeled, including plants, mammalian tissues, and tumors.

Conversion↗

Spectral Properties of Globally Distributed ENA Fluxes across Diverse Regions of the Heliosphere

This study analyzes energetic neutral atom (ENA) spectral properties across distinct regions of globally distributed flux (GDF) sky maps, using Interstellar Boundary Explorer data from a full solar cycle, corrected for time dispersion. By time-shifting the data to the heliosheath using GDF source distances from D. B. Reisenfeld et al., we achieve a more accurate representation of heliosheath GDF energy spectra. We quantify ENA spectral characteristics, heliosheath line-of-sight-integrated proton pressure, and heliosheath proton temperature, comparing these to solar wind properties at 1 au and interplanetary scintillation-derived solar wind data. Our findings show that the spectral index is generally anticorrelated with heliosheath proton temperature and pressure, except in the central tail, where a partial positive correlation is observed. The lowest spectral index values occur when high-latitude heliosheath regions are dominated by fast solar wind from polar coronal holes. The south pole exhibits the flattest energy spectra due to plasma heating from both fast solar wind and a late-2014 pressure pulse. The central tail shows shorter variability (5–6 yr) for spectral index and heliosheath proton temperature, while proton pressure follows the 11 yr solar cycle. Most spectral shapes exhibit a “knee” distribution, peaking during solar maximum, with an “ankle” shape observed only at the south pole during solar cycle transitions. Asymmetry in proton pressure in the lobes is driven by the draping effect of the local interstellar magnetic field. This study provides insights into the energetic properties of GDF across the heliosphere, enhancing our understanding of the heliospheric environment.

79 ASTRONOMY AND ASTROPHYSICS↗

Out-of-Distribution Detection and Radiological Data Monitoring Using Statistical Process Control

Abstract Machine learning (ML) models often fail with data that deviates from their training distribution. This is a significant concern for ML-enabled devices as data drift may lead to unexpected performance. This work introduces a new framework for out of distribution (OOD) detection and data drift monitoring that combines ML and geometric methods with statistical process control (SPC). We investigated different design choices, including methods for extracting feature representations and drift quantification for OOD detection in individual images and as an approach for input data monitoring. We evaluated the framework for both identifying OOD images and demonstrating the ability to detect shifts in data streams over time. We demonstrated a proof-of-concept via the following tasks: 1) differentiating axial vs. non-axial CT images, 2) differentiating CXR vs. other radiographic imaging modalities, and 3) differentiating adult CXR vs. pediatric CXR. For the identification of individual OOD images, our framework achieved high sensitivity in detecting OOD inputs: 0.980 in CT, 0.984 in CXR, and 0.854 in pediatric CXR. Our framework is also adept at monitoring data streams and identifying the time a drift occurred. In our simulations tracking drift over time, it effectively detected a shift from CXR to non-CXR instantly, a transition from axial to non-axial CT within few days, and a drift from adult to pediatric CXRs within a day—all while maintaining a low false positive rate. Through additional experiments, we demonstrate the framework is modality-agnostic and independent from the underlying model structure, making it highly customizable for specific applications and broadly applicable across different imaging modalities and deployed ML models.

Zamzmi, Ghada↗

Phylogenomics and the first higher taxonomy of Placozoa, an ancient and enigmatic animal phylum

Placozoa is an ancient phylum of extraordinarily unusual animals: miniscule, ameboid creatures that lack most fundamental animal features. Despite high genetic diversity, only recently have the second and third species been named. While prior genomic studies suffer from incomplete placozoan taxon sampling, we more than double the count with protein sequences from seven key genomes and produce the first nuclear phylogenomic reconstruction of all major placozoan lineages. This leads us to the first complete Linnaean taxonomic classification of Placozoa, over a century after its discovery: This may be the only time in the 21st century when an entire higher taxonomy for a whole animal phylum is formalized. Our classification establishes 2 new classes, 4 new orders, 3 new families, 1 new genus, and 1 new species, namely classes Polyplacotomia and Uniplacotomia; orders Polyplacotomea, Trichoplacea, Cladhexea, and Hoilungea; families Polyplacotomidae, Cladtertiidae, and Hoilungidae; and genus Cladtertia with species Cladtertia collaboinventa, nov. Our likelihood and gene content tree topologies refine the relationships determined in previous studies. Adding morphological data into our phylogenomic matrices suggests sponges (Porifera) as the sister to other animals, indicating that modest data addition shifts this node away from comb jellies (Ctenophora). Furthermore, by adding the first genomic protein data of the exceptionally distinct and branching Polyplacotoma mediterranea , we solidify its position as sister to all other placozoans; a divergence we estimate to be over 400 million years old. Yet even this deep split sits on a long branch to other animals, suggesting a bottleneck event followed by diversification. Ancestral state reconstructions indicate large shifts in gene content within Placozoa, with Hoilungia hongkongensis and its closest relatives having the most unique genetics.

54 ENVIRONMENTAL SCIENCES↗

The Design, Construction and Testing of the Straw Tracking Detectors for the E989 Muon g-2 Experiment at Fermilab.

The requirement of science, to stretch the boundaries and our understanding of the standard model,is the driving force for the continuous development of our knowledge of the building blocks of theUniverse. Experiments are designed, built and tested to support the theories of the mathematiciansof the physics community and the theorists help interpret the experimental results, looking foravenues of new physics. However, as is known in both science and human nature, nothing is everperfect - there is always room for improvement.The E989 Muon g-2 Collaboration experiment at Fermi National Accelerator Laboratory (otherwiseknown as Fermilab), aims to quantify the muon anomalous magnetic moment to unprecedentedprecision and looks to increase the accuracy of the measurement by fourfold of the predecessorexperiment - E821 at Brookhaven. The discrepancy between the theoretical prediction of theanomalous magnetic moment of the muon and the experimental results, has given tantalisingindication of new physics, prompting an ”upgrade” for systematic accuracy on the E821.The particular focus of this thesis details the design, construction and quality testing of the strawtracking modules for the E989 experiment. The author was one of the dedicated technicians whobuilt and tested the tracker modules at the University of Liverpool High Energy Physics (HEP)department, she built and quality tested the modules, wrote the procedural document for the buildand subsequently was involved in the Run I startup, shut down and data taking shifts (for Run Iand Run II) which contributed to the recent release of the first unblinded data.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Emerging Technologies for Privacy Preservation in Energy Systems

This study explores the intersection of digitalization and privacy within the energy sector, focusing on the emerging challenges and opportunities presented by integrating Distributed Energy Resources (DERs) and advanced metering infrastructure. The need for robust digital privacy measures has become crucial as the energy industry evolves towards a more decentralized, digitalized, and decarbonized future. This study delves into four cutting-edge privacy-preserving technologies—Homomorphic Encryption (HE), Secure Multiparty Computation (SMPC), Differential Privacy (DP), and Federated Learning (FL)—each offering unique solutions to safeguard consumer data by increasing digital connectivity and data exchange. Through a detailed examination of these methods, the study explains how each technology operates, its applications within the energy sector, and the specific privacy challenges it addresses. Homomorphic Encryption allows for secure computations on encrypted data, enabling data analysis without compromising privacy. Secure Multiparty Computation enables collaborative data analysis across different entities while protecting the confidentiality of the inputs. Differential Privacy introduces randomness into the assembled data set, preventing the identification of individual records in statistical databases. Lastly, Federated Learning offers a paradigm shift in data analysis, where machine learning models are trained at the edge, minimizing the centralization of sensitive data. The research underscores the significance of implementing these privacy-enhancing technologies to comply with strict data protection regulations, foster consumer trust, and enhance the security of the energy infrastructure. By providing a comprehensive overview of these methodologies and their practical implications for the energy sector, this study aims to contribute to the ongoing discourse on digital privacy, offering insights into how the energy industry can navigate the complexities of data privacy in the digital age.

Cali, Umit↗

COVID-19 Data Curation Effort: An Initial Analysis of the Data

During the COVID-19 pandemic of 2020, major case reporting outlets quickly coalesced around two or three primary vendors. Johns Hopkins University and The New York Times were among the more prominent, and all were of great value to the nation, particularly during the uncertain early stages of the pandemic. They primarily focused on three major attributes: number of new cases, deaths, and recovery. Recognizing that many states were reporting very detailed data sets (e.g., hospital beds) at a county level or finer, the ORNL Pandemic Modeling team embarked on a major data curation effort from March to June 2020 for the purpose of capturing this wealth of detailed data. The challenge of curating this data was daunting. The number of attributes reported by the states grew on almost on a weekly basis. States were routinely shifting their web tool strategies away from easily parsable HTML-based formatting to new Tableau and ArcGIS content. This growth in the sheer number of attributes, combined with the unpredictable shifts in data format, meant an aggressive and agile combination of automated scripting and manual scraping was required to capture new daily streams. Further, the team had to scale up staff and widen its approach for capture and storage. As a result, the team collected more than 11 million data points. Following the close of this data collection effort on June 30 th , 2020, the team embarked on a major effort to appraise what had been collected, including an inventory list, spatial completeness, temporal completeness, scale and geographic characteristics, and a determination. A report on this matter was submitted on September 15 th , 2020, titled “DOE COVID-19 Data Curation Effort: Overview of Data Collection Coverage”. Over 2000 unique attributes had been netted over a wide range of spatial scales, including state, county, zip codes, health regions, and census blocks. Over 11 million individual data points were collected across these attributes, and spatial coverage (in total) included all 50 states and multiple territories. What became apparent in the process is that in the absence of any data standards, many states reported a wide variety of unique attributes that were not always compatible with attributes reported in other states. As time continued, states began adding new attributes and offering finer grain detail in some older attributes. This meant that not all data streams existed for the entire time period; in fact, the number tended to increase dramatically towards the end. Often, states would begin an attribute series and then stop altogether. These highly variable and uncertain conditions illuminated the need for harmonization approaches that would reconcile and conflate changing attribute names and detail over time. For example, grouping racial data reported as either Black or African American, depending on the state, into a single harmonized attribute. These choices would make a within-state analysis possible during the time period and lead to potential between-state analytics later on. This was almost entirely a manual decision process, requiring some subjective decision-making at times, to prevent a fragmented, short-lived collection of time series fragments that would offer few insights into trends, patterns, and correlates. This report imports harmonized data for state and county into the World Spatio-Temporal Analytics and Mapping Project (WSTAMP). WSTAMP is a major space-time analysis and visualization tool developed at ORNL for the National Geospatial-Intelligence Agency specifically for this kind of exploratory analysis. WSTAMP offers a rich analytical and graphical environment consisting of a wide range of analytics. These include time series plots, statistical summaries, data mining techniques, trend and pattern detection, and hypothesis generation.

59 BASIC BIOLOGICAL SCIENCES↗

DOE COVID-19 Data Curation Effort: Overview of Initial Data Collection Coverage (March - June 2020)

During the COVID-19 pandemic of 2020, major case reporting outlets quickly coalesced around two or three primary vendors. Johns Hopkins University and The New York Times were among the more prominent, and all were of great value to the nation, particularly during the uncertain early stages of the pandemic. They primarily focused on three major attributes: number of new cases, deaths, and recovery, but only at the state level. Recognizing that many states were reporting very detailed data sets (e.g., hospital beds) at a count level or finer, the ORNL Pandemic Modeling team embarked on a major data curation effort from March to June 2020 for the purpose of capturing this wealth of detailed data. The challenge of curating this data was daunting. The number of attributes reported by the states grew on almost on a weekly basis. States were routinely shifting their web tool strategies away from easily parsable HTML-based formatting to new Tableau and ArcGIS content. This growth in the sheer number of attributes combined with the unpredictable shifts in data format meant an aggressive and agile combination of automated scripting and manual scraping was required to capture new daily streams. To keep up, the team had to scale up staff and widen its approach for capture and storage. The DOE COVID-19 data collection effort resulted in over 11 million data points being collected, covering over 13,000 unique geographies and over 2,000 unique attributes that spanned predominantly from early March through the end of June 2020.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗