Search NASA⌕ Search

SEARCH · Search NASA

Results for “archiving”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Integrating Characteristic Arctic Vegetation in a Land Surface Model Improves Representation of Carbon Dynamics Across a Tundra Landscape: Modeling Archive

This modeling archive is in support of the Next-Generation Ecosystem Experiments in the Arctic (NGEE Arctic) publication "Integrating Characteristic Arctic Vegetation in a Land Surface Model Improves Representation of Carbon Dynamics Across a Tundra Landscape", by Murphy et al. (2025). This archive contains model input files and outputs from landscape-scale simulations conducted using ELM, the land model component of the Department of Energy’s Energy Exascale Earth System Model (E3SM), at the Council NGEE Arctic field site (Council Road mile marker 71) on Alaska’s Seward Peninsula. Input data and model output from two sets of ELM simulations are provided. The first set of simulations were conducted with the two default ELM Arctic plant functional types (PFTs; broadleaf deciduous boreal shrub and a C3 grass) and the second set of simulations were conducted with a set of nine Arctic-specific PFTs including nonvascular mosses and lichens, graminoids, forbs, evergreen dwarf shrubs, three height classes of deciduous shrubs (dwarf, low, and low to tall), and deciduous alder shrubs (Sulman et al., 2021). Parameter names and major parameter changes in the Arctic-specific PFT configuration are described in Sulman et al. (2021) and archived in the Sulman et al. (2021) dataset (see below). Simulations were spatially explicit, covering an approximately 6.4X3.3 km domain at the Council site with a spatial resolution of 100 m for a total of 2,112 simulated grid cells under each ELM PFT configuration. The modeling archive contains meteorological forcing (seven *.nc files and one *.txt file), a domain definition file (one *.nc files), land surface configuration files (two *.nc files), parameter files (two *.nc files), annual ELM output files spanning 1980-2014 (68 *.nc files), and a User’s Guide (*pdf file). Additional information on the provided files is in the “Modeling Archive Contents” section of the User’s Guide. Model outputs are aggregated to the column scale (i.e. PFT-specific outputs are not provided here).

Murphy, Bailey [ORNL] (ORCID:0000000203995221)↗
DOE OSTI2021

Developing Bloom Filters for Web Archives’ Holdings (Final Project Report)

The main goal of the project was to develop a framework for web archives to create Bloom filters based on their holdings of archived web resources. A Bloom filter (BF) is a data structure that, in our scenario, contains hash values of all (or a subset of) URLs, of which an archive has one or more archival copies. Two main use cases fall in scope for this project and are supported by a BF implementation: 1) Sharing of an archive’s holdings (URLs) and 2) Querying the holdings of one or more archives. Since URL strings are hashed before ingested into the BF, the index of an archive is not shared in plain text when the BF is shared with trusted parties. On the other hand, a BF implementation does allow queries for a URL to confirm if an archive indeed has one or more archival copies of that URL. This collaborative project between Los Alamos National Laboratory (LANL) and the Croatian Web Archive (HAW), from the National and University Library in Zagreb (NSK), developed by NSK and University of Zagreb University Computing Center SRCE), aimed at developing software to create BFs, evaluate the scalability of the approach, pilot a search service based on BFs, and design a framework for archives to share their filters with trusted parties. In the remainder of this document, we will report on the work completed with respect to the individual deliverables, outline aspects of future work, and conclude with our recommendations for the use of BFs for the web archiving community.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

ESS-DIVE guidelines for archiving terrestrial model data

This dataset contains supporting documents and images for ESS-DIVE terrestrial model data archiving guidelines.Terrestrial models are broadly defined as numerical models that couple both land dynamics and energy, water, carbon, or nutrient fluxes. We created these guidelines based on input from the U.S. Department of Energy’s Biological and Environmental Research land modeling community. The guidelines are intended to help modelers determine which components of their terrestrial model data associated with publication should be archived. Based on input from the land modeling community, the guidelines recommend archiving both model input and testing data, as well as code, script, and metadata. The guidelines also recommend archiving model data output, depending on the limitations set by data repositories. Lastly, we provide recommendations for bundling data files for publication as well as a discussion about tools that can facilitate model data archiving and reuse.This dataset is an archive of the associated GitHub repository for our model archiving guidelines (https://github.com/ess-dive-community/essdive-model-data-archiving-guidelines). The ‘README.pdf’ file gives a general introduction to the guidelines, and the ‘instructions.pdf’ file provides more detailed steps for following the guidelines. We also provide 2 figures in this data package: 1) a decision tree (model_data_guidelines_decision_tree.png) that can help users determine which components of their model data to archive. and 2) the ‘model_data_guidelines_flmd.png’ file depicts the different files that can be archived in addition to the model data itself. Lastly, we include 3 digitized tables from our associated manuscript and 3 CSV files with anonymized input from DOE scientists about the importance of different aspects of model data archiving from which we developed the guidelines.Dataset updates for v1.1.0: We updated this data package on 2021-11-22 in response to review comments on our related manuscript. In this update we removed one figure so that the model archiving guidelines are conveyed in text rather than an image. We updated the file-level metadata (FLMD) figure to be in accord with the most recent FLMD recommendations. We made minor edits to the README file to update the recommended citation and added two co-authors. We also added 6 new data files (3 are anonymized input from DOE scientists that helped to inform guidelines, and 3 are digitized tables from our manuscript.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Hashes are not suitable to verify fixity of the public archived web

Web archives, such as the Internet Archive, preserve the web and allow access to prior states of web pages. We implicitly trust their versions of archived pages, but as their role moves from preserving curios of the past to facilitating present day adjudication, we are concerned with verifying the fixity of archived web pages, or mementos, to ensure they have always remained unaltered. A widely used technique in digital preservation to verify the fixity of an archived resource is to periodically compute a cryptographic hash value on a resource and then compare it with a previous hash value. If the hash values generated on the same resource are identical, then the fixity of the resource is verified. We tested this process by conducting a study on 16,627 mementos from 17 public web archives. We replayed and downloaded the mementos 39 times using a headless browser over a period of 442 days and generated a hash for each memento after each download, resulting in 39 hashes per memento. The hash is calculated by including not only the content of the base HTML of a memento but also all embedded resources, such as images and style sheets. We expected to always observe the same hash for a memento regardless of the number of downloads. However, our results indicate that 88.45% of mementos produce more than one unique hash value, and about 16% (or one in six) of those mementos always produce different hash values. We identify and quantify the types of changes that cause the same memento to produce different hashes. These results point to the need for defining an archive-aware hashing function, as conventional hashing functions are not suitable for replayed archived web pages.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2016

WE-H-204-02: Part 1: History and Archives Resources at AIP for AAPM and Its Members

“William D. Coolidge, Inventor of the Modern X-ray Tube” David J. Allard, M.S., CHP - Director, PA DEP Bureau of Radiation Protection William David Coolidge 1873–1975 was a research scientist and inventor of the modern X-ray tube. Besides Roentgen, with his 1895 discovery and subsequent studies of X-rays, perhaps no other individual contributed more to the advancement of X-ray technology than did Coolidge. He was born in Hudson, MA and received his Bachelor of Science degree from MIT in 1896. That same year he went to Europe to study under renowned physicists of the time. Coolidge received his Ph.D. summa cum laude from the University of Leipzig in 1899 and soon after joined the staff of MIT. While studying at Leipzig, he met Roentgen. In 1905 he was asked to join the newly established General Electric Research Laboratory in Schenectady, NY. He promptly began fundamental work on the production of ductile tungsten filaments as a replacement for fragile carbon filaments used in incandescent light bulbs. This improved light bulb was brought to market by GE in 1911. It was subsequent application of his tungsten work that led Coolidge to his studies in X ray production. Circa 1910, the state-of-the-art X-ray tube was a “gas tube” or “cold cathode” type tube. These crude X-ray tubes relied on residual gas molecules as a source of electrons for bombardment of low to medium atomic number metal targets. In 1912 Coolidge described the use of tungsten as an improved anode target material for X-ray tubes. Shortly after in 1913 he published a paper in Physical Review describing “A Powerful Roentgen Ray Tube With a Pure Electron Discharge.” This tube used a tungsten filament as a thermionic source of electrons under high vacuum to bombard a tungsten anode target. Great improvements in X-ray tube stability, output and performance were obtained with the “hot cathode” or “Coolidge tube.” With some variation in filament and target geometry, this 100 year old invention is the same basic X-ray tube used today in medicine, research and industry. In 1932 Coolidge became Director of the GE Laboratory, then in 1940 Vice-President and Director of Research. In 1941 he was a member of a small committee, appointed by President Franklin D. Roosevelt, to evaluate the military importance of research on uranium. This committee’s report led to the establishment of the Manhattan Engineering District for nuclear weapons development during WWII. Coolidge lived to be over 100 years old, he had 83 patents to his credit, numerous awards and honorary degrees, and in 1975 was elected to the National Inventor’s Hall of Fame. At the time he was the only inventor to receive this honor in his lifetime. Dr. Coolidge was also the first recipient of the AAPM’s highest science award - named in his honor. From notes of a day-long interview with Coolidge’s son Lawrence in the mid-1990s, previous biographies, publications, books, GE literature, historic photographs, e.g., a wonderful 1874 photo stereoview card with 1 year old baby “Willie Coolidge”, and other artifacts in the author’s collection, this presentation will review Dr. Coolidge’s amazing life, work, accomplishments and awards. “History and Archives Resources at AIP for AAPM and its Members” Gregory A. Good, Ph.D. - Director, AIP Center for History of Physics Melanie J. Mueller, MLIS - Acting Director, AIP Niels Bohr Library & Archives The American Institute of Physics established the Center for History of Physics and the Niels Bohr Library & Archives in the 1960s. Our shared mission is: To preserve and make known the history of the physical sciences. This talk will explore the many ways that AIP’s two history programs support the historical and archival activities of AAPM. Topics will include our ongoing oral history program, web outreach through exhibits and teaching guides, and archiving for AAPM and other Member Societies. We will focus in particular on materials in our collections related to the history of medical physics and to the history of AAPM. We will unveil and demonstrate a new “Archives Portal” that we are designing specifically to be useful to AAPM and its members. Learning Objectives: Study the background of the medical physicist - William David Coolidge Examine the time-line for his success Review the publications conceptualizing his works and progressions Realize what he invented Evaluate the importance of the invention Relate the success to national prominence Uncover how he influenced medical physicists today Find out how he was celebrated by the AAPM View the AIP established Center for History of Physics Consider the significant efforts and vision to preserve the history of medical physics Learn about the Niels Bohr Library & Archives Look back in time at medical physics in the 1960s Unveil and demonstrate a new “Archives Portal” that will be useful to AAPM.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2016

WE-H-204-03: Part 2: History and Archives Resources at AIP for AAPM and Its Members

“William D. Coolidge, Inventor of the Modern X-ray Tube” David J. Allard, M.S., CHP - Director, PA DEP Bureau of Radiation Protection William David Coolidge 1873–1975 was a research scientist and inventor of the modern X-ray tube. Besides Roentgen, with his 1895 discovery and subsequent studies of X-rays, perhaps no other individual contributed more to the advancement of X-ray technology than did Coolidge. He was born in Hudson, MA and received his Bachelor of Science degree from MIT in 1896. That same year he went to Europe to study under renowned physicists of the time. Coolidge received his Ph.D. summa cum laude from the University of Leipzig in 1899 and soon after joined the staff of MIT. While studying at Leipzig, he met Roentgen. In 1905 he was asked to join the newly established General Electric Research Laboratory in Schenectady, NY. He promptly began fundamental work on the production of ductile tungsten filaments as a replacement for fragile carbon filaments used in incandescent light bulbs. This improved light bulb was brought to market by GE in 1911. It was subsequent application of his tungsten work that led Coolidge to his studies in X ray production. Circa 1910, the state-of-the-art X-ray tube was a “gas tube” or “cold cathode” type tube. These crude X-ray tubes relied on residual gas molecules as a source of electrons for bombardment of low to medium atomic number metal targets. In 1912 Coolidge described the use of tungsten as an improved anode target material for X-ray tubes. Shortly after in 1913 he published a paper in Physical Review describing “A Powerful Roentgen Ray Tube With a Pure Electron Discharge.” This tube used a tungsten filament as a thermionic source of electrons under high vacuum to bombard a tungsten anode target. Great improvements in X-ray tube stability, output and performance were obtained with the “hot cathode” or “Coolidge tube.” With some variation in filament and target geometry, this 100 year old invention is the same basic X-ray tube used today in medicine, research and industry. In 1932 Coolidge became Director of the GE Laboratory, then in 1940 Vice-President and Director of Research. In 1941 he was a member of a small committee, appointed by President Franklin D. Roosevelt, to evaluate the military importance of research on uranium. This committee’s report led to the establishment of the Manhattan Engineering District for nuclear weapons development during WWII. Coolidge lived to be over 100 years old, he had 83 patents to his credit, numerous awards and honorary degrees, and in 1975 was elected to the National Inventor’s Hall of Fame. At the time he was the only inventor to receive this honor in his lifetime. Dr. Coolidge was also the first recipient of the AAPM’s highest science award - named in his honor. From notes of a day-long interview with Coolidge’s son Lawrence in the mid-1990s, previous biographies, publications, books, GE literature, historic photographs, e.g., a wonderful 1874 photo stereoview card with 1 year old baby “Willie Coolidge”, and other artifacts in the author’s collection, this presentation will review Dr. Coolidge’s amazing life, work, accomplishments and awards. “History and Archives Resources at AIP for AAPM and its Members” Gregory A. Good, Ph.D. - Director, AIP Center for History of Physics Melanie J. Mueller, MLIS - Acting Director, AIP Niels Bohr Library & Archives The American Institute of Physics established the Center for History of Physics and the Niels Bohr Library & Archives in the 1960s. Our shared mission is: To preserve and make known the history of the physical sciences. This talk will explore the many ways that AIP’s two history programs support the historical and archival activities of AAPM. Topics will include our ongoing oral history program, web outreach through exhibits and teaching guides, and archiving for AAPM and other Member Societies. We will focus in particular on materials in our collections related to the history of medical physics and to the history of AAPM. We will unveil and demonstrate a new “Archives Portal” that we are designing specifically to be useful to AAPM and its members. Learning Objectives: Study the background of the medical physicist - William David Coolidge Examine the time-line for his success Review the publications conceptualizing his works and progressions Realize what he invented Evaluate the importance of the invention Relate the success to national prominence Uncover how he influenced medical physicists today Find out how he was celebrated by the AAPM View the AIP established Center for History of Physics Consider the significant efforts and vision to preserve the history of medical physics Learn about the Niels Bohr Library & Archives Look back in time at medical physics in the 1960s Unveil and demonstrate a new “Archives Portal” that will be useful to AAPM.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2022

The DSA Toolkit Shines Light Into Dark and Stormy Archives

Web archive collections are created with a particular purpose in mind. A curator selects seeds, or original resources, which are then captured by an archiving system and stored as archived web pages, or mementos. The systems that build web archive collections are often configured to revisit the same original resource multiple times. This is incredibly useful for understanding an unfolding news story or the evolution of an organization. Unfortunately, over time, some of these original resources can go off-topic and no longer suit the purpose for which the collection was originally created. They can go off-topic due to web site redesigns, changes in domain ownership, financial issues, hacking, technical problems, or because their content has moved on from the original topic. Even though they are off-topic, the archiving system will still capture them, thus it becomes imperative to anyone performing research on these collections to identify these off-topic mementos. Hence, we present the Off-Topic Memento Toolkit, which allows users to detect off-topic mementos within web archive collections. The mementos identified by this toolkit can then be separately removed from a collection or merely excluded from downstream analysis. The following similarity measures are available: byte count, word count, cosine similarity, Jaccard distance, Sørensen-Dice distance, Simhash using raw text content, Simhash using term frequency, and Latent Semantic Indexing via the gensim library. We document the implementation of each of these similarity measures. We possess a gold standard dataset generated by manual analysis, which contains both off-topic and on-topic mementos. Using this gold standard dataset, we establish a default threshold corresponding to the best F1 score for each measure. We also provide an overview of potential future directions that the toolkit may take.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2019

Profiling the Usage of an Extreme-Scale Archival Storage System

Profiling the archival storage system in scientific computing environments has received much less attention compared to the parallel file system, but is equally important since it stores the final data products safely, for a long duration. In this paper, we analyze eight years worth of data transfer logs for accessing the archival file system (HPSS) in the Oak Ridge Leadership Computing Facility (OLCF), which has been hosting the world's largest supercomputers and file systems. Our analysis encompasses about 135 million data transfer activities to the 80 PB High Performance Storage System (HPSS), between 2010 and 2017. We analyze the logs from several dimensions, including studying the workload characteristics (e.g., access patterns, frequency of accesses and temporal behavior), file system characteristics (e.g., directory depth, file system scaling trends, file types), and scientific user behavior (e.g., domain-specific usage and organization). Based on the analysis, we derive insights into the future evolution of the archive in terms of provisioning, desired features and functionality from the archive software, role and right sizing of the archive tiers, quota management, and the importance of smart and efficient metadata and storage management. We believe our study will prove useful for both operating current archival storage and better provisioning future systems.

Sim, Hyogi↗
DOE OSTI2016

Milestone Report - Level-2 Milestone 5589: Modernization and Expansion of LLNL Archive Disk Cache

This report documents Livermore Computing (LC) activities in support of ASC L2 milestone 5589: Modernization and Expansion of LLNL Archive Disk Cache, due March 31, 2016. The full text of the milestone is included in Attachment 1. The description of the milestone is: Description: Configuration of archival disk cache systems will be modernized to reduce fragmentation, and new, higher capacity disk subsystems will be deployed. This will enhance archival disk cache capability for ASC archive users, enabling files written to the archives to remain resident on disk for many (6–12) months, regardless of file size. The milestone was completed in three phases. On August 26, 2015 subsystems with 6PB of disk cache were deployed for production use in LLNL’s unclassified HPSS environment. Following that, on September 23, 2015 subsystems with 9 PB of disk cache were deployed for production use in LLNL’s classified HPSS environment. On January 31, 2016, the milestone was fully satisfied when the legacy Data Direct Networks (DDN) archive disk cache subsystems were fully retired from production use in both LLNL’s unclassified and classified HPSS environments, and only the newly deployed systems were in use.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Guidelines for Publicly Archiving Terrestrial Model Data to Enhance Usability, Intercomparison, and Synthesis

Scientific communities are increasingly publishing data to evaluate, accredit, and build on published research. However, guidelines for curating data for publication are sparse for model-related research, limiting the usability of archived simulation data. In particular, there are no established guidelines for archiving data related to terrestrial models that simulate land processes and their coupled interactions with climate. Terrestrial modelers have a unique set of challenges when publishing data due to the diversity of scientific domains, research questions, and the types and scales of simulations. Researchers in the U.S. Department of Energy’s (DOE) projects use a variety of multiscale models to advance robust predictions of terrestrial and subsurface ecosystem processes. Here, we synthesize archiving needs for data associated with different DOE models, and provide guidelines for publishing terrestrial model data components following FAIR (Findable, Accessible, Interoperable, Reusable) principles. The guidelines recommend archiving model inputs and testing data used in final simulation runs along with associated codes, workflow scripts, and metadata in public repositories. Researchers should consider archiving model outputs if they are within the storage limits of the repository. We also provide considerations for how to bundle files into different data publications with citable digital object identifiers. Finally, we identify repository features and tools that would enable storage and reuse of model data. Given the diversity of DOE terrestrial models, these guidelines are transferable to other model types and will enable efficient reuse of simulation data for purposes such as model intercomparisons, initialization, benchmarking, synthesis, and comparisons with field observations.

58 GEOSCIENCES↗
DOE OSTI2025

Digitizing and Enhancing Accessibility of the Fusion Safety Archives

This project focuses on the digitization and public accessibility to the Fusion Safety Archives at the Idaho National Laboratory. The first phase involves a thorough review of each document in the physical archives to determine its online availability. For documents that are available online, PDF copies and unique identifiers are collected for database integration. Documents not available online are delivered to Red Inc. for digitization. Additionally, defunct storage devices such as diskettes are sent to INL’s archival department for data retrieval where possible. The second phase of the project involves the creation of a comprehensive database to house the digital copies of the archives. The database will facilitate easy access and management of the digitized documents. Following the database creation, we plan to train a Retrieval-Augmented Generation (RAG) based AI on publicly available documents. The trained AI will be integrated into a front-facing application, allowing the public to easily access information from the Fusion Safety Archives. This project aims to preserve valuable historical data, improve accessibility, and promote transparency in fusion safety research.

70 - PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2016

Identification of mycotoxins by UHPLC–QTOF MS in airborne fungi and fungi isolated from industrial paper and antique documents from the Archive of Bogotá

Mold deterioration of historical documents in archives and libraries is a frequent and complex phenomenon that may have important economic and cultural consequences. In addition, exposure to toxic fungal metabolites might produce health problems. In this work, samples of broths of fungal species isolated from the documentary material and from indoor environmental samples of the Archive of Bogotá have been analyzed to investigate the presence of mycotoxins. High resolution mass spectrometry made possible to search for a large number of mycotoxins, even without reference standards available at the laboratory. For this purpose, a screening strategy based on ultra-high pressure liquid chromatography coupled to quadrupole-time of flight mass spectrometry (UHPLC–QTOF MS) under MS{sup E} mode was applied. A customized home-made database containing elemental composition for around 600 mycotoxins was compiled. The presence of the (de)protonated molecule measured at its accurate mass was evaluated in the samples. When a peak was detected, collision induced dissociation fragments and characteristic isotopic ions were also evaluated and used for tentative identification, based on structure compatibility and comparison with literature data (if existing). Up to 44 mycotoxins were tentatively identified by UHPLC–QTOF MS. 34 of these tentative compounds were confirmed by subsequent analysis using a targeted LC–MS/MS method, supporting the strong potential of QTOF MS for identification/elucidation purposes. The presence of mycotoxins in these samples might help to reinforce safety measures for researchers and staff who work on reception, restoration and conservation of archival material, not only at the Archive of Bogotá but worldwide. - Highlights: • Mold deterioration of historical documents is a frequent and complex phenomenon. • Samples of broths of fungal species isolated from Archive of Bogotá analyzed. • UHPLC–QTOF MS (MS{sup E}) applied for mycotoxins screening, without reference standards. • Customized home-made database for around 600 mycotoxins compiled. • 44 mycotoxins tentatively identified, 34 of which confirmed by LC–MS/MS.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2016

The Alaska Arctic Vegetation Archive (AVA-AK)

The Alaska Arctic Vegetation Archive (AVA-AK, GIVD-ID: NA-US-014) is a free, publically available database archive of vegetation-plot data from the Arctic tundra region of northern Alaska. The archive currently contains 24 datasets with 3,026 non-overlapping plots. Of these, 74% have geolocation data with 25-m or better precision. Species cover data and header data are stored in a Turboveg database. A standardized Pan Arctic Species List provides a consistent nomenclature for vascular plants, bryophytes, and lichens in the archive. A web-based online Alaska Arctic Geoecological Atlas (AGA-AK) allows viewing and downloading the species data in a variety of formats, and provides access to a wide variety of ancillary data. We conducted a preliminary cluster analysis of the first 16 datasets (1,613 plots) to examine how the spectrum of derived clusters is related to the suite of datasets, habitat types, and environmental gradients. Here, we present the contents of the archive, assess its strengths and weaknesses, and provide three supplementary files that include the data dictionary, a list of habitat types, an overview of the datasets, and details of the cluster analysis.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2018

Epi Archive: Automated Synthesis of Global Notifiable Disease Data

ObjectiveLANL has built software that automatically collects global notifiable disease data, synthesizes the data, and makes it available to humans and computers within the Biosurveillance Ecosystem (BSVE) as a novel data stream. These data have many applications including improving the prediction and early warning of disease events.IntroductionMost countries do not report national notifiable disease data in a machine-readable format. Data are often in the form of a file that contains text, tables and graphs summarizing weekly or monthly disease counts. This presents a problem when information is needed for more data intensive approaches to epidemiology, biosurveillance and public health.While most nations likely store incident data in a machine-readable format, governments are often hesitant to share data openly for a variety of reasons that include technical, political, economic, and motivational issues1.A survey conducted by LANL of notifiable disease data reporting in over fifty countries identified only a few websites that report data in a machine-readable format. The majority (>70%) produce reports as PDF files on a regular basis. The bulk of the PDF reports present data in a structured tabular format, while some report in natural language.The structure and format of PDF reports change often; this adds to the complexity of identifying and parsing the desired data. Not all websites publish in English, and it is common to find typos and clerical errors.LANL has developed a tool, Epi Archive, to collect global notifiable disease data automatically and continuously and make it uniform and readily accessible.MethodsWe conducted a survey of the national notifiable disease reporting systems notating how the data are reported and in what formats. We determined the minimal metadata that is required to contextualize incident counts properly, as well as optional metadata that is commonly found.The development of software to regularly ingest notifiable disease data and make it available involves three or four main steps: scraping, detecting, parsing and persisting.Scraping: we examine website design and determine reporting mechanisms for each country/website, as well as what varies across the reporting mechanisms. We then designed and wrote code to automate the downloading of the data for each country. We store all artifacts presented as files (PDF, XLSX, etc.) in their original form, along with appropriate metadata for parsing and data provenance.Detecting: This step is required when parsing structured non-machine-readable data such as tabular data in PDF files. We combined the Nurminen methodology of PDF table detection with in-house heuristics to find the desired data within PDF reports2.Parsing: We determined what to extract from each dataset and parsed these data into uniform data structures, correctly accommodating the variations in metadata (e.g., time interval definitions) and the various human languages.Persisting: We store the data in the Epi Archive database and make it available on the internet and through the BSVE. The data is persisted into a structured and normalized SQL database.ResultsThe Epi Archive tool currently contains national and/or subnational notifiable disease data from twenty nations. When a user accesses the Epi Archive site, they are prompted with four fields: country, subregion, disease of interest, and date duration. Upon form submission, a time series is generated from the users’ specifications. The generated graph can then be downloaded into a CSV file if a user is interested in performing personal analysis. Additionally, the data from Epi Archive can be reached through a REST API (Representational State Transfer Application Programming Interface).ConclusionsLANL, as part of a currently funded DTRA effort, is automatically and continually collecting global notifiable disease data. While 20 nations are in production, more are being brought online in the near future. These data are already being utilized and will have many applications including improving the prediction and early warning of disease events.References[1] van Panhuis WG, Paul P, Emerson C, et al. A systematic review of barriers to data sharing in public health. BMC Public Health. 2014. 14:1144. doi:10.1186/1471-2458-14-1144[2] Nurminen, Anssi. "Algorithmic extraction of data in tables in PDF documents." (2013).

Khalsa, Hari S.↗
DOE OSTI2021

Detecting and Analyzing Power System Disturbances in PMU Data with the Open-Source Archive Walker Tool

As the number of Phasor Measurement Units (PMUs) deployed in power systems increases, so do the archives of synchrophasor measurements stored by utilities. Significant value can be extracted from these archives by using them to support studies such as frequency response analysis, small-signal stability analysis, and power plant model validation. To better enable utilities to perform these studies, the open-source Archive Walker tool was developed. This tool examines PMU data for grid disturbances and periods of interest. Once identified, the data can be further analyzed inside Archive Walker or exported to specialty tools. In this paper, the various disturbance detectors available in Archive Walker are described, and its interactions with other openly-available tools are highlighted.

Follum, James D.↗