Search NASA⌕ Search

SEARCH · Search NASA

Results for “Sample curation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21

NASA GeneLab Project: Bridging Space Radiation Omics with Ground Studies

Accurate assessment of risk factors for long-term space missions is critical for human space exploration: therefore it is essential to have a detailed understanding of the biological effects on humans living and working in deep space. Ionizing radiation from Galactic Cosmic Rays (GCR) is one of the major risk factors factor that will impact health of astronauts on extended missions outside the protective effects of the Earth's magnetic field. Currently there are gaps in our knowledge of the health risks associated with chronic low dose, low dose rate ionizing radiation, specifically ions associated with high (H) atomic number (Z) and energy (E). The GeneLab project (genelab.nasa.gov) aims to provide a detailed library of Omics datasets associated with biological samples exposed to HZE. The GeneLab Data System (GLDS) currently includes datasets from both spaceflight and ground-based studies, a majority of which involve exposure to ionizing radiation. In addition to detailed information for ground-based studies, we are in the process of adding detailed, curated dosimetry information for spaceflight missions. GeneLab is the first comprehensive Omics database for space related research from which an investigator can generate hypotheses to direct future experiments utilizing both ground and space biological radiation data. In addition to previously acquired data, the GLDS is continually expanding as Omics related data are generated by the space life sciences community. Here we provide a brief summary of space radiation related data available at GeneLab.

Genelab↗

Post-composing ontology terms for efficient phenotyping in plant breeding

Abstract Ontologies are widely used in databases to standardize data, improving data quality, integration, and ease of comparison. Within ontologies tailored to diverse use cases, post-composing user-defined terms reconciles the demands for standardization on the one hand and flexibility on the other. In many instances of Breedbase, a digital ecosystem for plant breeding designed for genomic selection, the goal is to capture phenotypic data using highly curated and rigorous crop ontologies, while adapting to the specific requirements of plant breeders to record data quickly and efficiently. For example, post-composing enables users to tailor ontology terms to suit specific and granular use cases such as repeated measurements on different plant parts and special sample preparation techniques. To achieve this, we have implemented a post-composing tool based on orthogonal ontologies providing users with the ability to introduce additional levels of phenotyping granularity tailored to unique experimental designs. Post-composed terms are designed to be reused by all breeding programs within a Breedbase instance but are not exported to the crop reference ontologies. Breedbase users can post-compose terms across various categories, such as plant anatomy, treatments, temporal events, and breeding cycles, and, as a result, generate highly specific terms for more accurate phenotyping.

Mathematical & Computational Biology↗

Cometary Glycine Detected in Samples Returned by Stardust

Our previous analysis of cometary samples returned to Earth by NASA's Stardust spacecraft showed several amines and amino acids, but the or igin of these compounds could not be firmly established. Here, we pre sent the stable carbon isotopic ratios of glycine and E-amino-n-caproic acid (EACA), the two most abundant amino acids identified in Stardu st-returned foil samples measured by gas chromatography-mass spectrom etry coupled with isotope ratio mass spectrometry. The Delta C-13 value for glycine of +29 +/- ? 6%: strongly suggests an extraterrestrial origin For glycine, while the Delta C-13 value for EACA of -25 +/-2 % indicates terrestrial contamination by Nylon-6 during curation. This represents the first detection of a cometary amino acid.

Elsila, Jamie E.↗

Maximizing Spaceflight Biological Data with Omics Analytics: The NASA GeneLab Database

NASA’s GeneLab includes an open-access repository of some 250+ omics datasets generated by biological experiments relevant to spaceflight including simulated cosmic radiation and microgravity. In order to maximize the intelligibility of these data, particularly for users with limited bioinformatics background, GeneLab has become a knowledgebase platform converting raw genetic and proteomic signatures found in flight samples into biological and physiological meanings. A large community of more than 100 scientists has rallied behind GeneLab and organized into four Analysis Working Groups (AWGs: Animal, Plant, Microbe, and Multi-Omics). Together, the AWGs have gained scientific recognition worldwide by establishing a consortium in charge of adopting new complex standards for data analysis workflows and omics sample processing in a rapidly evolving field. We will demonstrate the usage of the repository with smart search capability, an online controlled-access toolshed "Galaxy" to process user data with vetted standard workflows, a workspace for data sharing and a data submission portal with ontology control for better metadata curation. The GeneLab visualization portal will also be demonstrated, showing how anyone without formal training in bioinformatics can now browse the space biology omics data to discover new biology and potential solutions to improve life in space.

Sylvain Vincent Costes↗

GeneLab: The NASA System Biology Platform for Space Omics Repository, Analysis and Visualization

NASA’s GeneLab includes an open-access repository of some 250+ omics datasets generated by biological experiments relevant to spaceflight including simulated cosmic radiation and microgravity. In order to maximize the intelligibility of these data, particularly for users with limited bioinformatics background, GeneLab has become a knowledgebase platform converting raw genetic and proteomic signatures found in flight samples into biological and physiological meanings. A large community of more than 100 scientists has rallied behind GeneLab and organized into four Analysis Working Groups (AWGs: Animal, Plant, Microbe, and Multi-Omics). Together, the AWGs have gained scientific recognition worldwide by establishing a consortium in charge of adopting new complex standards for data analysis workflows and omics sample processing in a rapidly evolving field. We will demonstrate the usage of the repository with smart search capability, an online controlled-access toolshed "Galaxy" to process user data with vetted standard workflows, a workspace for data sharing and a data submission portal with ontology control for better metadata curation. The GeneLab visualization portal will also be demonstrated, showing how anyone without formal training in bioinformatics can now browse the space biology omics data to discover new biology and potential solutions to improve life in space.

GeneLab↗

NASA GeneLab Platform Utilized for Biological Response to Space Radiation in Animal Models

Ionizing radiation from Galactic Cosmic Rays (GCR) is one of the major risk factors that will impact the health of astronauts on extended missions outside the protective effects of Earth’s magnetic field. The NASA GeneLab project has detailed information on radiation exposure using animal models with curated dosimetry information for spaceflight experiments. We analyzed multiple GeneLab omics datasets associated with both ground-based and spaceflight radiation studies that included in vivo and in vitro approaches. A range of ions from protons to iron particles with doses from 0.1 Gy to 1.0 Gy for ground studies and samples flown in Low Earth Orbit (LEO) with total doses of 1.0 mGy to 30 mGy were utilized From this analysis we were able to identify distinct biological signatures associating specific ions with specific biological responses due to radiation exposure in space. For example, we discovered changes in mitochondrial function, ribosomal assembly, and immune pathways as a function of dose. We provided a summary of how the GeneLab’s rich database of omics experiments with animal models can be used to generate novel hypotheses to better understand human health risks from GCR exposures.

Afshin Beheshti↗

NASA GeneLab Platform Utilized for Space Radiation Dosimetry Biological Response Compared to Radiation Ground Studies

Ionizing radiation from Galactic Cosmic Rays (GCR) is one of the major risk factors that will impact the health of astronauts on extended missions outside the protective effects of Earth’s magnetic field. The NASA GeneLab project has detailed information on radiation exposure using animal models with curated dosimetry information for spaceflight experiments. We analyzed multiple GeneLab omics datasets associated with both ground-based and spaceflight radiation studies that included in vivo and in vitro approaches. A range of ions from protons to iron particles with doses from 0.1 Gy to 1.0 Gy for ground studies and samples flown in Low Earth Orbit (LEO) with total doses of 1.0 mGy to 30 mGy were utilized From this analysis we were able to identify distinct biological signatures associating specific ions with specific biological responses due to radiation exposure in space. For example, we discovered changes in mitochondrial function, ribosomal assembly, and immune pathways as a function of dose. We provided a summary of how the GeneLab’s rich database of omics experiments with animal models can be used to generate novel hypotheses to better understand human health risks from GCR exposures.

Afshin Beheshti↗

Stardust Interstellar Preliminary Examination II: Curating the Interstellar Dust Collector, Picokeystones, and Sources of Impact Tracks

We discuss the inherent difficulties that arise during "ground truth" characterization of the Stardust interstellar dust collector. The challenge of identifying contemporary interstellar dust impact tracks in aerogel is described within the context of background spacecraft secondaries and possible interplanetary dust particles and beta-meteoroids. In addition, the extraction of microscopic dust embedded in aerogel is technically challenging. Specifically, we provide a detailed description of the sample preparation techniques developed to address the unique goals and restrictions of the Interstellar Preliminary Exam. These sample preparation requirements and the scarcity of candidate interstellar impact tracks exacerbate the difficulties. We also illustrate the role of initial optical imaging with critically important examples, and summarize the overall processing of the collection to date.

Dust↗

Cleaning Genesis Sample Return Canister for Flight: Lessons for Planetary Sample Return

Sample return missions require chemical contamination to be minimized and potential sources of contamination to be documented and preserved for future use. Genesis focused on and successfully accomplished the following: - Early involvement provided input to mission design: a) cleanable materials and cleanable design; b) mission operation parameters to minimize contamination during flight. - Established contamination control authority at a high level and developed knowledge and respect for contamination control across all institutions at the working level. - Provided state-of-the-art spacecraft assembly cleanroom facilities for science canister assembly and function testing. Both particulate and airborne molecular contamination was minimized. - Using ultrapure water, cleaned spacecraft components to a very high level. Stainless steel components were cleaned to carbon monolayer levels (10 (sup 15) carbon atoms per square centimeter). - Established long-term curation facility Lessons learned and areas for improvement, include: - Bare aluminum is not a cleanable surface and should not be used for components requiring extreme levels of cleanliness. The problem is formation of oxides during rigorous cleaning. - Representative coupons of relevant spacecraft components (cut from the same block at the same time with identical surface finish and cleaning history) should be acquired, documented and preserved. Genesis experience suggests that creation of these coupons would be facilitated by specification on the engineering component drawings. - Component handling history is critical for interpretation of analytical results on returned samples. This set of relevant documents is not the same as typical documentation for one-way missions and does include data from several institutions, which need to be unified. Dedicated resources need to be provided for acquiring and archiving appropriate documents in one location with easy access for decades. - Dedicated, knowledgeable contamination control oversight should be provided at sites of fabrication and integration. Numerous excellent Genesis chemists and analytical facilities participated in the contamination oversight; however, additional oversight at fabrication sites would have been helpful.

Allton, J. H.↗

CABO-16S—a Combined Archaea, Bacteria, Organelle 16S rRNA database framework for amplicon analysis of prokaryotes and eukaryotes in environmental samples

Abstract Identification of both prokaryotic and eukaryotic microorganisms in environmental samples is currently challenged by the need for additional sequencing to obtain separate 16S and 18S ribosomal RNA (rRNA) amplicons or the constraints imposed by “universal” primers. Organellar 16S rRNA sequences are amplified and sequenced along with prokaryote 16S rRNA and provide an alternative method to identify eukaryotic microorganisms. CABO-16S combines bacterial and archaeal sequences from the SILVA database with 16S rRNA sequences of plastids and other organelles from the PR2 database to enable identification of all 16S rRNA sequences. Comparison of CABO-16S with SILVA 138.2 results in equivalent taxonomic classification of mock communities and increased classification of diverse environmental samples. In particular, identification of phototrophic eukaryotes in shallow seagrass environments, marine waters, and lake waters was increased. The CABO-16S framework allows users to add custom sequences for further classification of underrepresented clades and can be easily updated with future releases of reference databases. Addition of sequences obtained from Sanger sequencing of methane seep sediments and curated sequences of the polyphyletic SEEP-SRB1 clade resulted in differentiation of syntrophic and non-syntrophic SEEP-SRB1 in hydrothermal vent sediments. CABO-16S highlights the benefit of combining and amending existing training sets when studying microorganisms in diverse environments.

Eitel, Eryn M. (ORCID:0009000723919297)↗

A robust synthetic data generation framework for machine learning in high-resolution transmission electron microscopy (HRTEM)

Machine learning techniques are attractive options for developing highly-accurate analysis tools for nanomaterials characterization, including high-resolution transmission electron microscopy (HRTEM). However, successfully implementing such machine learning tools can be difficult due to the challenges in procuring sufficiently large, high-quality training datasets from experiments. In this work, we introduce Construction Zone, a Python package for rapid generation of complex nanoscale atomic structures which enables fast, systematic sampling of realistic nanomaterial structures and can be used as a random structure generator for large, diverse synthetic datasets. Using Construction Zone, we develop an end-to-end machine learning workflow for training neural network models to analyze experimental atomic resolution HRTEM images on the task of nanoparticle image segmentation purely with simulated databases. Further, we study the data curation process to understand how various aspects of the curated simulated data—including simulation fidelity, the distribution of atomic structures, and the distribution of imaging conditions—affect model performance across three benchmark experimental HRTEM image datasets. Using our workflow, we are able to achieve state-of-the-art segmentation performance on these experimental benchmarks and, further, we discuss robust strategies for consistently achieving high performance with machine learning in experimental settings using purely synthetic data. Construction Zone and its documentation are available at https://github.com/lerandc/construction_zone.

36 MATERIALS SCIENCE↗

Wildfire Segmentation From Remotely Sensed Data Using Quantum-Compatible Conditional Vector Quantized-Variational Autoencoders

Wildfires represent a critical environmental hazard with multifaceted implications for ecosystems, communities, and public health [1]. The escalating frequency and intensity of wildfires globally have intensified the urgency for robust segmentation methodologies to facilitate effective mitigation, response, and recovery strategies [2]. Accurate wildfire segmentation is pivotal for delineating fire boundaries, assessing progression patterns, and prioritizing resource allocation during emergency scenarios. Furthermore, precise segmentation enables stakeholders, including policymakers, environmental scientists, and emergency responders, to formulate evidence-based strategies, thereby minimizing socio-economic disruptions and ecological degradation. Consequently, advancing wildfire segmentation techniques through innovative technological interventions remains a paramount research imperative. Although foundational in wildfire segmentation, traditional deterministic models exhibit inherent limitations that compromise their efficacy in dynamic and uncertain environments. These models often operate on rigid algorithms prioritizing deterministic classifications, thereby overlooking the inherent complexities and uncertainties associated with wildfire behavior and satellite data variability. Such deterministic frameworks tend to produce oversimplified representations that fail to capture the intricate nuances of evolving fire dynamics, spatial heterogeneity, and environmental interactions [1]. Consequently, the deterministic approach’s propensity for uncertainty collapsing [1, 3] hampers the accuracy, reliability, and applicability of segmentation outcomes in real-world scenarios. Contrastingly, stochastic models offer a more nuanced and adaptable framework for wildfire segmentation. By integrating probabilistic elements into the modeling paradigm, stochastic approaches, particularly probabilistic approaches such as variational auto encoders (VAEs) [4], facilitate comprehensive uncertainty assessment, enabling researchers to quantify and incorporate uncertainties into segmentation outcomes effectively. This probabilistic nature empowers stochastic models to encapsulate variability, account for data inconsistencies, and adapt to evolving environmental conditions, enhancing segmentation accuracy, reliability, and robustness. Embracing stochastic methodologies thus catalyzes advancements in wildfire science by fostering a more holistic, adaptive, and resilient segmentation framework. Despite VAEs demonstrating significant promise in various applications, they come with inherent limitations that have garnered attention within the machine learning community. One of the primary drawbacks lies in their reliance on static priors, which essentially assume a fixed distribution for latent variables, thereby limiting the model’s flexibility to capture complex data structures effectively [5]. This static nature leads to suboptimal representations, especially when dealing with complex and high-dimensional data. Additionally, VAEs often struggle with generating sharp and realistic samples, a phenomenon commonly referred to as mode collapse [5, 7, 6]. Furthermore, the optimization process in VAEs, which involves balancing the reconstruction loss and the regularization term, can sometimes be challenging to fine-tune [7]. In recent efforts to address these shortcomings, alternative approaches like Vector Quantized Variational Auto encoders(VQ-VAEs) [7], address the challenges by incorporating discrete latent variables and leveraging techniques that enhance the quality and diversity of generated samples while maintaining efficient training dynamics. VQ-VAEs propose a dynamic prior distribution generation mechanism that diverges from the static priors commonly associated with traditional VAEs. This dynamic approach allows for more adaptive and context-aware latent variable representations, thereby potentially capturing complex data structures more effectively. Unlike autoregressive prior models such as PixelCNN, which, despite their ability to model dependencies across data dimensions, suffer from significant computational inefficiencies and lack flexibility in handling diverse datasets. In our work, we propose to use a generative quantum-compatible approach to help alleviate the shortcomings of autoregressive prior model in VQ-VAEs. Restricted Boltzmann Machines (RBMs) are a viable alternative prior model that can learn prior distributions in a faster and more flexible manner. In this research endeavor, we meticulously curate a state-of-the-art dataset leveraging satellite MODIS data in conjunction with VIIRS fire masks, derived from Fire Radiative Power (FRP), thereby encapsulating diverse wildfire scenarios and environmental contexts. We developed a conditional VQ-VAE architecture with the RBM prior model that is trained in a supervised manner for segmenting wildfire masks. This innovative approach synergistically harnesses deep learning capabilities, enabling the generation of segmentation maps characterized by heightened precision, granularity, and contextual relevance. Furthermore, replacing the autoregressive prior learning method proposed by the original VQ-VAE with a prior density approximation via quantum-compatible RBM facilitates expedited inference processes, augments flexibility in prior sampling, optimizes computational efficiency and establishes a groundbreaking benchmark in wildfire segmentation methodologies.

quantum machine learning↗

Proton NMR spectra of lignin isolated from field grown transgenic poplar

Here we present a curated dataset of a series of 1H nuclear magnetic resonance (NMR) spectra of lignin isolated from transgenic monolignol 4-O-methyltransferase (MOMT4) engineered poplar. The transgenic poplar was collected from a 2-year-old rotation trees within a three-year field trial experiment. Two replicates were collected for each transgenic poplar for the 1H NMR analysis. The poplar samples were Soxhlet-extracted with toluene/ethanol to remove the extractives and the extractives-free poplar was then ball-milled in a Retsch PM100 planetary ball mill using a porcelain jar with ceramic balls at 600 rpm for 2 h. The ball-milled materials were subjected to enzymatic hydrolysis for 48 h followed by centrifugation and washing with deionized water. The solid residue was extracted twice with 96:4 (v/v) 1,4-dioxane/water mixture at room temperature overnight. The extracts were combined, rotary evaporated, and freeze-dried to recover lignin. The dry lignin samples were dissolved in deuterated dimethyl sulfoxide and transferred into a 5 mm NMR tube. 1H NMR experiments were performed in a Bruker Avance III HD 500 MHz NMR spectrometer operating at a frequency of 125.12 MHz for the 13C nucleus using a standard Bruker pulse sequence (zg) on a Prodigy platform cryoprobe. The NMR spectra were acquired with 16 ppm spectra width, 32k data points, 3s pulse delay, and 16 scans. All the data was processed using the Bruker’s TopSpin 3.6 software. Additional meta data is embedded in the raw spectra files.

1H NMR, lignin, poplar, field trial, MOMT4, CBI↗

Recommendations for developing, documenting, and distributing data products derived from NEON data

The National Ecological Observatory Network (NEON) provides over 180 distinct data products from 81 sites (47 terrestrial and 34 freshwater aquatic sites) within the United States and Puerto Rico. These data products include both field and remote sensing data collected using standardized protocols and sampling schema, with centralized quality assurance and quality control (QA/QC) provided by NEON staff. Such breadth of data creates opportunities for the research community to extend basic and applied research while also extending the impact and reach of NEON data through the creation of derived data products—higher level data products derived by the user community from NEON data. Derived data products are curated, documented, reproducibly-generated datasets created by applying various processing steps to one or more lower level data products—including interpolation, extrapolation, integration, statistical analysis, modeling, or transformations. Derived data products directly benefit the research community and increase the impact of NEON data by broadening the size and diversity of the user base, decreasing the time and effort needed for working with NEON data, providing primary research foci through the development via the derivation process, and helping users address multidisciplinary questions. Creating derived data products also promotes personal career advancement to those involved through publications, citations, and future grant proposals. However, the creation of derived data products is a nontrivial task. Here we provide an overview of the process of creating derived data products while outlining the advantages, challenges, and major considerations.

54 ENVIRONMENTAL SCIENCES↗

The Zooplankton International Geospatial (ZIG) dataset: A global repository of spatiotemporal freshwater zooplankton community composition data to support ecological research

Zooplankton play critical roles in aquatic ecosystem function and food webs. Nevertheless, global syntheses of their abundance and community dynamics are challenging due to methodological differences across monitoring programs, taxonomic inconsistencies, and a lack of standardized metadata. To reconcile these challenges, we assembled, curated, validated, and harmonized the Zooplankton International Geospatial (ZIG) dataset, which includes co-located and contemporaneous zooplankton, water chemistry, and limnological data from 307 lakes and reservoirs. ZIG includes waterbodies from each major lake thermal region and range in size from 0.8-2,805,8600 hectares. Temporal coverage for individual waterbodies ranges between 1-60 years of data (median = 4 years) with sampling from once annually to weekly. ZIG is publicly available and can be used to understand freshwater biodiversity change and its drivers at unprecedented scales, and we consider it to be a cornerstone for future investigations of freshwater biology, chemistry, and ecology.

Figary, Stephanie [Cornell University, Ithaca, NY]↗

Preliminary Quantification of Image Color Gradient on Genesis Concentrator Silicon Carbine Target 60001

The Genesis spacecraft concentrator was a device to focus solar wind ions onto a 6-cm diameter target area, thus concentrating the solar wind by 20X [1]. The target area was comprised of 4 quadrants held in place by a gold-coated stainless steel "cross" (Fig. 1). To date, two SiC and one chemical vapor deposited (CVD) quadrants have been imaged at 5X using a Leica DM-6000M in autoscan mode. Complete imaging of SiC sample 60001 required 1036 images. The mosaic of images is shown in Fig. 2 and position of analyzed areas in Fig. 3. This mosaic imaging is part of the curatorial documentation of surface condition and mapping of contamination. Higher magnification (50X) images of selected areas of the target and individual contaminant particles are compiled into reports which may be requested from the Genesis Curator [2].

Allton, J. H.↗

Spaceflight Biospecimen Sharing in Support of Science Discovery and Exploration

For decades, NASA and international partners have flown non-human biological experiments in space to understand the effects of spaceflight and address potential biological hazards. Sending organisms into space is a costly endeavor which makes space-flown biological specimens a valuable resource. To enable maximum scientific return, samples not required by the Principal Investigators are harvested and collected mostly by NASA’s Space Biology Biospecimen Sharing Program. These specimens are collected according to well-established SOPs that maintain quality and integrity. The specimens are then preserved, archived, and made available to the international scientific community through NASA’s Institutional Scientific Collection (ISC) at Ames Research Center (ARC). The ISC-ARC biospecimens and descriptive metadata are findable and accessible for request through the Life Sciences Data Archive (LSDA). The NASA ISC-ARC currently stores over 32,000 specimens from Shuttle, International Space Station, and ground-based investigations (spaceflight analog experiments involving either hindlimb unloading, centrifugation, or partial weight-bearing study designs). Tissues are predominantly from mice and rats, though samples are also available from bacteria and quail. The specimens include tissues from many physiological systems including musculoskeletal, neurosensory, reproductive, respiratory, circulatory, and digestive. Tissues are stored at -80°C, -20°C, +4°C, or ambient and preserved in various fixatives. Descriptive metadata is available for all samples. Historically, these tissues have been used for a wide range of analyses, including histology, genomics, and transcriptomics. Plans are underway to expand the ISC-ARC beyond the mostly-rodent contents, to include a space-relevant microbial culture collection including bacteria, fungi, and yeast. This expansion of the ISC-ARC will now involve identifying and standardizing best practices for microbial curations. To ensure safe long-term storage of microbial isolates, a microbiology laboratory will be dedicated for identification, cell culture, and lyophilization. Awarding of tissue to public science investigators has resulted in 33 publications since 2011, with 48 requests being submitted since 2016. Of note, NASA GeneLab has been awarded ISC-ARC biospecimens in the past few years. GeneLab processes the biospecimens to generate various levels of ‘omics’ data, which are published on GeneLab’s open access online platform for bioinformatics analysis and visualization. This has helped a systems biology community grow around the processed-biospecimens’ datasets, resulting in many new publications and insights. Websites: https://www.nasa.gov/ames/research/space-biosciences/isc-bsp ; https://lsda.jsc.nasa.gov/Biospecimen

Ryan T. Scott↗

Datum: A Scientific Metadata Catalog

The data catalog market is currently flooded with a myriad of different products, but none serve the scientific community well. There are cloud-native tools like Databricks, Snowflake,to on-premise solutions like Collibra and Datahub. The common failing of all these tools however, is their inability to serve the scientific data community directly. Most catalogs are targeted towards financial, health, or user data - not sensor or scientific domain data. They also prioritize integrations that often don’t exist or are just starting to be used in the scientific realm - all while ignoring common scientific tools and file types. Datum is a catalog which targets the scientific data directly, including the tools and networks in which those tools are used. We work with the producers and consumers of the data where they are, targeting cloud and on-premise with a focus on classified networks. Datum is an Erlang/Elixir application. Technical Features Note: The features listed below are still under development and may change, slightly, upon final delivery of the product. File Formats - Datum has the ability to read additional metadata and provides processing pipelines for the following file formats: Plain Text, PDF, LaTeX, HTML, Open Document Format (.odt), XML, CSV/TSV (and other standard delimiters), OpenDocument Database and Spreadsheets, Geo-Referenced TIFF, Common Data Format, HDF/HDF5, LabView TDMS, Excel, DeltaTables, Parquet, Apache Iceberg, Apache Hudi and many others. Metadata Collection - Scanners for the local and networked file systems and cloud storage providers. Network integration with common databases such as MSSQL and MySQL. User Plugin System - Users are able to provide either file processing, metadata extraction, or sampling plugins in the programming language of their choice. Authentication/Authorization -: OIDC integration, SCIM provisioning and EntraID integration out of the box. Full user and group management system with a “least privilege” operating mode. Governance - Customizable data governance platform; dictate and enforce required metadata, enforce data embargos, and enforce user agreements and NDAs before data access. Ability to create health checks on data, rejecting abandoned or poorly curated data and automatically removing it from the search index. Ability for users to submit corrections. Search - Semantic search is a first class citizen. No licenses to expensive, external software required. Integrated use of vectors and vector-based search allows for AI agent integration at all levels of operation. Metadata Model - Display and control data’s lineage and connections to other data and data directories. Data is modeled after a filesystem - an organization instantly recognizable and navigable by most any user. CLI and SDK - Ships with a Command Line Interface (CLI) tool and with a fully-featured Python SDK. This allows for rapid and programmatic use of Datum by every level of user. Minimal Infrastructure - Datum ships as a single executable file and can be run on any operating system and most CPU architectures. Datum has no reliance on external databases, search indexing tools, or other outside services - and it runs equally well on edge computing devices, cloud services, or in a clustered HPC environment.

darrington, john↗