Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data lineage”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Tracking materials science data lineage to manage millions of materials experiments and analyses

Abstract In an era of rapid advancement of algorithms that extract knowledge from data, data and metadata management are increasingly critical to research success. In materials science, there are few examples of experimental databases that contain many different types of information, and compared with other disciplines, the database sizes are relatively small. Underlying these issues are the challenges in managing and linking data across disparate synthesis and characterization experiments, which we address with the development of a lightweight data management framework that is generally applicable for experimental science and beyond. Five years of managing experiments with this system has yielded the Materials Experiment and Analysis Database (MEAD) that contains raw data and metadata from millions of materials synthesis and characterization experiments, as well as the analysis and distillation of that data into property and performance metrics via software in an accompanying open source repository. The unprecedented quantity and diversity of experimental data are searchable by experiment and analysis attributes generated by both researchers and data processing software. The search web interface allows users to visualize their search results and download zipped packages of data with full annotations of their lineage. The enormity of the data provides substantial challenges and opportunities for incorporating data science in the physical sciences, and MEAD’s data and algorithm management framework will foster increased incorporation of automation and autonomous discovery in materials and chemistry research.

Soedarmadji, Edwin↗

express: Extensible, high-level workflows for swifter ab initio materials modeling

In this work, we introduce an open-source Julia project, express, an extensible, lightweight, high-throughput, high-level workflow framework that aims to automate ab initio calculations for the materials science community. express is shipped with well-tested workflow templates, including structure optimization, equation of state (EOS) fitting, phonon spectrum (lattice dynamics) calculation, and thermodynamic property calculation in the framework of the quasi-harmonic approximation (QHA). It is designed to be highly modularized so that its components can be reused across various occasions, and customized workflows can be built on top of that. Users can also track the status of workflows in real-time, and rerun failed jobs thanks to the data lineage feature express provides. Finally, two working examples, i.e., all workflows applied to lime and akimotoite, are also presented in the code and this paper.

36 MATERIALS SCIENCE↗

Issues and Solutions for Bringing Heterogeneous Water Cycle Data Sets Together

The water cycle research community has generated many regional to global scale products using data from individual NASA missions or sensors (e.g., TRMM, AMSR-E); multiple ground- and space-based data sources (e.g., Global Precipitation Climatology Project [GPCP] products); and sophisticated data assimilation systems (e.g., Land Data Assimilation Systems [LDAS]). However, it is often difficult to access, explore, merge, analyze, and inter-compare these data in a coherent manner due to issues of data resolution, format, and structure. These difficulties were substantiated at the recent Collaborative Energy and Water Cycle Information Services (CEWIS) Workshop, where members of the NASA Energy and Water cycle Study (NEWS) community gave presentations, provided feedback, and developed scenarios which illustrated the difficulties and techniques for bringing together heterogeneous datasets. This presentation reports on the findings of the workshop, thus defining the problems and challenges of multi-dataset research. In addition, the CEWIS prototype shown at the workshop will be presented to illustrate new technologies that can mitigate data access roadblocks encountered in multi-dataset research, including: (1) Quick and easy search and access of selected NEWS data sets. (2) Multi-parameter data subsetting, manipulation, analysis, and display tools. (3) Access to input and derived water cycle data (data lineage). It is hoped that this presentation will encourage community discussion and feedback on heterogeneous data analysis scenarios, issues, and remedies.

Acker, James↗

Addressing and Presenting Quality of Satellite Data via Web-Based Services

With the recent attention to climate change and proliferation of remote-sensing data utilization, climate model and various environmental monitoring and protection applications have begun to increasingly rely on satellite measurements. Research application users seek good quality satellite data, with uncertainties and biases provided for each data point. However, different communities address remote sensing quality issues rather inconsistently and differently. We describe our attempt to systematically characterize, capture, and provision quality and uncertainty information as it applies to the NASA MODIS Aerosol Optical Depth data product. In particular, we note the semantic differences in quality/bias/uncertainty at the pixel, granule, product, and record levels. We outline various factors contributing to uncertainty or error budget; errors. Web-based science analysis and processing tools allow users to access, analyze, and generate visualizations of data while alleviating users from having directly managing complex data processing operations. These tools provide value by streamlining the data analysis process, but usually shield users from details of the data processing steps, algorithm assumptions, caveats, etc. Correct interpretation of the final analysis requires user understanding of how data has been generated and processed and what potential biases, anomalies, or errors may have been introduced. By providing services that leverage data lineage provenance and domain-expertise, expert systems can be built to aid the user in understanding data sources, processing, and the suitability for use of products generated by the tools. We describe our experiences developing a semantic, provenance-aware, expert-knowledge advisory system applied to NASA Giovanni web-based Earth science data analysis tool as part of the ESTO AIST-funded Multi-sensor Data Synergy Advisor project.

Leptoukh, Gregory↗

Data Quality Challenges for Analysis Ready Data (ARD)

Data quality plays a critical role in research and applications. The Earth Science Information Partners (ESIP) Information Quality Cluster (IQC) defines four aspects of information quality: Science, Product, Stewardship, and Services. The ESIP IQC has become internationally recognized as an authoritative and responsive resource of information and guidance to data producers and distributors on how to implement data quality standards and best practices for their science data systems, datasets, and data/metadata dissemination services. In recent years, cloud computing environments have provided scale-up capabilities such as data archives and services, enabling interdisciplinary science and applications. More value-added products are expected from data service providers, including Analysis Ready Data (ARD). ARD refers to data that has been preprocessed into a form that allows immediate analysis by the end user, processed to a minimum set of requirements and provides interoperability over time and across multiple datasets. Once a dataset has been developed from its original form to produce ARD, what quality characteristics should the derived dataset or ARD possess? Also, is it safe to assume that the quality of the ARD is consistent with the quality of the source data, or are there special attributes to an ARD that would warrant a secondary, independent quality assessment? What provenance (also called “data lineage”) information needs to be included in ARD? It is important to answer these questions, especially given the ease of use of ARD, and the consequent temptation by users to trust ARD without understanding the limitations or possible variations in quality compared to the source data. In this presentation, we will discuss data quality challenges for ARD products and services and introduce IQC for participation.

data quality↗

Datum: A Scientific Metadata Catalog

The data catalog market is currently flooded with a myriad of different products, but none serve the scientific community well. There are cloud-native tools like Databricks, Snowflake,to on-premise solutions like Collibra and Datahub. The common failing of all these tools however, is their inability to serve the scientific data community directly. Most catalogs are targeted towards financial, health, or user data - not sensor or scientific domain data. They also prioritize integrations that often don’t exist or are just starting to be used in the scientific realm - all while ignoring common scientific tools and file types. Datum is a catalog which targets the scientific data directly, including the tools and networks in which those tools are used. We work with the producers and consumers of the data where they are, targeting cloud and on-premise with a focus on classified networks. Datum is an Erlang/Elixir application. Technical Features Note: The features listed below are still under development and may change, slightly, upon final delivery of the product. File Formats - Datum has the ability to read additional metadata and provides processing pipelines for the following file formats: Plain Text, PDF, LaTeX, HTML, Open Document Format (.odt), XML, CSV/TSV (and other standard delimiters), OpenDocument Database and Spreadsheets, Geo-Referenced TIFF, Common Data Format, HDF/HDF5, LabView TDMS, Excel, DeltaTables, Parquet, Apache Iceberg, Apache Hudi and many others. Metadata Collection - Scanners for the local and networked file systems and cloud storage providers. Network integration with common databases such as MSSQL and MySQL. User Plugin System - Users are able to provide either file processing, metadata extraction, or sampling plugins in the programming language of their choice. Authentication/Authorization -: OIDC integration, SCIM provisioning and EntraID integration out of the box. Full user and group management system with a “least privilege” operating mode. Governance - Customizable data governance platform; dictate and enforce required metadata, enforce data embargos, and enforce user agreements and NDAs before data access. Ability to create health checks on data, rejecting abandoned or poorly curated data and automatically removing it from the search index. Ability for users to submit corrections. Search - Semantic search is a first class citizen. No licenses to expensive, external software required. Integrated use of vectors and vector-based search allows for AI agent integration at all levels of operation. Metadata Model - Display and control data’s lineage and connections to other data and data directories. Data is modeled after a filesystem - an organization instantly recognizable and navigable by most any user. CLI and SDK - Ships with a Command Line Interface (CLI) tool and with a fully-featured Python SDK. This allows for rapid and programmatic use of Datum by every level of user. Minimal Infrastructure - Datum ships as a single executable file and can be run on any operating system and most CPU architectures. Datum has no reliance on external databases, search indexing tools, or other outside services - and it runs equally well on edge computing devices, cloud services, or in a clustered HPC environment.

darrington, john↗

Genome-centric view of carbon processing in thawing permafrost

As global temperatures rise, large amounts of carbon sequestered in permafrost are becoming available for microbial degradation. However, accurate prediction of carbon gas emissions produced from thawing permafrost is limited by our understanding of the resident microbial communities and their associated carbon metabolism. Here, metagenomic sequencing of 214 samples from intact, thawing and thawed sites collected over three years enabled the recovery of 1,529 metagenome-assembled genomes (MAGs), including many from phyla with poor genomic representation. These genomes were shown to broadly reflect the diversity of this complex ecosystem with genus-level representatives recovered for >60% of the community, constituting a two orders of magnitude increase in the number of genomes available for understanding carbon processing in this environment. Metabolic reconstruction, supported by metatranscriptomic and metaproteomic data, revealed key populations involved in organic matter degradation, including bacteria encoding a pathway for xylose degradation only previously identified in fungi. Combined analysis of the microbial communities and geochemical data highlighted lineages correlated with the production of greenhouse gases and suggest novel syntrophic relationships. Our findings link changing biogeochemistry to specific microbial lineages involved in each stage of carbon processing, providing key information for predicting the impact of climate change on these systems.

Woodcroft, Ben J.↗

SU-D-BRD-04: A Logical Organizational Approach to Clinical Information Management

Purpose: To develop a clinical information management system (CIMS) that collects, organizes physician inputs logically and supports analysis functionality. Methods: In a conventional electronic medical record system (EMR), the document manager component stores data in a pool of standalone .docx or .pdf files. The lack of a content-based logical organization makes cross-checking, reference or automatic inheritance of information challenging. We have developed an information-oriented clinical record system that addresses this shortcoming. In CIMS, a parent library predefines a set of available questions along with the data types of their expected answers. The creation of a questionnaire template is achieved by selecting questions from this parent library to form a virtual group. Instances of the same data field in different documents are linked by their question identifier. This design allows for flexible data sharing and inheritance among various forms using a longitudinal lineage of data indexed according to the modification time stamps of the documents. CIMS is designed with a web portal to facilitate querying, data entry and modification, aggregate report generation, and data adjudication. The current implementation addresses diagnostic data, medical history, vital signs, and various quantities in consult note and treatment summaries. Results: CIMS is currently storing treatment summary information of over 1,000 patients who have received treatment at UCLA Radiation Oncology between March 1, 2013 and January 31, 2014. We are in the process of incorporating a DICOM-RT dosimetry parser and patient reporting applications into CIMS, as well as continuing to define document templates to support additional forms. Conclusion: We are able to devise an alternative storage paradigm which results in an improvement in the accuracy and organizational structure of clinical information.

60 APPLIED LIFE SCIENCES↗

CovTransformer: A transformer model for SARS-CoV-2 lineage frequency forecasting

With hundreds of SARS-CoV-2 lineages circulating in the global population, there is an ongoing need for predicting and forecasting lineage frequencies and thus identifying rapidly expanding lineages. Accurate prediction would allow for more focused experimental efforts to understand pathogenicity of future dominating lineages and characterize the extent of their immune escape. Here, we first show that the inherent noise and biases in lineage frequency data make a commonly-used regression-based approach unreliable. To address this weakness, we constructed a machine learning model for SARS-CoV-2 lineage frequency forecasting, called CovTransformer, based on the transformer architecture. We designed our model to navigate challenges such as a limited amount of data with high levels of noise and bias. We first trained and tested the model using data from the UK and the USA, and then tested the generalization ability of the model to many other countries and US states. Remarkably, the trained model makes accurate predictions two months into the future with high levels of accuracy both globally (in 31 countries with high levels of sequencing effort) and at the US-state level. Our model performed substantially better than a widely used forecasting tool, the multinomial regression model implemented in Nextstrain, demonstrating its utility in SARS-CoV-2 monitoring. Assuming a newly emerged lineage is identified and assigned, our test using retrospective data shows that our model is able to identify the dominating lineages 7 weeks in advance on average before they became dominant. Overall, our work demonstrates that transformer models represent a promising approach for SARS-CoV-2 forecasting and pandemic monitoring.

60 APPLIED LIFE SCIENCES↗

Genetic Predictive Factors for Nonsusceptible Phenotypes and Multidrug Resistance in Expanded-Spectrum Cephalosporin-Resistant Uropathogenic Escherichia coli from a Multicenter Cohort: Insights into the Phenotypic and Genetic Basis of Coresistance

Antimicrobial resistance in urinary tract infections (UTIs) is a major public health concern. This study aims to characterize the phenotypic and genetic basis of multidrug resistance (MDR) among expanded-spectrum cephalosporin-resistant (ESCR) uropathogenic Escherichia coli (UPEC) causing UTIs in California patient populations. Between February and October 2019, 577 ESCR UPEC isolates were collected from patients at 6 clinical laboratory sites across California. Lineage and antibiotic resistance genes were determined by analysis of whole-genome sequence data. The lineages ST131, ST1193, ST648, and ST69 were predominant, representing 46%, 5.5%, 4.5%, and 4.5% of the collection, respectively. Overall, 527 (91%) isolates had an expanded-spectrum β-lactamase (ESBL) phenotype, with bla CTX-M-15 , bla CTX-M-27 , bla CTX-M-55 , and bla CTX-M-14 being the most prevalent ESBL genes. In the 50 non-ESBL phenotype isolates, 40 (62%) contained bla CMY-2 , which was the predominant plasmid-mediated AmpC (pAmpC) gene. Narrow-spectrum β-lactamases, bla TEM-1B and bla OXA-1 , were also found in 44.9% and 32.1% of isolates, respectively. Among ESCR UPEC isolates, isolates with an ESBL phenotype had a 1.7-times-greater likelihood of being MDR than non-ESBL phenotype isolates (P < 0.001). The cooccurrence of bla CTX-M-15 , bla OXA-1 , and aac(6')-Ib-cr within ESCR UPEC isolates was strongly correlated. Cooccurrence of bla CTX-M-15 , bla OXA-1 , and aac(6')-Ib-cr was associated with an increased risk of nonsusceptibility to piperacillin-tazobactam, cefepime, fluoroquinolones, and amikacin as well as MDR. Multivariate regression revealed the presence of bla CTX-M-55 , bla TEM-1B , and the ST131 genotype as predictors of MDR.

59 BASIC BIOLOGICAL SCIENCES↗

Carbonate-hosted microbial communities are prolific and pervasive methane oxidizers at geologically diverse marine methane seep sites

At marine methane seeps, vast quantities of methane move through the shallow subseafloor, where it is largely consumed by microbial communities. This process plays an important role in global methane dynamics, but we have yet to identify all of the methane sinks in the deep sea. Here, we conducted a continental-scale survey of seven geologically diverse seafloor seeps and found that carbonate rocks from all sites host methane-oxidizing microbial communities with substantial methanotrophic potential. In laboratory-based mesocosm incubations, chimney-like carbonates from the newly described Point Dume seep off the coast of Southern California exhibited the highest rates of anaerobic methane oxidation measured to date. After a thorough analysis of physicochemical, electrical, and biological factors, we attribute this substantial metabolic activity largely to higher cell density, mineral composition, kinetic parameters including an elevated V max , and the presence of specific microbial lineages. Our data also suggest that other features, such as electrical conductance, rock particle size, and microbial community alpha diversity, may influence a sample’s methanotrophic potential, but these factors did not demonstrate clear patterns with respect to methane oxidation rates. Based on the apparent pervasiveness within seep carbonates of microbial communities capable of performing anaerobic oxidation of methane, as well as the frequent occurrence of carbonates at seeps, we suggest that rock-hosted methanotrophy may be an important contributor to marine methane consumption.

58 GEOSCIENCES↗

Evolutionary significance of seed structure in Alpinioideae (Zingiberaceae): Seed Structure in Alpinioideae

Alpinioideae is the largest of the four subfamilies of Zingiberaceae and is widely distributed throughout the New and Old World tropics. Recent molecular studies have shown that, although Alpinioideae is a strongly supported monophyletic subfamily with two distinct tribes (Alpinieae and Riedelieae), large genera, such as Alpinia and Amomum, are polyphyletic and are in need of revision. Alpinia and Amomum have been shown to form seven and three distinct clades, respectively, but, for many of these clades, traditional vegetative and floral synapomorphies have not been found. A broad survey of seeds in Alpinioideae using light microscopy and synchrotron-based X-ray tomographic microscopy has shown that many clades have distinctive seed structures that serve as distinctive apomorphies. Tribes Riedelieae and Alpinieae can be distinguished on the basis of operculum structure, with the exception of three taxa analysed. The most significant seed characters were found to be various modifications of the micropylar and chalazal ends, the cell shape of the endotesta and exotesta, and the location of an endotestal gap. A chalazal chamber and hilar rim are reported for the first time in Zingiberaceae. In addition to characterizing clades of extant lineages, these data offer insights into the taxonomic placement of many fossil zingiberalean seeds that are critical to understanding the origin and evolution of Alpinioideae and Zingiberales as a whole.(c) 2015 The Linnean Society of London, Botanical Journal of the Linnean Society, 2015, 178, 441-466..

59 BASIC BIOLOGICAL SCIENCES↗

Nonparallel transcriptional divergence during parallel adaptation

Abstract How underlying mechanisms bias evolution toward predictable outcomes remains an area of active debate. In this study, we leveraged phenotypic plasticity and parallel adaptation across independent lineages of Trinidadian guppies ( Poecilia reticulata ) to assess the predictability of gene expression evolution during parallel adaptation. Trinidadian guppies have repeatedly and independently adapted to high‐ and low‐predation environments in the wild. We combined this natural experiment with a laboratory breeding design to attribute transcriptional variation to the genetic influences of population of origin and developmental plasticity in response to rearing with or without predators. We observed substantial gene expression plasticity, as well as the evolution of expression plasticity itself, across populations. Genes exhibiting expression plasticity within populations were more likely to also differ in expression between populations, with the direction of population differences more likely to be opposite those of plasticity. While we found more overlap than expected by chance in genes differentially expressed between high‐ and low‐predation populations from distinct evolutionary lineages, the majority of differentially expressed genes were not shared between lineages. Our data suggest alternative transcriptional configurations associated with shared phenotypes, highlighting a role for transcriptional flexibility in the parallel phenotypic evolution of a species known for rapid adaptation.

Fischer, Eva K.↗

Provenance In Sensor Data Management: A Cohesive, Independent Solution

In today's information-driven workplaces, data is constantly undergoing transformations and being moved around. The typical business-as-usual approach is to use email attachments, shared network locations, databases, and now, the cloud. More often than not, there are multiple versions of the data sitting in different locations and users of this data are confounded by the lack of metadata describing its provenance, or in other words, its lineage. Our project is aimed to solve this issue in the context of sensor data. The Oak Ridge National Laboratory's Building Technologies Research and Integration Center has reconfigurable commercial buildings deployed on the Flexible Research Platforms (FRPs). These FRPs are instrumented with a large number of sensors which measure a number of variables such as HVAC efficiency, relative humidity, and temperature gradients across doors, windows, and walls. Sub-minute resolution data from hundreds of channels is acquired. This sensor data, traditionally, was saved to a shared network location which was accessible to a number of scientists for performing complicated simulation and analysis tasks. The sensor data also participates in elaborate quality assurance exercises as a result of inherent faults. Sometimes, faults are induced to observe building behavior. It became apparent that proper scientific controls required not just managing the data acquisition and delivery, but to also manage the metadata associated with temporal subsets of the sensor data. We built a system named ProvDMS, or Provenance Data Management System for the FRPs, which would both allow researchers to retrieve data of interest as well as trace data lineage. This provides researchers a one-stop shop for comprehensive views of various data transformation allowing researchers to effectively trace their data to its source so that experiments, and derivations of experiments, may be reused and reproduced without much overhead of the repeatability of experiments that use it. Using these traces, researchers can determine exactly what happens to data as it moves through its life cycle.

Hensley, Zachary P↗

Towards an Open (Data) Science Analytics-Hub for Reproducible Multi-Model Climate Analysis at Scale

Open Science is key to future scientific research and promotes a deep transformation in the whole scientific research process encouraging the adoption of transparent and collaborative scientific approaches aimed at knowledge sharing. Open Science is increasingly gaining attention in the current and future research agenda worldwide. To effectively address Open Science goals, besides Open Access to results and data, it is also paramount to provide tools or environments to support the whole research process, in particular the design, execution and sharing of transparent and reproducible experiments, including data provenance (or lineage) tracking. This work introduces the Climate Analytics-Hub, a new component on top of the Earth System Grid Federation (ESGF), which joins big data approaches and parallel computing paradigms to provide an Open Science environment for reproducible multi-model climate change data analytics experiments at scale. An operational implementation has been set up at the SuperComputing Centre of the Euro- Mediterranean Center on Climate Change, with the main goal of becoming a reference Open Science hub in the climate community regarding the multi-model analysis based on the Coupled Model Intercomparison Project (CMIP).

analytics-hub↗

Sensor Data Management, Validation, Correction, and Provenance for Building Technologies

Oak Ridge National Laboratory (ORNL) conducts research on technologies that use a wide range of sensors to develop and characterize building energy performance. The management of high-resolution sensor data, analysis, and tracing lineage of such activities is challenging. Missing or corrupt data due to sensor failure, fouling, drifting, calibration error, or data logger failure is another issue. This paper focuses on sensor data management, validation, correction, and provenance to combat these issues, ensuring complete and accurate sensor datasets for building technologies applications and research. The design and development of two integrated software products are discussed: Sensor Data Validation and Correction (SensorDVC) and the Provenance Data Management System (ProvDMS) platform.

Data management↗

Towards Exascale Scientific Metadata Management

Advances in technology and computing hardware are enabling scientists from all areas of science to produce massive amounts of data using large-scale simulations or observational facilities. In this era of data deluge, effective coordination between the data production and the analysis phases hinges on the availability of metadata that describe the scientific datasets. Existing workflow engines have been capturing a limited form of metadata to provide provenance information about the identity and lineage of the data. However, much of the data produced by simulations, experiments, and analyses still need to be annotated manually in an ad hoc manner by domain scientists. Systematic and transparent acquisition of rich metadata becomes a crucial prerequisite to sustain and accelerate the pace of scientific innovation. Yet, ubiquitous and domain-agnostic metadata management infrastructure that can meet the demands of extreme-scale science is notable by its absence. To address this gap in scientific data management research and practice, we present our vision for an integrated approach that (1) automatically captures and manipulates information-rich metadata while the data is being produced or analyzed and (2) stores metadata within each dataset to permeate metadata-oblivious processes and to query metadata through established and standardized data access interfaces. We motivate the need for the proposed integrated approach using applications from plasma physics, climate modeling and neuroscience, and then discuss research challenges and possible solutions.

97 MATHEMATICS AND COMPUTING↗