Multi-Path Fusion: A Hierarchical Machine Learning Approach for Combining Diverse Data Sets for A Forest Monitoring New Observing System
Explore the source record for details and available documents.
Data fusion, Data science, deep learning↗
SEARCH · Search NASA
Results for “data diversity”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
STARE for scalable unification of diverse data within Earth, Space, and Planetary Science
Explore the source record for details and available documents.
Michael Rilee↗
Data Diversity as a Tool In the Search for d-Pu’s Missing Magnetism
Explore the source record for details and available documents.
Tutchton, Roxanne M'liss↗
BASIN-3D: A brokering framework to integrate diverse environmental data
Diverse observational and simulation datasets are needed to understand and predict complex ecosystem behavior over seasonal to decadal and century time-scales. Integration of these datasets poses a major barrier towards advancing environmental science, particularly due to differences in the structure and formats of data provided by various sources. Here, we describe BASIN-3D (Broker for Assimilation, Synthesis and Integration of eNvironmental Diverse, Distributed Datasets), a data integration framework designed to dynamically retrieve and transform heterogeneous data from different sources into a common format to provide an integrated view. BASIN-3D enables users to adopt a standardized approach for data retrieval and avoid customizations for the data type or source. We demonstrate the value of BASIN-3D with two use cases that require integration of data from regional to watershed spatial scales. The first application uses the BASIN-3D Python library to integrate time-series hydrological and meteorological data to provide standardized inputs to analytical and machine learning codes in order to predict the impacts of hydrological disturbances on large river corridors of the United States. The second application uses the BASIN-3D Django framework to integrate diverse time-series data in a mountainous watershed in East River, Colorado, United States to enable scientific researchers to explore and download data through an interactive web portal. Thus, BASIN-3D can be used to support data integration for both web-based tools, as well as data analytics using Python scripting and extensions like Jupyter notebooks. The framework is expected to be transferable to and useful for many other field and modeling studies.
Varadharajan, C↗
Optimizing Error-Bounded Lossy Compression for Scientific Data With Diverse Constraints
Vast volumes of data are produced by today's scientific simulations and advanced instruments. These data cannot be stored and transferred efficiently because of limited I/O bandwidth, network speed, and storage capacity. Error-bounded lossy compression can be an effective method for addressing these issues: not only can it significantly reduce data size, but it can also control the data distortion based on user-defined error bounds. In practice, many scientific applications have specific requirements or constraints for lossy compression, in order to guarantee that the reconstructed data are valid for post hoc analysis. For example, some datasets contain irrelevant data that should be isolated in particular and users often have intuition regarding value ranges, geospatial regions, and other data subsets that are crucial for subsequent analysis. Existing state-of-the-art error-bounded lossy compressors, however, do not consider these constraints during compression, resulting in inferior compression ratios with respect to user's post hoc analysis, due to the fact that the data itself provides little or no value for post hoc analysis. In this work we address this issue by proposing an optimized framework that can preserve diverse constraints during the error-bounded lossy compression, e.g., cleaning the irrelevant data, efficiently preserving different precision for multiple value intervals, and allowing users to set diverse precision over both regular and irregular regions. We perform our evaluation on a supercomputer with up to 2,100 cores. Experiments with six real-world applications show that our proposed diverse constraints based error-bounded lossy compressor can obtain a higher visual quality or data fidelity on reconstructed data with the same or even higher compression ratios compared with the traditional state-of-the-art compressor SZ. Furthermore, our experiments also demonstrate very good scalability in compression performance compared with the I/O throughput of the parallel file system.
97 MATHEMATICS AND COMPUTING↗
Uniform Data Access Using GXD
This paper gives an overview of GXD, a framework facilitating publication and use of data from diverse data sources. GXD defines an object-oriented data model designed to represent a wide range of things including data, its metadata, resources and query results. GXD also defines a data transport language. a dialect of XML, for representing instances of the data model. This language allows for a wide range of data source implementations by supporting both the direct incorporation of data and the specification of data by various rules. The GXD software library, proto-typed in Java, includes client and server runtimes. The server runtime facilitates the generation of entities containing data encoded in the GXD transport language. The GXD client runtime interprets these entities (potentially from many data sources) to create an illusion of a globally interconnected data space, one that is independent of data source location and implementation.
Vanderbilt, Peter↗
An Innovative Infrastructure with a Universal Geo-Spatiotemporal Data Representation Supporting Cost-Effective Integration of Diverse Earth Science Data
The SpatioTemporal Adaptive Resolution Encoding (STARE) is a unifying scheme encoding geospatial and temporal information for organizing data on scalable computing/storage resources, minimizing expensive data transfers. STARE provides a compact representation that turns set-logic functions into integer operations, e.g. conditional sub-setting, taking into account representative spatiotemporal resolutions of the data in the datasets. STARE geo-spatiotemporally aligns data placements of diverse data on massive parallel resources to maximize performance. Automating important scientific functions (e.g. regridding) and computational functions (e.g. data placement) allows scientists to focus on domain-specific questions instead of expending their efforts and expertise on data processing. With STARE-enabled automation, SciDB (Scientific Database) plus STARE provides a database interface, reducing costly data preparation, increasing the volume and variety of interoperable data, and easing result sharing. Using SciDB plus STARE as part of an integrated analysis infrastructure dramatically eases combining diametrically different datasets.
Rilee, Michael Lee↗
VISAGE - A Visualization and Exploration Framework for Environmental Data
Diverse airborne and ground-based environmental observations are important technologies for disaster assessment and response, as well as for the validation of environmental satellite observations and atmospheric models which can improve forecasts. The VISAGE (Visualization for Integrated Satellite, Airborne and Ground-based data Exploration) project is working to provide three-dimensional visualization and basic analytics capabilities for such datasets in an interactive user interface. The use of cloud-native, server less technologies for analysis optimized data storage will position VISAGE for integration with other technologies into a Data Analytic Center Framework.
Conover, Helen↗
Novel Observing Strategies (NOS) and Earth System Digital Twins (ESDT) for Disaster Resilience
"Earth systems have now been observed continuously for more than 50 years, not only from space but also from airplanes, balloons and in-situ sensors. With the addition of commercial remote sensing providers, many more Internet-of-Things sensors and new NASA and international observatories being planned, these incredible amounts of data will soon be augmented by even larger amounts of diverse data and therefore will become more and more difficult to access, integrate, understand and utilize. At the same time, because of climate change and its impacts, in order to predict, manage, and mitigate the effects of extreme science events and disasters, the information produced by all of this data needs to be optimized, organized and analyzed in such a way that it can be utilized by traditional as well as many new non-traditional users. This calls for the development of observing systems that are agile, coordinated and responsive to events of interest, and for information systems to be dynamic, interactive and fast. With these objectives in mind, the Advanced Information Systems Technology (AIST) Program has been developing technologies that will allow for the development of Novel Observing Strategies (NOS) in a distributed, coordinated fashion (i.e., similar to an “Internet-of-Earth-Things”), as well as for the development of novel information systems or Earth System Digital Twins (ESDT) that will build a digital replica of the past and current states of the Earth systems, will derive forecasts of future states under nominal assumptions and will also offer the capability to investigate many hypothetical evolution scenarios under varying impact assumptions. Both NOS and ESDT will be facilitated by the unprecedented advances of Artificial Intelligence technologies, especially Machine Learning (ML), for extracting relevant information from these large amounts of data, for fusing and assimilating diverse data together and for running complex models faster. Together NOS and ESDT will help build the capabilities needed to anticipate, predict and mitigate the effects of future disasters."
Earth Science Remote Sensing; Information Systems↗
A Feasibility Study on the Integration of Human Performance Data From Diverse Sources Based on the Complexity of a Proceduralized Task
Securing the safety of socio-technical systems including nuclear facilities is the upmost goal to ensure their sustainability because historical records demonstrate that the performance degradation of human operators (e.g., human errors) is one of the crucial contributors to the occurrence of unexpected events resulting in extensive casualties and financial losses. This implies that the collection of human performance data in diverse conditions with which they could be faced during the operation of nuclear facilities. As this collection requires significant resources, it is necessary to resolve how to accomplish it with limited resources. To address this challenge, as suggested in the SHEEP framework, it is indispensable to extract valuable insights after integrating various kinds of human performance data obtained from different sources. However, a practical method to soundly integrate them seems to be still incomplete. Accordingly, the applicability of TACOM (Task Complexity) measure is investigated as a tool to identify useful information based on the integration of human performance data observed from different simulation conditions. As a result, it is expected that the TACOM measure would play an important role in addressing the technical challenge in securing human performance data.
99 GENERAL AND MISCELLANEOUS↗
Finding the forest in the trees: The challenge of combining diverse environmental data
It has become increasingly important to conduct interdisciplinary environmental research assessments, both nationally and internationally. For this reason, the Committee for a Pilot Study on Database Interfaces was charged to review and advise on data interfacing activities. The committee used six case studies (1) to identify and understand the most important problems associated with collecting, integrating, and analyzing environmental data from local to global spatial scales and over a very wide range of temporal scales; and (2) to elaborate the common barriers to interfacing data of disparate sources and types. Consistent with the committee's charge, the primary focus was the interfacing of geophysical and ecological data. The case studies used by the committee were: The Impact Assessment Project for Drought Early Warning in the Sahel, The National Acid Precipitation Assessment Program, The H.J. Andrew Experimental Forest Long-Term Ecological Research Site, The Carbon Dioxide Information Analysis Center, The First International Satellite Land Surface Climatology Project (ISLSCP) Field Experiment, and The California Cooperative Oceanic Fisheries Investigation. The committee derived a number of lessons from the case studies, and these lessons are summarized at the end of each case study and analyzed in the last chapter. Some are generic in nature; others are more specific to a discipline or project. The conclusions and recommendations are based on the committee's analysis of the case studies and additional research. They are organized according to four major areas of barriers or challenges to the effective interfacing of diverse environmental data. In the final section the committee offers a set of broadly applicable principles (Ten Keys to Success) that can be used by scientists and data managers in planning and conducting interfacing activities.
Source record↗
Reviews and syntheses: The promise of big diverse soil data, moving current practices towards future potential
Abstract. In the age of big data, soil data are more available and richer than ever, but – outside of a few large soil survey resources – they remain largely unusable for informing soil management and understanding Earth system processes beyond the original study. Data science has promised a fully reusable research pipeline where data from past studies are used to contextualize new findings and reanalyzed for new insight. Yet synthesis projects encounter challenges at all steps of the data reuse pipeline, including unavailable data, labor-intensive transcription of datasets, incomplete metadata, and a lack of communication between collaborators. Here, using insights from a diversity of soil, data, and climate scientists, we summarize current practices in soil data synthesis across all stages of database creation: availability, input, harmonization, curation, and publication. We then suggest new soil-focused semantic tools to improve existing data pipelines, such as ontologies, vocabulary lists, and community practices. Our goal is to provide the soil data community with an overview of current practices in soil data and where we need to go to fully leverage big data to solve soil problems in the next century.
54 ENVIRONMENTAL SCIENCES↗
Tutorial: Lessons Learned for Behavior Analysts from Data Scientists
Big data is a computing term used to refer to large and complex data sets, typically consisting of terabytes or more of diverse data that is produced rapidly. The analysis of such complex data sets requires advanced analysis techniques with the capacity to identify patterns and abstract meanings from the vast data. The field of data science combines computer science with mathematics/statistics and leverages artificial intelligence, in particular machine learning, to analyze big data. This field holds great promise for behavior analysis, where both clinical and research studies produce large volumes of diverse data at a rapid pace (i.e., big data). This article presents basic lessons for the behavior analytic researchers and clinicians regarding integration of data science into the field of behavior analysis. We provide guidance on how to collect, protect, and process the data, while highlighting the importance of collaborating with data scientists to select a proper machine learning model that aligns with the project goals and develop models with input from human experts. Here, we hope this serves as a guide to support the behavior analysts interested in the field of data science to advance their practice or research, and helps them avoid some common pitfalls.
42 ENGINEERING↗
A Graphical Model for Fusing Diverse Microbiome Data
This paper develops a Bayesian graphical model for fusing disparate types of count data. The motivating application is the study of bacterial communities from diverse high-dimensional features, in this case, transcripts, collected from different treatments. In such datasets, there are no explicit correspondences between the communities and each corresponds to different factors, making data fusion challenging. We introduce a flexible multinomial-Gaussian generative model for jointly modeling such count data. This latent variable model jointly characterizes the observed data through a common multivariate Gaussian latent space that parameterizes the set of multinomial probabilities of the transcriptome counts. The covariance matrix of the latent variables induces a covariance matrix of co-dependencies between all the transcripts, effectively fusing multiple data sources. We present a computationally scalable variational Expectation-Maximization (EM) algorithm for inferring the latent variables and the parameters of the model. Here, the inferred latent variables provide a common dimensionality reduction for visualizing the data and the inferred parameters provide a predictive posterior distribution. In addition to simulation studies that demonstrate the variational EM procedure, we apply our model to a bacterial microbiome dataset.
59 BASIC BIOLOGICAL SCIENCES↗
NASA Open Science Data Repository: Open Science for Life in Space
Space biology and health data are critical for the success of deep space missions and sustainable human presence off-world. At the core of effectively managing biomedical risks is the commitment to open science principles, which ensure that data are findable, accessible, interoperable, reusable, reproducible and maximally open. The 2021 integration of the Ames Life Sciences Data Archive with GeneLab to establish the NASA Open Science Data Repository significantly enhanced access to a wide range of life sciences, biomedical-clinical, and mission telemetry data alongside existing ‘omics data from GeneLab. This paper describes the new database, its architecture, and new data streams supporting diverse data types and enhancing data submission, retrieval, and analysis. Features include the Biological Data Management Environment for improved data submission, a new user interface, controlled data access, an enhanced API, and comprehensive public visualization tools for environmental telemetry, radiation dosimetry data, and ‘omics analyses. By fostering global collaboration through its Analysis Working Groups and training programs, the Open Science Data Repository promotes widespread engagement in space biology, ensuring transparency and inclusivity in research. It supports the global scientific community in advancing our understanding of spaceflight's impact on biological systems, ensuring humans will thrive in future deep space missions.
OSDR↗
A Balanced Model for Exploration of the Terrestrial Planets: Lessons from the Lunar Experience
The Moon is the only extraterrestrial rocky body for which we have a combination of surface-selected samples, high-resolution orbital photography (Lunar Orbiter), manned and robotic surface exploration (Surveyor, Apollo, Luna), and global compositional, mineralogical, and geophysical data (Galileo, Clementine, Lunar Prospector). Beginning in 1998, CAPTEM organized a series of workshops and conference sessions aimed at integrating these diverse data sets. The insights gained by bringing together scientists from the remote-sensing and sample-analysis communities have been singularly rewarding. Not least of these has been the recognition by both groups that having both kinds of data maximizes the scientific return and permits reconciling information from diverse scales and perspectives. The 20-20 hindsight of the Lunar experience thus provides important lessons; learning from mistakes as well as successes, we can derive a sensible scientific program for Mars exploration. In this abstract, we describe examples of key information from (a) in-situ geologic investigation, (b) laboratory analysis of returned samples whose geologic context and location are known, and (c) global remote sensing of mineralogy, composition, and geophysical parameters. We then show the value of integrating these diverse data sets.
Jolliff, B. L.↗
Statistical relationship between metabolic decomposition and chemical uptake predicts bioconcentration factor data for diverse chemical exposures
Background: A challenge of in vitro to in vivo extrapolation (IVIVE) is to predict the physical state of organisms exposed to chemicals in the environment from in vitro exposure assay data. Although toxicokinetic modeling approaches promise to bridge in vitro screening data with i Danio rerio ) based upon particle swarm optimizations (PSO) of the chemical uptake and degradation rates that predict bioconcentration factors (BCF) for a broad range of chemicals. PSO reveals a relationship between chemical uptake and decomposition parameter values that predicts chemical-specific BCF values with moderate statistical agreement to a limited yet diverse chemical dataset, and all without a need to retrain the model to new data. Conclusions: The presented model requires only the octanol-water partitioning ratio to predict BCFs to a fidelity consistent with existing QSAR models. This success begs re-evaluation of the modeling assumptions; specifically, it suggests that chemical uptake into arterial blood may be limited by transport across gill membranes (diffusion) rather than by counter-current flow between gill lamellae (convection). Therefore, more detailed molecular modeling of aquatic respiration may further improve predictive accuracy of the rTK approach.
59 BASIC BIOLOGICAL SCIENCES↗
A digital twin platform for building performance monitoring and optimization: Performance simulation and case studies
Advancements in sensor technology, data analytics, affordable compute, and communication infrastructure have paved the way for Digital Twin technology in optimizing building operations and controls. This study presents the development of an open and interoperable web-based Digital Twin platform for integrating diverse data streams and facilitating effective user interactions. The platform utilizes modern technologies for the web framework and time-series data management, ensuring scalability and responsiveness. The backend supports seamless integration of diverse data sources and emulators, incorporating data from building sensors and meters, external weather Application Programming Interfaces, and advanced EnergyPlus simulation models of the building and its energy systems including the Distributed Energy Resources that are formulated in Functional Mockup Units. A simulation case study was conducted with FlexLab, a test facility on Lawrence Berkeley National Laboratory campus. The case study includes normal operations, Distributed Energy Resource integration, and power outage scenarios, to illustrate the Digital Twin’s ability to provide critical insights into energy performance and thermal resilience. The results demonstrated the platform’s potential as a decision-support tool for optimizing building energy performance and enhancing resilience against extreme weather events. Future work will focus on deploying the Digital Twin platform to a real building for field validation, extending its capabilities to cover more scenarios such as bidirectional Electric Vehicle interactions, and enhancing user engagement.
EnergyPlus↗