Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data science, data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
NASA NTRS1990

The Planetary Data System - A solution to data management for the planetary science community

An overview of the first release of the Planetary Data System (PDS) is presented, and some of the challenges encountered during development of the system are described. The principal goals of the PDS are to distribute planetary science data and information about these data to NASA, to provide scientific knowledge to users of these data, and to provide for permanent storage. The current architecture and capabilities of the PDS (Version 1.0) are examined, and some of the special challenges encountered and lessons learned during the application are highlighted. Finally, implications for future versions of the PDS as well as for other science data systems are discussed.

Dobinson, Elaine R.↗
DOE OSTI2026

Aligning NASA Earth Science Data Stewardship with FAIR Principles: Outcomes, Recommendations, and Future Directions

The FAIR Principles—Findable, Accessible, Interoperable, and Reusable—offer a widely accepted framework for improving the sharing and reuse of digital scientific data by both human and machine users. Following these principles is critical for effective scientific data stewardship, broader scientific collaboration, and compliance with federal and agency data policies. This paper, based on the work of NASA’s Open, Free, and FAIR Working Group (O’FAIR WG) under the Earth Science Data Systems Program, presents an overview of how FAIR is being applied within NASA’s Earth science data landscape. It highlights ongoing progress and challenges, identifies FAIR-enabling resources, and offers recommendations and strategic actions to enhance the FAIRness of NASA-funded open and free Earth science data products. The FAIR-enabling resources identified underscore the vital role of NASA's existing enterprise processes, standards, tools, and infrastructures in supporting FAIR implementation. Our findings show strong performance in making NASA Earth science data more findable and accessible. However, further work is needed—especially in enhancing interoperability, so that different systems and tools can better understand and exchange data. This is especially important for enabling machine-driven discovery and analysis. We emphasize the importance of a balanced strategy that combines a centralized, top-down approach—focused on building enterprise-level capabilities and processes—with a decentralized, bottom-up approach driven by discipline-specific needs and community practices. We advocate for coordinated efforts to enhance (meta)data interoperability to facilitate seamless data and information sharing and exchange of Earth science data both within NASA and across other agencies managing Earth science data.

Data Product↗
NASA NTRS1998

The Planetary Data System - A Case Study in the Development and Management of Meta-Data for a Scientific Digital Library

The Planetary Data System (PDS) is an active science data archive managed by scientists for NASA's planetary science community. With the advent of the World Wide Web the majority of the archive has been placed on-line as a science digital libraty for access by scientists, the educational community, and the general public.

Data System Meta-data scientific digital library↗
NASA NTRSDate not supplied

Enabling Exchange and Adequate Use of Data for Observation Based Atmospheric Research

Systematic long-term field observations have played a vital role in advancing atmospheric research over the past several decades. The use of these observations has expanded from primarily characterizing atmospheric processes and trends to evaluating satellite measurements, assessing models, and improving air quality forecasts. Consequently, the demand for atmospheric chemistry observational data have dramatically increased in terms of scope and coverage of measurements (i.e., parameters/species, spatiotemporal extent). In addition to high quality measurements, certain data reporting standards need to be agreed to ensure the data can be readily exchanged and are sufficiently documented to enable adequate use in different research activities. To this end, WMO has developed and implemented measurement guidelines and community practices for meteorology, climatology, atmospheric and hydrological sciences. In addition, the WMO Expert Team on Metadata Standards manages and evolves the existing metadata standards for the WMO Information System WIS and WMO Integrated Global Observing System WIGOS to support consistent and interoperable data descriptions, ensure relevance to research, and to apply data science principles. This team draws on a wide range of expertise from the research community, including atmospheric measurements, modeling, data management, and data science. The current activities include development of key performance indicators, vocabularies for metadata and the evolution of metadata standards to lower the barrier of application to weather/climate/water/environment data for all communities and the weather enterprise. This presentation intends to promote awareness of ongoing progress and actively solicit community feedback.

Field Observations↗
NASA NTRS2018

The Importance of User Feedback in Sustaining Trusted Repositories

The NASA Earth Observation System Data and Information System (EOSDIS) has been operating since 1994 and is serving a global user community with well-managed Earth science data in a variety of scientific disciplines. EOSDIS processes, archives and distributes data and information products resulting from spaceborne and airborne instruments as well as in situ measurements from field campaigns. The Earth Science Data and Information System (ESDIS) Project at the NASA Goddard Space Flight Center manages EOSDIS with its 12 Distributed Active Archive Centers (DAACs) located across the United States. During the entire life of EOSDIS, the ESDIS Project and the DAACs have deployed many different mechanisms for user feedback, which have proven extremely valuable to their evolution and performance in their service to user communities. Some of the inputs from our user groups have resulted in fundamental changes in the architecture, design and operations of EOSDIS, while others have provided novel ideas for incremental changes. The EOSDIS DAACs have User Working Groups (UWGs) that represent broad user communities in the Earth science disciplines served by the DAACs. The UWGs meet periodically to assess and provide feedback on dataset and service priorities. As regular users of the data and services of the DAAC and experts in the scientific disciplines, the UWG members provide valuable inputs for planning and prioritizing the services, as well as addition of new datasets, for the benefit of the community. The EOSDIS is evaluated annually through an independently administered survey of its users resulting in the American Customer Satisfaction Index (ACSI). The survey provides an ACSI score as well as free-text suggestions from users, which are also helpful in making specific system improvements. In addition, each of the DAACs has a user services group that address on-going requests for help and other comments from users. The ESDIS Project has established a mechanism through the "earthdata" website (http://earthdata.nasa.gov) for users to provide feedback of any kind and these questions/comments are routed to the appropriate individuals in the Project or the DAACs. These various forms of receiving user feedback and responding to them continue to be extremely valuable in evolving and sustaining our Earth Science repository.

Behnke, Jeanne↗
DOE OSTI2020

DMTN-107: Options for Alert Production in LSST Operations Year 1

This document reviews five options for alert production in LSST Operations Year 1 (LOY1), taking into account any implications on LSST formal requirements including up-scopes, down-scopes or explicit violations. The Data Management System Science Team's preferred option for maximizing LSST science is to generate template images from as much of the data from the commissioning and/or science verification phases as possible and use them to run Difference Image Analysis and alert production during LOY1. A proposal to increase the sky area covered during commissioning via a "filler" scheduler program is also presented. As a potential moderate up-scope, this study presents an option to build interim templates on a $\sim$monthly basis during LOY1, which could increase the accessible sky area by ~1000-2000 deg**2 per month, and should be reconsidered closer to the start of Operations.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRSDate not supplied

ImageLabler: Labeling and Managing Image Data for Machine Learning in the Earth Sciences

While machine learning techniques for image classification have been around for a long time, storing and managing the vast number of images required as training data is still a problem for scientists. This is especially true for the field of Earth science, where only recently have experts begun using machine learning techniques for image-based phenomena classification. Image Labeler, a fast and scalable cloud-based tagging platform for Earth science images, seeks to improve upon existing methods of managing images and associated metadata, such as maintaining categorized folders of images on a local machine, a process that can be cumbersome and difficult to scale. The platform facilitates rapid development of image-based Earth science phenomena training datasets by allowing scientists to upload their existing imagery as well as extract new samples from open satellite imagery services made available through NASA’s Global Imagery Browse Service (GIBS). Image Labeler also supports GeoTIFF data, with capabilities such as displaying GeoTIFFs on an interactive map, drawing shapefiles over them, and tagging them with additional metadata. This allows scientists to perform spatiotemporal subsetting with geographic information and develop training data more quickly. Built using modern web technologies, Image Labeler includes additional capabilities such as team collaboration for large-scale image tagging projects. Users can download their data in a machine-learning-ready format, allowing scientists to spend time on experimentation rather than on the collection of training data. In this presentation, we demonstrate how Image Labeler seeks to become a one-stop image data management solution for machine learning applications in Earth science.

Ashish Acharya↗
NASA NTRS1986

Issues and recommendations associated with distributed computation and data management systems for the space sciences

The primary purpose of the report is to explore management approaches and technology developments for computation and data management systems designed to meet future needs in the space sciences.The report builds on work presented in previous reports on solar-terrestrial and planetary reports, broadening the outlook to all of the space sciences, and considering policy issues aspects related to coordiantion between data centers, missions, and ongoing research activities, because it is perceived that the rapid growth of data and the wide geographic distribution of relevant facilities will present especially troublesome problems for data archiving, distribution, and analysis.

Source record↗
NASA NTRS2016

Benchmark Comparison of Cloud Analytics Methods Applied to Earth Observations

Cloud computing has the potential to bring high performance computing capabilities to the average science researcher. However, in order to take full advantage of cloud capabilities, the science data used in the analysis must often be reorganized. This typically involves sharding the data across multiple nodes to enable relatively fine-grained parallelism. This can be either via cloud-based file systems or cloud-enabled databases such as Cassandra, Rasdaman or SciDB. Since storing an extra copy of data leads to increased cost and data management complexity, NASA is interested in determining the benefits and costs of various cloud analytics methods for real Earth Observation cases. Accordingly, NASA's Earth Science Technology Office and Earth Science Data and Information Systems project have teamed with cloud analytics practitioners to run a benchmark comparison on cloud analytics methods using the same input data and analysis algorithms. We have particularly looked at analysis algorithms that work over long time series, because these are particularly intractable for many Earth Observation datasets which typically store data with one or just a few time steps per file. This post will present side-by-side cost and performance results for several common Earth observation analysis operations.

science data management↗
NASA NTRS2020

NASA ESDS Citizen Science Data Working Group

This document provides guidelines for legal, policy, and ethical issues; standards for citizen science data collection and management; information on ensuring usability of citizen science data and communication regarding its use; and best practices for long-term archival of citizen science data.Section1 contains a detailed discussion of policy, ethical, and legal considerations influencing citizen science data collection. Section 2 considers standards for documentation, including documentation of instrumentation, procedures, and the data itself. It concludes with a discussion of how citizen science data should be attributed. Section 3 provides guidance about how to ensure citizen science data are collected and stored in a useable way. It also considers how NASA and data producers should notify the scientific community, including citizen scientists and the public, about citizen science datasets and the scientific conclusions reached using them. Finally, Section 4 provides detailed information regarding what should be archived from projects using a citizen science approach, including data and code. It provides guidance about archive location, process, and timeframe, as well as information about data access and distribution services provided by NASA that may be relevant to data producers working with citizen scientists.

Citizen Science↗
DOE OSTI2021

The future low-temperature geochemical data-scape as envisioned by the U.S. geochemical community

Data sharing benefits the researcher, the scientific community, and the public by allowing the impact of data to be generalized beyond one project and by making science more transparent. However, many scientific communities have not developed protocols or standards for publishing, citing, and versioning datasets. One community that lags in data management is that of low-temperature geochemistry (LTG). This paper resulted from an initiative from 2018 through 2020 to convene LTG and data scientists in the U.S. to strategize future management of LTG data. Through webinars, a workshop, a preprint, a townhall, and a community survey, the group of U.S. scientists discussed the landscape of data management for LTG – the data-scape. Currently this data-scape includes a “street bazaar” of data repositories. This was deemed appropriate in the same way that LTG scientists publish articles in many journals. The variety of data repositories and journals reflect that LTG scientists target many different scientific questions, produce data with extremely different structures and volumes, and utilize copious and complex metadata. Nonetheless, the group agreed that publication of LTG science must be accompanied by sharing of data in publicly accessible repositories, and, for sample-based data, registration of samples with globally unique persistent identifiers. LTG scientists should use certified data repositories that are either highly structured databases designed for specialized types of data, or unstructured generalized data systems. Recognizing the need for tools to enable search and cross-referencing across the proliferating data repositories, the group proposed that the overall data informatics paradigm in LTG should shift from “build data repository, data will come” to “publish data online, cybertools will find”. Funding agencies could also provide portals for LTG scientists to register funded projects and datasets, and forge approaches that cross national boundaries. Finally, the needed transformation of the LTG data culture requires emphasis in student education on science and management of data.

58 GEOSCIENCES↗
NASA NTRS1993

Data management for interdisciplinary field experiments: OTTER project support

The ability of investigators of an interdisciplinary science project to properly manage the data that are collected during the experiment is critical to the effective conduct of science. When the project becomes large, possibly including several scenes of large-format remotely sensed imagery shared by many investigators requiring several services, the data management effort can involve extensive staff and computerized data inventories. The OTTER (Oregon Transect Ecosystem Research) project was supported by the PLDS (Pilot Land Data System) with several data management services, such as data inventory, certification, and publication. After a brief description of these services, experiences in providing them are compared with earlier data management efforts and some conclusions regarding data management in support of interdisciplinary science are discussed. In addition to providing these services, a major goal of this data management capability was to adopt characteristics of a pro-active attitude, such as flexibility and responsiveness, believed to be crucial for the effective conduct of active, interdisciplinary science. These are also itemized and compared with previous data management support activities. Identifying and improving these services and characteristics can lead to the design and implementation of optimal data management support capabilities, which can result in higher quality science and data products from future interdisciplinary field experiments.

Angelici, Gary↗
NASA NTRS1975

General purpose simulation system of the data management system for space shuttle mission 18

The simulation program of the science and engineering data management system for the space shuttle is presented. The programming language used was General Purpose Simulation System V (OS). The data flow was modeled from its origin at the experiments or subsystems to transmission from the space shuttle. Mission 18 was the particular flight chosen for simulation. First, the general structure of the program is presented and the trade studies which were performed are identified. Inputs required to make runs are discussed followed by identification of the output statistics. Some areas for model modifications are pointed out. A detailed model configuration, program listing and results are included.

Bengtson, N. M.↗
NASA NTRS2019

The Cumulus Ecosystem: Open Source and Beyond to Foster Collaboration

NASA’s Earth Observing System Data and Information System (EOSDIS) open source Cumulus software is designed as a common set of code and services that can be used to create a pipeline to deliver and manage earth science data in the cloud. Cumulus strives to create an ecosystem on the foundation of open source that unites those with shared problems and goals by encouraging users to contribute solutions back to the platform. Large parts of ingesting and managing data are common and much of what is created can be used by others. Our goal is to maximize collaboration and code reuse while allowing users to design a custom solution that meets their needs without having to take on extraneous functionality. In this talk we will describe how the Cumulus ecosystem works beyond just open source software. We will review the technology, the successes and challenges, and the evolution and future of Cumulus as an ecosystem.

Cumulus↗
NASA NTRS2011

Generic, Extensible, Configurable Push-Pull Framework for Large-Scale Science Missions

The push-pull framework was developed in hopes that an infrastructure would be created that could literally connect to any given remote site, and (given a set of restrictions) download files from that remote site based on those restrictions. The Cataloging and Archiving Service (CAS) has recently been re-architected and re-factored in its canonical services, including file management, workflow management, and resource management. Additionally, a generic CAS Crawling Framework was built based on motivation from Apache s open-source search engine project called Nutch. Nutch is an Apache effort to provide search engine services (akin to Google), including crawling, parsing, content analysis, and indexing. It has produced several stable software releases, and is currently used in production services at companies such as Yahoo, and at NASA's Planetary Data System. The CAS Crawling Framework supports many of the Nutch Crawler's generic services, including metadata extraction, crawling, and ingestion. However, one service that was not ported over from Nutch is a generic protocol layer service that allows the Nutch crawler to obtain content using protocol plug-ins that download content using implementations of remote protocols, such as HTTP, FTP, WinNT file system, HTTPS, etc. Such a generic protocol layer would greatly aid in the CAS Crawling Framework, as the layer would allow the framework to generically obtain content (i.e., data products) from remote sites using protocols such as FTP and others. Augmented with this capability, the Orbiting Carbon Observatory (OCO) and NPP (NPOESS Preparatory Project) Sounder PEATE (Product Evaluation and Analysis Tools Elements) would be provided with an infrastructure to support generic FTP-based pull access to remote data products, obviating the need for any specialized software outside of the context of their existing process control systems. This extensible configurable framework was created in Java, and allows the use of different underlying communication middleware (at present, both XMLRPC, and RMI). In addition, the framework is entirely suitable in a multi-mission environment and is supporting both NPP Sounder PEATE and the OCO Mission. Both systems involve tasks such as high-throughput job processing, terabyte-scale data management, and science computing facilities. NPP Sounder PEATE is already using the push-pull framework to accept hundreds of gigabytes of IASI (infrared atmospheric sounding interferometer) data, and is in preparation to accept CRIMS (Cross-track Infrared Microwave Sounding Suite) data. OCO will leverage the framework to download MODIS, CloudSat, and other ancillary data products for use in the high-performance Level 2 Science Algorithm. The National Cancer Institute is also evaluating the framework for use in sharing and disseminating cancer research data through its Early Detection Research Network (EDRN).

Foster, Brian M.↗
NASA NTRS2009

Approach to Managing MeaSURES Data at the GSFC Earth Science Data and Information Services Center (GES DISC)

A major need stated by the NASA Earth science research strategy is to develop long-term, consistent, and calibrated data and products that are valid across multiple missions and satellite sensors. (NASA Solicitation for Making Earth System data records for Use in Research Environments (MEaSUREs) 2006-2010) Selected projects create long term records of a given parameter, called Earth Science Data Records (ESDRs), based on mature algorithms that bring together continuous multi-sensor data. ESDRs, associated algorithms, vetted by the appropriate community, are archived at a NASA affiliated data center for archive, stewardship, and distribution. See http://measures-projects.gsfc.nasa.gov/ for more details. This presentation describes the NASA GSFC Earth Science Data and Information Services Center (GES DISC) approach to managing the MEaSUREs ESDR datasets assigned to GES DISC. (Energy/water cycle related and atmospheric composition ESDRs) GES DISC will utilize its experience to integrate existing and proven reusable data management components to accommodate the new ESDRs. Components include a data archive system (S4PA), a data discovery and access system (Mirador), and various web services for data access. In addition, if determined to be useful to the user community, the Giovanni data exploration tool will be made available to ESDRs. The GES DISC data integration methodology to be used for the MEaSUREs datasets is presented. The goals of this presentation are to share an approach to ESDR integration, and initiate discussions amongst the data centers, data managers and data providers for the purpose of gaining efficiencies in data management for MEaSUREs projects.

Vollmer, Bruce↗
NASA NTRS2023

Data Mining for Science of the Sun-Earth Connection as a Single System

Establishing the Sun-Earth connection requires overcoming the challenges of exploring the data from past and current missions and leveraging tools and models (data mining) to create an efficient system treatment of the Sun and heliosphere. However, solar and heliospheric environment data constitute a vast source of information whose potential is far from being optimally exploited. In the next decade, the solar and heliospheric community will have to manage the increasing amount of information coming from new missions, improve reanalysis of data from past and current missions, and create new data products from the application of new methodologies. This complex task is further complicated by practical challenges such as different datasets and catalogs in different formats that may require different pre-processing and analysis tools, and the need for numerous analysis approaches that are not all fully optimized for large volumes of data. While several ongoing efforts aim at addressing these problems, the available datasets and tools are not always used to their full potential often due to lack of awareness of available resources. In this paper, we summarize the issues raised and goals discussed by members of the community during recent conference sessions focused on data mining for science.

Sun-Earth connection↗