Expanding Biological Repository Data Availability for Sharing and Knowledge Discovery
Explore the source record for details and available documents.
Data↗
SEARCH · Search NASA
Results for “Data science, data management”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
NASA DEVELOP’s Communication Strategy for Sharing Applied Science Outcomes
NASA’s DEVELOP Program conducts rapid feasibility projects that enable the future workforce and current decision makers to collaborate and build capacity to use Earth science data to enhance environmental management. The program communicates its results and applications to a broad audience through multiple methods: social media, website blog posts, print materials, short project videos, an “Ambassador Corps,” scientific and policy conference presentations, community engagement activities, end-of-project presentations, project “hand-offs,” traditional publications (both gray literature and peer-reviewed), and a project archive gallery. This presentation will describe the various methods employed by DEVELOP to communicate the program’s scientific outputs, target audiences, general statistics, community response, and lessons learned.
Capacity Building↗
Computational Math Problems for a Clean Energy Future
Cutting edge computational mathematics are ubiquitous in renewable energy research. Problems in resilient and reliable electric grid operations, infrastructure planning, wind farm yaw control, and more demand sophisticated and scalable computational tools that enable the transition of renewable energy technologies from proof of concept to deployment into our energy system. The mission of the Computational Science Center at NREL is to lead the lab's efforts to solve energy challenges using high-performance computing (HPC), computational science, applied mathematics, scientific data management, visualization, and informatics. In this poster, we provide a short overview of three areas of computational mathematics research at NREL: wind power scenario generation for stochastic grid operations and infrastructure planning, improved rational function approximations for electromagnetic transients codes, and wind farm yaw control using a combination of the Alternating Direction Method of Multipliers (ADMM) and reinforcement learning (RL). Increasing penetrations of renewable energy into power grids motivate the investigation of new approaches to characterizing uncertainty for five-minute economic dispatch problems. Similarly, as the penetration of distributed energy resources on power grids increases, it becomes important to revisit our methods of modelling transient phenomena, i.e. electromagnetic transients programs. Finally, the combination of ADMM and RL for wind farm yaw control presented here can potentially increase the efficiency of the deployed distributed controllers by orders of magnitude.
ADMM↗
Discovering Research Areas in Dataset Applications Through Knowledge Graphs and Large Language Models
Scientific datasets are increasingly cited in peer-reviewed journal publications, facilitating easy access to research utilizing those datasets. Datasets undergo a life cycle where older versions of datasets are replaced by newer versions often due to improvements in data resolution, algorithms, and other factors. Unlike peer reviewed documents registered with a single Digital Unique Identifier (DOI), datasets can be updated over time and the newer version of the datasets are registered with a new DOI which is not necessarily linked to the previous version of the dataset. It is challenging when publications citing a dataset need to be traced over the entire life cycle of that dataset. We provide an innovative approach to link the dataset versions and publications using a knowledge graph (KG). KG can help to trace the dataset cited in publications over the entire dataset life cycle and shed light into dataset usage in various applied research areas. We fine-tuned the pretrained NASA IMPACTINDUS Large Language Model (LLM) on a set of labeled publications abstracts. Our results showed that 87% of the publications were classified into one of twenty applied research areas, while the remaining 13% were classified into non-applied research areas. By linking datasets to applied research areas through the KG and employing Global Change Master Directory(GCMD), a well-established controlled vocabulary of scientific keywords describing Earth science datasets, we contribute to a transparent and advanced search and discovery mechanism for datasets across the Earth data ecosystem. The integrated KG and LLM approach is now incorporated and operational in dataset publication management at one of NASA’s Earth science data archival centers.
data provenance↗
NASA Ames Institutional Scientific Collection (ISC)
NASA's current human space flight research is directed towards enabling human space exploration beyond Low Earth Orbit (LEO). The Space Flight Payload Projects; Rodent Research, Cell Science, and Microbial Labs, flown on the International Space Station (ISS), benefit both the global life sciences and commercial space communities. Verified data sets, science results, peer-reviewed publications, and returned biospecimens, collected and analyzed for flight and ground investigations, are all part of the knowledge base within NASA’s Human Exploration and Operations Mission Directorate’s Space Life and Physical Sciences Research and Applications (SLPSRA) Division, specifically the Human Research and Space Biology Programs. These data and biospecimens are made available through the public LSDA website. The Ames Institutional Scientific Collection (ISC), or ARC Biobank, stores flight and ground biospecimens from Space Shuttle and ISS programs. These specimens are curated and managed by the Ames Life Sciences Data Archive (ALSDA), an internal node of NASA's Life Sciences Data Archive (LSDA). The ARC Biolbank stores over 15,000 specimens from experiments dating from 1984 to present. Currently available specimens include tissues from the circulatory, digestive, endocrine, excretory, integumentary, muscular, neurosensory, reproductive, respiratory and skeletal systems. The most recent contributions include RNA, DNA and protein extracts from Rodent Research 1 and tissues from Rodent Research 4. NASA's biospecimen collection represents a unique and limited resource. The use of these biospecimens maximizes utilization and scientific return from these unique spaceflight payload and ground control research subjects. These biospecimens are harvested following complex, costly NASA research activities to meet primary scientific objectives. Once the primary scientific objectives have been met, the remaining specimens are made available to provide secondary opportunities for complementary studies or new investigations to broaden research without large expenditures of time or resources. Innovative ways of sharing this information ultimately advances the frontiers of human space exploration as well as scientific understanding of the effects of gravity on life on earth.
French, Alison J.↗
Report on the Global Data Assembly Center (GDAC) to the 12th GHRSST Science Team Meeting
In 2010/2011 the Global Data Assembly Center (GDAC) at NASA's Physical Oceanography Distributed Active Archive Center (PO.DAAC) continued its role as the primary clearinghouse and access node for operational Group for High Resolution Sea Surface Temperature (GHRSST) datastreams, as well as its collaborative role with the NOAA Long Term Stewardship and Reanalysis Facility (LTSRF) for archiving. Here we report on our data management activities and infrastructure improvements since the last science team meeting in June 2010.These include the implementation of all GHRSST datastreams in the new PO.DAAC Data Management and Archive System (DMAS) for more reliable and timely data access. GHRSST dataset metadata are now stored in a new database that has made the maintenance and quality improvement of metadata fields more straightforward. A content management system for a revised suite of PO.DAAC web pages allows dynamic access to a subset of these metadata fields for enhanced dataset description as well as discovery through a faceted search mechanism from the perspective of the user. From the discovery and metadata standpoint the GDAC has also implemented the NASA version of the OpenSearch protocol for searching for GHRSST granules and developed a web service to generate ISO 19115-2 compliant metadata records. Furthermore, the GDAC has continued to implement a new suite of tools and services for GHRSST datastreams including a Level 2 subsetter known as Dataminer, a revised POET Level 3/4 subsetter and visualization tool, a Google Earth interface to selected daily global Level 2 and Level 4 data, and experimented with a THREDDS catalog of GHRSST data collections. Finally we will summarize the expanding user and data statistics, and other metrics that we have collected over the last year demonstrating the broad user community and applications that the GHRSST project continues to serve via the GDAC distribution mechanisms. This report also serves by extension to summarize the activities of the GHRSST Data Assembly and Systems Technical Advisory Group (DAS-TAG).
sea surface temperature (SST)↗
Position Papers for the ASCR Workshop on the Management and Storage of Scientific Data
The purpose of this workshop is to identify priority research directions in the area of data management for high-performance and scientific computing above and beyond HPC’s traditional "the parallel file system is the data-management system" model. Supporting the breadth of the DOE mission, including the explosion of AI uses and the growing needs of experimental and observational science, motivates revisiting our assumptions about data management. There are many facets of this topic to explore including: (1) Interfaces for accessing data that resides on traditional persistent storage as well as memory devices; (2) Storage-system architecture design that supports scientific workflows on varied hierarchical storage and networking devices; (3) Devising metadata management infrastructure to support FAIR principles (Findability, Accessibility, Interoperability, and Reusability); (4) Capturing provenance information about scientific data; (5) Utilizing AI to learn I/O patterns of emerging workloads for efficient data management; (6) Providing data management support for AI and complex workflows; and (7) Understanding the overlap between traditional storage systems and I/O (SSIO) efforts and data management. While the program committee has identified these topics as important areas for discussion, we welcome position papers from the community that propose additional topics of interest for discussion at the workshop. The workshop agenda will include breakout sessions for discussing these and selected topic areas to inform priority research directions for data management for high-performance and scientific computing.
97 MATHEMATICS AND COMPUTING↗
Report for the ASCR Workshop on the Management and Storage of Scientific Data
The purpose of this workshop is to identify priority research directions in the area of data management for high-performance and scientific computing above and beyond HPC’s traditional "the parallel file system is the data-management system" model. Supporting the breadth of the DOE mission, including the explosion of AI uses and the growing needs of experimental and observational science, motivates revisiting our assumptions about data management. There are many facets of this topic to explore including: (1) Interfaces for accessing data that resides on traditional persistent storage as well as memory devices; (2) Storage-system architecture design that supports scientific workflows on varied hierarchical storage and networking devices; (3) Devising metadata management infrastructure to support FAIR principles (Findability, Accessibility, Interoperability, and Reusability); (4) Capturing provenance information about scientific data; (5) Utilizing AI to learn I/O patterns of emerging workloads for efficient data management; (6) Providing data management support for AI and complex workflows; and (7) Understanding the overlap between traditional storage systems and I/O (SSIO) efforts and data management. While the program committee has identified these topics as important areas for discussion, we welcome position papers from the community that propose additional topics of interest for discussion at the workshop. The workshop agenda will include breakout sessions for discussing these and selected topic areas to inform priority research directions for data management for high-performance and scientific computing.
97 MATHEMATICS AND COMPUTING↗
Data Management as a Cluster Middleware Centerpiece
Through earth and space modeling and the ongoing launches of satellites to gather data, NASA has become one of the largest producers of data in the world. These large data sets necessitated the creation of a Data Management System (DMS) to assist both the users and the administrators of the data. Halcyon Systems Inc. was contracted by the NASA Center for Computational Sciences (NCCS) to produce a Data Management System. The prototype of the DMS was produced by Halcyon Systems Inc. (Halcyon) for the Global Modeling and Assimilation Office (GMAO). The system, which was implemented and deployed within a relatively short period of time, has proven to be highly reliable and deployable. Following the prototype deployment, Halcyon was contacted by the NCCS to produce a production DMS version for their user community. The system is composed of several existing open source or government-sponsored components such as the San Diego Supercomputer Center s (SDSC) Storage Resource Broker (SRB), the Distributed Oceanographic Data System (DODS), and other components. Since Data Management is one of the foremost problems in cluster computing, the final package not only extends its capabilities as a Data Management System, but also to a cluster management system. This Cluster/Data Management System (CDMS) can be envisioned as the integration of existing packages.
Zero, Jose↗
2020 Multiscale Microbial Dynamics Modeling Course
The 2020 Multiscale Microbial Dynamics course is adapted from the virtual 2020 Mutliscale Microbial Dynamics Summer School that was hosted by Environmental Molecular Sciences Laboratory (EMSL), a U.S. Department of Energy (DOE) science user facility located on the Pacific Northwest National Laboratory (PNNL) campus, in collaboration with the Joint Genome Institute (JGI) and the DOE Systems Biology Knowledgebase (KBase). The course course covers how to incorporate microbial metagenomic and environmental metabolite data from watershed ecosystems into metabolic and community modeling using computational frameworks, such as KBase and PFLOTRAN. The curriculum includes lectures and software and data analysis tutorials. All materials are freely accessible to the community as part of the 2020 Microbial Dynamics Summer School Organization in KBase.
54 ENVIRONMENTAL SCIENCES↗
The Echoes of Earth Science
NASA s Earth Observing System Data and Information System (EOSDIS) acquires, archives, and manages data from all of NASA s Earth science satellites, for the benefit of the Space Agency and for the benefit of others, including local governments, first responders, the commercial remote sensing industry, teachers, museums, and the general public. EOSDIS is currently handling an extraordinary amount of NASA scientific data. To give an idea of the volume of information it receives, NASA s Terra Earth-observing satellite, just one of many NASA satellites sending down data, sends it hundreds of gigabytes a day, almost as much data as the Hubble Space Telescope acquires in an entire year, or about equal to the amount of information that could be found in hundreds of pickup trucks filled with books. To make EOSDIS data completely accessible to the Earth science community, NASA teamed up with private industry in 2000 to develop an Earth science "marketplace" registry that lets public users quickly drill down to the exact information they need. It also enables them to publish their research and resources alongside of NASA s research and resources. This registry is known as the Earth Observing System ClearingHOuse, or ECHO. The charter for this project focused on having an infrastructure completely independent from EOSDIS that would allow for more contributors and open up additional data access options. Accordingly, it is only fitting that the term ECHO is more than just an acronym; it represents the functionality of the system in that it can echo out and create interoperability among other systems, all while maturing with time as industry technologies and standards change and improve.
Source record↗
RAPIDS: Reconciling Availability, Accuracy, and Performance in Managing Geo-Distributed Scientific Data
In modern science, big data plays an increasingly important role. Many scientific applications, such as running simulations on supercomputers or conducting experiments on advanced instruments, produce huge amount of data at unprecedented speed. Analyzing and understanding such big data is the key for scientists to make scientific breakthroughs. However, data might become unavailable for scientists to access when outages or maintenance of the storage system occur, which severely hinders scientific discovery. To improve the data availability, data duplication and erasure coding (EC) are often used. But as the scientific data gets larger, using these two methods can cause considerable storage and network overhead.In this paper, we propose RAPIDS, a hybrid approach that combines the multigrid-based error-bounded lossy compression with erasure coding, to significantly reduce the storage and network overhead required for maintaining high data availability. Our experiments show that RAPIDS reduces the storage overhead by up to 7.5x and network overhead by up to 3x to achieve the same level of availability compared to the regular EC method. We improve RAPIDS by building two models to optimize the fault tolerance configurations and data gathering strategy. We demonstrate that RAPIDS significantly improves performance when running on many CPU cores in parallel or on GPUs.
Wan, Lipeng↗
Assessment of the solar-terrestrial data base management system
Members of the space science community used the present MSFC NEEDS network. The design and development of improvements and extension of the NEEDS network were designed and development of improvements and extension of the NEEDS network were discussed.
Horwitz, J. L.↗
GES DISC Data Recipes in Jupyter Notebooks
The Earth Science Data and Information System (ESDIS) Project manages twelve Distributed Active Archive Centers (DAACs) which are geographically dispersed across the United States. The DAACs are responsible for ingesting, processing, archiving, and distributing Earth science data produced from various sources (satellites, aircraft, field measurements, etc.). In response to projections of an exponential increase in data production, there has been a recent effort to prototype various DAAC activities in the cloud computing environment. This, in turn, led to the creation of an initiative, called the Cloud Analysis Toolkit to Enable Earth Science (CATEES), to develop a Python software package in order to transition Earth science data processing to the cloud. This project, in particular, supports CATEES and has two primary goals. One, to transition data recipes created by the Goddard Earth Science Data and Information Service Center (GES DISC) into an interactive and educational environment using JupyterNotebooks. Two, to acclimate Earth scientists to cloud computing. To accomplish these goals, we create JupyterNotebooks to compartmentalize the different steps of data analysis and help users obtain and parse data from the command line. We also develop a Docker container, comprised of Jupyter Notebooks, Python dependencies, and command line tools, and configure it into an easy-to-deploy package. The end result is an end-to-end product that simulates the use case of end users working in the cloud computing environment.
discoverability↗
A Contrast in Use of Metrics in Earth Science Data Systems
In recent years there has been a surge in the number of systems for processing, archiving and distributing remotely sensed data. Such systems, working independently as well as in collaboration, have been contributing greatly to the advances in the scientific understanding of the Earth system, as well as utilization of the data for nationally and internationally important applications. Among such systems, we consider those that are developed by or under the sponsorship of NASA to fulfill one of its strategic objectives: "Study Earth from space to advance scientific understanding and meet societal needs." NASA's Earth science data systems are of varying size and complexity depending on the requirements they are intended to meet. Some data systems are regarded as NASA's "Core Capabilities" that provide the basic infrastructure for processing, archiving and distributing a set of data products to a large and diverse user community in a robust and reliable manner. Other data systems constitute "Community Capabilities". These provide specialized and innovative services to data users and/or research products offering new scientific insight. Such data systems are generally supported by NASA through peer reviewed competition. Examples of Core Capabilities are 1. Earth Observing Data and Information System (EOSDIS) with its Distributed Active Archive Centers (DAACs), Science Investigator-led Processing Systems (SIPSs), and the EOS Clearing House (ECHO); 2. Tropical Rainfall Measurement Mission (TRMM) Science Data and Information System (TSDIS); 3. Ocean Data Processing System (ODPS); and 4. CloudSat Data Processing Center. Examples of Community Capabilities are projects under the Research, Education and Applications Solutions Network (REASON), and Advancing Collaborative Connections for Earth System Science (ACCESS) Programs. In managing these data system capabilities, it is necessary to have well-established goals and to measure progress relative to them. Progress is measured through "metrics", which can be a combination of quantitative as well as qualitative assessments. The specific metrics of interest depend on the user of the metrics as well as the type of data system. The users of metrics can be data system managers, program managers, funding agency or the public. Data system managers need metrics for assessing and improving the performance of the system and for future planning. Program managers need metrics to assess progress and the value of the data systems sponsored by them. Also, there is a difference in the metrics needed for core capabilities that tend to be more complex, larger and longer-term compared to community capabilities and the community capabilities that tend to be simpler, smaller and shorter-term. Even among community capabilities there are differences; hence the same set of metrics does not apply to all. Some provide data products to users, some provide services that enable better utilization of data or interoperability among other systems, and some are a part of a larger project where provision of data or services is only a minor activity. There is also a contrast between metrics used for internal and external purposes. Examples of internal purposes are: ensuring that the system meets its requirements, and planning for evolution and growth. Examples of external purposes are: providing to sponsors indicators of success of the systems, demonstrating the contributions of the system to overall program success, etc. This paper will consider EOSDIS, REASON and ACCESS programs to show the various types of metrics needed and how they need to be tailored to the types of data systems while maintaining the overall management goals of measuring progress and contributions made by the data systems.
Ramapriyan, Hampapuram↗
Applying Satellite Data to Support Disaster Response and Emergency Management Decision Making
Using the vantage point of space, satellite observations provide information about the Earth that can serve a critical role in building situational awareness and filling in data gaps during disaster response. NASA’s Earth Science Division ( studies the Earth as a system and develops technologies to improve the quality of life here on our home planet. Within NASA ESD, the Disasters Program and its Disasters Response Coordination System (DRCS) aims to advance Earth science data and information to support management decisions that prevent or mitigate the impacts of disasters. Using a whole-of-NASA approach to coordinate and mobilize the Agency’s assets and expertise to provide geospatial information during disasters, this work brings the utility of Earth observation information to emergency management and disaster response and reduces the impacts of disasters on lives and livelihoods .This poster will introduce the utility of satellite and geospatial information to disaster response through examples of recent DRCS incident response activations and highlight the DRCS model that employs a user-centered activation framework beginning with direct requests from responders and ending with after-action assessments that feed lessons learned and process improvements.
Remote Sensing↗
How NASA is Building a Petabyte Scale Geospatial Archive in the Cloud
NASA's Earth Observing System Data and Information System (EOSDIS) is working towards a vision of a cloud-based, highly-flexible, ingest, archive, management, and distribution system for its ever-growing and evolving data holdings. This free and open source system, Cumulus, is emerging from its prototyping stages and is poised to make a huge impact on how NASA manages and disseminates its Earth science data. This talk outlines the motivation for this work, present the achievements and hurdles of the past 18 months and charts a course for the future expansion of Cumulus. We explore not just the technical, but also the socio-technical challenges that we face in evolving a system of this magnitude into the cloud. The NASA EOSDIS archive is currently at nearly 30 PBs and will grow to over 300PBs in the coming years. We've presented progress on this effort at AWS re:Invent and the American Geophysical Union (AGU) Fall Meeting in 2017 and hope to have the opportunity to share with FOSS4G attendees information on the availability of the open sourced software and how NASA intends on making its Earth Observing Geospatial data available for free to the public in the cloud.
NASA Archive↗
Earth-Mars Telecommunications and Information Management System (TIMS): Antenna Visibility Determination, Network Simulation, and Management Models
This report presents the results of our research on Earth-Mars Telecommunications and Information Management System (TIMS) network modeling and unattended network operations. The primary focus of our research is to investigate the feasibility of the TIMS architecture, which links the Earth-based Mars Operations Control Center, Science Data Processing Facility, Mars Network Management Center, and the Deep Space Network of antennae to the relay satellites and other communication network elements based in the Mars region. The investigation was enhanced by developing Build 3 of the TIMS network modeling and simulation model. The results of several 'what-if' scenarios are reported along with reports on upgraded antenna visibility determination software and unattended network management prototype.
Odubiyi, Jide↗