Search NASA⌕ Search
Home/Archive/NASA NTRS

NASA NTRS · 20180008457

Cumulus: NASA Archives in the Cloud

Frederick, Lauren P.
View original source ↗Export metadata ↓

Abstract

NASA's Earth Observing System Data and Information System (EOSDIS) houses nearly 30PBs of critical Earth Science data and with upcoming missions is expected to balloon to between 200PBs-300PBs over the next seven years. The magnitude of data collected makes it infeasible to download data and process it locally, forcing us to re-think how we store and work with earth science data. NASA has looked to the cloud to address this, building its open source Cumulus software to manage the ingest of diverse data in a wide variety of formats into the cloud and provide services to manage and access the data. In this talk, we will describe how Cumulus provides common features needed to manage a cloud archive in the realm of ingest, data stewardship, and cost-controlled distribution of science data to users and services.

Explore related subjects

EOSDISCumulusAWSDistributionCloud

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Frederick, Lauren P.. 2018-12-10. Cumulus: NASA Archives in the Cloud. https://ntrs.nasa.gov/citations/20180008457

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related reports

NASA NTRS2021

EED-2 Contract Final Report

This EED-2 contract provides for development and sustaining engineering of software and hardware systems that provide science data management for the ESDIS Project. A major activity under this contract will be for evolution and development engineering of the EOSDIS Core System (ECS), Earthdata platform, Common Metadata Repository (CMR), NASA-Compliant General Application Platform (NGAP) cloud environment, and other EOSDIS elements that provide the common capabilities and infrastructure of EOSDIS.

EOSDIS↗
NASA NTRS2019

Global Change Master Directory (GCMD) Keyword Management Process and Lifecycle

The Global Change Master Directory (GCMD) keywords are a hierarchical set of controlled vocabulary covering the Earth science disciplines that have been evolving for over 25 years. The process for how these keywords have been curated and reviewed has also evolved. This presentation will convey the process for reviewing and approving the GCMD keywords, including fast track and yearly reviews through the ESDIS Standards Office, and how Earth science users can influence keyword additions and modifications. The presentation will also highlight how keywords facilitate the discovery of EOSDIS data and services and how other organizations are using the GCMD keywords.

EOSDIS↗
NASA NTRS2018

Data, Meet Compute: NASA's Cumulus Ingest Architecture

NASA's Earth Observing System Data and Information System (EOSDIS) houses nearly 30PBs of critical Earth Science data and with upcoming missions is expected to balloon to between 200PBs-300PBs over the next seven years. In addition to the massive increase in data collected, researchers and application developers want more and faster access - enabling complex visualizations, long time-series analysis, and cross dataset research without needing to copy and manage massive amounts of data locally. NASA has looked to the cloud to address these needs, building its Cumulus system to manage the ingest of diverse data in a wide variety of formats into the cloud. In this talk, we look at what Cumulus is from a high level and then take a deep dive into how it manages complexity and versioning associated with multiple AWS Lambda and ECS microservices communicating through AWS Step Functions across several disparate installations

EOSDIS↗