Search NASA⌕ Search

SEARCH · Search NASA

Results for “AWS cost”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Cloud Giovanni: Reining in Costs and Improving Performance with Analytical Data Stores Using Scalable Serverless Architecture

Giovanni is the Geospatial Interactive Online Visualization ANd aNalysis Infrastructure developed at NASA GES DISC which provides a simple and intuitive way to visualize, analyze, and access vast amounts of Earth science data. It receives large number of user requests each day for a variety of analysis and visualization services, which leads to the big data challenge of serving gradually increasing large data volumes with diverse statistical algorithms. We hereby propose a multi-dimensional accumulation method which provides fast and cost-efficient cloud analysis for diverse services including both area averaging and time averaging. This method involves the weighted volume integration over multiple variable dimensions (time and space), and is implemented in AWS using Athena providing serverless and highly scalable data analysis. Compared to the standard method, this approach dramatically reduced the computational time by order of magnitude with a minimal AWS cost incurred. For example, for a benchmark of 10-year area averaging over the 1x1 degree daily variable, the computational time was reduced from minutes to seconds, and the Athena cost is only $5 for 100,000 requests.

Zhang, Hailiang↗

Trade Study: Storing NASA HDF5/netCDF-4 Data in the Amazon Cloud and Retrieving Data Via Hyrax Server Data Server

This study explored three candidate architectures with different types of objects and access paths for serving NASA Earth Science HDF5 data via Hyrax running on Amazon Web Services (AWS). We studied the cost and performance for each architecture using several representative Use-Cases. The objectives of the study were: Conduct a trade study to identify one or more high performance integrated solutions for storing and retrieving NASA HDF5 and netCDF4 data in a cloud (web object store) environment. The target environment is Amazon Web Services (AWS) Simple Storage Service (S3). Conduct needed level of software development to properly evaluate solutions in the trade study and to obtain required benchmarking metrics for input into government decision of potential follow-on prototyping. Develop a cloud cost model for the preferred data storage solution (or solutions) that accounts for different granulation and aggregation schemes as well as cost and performance trades.We will describe the three architectures and the use cases along with performance results and recommendations for further work.

AWS cost↗

Task 28: Web Accessible APIs in the Cloud Trade Study

This study explored three candidate architectures for serving NASA Earth Science Hierarchical Data Format Version 5 (HDF5) data via Hyrax running on Amazon Web Services (AWS). We studied the cost and performance for each architecture using several representative Use-Cases. The objectives of the project are: Conduct a trade study to identify one or more high performance integrated solutions for storing and retrieving NASA HDF5 and Network Common Data Format Version 4 (netCDF4) data in a cloud (web object store) environment. The target environment is Amazon Web Services (AWS) Simple Storage Service (S3).Conduct needed level of software development to properly evaluate solutions in the trade study and to obtain required benchmarking metrics for input into government decision of potential follow-on prototyping. Develop a cloud cost model for the preferred data storage solution (or solutions) that accounts for different granulation and aggregation schemes as well as cost and performance trades.

cost model↗

Lessons Learned and Cost Analysis of Hosting a Full Stack Open Data Cube (ODC) Application on the Amazon Web Services (AWS)

The Open Data Cube (ODC) initiative, with support from the Committee on Earth Observation Satellites (CEOS) System Engineering Office (SEO) has developed a state-of-the-art suite of software tools and products to facilitate the analysis of Earth Observation data. This paper presents a short summary and cost analysis of our experience using Amazon Web Services (AWS) to host one such software product, the CEOS Data Cube (CDC) web-based User Interface (UI). In order to provide adaptability, flexibility, scalability, and robustness, we leverage widely-adopted and well-supported technologies such as the Django web framework and the AWS Cloud platform. The UI has empowered users by providing features that assist with streamlining data preparation, data processing, data visualization, and the sub-setting of Analysis Ready Data (ARD) products in order to achieve a wide variety of Earth imaging objectives.

Rizvi, Syed R.↗

Giovanni in the Cloud: Earth Science Data Exploration in Amazon Web Services

Giovanni is an exploration tool at the NASA Goddard Earth Sciences Data Information Services Center (GES DISC), providing 22 analysis and visualization services for over 1600 Earth Science data variables. Owing to its popularity, Giovanni has experienced a consistent growth in overall demand, with periodic usage spikes attributed to trainings by education organizations, extensive data analysis in response to natural disasters, preparations for science meetings, etc. Furthermore, the new generation of spaceborne sensors and high resolution models have resulted in an exponential growth in data volume with data distributed across the traditional boundaries of data centers. Seamless exploration of data (without users having to worry about data center boundaries) has been a key recommendation of the GES DISC User Working Group. These factors have required new strategies for delivering acceptable performance. The cloud-based Giovanni, built on Amazon Web Services (AWS), evaluates (1) AWS native solutions to provide a scalable, serverless architecture; (2) open standards for data storage in the Cloud; (3) a cost model for operations; and (4) end-user performance. Our preliminary findings indicate that the use of serverless architecture has a potential to significantly reduce development and operational cost of Giovanni. The combination of using AWS managed services, storage of data in open standards, and schema-on-read data access strategy simplifies data access and analytics, in addition to making data more accessible to the end users of Giovanni through popular programming languages.

Giovanni↗

GeneLab Analysis Working Group Kick-Off Meeting

Goals to achieve for GeneLab AWG - GL vision - Review of GeneLab AWG charter Timeline and milestones for 2018 Logistics - Monthly Meeting - Workshop - Internship - ASGSR Introduction of team leads and goals of each group Introduction of all members Q/A Three-tier Client Strategy to Democratize Data Physiological changes, pathway enrichment, differential expression, normalization, processing metadata, reproducibility, Data federation/integration with heterogeneous bioinformatics external databases The GLDS currently serves over 100 omics investigations to the biomedical community via open access. In order to expand the scope of metadata record searches via the GLDS, we designed a metadata warehouse that collects and updates metadata records from external systems housing similar data. To demonstrate the capabilities of federated search and retrieval of these data, we imported metadata records from three open-access data systems into the GLDS metadata warehouse: NCBI's Gene Expression Omnibus (GEO), EBI's PRoteomics IDEntifications (PRIDE) repository, and the Metagenomics Analysis server (MG-RAST). Each of these systems defines metadata for omics data sets differently. One solution to bridge such differences is to employ a common object model (COM) to which each systems' representation of metadata can be mapped. Warehoused metadata records are then transformed at ETL to this single, common representation. Queries generated via the GLDS are then executed against the warehouse, and matching records are shown in the COM representation (Fig. 1). While this approach is relatively straightforward to implement, the volume of the data in the omics domain presents challenges in dealing with latency and currency of records. Furthermore, the lack of a coordinated has been federated data search for and retrieval of these kinds of data across other open-access systems, so that users are able to conduct biological meta-investigations using data from a variety of sources. Such meta-investigations are key to corroborating findings from many kinds of assays and translating them into systems biology knowledge and, eventually, therapeutics.

GeneLab↗

Cost Controls in the Cloud

The pay-as-you-go model used by most Cloud Service Providers shifts ownership of resource procurement from finance and management teams to IT, engineering, development and other stakeholders. This requires individual contributors to adopt a more "cost conscious" approach to development, and for projects to invest time into the creation of automated cost controls and cost-focused monitoring. This talk will presents some patterns and examples of cost controls based on our experience managing a multi-tenant cloud environment.

circuit breaker↗

NASA Earthdata Cloud

This presentation is an overview OF EOSDIS and motivation for commercial cloud and will discuss components of the Earthdata cloud, AWS accounts metric, account structure architecture and cloud resource and cost model.

McInerney, Mark A.↗

SatCORPS Global Cloud Composite (GCC): the Design and Delivery of A High Quality, High Resolution, Global Cloud Product Available in Near-Real Time

The NASA Satellite ClOud and Radiation Property retrieval System (SatCORPS) supports the development of an analysis ready and cloud-optimized data transformation pipeline and geospatial service enablement of a global cloud composite (GCC) product derived from global geostationary satellite imagery. This geospatial service will be available at high temporal and spatial resolution via the SatCORPS web mapping application for visualization and analysis as well as direct ingestion to common geospatial software and custom programming. The resulting global cloud composite products from the processing pipeline can then be geospatially-service enabled as ArcGIS Image Services and Open Geospatial Consortium (OGC) Web Mapping/Coverage Services for visualization and analysis via a web mapping application and common geospatial software. Near real time global observations are created through the composition of five geostationary satellites that provides modelling and forecasting communities with the capability to provide high quality and timely information to start the projection process. The Global Cloud Composite product combines information from geostationary satellites, GOES-16, GOES-17, Himawari-8, Meteosat-11 and Meteosat-9 to create a single global composite netcdf file and images using the different products within the netcdf file. The SatCORPS team, though our Global Cloud Composite (GCC) product and web-based visualization tools including Geographical Information System (GIS) services provide near real time global cloud product information to both automated processes and traditional web users that is timely and high quality derived from geostationary satellites. The Global Cloud Composite product takes advantage of the scalable processing resources provided by the AWS batch service to provide new composites every thirty minutes. Because information from each of the low earth orbiting satellites is available on schedules tuned to the specific satellite, the processing algorithm temporally composites the final dataset as each satellite’s information becomes available. The SatCORPS team has leveraged our experience using Amazon Web Services (AWS) to build a low latency high availability tool that allows end users both human and automated to acquire high quality and high-resolution Geostationary Earth Orbiting (GEO) information at zero cost to the end user. This presentation will describe how we architected and implemented the service as well as lessons learned based on our experiences both developing and operating the system. The lessons learned include how we integrated multiple services including Amazon Batch, Amazon S3 and Amazon Lambda service to create a low cost but high-performance processing system that is capable of identifying and processing the most appropriate satellite overpass information into global cloud composites. We will also describe our web-based tools including our Geographic Information System that can be used for visualization and analysis. The products from the processing can be geospatially-service enabled as ArcGIS Image Services and Open Geospatial Consortium (OGC) Web Mapping/Coverage Services for visualization and analysis via a web mapping application and common geospatial software. The SatCORPS Global Composite Cloud product provides sophisticated global composited cloud research products with very low latency that we see that as filling a rapidly growing need in the research and modelling community with no up-front nor ongoing costs associated with downloading or using the information.

AWS AMCE SMCE GCC SATCORPS GLOBAL CLOUD COMPOSITE ↗

Air and Water System (AWS) Design and Technology Selection for the Vision for Space Exploration

This paper considers technology selection for the crew air and water recycling systems to be used in long duration human space exploration. The specific objectives are to identify the most probable air and water technologies for the vision for space exploration and to identify the alternate technologies that might be developed. The approach is to conduct a preliminary first cut systems engineering analysis, beginning with the Air and Water System (AWS) requirements and the system mass balance, and then define the functional architecture, review the International Space Station (ISS) technologies, and discuss alternate technologies. The life support requirements for air and water are well known. The results of the mass flow and mass balance analysis help define the system architectural concept. The AWS includes five subsystems: Oxygen Supply, Condensate Purification, Urine Purification, Hygiene Water Purification, and Clothes Wash Purification. AWS technologies have been evaluated in the life support design for ISS node 3, and in earlier space station design studies, in proposals for the upgrade or evolution of the space station, and in studies of potential lunar or Mars missions. The leading candidate technologies for the vision for space exploration are those planned for Node 3 of the ISS. The ISS life support was designed to utilize Space Station Freedom (SSF) hardware to the maximum extent possible. The SSF final technology selection process, criteria, and results are discussed. Would it be cost-effective for the vision for space exploration to develop alternate technology? This paper will examine this and other questions associated with AWS design and technology selection.

Jones, Harry↗

Lessons from Earth Aerobiology for Venus Astrobiology

Venus’s clouds have often suggested as a possible habitat. The constraints governing putative airborne life in such a habitat in turn inform priorities and strategies for remote and in situ exploration, methods by which resulting biosignatures might be detected, exoplanet habitability assessment, and planetary protection concerns. Lessons drawn from studying Earth’s aerobiosphere can help improve this understanding. There are altitude ranges within Venus’s clouds in which temperature, pressure, particle size, and radiation appear to be within the limits of microbial life on Earth, and life cycles involving S- and Fe-based redox metabolism have been proposed. However, given the lack of a habitable surface reservoir, a long-term stable Venus aerobiosphere would require that the reproduction rate of the airborne microbes be faster than the settling rate of airborne microbes due to gravity, or eventually the population would be depleted; put another way, the mean generation time would need to exceed the mean residence time. This creates a joint constraint of aerosol dynamics, potential nutrient availability and energy influx, and bioenergetic costs such as desiccation and radiation damage. Even at an optimistic estimate of 75% H2SO4, Venus aerosol water activity (aw) is still ~0.02, far below the observed microbial growth limit of ~0.6. Long-term desiccation with brief spurts of repair and growth in response to transient water influx, such as from volcanism, is the most likely model for Earth-like life on Venus – a ‘desert bloom’ scenario. Several high-priority science goals – cloud aerosol composition, internal radiative flux, and aerosol residence time and circulation models – thus will also improve our understanding of Venus in an astrobiology context. In Earth’s troposphere, warm water clouds can carry 103 – 105 cells/mL, some metabolically active. However, Earth’s stratospheric sulfate aerosol layer may be a better analogue: supercooled sulfuric acidaerosols (acid weight fraction 0.6 – 0.85, 0.1 – 1 μm diameter, 0.1 – 1 cm-3) with little water activity, long residence times, high UV radiation, and only sporadic influx from surface particle sources. Though ‘hot spots’ can occur associated with tropospheric mixing, viable cells in stratospheric samples are rare (~102 cells/m3), and primarily inactive forms such as spores. It is not yet clear whether such bioaerosols are associated with sulfate aerosols or simply co-located, and reproduction in situ has not yet been observed. In this model of a sparse, largely dormant Venus ecosystem, a single transect on descent is likely to pass through a low-water, inactive region, missing potential signs of habitability or biosignatures. A targeted strategy would sample through an aerial region with some upwelling from surface sources, andtake multiple transects separated in time and space. This is compatible with other in situ science goals seeking to understand the dynamics and heterogeneities of Venus’s clouds.

Lower atmosphere↗

NASA Archives and Data Stewardship in the Cloud with Cumulus

NASA's Earth Observing System Data and Information System (EOSDIS) houses nearly 30PB (petabytes) of critical Earth Science data and with upcoming missions is expected to balloon to between 200PBs-300PBs over the next seven years. The magnitude of data collected makes it infeasible to download data and process it locally, forcing us to re-think how we store and work with earth science data. NASA has looked to the cloud to address this, building its open source Cumulus software to manage the ingest of diverse data in a wide variety of formats into the cloud and provide services to manage and access the data. In this talk, we will describe how Cumulus provides common features needed to manage a cloud archive in the realm of ingest, data stewardship, and cost-controlled distribution of science data to users and services.

NASA↗

Cumulus: NASA Archives in the Cloud

NASA's Earth Observing System Data and Information System (EOSDIS) houses nearly 30PBs of critical Earth Science data and with upcoming missions is expected to balloon to between 200PBs-300PBs over the next seven years. The magnitude of data collected makes it infeasible to download data and process it locally, forcing us to re-think how we store and work with earth science data. NASA has looked to the cloud to address this, building its open source Cumulus software to manage the ingest of diverse data in a wide variety of formats into the cloud and provide services to manage and access the data. In this talk, we will describe how Cumulus provides common features needed to manage a cloud archive in the realm of ingest, data stewardship, and cost-controlled distribution of science data to users and services.

EOSDIS↗

Lessons from Earth Bioaerosol Sampling for Venus Habitability & Biosignature Science

Venus’s clouds are often discussed as a potential habitat. Several high-priority science goals – cloud aerosol composition, internal radiative flux, and aerosol residence time and circulation models – will improve our understanding of Venus in an astrobiology context. Lessons from Earth’s aerobiosphere can inform these requirements and interpretation of results. A stable Venus aerobiosphere would need microorganisms to reproduce faster than they settle out due to gravity. This is a joint constraint of the potential nutrients and energy influx, bioenergetic costs such as desiccation and radiation damage, and aerosol dynamics. Though life cycles involving S- and Fe-based redox metabolism have been proposed, at an optimistic estimate of 75% H2SO4, Venus aerosol water activity (aw) is still ~0.02, far below the observed microbial growth limit of ~0.6. Long-term desiccation with brief spurts of repair, growth and reproduction in response to transient water influx (perhaps volcanism) is the most likely model for Earth-like life on Venus – a ‘desert bloom’ scenario. In Earth’s troposphere, clouds can carry 103 – 105 cells/mL, some metabolically active. A better analogue may be Earth’s stratospheric sulfate aerosol layer: supercooled sulfuric acid aerosols (acid weight fraction 0.6 – 0.85, 0.1 – 1 µm diameter, 0.1 – 1 cm-3) with little water activity, long residence times, high UV radiation, and only sporadic influx from surface particle sources. Viable cells isolated from this region are rare (~102 cells/m3), though ‘hot spots’ at tropospheric mixing occur. Cells are primarily in inactive forms such as spores, and it is unclear if any are associated with sulfate aerosols or simply co-located. Given the heterogeneities in Venus clouds and the model of a sparse, largely dormant ecosystem, a single transect on descent would most likely pass through a low-water, inactive region, missing potential signs of habitability or biosignatures. Even a passive aerial platform (balloon) may end up following a single air mass over its lifetime. A better strategy for Venus would target an aerial region with some upwelling from surface sources, and take multiple transects separated in time and space. Earth’s stratospheric bioaerosol population could provide a starting point for estimating the required lower volume and detection limits.

Earth↗

Geonex: A NASA-NOAA Collaboration for Producing Land Surface Products from Geostationary Sensors Using Cloud Computing

The latest generation of geostationary satellites carry sensors such as the Advanced Baseline Imager (GOES-16/17) and the Advanced Himawari Imager (Himawari-8/9) that closely mimic the spatial and spectral characteristics of MODIS and VIIRS, useful for monitoring land surface conditions. The NASA Earth Exchange (NEX) team at Ames Research Center has embarked on a collaborative effort among scientists from NASA and NOAA exploring the feasibility of producing operational land surface products similar to those from MODIS/VIIRS. The team built a processing pipeline called GEONEX that is capable of converting raw geostationary data into routine products of Fires, surface reflectances, vegetation indices, LAI/FPAR, ET and GPP/NPP using algorithms adapted from both NASA/EOS and NOAA/GOES-R programs. The GEONEX pipeline has been deployed on Amazon Web Services cloud platform and it currently leverages near-realtime geostationary data hosted in AWS public datasets under a NOAA-AWS agreement.Initial analyses of various products from ABI/AHI sensors suggest that they are comparable to those from MODIS in representing the spatio-temporal dynamics of land conditions. Cloud computing offers a variety of options for deploying the GEONEX pipeline including choice CPUs, storage media, and automation. We estimate the cost of deploying GEONEX to be $400 - 750 a month for processing data (every 30 minutes) and producing products over the conterminous US. For products such as Fire, latency can be as little as 10 minutes from the time of data acquisition.

Geostationary↗

NASA Space Communications and Navigation Approach to Cloud Capability Implementation

The integration of cloud services and capabilities across government and commercial sectors has continued to increase over the last several years. Cloud architectures provide on-demand process, storage, and transfer of data as a service without the need to purchase expensive hardware, infrastructure, and their associated maintenance costs. Most cloud environments charge the user for only the resources they use, making cloud implementations highly scalable and easily adjustable to the user’s changing demands.

cloud computing↗

Use of Schema on Read in Earth Science Data Archives

Traditionally, NASA Earth Science data archives have file-based storage using proprietary data file formats, such as HDF and HDF-EOS, which are optimized to support fast and efficient storage of spaceborne and model data as they are generated. The use of file-based storage essentially imposes an indexing strategy based on data dimensions. In most cases, NASA Earth Science data uses time as the primary index, leading to poor performance in accessing data in spatial dimensions. For example, producing a time series for a single spatial grid cell involves accessing a large number of data files. With exponential growth in data volume due to the ever-increasing spatial and temporal resolution of the data, using file-based archives poses significant performance and cost barriers to data discovery and access. Storing and disseminating data in proprietary data formats imposes an additional access barrier for users outside the mainstream research community. At the NASA Goddard Earth Sciences Data Information Services Center (GES DISC), we have evaluated applying the schema-on-read principle to data access and distribution. We used Apache Parquet to store geospatial data, and have exposed data through Amazon Web Services (AWS) Athena, AWS Simple Storage Service (S3), and Apache Spark. Using the schema-on-read approach allows customization of indexing spatially or temporally to suit the data access pattern. The storage of data in open formats such as Apache Parquet has widespread support in popular programming languages. A wide range of solutions for handling big data lowers the access barrier for all users. This presentation will discuss formats used for data storage, frameworks with This presentation will discuss formats used for data storage, frameworks with support for schema-on-read used for data access, and common use cases covering data usage patterns seen in a geospatial data archive.

cloud applications↗

Evaluation of Margins of Safety in Brazed Joints

One of the essential steps in assuring reliable performance of high cost critical brazed structures is the assessment of the Margin of Safety (MS) of the brazed joints. In many cases the experimental determination of the failure loads by destructive testing of the brazed assembly is not practical and cost prohibitive. In such cases the evaluation of the MS is performed analytically by comparing the maximum design loads with the allowable ones and incorporating various safety or knock down factors imposed by the customer. Unfortunately, an industry standard methodology for the design and analysis of brazed joints has not been developed. This paper provides an example of an approach that was used to analyze an AlBeMet 162 (38%Be-62%Al) structure brazed with the AWS BAlSi-4 (Al-12%Si) filler metal. A practical and conservative interaction equation combining shear and tensile allowables was developed and validated to evaluate an acceptable (safe) combination of tensile and shear stresses acting in the brazed joint. These allowables are obtained from testing of standard tensile and lap shear brazed specimens. The proposed equation enables the assessment of the load carrying capability of complex brazed joints subjected to multi-axial loading.

Flom, Yury↗