Search NASA⌕ Search

SEARCH · Search NASA

Results for “data volume”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26
All sourcesNASA NTRSDOE OSTI
NASA NTRS2018

Future of Big Earth Data Analytics

The state of the art of Big Earth Data Analytics can be expected to evolve rapidly in the coming years. The forces driving evolution come from both growth in the data and advancement in the field of data analytics. In the data area, advances in sensor instrumentation and platform miniaturization are increasing both data resolution and coverage, resulting in enormous growth in data Volume. Increases in temporal resolution in particular also generate demands for higher data Velocity. At the same time, the proliferation of instruments and the platforms on which they reside is increasing the Variety of datasets. The Variety increase in turn leads to questions about the Veracity of the data. In the algorithm area, powerful machine learning methods are coming to the fore, particularly Deep Neural Networks. These are powerful at detecting interesting features in the data, integrating many different measurements (i.e., data fusion), and classification problems. However, they are still challenging when seeking explanations of how natural or socio-economic phenomena work using Earth Observations. Thus, classical analysis techniques will remain relevant when the emphasis is on forming or testing explanations, as well as to support interactive data exploration.

Lynnes, Christopher↗
NASA NTRS2019

Evolving HPC and Application Design Toward a Coupled Data Assimilation System at NASA Suitable for Emerging Exascale Platforms

The prediction capabilities of global models have continuously evolved from the traditional medium-range global weather prediction application to span scales in support of hourly prediction of convective scale storms to seasonal Earth system prediction. This evolution has increased the demands on the system infrastructure design and workflow to achieve the required performance on modern high-performance computing (HPC) platforms. The planned evolution of the Goddard Earth Observing System (GEOS) modeling and assimilation system will stress the capabilities of conventional HPC overwhelming the available compute cycles at the NASA Center for Climate Simulation (NCCS) at the NASA Goddard Space Flight Center in the coming 5-10 years. This has led to the re-design of key elements of the assimilation and modeling systems to achieve significant gains in performance on anticipated Exacale platforms. The transition of the assimilation system to the Joint Effort for Data assimilation Integration (JEDI) framework has positioned GEOS to exploit new efficient algorithms for data assimilation (DA) in a fully-coupled Earth system context. The suitability of the GEOS model to leverage a domain specific language (DSL) approach and artificial intelligence (AI) is being explored to accelerate computational performance and data exchange efficiency of the coupled Earth system model. The storage and processing of large data volumes produced by these advance systems is being redesigned with a data-centric cloud-based approach. We will highlight the recent efforts in these areas and emphasize the demand for further development and re-design to achieve the science objectives in support of NASA's Earth system modeling and assimilation missions.

Putman, Bill↗
NASA NTRSDate not supplied

GeoNEX-SUBSETS: Geostationary Satellite Observations Over International Observing Network Sites

Satellite remote sensing is crucial for monitoring the Earth’s surface and for simulating global carbon and water cycles. Subsets of satellite data from sensors such as MODIS over established observing networks have been valuable for researchers for comparing the ground observed phenomena with satellite observations. Here, we introduce new subsets of the GeoNEX geostationary satellite datasets over locations of several ground observation networks. The NASA Earth Exchange group has been generating geostationary satellite products with our international partners and universities to cover the entire globe. In comparison to polar-orbiting satellite sensors such as MODIS, the new generation geostationary satellites observe target areas at a higher frequency (5-10 minutes), which also significantly increased the data volume. To reduce the burden of downloading and extracting geostationary time series data and provide easy access to the community, we provide subsets of GeoNEX products through our data portal [www.data.nas.nasa.gov]. The ready-to-use subset follows the same format as the MODIS fixed sites subset tools for users who are familiar with MODIS subset data and software. The selected networks include Fluxnet, AERONET, and Phenocam over the conterminous US. We demonstrate the usage of the GeoNEX fixed-site subset data and showcase thir advantages with three example studies. The first example is simulating the diurnal cycle of plant ecophysiology at Fluxnet sites. The high frequency of the GeoNEX time series allows us to run ecological models at sub-hour time steps and directly compare the simulated carbon and water fluxes with half-hourly Fluxnet data. The second example uses the geostationary data to track phenological changes. It highlights the high-frequent observations of geostationary satellites in helping mask cloud covers and capture the quick responses of vegetation to environmental changes. As such, these examples demonstrate the value of ready-to-use GeoNEX subsets data in terrestrial ecophysiology research.

GeoNEX↗
NASA NTRS1995

The CCD/Transit Instrument (CTI) data-analysis system

The automated software system for archiving, analyzing, and interrogating data from the CCD/Transit Instrument (CTI) is described. The CTI collects up to 450 Mbytes of image-data each clear night in the form of a narrow strip of sky observed in two colors. The large data-volumes and the scientific aims of the project make it imperative that the data are analyzed within the 24-hour period following the observations. To this end a fully automatic and self evaluating software system has been developed. The data are collected from the telescope in real-time and then transported to Tucson for analysis. Verification is performed by visual inspection of random subsets of the data and obvious cosmic rays are detected and removed before permanent archival is made to the optical disc. The analysis phase is performed by a pair of linked algorithms, one operating on the absolute pixel-values and the other on the spatial derivative of the data. In this way both isolated and merged images are reliably detected in a single pass. In order to isolate the latter algorithm from the effects of noise spikes a 3x3 Hanning filter is applied to the raw data before the analysis is run. The algorithms reduce the input pixel-data to a database of measured parameters for each image which has been found. A contrast filter is applied in order to assign a detection-probability to each image and then x-y calibration and intensity calibration are performed using known reference stars in the strip. These are added to as necessary by secondary standards boot-strapped from the CTI data itself. The final stages involve merging the new data into the CTI Master-list and History-list and the automatic comparison of each new detection with a set of pre-defined templates in parameter-space to find interesting objects such as supernovae, quasars and variable stars. Each stage of the processing from verification to interesting image selection is performed under a data-logging system which both controls the pipe-lining of data through the system and records key performance monitor parameters which are built into the software. Furthermore, the data from each stage are stored in databases to facilitate evaluation, and all stages offer the facility to enter keyword-indexed free-format text into the data-logging system. In this way a large measure of certification is built into the system to provide the necessary confidence in the end results.

Cawson, M. G. M.↗
NASA NTRS1995

Reduction and coding of synthetic aperture radar data with Fourier transforms

Recently, aboard the Space Radar Laboratory (SRL), the two roles of Fourier Transforms for ocean image synthesis and surface wave analysis have been implemented with a dedicated radar processor to significantly reduce Synthetic Aperture Radar (SAR) ocean data before transmission to the ground. The object was to archive the SAR image spectrum, rather than the SAR image itself, to reduce data volume and capture the essential descriptors of the surface wave field. SAR signal data are usually sampled and coded in the time domain for transmission to the ground where Fourier Transforms are applied both to individual radar pulses and to long sequences of radar pulses to form two-dimensional images. High resolution images of the ocean often contain no striking features and subtle image modulations by wind generated surface waves are only apparent when large ocean regions are studied, with Fourier transforms, to reveal periodic patterns created by wind stress over the surface wave field. Major ocean currents and atmospheric instability in coastal environments are apparent as large scale modulations of SAR imagery. This paper explores the possibility of computing complex Fourier spectrum codes representing SAR images, transmitting the coded spectra to Earth for data archives and creating scenes of surface wave signatures and air-sea interactions via inverse Fourier transformations with ground station processors.

Tilley, David G.↗
NASA NTRS2003

The Athena Microscopic Imager Investigation

The Athena science payload on the Mars Exploration Rovers (MER) includes the Microscopic Imager (MI) [1]. The MI is a fixed-focus camera mounted on the end of an extendable instrument arm, the Instrument Deployment Device (IDD; see Figure 1).The MI was designed to acquire images at a spatial resolution of 30 microns/pixel over a broad spectral range (400 - 700 nm; see Table 1). Technically, the microscopic imager is not a microscope: it has a fixed magnification of 0.4 and is intended to produce images that simulate a geologist s view through a common hand lens. In photographers parlance, the system makes use of a macro lens. The MI uses the same electronics design as the other MER cameras [2, 3] but has optics that yield a field of view of 31 31 mm across a 1024 1024 pixel CCD image (Figure 2). The MI acquires images using only solar or skylightillumination of the target surface. A contact sensor is used to place the MI slightly closer to the target surface than its best focus distance (about 66 mm), allowing concave surfaces to be imaged in good focus. Because the MI has a relatively small depth of field (3 mm), a single MI image of a rough surface will contain both focused and unfocused areas. Coarse focusing will be achieved by moving the IDD away from a rock target after the contact sensor is activated. Multiple images taken at various distances will be acquired to ensure good focus on all parts of rough surfaces. By combining a set of images acquired in this way, a completely focused image can be assembled. Stereoscopic observations can be obtained by moving the MI laterally relative to its boresight. Estimates of the position and orientation of the MI for each acquired image will be stored in the rover computer and returned to Earth with the image data. The MI optics will be protected from the Martian environment by a retractable dust cover. The dust cover includes a Kapton window that is tinted orange to restrict the spectral bandpass to 500-700 nm, allowing color information to be obtained by taking images with the dust cover open and closed. The MI will image the same materials measured by other Athena instruments (including surfaces prepared by the Rock Abrasion Tool), as well as rock and soil targets of opportunity. Subsets of the full image array can be selected and/or pixels can be binned to reduce data volume. Image compression will be used to maximize the information contained in the data returned to Earth. The resulting MI data will place other MER instrument data in context and aid in petrologic and geologic interpretations of rocks and soils on Mars.

K E Herkenhoff↗
NASA NTRS2011

Global Precipitation Mission Visualization Tool

The Global Precipitation Mission (GPM) software provides graphic visualization tools that enable easy comparison of ground- and space-based radar observations. It was initially designed to compare ground radar reflectivity from operational, ground-based, S- and C-band meteorological radars with comparable measurements from the Tropical Rainfall Measuring Mission (TRMM) satellite's precipitation radar instrument. This design is also applicable to other groundbased and space-based radars, and allows both ground- and space-based radar data to be compared for validation purposes. The tool creates an operational system that routinely performs several steps. It ingests satellite radar data (precipitation radar data from TRMM) and groundbased meteorological radar data from a number of sources. Principally, the ground radar data comes from national networks of weather radars (see figure). The data ingested by the visualization tool must conform to the data formats used in GPM Validation Network Geometry-matched data product generation. The software also performs match-ups of the radar volume data for the ground- and space-based data, as well as statistical and graphical analysis (including two-dimensional graphical displays) on the match-up data. The visualization tool software is written in IDL, and can be operated either in the IDL development environment or as a stand-alone executable function.

Schwaller, Mathew↗
NASA NTRS2016

A Spatiotemporal Indexing Approach for Efficient Processing of Big Array-Based Climate Data with MapReduce

Climate observations and model simulations are producing vast amounts of array-based spatiotemporal data. Efficient processing of these data is essential for assessing global challenges such as climate change, natural disasters, and diseases. This is challenging not only because of the large data volume, but also because of the intrinsic high-dimensional nature of geoscience data. To tackle this challenge, we propose a spatiotemporal indexing approach to efficiently manage and process big climate data with MapReduce in a highly scalable environment. Using this approach, big climate data are directly stored in a Hadoop Distributed File System in its original, native file format. A spatiotemporal index is built to bridge the logical array-based data model and the physical data layout, which enables fast data retrieval when performing spatiotemporal queries. Based on the index, a data-partitioning algorithm is applied to enable MapReduce to achieve high data locality, as well as balancing the workload. The proposed indexing approach is evaluated using the National Aeronautics and Space Administration (NASA) Modern-Era Retrospective Analysis for Research and Applications (MERRA) climate reanalysis dataset. The experimental results show that the index can significantly accelerate querying and processing (10 speedup compared to the baseline test using the same computing cluster), while keeping the index-to-data ratio small (0.0328). The applicability of the indexing approach is demonstrated by a climate anomaly detection deployed on a NASA Hadoop cluster. This approach is also able to support efficient processing of general array-based spatiotemporal data in various geoscience domains without special configuration on a Hadoop cluster.

big data↗
NASA NTRS2017

Use of Schema on Read in Earth Science Data Archives

Traditionally, NASA Earth Science data archives have file-based storage using proprietary data file formats, such as HDF and HDF-EOS, which are optimized to support fast and efficient storage of spaceborne and model data as they are generated. The use of file-based storage essentially imposes an indexing strategy based on data dimensions. In most cases, NASA Earth Science data uses time as the primary index, leading to poor performance in accessing data in spatial dimensions. For example, producing a time series for a single spatial grid cell involves accessing a large number of data files. With exponential growth in data volume due to the ever-increasing spatial and temporal resolution of the data, using file-based archives poses significant performance and cost barriers to data discovery and access. Storing and disseminating data in proprietary data formats imposes an additional access barrier for users outside the mainstream research community. At the NASA Goddard Earth Sciences Data Information Services Center (GES DISC), we have evaluated applying the schema-on-read principle to data access and distribution. We used Apache Parquet to store geospatial data, and have exposed data through Amazon Web Services (AWS) Athena, AWS Simple Storage Service (S3), and Apache Spark. Using the schema-on-read approach allows customization of indexing spatially or temporally to suit the data access pattern. The storage of data in open formats such as Apache Parquet has widespread support in popular programming languages. A wide range of solutions for handling big data lowers the access barrier for all users. This presentation will discuss formats used for data storage, frameworks with This presentation will discuss formats used for data storage, frameworks with support for schema-on-read used for data access, and common use cases covering data usage patterns seen in a geospatial data archive.

cloud applications↗
NASA NTRS1999

ICESat (GLAS) Science Processing Software Document Series: Science Data Management Plan - Volume 2

This document provides the Data Management Plan for the GLAS Standard Data Software (SDS) supporting the GLAS instrument of the EOS ICESat Spacecraft. The SDS encompasses the ICESat Science Investigator-led Processing System (I-SIPS) Software and the Instrument Support Facility (ISF) Software. This Plan addresses the identification, authority, and description of the interface nodes associated with the GLAS Standard Data Products and the GLAS Ancillary Data.

Jester, Peggy L.↗
NASA NTRS2002

Global Change Data Center: Mission, Organization, Major Activities, and 2001 Highlights

Rapid efficient access to Earth sciences data is fundamental to the Nation's efforts to understand the effects of global environmental changes and their implications for public policy. It becomes a bigger challenge in the future when data volumes increase further and missions with constellations of satellites start to appear. Demands on data storage, data access, network throughput, processing power, and database and information management are increased by orders of magnitude, while budgets remain constant and even shrink. The Global Change Data Center's (GCDC) mission is to provide systems, data products, and information management services to maximize the availability and utility of NASA's Earth science data. The specific objectives are (1) support Earth science missions be developing and operating systems to generate, archive, and distribute data products and information; (2) develop innovative information systems for processing, archiving, accessing, visualizing, and communicating Earth science data; and (3) develop value-added products and services to promote broader utilization of NASA Earth Sciences Enterprise (ESE) data and information. The ultimate product of GCDC activities is access to data and information to support research, education, and public policy.

Wharton, Stephen W.↗
NASA NTRS2009

Uncertainty Management in Remote Sensing of Climate Data. Summary of A Workshop

Great advances have been made in our understanding of the climate system over the past few decades, and remotely sensed data have played a key role in supporting many of these advances. Improvements in satellites and in computational and data-handling techniques have yielded high quality, readily accessible data. However, rapid increases in data volume have also led to large and complex datasets that pose significant challenges in data analysis (NRC, 2007). Uncertainty characterization is needed for every satellite mission and scientists continue to be challenged by the need to reduce the uncertainty in remotely sensed climate records and projections. The approaches currently used to quantify the uncertainty in remotely sensed data, including statistical methods used to calibrate and validate satellite instruments, lack an overall mathematically based framework.

Remote sensing↗
NASA NTRS2009

TRMM .25 deg x .25 deg Gridded Precipitation Text Product

Since the launch of the Tropical Rainfall Measuring Mission (TRMM), the Precipitation Measurement Missions science team has endeavored to provide TRMM precipitation retrievals in a variety of formats that are more easily usable by the broad science community than the standard Hierarchical Data Format (HDF) in which TRMM data is produced and archived. At the request of users, the Precipitation Processing System (PPS) has developed a .25 x .25 gridded product in an easily used ASCII text format. The entire TRMM mission data has been made available in this format. The paper provides the details of this new precipitation product that is designated with the TRMM designator 3G68.25. The format is packaged into daily files. It provides hourly precipitation information from the TRMM microwave imager (TMI), precipitation radar (PR), and TMI/PR combined rain retrievals. A major advantage of this approach is the inclusion only of rain data, compression when a particular grid has no rain from the PR or combined, and its direct ASCII text format. For those interested only in rain retrievals and whether rain is convection or stratiform, these products provide a huge reduction in the data volume inherent in the standard TRMM products. This paper provides examples of the 3G68 data products and their uses. It also provides information about C tools that can be used to aggregate daily files into larger time samples. In addition, it describes the possibilities inherent in the spatial sampling which allows resampling into coarser spatial sampling. The paper concludes with information about downloading the gridded text data products.

Stocker, Erich↗
NASA NTRS2010

Design of Control Software for a High-Speed Coherent Doppler Lidar System for CO2 Measurement

The design of the software for a 2-micron coherent high-speed Doppler lidar system for CO2 measurement at NASA Langley Research Center is discussed in this paper. The specific strategy and design topology to meet the requirements of the system are reviewed. In order to attain the high-speed digitization of the different types of signals to be sampled on multiple channels, a carefully planned design of the control software is imperative. Samples of digitized data from each channel and their roles in data analysis post processing are also presented. Several challenges of extremely-fast, high volume data acquisition are discussed. The software must check the validity of each lidar return as well as other monitoring channel data in real-time. For such high-speed data acquisition systems, the software is a key component that enables the entire scope of CO2 measurement studies using commercially available system components.

Vanvalkenburg, Randal L.↗
NASA NTRS1985

The magnetic field of the earth - Performance considerations for space-based observing systems

Basic problems inherent in carrying out observations of the earth magnetic field from space are reviewed. It is shown that while useful observations of the core and crustal fields are possible at the peak of the solar cycle, the greatest useful data volume is obtained during solar minimum. During the last three solar cycles, the proportion of data with a planetary disturbance index of less than 2 at solar maximum was in the range 0.4-0.8 in comparison with solar minimum. It is found that current state of the art orbit determination techniques should eliminate orbit error as a problem in gravitational field measurements from space. The spatial resolution obtained for crustal field anomalies during the major satellite observation programs of the last 30 years are compared in a table. The relationship between observing altitude and the spatial resolution of magnetic field structures is discussed. Reference is made to data obtained using the Magsat, the Polar Orbiting Geophysical Observatory (POGO), and instruments on board the Space Shuttle.

Webster, W. J., Jr.↗
NASA NTRS1993

Compression through decomposition into browse and residual images

Economical archival and retrieval of image data is becoming increasingly important considering the unprecedented data volumes expected from the Earth Observing System (EOS) instruments. For cost effective browsing the image data (possibly from remote site), and retrieving the original image data from the data archive, we suggest an integrated image browse and data archive system employing incremental transmission. We produce our browse image data with the JPEG/DCT lossy compression approach. Image residual data is then obtained by taking the pixel by pixel differences between the original data and the browse image data. We then code the residual data with a form of variable length coding called diagonal coding. In our experiments, the JPEG/DCT is used at different quality factors (Q) to generate the browse and residual data. The algorithm has been tested on band 4 of two Thematic mapper (TM) data sets. The best overall compression ratios (of about 1.7) were obtained when a quality factor of Q=50 was used to produce browse data at a compression ratio of 10 to 11. At this quality factor the browse image data has virtually no visible distortions for the images tested.

Novik, Dmitry A.↗
NASA NTRS2012

Investigation into Cloud Computing for More Robust Automated Bulk Image Geoprocessing

Geospatial resource assessments frequently require timely geospatial data processing that involves large multivariate remote sensing data sets. In particular, for disasters, response requires rapid access to large data volumes, substantial storage space and high performance processing capability. The processing and distribution of this data into usable information products requires a processing pipeline that can efficiently manage the required storage, computing utilities, and data handling requirements. In recent years, with the availability of cloud computing technology, cloud processing platforms have made available a powerful new computing infrastructure resource that can meet this need. To assess the utility of this resource, this project investigates cloud computing platforms for bulk, automated geoprocessing capabilities with respect to data handling and application development requirements. This presentation is of work being conducted by Applied Sciences Program Office at NASA-Stennis Space Center. A prototypical set of image manipulation and transformation processes that incorporate sample Unmanned Airborne System data were developed to create value-added products and tested for implementation on the "cloud". This project outlines the steps involved in creating and testing of open source software developed process code on a local prototype platform, and then transitioning this code with associated environment requirements into an analogous, but memory and processor enhanced cloud platform. A data processing cloud was used to store both standard digital camera panchromatic and multi-band image data, which were subsequently subjected to standard image processing functions such as NDVI (Normalized Difference Vegetation Index), NDMI (Normalized Difference Moisture Index), band stacking, reprojection, and other similar type data processes. Cloud infrastructure service providers were evaluated by taking these locally tested processing functions, and then applying them to a given cloud-enabled infrastructure to assesses and compare environment setup options and enabled technologies. This project reviews findings that were observed when cloud platforms were evaluated for bulk geoprocessing capabilities based on data handling and application development requirements.

Brown, Richard B.↗
NASA NTRS2015

Development of the Science Data System for the International Space Station Cold Atom Lab

Cold Atom Laboratory (CAL) is a facility that will enable scientists to study ultra-cold quantum gases in a microgravity environment on the International Space Station (ISS) beginning in 2016. The primary science data for each experiment consists of two images taken in quick succession. The first image is of the trapped cold atoms and the second image is of the background. The two images are subtracted to obtain optical density. These raw Level 0 atom and background images are processed into the Level 1 optical density data product, and then into the Level 2 data products: atom number, Magneto-Optical Trap (MOT) lifetime, magnetic chip-trap atom lifetime, and condensate fraction. These products can also be used as diagnostics of the instrument health. With experiments being conducted for 8 hours every day, the amount of data being generated poses many technical challenges, such as downlinking and managing the required data volume. A parallel processing design is described, implemented, and benchmarked. In addition to optimizing the data pipeline, accuracy and speed in producing the Level 1 and 2 data products is key. Algorithms for feature recognition are explored, facilitating image cropping and accurate atom number calculations.

bose einstein condensate↗