Search NASA⌕ Search

SEARCH · Search NASA

Results for “data challenges”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22
All sourcesNASA NTRSDOE OSTI
NASA NTRS2012

Bridging Informatics and Earth Science: a Look at Gregory Leptoukh's Contributions

With the tragic passing this year of Gregory Leptoukh, the Earth and Space Sciences community lost a tireless participant in--and advocate for--science informatics. Throughout his career at NASA, Dr. Leptoukh established a theme of bridging the gulf between the informatics and science communities. Nowhere is this more evident than his leadership in the development of Giovanni (GES DISC Interactive Online Visualization ANd aNalysis Infrastructure). Giovanni is an online tool that serves to hide the often-complex technical details of data format and structure, making science data easier to explore and use by Earth scientists. To date Giovanni has been acknowledged as a contributor in 500-odd scientific articles. In recent years, Leptoukh concentrated his efforts on multi-sensor data inter-comparison, merging and fusion. This work exposed several challenges at the intersection of data and science. One of these was the ease with which a naive user might generate spurious comparisons, a potential hazard that was the genesis of the Multi-sensor Data Synergy Advisor (MDSA). The MDSA uses semantic ontologies and inference rules to organize knowledge about dataset quality and other salient characteristics in order to advise users on potential caveats for comparing or merging two datasets. Recently, Leptoukh also led the development of AeroStat, an online Giovanni instance to investigate aerosols via statistics from station and satellite comparisons and merged maps of data from more than one instrument. Aerostat offers a neural net based bias adjustment to harmonize the data by removing systematic offsets between datasets before merging. These examples exhibit Leptoukh's talent for adopting advanced computer technologies in the service of making science data more accessible to researchers. In this, he set an example that is at once both vital and challenging for the ESSI community to emulate.

Source record↗
NASA NTRS2019

Providing Data Access and Analysis Capabilities to SERVIR’s Data-Sparse Regions

In developing regions of the world, the communications infrastructure pose enormous challenges for using Earth observation data. Limited internet bandwidth along with the high costs make it almost impossible to process and extract zonal statistics over large periods of time for even small geographic areas. In such cases, downloading daily rainfall data or dekadal series of NDVI data would take days and consume all the bandwidth allocated to an organization (for reference, internet connections in Niger would cost thousands of dollars per month at a maximum - and unreliable - bandwidth of just 10 Mbps). Running crop models or hydrological models typically require several years of historic data over the area of interest (AOI). In some cases, these AOIs are relatively small compared to the footprint of individual earth observation granules. Hence, systems that let the stakeholders subset the data to download to a user specified area, or even submit processing requests that let them download small result files for the AOI become critical. The SERVIR program has developed a tool to provide this type of access to help decision makers in developing regions use long time series of adjusted rainfall data (CHIRPS), NDVI values, seasonal weather forecasts, evaporative stress indices and others in a very efficient manner. This system, named ClimateSERV (https://climateserv.servirglobal.net) ingests the datasets in an automated fashion and allows interactive access (through a web application), or automated access through a simple API that developers can quickly incorporate in independent applications. This way, the extraction of daily averages of rainfall over a 50 square Km area through 30 years of archived data takes only a few seconds to process, and the results can be presented on an online chart or downloaded in a comma separated file that's only a few Kb.

Ashmall, William↗
NASA NTRS2015

Taming Big Data Variety in the Earth Observing System Data and Information System

Although the volume of the remote sensing data managed by the Earth Observing System Data and Information System is formidable, an oft-overlooked challenge is the variety of data. The diversity in satellite instruments, science disciplines and user communities drives cost as much or more as the data volume. Several strategies are used to tame this variety: data allocation to distinct centers of expertise; a common metadata repository for discovery, data format standards and conventions; and services that further abstract the variations in data.

Information Systems↗
NASA NTRS2017

Collaborative Data Publication Utilizing the Open Data Repository's (ODR) Data Publisher

Introduction: For small communities in diverse fields such as astrobiology, publishing and sharing data can be a difficult challenge. While large, homogenous fields often have repositories and existing data standards, small groups of independent researchers have few options for publishing standards and data that can be utilized within their community. In conjunction with teams at NASA Ames and the University of Arizona, the Open Data Repository's (ODR) Data Publisher has been conducting ongoing pilots to assess the needs of diverse research groups and to develop software to allow them to publish and share their data collaboratively. Objectives: The ODR's Data Publisher aims to provide an easy-to-use and implement software tool that will allow researchers to create and publish database templates and related data. The end product will facilitate both human-readable interfaces (web-based with embedded images, files, and charts) and machine-readable interfaces utilizing semantic standards. Characteristics: The Data Publisher software runs on the standard LAMP (Linux, Apache, MySQL, PHP) stack to provide the widest server base available. The software is based on Symfony (www.symfony.com) which provides a robust framework for creating extensible, object-oriented software in PHP. The software interface consists of a template designer where individual or master database templates can be created. A master database template can be shared by many researchers to provide a common metadata standard that will set a compatibility standard for all derivative databases. Individual researchers can then extend their instance of the template with custom fields, file storage, or visualizations that may be unique to their studies. This allows groups to create compatible databases for data discovery and sharing purposes while still providing the flexibility needed to meet the needs of scientists in rapidly evolving areas of research. Research: As part of this effort, a number of ongoing pilot and test projects are currently in progress. The Astrobiology Habitable Environments Database Working Group is developing a shared database standard using the ODR's Data Publisher and has a number of example databases where astrobiology data are shared. Soon these databases will be integrated via the template-based standard. Work with this group helps determine what data researchers in these diverse fields need to share and archive. Additionally, this pilot helps determine what standards are viable for sharing these types of data from internally developed standards to existing open standards such as the Dublin Core (http://dublincore.org) and Darwin Core (http://rs.twdg.org) metadata standards. Further studies are ongoing with the University of Arizona Department of Geosciences where a number of mineralogy databases are being constructed within the ODR Data Publisher system. Conclusions: Through the ongoing pilots and discussions with individual researchers and small research teams, a definition of the tools desired by these groups is coming into focus. As the software development moves forward, the goal is to meet the publication and collaboration needs of these scientists in an unobtrusive and functional way.

easy to use and implement software tool↗
NASA NTRS2019

An Integrated Data Analytics Platform

An Integrated Science Data Analytics Platform is an environment that enables the confluence of resources for scientific investigation. It harmonizes data, tools and computational resources which subsequently enable the research community to focus on the investigation rather than spending time on security, data preparation, management, etc. OceanWorks is a NASA technology integration project to establish a cloud-based Integrated Ocean Science Data Analytics Platform at NASA’s Physical Oceanography Distributed Active Archive Center (PO.DAAC) for big ocean science. It focuses on advancement and maturity by bringing together several NASA open-source, big data projects for parallel analytics, anomaly detection, in-situ to satellite data matchup, quality-screened data subsetting, search relevancy, and data discovery. Our communities are relying on data distributed through data centers such as the PO.DAAC, COAPS, NCAR, and many others to conduct their research. In typical investigations, scientists would engage in: search for data, evaluate the relevance of that data, download it, and then apply algorithms to identify trends. Such workflow cannot scale if the research involves a massive amount of data or multi-variate measurements. NASA’s Surface Water and Ocean Topography (SWOT) mission is expected to produce massive amount of observational data during its 3-year nominal mission. Collections like SWOT challenges all existing Earth Science data archival, distribution and analysis paradigms. In this paper, we will discuss how OceanWorks enhances the analysis of physical ocean data where the computation is done on an elastic cloud platform next to the archive to deliver fast, web-accessible services for working with oceanographic measurements.

Yang, Chaowei↗
NASA NTRS2011

Provenance Challenges for Earth Science Dataset Publication

Modern science is increasingly dependent on computational analysis of very large data sets. Organizing, referencing, publishing those data has become a complex problem. Published research that depends on such data often fails to cite the data in sufficient detail to allow an independent scientist to reproduce the original experiments and analyses. This paper explores some of the challenges related to data identification, equivalence and reproducibility in the domain of data intensive scientific processing. It will use the example of Earth Science satellite data, but the challenges also apply to other domains.

Tilmes, Curt↗
NASA NTRS2019

Uncertainties for Pre- and Post-Launch Radiometric Calibration of Imaging Spectrometers for Multi-Sensor Applications

An important aspect to using imaging spectrometer data is the radiometric characterization and calibration of the sensors and validation of their data products and doing so with error budgets with known traceability. The radiometric accuracy of a given sensor is important for demonstrating the expected quality of data from the sensor. Known traceability allows data from multiple sensors to be directly comparable as will become more important in the near future with the expected launches of multiple imaging spectrometers from multiple countries, agencies, and commercial entities. The current work describes the state of pre- and post-launch radiometric absolute and relative uncertainties and their role in harmonising on-orbit data. Examples of prelaunch uncertainties based on the calibration of EnMAP and the calibration planned for the CLARREO Pathfinder Mission are presented highlighting recent work in the area of detector-based approaches using tunable laser sources. Post-launch calibration approaches for Pathfinder, EnMAP, CHIME, and DESIS including traditional vicarious calibration methods and the challenges of working with commercial data are presented. The vicarious calibration discussion relies on the example of the recently-available RadCalNet data to describe typical methods and challenges that will be faced when harmonising data between imaging spectrometers as well as with multispectral sensors.

Thome, K.↗
NASA NTRS1992

TOVS Pathfinder product validation and intercomparison

Validation and intercomparison is an essential part of the Pathfinder program. Since its organization within the International TOVS Working Group, the TOVS community has continuously emphasized this aspect of the product retrieval problem. In the previous sections of this report, the TOVS Pathfinder SWG has recommended a careful and coherent reorganization and archiving of TOVS radiance data, and three distinct pathways for deriving product variables from this TOVS radiance archive. The importance in taking this multiple path approach to creating climate datasets rests in our firm belief that there is now global 'absolute truth' data for any of the derived physical parameters. Each of the selected methods is based upon a different set of assumptions. The challenge in interpreting Pathfinder data is to determine what alterations in the physical attributes of the environment contribute to evidence of global change extracted from observed or derived Pathfinder datasets. The Pathfinder validation and intercomparison activity must include validation of the forward problem, by which one calculates an estimate of upwelling radiance information from given earth and atmospheric data, and validation of the inverse problem, by which one calculates estimates of earth and atmospheric information from given upwelling satellite spectral radiance data.

Source record↗
NASA NTRSDate not supplied

Assessment of Retrieved GMI Emissivity Over Land, Snow and Sea Ice in the GEOS System

Measurements from microwave sounders and imagers provide a valuable source of information including atmospheric temperature and water vapor in Numerical Weather Prediction (NWP) systems that assimilate these observations directly over water surfaces (oceans and other large water bodies). In a recent decadal survey, targeted observables in the Planetary Boundary Layer (PBL) were cited as a key need for future observations (NASEM, 2018). Microwave observations which sense in the PBL are currently available, however, utilizing surface-sensitive microwave observations for atmospheric data assimilation remains a challenge over land, snow and sea ice. This is in part due to the inability of surface emissivity models used by NWP data assimilation systems to simulate observations with sufficient accuracy. The GEOS-ADAS (Todling and el Akkraoui, 2018) which utilizes the Community Radiative Transfer Model (CRTM) (Han, 2006; Chen 2009) is no exception. The ECMWF system has retrieved instantaneous surface emissivity from surface-sensitive channels for SSMI/S and MHS radiance observations, and apply these estimates to the closest channels higher in frequency (Baordo and Geer 2016) in the calculation of simulated radiances. This approach currently is also being tested in the GEOS-ADAS for AMSU-A and ATMS radiances (Zhu et al. 2021). No or minimal emissivity spectral variability has been assumed in the above-mentioned studies. Recently, work by Munchak et al., 2020 (hereby referred to as M2020) provided a new database for emissivity over land, snow and sea ice retrieved from the NASA Global Precipitation Mission (GPM). Compared with Tool to Estimate Land Surface Emissivities at Microwave (TELSEM2; Wang et al., 2017), M2020 provides emissivities for more frequencies(i.e., 10.7 GHz V/H). Moreover, this database is unique in that it utilizes both active and passive data to retrieve surface emissivity and normalized radar cross section. While the emissivity values may be useful for other sensors, they are most applicable to the GPM Microwave Imager (GMI). In this work the GEOS-ADAS is modified to utilize emissivity values from Munchak et al, 2020 in place of values used by CRTM. Presently, only GMI radiances over ocean are used in the operational GEOS-ADAS. This study will focus on the GMI radiances over land, snow, and ice, as a first attempt to evaluate GMI radiances over these non-water surface types. Two cases are then presented, one with one week of observation minus background departures using the modified GEOS-ADAS, and one utilizing the original GEOS-ADAS. It should be noted that the surface emissivity models in CRTM are not state of the art and are scheduled to be replaced by the Community Surface Emissivity Module (CSEM; Chen and Weng, 2016). Simulations using default CRTM emissivity values are used merely as reference comparing against M2020, and is not a thorough comparison against other more state of the art modules such as CSEM.

GMI↗
NASA NTRS2002

CHAMP Tracking and Accelerometer Data Analysis Results

The CHAMP (Challenging Minisatellite Payload) mission's unique combination of sensors and orbit configuration will enable unprecedented improvements in modeling and understanding the Earth's static gravity field and its temporal variations. CHAMP is the first of two missions (GRACE (Gravity Recovery and Climate Experiment) to be launched in the later part of '01) that combine a new generation of GPS (Global Positioning System) receivers, a high precision three axis accelerometer, and star cameras for the precision attitude determination. In order to isolate the gravity signal for science investigations, it is necessary to perform a detailed reduction and analysis of the GPS and SLR tracking data in conjunction with the accelerometer and attitude data. Precision orbit determination based on the GPS and SLR (Satellite Laser Ranging) tracking data will isolate the orbit perturbations, while the accelerometer data will be used to distinguish the surface forces from those due to the geopotential (static, and time varying). In preparation for the CHAMP and GRACE missions, extensive modifications have been made to NASA/GSFC's GEODYN orbit determination software to enable the simultaneous reduction of spacecraft tracking (e.g. GPS and SLR), three axis accelerometer and precise attitude data. Several weeks of CHAMP tracking and accelerometer data have been analyzed and the results will be presented. Precision orbit determination analysis based on tracking data alone in addition to results based on the simultaneous reduction of tracking and accelerometer data will be discussed. Results from a calibration of the accelerometer will be presented along with the results from various orbit determination strategies. Gravity field modeling status and plans will be discussed.

Lemoine, Frank G.↗
NASA NTRS2018

Enabling Reanalysis Research Using the Collaborative Reanalysis Technical Environment (CREATE)

Modern atmospheric and oceanic reanalysis are valuable assets for atmospheric research and climate monitoring (Kalnay et al. 1996). Now that most reanalysis records are more than 36 years long, the data have become more useful for climate modeling research (Dole et al. 2008). For investigators who need to use multiple reanalysis, a common challenge is that the data are distributed at various sites and often in different formats. The NASA CREATE system provides access to the data in one location in a standard format (one variable per file and standardized metadata in the CMIP5 style; see Table 1 for a list of the key acronyms used in this paper). The collection includes monthly and 6-hourly data from the seven major atmospheric reanalysis: CFSR (Saha et al. 2010), ERA-Interim (Dee et al. 2011), MERRA (Rienecker et al. 2011), MERRA-2 (Gelaro et al. 2017), JRA-25 (Onogi et al. 2007), JRA-55 (Kobayashi et al. 2014), and 20CRv2c (Compo et al. 2011). An ancillary portion of CREATE includes eight ocean reanalysis: NCEP CFSR, CMCC C-GLORSv5 (Storto et al. 2016), ECMWF ORAS4 (Balmaseda et al.2013), ECMWF ORAP5.0 (Zuo et al. 2015), University of Hamburg GECCO2 (Köhl 2015), GFDL ECDA (Zhang et al. 2007), NOAA GODAS (Saha et al. 2010), and MOVE/MRI.COM-G2i (Toyoda et al. 2016). The ocean state variables were similarly reformatted but were then also regridded onto a common horizontal and vertical grid. This approach facilitated the calculation of an ensemble average and spread that is also published alongside the native gridded data. A third reanalysis product is a global hourly 0.5° land surface air temperature dataset constructed by Wang and Zeng (2013). All three datasets are distributed through the ESGF in a format consistent with the CMIP style described by Cinquini et al. (2014).

Potter, Gerald L.↗
NASA NTRS2019

Understanding Machine Learning in Earth Science: A Natural Language Processing Approach

Machine learning (ML) is being increasingly utilized in Earth science research. Benefits of ML include efficiency, reduction of human error, and ability to extract hidden patterns within data. However, the mutual lack of each other’s domain knowledge by ML and Earth science stands as a barrier to timely and effective implementation. Earth science, in particular, faces challenges in generating sample data, compared to those of traditional ML problems such as face recognition or stock predictions, where data is abundant and not lacking in ground truth, which is necessary for labeling. Earth science data are more varying in formats, such as HDF5 and image resolutions, and are not standardized across instruments, even within a given Earth science discipline. Previous studies have been done to outline the specific challenges that Earth science faces with ML, while others have focused on using existing publications to mine information efficiently. Other resources such as Scikit-Learn have developed decision trees for choosing appropriate machine learning algorithms, but application within Earth science subjects becomes much more complex. For the current study, we propose a methodology and tool that aids in implementation of ML in Earth science using natural language processing (NLP). Our work comprises three main parts: (1) analyzing existing publications related to ML and Earth science, using natural language processing: (2) extracting from the publications information on ML models subjects in Earth Science: and (3) visualizing the extracted relationships as a network graph. The resulting network graph should aid the Earth science communities in applying optimal ML algorithms and guiding data preparation through visualization of similar studies. The network graph and analysis of document similarity will be the basis of our next step, which is to develop a decision tree for selecting optimal machine learning methodologies for specified Earth science applications.

Zheng, Laura↗
NASA NTRS1978

Advanced marine information delivery

The paper examines the information delivery challenges of the follow-on programs to Seasat-1 as they may progress through the next decade. These challenges include coping with the vast amounts of data to be transferred, fulfilling the temporal requirements on data delivery, and the tradeoffs and developments needed to accomplish the various levels of processing required to convert sensor output into useful information. A need for critical development is clearly identifiable in the areas of (1) low cost ground terminals capable of image extraction and image correlation; (2) dynamic data assimilation to accommodate forecasters; (3) low resolution onboard correlators; and (4) low cost user advisory (display) terminals. The system planners for the Ocean Satellite advanced programs are using an end-to-end data systems approach in meeting these challenges.

Durstenfeld, R.↗
NASA NTRS2004

An Overview of Ecological Modeling and Machine Learning Research Within the U.S. National Aeronautics and Space Administration

In the early 1980 s NASA began research to understand global habitability and quantify the processes and fluxes between the Earth's vegetation and the biosphere. This effort evolved into the Earth Observing System Program which current encompasses 18 platforms and 80 sensors. During this time, the global environmental research community has evolved from a data poor to a data rich research area and is challenged to provide timely use of these new data. This talk will outline some of the data mining research NASA has funded in support for the environmental sciences in the Intelligent Systems project and will give a specific example in ecological forecasting, predicting the land surface properties given nowcasts and weather forecasts, using the Terrestrial Observation and Prediction System (TOPS).

Coughlan, Joseph C.↗
NASA NTRS2015

Development of the Science Data System for the International Space Station Cold Atom Lab

Cold Atom Laboratory (CAL) is a facility that will enable scientists to study ultra-cold quantum gases in a microgravity environment on the International Space Station (ISS) beginning in 2016. The primary science data for each experiment consists of two images taken in quick succession. The first image is of the trapped cold atoms and the second image is of the background. The two images are subtracted to obtain optical density. These raw Level 0 atom and background images are processed into the Level 1 optical density data product, and then into the Level 2 data products: atom number, Magneto-Optical Trap (MOT) lifetime, magnetic chip-trap atom lifetime, and condensate fraction. These products can also be used as diagnostics of the instrument health. With experiments being conducted for 8 hours every day, the amount of data being generated poses many technical challenges, such as downlinking and managing the required data volume. A parallel processing design is described, implemented, and benchmarked. In addition to optimizing the data pipeline, accuracy and speed in producing the Level 1 and 2 data products is key. Algorithms for feature recognition are explored, facilitating image cropping and accurate atom number calculations.

bose einstein condensate↗
NASA NTRS2017

Data Recovery Effort of Nimbus Era Observations by the NASA GES DISC

NASA launched seven Nimbus meteorological satellites in the 1960s and 70s. These satellites carried instruments for making observations of the Earth in the visible, infrared, ultraviolet, and microwave wavelengths. The original data archive consisted of a combination of magnetic tapes and various film media. As these media are well past their expected end of life, the valuable data they contain are now being migrated to the GES DISC modern online archive. The process involves recovering the digital data files from the tapes as well as scanning images of the data from film strips. This presentation will address the status and challenges of recovering the Nimbus data. The old data products were written on now obsolete hardware systems and outdated file formats. They lack any metadata standards and each product is often written in its own proprietary file structure. This requires creating metadata by reading the contents of the old data files. The job is tedious and laborious, as documentation may be incomplete, data files and tapes are sometimes corrupted, or were improperly copied at the time they were created.

Nimbus↗
NASA NTRS2014

Automating the SMAP Ground Data System to Support Lights-Out Operations

The Soil Moisture Active Passive (SMAP) Mission is a first tier mission in NASA's Earth Science Decadal Survey. SMAP will provide a global mapping of soil moisture and its freeze/thaw states. This mapping will be used to enhance the understanding of processes that link the terrestrial water, energy, and carbon cycles, and to enhance weather and forecast capabilities. NASA's Jet Propulsion Laboratory has been selected as the lead center for the development and operation of SMAP. The Jet Propulsion Laboratory (JPL) has an extensive history of successful deep space exploration. JPL missions have typically been large scale Class A missions with significant budget and staffing. SMAP represents a new area of JPL focus towards low cost Earth science missions. Success in this new area requires changes to the way that JPL has traditionally provided the Mission Operations System (MOS)/Ground Data System (GDS) functions. The operation of SMAP requires more routine operations activities and support for higher data rates and data volumes than have been achieved in the past. These activities must be addressed by a reduced operations team and support staff. To meet this challenge, the SMAP ground data system provides automation that will perform unattended operations, including automated commanding of the SMAP spacecraft.

GDS↗
NASA NTRSDate not supplied

Diving into the Data Plane of Delay Tolerant Networks

This presentation details early research towards applying data plane programming and software defined networking (SDN) to delay tolerant networking (DTN). This involved using the Programming Protocol-Independent Packet Processors (P4) language to program the data plane to handle DTN bundles natively in hardware. The results presented include a proof of concept DTN Bundle Protocol (BP) translator network middlebox, the challenges of programming a data plane to process BP, a software-defined DTN (SDDTN) architecture, and potential future applications of SDDTN. It is demonstrated that SDN and the P4 language has high potential for improving the scalability of DTN networks, although further research must be done to enable this.

Dominick Ta↗