Search NASA⌕ Search

SEARCH · Search NASA

Results for “data challenges”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
NASA NTRS2019

Federated Cloud Challenges in NASA's Earth Science Data Systems (Why So Difficult?)

NASA is presented with a number of opportunities and challenges in federating its Earth Science Data Systems in the burgeoning world of cloud computing. Cloud hosting of Earth Science data provides a new way of bringing data together, at least from a virtual location sense, and is one of the main motives for NASA to host data there. However, NASA is also faced with a Big Data Variety challenge, brought on by the variety of the EO datasets in its archives. This diversity requires many diverse science archives to service the different science communities. As a result, nearly every major function in the Earth Observing System Data and Information System (EOSDIS) must also be federated across its data centers. This pattern is repeated with many of the outside agencies and organizations that EOSDIS federates with, such as the Committee for Earth Observing Satellites, leading to pioneering work on "deep federation" in a joint project with the European Space Agency to develop a Multi-Mission Algorithm and Analysis Platform.

Lynnes, Christopher↗
NASA NTRS1991

Information management challenges of the EOS Data and Information System

An overview of the current information management concepts that are embodied in the plans for the Earth Observing System Data and Information System (EOSDIS) is presented, and some of the technology development and application areas that are envisioned to be particularly challenging are introduced. The Information Management System (IMS) is the EOSDIS element that provides the primary interface between the science users and the data products and services of EOSDIS. The goals of IMS are to define a clear and complete set of functional requirements and to apply innovative methods and technologies to satisfy them. The information management functions are described in detail, and some applicable technolgies are discussed. Some of the general issues affecting the successful development and operation of the information management element are addressed.

Mcdonald, Kenneth R.↗
NASA NTRS2010

Aerosol Observability and Predictability: From Research to Operations for Chemical Weather Forecasting. Lagrangian Displacement Ensembles for Aerosol Data Assimilation

A challenge common to many constituent data assimilation applications is the fact that one observes a much smaller fraction of the phase space that one wishes to estimate. For example, remotely sensed estimates of the column average concentrations are available, while one is faced with the problem of estimating 3D concentrations for initializing a prognostic model. This problem is exacerbated in the case of aerosols because the observable Aerosol Optical Depth (AOD) is not only a column integrated quantity, but it also sums over a large number of species (dust, sea-salt, carbonaceous and sulfate aerosols. An aerosol transport model when driven by high-resolution, state-of-the-art analysis of meteorological fields and realistic emissions can produce skillful forecasts even when no aerosol data is assimilated. The main task of aerosol data assimilation is to address the bias arising from inaccurate emissions, and Lagrangian misplacement of plumes induced by errors in the driving meteorological fields. As long as one decouples the meteorological and aerosol assimilation as we do here, the classic baroclinic growth of error is no longer the main order of business. We will describe an aerosol data assimilation scheme in which the analysis update step is conducted in observation space, using an adaptive maximum-likelihood scheme for estimating background errors in AOD space. This scheme includes e explicit sequential bias estimation as in Dee and da Silva. Unlikely existing aerosol data assimilation schemes we do not obtain analysis increments of the 3D concentrations by scaling the background profiles. Instead we explore the Lagrangian characteristics of the problem for generating local displacement ensembles. These high-resolution state-dependent ensembles are then used to parameterize the background errors and generate 3D aerosol increments. The algorithm has computational complexity running at a resolution of 1/4 degree, globally. We will present the result of assimilating AOD retrievals from MODIS (on both Aqua and TERRA satellites) from AERONET for validation. The impact on the GEOS-5 Aerosol Forecasting will be fully documented.

da Silva, Arlindo↗
NASA NTRS2015

Managing Large Datasets for Atmospheric Research

Since the mid-1980s, airborne and ground measurements have been widely used to provide comprehensive characterization of atmospheric composition and processes. Field campaigns have generated a wealth of insitu data and have grown considerably over the years in terms of both the number of measured parameters and the data volume. This can largely be attributed to the rapid advances in instrument development and computing power. The users of field data may face a number of challenges spanning data access, understanding, and proper use in scientific analysis. This tutorial is designed to provide an introduction to using data sets, with a focus on airborne measurements, for atmospheric research. The first part of the tutorial provides an overview of airborne measurements and data discovery. This will be followed by a discussion on the understanding of airborne data files. An actual data file will be used to illustrate how data are reported, including the use of data flags to indicate missing data and limits of detection. Retrieving information from the file header will be discussed, which is essential to properly interpreting the data. Field measurements are typically reported as a function of sampling time, but different instruments often have different sampling intervals. To create a combined data set, the data merge process (interpolation of all data to a common time base) will be discussed in terms of the algorithm, data merge products available from airborne studies, and their application in research. Statistical treatment of missing data and data flagged for limit of detection will also be covered in this section. These basic data processing techniques are applicable to both airborne and ground-based observational data sets. Finally, the recently developed Toolsets for Airborne Data (TAD) will be introduced. TAD (tad.larc.nasa.gov) is an airborne data portal offering tools to create user defined merged data products with the capability to provide descriptive statistics and the option to treat measurement uncertainty.

Chen, Gao↗
NASA NTRSDate not supplied

Modern Scientific Data Governance Framework

Science has entered the era of Big Data with new challenges related to data governance, stewardship, and management. The existing data governance practices must catch up to ensure proper data management. Existing data governance policies and stewardship best practices tend to be disconnected from operational data management practices and enforcement and mainly exist in well-meaning documents or reports. These governance policies are, at best, partially implemented and rarely monitored or audited. In addition, existing governance policies keep adding additional data management steps that require a human, ‘a data steward’, in the loop, and the cost of data management can no longer scale proportionately with the current and future increased data volume and complexity. The goal for developing an updated data governance framework is to modernize scientific data governance to the reality of Big data and align it with the current technology trends such as cloud computing and AI. The goals of this framework are two folds. One is to ensure thoroughness that the governance adequately covers the entire data life cycle. Two, provide a practical approach that offers a consistent and repeatable process for different projects. Three core principles ground this framework. First, focus on just enough governance and prevent data governance from becoming a roadblock toward the scientific process. Remove any unnecessary processes and steps. Second, automate data management steps where possible. Actively remove steps that require ‘human in the loop’ within the management process to be efficient and scale with increasing data. Third, all the processes should continually be optimized using quantified metrics to streamline the monitoring and auditing workflows.

Rahul Ramachandran↗
NASA NTRS2003

Multi-mission ground data systems - breakthroughs and challenges

Given the cost-constrained nature of JPL Flight Projects, especially Discovery Class missions, there is more and more pressure to reduce the costs associated with mission operations, particularly the costs for the Ground Data System. This paper explores the successes (and failures) of using the Mission Management Office (MMO) GDS team to provide a common set of services, tools, procedures, and products to JPL Flight Projects.

GDS↗
NASA NTRS2016

Big Data in the Earth Observing System Data and Information System

Approaches that are being pursued for the Earth Observing System Data and Information System (EOSDIS) data system to address the challenges of Big Data were presented to the NASA Big Data Task Force. Cloud prototypes are underway to tackle the volume challenge of Big Data. However, advances in computer hardware or cloud won't help (much) with variety. Rather, interoperability standards, conventions, and community engagement are the key to addressing variety.

data systems↗
NASA NTRS1987

Comments on the challenge of using mesoscale data in mesoscale numerical models

The dangers of addressing the initialization issues for limited-area mesoscale models by extending the lessons learned during the development of global analysis and prediction systems are discussed. Lack of impact with data inserts at one time suggests that the lateral boundary conditions imposed on the limited-area models might force the model simulation toward a preferred solution, work against the new data being inserted into the model and, therefore, limit the potential impact that this data can have on the model system. The second potential pitfall involves the imposition of balance constraints on the data that are being inserted into the model to compute winds from temperature data and/or temperature from wind data.

Uccellini, Louis W.↗
NASA NTRS2008

Challenges of AVHRR Vegetation Data for Real Time Applications

Remote sensing data has long been used to monitor global ecosystems for floods and droughts and AVHRR data, as one of the first product, has many users interested in receiving the data within hours of acquisition. With the introduction of a new series of sensors in 2000 (the AVHRR/3 series), the quality of the NDVI datasets available for real time environmental monitoring has declined. This paper provides evidence of problems of cloud contamination, calibration and noise in the real time data which are not present in the historical AVHRR NDVIg dataset. These differences introduce significant uncertainty in the use of the real time data, degrading their utility for detecting climate variations in near real time.

Brown, Molly↗
NASA NTRS2012

The Human Dimension of Closing the Training Gap for Fifth-Generation Fighters

Based on a review of the recent technical literature there is little question that a serious training gap exists for fifth-generation fighters, primarily arising from the need to provide their own red-air. There are several methods for reducing this gap, including injecting virtual and constructive threats into the live cockpit. This live-virtual-constructive (LVC) training approach provides a cost effective means for addressing training needs but faces several challenges. Technical challenges include data links and information assurance. A more serious challenge may be the human factors dimension of representing virtual and constructive entities in the cockpit while ensuring safety-of-flight. This also needs to happen without increasing pilot workload. This paper discusses the methods Rockwell Collins and the University of Iowa's Operator Performance Lab use to assess pilot workload and training fidelity measures in an LVC training environment and the research we are conducting in safety-of-flight requirements of integrated LVC symbology.

Hoke, Jaclyn↗
NASA NTRS2020

Accelerating Space Life Sciences: Successes and Challenges of Biospecimen and Data Sharing

NASA's current human space flight research is directed towards enabling human space exploration beyond Low Earth Orbit (LEO). To that end, NASA Space Flight Payload Projects; Rodent Research, Cell Science, and Microbial Labs, flown on the International Space Station (ISS), benefit the global life sciences and commercial space communities. Verified data sets, science results, peer-reviewed publications, and returned biospecimens, collected and analyzed for flight and ground investigations, are all part of the knowledge base collected by NASA's Human Exploration and Operations Mission Directorate's Space Life and Physical Sciences Research and Applications (SLPSRA) Division, specifically the Human Research and Space Biology Programs. These data and biospecimens are made available through the public Life Sciences Data Archive (LSDA) website to promote basic discovery, pre-clinical and clinical science.The NASA Institutional Scientific Collection (ISC), stores flight and ground biospecimens from Space Shuttle and ISS programs. These specimens are curated and managed by the Ames Life Sciences Data Archive (ALSDA), an internal node of NASA's LSDA. The ISC stores over 30,000 specimens from experiments dating from 1984 to present. Currently available specimens include tissues from the circulatory, digestive, endocrine, excretory, integumentary, muscular, neurosensory, reproductive, respiratory and skeletal systems.NASA's biospecimen collection represents a unique and limited resource of unique spaceflight payload and ground control research subjects. These specimens are harvested according to well established SOPs that maintain their quality and integrity. Once the primary scientific objectives have been met, the remaining specimens are made available to provide secondary opportunities for complementary studies or new investigations to broaden research without large expenditures of time or resources. Website: https://lsda.jsc.nasa.gov/

Scott, Ryan T.↗
NASA NTRS2023

Comparative Analysis of Empirical and Machine Learning Models for Chla Extraction Using Sentinel-2 and Landsat OLI Data: Opportunities, Limitations, and Challenges

Remote retrieval of near-surface chlorophyll-a (Chla) concentration in small inland waters is challenging due to substantial optical interferences of various water constituents and uncertainties in the atmospheric correction (AC) process. Although various algorithms have been developed to estimate Chla from moderate-resolution terrestrial missions (∼10–60 m), the production of both accurate distribution maps and time series of Chla has proven challenging, limiting the use of remote analyses for lake monitoring. Here, we develop a support vector regression (SVR) model, which uses satellite-derived remote-sensing reflectance spectra () from Sentinel-2 and Landsat-8 images as input for Chla retrieval in a representative eutrophic prairie lake, Buffalo Pound Lake (BPL), Saskatchewan, Canada. Validated against in situ Chla from seven ice-free seasons (N ∼ 200; 2014–2020), the SVR model outperformed both locally tuned, -fed empirical models (Normalized Difference Chlorophyll Index, 2- and 3-band, and OC3) and Mixture Density Networks (MDNs) by 15–65%, while exhibiting comparable performance to a locally trained MDN, with an error of ∼35%. Comparison of Chla retrieval models, AC processors (iCOR, ACOLITE), and radiometric products (Rayleigh-corrected, surface, and top-of-atmosphere reflectance) showed that the best Chla maps and optimal time series (up to 100 mg m−3) were produced using a coupled SVR-iCOR system.

algal blooms↗
NASA NTRS2018

NASA's Earth Observing Data and Information System - Near-Term Challenges

NASA's Earth Observing System Data and Information System (EOSDIS) has been a central component of the NASA Earth observation program since the 1990's. EOSDIS manages data covering a wide range of Earth science disciplines including cryosphere, land cover change, polar processes, field campaigns, ocean surface, digital elevation, atmosphere dynamics and composition, and inter-disciplinary research, and many others. One of the key components of EOSDIS is a set of twelve discipline-based Distributed Active Archive Centers (DAACs) distributed across the United States. Managed by NASA's Earth Science Data and Information System (ESDIS) Project at Goddard Space Flight Center, these DAACs serve over 3 million users globally. The ESDIS Project provides the infrastructure support for EOSDIS, which includes other components such as the Science Investigator-led Processing systems (SIPS), common metadata and metrics management systems, specialized network systems, standards management, and centralized support for use of commercial cloud capabilities. Given the long-term requirements, and the rapid pace of information technology and changing expectations of the user community, EOSDIS has evolved continually over the past three decades. However, many challenges remain. Challenges addressed in this paper include: growing volume and variety, achieving consistency across a diverse set of data producers, managing information about a large number of datasets, migration to a cloud computing environment, optimizing data discovery and access, incorporating user feedback from a diverse community, keeping metadata updated as data collections grow and age, and ensuring that all the content needed for understanding datasets by future users is identified and preserved.

Remote Sensing↗
NASA NTRS2001

Large Aperture Scanning Lidar Based on Holographic Optical Elements

Lidar remote sensing instruments can make a significant contribution to satisfying many of the required measurements of atmospheric and surface parameters for future spaceborne platforms, including topographic altimeters, atmospheric profiles of, wind, humidity, temperature, trace molecules, aerosols, and clouds. It is highly desirable to have wide measurement swaths for rapid coverage rather than just the narrow ribbon of data that is obtained with a nadir only observation. For most applications global coverage is required, and for wind measurements scanning or pointing is required in order to retrieve the full 3-D wind vector from multiple line-of-sight Doppler measurements. Conventional lidar receivers make up a substantial portion of the instrument's size and weight. Wide angle scanning typically requires a large scanning mirror in front of the receiver telescope, or pointing the entire telescope and aft optics assembly, Either of these methods entails the use of large bearings, motors, gearing and their associated electronics. Spaceborne instruments also need reaction wheels to counter the torque applied to the spacecraft by these motions. NASA has developed simplified conical scanning telescopes using Holographic Optical Elements (HOEs) to reduce the size, mass, angular momentum, and cost of scanning lidar systems. NASA has developed two operating lidar systems based on 40 cm diameter HOEs. The first such system, named Prototype Holographic Atmospheric Scanner for Environmental Remote Sensing (PHASERS) was a joint development between NASA Goddard Space Flight Center (GSFC) and the University of Maryland College Park. PHASERS is based on a reflection HOE for use at the doubled Nd:YAG laser wavelength of 532 nm and has recently undergone a number of design changes in a collaborative effort between GSFC and Saint Anselm College in New Hampshire. The next step was to develop IR transmission HOEs for use with the Nd:YAG fundamental in the Holographic Airborne Rotating Lidar Instrument Experiment (HARLIE). The HOE spins like a compact disk in a large ring ball bearing. In an aircraft the HOE faces down, looking out through a window at an angle of 45 degrees off-nadir. The HOE diffracts 85% of the incident 532 nm light into a 160 micron spot at a focal length of 1 meter. HARLIE is a field deployable lidar measuring aerosol, cloud, and boundary layer backscatter for atmospheric research. It has flown several times and is also used from a ground-based trailer in an upward-looking mode. The HOE generates a 45 degree conical scan pattern by rotating at speeds up to 30 rpm. Like PHASERS, the HOE in HARLIE serves both as the laser collimating lens as well as the receiver telescope primary optic. The telescope is coupled to the receiver package via fiber optic. The transmitter is a diode pumped Nd:YAG laser operating at 1064 nm, delivering 1 mJ pulses at a 5 KHz rep-rate. The receiver has a 200 microradian field-of-view and a 0.5 nm optical bandpass. The photon counting data system utilizes a single Geiger-mode silicon avalanche photodiode detector, This new technology has also presented us with new data visualization challenges as well as new measurement techniques. The backscatter data obtained from a stationary (i.e. ground-based) scanning HOE lidar is on the surface of a cone, which when viewed over many consecutive scans can reveal atmospheric motions on this surface over time as the atmosphere advects over the site. In a moving platform such as an airplane or satellite, the data from consecutive scans cover different areas under the flight path, revealing atmospheric structure in 3-dimensions. An example of a visualization of HARLIE ground-based data is presented, showing aerosol backscatter on a 90 degree conical surface generated from one 360 degree scan of the lidar during the HOLO-1 field campaign on the afternoon of 10 March 1999. Higher backscatter levels are rendered as lighter signal against a dark background. Breaking Kelvin-Helmholtz waves are evident on the north side of the scan at an altitude of 10-11 km. Time series of successive scans made at regular intervals render unique views of atmospheric motions, from which vertical profiles of atmospheric wind vectors can be obtained using a unique data analysis approach. Wind vectors obtained from the lidar were compared with co-located radiosonde wind profiles during an intensive operating period in September-October 2000 at the Atmospheric Radiation Measurement Program's Southern Great Plains Central Facility.

Schwemmer, Geary K.↗
NASA NTRS2010

Evolving the Technical Infrastructure of the Planetary Data System for the 21st Century

The Planetary Data System (PDS) was established in 1989 as a distributed system to assure scientific oversight. Initially the PDS followed guidelines recommended by the National Academies Committee on Data Management and Computation (CODMAC, 1982) and placed emphasis on archiving validated datasets. But overtime user demands, supported by increased computing capabilities and communication methods, have placed increasing demands on the PDS. The PDS must add additional services to better enable scientific analysis within distributed environments and to ensure that those services integrate with existing systems and data. To face these challenges the Planetary Data System (PDS) must modernize its architecture and technical implementation. The PDS 2010 project addresses these challenges. As part of this project, the PDS has three fundamental project goals that include: (1) Providing more efficient client delivery of data by data providers to the PDS (2) Enabling a stable, long-term usable planetary science data archive (3) Enabling services for the data consumer to find, access and use the data they require in contemporary data formats. In order to achieve these goals, the PDS 2010 project is upgrading both the technical infrastructure and the data standards to support increased efficiency in data delivery as well as usability of the PDS. Efforts are underway to interface with missions as early as possible and to streamline the preparation and delivery of data to the PDS. Likewise, the PDS is working to define and plan for data services that will help researchers to perform analysis in cost-constrained environments. This presentation will cover the PDS 2010 project including the goals, data standards and technical implementation plans that are underway within the Planetary Data System. It will discuss the plans for moving from the current system, version PDS 3, to version PDS 4.

Beebe, Reta F.↗
NASA NTRS2010

Examples L-Band Interference will be Presented and Discussed, as well as the Importance of L-Band Soil Moisture Observations

Examples of L-band interference will be presented and discussed, as well as the importance of L-band soil moisture observations, as part of this one-day GEOSS workshop XXXVII on "Data Quality and Radio Spectrum Allocation Impact on Earth Observations" will address the broad challenges of data quality and the impact of generating reliable information for decision makers who are Earth data users but not necessarily experts in the Earth observation field. GEO has initiated a data quality assessment task (DA-09-01a) and workshop users will review and debate the directions and challenges of this effort. Radio spectrum allocation is an element of data availability and data quality, and is also associated with a GEO task (AR-06-11). A recent U.S. National Research Council report on spectrum management will be addressed as part of the workshop. Key representatives from industry, academia, and government will provide invited talks on these and related issues that impact GEOSS implementation.

Kim, Edward↗