Search NASA⌕ Search

SEARCH · Search NASA

Results for “data challenges”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
NASA NTRS2009

Porting Gravitational Wave Signal Extraction to Parallel Virtual Machine (PVM)

Laser Interferometer Space Antenna (LISA) is a planned NASA-ESA mission to be launched around 2012. The Gravitational Wave detection is fundamentally the determination of frequency, source parameters, and waveform amplitude derived in a specific order from the interferometric time-series of the rotating LISA spacecrafts. The LISA Science Team has developed a Mock LISA Data Challenge intended to promote the testing of complicated nested search algorithms to detect the 100-1 millihertz frequency signals at amplitudes of 10E-21. However, it has become clear that, sequential search of the parameters is very time consuming and ultra-sensitive; hence, a new strategy has been developed. Parallelization of existing sequential search algorithms of Gravitational Wave signal identification consists of decomposing sequential search loops, beginning with outermost loops and working inward. In this process, the main challenge is to detect interdependencies among loops and partitioning the loops so as to preserve concurrency. Existing parallel programs are based upon either shared memory or distributed memory paradigms. In PVM, master and node programs are used to execute parallelization and process spawning. The PVM can handle process management and process addressing schemes using a virtual machine configuration. The task scheduling and the messaging and signaling can be implemented efficiently for the LISA Gravitational Wave search process using a master and 6 nodes. This approach is accomplished using a server that is available at NASA Ames Research Center, and has been dedicated to the LISA Data Challenge Competition. Historically, gravitational wave and source identification parameters have taken around 7 days in this dedicated single thread Linux based server. Using PVM approach, the parameter extraction problem can be reduced to within a day. The low frequency computation and a proxy signal-to-noise ratio are calculated in separate nodes that are controlled by the master using message and vector of data passing. The message passing among nodes follows a pattern of synchronous and asynchronous send-and-receive protocols. The communication model and the message buffers are allocated dynamically to address rapid search of gravitational wave source information in the Mock LISA data sets.

Thirumalainambi, Rajkumar↗
NASA NTRS2019

Overcoming the Challenges of Data Integration and Automation

Artificial Intelligence (AI) and machine learning (ML) are gaining increased attention as a way to leverage the world's data to solve problems. Although AI and ML offer much potential, there are often misconceptions about the application of such techniques. This panel discussion includes speakers from airlines and the research community who will present machine learning approaches they have developed on a variety of aviation data including digital flight data, safety reporting data, and traffic flow data. They will explain the purpose of the application, the data used, and the lessons learned in the development and deployment of their solutions. The panel discussion will focus on common pitfalls in developing and AI solution, the dangers of the current hype around AI, tips for gaining value from a machine learning solution, how to determine whether a machine learning approach is appropriate for a problem, and more.

Matthews, Bryan L.↗
NASA NTRSDate not supplied

Pyroscopegridding: Geo-Leo Aerosol Data Fusion (an Open-Source Package)

The retrieval of aerosol optical depths (AODs) from sun-synchronous polar orbiting satellites, such as MODISs, VIIRSs, OMI, TROPOMI, etc., has been widely adopted as a method for obtaining information regarding particulate matter (PM) and related atmospheric processes. However, the advent of recently launched geostationary satellites, such as GOES-16/17/18, Himawari-8/9, and Meteosat Third Generation (MTG), has led to an increased temporal resolution of AOD observations (order of 10 minutes), resulting in typically one or more images per hour during daylight hours, compared to the once-per-day observations obtained from LEO satellites. By integrating these observations, the diurnal cycle of global AOD can be characterized at local, regional, and global scales. The scientific community is still examining the novel data from geostationary satellite observations and evaluating methods for effectively merging these observations with differing spatial and temporal resolutions. This presents a significant ""Big Data"" challenge, encompassing not only data storage, but also data discoverability, accessibility, and migration within cloud computing environments. This study presents our attempts at fusing Level 2 aerosol data from six satellites, three of which are geostationary (GOES-16/17 and Himawari-8) and three of which are polar orbiting (TERRA/MODIS, AQUA/MODIS, and SNPP-VIIRS), using the Dark Target aerosol retrieval algorithm. The ability to fuse remote sensing products on demand into desired temporal and spatial domains empowers researchers and practitioners to more efficiently work with satellite and sensor data. It is our hope that through making our open-source package and accompanying functionality available, the scientific community will have improved access to aerosol data processing resources.

Jennifer Wei↗
NASA NTRSDate not supplied

Breaking Barriers: Integrating Geo-Leo Aerosol Data with an Open-Source Approach

The scientific community is still examining the novel data from geostationary satellite observations and evaluating methods for effectively fusing the polar observations with various spatial and temporal resolutions. However, the merged data will present a significant ""Big Data"" challenge, including processing, storage, data discoverability, accessibility, and migration within cloud computing environments. We have developed an open-source package to fuse aerosol optical depths (AOD) products from six satellite sensors in the past four years (2019~2023), and this presentation will update our recent progress. Using this Python-based package, we produced a level 3 global (AOD) product in a quarter-degree spatial resolution every half-hour, fusing the Level 2 AOD data with the Dark Target aerosol retrieval algorithm from six satellites: three geostationary (GOES-16/17 and Himawari-8) with high temporal resolution, and three polar orbiting (TERRA/MODIS, AQUA/MODIS, and SNPP-VIIRS) with global coverage. By integrating these observations, the diurnal cycle of global AOD in this fused product can be characterized at local, regional, and global scales. Furthermore, we are committed to openness and transparency by providing our package and its associated functionalities as open-source. Our dedication to adhering to the FAIR, CARE, and TRUST principles ensures that our users can rely on the integrity and ethical standards of our work. For instance of Interoperability, this package fuses remote sensing products on demand into desired temporal and spatial domains. It can be run in a central processing unit (CPU) or a Graphics processing unit (GPU) mode. This package will empower researchers and practitioners to use satellite and sensor data efficiently in various applications and research.

Xiaohua Pan↗
NASA NTRS2008

Damage Propagation Modeling for Aircraft Engine Prognostics

This paper describes how damage propagation can be modeled within the modules of aircraft gas turbine engines. To that end, response surfaces of all sensors are generated via a thermo-dynamical simulation model for the engine as a function of variations of flow and efficiency of the modules of interest. An exponential rate of change for flow and efficiency loss was imposed for each data set, starting at a randomly chosen initial deterioration set point. The rate of change of the flow and efficiency denotes an otherwise unspecified fault with increasingly worsening effect. The rates of change of the faults were constrained to an upper threshold but were otherwise chosen randomly. Damage propagation was allowed to continue until a failure criterion was reached. A health index was defined as the minimum of several superimposed operational margins at any given time instant and the failure criterion is reached when health index reaches zero. Output of the model was the time series (cycles) of sensed measurements typically available from aircraft gas turbine engines. The data generated were used as challenge data for the Prognostics and Health Management (PHM) data competition at PHM 08.

Saxena, Abhinav↗
NASA NTRS2016

Ontologies for Aviation Data Management

Managing complex aviation data can be a significant challenge for any enterprise – whether a government agency, airline, airframe manufacturer, or aviation service provider. To handle this challenge, data models are typically developed to characterize and manage the data generated, used, and stored by a given enterprise. Unfortunately, different data providers employ qualitatively different data models, and this gives rise to problems exchanging data across organizational boundaries. Over the past decade, these problems have motivated data producers and consumers to look toward standardized data exchange models to address data interoperability. In this paper we examine some of these standardized data exchange models and compare them with a new type of data model based on ontologies. Ontology models have emerged in recent years from a confluence of research in the artificial intelligence, semantic web, and information science communities. This paper introduces ontology models, provides several use cases for ontologies relevant to aviation data management, and summarizes state of the art aviation prototype applications that utilize ontologies.

artificial intelligence↗
NASA NTRS2019

NASA GES DISC Giovanni: Current and Future

Giovanni (Geospatial Interactive Online Visualization and Analysis Infrastructure), developed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), has established a reputation among NASA users for easy access, analysis, and visualization of NASA Earth science data. Currently, Giovanni supports over 1900 variables in eight disciplinary areas. Like any other enterprise application, Giovanni faces big data challenges, such as servicing increasingly large data volumes and more complex data types, while at the same time addressing the demands of a more diverse user community, e.g., placing requests for long-term time series from multiple spatially and temporally dense data records. I will present how Giovanni has been evolving from an on-premises, monolithic software application towards a cloud-enabled implementation to address these challenges.

Analytics↗
NASA NTRS2019

Cloud Giovanni: Reining in Costs and Improving Performance with Analytical Data Stores Using Scalable Serverless Architecture

Giovanni is the Geospatial Interactive Online Visualization ANd aNalysis Infrastructure developed at NASA GES DISC which provides a simple and intuitive way to visualize, analyze, and access vast amounts of Earth science data. It receives large number of user requests each day for a variety of analysis and visualization services, which leads to the big data challenge of serving gradually increasing large data volumes with diverse statistical algorithms. We hereby propose a multi-dimensional accumulation method which provides fast and cost-efficient cloud analysis for diverse services including both area averaging and time averaging. This method involves the weighted volume integration over multiple variable dimensions (time and space), and is implemented in AWS using Athena providing serverless and highly scalable data analysis. Compared to the standard method, this approach dramatically reduced the computational time by order of magnitude with a minimal AWS cost incurred. For example, for a benchmark of 10-year area averaging over the 1x1 degree daily variable, the computational time was reduced from minutes to seconds, and the Athena cost is only $5 for 100,000 requests.

Zhang, Hailiang↗
NASA NTRS2005

The Challenges of Interoperable Data Discovery

The Global Change Master Directory (GCMD) assists the oceanographic community in data discovery and access through its online metadata directory. The directory also offers data holders a means to post and search their oceanographic data through the GCMD portals, i.e. online customized subset metadata directories. The Gulf of Maine Ocean Data Partnership (GoMODP) has expressed interest in using the GCMD portals to increase the visibility of their data holding throughout the Gulf of Maine region and beyond. The purpose of the Gulf of Maine Ocean Data Partnership (GoMODP) is to "promote and coordinate the sharing, linking, electronic dissemination, and use of data on the Gulf of Maine region". The participants have decided that a "coordinated effort is needed to enable users throughout the Gulf of Maine region and beyond to discover and put to use the vast and growing quantities of data in their respective databases". GoMODP members have invited the GCMD to discuss further collaborations in view of this effort. This presentation. will focus on the GCMD GoMODP Portal - demonstrating its content and use for data discovery, and will discuss the challenges of interoperable data discovery. interoperability among metadata standards and vocabularies will be discussed. A short overview of the lessons learned at the Marine Metadata Interoperability (MMI) metadata workshop held in Boulder, Colorado on August 9-11, 2005 will be given.

Meaux, Melanie F.↗
NASA NTRS2006

Zero to Integration in Eight Months, the Dawn Ground Data System Engineering Challenge

The Dawn Project has presented the Ground Data System (GDS) with technical challenges driven by cost and schedule constraints commonly associated with National Aeronautics and Space Administration (NASA) Discovery Projects. The Dawn mission consists of a new and exciting Deep Space partnership among: the Jet Propulsion Laboratory (JPL), manages the project and is responsible for flight operation; Orbital Sciences Corporation (OSC), is the spacecraft builder and is responsible for flight system test and integration; and the University of California, at Los Angeles (UCLA), is responsible for science planning and operations. As a cost-capped mission, one of Dawn's implementation strategies is to leverage from both flight and ground heritage. OSC's ground data system is used for flight system test and integration as part of the flight heritage strategy. Mission operations, however, are to be conducted with JPL's ground system. The system engineering challenge of dealing with two heterogeneous ground systems emerged immediately. During the first technical interchange meeting between the JPL's GDS Team and OSC's Flight Software Team, August 2003, the need to integrate the ground system with the flight software was brought to the table. This need was driven by the project's commitment to enable instrument engineering model integration in a spacecraft simulator environment, for both demonstration and risk mitigation purposes, by April 2004. This paper will describe the system engineering approach that was undertaken by JPL's GDS Team in order to meet the technical challenge within a non-negotiable eight-month schedule. Key to the success was adherence to fundamental systems engineering practices: decomposition of the project request into manageable requirements; integration of multiple ground disciplines and experts into a focused team effort; definition of a structured yet flexible development process; definition of an in-process risk reduction plan; and aggregation of the intermediate products to an integrated final product. In addition, this paper will highlight the role of lessons learned from the integration experience. The lessons learned from an early GDS deployment have served as the foundation for the design and implementation of the Dawn Ground Data System.

Ground Data System (GDS)↗
NASA NTRS2024

End-to-End Mission Design & Trajectory Optimization

Need: A need exists for a generalized, robust, user-friendly and accessible end-to-end mission design optimization tool. Solution: Our solution to developing this capability was to interface two JSC tools—Copernicus and Genesis. Each of these tools has a specific area of the mission design process that it excels at. By utilizing them both, we can gain performance benefits not seen by either on their own. - Copernicus is a trajectory design and optimization software used for in-space trajectories around multiple bodies. - Genesis is a flight mechanics tool used to model ascent, entry, descent, and landing trajectories around a single planetary body. Year 1 was focused on combining these 2 software packages—allowing Copernicus to incorporate the ascent/descent capabilities of Genesis into the optimization problem—and developing this end-to-end mission design capability. Year 2 we focused on increasing the robustness of this capability by building the initial guess generator (IGG), which produces initial guesses based on simplifying assumptions and the physics of the problem. Year 3 of our project focused on utilizing the end-to-end mission design and optimization capabilities developed in the previous 2 years to analyze specific mission scenarios—scaling up from proof of concept to real analyses—capturing any resulting performance benefits, as well as addressing the Big Data challenges we’re faced with—namely, how we’re going to manage and interpret all the data that’s generated.

Kristin Nichols↗
NASA NTRS2012

Global Characterization and Monitoring of Forest Cover Using Landsat Data: Opportunities and Challenges

The compilation of global Landsat data-sets and the ever-lowering costs of computing now make it feasible to monitor the Earth's land cover at Landsat resolutions of 30 m. In this article, we describe the methods to create global products of forest cover and cover change at Landsat resolutions. Nevertheless, there are many challenges in ensuring the creation of high-quality products. And we propose various ways in which the challenges can be overcome. Among the challenges are the need for atmospheric correction, incorrect calibration coefficients in some of the data-sets, the different phenologies between compilations, the need for terrain correction, the lack of consistent reference data for training and accuracy assessment, and the need for highly automated characterization and change detection. We propose and evaluate the creation and use of surface reflectance products, improved selection of scenes to reduce phenological differences, terrain illumination correction, automated training selection, and the use of information extraction procedures robust to errors in training data along with several other issues. At several stages we use Moderate Resolution Spectroradiometer data and products to assist our analysis. A global working prototype product of forest cover and forest cover change is included.

Global↗
NASA NTRS1996

Climate Data Assimilation on a Massively Parallel Supercomputer

We have designed and implemented a set of highly efficient and highly scalable algorithms for an unstructured computational package, the PSAS data assimilation package, as demonstrated by detailed performance analysis of systematic runs on up to 512-nodes of an Intel Paragon. The preconditioned Conjugate Gradient solver achieves a sustained 18 Gflops performance. Consequently, we achieve an unprecedented 100-fold reduction in time to solution on the Intel Paragon over a single head of a Cray C90. This not only exceeds the daily performance requirement of the Data Assimilation Office at NASA's Goddard Space Flight Center, but also makes it possible to explore much larger and challenging data assimilation problems which are unthinkable on a traditional computer platform such as the Cray C90.

supercomputer↗
NASA NTRS2014

Climate Analytics as a Service

Climate science is a big data domain that is experiencing unprecedented growth. In our efforts to address the big data challenges of climate science, we are moving toward a notion of Climate Analytics-as-a-Service (CAaaS). CAaaS combines high-performance computing and data-proximal analytics with scalable data management, cloud computing virtualization, the notion of adaptive analytics, and a domain-harmonized API to improve the accessibility and usability of large collections of climate data. MERRA Analytic Services (MERRA/AS) provides an example of CAaaS. MERRA/AS enables MapReduce analytics over NASA's Modern-Era Retrospective Analysis for Research and Applications (MERRA) data collection. The MERRA reanalysis integrates observational data with numerical models to produce a global temporally and spatially consistent synthesis of key climate variables. The effectiveness of MERRA/AS has been demonstrated in several applications. In our experience, CAaaS is providing the agility required to meet our customers' increasing and changing data management and data analysis needs.

big data↗