Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Challenge”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 901 records · Page 50

Weather Prediction Improvement Using Advanced Satellite Technology

We discuss in this paper some of the problems that exist today in the fall utilization of satellite data to improve weather forecasts and we propose specific recommendations to solve them. This discussion can be viewed as an aspect of the general debate on how best to organize the transition from research to operational satellites and how to evaluate the impact of a research instrument on numerical weather predictions. A method for providing this transition is offered by the National Polar-Orbiting Operational Environmental Satellite System (NPOESS) Preparatory Project (NPP). This mission will bridge the time between the present NOAA and Department of Defense (DOD) polar orbiting missions and the initiation of the converged NPOESS series and will evaluate some of the Earth Observing System (EOS) instruments as appropriate for operational missions. Thus, this mission can be viewed as an effort to meet the operational requirements of NOAA and DOD and the research requirements of NASA. More generally, however, it can be said that the process of going from the conception of new, more advanced instruments to their operational implementation and full utilization by the weather forecast communities is not optimal. Instruments developed for research purposes may have insufficient funding to explore their potential operational capabilities. Furthermore, instrument development programs designed for operational satellites typically have insufficient funding for assimilation algorithms needed to transform the satellite observations into data that can be used by sophisticated global weather forecast models. As a result, years often go by before satellite data are efficiently used for operational forecasts. NASA and NOAA each have unique expertise in the design of satellite instruments, their use for basic and applied research and their utilization in weather and climate research. At a time of limited resources, the two agencies must combine their efforts to work toward common goals of full utilization of satellite data. This is a challenge that requires the assimilation of myriad new data into increasingly sophisticated numerical forecast models that run on increasingly sophisticated computer systems. In section II, we briefly outline the impact of satellite data on the quality of the National Centers for Environmental Prediction (NCEP) forecasts. In section III, we describe the present status of the utilization of satellite data in NCEP models and the challenges that lie ahead. In section IV, we propose solutions whose goals are summarized in section V.

Einaudi, Franco↗

A Computational Approach for Model Update of an LS-DYNA Energy Absorbing Cell

NASA and its contractors are working on structural concepts for absorbing impact energy of aerospace vehicles. Recently, concepts in the form of multi-cell honeycomb-like structures designed to crush under load have been investigated for both space and aeronautics applications. Efforts to understand these concepts are progressing from tests of individual cells to tests of systems with hundreds of cells. Because of fabrication irregularities, geometry irregularities, and material properties uncertainties, the problem of reconciling analytical models, in particular LS-DYNA models, with experimental data is a challenge. A first look at the correlation results between single cell load/deflection data with LS-DYNA predictions showed problems which prompted additional work in this area. This paper describes a computational approach that uses analysis of variance, deterministic sampling techniques, response surface modeling, and genetic optimization to reconcile test with analysis results. Analysis of variance provides a screening technique for selection of critical parameters used when reconciling test with analysis. In this study, complete ignorance of the parameter distribution is assumed and, therefore, the value of any parameter within the range that is computed using the optimization procedure is considered to be equally likely. Mean values from tests are matched against LS-DYNA solutions by minimizing the square error using a genetic optimization. The paper presents the computational methodology along with results obtained using this approach.

Horta, Lucas G.↗

Web Coverage Service Challenges for NASA's Earth Science Data

In an effort to ensure that data in NASA's Earth Observing System Data and Information System (EOSDIS) is available to a wide variety of users through the tools of their choice, NASA continues to focus on exposing data and services using standards based protocols. Specifically, this work has focused recently on the Web Coverage Service (WCS). Experience has been gained in data delivery via GetCoverage requests, starting out with WCS v1.1.1. The pros and cons of both the version itself and different implementation approaches will be shared during this session. Additionally, due to limitations with WCS v1.1.1 ability to work with NASA's Earth science data, this session will also discuss the benefit of migrating to WCS 2.0.1 with EO-x to enrich this capability to meet a wide range of anticipated user's needs This will enable subsetting and various types of data transformations to be performed on a variety of EOS data sets.

Web Coverage Services↗

Solar Magnetic Activity and Solar-Stellar Connections

Prediction of solar magnetic activity on various temporal scales is a fundamental element of space weather, which requires a wide range of theoretical and observational expertise in solar phenomena from the deep interior to the corona. Historical observations have revealed many features of cyclic variations of the solar activity; but these data are dramatically insufficient to draw a physical picture of global magnetic field evolution. New observational data, currently available from space missions and ground-based observatories, provide us with detailed information about solar dynamics and magnetism. However, because of the relatively short duration of data series and the great variety of data types and quality, it is challenging to assimilate these data in theoretical models and make reliable forecasts. The recent unexpectedly weak solar activity cycles, as well as observations of rotational and magnetic topology transitions in solar-type stars, suggest that the Sun and its magnetic dynamo are currently in a very interesting evolutionary stage. This could relate to the difficulty in getting a model of the Sun to produce solar-like – rather than anti-solar-like – differential rotation, to reproduce the rotation profile obtained from helioseismology, and to predict solar activity cycles.

magnetic↗

Computing contingency statistics in parallel : design trade-offs and limiting cases.

Statistical analysis is typically used to reduce the dimensionality of and infer meaning from data. A key challenge of any statistical analysis package aimed at large-scale, distributed data is to address the orthogonal issues of parallel scalability and numerical stability. Many statistical techniques, e.g., descriptive statistics or principal component analysis, are based on moments and co-moments and, using robust online update formulas, can be computed in an embarrassingly parallel manner, amenable to a map-reduce style implementation. In this paper we focus on contingency tables, through which numerous derived statistics such as joint and marginal probability, point-wise mutual information, information entropy, and X{sup 2} independence statistics can be directly obtained. However, contingency tables can become large as data size increases, requiring a correspondingly large amount of communication between processors. This potential increase in communication prevents optimal parallel speedup and is the main difference with moment-based statistics (which we discussed in [1]) where the amount of inter-processor communication is independent of data size. Here we present the design trade-offs which we made to implement the computation of contingency tables in parallel.We also study the parallel speedup and scalability properties of our open source implementation. In particular, we observe optimal speed-up and scalability when the contingency statistics are used in their appropriate context, namely, when the data input is not quasi-diffuse.

99 GENERAL AND MISCELLANEOUS↗

Computing contingency statistics in parallel : design trade-offs and limiting cases.

Statistical analysis is typically used to reduce the dimensionality of and infer meaning from data. A key challenge of any statistical analysis package aimed at large-scale, distributed data is to address the orthogonal issues of parallel scalability and numerical stability. Many statistical techniques, e.g., descriptive statistics or principal component analysis, are based on moments and co-moments and, using robust online update formulas, can be computed in an embarrassingly parallel manner, amenable to a map-reduce style implementation. In this paper we focus on contingency tables, through which numerous derived statistics such as joint and marginal probability, point-wise mutual information, information entropy, and {chi}{sup 2} independence statistics can be directly obtained. However, contingency tables can become large as data size increases, requiring a correspondingly large amount of communication between processors. This potential increase in communication prevents optimal parallel speedup and is the main difference with moment-based statistics (which we discussed in [1]) where the amount of inter-processor communication is independent of data size. Here we present the design trade-offs which we made to implement the computation of contingency tables in parallel. We also study the parallel speedup and scalability properties of our open source implementation. In particular, we observe optimal speed-up and scalability when the contingency statistics are used in their appropriate context, namely, when the data input is not quasi-diffuse.

99 GENERAL AND MISCELLANEOUS↗

Computing contingency statistics in parallel.

Statistical analysis is typically used to reduce the dimensionality of and infer meaning from data. A key challenge of any statistical analysis package aimed at large-scale, distributed data is to address the orthogonal issues of parallel scalability and numerical stability. Many statistical techniques, e.g., descriptive statistics or principal component analysis, are based on moments and co-moments and, using robust online update formulas, can be computed in an embarrassingly parallel manner, amenable to a map-reduce style implementation. In this paper we focus on contingency tables, through which numerous derived statistics such as joint and marginal probability, point-wise mutual information, information entropy, and {chi}{sup 2} independence statistics can be directly obtained. However, contingency tables can become large as data size increases, requiring a correspondingly large amount of communication between processors. This potential increase in communication prevents optimal parallel speedup and is the main difference with moment-based statistics where the amount of inter-processor communication is independent of data size. Here we present the design trade-offs which we made to implement the computation of contingency tables in parallel.We also study the parallel speedup and scalability properties of our open source implementation. In particular, we observe optimal speed-up and scalability when the contingency statistics are used in their appropriate context, namely, when the data input is not quasi-diffuse.

99 GENERAL AND MISCELLANEOUS↗

Big Data Processing for Power Grid Event Detection

In this paper we present the application of big data processing for the development of machine learning (ML) models to detect relevant events in power grid operations. This is based on almost 20TB of phasor measurement unit data corresponding to up to two years of operation of three grid interconnections which provide power to most of the United States. A significant aspect of the work consists in having all data processing performed on a single standard GPU server, from pre-processing to ML model training and testing. We describe the data and computational infrastructure, challenges faced and methods used in data processing, main findings and results. The ML approach employed for best utilization of the big data is also discussed, including sample results.

Paes Leao, Bruno↗

Sensor enabled data-driven predictive analytics for modeling and control with high penetration of DERs in distribution systems

The electric power grid is undergoing a tremendous transformation due to the increasing penetration of renewable energy resources beginning with wind and more recently with the distributed energy resources (DERs) such as solar and battery storage. DERs have dramatically changed the role of the distribution systems in the overall power grid, and they are expected to contribute a significant portion of power generation in the future. If current trends for DERs continue, system operation and control will need to change dramatically for improved grid reliability and resiliency. As renewable resources increase in penetration, new and challenging operational, planning, and design problems are expected to emerge. Some of the key challenges that arise in the planning and operation of the future grid are: 1) Quantifying the impact of high DER penetration in distribution systems on bulk grid behavior over multiple time scales. 2) Identifying whether a particular DER configuration/settings have a large impact on the overall grid behavior. These challenges can be addressed in an offline manner using detailed T&D grid models and they can also be addressed in an online manner using sensor measurements. In particular, the advancement and planned growth in sensor technology in power grid over various voltage levels provide us with a unique opportunity to tackle these challenges from a data analytic perspective without needing detailed T&D grid models. A few questions that naturally arise when addressing the challenges from DERs using sensor data are: 1) How can we use limited sensor measurements to monitor & control voltage stability and small signal stability of the bulk system? 2) How can we ensure that the developed data analytic methods are robust to data availability and quality issues? 3) How can we compute the developed analytics in a scalable manner using streaming measurements? In this project, we addressed the aforementioned challenges arising from DERs and answered the questions raised above on how to effectively use the sensor measurements to enhance the reliability and performance of the electric grid. Thus, the overarching goal of this project is to develop effective reduced/representative system models from data that make the computational complexity sufficiently manageable so as to be useful to simulate, analyze, and even control complex non-linear power systems dynamics with large penetrations of DERs. In order to achieve the objective, the project team established a four-fold technical approach 1) Formulated a combined transmission-distribution co-simulation framework for data generation and validation, 2) Derived reduced/representative models of power systems based on data-driven methods for efficient computation and appropriate representation of system behavior, 3) Developed data driven characterization of power system behavior based on transfer operator theory, machine learning and optimization for model estimation, 4) Incorporated a scalable data management and processing architecture using distributed Kafka streaming applications that coordinate input data streams to the developed data analytics. The key accomplishments of the project are: 1) Development of a scalable multi-timescale T&D co-simulation framework (both for steady state and for dynamic co-simulation) using commercial solvers (PSSE and GridLAB-D). The steady-state T&D co-simulation interface is shared with our industry partner (PJM). 2) A structured reduced order dynamic model of distribution systems that can represent partial motor stalling along with a systematic procedure to derive the model parameters. 3) A PMU based online method to monitor, localize and mitigate fault-induced delayed voltage recovery using DER reactive support and load control in distribution systems. 4) Development of linear operator based robust methodologies for dynamic state estimation, uncertainty quantification, system identification and trajectory prediction for power system dynamics. 5) An adaptive damping control for utilizing wind energy resources to provide oscillation damping and system stability. 6) Implementation of Kafka-based framework for efficient processing of streaming data using Linux-based local virtual environment.

DER integration↗

Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows, In: SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

As system scales and application complexity grow, managing and processing simulation data has become a significant challenge. While recent approaches based on data staging and in-situ/in-transit data processing are promising, dynamic data volumes and distributions, such as those occurring in AMR-based simulations, make the efficient use of these techniques challenging. In this paper we propose cross-layer adaptations that address these challenges and respond at runtime to dynamic data management requirements. Specifically we explore (1) adaptations of the spatial resolution at which the data is processed, (2) dynamic placement and scheduling of data processing kernels, and (3) dynamic allocation of in-transit resources. We also exploit coordinated approaches that dynamically combine these adaptations at the different layers. We evaluate the performance of our adaptive cross-layer management approach on the Intrepid IBM-BlueGene/P and Titan Cray-XK7 systems using Chombo-based AMR applications, and demonstrate its effectiveness in improving overall time-to-solution and increasing resource efficiency.

Computer Science↗

A scalable messaging system for accelerating discovery from large scale scientific simulations

Emerging scientific and engineering simulations running at scale on leadership-class High End Computing (HEC) environments are producing large volumes of data, which has to be transported and analyzed before any insights can result from these simulations. The complexity and cost (in terms of time and energy) associated with managing and analyzing this data have become significant challenges, and are limiting the impact of these simulations. Recently, data-staging approaches along with in-situ and in-transit analytics have been proposed to address these challenges by offloading I/O and/or moving data processing closer to the data. However, scientists continue to be overwhelmed by the large data volumes and data rates. In this paper we address this latter challenge. Specifically, we propose a highly scalable and low-overhead associative messaging framework that runs on the data staging resources within the HEC platform, and builds on the staging-based online in-situ/in- transit analytics to provide publish/subscribe/notification-type messaging patterns to the scientist. Rather than having to ingest and inspect the data volumes, this messaging system allows scientists to (1) dynamically subscribe to data events of interest, e.g., simple data values or a complex function or simple reduction (max()/min()/avg()) of the data values in a certain region of the application domain is greater/less than a threshold value, or certain spatial/temporal data features or data patterns are detected; (2) define customized in-situ/in-transit actions that are triggered based on the events, such as data visualization or transformation; and (3) get notified when these events occur. The key contribution of this paper is a design and implementation that can support such a messaging abstraction at scale on high- end computing (HEC) systems with minimal overheads. We have implemented and deployed the messaging system on the Jaguar Cray XK6 machines at Oak Ridge National Laboratory and the Lonestar system at the Texas Advanced Computing Center (TACC), and we present the experimental performance evaluation using these HEC platforms in the paper.

Jin, Tong↗

Flexible Data Fusion for Air Quality Estimation and Forecasting in Google Earth Engine to support Global Health Management Needs

The assessment and forecasting of air quality around the world at high spatial and temporal resolution can be enhanced by integrating data from multiple sources including models, satellites, regulatory monitors, and low-cost sensors. Such integration is subject to numerous technical challenges, however, including heterogeneous data resolution and formatting, different levels of data availability and reliability, and computational and capacity challenges to developing data fusion tools and platforms. This presentation will provide an overview of a NASA-funded effort to develop a data fusion system within the Google Earth Engine platform which integrates these air quality data sources to produce comprehensive assessments and forecasts of key air pollutants at sub-daily and sub-city scales. The system is being developed in collaboration with city- and regional-level air quality managers, and will provide them with information to the assess and anticipate the health impacts of poor air quality, track local changes in air quality due to ongoing transportation and land use changes, and identify potential gaps in their current air quality monitoring strategies. The presentation will report advances achieved through the project, including bringing local air quality monitoring data into Google Earth Engine, quantifying uncertainties in air quality estimates and forecasts, and tailored communications tools providing integration into end-user processes to meet their needs.

Carl Malings↗

Flexible Data Fusion for Air Quality Estimation and Forecasting in Google Earth Engine to support Global Health Management Needs

The assessment and forecasting of air quality around the world at high spatial and temporal resolution can be enhanced by integrating data from multiple sources including models, satellites, regulatory monitors, and low-cost sensors. Such integration is subject to numerous technical challenges, however, including heterogeneous data resolution and formatting, different levels of data availability and reliability, and computational and capacity challenges to developing data fusion tools and platforms. This presentation will provide an overview of a NASA-funded effort to develop a data fusion system within the Google Earth Engine platform which integrates these air quality data sources to produce comprehensive assessments and forecasts of key air pollutants at sub-daily and sub-city scales. The system is being developed in collaboration with city- and regional-level air quality managers and will provide them with information to the assess and anticipate the health impacts of poor air quality, track local changes in air quality due to ongoing transportation and land use changes, and identify potential gaps in their current air quality monitoring strategies. The presentation will report advances achieved through the project, including bringing local air quality monitoring data into Google Earth Engine, quantifying uncertainties in air quality estimates and forecasts, and tailored communications tools providing integration into end-user processes to meet their needs.

Nathan R. Pavlovic↗