Search NASA⌕ Search

SEARCH · Search NASA

Results for “Scientific Data Analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
NASA NTRS2002

The Intelligent Data Understanding Element of NASA's Intelligent Systems Program

Within the NASA Intelligent Systems Program, the Intelligent Data Understanding (IDU) element develops techniques for transforming data into scientific understanding. Automating such tools is critical for space science, space-based earth science, and planetary exploration with onboard scientific data analysis. Intelligent data understanding (IDU) is about extracting meaning from large, diverse science and engineering databases, via autonomous techniques that transform very large datasets into understanding. The earth science community in particular needs new tools for analyzing multi-formatted and geographically distributed datasets and for identifying cause-effect relationships in the complex data. Research within the IDU program element seeks to automate data analysis tasks so that humans can focus on creative hypothesis generation and knowledge synthesis. It may also enable NASA space missions in which autonomous agents must generate knowledge and take actions, and missions where limited bandwidth permits transmission of only the most interesting scientific observations, summaries, and conclusions. Twenty-seven research projects are-currently funded.

Coughlan, Joseph C.↗
DOE OSTI2024

Probabilistic data fusion and physics-informed machine learning: A new paradigm for modeling under uncertainty, and its application to accelerating the discovery of new materials

In this report we summarize the work conducted by PI Perdikaris and his group under this Early Career project DE–SC0019116 during the period of 09/01/2018 – 08/31/2023. The central aim of the work was to introduce a new paradigm for scientific data analysis that can seamlessly synthesize rigorous mathematical modeling with data of variable fidelity (e.g., measurements at multiple scales/resolutions or predictions of variable fidelity models) and multiple modalities (e.g., images, time–series, or scattered measurements). The setting we are interested in involves complex systems that are partially observed and whose dynamical behavior could be hard to model or totally unknown. The inherent uncertainty associated with this setting necessitates a departure from the classical deterministic realm of modeling and scientific computation, and, consequently, our main building blocks can no longer be crisp deterministic numbers and governing laws, but instead we must operate with probabilistic models.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Assessing data change in scientific datasets

Summary Scientific datasets are growing rapidly and becoming critical to next‐generation scientific discoveries. The validity of scientific results relies on the quality of data used and data are often subject to change, for example, due to observation additions, quality assessments, or processing software updates. The effects of data change are not well understood and difficult to predict. Datasets are often repeatedly updated and recomputing derived data products quickly becomes time consuming and resource intensive and may in some cases not even be necessary, thus delaying scientific advance. Despite its importance, there is a lack of systematic approaches for best comparing data versions to quantify the changes, and ad‐hoc or manual processes are commonly used. In this article, we propose a novel hierarchical approach for analyzing data changes, including real‐time (online) and offline analyses. We employ a variety of fast‐to‐compute numerical analyses, graphical data change representations, and more resource‐intensive recomputations of a subset of the data product. We illustrate the application of our approach using three scientific diverse use cases, namely, satellite, cosmological, and x‐ray data. The results show that a variety of data change metrics should be employed to enable a comprehensive representation and qualitative evaluation of data changes.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1994

BATSE data analysis

The Burst and Transient Source Experiment (BATSE) is one of four instruments on the Compton Observatory which was launched by the space shuttle Atlantis on 5 Apr. 1991. As of the end of Nov. 1993, BATSE detected more than 830 cosmic gamma ray bursts and more than 690 solar flares. Pulsed gamma rays have been detected from at least 16 sources and emission from at least 28 sources (including most of the pulsed sources) has been detected by the earth occultation technique. The daily BATSE operations tasks represent a substantial level of effort and involve a large team which includes MSFC personnel as well as contractors such as UAH. The effort is naturally divided into several areas: data operations, burst operations, occultation operations, and pulsar operations. UAH personnel have been involved to some extent in all of these as well as contributing to various areas of scientific data analysis.

Paciesas, William S.↗
NASA NTRS1997

The Next Generation Space Telescope: Visiting a Time When Galaxies Were Young

In the spring and summer of 1996, three independent teams studied the feasibility of a large aperture space telescope to follow the Hubble Space Telescope. The scientific goals for the new telescope had been laid out in a report by the HST & Beyond Committee, a group appointed by the Association of Universities for Research in Astronomy to consider the needs of the astronomical community after the nominal end of the HST mission in 2005. The technical capabilities and constraints on the new observatory were daunting: the telescope optics should be at least 4 meters in diameter and passively cooled to achieve optimum sensitivity in the near-infrared portion of the spectrum. Moreover, the costs should be kept within a fraction of those for the HST: approximately $500M for construction and $900M for lifetime costs, not including support for scientific data analysis. In their presentations to NASA on 19-21 August, 1996, the teams led by Lockheed Martin, TRW, and the Goddard Space Flight Center concluded that a Next Generation Space Telescope (NGST) was not only feasible and affordable, but that it could be made more powerful using recent breakthroughs in space technologies. Coming on the heels of breathtaking HST observations of distant galaxies in the process of formation, such an NGST could bridge the gap in our understanding of the earliest origins of stars, galaxies, and the elements that are the foundations of Life. This report presents the findings of the three teams and the technological roadmap which will guide us to the successful development of the NGST over the next decade. We have made liberal use of the written material, tables, and diagrams prepared by the three study teams. We have also taken advantage of the knowledge and ideas of our colleagues in government, industry and academia. In Appendix A, we list the members of the three study groups, the NGST Science Working Group and the NGST Scientific Oversight Committee. We deeply appreciate their assistance, advice, and enthusiasm. The scientific and technological goals of NGST are part of the Origins initiative in the Office of Space Science, NASA Headquarters. We are pleased to acknowledge the support and leadership of Edward Weiler, a steadfast friend of HST, Harley Thronson, a proponent of all things infrared, and Mike Kaplan, a tireless advocate of new technology. John Campbell, Project Manager for HST, initiated the NGST study at GSFC and we deeply appreciate his formative efforts and continued support. We are also grateful for the foresight of Riccardo Giacconi, Director General of the European Southern Observatory (ESO), and the efforts of his staff. We note that European Space Agency (ESA) staff at the Space Telescope Science Institute and ESO played important roles in the NGST scientific and technical studies. We look forward to future collaboration with ESA, ESO, and other international partners.

H.S. Stockman↗
DOE OSTI2016

πScope: Python based scientific workbench with MDSplus data visualization tool

In this work, a newly developed python based scientific data analysis and visualization tool, πScope (http://piscope.psfc.mit.edu), is reported. The primary motivation is 1) to provide an updated tool to browse the MDSplus data beyond existing dwscope/jScope and 2) to realize a universal foundation to construct interface tools to perform computer modeling from experimental data. To visualize MDSplus data, πScope has many features including overplotting different signals and discharges, generating various plot types (line, contour, image, etc.), performing in-panel data analysis using python scripts, and producing publication quality graphics. The logic to generate multi-panel plots is designed to be backward compatible with dwscope, enabling smooth migration for users. πScope uses multi-threading in data loading, and is easy to modify and expand due to its object-oriented design. Furthermore, A user can access the data structure both from a GUI and a script, enabling relatively complex data analysis workflow built quickly on πScope.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2016

On-demand data analytics in HPC environments at leadership computing facilities: Challenges and experiences

Scaling-up scientific data analysis and machine learning algorithms for data-driven discovery is a grand challenge that we face today. Despite the growing need for analysis from science domains that are generating `Big Data' from instruments and simulations, building high-performance analytical workflows of data-intensive algorithms have been daunting because: (i) the `Big Data' hardware and software architecture landscape is constantly evolving, (ii) newer architectures impose new programming models, and (iii) data-parallel kernels of analysis algorithms and their performance facets on different architectures are poorly understood. To address these problems, we have: (i) identified scalable data-parallel kernels of popular data analysis algorithms, (ii) implemented `Mini-Apps' of those kernels using different programming models (e.g. Map Reduce, MPI, etc.), (iii) benchmarked and validated the performance of the kernels in diverse architectures. In this paper, we discuss two of those Mini-Apps and show the execution of principal component analysis built as a workflow of the Mini-Apps. We show that Mini-Apps enable scientists to (i) write domain-specific data analysis code that scales on most HPC hardware and (ii) and offers the ability (most times with over a 10x speed-up) to analyze data sizes 100 times the size of what off-the-shelf desktop/workstations of today can handle.

Harney, John↗
DOE OSTI2024

To Derive or Not to Derive: I/O Libraries Take Charge of Derived Quantities Computation

The ever-increasing volume of data produced by HPC simulations necessitates scalable methods for data exploration and knowledge extraction. Scientific data analysis often involves complex queries across distributed datasets, requiring manipulation of multiple primary variables and generating derived data that needs to be handled efficiently, creating challenges for applications that need to parse many large datasets. Relying on individual applications to handle all intermediate data generally leads to redundant computations across studies and unnecessary data transfers. In this paper, we investigate the performance of different approaches where applications define derived variables as quantities of interest (QoIs) and offload the computation and transfer of these QoIs to the I/O library. This significantly reduces redundancy and optimizes data movement across the distributed storage and processing infrastructure by allowing control over when and where derived variables are computed. We present a detailed analysis of the performance-storage trade-offs associated with different solutions and showcase results for our study on two large-scale datasets created from climate and combustion simulations.

Gainaru, Ana↗
DOE OSTI2019

Workshop Report on Basic Research Needs for Scientific Machine Learning: Core Technologies for Artificial Intelligence

Scientific Machine Learning (SciML) and Artificial Intelligence (AI) will have broad use and transformative effects across the Department of Energy. Accordingly, the January 2018 Basic Research Needs workshop identified six Priority Research Directions (PRDs). The first three PRDs describe foundational research themes that correspond to the need for domain-awareness (PRD #1), interpretability (PRD #2), and robustness (PRD #3). The other three PRDs describe capability research themes and correspond to the three major use cases for massive scientific data analysis (PRD #4), machine learning-enhanced modeling and simulation (PRD #5), and intelligent automation and decision-support for complex systems (PRD #6). The Priority Research Directions provide a sound basis for a coherent, long-term research and development strategy in SciML and AI. Over the last decade, DOE investments in applied mathematics have laid the groundwork for the type of basic research that will underpin key advances in the six PRDs. Such advances will build on the work from leading researchers in optimization, linear algebra, high-performance solvers and algorithms, multiscale modeling and simulation, complex systems research, uncertainty quantification, and the new basic research areas that will emerge from the pursuit of transformative technologies.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Brochure on Basic Research Needs for Scientific Machine Learning: Core Technologies for Artificial Intelligence [Slides]

Scientific Machine Learning (SciML) and Artificial Intelligence (AI) will have broad use and transformative effects across the Department of Energy. Accordingly, the January 2018 Basic Research Needs workshop identified six Priority Research Directions (PRDs). The first three PRDs describe foundational research themes that correspond to the need for domain-awareness (PRD #1), interpretability (PRD #2), and robustness (PRD #3). The other three PRDs describe capability research themes and correspond to the three major use cases for massive scientific data analysis (PRD #4), machine learning-enhanced modeling and simulation (PRD #5), and intelligent automation and decision-support for complex systems (PRD #6). The Priority Research Directions provide a sound basis for a coherent, long-term research and development strategy in SciML and AI. Over the last decade, DOE investments in applied mathematics have laid the groundwork for the type of basic research that will underpin key advances in the six PRDs. Such advances will build on the work from leading researchers in optimization, linear algebra, high-performance solvers and algorithms, multiscale modeling and simulation, complex systems research, uncertainty quantification, and the new basic research areas that will emerge from the pursuit of transformative technologies.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1983

U.S. data processing for the IRAS project

The JPL's Scientific Data Analysis System (SDAS), which will process IRAS data and produce a catalogue of perhaps a million infrared sources in the sky, as well as other information for astronomical records, is described. The purposes of SDAS are discussed, and the major SDAS processors are shown in block diagram. The catalogue processing is addressed, mentioning the basic processing steps which will be applied to raw detector data. Signal reconstruction and conversion to astrophysical units, source detection, source confirmation, data management, and survey data products are considered in detail.

Duxbury, J. H.↗
NASA NTRS1991

AIAA/NASA International Symposium on Space Information Systems, 2nd, Pasadena, CA, Sept. 17-19, 1990, Proceedings. Vols. 1 & 2

These proceedings overview major space information system projects and lessons learned from current missions. Other topics include the science information system requirements for the 1990s, an information systems design approach for major programs, the technology needs and projections, the standards for space data information systems, the artificial intelligence technology and applications, international interoperability, and spacecraft data systems and architectures advanced communications. Other topics include the software engineering technology and applications, the multimission multidiscipline information system architectures, the distributed planning and scheduling systems and operations, and the computer and information systems architectures. Paper presented include prospects for scientific data analysis systems for solar-terrestrial physics in the 1990s, the Columbus data management system, data storage technologies for the future, the German aerospace research establishment, and launching artificial intelligence in NASA ground systems.

Tavenner, Leslie A.↗
NASA NTRS1998

Investigation of Coronal Large Scale Structures Utilizing Spartan 201 Data

Two telescopes aboard Spartan 201, a small satellite has been launched from the Space Shuttles, on April 8th, 1993, September 8th, 1994, September 7th, 1995 and November 20th, 1997. The main objective of the mission was to answer some of the most fundamental unanswered questions of solar physics-What accelerates the solar wind and what heats the corona? The two telescopes are 1) Ultraviolet Coronal Spectrometer (UVCS) provided by the Smithsonian Astrophysical Observatory which uses ultraviolet emissions from neutral hydrogen and ions in the corona to determine velocities of the coronal plasma within the solar wind source region, and the temperature and density distributions of protons and 2) White Light Coronagraph (WLC) provided by NASA's Goddard Space Flight Center which measures visible light to determine the density distribution of coronal electrons within the same region. The PI has had the primary responsibility in the development and application of computer codes necessary for scientific data analysis activities, end instrument calibration for the white-light coronagraph for the entire Spartan mission. The PI was responsible for the science output from the WLC instrument. PI has also been involved in the investigation of coronal density distributions in large-scale structures by use of numerical models which are (mathematically) sufficient to reproduce the details of the observed brightness and polarized brightness distributions found in SPARTAN 201 data.

Guhathakurta, Madhulika↗
NASA NTRS2003

A Step Beyond Simple Keyword Searches: Services Enabled by a Full Content Digital Journal Archive

The problems of managing and searching large archives of scientific journal articles can potentially be addressed through data mining and statistical techniques matured primarily for quantitative scientific data analysis. A journal paper could be represented by a multivariate descriptor, e.g., the occurrence counts of a number key technical terms or phrases (keywords), perhaps derived from a controlled vocabulary ( e . g . , the American Meteorological Society's Glossary of Meteorology) or bootstrapped from the journal archive itself. With this technique, conventional statistical classification tools can be leveraged to address challenges faced by both scientists and professional societies in knowledge management. For example, cluster analyses can be used to find bundles of "most-related" papers, and address the issue of journal bifurcation (when is a new journal necessary, and what topics should it encompass). Similarly, neural networks can be trained to predict the optimal journal (within a society's collection) in which a newly submitted paper should be published. Comparable techniques could enable very powerful end-user tools for journal searches, all premised on the view of a paper as a data point in a multidimensional descriptor space, e.g.: "find papers most similar to the one I am reading", "build a personalized subscription service, based on the content of the papers I am interested in, rather than preselected keywords", "find suitable reviewers, based on the content of their own published works", etc. Such services may represent the next "quantum leap" beyond the rudimentary search interfaces currently provided to end-users, as well as a compelling value-added component needed to bridge the print-to-digital-medium gap, and help stabilize professional societies' revenue stream during the print-to-digital transition.

Boccippio, Dennis J.↗
NASA NTRS2012

JPL Genesis and Rapid Intensification Processes (GRIP) Portal

Satellite observations can play a very important role in airborne field campaigns, since they provide a comprehensive description of the environment that is essential for the experiment design, flight planning, and post-experiment scientific data analysis. In the past, it has been difficult to fully utilize data from multiple NASA satellites due to the large data volume, the complexity of accessing NASA s data in near-real-time (NRT), as well as the lack of software tools to interact with multi-sensor information. The JPL GRIP Portal is a Web portal that serves a comprehensive set of NRT observation data sets from NASA and NOAA satellites describing the atmospheric and oceanic environments related to the genesis and intensification of the tropical storms in the North Atlantic Ocean. Together with the model forecast data from four major global atmospheric models, this portal provides a useful tool for the scientists and forecasters in planning and monitoring the NASA GRIP field campaign during the 2010 Atlantic Ocean hurricane season. This portal uses the Google Earth plug-in to visualize various types of data sets, such as 2D maps, wind vectors, streamlines, 3D data sets presented at series of vertical cross-sections or pointwise vertical profiles, and hurricane best tracks and forecast tracks. Additionally, it allows users to overlap multiple data sets, change the opacity of each image layer, generate animations on the fly with selected data sets, and compare the observation data with the model forecast using two independent calendars. The portal also provides the capability to identify the geographic location of any point of interest. In addition to supporting the airborne mission planning, the NRT data and portal will serve as a very rich source of information during the post-field campaign analysis stage of the airborne experiment. By including a diverse set of satellite observations and model forecasts, it provides a good spatial and temporal context for the high-resolution, but limited in space and time, airborne observations.

Knosp, Brian W.↗
DOE OSTI2011

Parallel In Situ Indexing for Data-intensive Computing

As computing power increases exponentially, vast amount of data is created by many scientific re- search activities. However, the bandwidth for storing the data to disks and reading the data from disks has been improving at a much slower pace. These two trends produce an ever-widening data access gap. Our work brings together two distinct technologies to address this data access issue: indexing and in situ processing. From decades of database research literature, we know that indexing is an effective way to address the data access issue, particularly for accessing relatively small fraction of data records. As data sets increase in sizes, more and more analysts need to use selective data access, which makes indexing an even more important for improving data access. The challenge is that most implementations of in- dexing technology are embedded in large database management systems (DBMS), but most scientific datasets are not managed by any DBMS. In this work, we choose to include indexes with the scientific data instead of requiring the data to be loaded into a DBMS. We use compressed bitmap indexes from the FastBit software which are known to be highly effective for query-intensive workloads common to scientific data analysis. To use the indexes, we need to build them first. The index building procedure needs to access the whole data set and may also require a significant amount of compute time. In this work, we adapt the in situ processing technology to generate the indexes, thus removing the need of read- ing data from disks and to build indexes in parallel. The in situ data processing system used is ADIOS, a middleware for high-performance I/O. Our experimental results show that the indexes can improve the data access time up to 200 times depending on the fraction of data selected, and using in situ data processing system can effectively reduce the time needed to create the indexes, up to 10 times with our in situ technique when using identical parallel settings.

97 MATHEMATICS AND COMPUTING↗