Search NASA⌕ Search

SEARCH · Search NASA

Results for “Big data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

BIG MAC: A bolometer array for mid-infrared astronomy, Center Director's Discretionary Fund

The infrared array referred to as Big Mac (for Marshall Array Camera), was designed for ground based astronomical observations in the wavelength range 5 to 35 microns. It contains 20 discrete gallium-doped germanium bolometer detectors at a temperature of 1.4K. Each bolometer is irradiated by a square field mirror constituting a single pixel of the array. The mirrors are arranged contiguously in four columns and five rows, thus defining the array configuration. Big Mac utilized cold reimaging optics and an up looking dewar. The total Big Mac system also contains a telescope interface tube for mounting the dewar and a computer for data acquisition and processing. Initial astronomical observations at a major infrared observatory indicate that Big Mac performance is excellent, having achieved the design specifications and making this instrument an outstanding tool for astrophysics.

Telesco, C. M.↗

Using Docker Containers to Extend Reproducibility Architecture for the NASA Earth Exchange (NEX)

NASA Earth Exchange (NEX) is a data, supercomputing and knowledge collaboratory that houses NASA satellite, climate and ancillary data where a focused community can come together to address large-scale challenges in Earth sciences. As NEX has been growing into a petabyte-size platform for analysis, experiments and data production, it has been increasingly important to enable users to easily retrace their steps, identify what datasets were produced by which process chains, and give them ability to readily reproduce their results. This can be a tedious and difficult task even for a small project, but is almost impossible on large processing pipelines. We have developed an initial reproducibility and knowledge capture solution for the NEX, however, if users want to move the code to another system, whether it is their home institution cluster, laptop or the cloud, they have to find, build and install all the required dependencies that would run their code. This can be a very tedious and tricky process and is a big impediment to moving code to data and reproducibility outside the original system. The NEX team has tried to assist users who wanted to move their code into OpenNEX on Amazon cloud by creating custom virtual machines with all the software and dependencies installed, but this, while solving some of the issues, creates a new bottleneck that requires the NEX team to be involved with any new request, updates to virtual machines and general maintenance support. In this presentation, we will describe a solution that integrates NEX and Docker to bridge the gap in code-to-data migration. The core of the solution is saemi-automatic conversion of science codes, tools and services that are already tracked and described in the NEX provenance system, to Docker - an open-source Linux container software. Docker is available on most computer platforms, easy to install and capable of seamlessly creating and/or executing any application packaged in the appropriate format. We believe this is an important step towards seamless process deployment in heterogeneous environments that will enhance community access to NASA data and tools in a scalable way, promote software reuse, and improve reproducibility of scientific results.

earth exchange↗

Testing the Standard Model with the Primordial Inflation Explorer

The Primordial Inflation Explorer is an Explorer-class mission to measure the gravity-wave signature of primordial inflation through its distinctive imprint on the linear polarization of the cosmic microwave background. PIXIE uses an innovative optical design to achieve background-limited sensitivity in 400 spectral channels spanning 2.5 decades in frequency from 30 GHz to 6 THz (1 cm to 50 micron wavelength). The principal science goal is the detection and characterization of linear polarization from an inflationary epoch in the early universe, with tensor-to-scalar ratio r < 10A{-3) at 5 standard deviations. The rich PIXIE data set will also constrain physical processes ranging from Big Bang cosmology to the nature of the first stars to physical conditions within the interstellar medium of the Galaxy. I describe the PIXIE instrument and mission architecture needed to detect the inflationary signature using only 4 semiconductor bolometers.

Kogut, Alan J.↗

The Primordial Inflation Explorer (PIXIE): A Nulling Polarimeter for Cosmic Microwave Background Observations

The Primordial Inflation Explorer (PIXIE) is a concept for an Explorer-class mission to measure the gravity-wave signature of primordial inflation through its distinctive imprint on the linear polarization of the cosmic microwave background. The instrument consists of a polarizing Michelson interferometer configured as a nulling polarimeter to measure the difference spectrum between orthogonal linear polarizations from two co-aligned beams. Either input can view the sky or a temperature-controlled absolute reference blackbody calibrator. Rhe proposed instrument can map the absolute intensity and linear polarization (Stokes I, Q, and U parameters) over the full sky in 400 spectral channels spanning 2.5 decades in frequency from 30 GHz to 6 THz (1 cm to 50 micron wavelength). Multi-moded optics provide background-limited sensitivity using only 4 detectors, while the highly symmetric design and multiple signal modulations provide robust rejection of potential systematic errors. The principal science goal is the detection and characterization of linear polarization from an inflationary epoch in the early universe, with tensor-to-scalar ratio r < 10..3 at 5 standard deviations. The rich PIXIE data set can also constrain physical processes ranging from Big Bang cosmology to the nature of the first stars to physical conditions within the interstellar medium of the Galaxy.

Kogut, Alan J.↗

Inflation, Reionization, and All That: The Primordial Inflation Explorer

The Primordial Inflation Explorer is an Explorer-class mission to measure the gravity-wave signature of primordial inflation through its distinctive imprint on the linear polarization of the cosmic microwave background. PIXIE uses an innovative optical design to achieve background-limited sensitivity in 400 spectral channels spanning 2.5 decades in frequency from 30 GHz to 6 THz (1 cm to 50 micron wavelength). The principal science goal is the detection and characterization of linear polarization from an inflationary epoch in the early universe, with tensor-to-scalar ratio r less than l0^{-3) at 5 standard deviations. The rich PIXIE data set will also constrain physical processes ranging from Big Bang cosmology to the nature of the first stars to physical conditions within the interstellar medium of the Galaxy. I describe the PIXIE instrument and mission architecture needed to detect the inflationary signature using only 4 semiconductor bolometers.

Kogut, Alan J.↗

Inflation, Reionization, and All That: The Primordial Inflation Explorer

The Primordial Inflation Explorer is an Explorer-class mission to measure the gravity-wave signature of primordial inflation through its distinctive imprint on the linear polarization of the cosmic microwave background. PIXIE uses an innovative optical design to achieve background-limited sensitivity in 400 spectral channels spanning 2.5 decades in frequency from 30 GHz to 6 THz (1 cm to 50 micron wavelength). The principal science goal is the detection and characterization of linear polarization from an inflationary epoch in the early universe, with tensor-to-scalar ratio r < 10(exp -3) at 5 standard deviations. The rich PIXIE data set will also constrain physical processes ranging from Big Bang cosmology to the nature of the first stars to physical conditions within the interstellar medium of the Galaxy. I describe the PIXIE instrument and mission architecture needed to detect the inflationary signature using only 4 semiconductor bolometers.

Kogut, Alan J.↗

The Primordial Inflation Explorer

The Primordial Inflation Explorer is an Explorer-class mission to measure the gravity-wave signature of primordial inflation through its distinctive imprint on the linear polarization of the cosmic microwave background. PIXIE uses an innovative optical design to achieve background-limited sensitivity in 400 spectral channels spanning 2.5 decades in frequency from 30 GHz to 6 THz (1 cm to 50 micron wavelength). The principal science goal is the detection and characterization of linear polarization from an inflationary epoch in the early universe, with tensor-to-scalar ratio r < 10(exp -3) at 5 standard deviations. The rich PIXIE data set will also constrain physical processes ranging from Big Bang cosmology to the nature of the first stars to physical conditions within the interstellar medium of the Galaxy. I describe the PIXIE instrument and mission architecture needed to detect the inflationary signature using only 4 semiconductor bolometers.

Kogut, Alan J.↗

Near 40 Years MERRA-2 Data at NASA GES DISC -Opportunity and Challenge to Support Extremes Study

To the end of 2019, 40 years NASA climate reanalysis data sets from the Modern Era Retrospective-analysis for Research and Applications, Version 2 (MERRA-2) will be available at NASA Goddard Earth Sciences Data and Information Services Center (GES DISC). MERRA-2 consists atmosphere, land, and ocean data, which may be used for the studies ranging from the short scale weather events to the large scale decadal vulnerabilities. The hourly products, such as precipitation, soil moisture, temperature, and aerosols etc., have been used widely to study extreme events.In supporting users from broad communities, GES DISC have developed various data access services, including subsetter for downloading only data of interest with preferred format; OPeNDAP - for machine-to-machine data access; and Giovanni- for online visualization and analysis, etc. A big challenge for extreme study is to downloading and processing long-term hourly or daily data. The data downloading performance is not very satisfied by many users with current services and the native archived data structure. Late June 2019, many people in Europe had experienced extreme heat waves. The temperatures in several countries exceeded 40°C (104°F). For example, MERRA-2 shows that the near surface daily maximum temperature of June 28 2019 over Marseille, a city in southern France, reached 41.1 °C (106°F), which is the record breaking temperature in the last 40 years. GES DISC is working together with domain science experts to improve the performance of long time series access, making analysis ready data sets in supporting application researches, such as extreme study. In this presentation, using Europe heat wave as an example, we will show prototype of the in developing service for finding extremes from near 40 years MERRA-2 data at a given location. MERRA-2 data can be accessed from NASA GES DISC(https://disc.gsfc.nasa.gov/ ) by search keyword "MERRA-2".

Shen, Suhung↗

Disruptive Technologies and Their Putative Impacts Upon Society and Aerospace- Entering The Virtual Age

Developments in technology over the recent decades have been extraordinary. They include the IT, bio, nano, and now quantum and energetics technology arenas and their many combinatorial interactions and impacts. In the main, these are at the frontiers of the small and in a combinational, synergistic feeding frenzy with each other. They fall under the broad category of Disruptive Technologies and have greatly altered society. The outlook for the runout of these and other technology developments augers mid-term to later alterations in components of the human existence theorem, including the requirement to work for our living and our physiological makeup and longevity (Ref 1). The IT revolution began in the 1950s with the development of solid-state electronics. The biologics revolution began later in the 1960s and 1970s with DNA and genomics, and the nano revolution in the 1990s with self-forming nano systems and carbon nanotubes. Quantum technology is now developing rapidly, aided by enabling nano systems, and the energetics revolution is providing ever more efficient and less expensive renewable energy sources. The IT revolution has produced improvements of an astounding eleven orders of magnitude in computing speed since the late 1950s. As we shift from silicon to biological, optical, nano, molecular, and atomic computing, improvements of some 4 orders of magnitude are evidently possible from either optical or DNA computing [Refs 2and 3], then there are combinatorials. Then there is quantum computing, under development worldwide for an increasing number of applications and proffering phenomenal capabilities. The current fastest computers are considerably beyond human brain speed. Machine intelligence is developing well after decades of inadequate machine capability, now no longer the case, and a detour into expert systems. Researchers in machine intelligence are now pursuing deep learning approaches using neural nets, which are proving to be extremely useful. Some believe the frontier of potential human-level machine intelligence may be found in biomimetics and brain-emulation approaches. There is even a possibility of “emergence”—i.e., when the machine intelligence is complex enough that it “wakes up,” as when human intelligence emerged via evolution during the million-plus years of the hunter-gatherer epoch [ Ref 4]. In fact, some posit that human intelligence can be improved upon and is only a cul-de-sac of what is conceivable. The IT revolution has produced massive changes in human society and economics—from the Internet, enabling the rapid expansion of knowledgeability (and even what is knowable), to an increasingly pervasive trend of “tele-everything.” The extraordinary compilation, storage, and availability of truly massive amounts of information could, when combined with AI and under the mantra of “big data,” greatly improve many of our technical and commercial processes and their content including elucidating new heuristic governing laws.

Dennis M. Bushnell↗

Big Data, Cloud, and Earth Science

Given the reality of a Big Data future, we need to reevaluate our ability to quickly process immense amounts of data while maintaining our responsibility of stewarding our archives. At National Aeronautics and Space Administration (NASA), our Science Missions Directorate and Earth Science Data Systems have been exploring the commercial cloud as a potential mechanism for data ingest, archive, and distribution since 2015. This talk will discuss our timeline and strategies being employed and will discuss the process of introducing commercial cloud entities into a largely on-premise hardware-reliant approach to ingest, archive, and distribution. We will address topics such as vendor lock-in, varying compute and costing strategies, and end-user adoption. Please join us for an open and authentic conversation about the opportunities we are embracing and the risks we are undertaking as a part of this evolution.

Informatics↗

Coevolution of Machine Learning and Process-Based Modelling to Revolutionize Earth and Environmental Sciences: A Perspective

Machine learning (ML) applications in Earth and environmental sciences (EES) have gained incredible momentum in recent years. However, these ML applications have largely evolved in ‘isolation’ from the mechanistic, process-based modelling (PBM) paradigms, which have historically been the cornerstone of scientific discovery and policy support. In this perspective, we assert that the cultural barriers between the ML and PBM communities limit the potential of ML, and even its ‘hybridization’ with PBM, for EES applications. Fundamental, but often ignored, differences between ML and PBM are discussed as well as their strengths and weaknesses in light of three overarching modelling objectives in EES, (1) nowcasting and prediction, (2) scenario analysis, and (3) diagnostic learning. The paper ponders over a ‘coevolutionary’ approach to model building, shifting away from a borrowing to a co-creation culture, to develop a generation of models that leverage the unique strengths of ML such as scalability to big data and high-dimensional mapping, while remaining faithful to process-based knowledge base and principles of model explainability and interpretability, and therefore, falsifiability.

Saman Razavi↗

Time and Measurement Days

Questions in data analysis involving the concepts of time and measurement are often pushed into the background or reserved for a philosophical discussion. Some examples are: a) Is causality a consequence of the laws of physics, or can the arrow of time be reversed? b) Can we determine the arrow of time of an event? c) Do we need the continuum hypothesis for the underlying function in any measurement process? d) Can we say anything about the analyticity of the underlying process of an event? e) Would it be valid to model a non-analytical process as function of time? f) What are the implications of all these questions for classical Fourier techniques? However, in the age of big data gathered either from space missions supplying ultra-precise long time series, or e.g. LIGO data from the ground, the moment to bring these questions to the foreground seems arrived. The limitations of our understanding of some fundamental processes is emphasized by the lack of solution for problems open for more than 2 decades, such as the non-detection of solar g-modes, or the modal identification of main sequence stellar pulsators like delta Scuti stars. Flicker noise or 1/f noise, for example, attributed in solar-like stars to granulation, is analyzed mostly only to apply noise reduction techniques, neither considering the classical problem of 1/f noise that was introduced a 100 years ago, nor taking into account ergodic or non-ergodic solutions that make inapplicable spectral analysis techniques in practice. This topic was discussed by Nicholas W. Watkins during the ITISE meeting held in Granada in 2016. There he presented preliminary results of his research on Mandelbrot's related work. We reproduce here his quotation of Mandelbrot (1999) "There is a sharp contrast between a highly anomalous ("non-white") noise that proceeds in ordinary clock time and a noise whose principal anomaly is that it is restricted to fractal time", suggesting a connection with the above proposed topics that could be phrased as the following additional questions:a) Is self-organized criticality (SOC) frequent in astrophysical phenomena? b) Could all fractals in nature be considered stochastic? c) Could we establish mathematical/physical relationships between chaotic and fractal behaviors in time series? d) Could the differences between fractals and chaos in terms of analyticity be used to understand the residuals of the fitting of stellar light curves? In this meeting we would like to approximate these problems from a holistic and multidisciplinary perspective, taking into account not only technical issues but also the deeper implications. In particular the concept of connectivity (introduced in Pascual-Granado et al. A&A, 2015) could be used to implement, within the framework of ARMA processes, an "arrow of time" (see attached document), and so studying the possible implications in the concept of time as envisaged by Watkins.

data analysis↗

Multiresolution With Super-Compact Wavelets

The solution data computed from large scale simulations are sometimes too big for main memory, for local disks, and possibly even for a remote storage disk, creating tremendous processing time as well as technical difficulties in analyzing the data. The excessive storage demands a corresponding huge penalty in I/O time, rendering time and transmission time between different computer systems. In this paper, a multiresolution scheme is proposed to compress field simulation or experimental data without much loss of important information in the representation. Originally, the wavelet based multiresolution scheme was introduced in image processing, for the purposes of data compression and feature extraction. Unlike photographic image data which has rather simple settings, computational field simulation data needs more careful treatment in applying the multiresolution technique. While the image data sits on a regular spaced grid, the simulation data usually resides on a structured curvilinear grid or unstructured grid. In addition to the irregularity in grid spacing, the other difficulty is that the solutions consist of vectors instead of scalar values. The data characteristics demand more restrictive conditions. In general, the photographic images have very little inherent smoothness with discontinuities almost everywhere. On the other hand, the numerical solutions have smoothness almost everywhere and discontinuities in local areas (shock, vortices, and shear layers). The wavelet bases should be amenable to the solution of the problem at hand and applicable to constraints such as numerical accuracy and boundary conditions. In choosing a suitable wavelet basis for simulation data among a variety of wavelet families, the supercompact wavelets designed by Beam and Warming provide one of the most effective multiresolution schemes. Supercompact multi-wavelets retain the compactness of Haar wavelets, are piecewise polynomial and orthogonal, and can have arbitrary order of approximation. The advantages of the multiresolution algorithm are that no special treatment is required at the boundaries of the interval, and that the application to functions which are only piecewise continuous (internal boundaries) can be efficiently implemented. In this presentation, Beam's supercompact wavelets are generalized to higher dimensions using multidimensional scaling and wavelet functions rather than alternating the directions as in the 1D version. As a demonstration of actual 3D data compression, supercompact wavelet transforms are applied to a 3D data set for wing tip vortex flow solutions (2.5 million grid points). It is shown that high data compression ratio can be achieved (around 50:1 ratio) in both vector and scalar data set.

Lee, Dohyung↗

The Effect of New Ozone Cross Sections Applied to SBUV and TOMS Retrievals

The ozone cross sections as measured by Bass and Paur have been used for processing of SBUV and TOMS data since 1986. While these cross sections were a big improvement over those previously available, there were known minor problems with accuracy for wavelengths longward of 330 nm and with the temperature dependance. Today's requirements to separate stratospheric ozone from tropospheric ozone and for the derivation of minor species such as BrO and N02 place stringent new requirements on the accuracy needed. The ozone cross section measurements of Brion, Daumont, and Malicet (BDM) are being considered for use in UV-based ozone retrievals. They have much better resolution, an extended wavelength range, and a more consistent temperature dependance. Tests show that BDM retrievals exhibit lower retrieval residuals in the satellite data; i.e., they explain our measured atmospheric radiances more accurately. Total column ozone retrieved by the TOMS instruments is about 1.5% higher than before. Ozone profiles retrieved from SBUV using the new cross sections are lower in the upper stratosphere and higher in the lower stratosphere and troposphere.

McPeters, Richard D.↗

LANDSAT demonstration/application and GIS integration in south central Alaska

Automated geographic information systems were developed for two sites in Southcentral Alaska to serve as tests for both the process of integrating classified LANDSAT data into a comprehensive environmental data base and the process of using automated information in land capability/suitability analysis and environmental planning. The Big Lake test site, located approximately 20 miles north of the City of Anchorage, comprises an area of approximately 150 square miles. The Anchorage Hillside test site, lying approximately 5 miles southeast of the central part of the city, extends over an area of some 25 square miles. Map construction and content is described.

Burns, A. W.↗

Digital Technologies at NASA for Science and Engineering

While scientific and engineering advancements used to rely primarily on theoretical studies and physical experiments, today digital technology enabled by petaflops-scale supercomputers is an equal, if not a greater, contributor to such achievements. In addition, computational modeling and simulation serves as a predictive tool that is not otherwise available. As a result, the use of high performance computing is integral to NASA's work in all mission areas such as space exploration, aeronautics, and scientific discovery. But traditional supercomputing alone is not sufficient for all of the space agency's needs. The success of many NASA missions depends on solving complex computing challenges, some of which are NP-hard (decision theory) if using classical solution methods. Quantum computing promises an unprecedented ability to solve such intractable problems by harnessing quantum mechanical effects such as tunneling, superposition, and entanglement. Another disruptive digital technology is neuromorphic computing that uses brain-inspired lessons to generate new architectures that are much more energy efficient, and capable of massive parallel processing and learning in-situ. Finally, with large amounts of observational and computational data sets, the opportunities of big data and data analytics can be leveraged to enable deep learning and knowledge discovery - it's all a massive digital transformation. This talk will be an overview how NASA utilizes digital technologies for its science and engineering efforts.

Biswas, Rupak↗

MERRA Analytic Services: Meeting the Big Data Challenges of Climate Science Through Cloud-enabled Climate Analytics-as-a-service

Climate science is a Big Data domain that is experiencing unprecedented growth. In our efforts to address the Big Data challenges of climate science, we are moving toward a notion of Climate Analytics-as-a-Service (CAaaS). We focus on analytics, because it is the knowledge gained from our interactions with Big Data that ultimately produce societal benefits. We focus on CAaaS because we believe it provides a useful way of thinking about the problem: a specialization of the concept of business process-as-a-service, which is an evolving extension of IaaS, PaaS, and SaaS enabled by Cloud Computing. Within this framework, Cloud Computing plays an important role; however, we it see it as only one element in a constellation of capabilities that are essential to delivering climate analytics as a service. These elements are essential because in the aggregate they lead to generativity, a capacity for self-assembly that we feel is the key to solving many of the Big Data challenges in this domain. MERRA Analytic Services (MERRAAS) is an example of cloud-enabled CAaaS built on this principle. MERRAAS enables MapReduce analytics over NASAs Modern-Era Retrospective Analysis for Research and Applications (MERRA) data collection. The MERRA reanalysis integrates observational data with numerical models to produce a global temporally and spatially consistent synthesis of 26 key climate variables. It represents a type of data product that is of growing importance to scientists doing climate change research and a wide range of decision support applications. MERRAAS brings together the following generative elements in a full, end-to-end demonstration of CAaaS capabilities: (1) high-performance, data proximal analytics, (2) scalable data management, (3) software appliance virtualization, (4) adaptive analytics, and (5) a domain-harmonized API. The effectiveness of MERRAAS has been demonstrated in several applications. In our experience, Cloud Computing lowers the barriers and risk to organizational change, fosters innovation and experimentation, facilitates technology transfer, and provides the agility required to meet our customers' increasing and changing needs. Cloud Computing is providing a new tier in the data services stack that helps connect earthbound, enterprise-level data and computational resources to new customers and new mobility-driven applications and modes of work. For climate science, Cloud Computing's capacity to engage communities in the construction of new capabilies is perhaps the most important link between Cloud Computing and Big Data.

Data Analytics↗

Communication Network Awareness Machine System Phase I Development: The Intelligent Party-Line Schema

As NextGen continues toward the full implementation of a Net-Centric Architecture (N-CA)it will inherently provide a continuous increase to the Three-Vs components (Volume, Velocity, and Variety) of big data . This will create an insurmountable environment for direct-action aviation personnel (DAAP)as the DAAP’s natural abilities to manage and process data into actionable information will be overmatched by the Three-Vs. Therefore, conducting operations within a N-CA requires that new tools and applications be researched and developed to aid the DAAP’s ability to understand and manage data, mitigate non-normals, create contingency plans and actions. This paper will describe a research area at NASA Langley Research Center known as the Intelligent Party-Line (IPL).

Intelligent Party-Line↗