SEARCH · Search NASA

Results for “Big data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14

All sources NASA NTRS DOE OSTI

DOE OSTI2025

High‐Resolution National‐Scale Water Modeling Is Enhanced by Multiscale Differentiable Physics‐Informed Machine Learning

Abstract The National Water Model (NWM) is a key tool for flood forecasting, planning, and water management. Key challenges facing the NWM include calibration and parameter regionalization when confronted with big data. We present two novel versions of high‐resolution (∼37 km 2 ) differentiable models (a type of hybrid model): one with implicit, unit‐hydrograph‐style routing and another with explicit Muskingum‐Cunge routing in the river network. The former predicts streamflow at basin outlets whereas the latter presents a discretized product that seamlessly covers rivers in the conterminous United States (CONUS). Both versions use neural networks to provide a multiscale parameterization and process‐based equations to provide a structural backbone, which were trained simultaneously (“end‐to‐end”) on 2,807 basins across the CONUS and evaluated on 4,997 basins. Both versions show great potential to elevate future NWM performance for extensively calibrated as well as ungauged sites: the median daily Nash‐Sutcliffe efficiency of all 4,997 basins is improved to around 0.68 from 0.48 of NWM3.0. As they resolve spatial heterogeneity, both versions greatly improved simulations in the western CONUS and also in the Prairie Pothole Region, a long‐standing modeling challenge. The Muskingum‐Cunge version further improved performance for basins >10,000 km 2 . Overall, our results show how neural‐network‐based parameterizations can improve NWM performance for providing operational flood predictions while maintaining interpretability and multivariate outputs. The modeling system supports the Basic Model Interface (BMI), which allows seamless integration with the next‐generation NWM. We also provide a CONUS‐scale hydrologic data set for further evaluation and use.

Song, Yalan [Civil and Environmental Engineering T↗

NASA NTRS1991

COBE ground segment attitude determination

The Cosmic Background Explorer (COBE) spacecraft was launched in November 1989 by NASA to survey the sky for primordial radiation left from the Big Bang explosion. The success of the mission requires an accurate determination of the spacecraft attitude. While the accuracy of the attitude obtained from the attitude sensors is adequate for two of the experiments, the higher accuracy required by the Diffuse Infrared Background Experiment (DIRBE) is obtained by using the DIRBE instrument as a special type of star sensor. Presented here is an overview of the attitude processing algorithms used at the Cosmology Data Analysis Center (CDAC) and the results obtained from the flight data.

Kumar, V. K.↗

DOE OSTI2019

A Scalable Graph Analytics Framework for Programming with Big Data in R (pbdR)

Many disciplines such as biology, economics, engineering, physics, and the social sciences represent their data as graphs to capture patterns, trends, and associations. There are are many commercially available graph libraries in different programming languages to analyze these complex graphs. But there is no distributed graph library package in R - the popular statistical programming language to analyze graphs that bigger than a single machine's memory. Many domain experts prefer R over the numerous other alternatives. Towards this, we present a distributed graph analytics framework for R called programming with big graph using R (pBGR.) Our proposed framework leverages the Programming with Big Data in R (pbdR) ecosystem that provides scalable R packages for distributed computing in data science. We present an early prototype implementation of this framework using the distributed-memory parallel graph library CombBLAS and evaluate the framework's performance on leadership class computing platforms. Our experimental results demonstrate that the proposed framework is capable of performing large-scale parallel graph mining through the easyto-use R language. This enhanced graph processing capability coupled with other statistical tools already available in R, should be valuable to many domain experts.

Hasan, S M Shamimul↗

NASA NTRS1999

Origin of Gamma-Ray Emissions from the MeV Blazars

More than sixty gamma-ray emitting blazars have been detected by the Compton Gamma Ray Observatory (CGRO). A sub-class of these blazars is known as MeV blazars that are most luminous at the MeV energies. Spectra of these sources show a break somewhere between 1 MeV and 30 MeV with break amplitude as large as DELTA-GAMMA = 1.5. Different models like Compton cooling jet models, synchrotron self-Compton models, etc., have been suggested to explain the observed properties of the MeV luminous blazars. However these models are unable to account for spectral break amplitudes in excess of 0.5. Here we suggest an inverse-Compton model based on recent published experimental data where MeV gamma rays were produced via Compton backscattering of UV photons with 500 MeV electrons. We show that the high energy electrons of the relativistic jet can collide with the copious ultraviolet photons of the big blue bump of the blazar, and produce photons peaked in the MeV region. Also we show that the X rays and the soft gamma rays in these sources are produced through the synchrotron self-Compton process. Flux computed from our model have been used to compare with the observed X-ray/gamma-ray spectra of the MeV luminous blazars. Details of the model are discussed in the present paper.

Ghosh, K. K.↗

NASA NTRS1998

Origin of Gamma-Ray Emissions from the MeV Blazars

More than sixty gamma-ray emitting blazars have been detected by the Compton Gamma Ray Observatory (CGRO). A sub-class of these blazars is known as MeV blazars that are most luminous at the MeV energies. Spectra of these sources show a break somewhere between 1 MeV and 30 MeV with break amplitude as large as Delta(Gamma) = 1.5. Different models like Compton cooling jet models, synchrotron self-Compton models, etc., have been suggested to explain the observed properties of the MeV luminous blazars. However these models are unable to account for spectral break amplitudes in excess of 0.5. Here we suggest an inverse-Compton model based on recent published experimental data where MeV gamma rays were produced via Compton backscattering of UV photons with 500 MeV electrons. We show that the high energy electrons of the relativistic jet can collide with the copious ultraviolet photons of the big blue bump of the blazar, and produce photons peaked in the MeV region. Also we show that the X rays and the soft gamma rays in these sources are produced through the synchrotron self-Compton process. Flux computed from our model have been used to compare with the observed X-ray/ gamma-ray spectra of the MeV luminous blazars. Details of the model are discussed in the present paper.

Ghosh, K. K.↗

NASA NTRS1981

Applications of digital image analysis capability in Idaho

The use of digital image analysis of LANDSAT imagery in water resource assessment is discussed. The data processing systems employed are described. The determination of urban land use conversion of agricultural land in two southwestern Idaho counties involving estimation and mapping of crop types and of irrigated land is described. The system was also applied to an inventory of irrigated cropland in the Snake River basin and establishment of a digital irrigation water source/service area data base for the basin. Application of the system to a determination of irrigation development in the Big Lost River basin as part of a hydrologic survey of the basin is also described.

Johnson, K. A.↗

NASA NTRS2015

A Heuristic Approach to Correlating ERAM Flight Data from Twenty Centers

Among its many other functions, the Federal Aviation Administration’s En Route Automation Modernization (ERAM) provides external systems with real-time air traffic data for flights in enroute airspace in the National Airspace System. It replaced the En Route Host computer and backup system used at 20 FAA Air Route Traffic Control Centers (Centers) nationwide. Among the new features of ERAM, its output data stream of flight plan and track data includes a unique identifier for a flight originating in any one of the 20 ERAM Centers. The unique identifier, called the Global Unique Flight Identifier (GUFI), is persistent across all the Centers that track the flight. However, certain factors make it difficult to correlate data using the GUFI. First, the value of the GUFI is only unique within a time window of seven days. Second, the GUFI is attached only to flight-plan related data messages. Finally, track positions reported by ERAM do not reference the GUFI. In order to correlate historical as well as real time flight-plan and position related ERAM data, an efficient, heuristic approach was developed, and a prototype was developed. The approach showed that the processing speed, through parallel processing, is sufficient to correlate ERAM data in real-time. As described in this paper, when there are multiple track positions reported from multiple Centers within a few seconds, each position is assigned with a weighted score to indicate the quality of the position relative to its last know position. The weighted score can be used to eliminate potentially duplicate track positions. The approach is database-agnostic, and can be implemented in a Big Data system such as an Apache Hadoop system, as well as in traditional database systems.

Correlating ERAM Flight Data↗

DOE OSTI2020

Global gross primary production simulated by three terrestrial biosphere models to analyse smoothing of limiting rates

Processed global terrestrial gross primary production (GPP) from three terrestrial biosphere models (TBMs). Two simulations were run with each model, one that selects the rate limiting process of photosynthesis based on the minimum of the potential rates (following Farquhar et al. 1980) and the other that uses quadratic smoothing between the potential rates (following Collatz et al. 1991). The three TBMs are: 1) the Energy Exascale Earth System Model (E3SM) land model (ELM) (release: v1.1.0) (Burrows et al., 2020), a coupled carbon, nitrogen, and phosphorus model with sun/shade big-leaf canopy photosynthesis scaling. 2) The Functionally Assembled Terrestrial Ecosystem Simulator (FATES, tag: sci.1.30.0_api.8.0.0) (Koven et al., 2019), coupled with the Community Land Model (CLM, version 5; Lawrence et al., 2019), a carbon-only vegetation demography model with multi-leaf and multi-canopy layers for scaling photosynthesis, and with a leaf area index (LAI) optimisation scheme. 3) The Sheffield Dynamic Global Vegetation Model (SDGVM, tag: Walkeretal2020_GCB), a carbon-only model with multi-leaf-layer canopy photosynthesis scaling and also with an LAI optimisation scheme (Walker et al., 2017; Woodward & Lomas, 2004). These data were generated to evaluate the effect of Collatz smoothing of limiting rates on simulated GPP across a range of terrestrial biosphere models. Analysis of results can be found in Walker et al. (2020). This dataset is comprised of R scripts to process data and produce figures and .RDS files, a binary format specific to R and read with the 'readRDS' function.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2024

Using Likwid and Byfl to Benchmark Hardware Performance

This paper outlines a benchmarking study conducted during my internship at LANL, focusing on CPU (Computer Processing Unit) and program performance assessment. The primary goal was to gather memory access data using three methods across five polybench kernels The data gathered would then be used to compare and contrast to one another and calculate operational intensity for performance comparisons. Benchmarking tools like Byfl and Likwid were employed, with Byfl offering hardware-independent data through LLVM compiler communication and Likwid directly interacting with computer hardware. The study considered various benchmarking factors, including optimization levels, Big O notation ((n)), CPU diversity and specific kernel equations. Big O notation was utilized to simplify code complexity, with detailed breakdwons of operations and memory components for each polybench application. Specific O(n) equations enabled nuanced kernel compariosns, facilitating the identification of performance variations. CPU efficiency assessments were conducted using Likwid tests on two CPUs. The central focus on code optimization aimed at achieving higher speeds and reduced memory usage through streamlined code. Future work propsoes creating a roofline model, synthesizing benchmarking data into a comprehensive data graph to assist in optimizing code and improving hardware performance. The potential impact on the laboratory or national mission was underscored, emphasizing the importance of optimizing applications and hardware to conserve resources and accelerate program execution. The specific relevance to LANL’s operations in math-intensive fields such as Nuclear Fission, Space Exploration, and Nanotechnology highlights the necessity of efficient benchmarking for resource conservation and proram speed. Overall, this study contributes to the understanding of CPU and program performance, providing insights for future optimization efforts in a laboratory setting

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2002

Seeing Red and Shooting Blanks: A Study of Red Quasars and Blank Field X-Ray Sources

We have identified a population of 'blank field sources' (or 'blanks') among the ROSAT (Roentgen Satellite) bright unidentified X-ray sources with faint optical counterparts. The extreme X-ray over optical flux ratio of blank field sources is not compatible with the main classes of X-ray emitters except for extreme BL Lacertae objects at fx/fv is equal to or less than 35. From the analysis of ROSAT archival data we found evidence for only three sources, out of 16, needing absorption in excess of the Galactic value and no indication of variability. We also found evidence for an extended nature for only one of the five blanks with a serendipitous HRI (High Resolution Imager) detection; this source (1WGA J1226.9+3332) was confirmed as a z=0.89 cluster of galaxies. Palomar images reveal the presence of a red (O - E is equal to or greater than 2) counterpart in the X-ray error circle for six blanks. The identification process brought to the discovery of another high z cluster of galaxies, one (possibly extreme) BL Lac and two apparently normal type 1 AGNs (Active Galactic Nuclei). These AGNs, together with four more AGN-like objects seem to form a well defined group: they present type 1 X-ray spectra but red Palomar counterparts. We discuss the possible explanations for the discrepancy between the X-ray and optical data, among which: a suppressed big blue bump emission, an extreme dust to gas (approximately 40 - 60 the Galactic ratio) ratio value and a high redshift (z is greater than or equal to 3.5) QSO (Quasi-Stellar Object) nature. These AGN-like blanks seem to be the bright (and easier to study) analogs of the sources which are being found in deep Chandra observations. Five more blanks have a still an unknown nature.

Elvis, Martin↗

NASA NTRS2019

SERVIR: Leveraging the Expertise of a Space Agency and a Development Agency to Increase Impact of Earth Observation in the Developing World

SERVIR is a joint initiative of the National Aeronautics and Space Administration (NASA) and the U.S. Agency for International Development (USAID), in collaboration with leading technical organizations around the world-- called SERVIR hubs--that serve and empower developing countries to use satellite data addressing critical challenges in food security and agriculture; water and water-related disasters; land cover, land use and ecosystems; and weather and climate. Over the past fourteen years, the program has worked with stakeholders in 50 countries across the world, partnered with 390 institutions, and generated and shared over 70 products from 27 satellites and sensors. In that process, around 7,400 specialists have been trained in the application of Earth observation data and technology. In its lifetime, SERVIR has been agile and innovative in shifting from what was essentially an incubator for testing and deploying Earth observation science and technology to making co-development the hallmark of its work, exemplified by both South-South and North-South scientific collaborations. SERVIR’s approach has embodied the concept that to solve really big problems, big, creative solutions are needed. SERVIR represents the world working together to address environmental challenges using spaced-based and geospatial technologies. Aligning with the very meaning of SERVIR, i.e. “to serve,” the program continues to be demand-driven in developing and deploying services (versus one-off products) which address development challenges using geospatial tools and Earth observation science. In 2016, as part of SERVIR’s evolution, USAID and NASA released the ‘SERVIR Service Planning Toolkit,’ a guidance document which provides a framework for how geospatial services can be used to tackle development challenges in a sustained manner. Since then, the Service Planning Toolkit’s systematic approach has begun to catch on in other Earth observation efforts. To improve access and use, SERVIR launched a Service Catalogue in February 2019, a searchable collection of demand-driven geospatial services that use Earth observations to support decision making. SERVIR implementing hub partners –include SERVIR-West Africa at the Agrometeorology, Hydrology and Meteorology (AGRHYMET) Regional Center, in Niamey, Niger; SERVIR-Eastern & Southern Africa at the Regional Centre for Mapping of Resources for Development in Nairobi, Kenya; SERVIR-Hindu Kush Himalaya at the International Centre for Integrated Mountain Development in Kathmandu, Nepal; SERVIR-Mekong at the Asian Disaster Preparedness Center in Bangkok, Thailand; and SERVIR’ -Amazonia, at the International Center for Tropical Agriculture (CIAT) in Cali, Colombia.

Searby, Nancy D.↗

NASA NTRS1993

RAID 7 disk array

Each RAID level reflects a different design architecture. Associated with each is a backdrop of imposed limitations, as well as possibilities which may be exploited within the architectural constraints of that level. There are three unique features that differentiate RAID 7 from all other levels. RAID 7 is asynchronous with respect to usage of I/O data paths. Each I/O drive (includes all data and one parity drives) as well as each host interface (there may be multiple host interfaces) has independent control and data paths. This means that each can be accessed completely, independently, of the other. This is facilitated by a separate device cache for each device/interface as well. RAID 7 is asynchronous with respect to device hierarchy and data bus utilization. Each drive and each interface is connected to a high speed data bus controlled by the embedded operating system to make independent transfers to and from central cache. RAID 7 is asynchronous with respect to the operation of an embedded real time process oriented operating system. This means that exclusive and independent of the host, or multiple host paths, the embedded OS manages all I/O transfers asynchronously across the data and parity drives. A key factor to consider is that of the RAID 7's ability to anticipate and match host I/O usage patterns. This yields the following benefits over RAID's built around micro-code based architectures. RAID 7 appears to the host as a normally connected Big Fast Disk (BFD). RAID 7 appears, from the perspective of the individual disk devices, to minimize the total number of accesses and optimize read/write transfer requests. RAID 7 smoothly integrates the random demands of independent users with the principles of spatial and temporal locality. This optimizes small, large, and time sequenced I/O requests which results in users having an I/O performance which approaches performance to that of main memory.

Stout, Lloyd↗

DOE OSTI2022

An Interactive Visualization Tool for Large-Scale Building Stock Modeling: Preprint

Recent advancements in data science and high-performance computing are making it easier to run millions of building simulations, but meaningful visualization of such large datasets remains a challenge. This paper presents a new tool developed to view the results of large-scale OpenStudio simulations of national, regional, or local building stocks. The tool processes millions of simulations to calculate measure savings, utility bills, carbon emissions, primary energy, and cost-effectiveness metrics at a high geographic resolution. Interactive visualizations of the building characteristics, consumption, and measure savings data include proportional symbol maps and histogram plots and can be filtered by any building characteristic.

big data↗

DOE OSTI2022

A Multi-Branch Decoder Network Approach to Adaptive Temporal Data Selection and Reconstruction for Big Scientific Simulation Data

A key challenge in scientific simulation is that the simulation outputs often require intensive I/O and storage space to store the results for effective post hoc analysis. This article focuses on a quality-aware adaptive temporal data selection and reconstruction problem where the goal is to adaptively select simulation data samples at certain key timesteps in situ and reconstruct the discarded samples with quality assurance during post hoc analysis. This problem is motivated by the limitation of current solutions that a significant amount of simulation data samples are either discarded or aggregated during the sampling process, leading to inaccurate modeling of the simulated phenomena. Two unique challenges exist: 1) the sampling decisions have to be made in situ and adapted to the dynamics of the complex scientific simulation data; 2) the reconstruction error must be strictly bounded to meet the application requirement. To address the above challenges, we develop DeepSample , an error-controlled convolutional neural network framework, that jointly integrates a set of coherent multi-branch deep decoders to effectively reconstruct the simulation data with rigorous quality assurance. The results on two real-world scientific simulation applications show that DeepSample significantly outperforms other state-of-the-art methods on both sampling efficiency and reconstructed simulation data quality.

Zhang, Yang↗

DOE OSTI2019

Provenance–aware workflow for data quality management and improvement for large continuous scientific data streams

Data quality assessment, management and improvement is integral part of any big data intensive scientific research to ensure accurate, reliable and reproducible science discoveries. The task of maintaining the quality of data, however, is non-trivial and pose a challenge for a program like Department of Energy’s Atmospheric Radiation Measurement (ARM) that collects data from hundreds of instruments across the world,and distributes thousands of streaming data products that are continuously growing in near-real-time in an archive 1.7 Petabyte in size and growing. We present a computational data processing workflow to collect the data quality issue via an easy and intuitive web-based portal that allows reporting of any quality issues for any site, facility or instruments at a granularity down to individual variable in the data files. Portal allows instrument specialists and scientists to provide corrective actions in form of symbolic equation. A parallel processing framework applies the data improvement to large volume of data in efficient parallel environment, while optimizing data transfer and file I/O operations. Corrected files are systematically versioned and archived. A provenance tracking module tracks and records any change made to the data during its entire life cycle which are communicated transparently to the scientific users. Developed in Python using open source technologies, software architecture enables efficient and fast management and improvement of data in an operational data center environment.

Kumar, Jitendra↗

DOE OSTI2021

Parallel I/O Evaluation Techniques and Emerging HPC Workloads: A Perspective

Emerging workloads such as artificial intelligence, big data analytics and complex multi-step workflows alongside future exascale applications are anticipated future HPC workloads, which will result in a more diverse I/O system workload and even less predictable I/O behavior and access patterns. Along with the ever increasing gap between the compute and storage performance capabilities, the in-depth understanding of extreme-scale I/O behavior and the I/O performance modeling and prediction are essential tools of the large-scale I/O evaluation process for addressing the needs of extreme-scale hybrid workloads. In this survey article, we focus on the state-of-the-art of the I/O behavior and performance analysis process for HPC systems in a 5-year time window and identify future research challenges.

Neuwirth, Sarah↗

NASA NTRSDate not supplied

NASA’s Prototype Spectral Water Inversion Processor and Emulator (SWIPE): Towards Global Coastal and Inland Water Quality and Algal Biodiversity Monitoring

Degradation of Earth’s inland water resources due to anthropogenic perturbations and climate anomalies at both local and global scales continues to place human health at substantial risk. There is now a growing necessity to develop pragmatic approaches that allow timely and effective extrapolation of local processes, to spatially resolved global products, and to promote operational and sustainable resource policy management. This presentation will provide updates on NASA’s prototype open-source aquatic modeling platform, Spectral Water Inversion Processor and Emulator (SWIPE), which is a comprehensive, multi-faceted modeling platform for both forward and inverse modeling of diverse aquatic ecosystems from the benthos to top-of-atmosphere (TOA). SWIPE provides a cohesive application which leverages recent advancements in particle modeling, Big Data analytics, and machine learning to develop a high-fidelity synthetic training ground for sensitivity studies and algorithm development for multispectral or upcoming hyperspectral missions. Some of the prominent features of SWIPE to be discussed include: 1. Advanced hyperspectral modeling of globally diverse algal and non-algal particles using a novel two-layer coated sphere scattering model and radiative transfer modeling, 2. Massive, highly detailed synthetic spectral libraries of Analysis-Ready-Data (ARD) which include spectral libraries of particle microphysics, water biogeophysical and optical properties, as well as surface and TOA reflectances at 1 nm resolution, 3. An ensemble of pre-built analytic, machine learning, and deep learning inversion algorithms for various water quality and biodiversity related retrieval parameters and uncertainty quantification, 4. Sensor-agnostic water quality inversion at wide ranging spatial and spectral resolutions including a codebase for seamless application in the Google Earth Engine and NASA Earth Exchange (NEX) for planetary scale analysis. SWIPE will be a fully open-source platform based in python with comprehensive documentation, tutorials, and options for distributed computing on high performance computing clusters or on single, local machines. Further, we will discuss how we envision SWIPE contributing towards a global analysis of coastal and inland water quality dynamics.

top-of-atmosphere (TOA)↗

DOE OSTI2019

Using Big Data Technologies for HEP Analysis

The HEP community is approaching an era were the excellent performances of the particle accelerators in delivering collision at high rate will force the experiments to record a large amount of information. The growing size of the datasets could potentially become a limiting factor in the capability to produce scientific results timely and efficiently. Recently, new technologies and new approaches have been developed in industry to answer to the necessity to retrieve information as quickly as possible to analyze PB and EB datasets. Providing the scientists with these modern computing tools will lead to rethinking the principles of data analysis in HEP, making the overall scientific process faster and smoother. In this paper, we are presenting the latest developments and the most recent results on the usage of Apache Spark for HEP analysis. The study aims at evaluating the efficiency of the application of the new tools both quantitatively, by measuring the performances, and qualitatively, focusing on the user experience. The first goal is achieved by developing a data reduction facility: working together with CERN Openlab and Intel, CMS replicates a real physics search using Spark-based technologies, with the ambition of reducing 1 PB of public data in 5 hours, collected by the CMS experiment, to 1 TB of data in a format suitable for physics analysis. The second goal is achieved by implementing multiple physics use-cases in Apache Spark using as input preprocessed datasets derived from official CMS data and simulation. By performing different end-analyses up to the publication plots on different hardware, feasibility, usability and portability are compared to the ones of a traditional ROOT-based workflow.

Cremonesi, Matteo↗