SEARCH · Search NASA

Results for “sparse data set”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

All sources NASA NTRS DOE OSTI

DOE OSTI2019

A sparse Gaussian sigmoid basis function approximation of hyperspectral data for detection of solids

We define a new characterization of emissivity and reflectance curves for compositional exploitation of hyperspectral data. Here, our method decomposes each spectrum into a sparse set of Gaussian sigmoid components using penalized regression. Detection is based on the combination of Gaussian sigmoid components unique to a target material. Focusing on the presence of spectral upslopes and downslopes rather than spectral correlations makes detection more robust to both target variation and spectral variability from atmosphere and background encountered during the collection process. We present simulation studies that demonstrate the potential to reduce false positive rates without compromising sensitivity. Characterization of long–wave infrared (LWIR) experimental data validates our method using minerals of different particle sizes, measurement angles, and collection conditions.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Applying Waveform Correlation to Reduce Seismic Analyst Workload Due to Repeating Mining Blasts

Agencies that monitor for underground nuclear tests are interested in techniques that automatically characterize mining blasts to reduce the human analyst effort required to produce high–quality event bulletins. Waveform correlation is effective in finding similar waveforms from repeating seismic events, including mining blasts. We report the results of an experiment to detect and identify mining blasts for two regions, Wyoming (U.S.A.) and Scandinavia, using waveform templates recorded by multiple International Monitoring System stations of the Preparatory Commission for the Comprehensive Nuclear–Test–Ban Treaty Organization (CTBTO PrepCom) for up to 10 yr prior to the time of interest. We discuss approaches for template selection, threshold setting, and event detection that are specialized for characterizing mining blasts using a sparse, global network. We apply the approaches to one week of data for each of the two regions to evaluate the potential for establishing a set of standards for waveform correlation processing of mining blasts that can be generally applied to operational monitoring systems with a sparse network. Here, we compare candidate events detected with our processing methods to the Reviewed Event Bulletin of the International Data Centre to assess potential reduction in analyst workload.

58 GEOSCIENCES↗

NASA NTRS2016

Soil Moisture Active Passive Mission L4_SM Data Product Assessment (Version 2 Validated Release)

During the post-launch SMAP calibration and validation (Cal/Val) phase there are two objectives for each science data product team: 1) calibrate, verify, and improve the performance of the science algorithm, and 2) validate the accuracy of the science data product as specified in the science requirements and according to the Cal/Val schedule. This report provides an assessment of the SMAP Level 4 Surface and Root Zone Soil Moisture Passive (L4_SM) product specifically for the product's public Version 2 validated release scheduled for 29 April 2016. The assessment of the Version 2 L4_SM data product includes comparisons of SMAP L4_SM soil moisture estimates with in situ soil moisture observations from core validation sites and sparse networks. The assessment further includes a global evaluation of the internal diagnostics from the ensemble-based data assimilation system that is used to generate the L4_SM product. This evaluation focuses on the statistics of the observation-minus-forecast (O-F) residuals and the analysis increments. Together, the core validation site comparisons and the statistics of the assimilation diagnostics are considered primary validation methodologies for the L4_SM product. Comparisons against in situ measurements from regional-scale sparse networks are considered a secondary validation methodology because such in situ measurements are subject to up-scaling errors from the point-scale to the grid cell scale of the data product. Based on the limited set of core validation sites, the wide geographic range of the sparse network sites, and the global assessment of the assimilation diagnostics, the assessment presented here meets the criteria established by the Committee on Earth Observing Satellites for Stage 2 validation and supports the validated release of the data. An analysis of the time average surface and root zone soil moisture shows that the global pattern of arid and humid regions are captured by the L4_SM estimates. Results from the core validation site comparisons indicate that "Version 2" of the L4_SM data product meets the self-imposed L4_SM accuracy requirement, which is formulated in terms of the ubRMSE: the RMSE (Root Mean Square Error) after removal of the long-term mean difference. The overall ubRMSE of the 3-hourly L4_SM surface soil moisture at the 9 km scale is 0.035 cubic meters per cubic meter requirement. The corresponding ubRMSE for L4_SM root zone soil moisture is 0.024 cubic meters per cubic meter requirement. Both of these metrics are comfortably below the 0.04 cubic meters per cubic meter requirement. The L4_SM estimates are an improvement over estimates from a model-only SMAP Nature Run version 4 (NRv4), which demonstrates the beneficial impact of the SMAP brightness temperature data. L4_SM surface soil moisture estimates are consistently more skillful than NRv4 estimates, although not by a statistically significant margin. The lack of statistical significance is not surprising given the limited data record available to date. Root zone soil moisture estimates from L4_SM and NRv4 have similar skill. Results from comparisons of the L4_SM product to in situ measurements from nearly 400 sparse network sites corroborate the core validation site results. The instantaneous soil moisture and soil temperature analysis increments are within a reasonable range and result in spatially smooth soil moisture analyses. The O-F residuals exhibit only small biases on the order of 1-3 degrees Kelvin between the (re-scaled) SMAP brightness temperature observations and the L4_SM model forecast, which indicates that the assimilation system is largely unbiased. The spatially averaged time series standard deviation of the O-F residuals is 5.9 degrees Kelvin, which reduces to 4.0 degrees Kelvin for the observation-minus-analysis (O-A) residuals, reflecting the impact of the SMAP observations on the L4_SM system. Averaged globally, the time series standard deviation of the normalized O-F residuals is close to unity, which would suggest that the magnitude of the modeled errors approximately reflects that of the actual errors. The assessment report also notes several limitations of the "Version 2" L4_SM data product and science algorithm calibration that will be addressed in future releases. Regionally, the time series standard deviation of the normalized O-F residuals deviates considerably from unity, which indicates that the L4_SM assimilation algorithm either over- or under-estimates the actual errors that are present in the system. Planned improvements include revised land model parameters, revised error parameters for the land model and the assimilated SMAP observations, and revised surface meteorological forcing data for the operational period and underlying climatological data. Moreover, a refined analysis of the impact of SMAP observations will be facilitated by the construction of additional variants of the model-only reference data. Nevertheless, the “Version 2” validated release of the L4_SM product is sufficiently mature and of adequate quality for distribution to and use by the larger science and application communities.

SMAP L4_SM↗

NASA NTRS1997

Applications of Data Assimilation to Analysis of the Ocean on Large Scales

It is commonplace to begin talks on this topic by noting that oceanographic data are too scarce and sparse to provide complete initial and boundary conditions for large-scale ocean models. Even considering the availability of remotely-sensed data such as radar altimetry from the TOPEX and ERS-1 satellites, a glance at a map of available subsurface data should convince most observers that this is still the case. Data are still too sparse for comprehensive treatment of interannual to interdecadal climate change through the use of models, since the new data sets have not been around for very long. In view of the dearth of data, we must note that the overall picture is changing rapidly. Recently, there have been a number of large scale ocean analysis and prediction efforts, some of which now run on an operational or at least quasi-operational basis, most notably the model based analyses of the tropical oceans. These programs are modeled on numerical weather prediction. Aside from the success of the global tide models, assimilation of data in the tropics, in support of prediction and analysis of seasonal to interannual climate change, is probably the area of large scale ocean modeling and data assimilation in which the most progress has been made. Climate change is a problem which is particularly suited to advanced data assimilation methods. Linear models are useful, and the linear theory can be exploited. For the most part, the data are sufficiently sparse that implementation of advanced methods is worthwhile. As an example of a large scale data assimilation experiment with a recent extensive data set, we present results of a tropical ocean experiment in which the Kalman filter was used to assimilate three years of altimetric data from Geosat into a coarsely resolved linearized long wave shallow water model. Since nonlinear processes dominate the local dynamic signal outside the tropics, subsurface dynamical quantities cannot be reliably inferred from surface height anomalies. Because of its potential for large scale synoptic coverage of the deep ocean, acoustic travel time data should be a natural complement to satellite altimetry. Satellite data give us vertical integrals associated with thermodynamic and dynamic processes.

Miller, Robert N.↗

NASA NTRS1974

Global detailed geoid computation and model analysis

Comparisons and analyses were carried out through the use of detailed gravimetric geoids which we have computed by combining models with a set of 26,000 1 deg x 1 deg mean free air gravity anomalies. The accuracy of the detailed gravimetric geoid computed using the most recent Goddard earth model (GEM-6) in conjunction with the set of 1 deg x 1 deg mean free air gravity anomalies is assessed at + or - 2 meters on the continents of North America, Europe, and Australia, 2 to 5 meters in the Northeast Pacific and North Atlantic areas, and 5 to 10 meters in other areas where surface gravity data are sparse. The R.M.S. differences between this detailed geoid and the detailed geoids computed using the other satellite gravity fields in conjuction with same set of surface data range from 3 to 7 meters.

Marsh, J. G.↗

NASA NTRS1984

Application of VISSR Atmospheric Sounder (VAS) data in weather analysis

A technique which analyzes irregularly spaced satellite data is described. An experiment with rawinsonde and VISSR Atmospheric Sounder (VAS) radiance measurements collected on March 6-7, 1982 is conducted to reveal the applicability of the technique. The rawinsonde data are analyzed on a 16 x 12 grid using the two pass analysis scheme of Barnes (1973). A scheme similar to the Barnes (1973) procedure is employed to produce gridded analysis of VAS data over a 200 x 15000 km region in central part of the U.S. The use of a correction pass on the initial gridded field is described; the technique is extremely effective on uniformly spaced observations. The incorporation of the limited fine mesh model to the scheme to analyze data in sparse and cloudy regions is examined. A comparison of rawinsonde data with VAS data is provided. The technique proves effective for studying cloudy and sparse areas with VAS data and produces a four-dimensional data set with significant mesoscale structure.

Jedlovec, G. J.↗

DOE OSTI2022

Dynamic signal recovery in distribution grids using compressive lossy measurements

Distribution system state estimation requires reliable aggregation of the measured data. However, the large volume of the measured data imposes a significant stress on the underlying communication infrastructure. With the challenges associated with measurement availability, current distribution systems are typically unobservable. To cope with the unobservability issue, compressive sensing theory allows us to recover system state information from a small number of measurements provided the states of the distribution system exhibit sparsity. In this paper, we evaluate the robustness of an updated Kalman filtered modified compressive sensing (KF-ModCS) technique that dynamically estimates the grid states using a small fraction of measured data. In practice, measurements used for sparsity based state estimation may also be intermittent due to communication network induced losses. Further, to understand the effect of packet losses on KF-ModCS, we provide an upper bound for the expected variances of the state estimation error for a given rate of information loss. This upper bound is further improved if the support set of the sparse signal that characterizes the state dynamics does not change over time and/or the reduced model is observable. Simulations based on two practical data sets collected from actual customers in a distribution grid validate the theoretical results.

24 POWER TRANSMISSION AND DISTRIBUTION↗

NASA NTRS2002

Southern Ocean Seasonal Net Production from Satellite, Atmosphere, and Ocean Data Sets

A new climatology of monthly air-sea O2 flux was developed using the net air-sea heat flux as a template for spatial and temporal interpolation of sparse hydrographic data. The climatology improves upon the previous climatology of Najjar and Keeling in the Southern Hemisphere, where the heat-based approach helps to overcome limitations due to sparse data coverage. The climatology is used to make comparisons with productivity derived from CZCS images. The climatology is also used in support of an investigation of the plausible impact of recent global warming an oceanic O2 inventories.

Keeling, Ralph F.↗

DOE OSTI2020

Recursive Dynamic Compressive Sensing in Smart Distribution Systems

With a transition to a smarter grid, we are witnessing a significant growth in smart metering infrastructure and sensor deployment in the distribution system. The underlying communication infrastructure is stressed due to the large volume of data that is generated by the smart meters/sensors. Furthermore, real time operations such as state estimation and control are impaired due to the lack of reliable aggregation of the data. In this paper, we exploit the underlying sparsity in grid data to implement two recursive dynamic compressive sensing (CS) approaches-streaming modified weighted-`1 CS and Kalman filtered CS. These approaches aim to reconstruct the sparse signal using the current underdetermined measurements and the prior information about the sparse signal and its support set. Slow signal and support change is in distribution grid data is validated using Pecan Street data. Both the IEEE 34 node test feeder system and PecanStreet data are considered as two examples to validate the superior performance of the two recursive CS techniques relative to classic CS.

Karimi, Hazhar Sufi↗

DOE OSTI2022

Bayesian-based response expansion and uncertainty quantification using sparse measurement sets

Systems subjected to dynamic loads often require monitoring of their vibrational response, but limitations on the total number and placement of the measurement sensors can hinder the data-collection process. Here, we present an indirect approach to estimate a system’s full-field dynamic response, including all uninstrumented locations, using response measurements from sensors sparsely located on the system. This approach relies on Bayesian inference that utilizes a system model to estimate the full-field response and quantify the uncertainty in these estimates. By casting the estimation problem in the frequency domain, this approach utilizes the modal frequency response functions as a natural, frequency-dependent weighting scheme for the system mode shapes to perform the expansion. This frequency-dependent weighting scheme enables an accurate expansion, even with highly correlated mode shapes that may arise from spatial aliasing due to the limited number of sensors, provided these correlated modes do not have natural frequencies that are closely spaced. Furthermore, the inherent regularization mechanism that arises in this Bayesian-based procedure enables the utilization of the full set of system mode shapes for the expansion, rather than any reduced subset. This approach can produce estimates when considering a single realization of the measured responses, and with some modification, it can also produce estimates for power spectral density matrices measured from many realizations of the responses from statistically stationary random processes. A simply supported beam provides an initial numerical validation, and a cylindrical test article excited by acoustic loads in a reverberation chamber provides experimental validation.

42 ENGINEERING↗

NASA NTRS1988

Integration of space and in situ observations to study global climate change

The use of model-based global data sets of atmospheric circulation for studying fundamental dynamical and physical processes is discussed, focusing on limitations of the available model-based data sets. Data from the Global Weather Experiment in 1979 were analyzed by two authorized level IIIb data centers in 1980 and in 1981. The analyses led to difference in data-sparse regions such as the tropics. Study areas which can be addressed by an internally-consistent long-term multivariate data set for the atmospheric circulation are considered, including mean climate, forcing for the ocean models, global hydrological cycle, atmospheric energetics, intraseasonal variability, land surface processes, and structure and variability of vertical velocity, divergence, and diabatic heating. It is concluded that the most comprehensive technique for integrating space and in situ observations to produce this type of data set would be a four-dimensional data assimilation system with a realistic physical model of the type employed in operational numerical weather prediction.

Bengtsson, L.↗

DOE OSTI2023

Development of the Senseiver for efficient field reconstruction from sparse observations

Abstract The reconstruction of complex time-evolving fields from sensor observations is a grand challenge. Frequently, sensors have extremely sparse coverage and low-resource computing capacity for measuring highly nonlinear phenomena. While numerical simulations can model some of these phenomena using partial differential equations, the reconstruction problem is ill-posed. Data-driven-strategies provide crucial disambiguation, but these suffer in cases with small amounts of data, and struggle to handle large domains. Here we present the Senseiver, an attention-based framework that excels in reconstructing complex spatial fields from few observations with low overhead. The Senseiver reconstructs n -dimensional fields by encoding arbitrarily sized sparse sets of inputs into a latent space using cross-attention, producing uniform-sized outputs regardless of the number of observations. This allows efficient inference by decoding only a sparse set of output observations, while a dense set of observations is needed to train. This framework enables training of data with complex boundary conditions and extremely large fine-scale simulations. We build on the Perceiver IO by enabling training models with fewer parameters, which facilitates field deployment, and a training framework that allows a flexible number of sensors as input, which is critical for real-world applications. We show that the Senseiver advances the state-of-the-art of field reconstruction in many applications.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Non-linear relationships between daily temperature extremes and US agricultural yields uncovered by global gridded meteorological datasets

Global agricultural commodity markets are highly integrated among major producers. Prices are driven by aggregate supply rather than what happens in individual countries in isolation. Furthermore, estimating the effects of weather-induced shocks on production, trade patterns and prices hence requires a globally representative weather data set. Recently, two data sets that provide daily or hourly records, GMFD and ERA5-Land, became available. Starting with the US, a data rich region, we formally test whether these global data sets are as good as more fine-scaled country-specific data in explaining yields and whether they estimate similar response functions. While GMFD and ERA5-Land have lower predictive skill for US corn and soybeans yields than the fine-scaled PRISM data, they still correctly uncover the underlying non-linear temperature relationship. All specifications using daily temperature extremes under any of the weather data sets outperform models that use a quadratic in average temperature. Correctly capturing the effect of daily extremes has a larger effect than the choice of weather data. In a second step, focusing on Sub Saharan Africa, a data sparse region, we confirm that GMFD and ERA5-Land have superior predictive power to CRU, a global weather data set previously employed for modeling climate effects in the region.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2022

Stellar kinematics of dwarf galaxies from multi-epoch spectroscopy: application to Triangulum II

ABSTRACT We present new MMT/Hectochelle spectroscopic measurements for 257 stars observed along the line of sight to the ultrafaint dwarf galaxy Triangulum II (Tri II). Combining results from previous Keck/DEIMOS spectroscopy, we obtain a sample that includes 16 likely members of Tri II, with up to 10 independent redshift measurements per star. To this multi-epoch kinematic data set, we apply methodology that we develop in order to infer binary orbital parameters from sparsely sampled radial velocity curves with as few as two epochs. For a previously identified (spatially unresolved) binary system in Tri II, we infer an orbital solution with period $296.0_{-3.3}^{+3.8} \rm ~ d$, semimajor axis $1.12^{+0.41}_{-0.24}\rm ~au$, and systemic velocity $-380.0 \pm 1.7 \rm ~km ~s^{-1}$ that we then use in the analysis of Tri II’s internal kinematics. Despite this improvement in the modelling of binary star systems, the current data remain insufficient to resolve the velocity dispersion of Tri II. We instead find a 95 per cent confidence upper limit of $\sigma _{v} \lesssim 3.4 \rm ~km~s^{-1}$.

Buttry, Rachel (ORCID:0000000155017609)↗

DOE OSTI2023

Recipes for when physics fails: recovering robust learning of physics informed neural networks

Abstract Physics-informed neural networks (PINNs) have been shown to be effective in solving partial differential equations by capturing the physics induced constraints as a part of the training loss function. This paper shows that a PINN can be sensitive to errors in training data and overfit itself in dynamically propagating these errors over the domain of the solution of the PDE. It also shows how physical regularizations based on continuity criteria and conservation laws fail to address this issue and rather introduce problems of their own causing the deep network to converge to a physics-obeying local minimum instead of the global minimum. We introduce Gaussian process (GP) based smoothing that recovers the performance of a PINN and promises a robust architecture against noise/errors in measurements. Additionally, we illustrate an inexpensive method of quantifying the evolution of uncertainty based on the variance estimation of GPs on boundary data. Robust PINN performance is also shown to be achievable by choice of sparse sets of inducing points based on sparsely induced GPs. We demonstrate the performance of our proposed methods and compare the results from existing benchmark models in literature for time-dependent Schrödinger and Burgers’ equations.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

Prong Segmentation using Point Set Transformers in Multiple View Neutrino Detectors

NOvA is a long-baseline neutrino experiment studying neutrino oscillations by detecting neutrinos from the NuMI beam at Fermilab. Its physics analysis relies on accurate prong segmentation, which involves matching each hit to its source particle and identifying the particle type. This task has commonly been addressed using a combination of traditional clustering algorithms and convolutional neural networks (CNNs). However, NOvA’s detector design presents data as two sparse and decoupled 2D images (XZ and YZ views) rather than a native 3D representation, posing a significant challenge for traditional CNN-based models. In this talk, we propose a novel neural network based on the Point Set Transformer. By treating detector hits as sparse point clouds and implementing a cross-view attention mechanism, our model enables efficient information mixing between both views. Evaluated on NOvA simulated data, our model achieves superior accuracy while requiring significantly fewer computational resources compared to other models. Furthermore, the model demonstrates great performance when applied to Liquid Argon Time Projection Chamber (LArTPC) data, which shows its potential as a universal prong segmentation algorithm for multiple view neutrino detectors.

Liu, Jiaxi [UC, Irvine]↗

NASA NTRS1997

The Influence of Correlated Crustal Signals in Modelling the Main Geomagnetic Field

Algorithms used in geomagnetic main-field modelling have for the most part treated the noise in the field measurements as if it were white. A major component of the noise consists of the field due to magnetization in the crust and it has been realized for some time that such signals are highly correlated at satellite altitude. Hence approximation by white noise, while of undoubted utility, is of unknown validity. In this paper we study two plausible statistical models for the crustal magnetization, in which the magnetization is a realization of a stationary, isotropic, random process. At a typical satellite altitude the associated fields exhibit significant correlation over ranges as great as 15 deg. or more, which introduces off-diagonal elements into the covariance matrix, elements that have usually been neglected in modelling procedures. Dealing with a full covariance matrix for a large data set would present a formidable computational challenge, but fortunately most of the entries in the covariance matrix are so small that they can be replaced by zeros. The resultant matrix comprises only about 3 per cent non-zero entries and thus we can take advantage of efficient sparse matrix techniques to solve the numerical system. We construct several main-field models based on vertical-component data from a selected 5 deg. by 5 deg. data set derived from the Magsat mission. Models with and without off-diagonal terms are compared.

Rygaard-Hjalsted, C.↗

DOE OSTI2020

A Satellite-Derived Upper-Ocean Stratification Data Set for the Tropical North Atlantic With Potential Applications for Hurricane Intensity Prediction

Upper-ocean stratification strongly impacts vertical mixing and the heat flux between the ocean and atmosphere, especially under extreme conditions of tropical cyclones (TCs). Knowledge of prestorm stratification is important for accurate TC intensity prediction. In situ observations of the tropical ocean have significantly increased in the past decade. However, they are still too sparse to resolve ocean stratification variability in near-real time and on small spatial scales. Here, in this study, based on long-term observations and an ocean reanalysis data set from 2004–2017, we investigate the possibility of retrieving upper-ocean stratification from sea surface temperature (SST), sea surface salinity (SSS), and sea surface height (SSH) using a simple regression method. It is found that more than 90% of the mean seasonal cycle and about 30% to 80% of temperature and salinity stratification anomalies can be reconstructed using surface data from either observations or an ocean reanalysis. Simple regression can be used with satellite observations to create a high-resolution, near-real-time-gridded ocean stratification data set that successfully reproduces both the large and mesoscale variability of ocean stratification. When used in a simple expression for TC-induced SST cooling, the satellite-derived stratification shows improvements over an ocean analysis in terms of variance explained of SST cooling, offering promise as a near-real-time indicator of the ocean's impact on TC intensification.

54 ENVIRONMENTAL SCIENCES↗