Search NASA⌕ Search

SEARCH · Search NASA

Results for “data corruption”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

Analysis of signals under compositional noise with applications to SONAR data

In this paper, we consider the problem of denoising and classification of SONAR signals observed under compositional noise, i.e., they have been warped randomly along the x-axis. The traditional techniques do not account for such noise and, consequently, cannot provide a robust classification of signals. We apply a recent framework that: 1) uses a distance-based objective function for data alignment and noise reduction; and 2) leads to warping-invariant distances between signals for robust clustering and classification. We use this framework to introduce two distances that can be used for signal classification: a) a y-distance, which is the distance between the aligned signals; and b) an x-distance that measures the amount of warping needed to align the signals. We focus on the task of clustering and classifying objects, using acoustic spectrum (acoustic color), which is complicated by the uncertainties in aspect angles at data collections. Small changes in the aspect angles corrupt signals in a way that amounts to compositional noise. As a result, we demonstrate the use of the developed metrics in classification of acoustic color data and highlight improvements in signal classification over current methods.

47 OTHER INSTRUMENTATION↗

Improving the trust in results of numerical simulations and scientific data analytics

This white paper investigates several key aspects of the trust that a user can give to the results of numerical simulations and scientific data analytics. In this document, the notion of trust is related to the integrity of numerical simulations and data analytics applications. This white paper complements the DOE ASCR report on Cybersecurity for Scientific Computing Integrity by (1) exploring the sources of trust loss; (2) reviewing the definitions of trust in several areas; (3) providing numerous cases of result alteration, some of them leading to catastrophic failures; (4) examining the current notion of trust in numerical simulation and scientific data analytics; (5) providing a gap analysis; and (6) suggesting two important research directions and their respective research topics. To simplify the presentation without loss of generality, we consider that trust in results can be lost (or the results’ integrity impaired) because of any form of corruption happening during the execution of the numerical simulation or the data analytics application. In general, the sources of such corruption are threefold: errors, bugs, and attacks. Current applications are already using techniques to deal with different types of corruption. However, not all potential corruptions are covered by these techniques. We firmly believe that the current level of trust that a user has in the results is at least partially founded on ignorance of this issue or the hope that no undetected corruptions will occur during the execution. This white paper explores the notion of trust and suggests recommendations for developing a more scientifically grounded notion of trust in numerical simulation and scientific data analytics. We first formulate the problem and show that it goes beyond previous questions regarding the quality of results such as V&V, uncertainly quantification, and data assimilation. We then explore the complexity of this difficult problem, and we sketch complementary general approaches to address it. This paper does not focus on the trust that the execution will actually complete. The product of simulation or of data analytic executions is the final element of a potentially long chain of transformations, where each stage has the potential to introduce harmful corruptions. These corruptions may produce results that deviate from the user-expected accuracy without notifying the user of this deviation. There are many potential sources of corruption before and during the execution; consequently, in this white paper we do not focus on the protection of the end result after the execution.

97 MATHEMATICS AND COMPUTING↗

Exploiting data representation for fault tolerance

Incorrect computer hardware behavior may corrupt intermediate computations in numerical algorithms, possibly resulting in incorrect answers. Prior work models misbehaving hardware by randomly flipping bits in memory. We start by accepting this premise, and present an analytic model for the error introduced by a bit flip in an IEEE 754 floating-point number. We then relate this finding to the linear algebra concepts of normalization and matrix equilibration. In particular, we present a case study illustrating that normalizing both vector inputs of a dot product minimizes the probability of a single bit flip causing a large error in the dot product's result. Moreover, the absolute error is either less than one or very large, which allows detection of large errors. Then, we apply this to the GMRES iterative solver. We count all possible errors that can be introduced through faults in arithmetic in the computationally intensive orthogonalization phase of GMRES, and show that when the matrix is equilibrated, the absolute error is bounded above by one.

97 MATHEMATICS AND COMPUTING↗

Quantifying the robustness of deep multispectral segmentation models against natural perturbations and data poisoning

In overhead image segmentation tasks, including additional spectral bands beyond the traditional RGB channels can improve model performance. However, it is still unclear how incorporating this additional data impacts model robustness to adversarial attacks and natural perturbations. For adversarial robustness, the additional in-formation could improve the model’s ability to distinguish malicious inputs, or simply provide new attack avenues and vulnerabilities. For natural perturbations, the additional information could better inform model decisions and weaken perturbation effects or have no significant influence at all. In this work, we seek to characterize the performance and robustness of a multispectral (RGB and near infrared) image segmentation model subjected to adversarial attacks and natural perturbations. While existing adversarial and natural robustness research has focused primarily on digital perturbations, we prioritize on creating realistic perturbations designed with physical world conditions in mind. For adversarial robustness, we focus on data poisoning attacks whereas for natural robustness, we focus on extending ImageNet-C common corruptions for fog and snow that coherently and self-consistently perturbs the input data. Overall, we find both RGB and multispectral models are vulnerable to data poisoning attacks regardless of input or fusion architectures and that while physically-realizable natural perturbations still degrade model performance, the impact differs based on fusion architecture and input data.

Deep learning, multispectral images, multimodal fu↗

PROTEUS: Machine Learning Driven Resilience for Extreme-scale Systems

The objective of this project is to design, develop, and evaluate scalable software to enhance resilience, data checkpointing, program restart, and analysis. The proposed tasks are to 1) develop scalable machine learning techniques to learn temporal change patterns in a scalable and in-situ manner, and to minimize data movement and maximize learning locally closest to data; 2) design a concise data representation and indexing mechanism to capture the distribution of changes in data that can guarantee point-wise user-defined tolerable errors while reducing the data storage requirements by an order of magnitude or more; 3) develop data reduction techniques as library modules; 4) exploit local SSD for minimizing data movement in storage hierarchy; 5) develop anomaly detection algorithms that can predict corruptions based on learning of emerging patterns; 6) develop software libraries to be incorporated within widely used data formats and APIs; and 7) evaluate the proposed software using DOE scientific applications. The outcomes of the proposed work are to satisfy many synergistic data reduction and resilience requirements for large-scale data intensive applications executed on extreme-scale computing systems. The developed mechanism for error-bound data approximation is directly applicable to existing scientific applications. Through machine learning from historical events and change distribution, this work will enable anomaly detection for DOE computer facility.

97 MATHEMATICS AND COMPUTING↗

MimicGAN: Robust Projection onto Image Manifolds with Corruption Mimicking

In the past few years, Generative Adversarial Networks (GANs) have dramatically advanced our ability to represent and parameterize high-dimensional, non-linear image manifolds. As a result, they have been widely adopted across a variety of applications, ranging from challenging inverse problems like image completion, to problems such as anomaly detection and adversarial defense. A recurring theme in many of these applications is the notion of projecting an image observation onto the manifold that is inferred by the generator. In this context, Projected Gradient Descent (PGD) has been the most popular approach, which essentially optimizes for a latent vector that minimizes the discrepancy between a generated image and the given observation. However, PGD is a brittle optimization technique that fails to identify the right projection (or latent vector) when the observation is corrupted, or perturbed even by a small amount. Such corruptions are common in the real world, for example images in the wild come with unknown crops, rotations, missing pixels, or other kinds of non-linear distributional shifts which break current encoding methods, rendering downstream applications unusable. To address this, we propose corruption mimicking—a new robust projection technique, that utilizes a surrogate network to approximate the unknown corruption directly at test time, without the need for additional supervision or data augmentation. The proposed method is significantly more robust than PGD and other competing methods under a wide variety of corruptions, thereby enabling a more effective use of GANs in real-world applications. Finally, more importantly, we show that our approach produces state-of-the-art performance in several GAN-based applications—anomaly detection, domain adaptation, and adversarial defense, that benefit from an accurate projection.

97 MATHEMATICS AND COMPUTING↗

Asi Nuclear Energy Sensors Data Portal Chatbot And Data Structuring Tool

The Idaho National Laboratory (INL) is advancing the development of an AI-powered chatbot and data structuring tool specifically designed to accelerate data mining processes for sensor-related information and seamlessly integrate the results into the ASI Sensors Data Portal (https://nes.energy.gov/). By doing so, the software aims to enhance the accessibility, usability, and organization of sensor data for nuclear energy applications. The software initial phase focuses on retrieving comprehensive datasets, prioritizing the past five years of publicly available information from the Office of Scientific and Technical Information (OSTI). These datasets will be meticulously processed to ensure compatibility, employing cleaning and preprocessing steps to eliminate irrelevant, incomplete, or corrupted information, thus establishing a robust foundation for subsequent AI use. The data will serve as the backbone for training an AI model and chatbot, which will act as an interactive tool enabling users to ask complex, context-specific questions and receive accurate, validated answers derived from constrained literature. In parallel, the project incorporates a data structuring process supported by AI to organize sensor information from multiple sources into a standardized format. This structured data will include detailed sensor specifications, such as measurement range, applications, accuracy, and operating conditions, generated and documented with AI. These specifications will be systematically integrated into the sensor portal. To maintain the highest levels of accuracy and relevance, all AI-generated outputs will be reviewed and validated by subject matter experts (SMEs), with additional fields or parameters added as needed. Future stages of the project aim to expand the dataset beyond OSTI to include other sources and potentially incorporate unclassified controlled information (UCI) with restricted access protocols to address security and confidentiality requirements.

Mapes, NormanJ. [Idaho National Laboratory (INL), ↗

Spy the Lie: Detecting Malicious Insiders

Insider threat is a hard problem. There is no ground truth, there are innumerable variables, and the data is sparse. The types of crimes and abuses associated with insider threats are significant; the most serious include espionage, sabotage, terrorism, embezzlement, extortion, bribery, and corruption. Malicious activities include an even broader range of exploits, such as negligent use of classified data, fraud, cybercrime, unauthorized access to sensitive information, and illicit communications with unauthorized recipients. Inadvertent action or inaction without malicious intent (e.g., disposing of sensitive documents incorrectly) can also cause harm to an organization. This review article will explore insider threat, specifically behaviors, beliefs, and current debates within the field. Additionally particular focus is given to deception, a significant behavioral component of the malicious insider. Finally, research and policy implications for law enforcement and the intelligence community are addressed.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

NEON AOP Survey of Upper East River CO Watersheds: LAZ Files, LiDAR Surface Elevation, Terrain Elevation, and Canopy Height Rasters

Lawrence Berkeley National Laboratory (LBNL) contracted the National Ecological Observatory Network Airborne Observation Platform (NEON AOP) to observe watersheds of interest surrounding Crested Butte, CO with remotely sensed data, including LiDAR. The flight box design encompassed the watersheds, surveying a total area of 334 km2 across 72 lines. The instrument used was an Optech Gemini, with a pulse density of 2-9 pulses m-2 across the study area (see final report document for detailed information (Goulden and Musinsky 2020). These LiDAR data are the primary data that were provided by NEON, and include both LiDAR source data and derived surface products. The source data are contained in 'crbu_laz.zip' and 'classified_point_cloud.zip. 'classified_point_cloud.zip' represents an additional processing level on 'crbu_laz.zip'. The derived products are 'dsm_mosaic_min_phase_me.tif', 'dtm_mosaic_min_phase_me.tif', and 'tch_mosaic_min_phase_me.tif'. These files are raster mosaics of a digital surface elevation model (DSM), digital terrain elevation model (DTM), and a canopy height model (CHM) at 1 m resolution with a height threshold of > 2 m, respectively, in geotiff (.tif) format. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231. [UPDATE] On 06/26/2026 the files 'crbu_laz.zip' and 'classified_point_cloud.zip' were replaced with original output files from NEON to address zip file corruption and naming errors. No changes were made to the underlying data. (See Methods Step 2 for details).

2018 NEON and 2025 CHESS Campaigns↗

Leipzig University W-Band Cloud Radar, Gothic (Colorado), SAIL Campaign Second Winter (15.11.2022 - 05.06.2023)

The instrument is a polarimetric W-Band simultaneous transmission simultaneous reception (STSR) Doppler Cloud Radar manufactured by Radiometer Physics GmbH (RPG, instrument type RPG-FMCW-94-DP). It was deployed about 500m away from the ARM M1 facility in Gothic (CO) near the Ore House between Nov 15, 2022 and Jun 5, 2023 at an altitude of 2913 m. The Doppler cloud radar was mounted on a prototype of a cold temperature scanner manufactured by RPG. The dataset contains common radar variables like equivalent radar reflectivity factor, mean Doppler velocity, Doppler spectrum width as well as common polarimetric radar variables like differential phase shift (PhiDP), specific differential phase shift (KDP), correlation coefficient (RHV) and differential reflectivity (ZDR). Between Nov 15, 2022, and Feb 7, 2023 the radar was operated at 40° constant elevation and 151° azimuth (towards the KAZR at the AMF1). Between Feb 7 and Mar 15, 2023 the following scan cycle was repeated in 15min intervals: zenith pointing measurements, RHI scans from 0° to 90° and 90° to 0° elevation at 151° azimuth, constant elevation measurements at 40° elevation and 151° azimuth and PPI scans at 85° elevation for wind retrievals (one PPI per hour). Between Mar 15 18:18 UTC and Apr 8, 2023 11:04 UTC, the scanner was stuck at 72° elevation, which resulted in corrupted files for this period. For that period, the files containing data are uploaded but should be treated with care. Between Apr 8 and Apr 14, 2023 the scanning pattern described above was performed again. Afterwards, except for May 17 and May 18, 2023 where scanning patterns as described above were performed, measurements continued at 40° elevation and 151° azimuth until May 22, 2023 09:00 UTC. The scanning pattern was then started again, but without hourly PPI measurements until May 31, 2023. Between May 31 and Jun 5, 2023 solely zenith-pointing observations were performed. The data provided here are the RPG Level 1 (L1) files, Doppler spectra Level 0 (L0) files are available upon request.

54 ENVIRONMENTAL SCIENCES↗

STUDIES IN ASTRONOMICAL TIME SERIES ANALYSIS. VI. BAYESIAN BLOCK REPRESENTATIONS

This paper addresses the problem of detecting and characterizing local variability in time series and other forms of sequential data. The goal is to identify and characterize statistically significant variations, at the same time suppressing the inevitable corrupting observational errors. We present a simple nonparametric modeling technique and an algorithm implementing it-an improved and generalized version of Bayesian Blocks-that finds the optimal segmentation of the data in the observation interval. The structure of the algorithm allows it to be used in either a real-time trigger mode, or a retrospective mode. Maximum likelihood or marginal posterior functions to measure model fitness are presented for events, binned counts, and measurements at arbitrary times with known error distributions. Problems addressed include those connected with data gaps, variable exposure, extension to piecewise linear and piecewise exponential representations, multivariate time series data, analysis of variance, data on the circle, other data modes, and dispersed data. Simulations provide evidence that the detection efficiency for weak signals is close to a theoretical asymptotic limit derived by Arias-Castro et al. In the spirit of Reproducible Research all of the code and data necessary to reproduce all of the figures in this paper are included as supplementary material.

79 ASTRONOMY AND ASTROPHYSICS↗

IDENTIFYING IONIZED REGIONS IN NOISY REDSHIFTED 21 cm DATA SETS

One of the most promising approaches for studying reionization is to use the redshifted 21 cm line. Early generations of redshifted 21 cm surveys will not, however, have the sensitivity to make detailed maps of the reionization process, and will instead focus on statistical measurements. Here, we show that it may nonetheless be possible to directly identify ionized regions in upcoming data sets by applying suitable filters to the noisy data. The locations of prominent minima in the filtered data correspond well with the positions of ionized regions. In particular, we corrupt semi-numeric simulations of the redshifted 21 cm signal during reionization with thermal noise at the level expected for a 500 antenna tile version of the Murchison Widefield Array (MWA), and mimic the degrading effects of foreground cleaning. Using a matched filter technique, we find that the MWA should be able to directly identify ionized regions despite the large thermal noise. In a plausible fiducial model in which {approx}20% of the volume of the universe is neutral at z {approx} 7, we find that a 500-tile MWA may directly identify as many as {approx}150 ionized regions in a 6 MHz portion of its survey volume and roughly determine the size of each of these regions. This may, in turn, allow interesting multi-wavelength follow-up observations, comparing galaxy properties inside and outside of ionized regions. We discuss how the optimal configuration of radio antenna tiles for detecting ionized regions with a matched filter technique differs from the optimal design for measuring power spectra. These considerations have potentially important implications for the design of future redshifted 21 cm surveys.

79 ASTRONOMY AND ASTROPHYSICS↗

Bi-sparsity pursuit: A paradigm for robust subspace recovery

The success of sparse models in computer vision and machine learning is due to the fact that, high dimensional data is distributed in a union of low dimensional subspaces in many real-world applications. The underlying structure may, however, be adversely affected by sparse errors. In this paper, we propose a bi-sparse model as a framework to analyze this problem, and provide a novel algorithm to recover the union of subspaces in the presence of sparse corruptions. We further show the effectiveness of our method by experiments on real-world vision data.

42 ENGINEERING↗

BI-sparsity pursuit for robust subspace recovery

Here, the success of sparse models in computer vision and machine learning in many real-world applications, may be attributed in large part, to the fact that many high dimensional data are distributed in a union of low dimensional subspaces. The underlying structure may, however, be adversely affected by sparse errors, thus inducing additional complexity in recovering it. In this paper, we propose a bi-sparse model as a framework to investigate and analyze this problem, and provide as a result , a novel algorithm to recover the union of subspaces in presence of sparse corruptions. We additionally demonstrate the effectiveness of our method by experiments on real-world vision data.

47 OTHER INSTRUMENTATION↗

Fitting local, low-dimensional parameterizations of optical turbulence modeled from optimal transport velocity vectors

This work exploits a connection between optimal transport theory and the physics of image propagation to yield a locally low-dimensional model of turbulence-corrupted imagery. Optimal transport produces an invertible, pixel-wise linear trajectories to approximate the globally nonlinear turbulence between a clean and turbulence corrupted image pair. We use the low-dimensional model to fit subsets of the optimal transport vector fields and stitch the local models into a surrogate for the global map to be used for image cleaning. Experiments are performed on laboratory generated data of beam propagation using different values of the Fried parameter (a scale measuring turbulence coherence) as well as a toy data set. The results suggest this is a fruitful direction, and first step, towards using multiple realizations of turbulence corrupted images to learn a blind surrogate for the optimal transport vector field for image cleaning.

47 OTHER INSTRUMENTATION↗

Attack-Resilient Weighted $\ell_{1}$ Observer with Prior Pruning

Security related questions for Cyber Physical Systems (CPS) have attracted much research attention in searching for novel methods for attack-resilient control and/or estimation. Specifically, false data injection attacks (FDIAs) have been shown to be capable of bypassing bad data detection (BDD), while arbitrarily compromising the integrity of state estimators and robust controller even with very sparse measurements corruption. Moreover, based on the inherent sparsity of pragmatic attack signals, ℓ1 -minimization scheme has been used extensively to improve the design of attack-resilient estimators. For this, the theoretical maximum for the percentage of compromised nodes that can be accommodated has been shown to be 50%. In order to guarantee correct state recoveries for larger percentage of attacked nodes, researchers have begun to incorporate prior information into the underlying resilient observer design framework. For the most pragmatic cases, this prior information is often obtained through some data-driven machine learning process. Existing results have shown strong positive correlation between the tolerated attack percentages and the precision of the prior information. In this paper, we present a pruning method to improve the precision of the prior information, given corresponding stochastic uncertainty characteristics of the underlying machine learning model. Then a weighted ℓ1 -minimization is proposed based on the pruned prior. The theoretical and simulation results show that the pruning method significantly improves the observer performance for much larger attack percentages, even when moderately accurate machine learning model used.

Resilient observer, Cyber-physical systems, prunin↗

Sensitivity of back‐projection algorithm (BPA) synthetic aperture radar (SAR) image formation to initial position, velocity, and attitude navigation errors

Abstract The Back‐Projection Algorithm (BPA) is a time‐domain‐matched filtering technique to form synthetic aperture radar (SAR) images. To produce high‐quality BPA images, precise navigation data for the radar platform must be known. Errors in position, velocity, or attitude result in improperly formed images that are corrupted by shifting and blurring. The contribution of this paper is the development of analytical expressions that characterise the relationship between navigation errors and image formation errors from an inertial navigation point of view, where trajectory estimation errors in position, velocity, and attitude propagate through time and cause compounding errors in the vehicle state vector. These analytical expressions are verified via simulated image formation and real‐data image formation.

Lindstrom, Colton↗

Fourier domain closed-form formulas for estimation of kinetic parameters in reversible multi-compartment models

Background: Compared with static imaging, dynamic emission computed tomographic imaging with compartment modeling can quantify in vivo physiologic processes, providing useful information about molecular disease processes. Dynamic imaging involves estimation of kinetic rate parameters. For multi-compartment models, kinetic parameter estimation can be computationally demanding and problematic with local minima. Methods: This paper offers a new perspective to the compartment model fitting problem where Fourier linear system theory is applied to derive closed-form formulas for estimating kinetic parameters for the two-compartment model. The proposed Fourier domain estimation method provides a unique solution, and offers very different noise response as compared to traditional non-linear chi-squared minimization techniques. Results: The unique feature of the proposed Fourier domain method is that only low frequency components are used for kinetic parameter estimation, where the DC (i.e., the zero frequency) component in the data is treated as the most important information, and high frequency components that tend to be corrupted by statistical noise are discarded. Computer simulations show that the proposed method is robust without having to specify the initial condition. The resultant solution can be fine tuned using the traditional iterative method. Conclusions: The proposed Fourier-domain estimation method has closed-form formulas. The proposed Fourier-domain curve-fitting method does not require an initial condition, it minimizes a quadratic objective function and a closed-form solution can be obtained. The noise is easier to control, simply by discarding the high frequency components, and emphasizing the DC component.

Nuclear medicine imaging↗