Search NASA⌕ Search

SEARCH · Search NASA

Results for “high dimensional data analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

System and Safety Analysis with SysAI A Statistical Learning Framework

This is a tutorial on how to use the SYSAI (System Analysis using Statistical AI), a flexible statistical learning framework for the V&V and analysis of complex and high-dimensional Aerospace systems with DNN and AI components. SYSAI provides functionality for a variety of analyses and V&V tasks, including statistical data analysis, high dimensional safety-envelope and time-series analysis, property checking, as well as intelligent test-case generation. The tutorial will demonstrate SYSAI with our industrial partner’s Autonomous Centerline Tracking system, which uses a DNN to enable autonomous aircraft taxiing as an example. Video & Tutorial

Statistical V&V for Complex safety-critical system↗
NASA NTRS1994

Effect of radiance-to-reflectance transformation and atmosphere removal on maximum likelihood classification accuracy of high-dimensional remote sensing data

Many analysis algorithms for high-dimensional remote sensing data require that the remotely sensed radiance spectra be transformed to approximate reflectance to allow comparison with a library of laboratory reflectance spectra. In maximum likelihood classification, however, the remotely sensed spectra are compared to training samples, thus a transformation to reflectance may or may not be helpful. The effect of several radiance-to-reflectance transformations on maximum likelihood classification accuracy is investigated in this paper. We show that the empirical line approach, LOWTRAN7, flat-field correction, single spectrum method, and internal average reflectance are all non-singular affine transformations, and that non-singular affine transformations have no effect on discriminant analysis feature extraction and maximum likelihood classification accuracy. (An affine transformation is a linear transformation with an optional offset.) Since the Atmosphere Removal Program (ATREM) and the log residue method are not affine transformations, experiments with Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) data were conducted to determine the effect of these transformations on maximum likelihood classification accuracy. The average classification accuracy of the data transformed by ATREM and the log residue method was slightly less than the accuracy of the original radiance data. Since the radiance-to-reflectance transformations allow direct comparison of remotely sensed spectra with laboratory reflectance spectra, they can be quite useful in labeling the training samples required by maximum likelihood classification, but these transformations have only a slight effect or no effect at all on discriminant analysis and maximum likelihood classification accuracy.

Hoffbeck, Joseph P.↗
NASA NTRS1997

Supervised Classification Techniques for Hyperspectral Data

The recent development of more sophisticated remote sensing systems enables the measurement of radiation in many mm-e spectral intervals than previous possible. An example of this technology is the AVIRIS system, which collects image data in 220 bands. The increased dimensionality of such hyperspectral data provides a challenge to the current techniques for analyzing such data. Human experience in three dimensional space tends to mislead one's intuition of geometrical and statistical properties in high dimensional space, properties which must guide our choices in the data analysis process. In this paper high dimensional space properties are mentioned with their implication for high dimensional data analysis in order to illuminate the next steps that need to be taken for the next generation of hyperspectral data classifiers.

Jimenez, Luis O.↗
NASA NTRS2006

Using Image Tour to Explore Multiangle, Multispectral Satellite Image

This viewgraph presentation reviews the use of Image Tour to explore the multiangle, multispectral satellite imagery. Remote sensing data are spatial arrays of p-dimensional vectors where each component corresponds to one of p variables. Applying the same R(exp p) to R(exp d) projection to all pixels creates new images, which may be easier to analyze than the original because d < p. Image grand tour (IGT) steps through the space of projections, and d=3 outputs a sequence of RGB images, one for each step. In this talk, we apply IGT to multiangle, multispectral data from NASA's MISR instrument. MISR views each pixel in four spectral bands at nine view angles. Multiple views detect photon scattering in different directions and are indicative of physical properties of the scene. IGT allows us to explore MISR's data structure while maintaining spatial context; a key requirement for physical interpretation. We report results highlighting the uniqueness of multiangle data and how IGT can exploit it.

visualization↗
DOE OSTI2025

Visualization of Noisy and Less Noisy Computational Basis States in Quantum Computing

Quantum computing technology holds substantial promise as a reliable computational paradigm. However, current noisy intermediate scale quantum (NISQ) systems, are significantly impacted by noise originating from hardware inconsistencies. This noise causes errors and lowers output fidelity. So we must find which basis states cause errors. However, there are two main challenges in analyzing noise corresponding to basis states. First, the noise distribution data is high dimensional in nature, thereby making its analysis challenging. Second, although functional box plots have been used in the state of the art research to understand such a high dimensional data, they suffer from clutter and occlusion issues because of overplotting. In this study, we introduce an innovative visualization pipeline to address the aforementioned challenges to provide a clear depiction of noisy and less-noisy basis states. Specifically, our proposed visualization pipeline comprises three stages namely, low dimensional embedding, clustering, and violin plot visualization, to reduce visual clutter and effectively analyze high-dimensional noise distribution data. Our analysis uses quantum machine learning (QML) circuits as case study for drawing a distinction between noisy and less noisy basis states.

Senapati, Priyabrata [Kent State University]↗
DOE OSTI2024

Predicting Drug Effects from High-dimensional Asymmetric Drug Data Sets using Graph Neural Networks: A Comprehensive Analysis of Multi-target Drug Effect Prediction

Graph neural networks (GNNs) have emerged as one of the most effective Machine learning (ML) techniques for drug effect prediction from drug molecular graphs. Despite having immense potential, GNN models lack performance when using data sets that contain high dimensional asymmetrically co-occurrent drug effects as targets with complex correlations between them. Training individual learning models for each drug effect and incorporating every prediction result for a wide spectrum of drug effects is beyond practicality. Such an implication provides a testbed to address this challenge as multi-target prediction problems, aiming to predict all drug effects at a time. We develop standard and hybrid graph neural networks (GNNs)to perform two separate tasks that are multi-regression for continuous values and multi-label classification for categorical values contained in our data sets. Since this step makes the target data even more sparse and introduces asymmetric label co-occurrence, the learning of multi-label classification models becomes difficult and heavily impacts the GNN's performance. To address these challenges, we propose a new data oversampling technique to improve multi-label classification performances on all the given imbalanced molecular graph data sets. Using the technique, we improve the data imbalance ratio of the drug effects better than before while protecting the data set's integrity. Finally, we evaluate multi-label classification performance using the best-performant hybrid GNN model on all the oversampled data sets obtained from the proposed oversampling technique. These results outperform those of other ML models including GNN models when they are trained on the original data sets or oversampled data sets using MLSMOTE (a well-known oversampling technique) in all evaluation metrics precision, recall, and F1 score by a significant margin.

Bose, Avishek [ORNL]↗
NASA NTRS2011

Parametric Testing of Launch Vehicle FDDR Models

For the safe operation of a complex system like a (manned) launch vehicle, real-time information about the state of the system and potential faults is extremely important. The on-board FDDR (Failure Detection, Diagnostics, and Response) system is a software system to detect and identify failures, provide real-time diagnostics, and to initiate fault recovery and mitigation. The ERIS (Evaluation of Rocket Integrated Subsystems) failure simulation is a unified Matlab/Simulink model of the Ares I Launch Vehicle with modular, hierarchical subsystems and components. With this model, the nominal flight performance characteristics can be studied. Additionally, failures can be injected to see their effects on vehicle state and on vehicle behavior. A comprehensive test and analysis of such a complicated model is virtually impossible. In this paper, we will describe, how parametric testing (PT) can be used to support testing and analysis of the ERIS failure simulation. PT uses a combination of Monte Carlo techniques with n-factor combinatorial exploration to generate a small, yet comprehensive set of parameters for the test runs. For the analysis of the high-dimensional simulation data, we are using multivariate clustering to automatically find structure in this high-dimensional data space. Our tools can generate detailed HTML reports that facilitate the analysis.

Schumann, Johann↗
NASA NTRS1977

Two-dimensional wind-tunnel tests of a NASA supercritical airfoil with various high-lift systems. Volume 1: Data analysis

High-lift systems for a NASA, 9.3%, method for calculating the viscous flow about two-dimensional multicomponent airfoils was evaluated by comparing its predictions with test data. High-lift systems derived from supercritical airfoils were compared in terms of performance to high-lift systems derived from conventional airfoils. The high-lift systems for the supercritical airfoil were designed to achieve maximum lift and consisted of: a single-slotted flap; a double-slotted flap and a leading-edge slat; and a triple-slotted flap and a leading-edge slat. Agreement between theoretical predictions and experimental results are also discussed.

Omar, E.↗
DOE OSTI2025

In Situ Data Analysis Through Physics-informed Tensor Decompositions (LDRD Final Report)

We introduce a new low-dimensional model of high-dimensional numerical simulation data based on low-rank tensor decompositions. Our new model aims to minimize differences between the model data and simulation data as well as functions of the model data and functions of the simulation data. This novel approach to dimensionality reduction of simulation data provides a means of directly incorporating quantities of interests and invariants associated with conservation principles associated with the simulation data into the low-dimensional model, thus enabling more accurate analysis of the simulation without requiring access to the full set of high-dimensional data. Computational results of applying this approach to two standard low-rank tensor decompositions of data arising from simulation of combustion and plasma physics are presented.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1982

A microprocessor-based one dimensional optical data processor for spatial frequency analysis

A high degree of accuracy was obtained in measuring the spatial frequency spectrum of known samples using an optical data processor based on a microprocessor, which reliably collected intensity versus angle data. Stray light control, system alignment, and angle measurement problems were addressed and solved. The capabilities of the instrument were extended by the addition of appropriate optics to allow the use of different wavelengths of laser radiation and by increasing the travel limits of the rotating arm to + or - 160 degrees. The acquisition, storage, and plotting of data by the computer permits the researcher a free hand in data manipulation such as subtracting background scattering from a diffraction pattern. Tests conducted to verify the operation of the processor using a 25 mm diameter pinhole, a 39.37 line pairs per mm series of multiple slits, and a microscope slide coated with 1.091 mm diameter polystyrene latex spheres are described.

Collier, R. L.↗
DOE OSTI2024

Deep learning-driven super-resolution in Raman hyperspectral imaging: Efficient high-resolution reconstruction from low-resolution data

Deep learning (DL) has become an indispensable tool in hyperspectral data analysis, automatically extracting valuable features from complex, high-dimensional datasets. Super-resolution reconstruction, an essential aspect of hyperspectral data, involves enhancing spatial resolution, particularly relevant to low-resolution hyperspectral data. Yet, the pursuit of super-resolution in hyperspectral analysis is fraught with challenges, including acquiring ground truth high-resolution data for training, generalization, and scalability. The pressing issue of extended spectral acquisition times, notably for high-resolution scans, is a significant roadblock in hyperspectral imaging. Super-resolution methods offer a promising solution by providing higher spatial resolution data to expedite data collection and yield more efficient outcomes. This paper delves into a practical application of these concepts using Raman imaging, where spectral acquisition times can be prohibitively long. In this context, DL-based super-resolution models demonstrate their efficacy by predicting and reconstructing high-resolution Raman data from low-resolution input, eliminating the need for resource-intensive high-resolution scans. While previous work often relied on substantial high-resolution datasets, this study showcases the ability to achieve similar outcomes even with limited data, presenting a more practical and cost-effective approach. In conclusion, the results offer a glimpse into the transformative potential of this technology to streamline hyperspectral imaging applications by saving valuable time and resources through the successful generation of high-resolution data from low-resolution inputs.

42 ENGINEERING↗
DOE OSTI2024

Blueprints for Training Information Bottlenecks for Collider Analyses

Dimensionality reduction is a crucial aspect of data analysis in high energy physics, even if accompanied by information loss. Several methods, including histogram- and kernel-based analyses, are only computationally feasible for low-dimensional data. Furthermore, simulation models used in HEP can often only be validated for low-dimensional data. We provide several blueprints for using machine learning to create low-dimensional data representations (continuous event variables and discrete classification labels) for use in signal discovery and parameter estimation tasks. We also describe how to design the learned representation to facilitate a) searches with unknown model parameters and b) validation of simulation models in data control regions.

43 PARTICLE ACCELERATORS↗
NASA NTRS1993

A tool for manual endmember selection and spectral unmixing

Sampling a continuous radiance spectrum in many narrow contiguous spectral bands results in a high covariance between the bands. Hence, the true dimensionality of imaging spectrometer data is not determined by the number of spectral bands, but by the number of spectrally unique signatures whose mixtures reproduce the spectral variance observed in an image. Methods to unmix high dimensional multispectral data use principal components analysis to reduce the dimensionality. The variance of the spectral data is modeled as a linear combination of a finite set of endmembers in the space of the eigen-vectors that account for most of the variance. The number and characteristics of these endmembers are determined not only by the number and characteristics of the spectrally unique materials on the surface but also by processes (e.g., illumination, atmospheric scattering and absorption) affecting the signal received by the sensor. Selection of endmember spectra has typically been from a library. However, since most libraries are incomplete and do not account for the processes mentioned above, we have devised a computer display that allows researchers to explore interactively the eigenvector space of a representative and mean-corrected subset of the image data in search of extreme spectra to designate as endmembers. This display, which is based on parallel coordinates, is unique in the area of multidimensional visualization in that it includes not only a passive view of higher dimensional data but also the capability to interact and move geometrical objects in higher dimensional spaces.

Bateson, C. Ann↗
DOE OSTI2025

Landscaper v1

Understanding the inner workings of machine learning models through their loss landscapes offers crucial insights into model properties, optimization dynamics, and generalizability. However, accessing these insights has traditionally required specialized mathematical expertise, limiting broader adoption. Landscaper is an open-source Python package designed to bridge this gap. Landscaper seamlessly integrates a suite of multi-dimensional loss landscape analyses with cutting-edge topological data analysis (TDA) methods. This powerful combination makes both fundamental loss landscape analysis and advanced TDA techniques accessible to the broader scientific ML community, without requiring deep pre-existing mathematical knowledge. Landscaper offers three key functionalities: * Construction: Builds detailed loss landscape representations through versatile low and high-dimensional sampling techniques. * Quantification: Applies advanced metrics, including a novel topological data analysis (TDA) based smoothness metric, enabling new perspectives on model behavior. * Visualization: Offers intuitive tools to visualize and interpret loss landscapes, providing actionable insights beyond traditional performance metrics.

Weber, Gunther [Lawrence Berkeley National Laborat↗
DOE OSTI2024

Machine learning analysis of high-repetition-rate two-dimensional Thomson scattering spectra from laser-produced plasmas

With the emergence of high-repetition-rate two-dimensional Thomson scattering (TS) measurements, improving spectral data analysis is a key area of interest. Here, we present a new way to derive the electron temperature and density of laser-driven blast waves in plasmas from their TS spectra with machine learning (ML). This analysis occurs in both the non-collective (α < 1) and collective (α > 1) scattering regimes with the goal of autonomously and more accurately determining T c and n e both where spectral data has been collected and to give the ability to predict these attributes in regions where data has not been collected. We introduce three ML models, one trained only on experimental data, one only on synthetic data, and one using transfer learning, and compare their speed and accuracy with the conventional TS inversion algorithms in the open source PlasmaPy python package.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2024

Accelerating particle-in-cell kinetic plasma simulations via reduced-order modeling of space-charge dynamics using dynamic mode decomposition

We present a data-driven reduced-order modeling of the space-charge dynamics for electromagnetic particle-in-cell (EMPIC) plasma simulations based on dynamic mode decomposition (DMD). The dynamics of the charged particles in kinetic plasma simulations such as EMPIC is manifested through the plasma current density defined along the edges of the spatial mesh. We showcase the efficacy of DMD in modeling the time evolution of current density through a low-dimensional feature space. Not only do such DMD based predictive reduced-order models help accelerate EMPIC simulations, they also have the potential to facilitate investigative analysis and control applications. Here, we demonstrate the proposed DMD-EMPIC scheme for reduced-order modeling of current density and speedup in EMPIC simulations involving electron beam under the influence of magnetic field, virtual cathode oscillations, and backward wave oscillator.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2025

Visualizing Temporal Topic Embeddings with a Compass

—Dynamic topic modeling is useful at discovering the development and change in latent topics over time. However, present methodology relies on algorithms that separate document and word representations. This prevents the creation of a meaningful embedding space where changes in word usage and documents can be directly analyzed in a temporal context. This paper proposes an expansion of the compass-aligned temporal Word2Vec methodology into dynamic topic modeling. Such a method allows for the direct comparison of word and document embeddings across time in dynamic topics. This enables the creation of visualizations that incorporate temporal word embeddings within the context of documents into topic visualizations. In experiments against the current state-of-the-art, our proposed method demonstrates overall competitive performance in topic relevancy and diversity across temporal datasets of varying size. Simultaneously, it provides insightful visualizations focused on temporal word embeddings while maintaining the insights provided by global topic evolution, advancing our understanding of how topics evolve over time.

Cluster analysis↗
NASA NTRSDate not supplied

Transcriptomics-based Machine Learning Analysis Predicts Space-Exposed Murine Livers

Limited sample sizes, high data dimensionality, and sensitivity to technical and biological variability of next generation sequencing (NGS), typically limits machine learning (ML) approaches in spaceflight studies that include radiation effects. However, pooling smaller studies while addressing intra- and inter-study variabilities allows for ML predictive modeling. Here, integration methods were applied to whole transcriptome shotgun sequencing (RNA-seq) data from six mouse liver GeneLab datasets (GLDS) (n ranging from 6 to 39 samples) from with a total of 81 spaceflight and ground-control samples to determine top features (i.e. genes) relevant to spaceflight including the effect of radiation exposure. RNASeq counts were normalized for each study, then merged and scaled across all datasets. Data dimensionality was reduced using a minimum redundancy maximum relevance (MRMR) methodology. Redundancy and relevance were computed using the Pearson correlation and F-statistic, respectively. The top 100 MRMR features were used to predict spaceflight vs. ground-control samples using Random Forest (RF), Support Vector Machine (SVM), and Linear Discriminant Analysis (LDA) classifiers with 5-fold cross validation (CV). Principal component analysis (PCA) on the complete feature set versus the MRMR features shows separation between spaceflight samples and ground controls (Figure 1A). The ML-based gene sets were compared against differential gene expression results obtained with DESeq2 from individual GLDS. Using all features or randomly sampled subsets at matching set sizes with MRMR, a maximum classifier accuracy of 69% was shown on the test set over 5 folds. For all classifiers, CV training using at least the top 30 MRMR genes show minimum 89% accuracy and 0.95 AUC value on the test set over 5 folds (Figure 1B). Baseline set analysis on differentially expressed genes (DEGs) identified using padj ≤ 0.05 show 295 DEGs that overlap at least two studies and 13 DEGs that overlap three studies (Figure 1C). Set analysis between the top 100 MRMR features and the DEGs showed 47 genes that overlap at least one study and 24 genes that overlap two studies. Over-representation analysis showed overlapping biological processes related to fatty acid and lipid metabolism which may indicate these processes in the response to spaceflight stressors. MRMR feature selection for the selected ML methods improve performance relative to a classifier built on all features or randomly sampled subsets. Permutation feature importance within the decorrelated MRMR features showed concordance in feature ranking between ML methods. A challenge of applying ML methods across heterogeneous NGS data is accounting for signal:noise. Here, signal validation across studies was shown by intersecting sets between top MRMR genes and DEGs from DESeq2 analysis. Non-intersecting sets introduce opportunity to explore genes relevant to differentiating space flight exposed groups and implementing ML methods across existing NGS datasets may overcome sample size limitations.

Machine Learning↗