Search NASA⌕ Search

SEARCH · Search NASA

Results for “data reduction algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Framework of compressive sensing and data compression for 4D-STEM

Four-dimensional Scanning Transmission Electron Microscopy (4D-STEM) is a powerful technique for high-resolution and high-precision materials characterization at multiple length scales, including the characterization of beam-sensitive materials. However, the field of view of 4D-STEM is relatively small, which in absence of live processing is limited by the data size required for storage. Furthermore, the rectilinear scan approach currently employed in 4D-STEM places a resolution- and signal-dependent dose limit for the study of beam sensitive materials. Improving 4D-STEM data and dose efficiency, by keeping the data size manageable while limiting the amount of electron dose, is thus critical for broader applications. Here we introduce a general method for reconstructing 4D-STEM data with subsampling in both real and reciprocal spaces at high fidelity. The approach is first tested on the subsampled datasets created from a full 4D-STEM dataset, and then demonstrated experimentally using random scan in real-space. The same reconstruction algorithm can also be used for compression of 4D-STEM datasets, leading to a large reduction (100 times or more) in data size, while retaining the fine features of 4D-STEM imaging, for crystalline samples.

4D-STEM↗
DOE OSTI2024

Scalable Tensor Methods for Nonuniform Hypergraphs

While multilinear algebra appears natural for studying the multiway interactions modeled by hypergraphs, tensor methods for general hypergraphs have been stymied by theoretical and practical barriers. A recently proposed adjacency tensor is applicable to nonuniform hypergraphs, but is prohibitively costly to form and analyze in practice. We develop tensor times same vector (TTSV) algorithms for this tensor which improve complexity from $O(n^r)$ to a low-degree polynomial in $r$, where $n$ is the number of vertices and $r$ is the maximum hyperedge size. Our algorithms are implicit, avoiding formation of the order $r$ adjacency tensor. Here, we demonstrate the flexibility and utility of our approach in practice by developing tensor-based hypergraph centrality and clustering algorithms. We also show these tensor measures offer complementary information to analogous graph-reduction approaches on data, and are also able to detect higher-order structure that many existing matrix-based approaches provably cannot.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1984

Radar data smoothing filter study

The accuracy of the current Wallops Flight Facility (WFF) data smoothing techniques for a variety of radars and payloads is examined. Alternative data reduction techniques are given and recommendations are made for improving radar data processing at WFF. A data adaptive algorithm, based on Kalman filtering and smoothing techniques, is also developed for estimating payload trajectories above the atmosphere from noisy time varying radar data. This algorithm is tested and verified using radar tracking data from WFF.

White, J. V.↗
NASA NTRS1985

Solar radiance models for determination of ERBE scanner filter factor

Shortwave spectral radiance models for use in the spectral correction algorithms for the ERBE Scanner Instrument are provided. The required data base was delivered to the ERBe Data Reduction Group in October 1984. It consisted of two sets of data files: (1) the spectral bidirectional angular models and (2) the spectral flux modes. The bidirectional models employ the angular characteristics of reflection by the Earth-atmosphere system and were derived from detailed radiance calculations using a finite difference model of the radiative transfer process. The spectral flux models were created through the use of a delta-Eddington model to economically simulate the effects of atmospheric variability. By combining these data sets, a wide range of radiances may be approximated for a number of scene types.

Arduini, R. F.↗
NASA NTRS1985

Least squares collocation applied to local gravimetric solutions from satellite gravity gradiometry data

An autonomous spaceborne gravity gradiometer mission is being considered as a post Geopotential Research Mission project. The introduction of satellite diometry data to geodesy is expected to improve solid earth gravity models. The possibility of utilizing gradiometer data for the determination of pertinent gravimetric quantities on a local basis is explored. The analytical technique of least squares collocation is investigated for its usefulness in local solutions of this type. It is assumed, in the error analysis, that the vertical gravity gradient component of the gradient tensor is used as the raw data signal from which the corresponding reference gradients are removed to create the centered observations required in the collocation solution. The reference gradients are computed from a high degree and order geopotential model. The solution can be made in terms of mean or point gravity anomalies, height anomalies, or other useful gravimetric quantities depending on the choice of covariance types. Selected for this study were 30 x 30 foot mean gravity and height anomalies. Existing software and new software are utilized to implement the collocation technique. It was determined that satellite gradiometry data at an altitude of 200 km can be used successfully for the determination of 30 x 30 foot mean gravity anomalies to an accuracy of 9.2 mgal from this algorithm. It is shown that the resulting accuracy estimates are sensitive to gravity model coefficient uncertainties, data reduction assumptions and satellite mission parameters.

Robbins, J. W.↗
NASA NTRS1997

Analysis of Particle Image Velocimetry (PIV) Data for Application to Subsonic Jet Noise Studies

Global velocimetry measurements were taken using Particle Image Velocimetry (PIV) in the subsonic flow exiting a 1 inch circular nozzle in an attempt to better understand the turbulence characteristics of its shear layer region. This report presents the results of the PIV analysis and data reduction portions of the test and details the processing that was done. Custom data analysis and data validation algorithms were developed and applied to a data ensemble consisting of over 750 PIV 70 mm photographs taken in the 0.85 mach flow facility. Results are presented detailing spatial characteristics of the flow including ensemble mean and standard deviation, turbulence intensities and Reynold's stress levels, and 2-point spatial correlations.

Blackshire, James L.↗
NASA NTRS2011

The GOES-R GeoStationary Lightning Mapper (GLM)

The Geostationary Operational Environmental Satellite (GOES-R) is the next series to follow the existing GOES system currently operating over the Western Hemisphere. Superior spacecraft and instrument technology will support expanded detection of environmental phenomena, resulting in more timely and accurate forecasts and warnings. Advancements over current GOES capabilities include a new capability for total lightning detection (cloud and cloud-to-ground flashes) from the Geostationary Lightning Mapper (GLM), and improved capability for the Advanced Baseline Imager (ABI). The Geostationary Lighting Mapper (GLM) will map total lightning activity (in-cloud and cloud-to-ground lighting flashes) continuously day and night with near-uniform spatial resolution of 8 km with a product refresh rate of less than 20 sec over the Americas and adjacent oceanic regions. This will aid in forecasting severe storms and tornado activity, and convective weather impacts on aviation safety and efficiency among a number of potential applications. In parallel with the instrument development (a prototype and 4 flight models), a GOES-R Risk Reduction Team and Algorithm Working Group Lightning Applications Team have begun to develop the Level 2 algorithms (environmental data records), cal/val performance monitoring tools, and new applications using GLM alone, in combination with the ABI, merged with ground-based sensors, and decision aids augmented by numerical weather prediction model forecasts. Proxy total lightning data from the NASA Lightning Imaging Sensor on the Tropical Rainfall Measuring Mission (TRMM) satellite and regional test beds are being used to develop the pre-launch algorithms and applications, and also improve our knowledge of thunderstorm initiation and evolution. An international field campaign planned for 2011-2012 will produce concurrent observations from a VHF lightning mapping array, Meteosat multi-band imagery, Tropical Rainfall Measuring Mission (TRMM) Lightning Imaging Sensor (LIS) overpasses, and related ground and in-situ lightning and meteorological measurements in the vicinity of Sao Paulo. These data will provide a new comprehensive proxy data set for algorithm and application development.

Goodman, Steven J.↗
NASA NTRS2008

Reducing Surface Clutter in Cloud Profiling Radar Data

An algorithm has been devised to reduce ground clutter in the data products of the CloudSat Cloud Profiling Radar (CPR), which is a nadir-looking radar instrument, in orbit around the Earth, that measures power backscattered by clouds as a function of distance from the instrument. Ground clutter contaminates the CPR data in the lowest 1 km of the atmospheric profile, heretofore making it impossible to use CPR data to satisfy the scientific interest in studying clouds and light rainfall at low altitude. The algorithm is based partly on the fact that the CloudSat orbit is such that the geodetic altitude of the CPR varies continuously over a range of approximately 25 km. As the geodetic altitude changes, the radar timing parameters are changed at intervals defined by flight software in order to keep the troposphere inside a data-collection time window. However, within each interval, the surface of the Earth continuously "scans through" (that is, it moves across) a few range bins of the data time window. For each radar profile, only few samples [one for every range-bin increment ((Delta)r = 240 m)] of the surface-clutter signature are available around the range bin in which the peak of surface return is observed, but samples in consecutive radar profiles are offset slightly (by amounts much less than (Delta)r) with respect to each other according to the relative change in geodetic altitude. As a consequence, in a case in which the surface area under examination is homogenous (e.g., an ocean surface), a sequence of consecutive radar profiles of the surface in that area contains samples of the surface response with range resolution (Delta)p much finer than the range-bin increment ((Delta)p << r). Once the high-resolution surface response has thus become available, the profile of surface clutter can be accurately estimated by use of a conventional maximum-correlation scheme: A translated and scaled version of the high-resolution surface response is fitted to the observed low-resolution profile. The translation and scaling factors that optimize the fit in a maximum-correlation sense represent (1) the true position of the surface relative to the sampled surface peak and (2) the magnitude of the surface backscatter. The performance of this algorithm has been tested on CloudSat data acquired over an ocean surface. A preliminary analysis of the test data showed a surface-clutter-rejection ratio over flat surfaces of >10 dB and a reduction of the contaminated altitude over ocean from about 1 km to about 0.5 km (over the ocean). The algorithm has been embedded in CloudSat L1B processing as of Release 04 (July 2007), and the estimated flat surface clutter is removed in L2B-GEOPROF product from the observed profile of reflectivity (see CloudSat product documentation for details and performance at http://www.cloudsat.cira.colostate.edu/ dataSpecs.php?prodid=1).

Tanelli, Simone↗
NASA NTRS2015

PISCES High Contrast Integral Field Spectrograph Simulations and Data Reduction Pipeline

The PISCES (Prototype Imaging Spectrograph for Coronagraphic Exoplanet Studies) is a lenslet array based integral field spectrograph (IFS) designed to advance the technology readiness of the WFIRST (Wide Field Infrared Survey Telescope)-AFTA (Astrophysics Focused Telescope Assets) high contrast Coronagraph Instrument. We present the end to end optical simulator and plans for the data reduction pipeline (DRP). The optical simulator was created with a combination of the IDL (Interactive Data Language)-based PROPER (optical propagation) library and Zemax (a MatLab script), while the data reduction pipeline is a modified version of the Gemini Planet Imager's (GPI) IDL pipeline. The simulations of the propagation of light through the instrument are based on Fourier transform algorithms. The DRP enables transformation of the PISCES IFS data to calibrated spectral data cubes.

The PISCES (Prototype Imaging Spectrograph for Cor↗
NASA NTRS1990

Kalman filter based range estimation for autonomous navigation using imaging sensors

The ability to detect and locate obstacles using on-board sensors and modify the nominal trajectory is necessary for safe landing of an autonomous lander on Mars. This paper examines some of the issues in the location of objects using a sequence of images from a passive sensor, and describes a Kalman filter approach to improve the range estimation to obstacles. The filter is also used to track features in the images leading to a significant reduction of search effort in the feature extraction step of the algorithm. The lack of suitable flight imagery data presents a problem in the verification of concepts for obstacle detection. An experiment is designed to acquire a sequence of images along with sensor motion parameters and the range estimation results using this imagery are presented.

Sridhar, B.↗
NASA NTRS1987

Algorithm Sorts Groups Of Data

For efficient sorting, algorithm finds set containing minimum or maximum most significant data. Sets of data sorted as desired. Sorting process simplified by reduction of each multielement set of data to single representative number. First, each set of data expressed as polynomial with suitably chosen base, using elements of set as coefficients. Most significant element placed in term containing largest exponent. Base selected by examining range in value of data elements. Resulting series summed to yield single representative number. Numbers easily sorted, and each such number converted back to original set of data by successive division. Program written in BASIC.

Evans, J. D.↗
NASA NTRS1986

Data inversion algorithm development for the hologen occultation experiment

The successful retrieval of atmospheric parameters from radiometric measurement requires not only the ability to do ideal radiometric calculations, but also a detailed understanding of instrument characteristics. Therefore a considerable amount of time was spent in instrument characterization in the form of test data analysis and mathematical formulation. Analyses of solar-to-reference interference (electrical cross-talk), detector nonuniformity, instrument balance error, electronic filter time-constants and noise character were conducted. A second area of effort was the development of techniques for the ideal radiometric calculations required for the Halogen Occultation Experiment (HALOE) data reduction. The computer code for these calculations must be extremely complex and fast. A scheme for meeting these requirements was defined and the algorithms needed form implementation are currently under development. A third area of work included consulting on the implementation of the Emissivity Growth Approximation (EGA) method of absorption calculation into a HALOE broadband radiometer channel retrieval algorithm.

Gordley, Larry L.↗
DOE OSTI2025

Identification of tau leptons using a convolutional neural network with domain adaptation

A tau lepton identification algorithm,DeepTau, based on convolutional neural network techniques, has been developed in the CMS experiment to discriminate reconstructed hadronic decays of tau leptons (τ h ) from quark or gluon jets and electrons and muons that are misreconstructed as τ h candidates. The latest version of this algorithm, v2.5, includes domain adaptation by backpropagation, a technique that reduces discrepancies between collision data and simulation in the region with the highest purity of genuine τh candidates. Additionally, a refined training workflow improves classification performance with respect to the previous version of the algorithm, with a reduction of 30–50% in the probability for quark and gluon jets to be misidentified as τ h candidates for given reconstruction and identification efficiencies. This paper presents the novel improvements introduced in theDeepTau algorithm and evaluates its performance in LHC proton-proton collision data at √(s) = 13 and 13.6 TeV collected in 2018 and 2022 with integrated luminosities of 60 and 35 fb -1 , respectively. Techniques to calibrate the performance of the τ h identification algorithm in simulation with respect to its measured performance in real data are presented, together with a subset of results among those measured for use in CMS physics analyses.

Large detector-systems performance↗
NASA NTRS2008

1D-VAR Retrieval Using Superchannels

Since modern ultra-spectral remote sensors have thousands of channels, it is difficult to include all of them in a 1D-var retrieval system. We will describe a physical inversion algorithm, which includes all available channels for the atmospheric temperature, moisture, cloud, and surface parameter retrievals. Both the forward model and the inversion algorithm compress the channel radiances into super channels. These super channels are obtained by projecting the radiance spectra onto a set of pre-calculated eigenvectors. The forward model provides both super channel properties and jacobian in EOF space directly. For ultra-spectral sensors such as Infrared Atmospheric Sounding Interferometer (IASI) and the NPOESS Airborne Sounder Testbed Interferometer (NAST), a compression ratio of more than 80 can be achieved, leading to a significant reduction in computations involved in an inversion process. Results will be shown applying the algorithm to real IASI and NAST data.

Liu, Xu↗
NASA NTRS2018

Traction Control Design and Integration Onboard the Mars Science Laboratory Curiosity Rover

The Mars Science Laboratory (MSL) Curiosity rover experienced increasing wheel damage beginning in October 2013. While the wheels were designed to operate with considerable damage, the rate at which damage was occurring was unexpected and raised concerns regarding wheel lifetime. The Jet Propulsion Laboratory (JPL) has now developed and deployed new software on Curiosity that reduces the forces acting on the wheels. Our new Traction Control algorithm adapts each wheel’s speed to fit the terrain it drives over. It does not rely on any a priori knowledge of the terrain, and instead leverages the rover’s measured attitude rates and suspension angles, together with a rigid-body kinematics model, to estimate the real-time wheel-terrain contact angles and ideal, no-slip wheel angular rates. In addition, free-floating “wheelies” are detected and autonomously corrected. In this paper, we describe the algorithm, its ground testing campaign and associated challenges, and finally its validation and performance in flight. Ground test data demonstrates reductions in the forces acting on the wheels and validates the wheelie-damping capability. Secondary benefits in some terrains include a reduction in heading deviations while climbing rocks, with a reduction in slip in certain sandy terrains. Preliminary validation from flight data confirms these findings.

Maimone, Mark↗
DOE OSTI2024

Event Detection and Classification Using Machine Learning Applied to PMU Data for the Western US Power System

Smart grid technology enhances our comprehension and reliability of the power grid, leveraging Phasor Measurement Unit (PMU) data—time-synchronized, high-frequency measurements gathered across the US power grid. This paper employs machine learning techniques to effectively analyze the vast PMU data in Wide Area Monitoring Systems (WAMS) for power grid event detection and classification. Analyzing several months of real-world PMU data, the paper focuses on machine learning for fast, precise event detection and classification, corroborated by utility event logs. Practical challenges like feature extraction, dimensionality reduction, and model selection are addressed. A novel feature yielding improved results is discovered, and a supplementary algorithm for detecting small power grid faults is developed. The final algorithm is validated using a month-long real PMU data set, demonstrating its capability in accurately identifying power grid events in near real-time.

machine learning, event detection, PMU↗
DOE OSTI2024

Embedded FPGA developments in 130 nm and 28 nm CMOS for machine learning in particle detector readout

Embedded field programmable gate array (eFPGA) technology allows the implementation of reconfigurable logic within the design of an application-specific integrated circuit (ASIC). This approach offers the low power and efficiency of an ASIC along with the ease of FPGA configuration, particularly beneficial for the use case of machine learning in the data pipeline of next-generation collider experiments. An open-source framework called "FABulous" was used to design eFPGAs using 130 nm and 28 nm CMOS technology nodes, which were subsequently fabricated and verified through testing. The capability of an eFPGA to act as a front-end readout chip was assessed using simulation of high energy particles passing through a silicon pixel sensor. A machine learning-based classifier, designed for reduction of sensor data at the source, was synthesized and configured onto the eFPGA. A successful proof-of-concept was demonstrated through reproduction of the expected algorithm result on the eFPGA with perfect accuracy. Finally, further development of the eFPGA technology and its application to collider detector readout is discussed.

47 OTHER INSTRUMENTATION↗
NASA NTRS1993

Scattered light in the echelle modes of the Goddard High Resolution Spectrograph aboard the Hubble Space Telescope. II - Analysis of in-flight spectroscopic observations

We report on an analysis of observations made with the GHRS aboard the HST for the purpose of calibrating the effects of scattered light. The data were obtained during the first 2 years of flight operations as part of the Science Verification, Science Assessment Observations, and Guaranteed Time Observations programs. Both the general level of background light near the spectra and the filling-in of the cores of saturated interstellar absorption lines have been determined as functions of grating, order, wavelength, and aperture. The behavior is in accordance with our detailed analysis of prelaunch laboratory data and validates both the observing procedures and the data reduction developed on the basis of that experience. We present and discuss the use of a background (scattered light) removal algorithm designed to produce net spectra properly corrected for the effects of grating scattered light. The coefficients of the scattered light removal algorithm have been determined, and recommended numerical values are presented.

Cardelli, Jason A.↗