Search NASA⌕ Search

SEARCH · Search NASA

Results for “preprocessing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

TEMPEST

This repository solves the problem of driver identification through vehicular and biometric data. Through an embedding-based approach and a novel loss function, we're able to distinguish between different drivers' behaviors. This also provides preprocessing for reproducibility of results.The code preprocesses vehicular data, trains neural networks, and outputs predictions.This code introduces a novel embedding-based neural network with a 91% rank-1 accuracy, as well as all code to reproduce training and results.

Musgrove, Kyle↗
DOE OSTI2024

Human limits in machine learning: prediction of potato yield and disease using soil microbiome data

Abstract Background The preservation of soil health is a critical challenge in the 21st century due to its significant impact on agriculture, human health, and biodiversity. We provide one of the first comprehensive investigations into the predictive potential of machine learning models for understanding the connections between soil and biological phenotypes. We investigate an integrative framework performing accurate machine learning-based prediction of plant performance from biological, chemical, and physical properties of the soil via two models: random forest and Bayesian neural network. Results Prediction improves when we add environmental features, such as soil properties and microbial density, along with microbiome data. Different preprocessing strategies show that human decisions significantly impact predictive performance. We show that the naive total sum scaling normalization that is commonly used in microbiome research is one of the optimal strategies to maximize predictive power. Also, we find that accurately defined labels are more important than normalization, taxonomic level, or model characteristics. ML performance is limited when humans can’t classify samples accurately. Lastly, we provide domain scientists via a full model selection decision tree to identify the human choices that optimize model prediction power. Conclusions Our study highlights the importance of incorporating diverse environmental features and careful data preprocessing in enhancing the predictive power of machine learning models for soil and biological phenotype connections. This approach can significantly contribute to advancing agricultural practices and soil health management.

Aghdam, Rosa↗
DOE OSTI2025

ORBIT-2 Dataset for Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling

This dataset release corresponds to the work conducted in ORBIT-2: Scaling Exascale Vision Foundation Models for Weather and Climate Downscaling, where large-scale AI methods were applied to improve climate and weather resolution. The collection integrates four widely used, publicly available datasets: ERA5, PRISM, DAYMET, and IMERG. To prepare the data for ORBIT-2 model training and evaluation, we applied a preprocessing pipeline that generates paired low-resolution and high-resolution samples, enabling supervised downscaling experiments. The transformation from coarse to fine scales was performed using bilinear regridding, consistent with the procedures described in WeatherBench2, a community benchmark for weather and climate AI models. This dataset supports the development and evaluation of foundation models designed for weather and climate downscaling at exascale. Additional details on methodology and applications can be found in Wang et al., ORBIT-2 (arXiv:2505.04802, 2025).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Dataset: "Widespread Drought-driven Declines in Streamflows and Water quality in the Upper Colorado River Basin (1998-2022)"

This data package contains the associated data and scripts for Nagamoto, E., Ombadi, M., Ciulla, F. et al. Widespread drought-driven declines in streamflows and water quality in the Upper Colorado River Basin during 1998-2022. Commun Earth Environ 7, 734 (2026). https://doi.org/10.1038/s43247-026-03890-5. This purpose of this study was to investigate the impact of the 21st century drought on water quantity and quality at catchments throughout the Upper Colorado River Basin (UCRB). We used stream flow, water temperature, specific conductance, air temperature, precipitation, and catchment attribute data for over 200 sites in the UCRB, collected from the National Water Information System using Basin3D (Varadharajan, 2023), GAGESII (Falcone, 2010), and the Google Earth Engine. We identified years of severe drought between 1998 and 2022 using the Standardized Precipitation Evaporation Index (SPEI), then calculated the relative change percentage of the stream flow, water temperature, and specific conductance from drought versus non-drought years. We used the attribute information from GAGESII to investigate what physical traits of catchments are associated streamflow vulnerability (greater relative change) or resilience to drought. We used land cover data from the National Land Cover Database (USGS, 2024) to assess any changes to physical attributes that may not be represented in the static attributes information in GAGESII. To increase data availability, we modeled stream temperature using methods from Willard, 2023. While the study period is water years 1998 to 2022, the raw water quantity and quality data extends to 1950 and the meteorological data extends to 1980. The data and code can be downloaded via the UCRB_drought.zip. Within the zip, the files are organized as follows: - INPUTS: Contains all input data used in UCRB_Drought_Workflow.ipynb - OUTPUTS: Contains all intermediate data created from UCRB_Drought_Workflow.ipynb as well as final products including the calculated Standardized Evapotranspiration Index (SPEI) - climatic_variables: The code used to collect meteorologic data from Google Earth Engine - feature_importance: The code used for the catchment attributes analysis - preprocessing: Code used in UCRB_Drought_Workflow_Preprocessing.ipynb - pyeto: Code used in UCRB_Drought_Workflow_Preprocessing.ipynb - calculations: Code used in UCRB_Drought_Workflow_Impacts.ipynb - plotting: Code used in UCRB_Drought_Workflow_Impacts.ipynb - README.md - UCRB_Drought_Workflow_Preprocessing.ipynb: The code used to prep raw data for the analysis - UCRB_Drought_Workflow_Impact.ipynb: The code which uses the prepped raw data for analysis, and plots all figures - requirements_ucrb-drought_v2.yml: The requirements file to create a virtual environment and Jupyter Lab kernel to run the code The INPUTS folder is organized into the following major directories and sub-directories. The "RDC_WT_SC_RAW" folder contains raw data for streamflow, water temperature, and specific conductance in a ".h5" file. The "NLCD_RAW" folder contains ".csv" files with annual land cover percentages for counties within the UCRB. The "MET_RAW" folder contains a ".csv" file with monthly meteorological data (air temperature and precipitation) for the sites in the UCRB which was obtained from code in the climatic_variables folder. The "GAGESII" folder contains ".csv" files with physical catchment attribute variables for catchments across the country. The "WT_LSTM_data" folder contains ".csv" files with calculated WT (Willard, 2023) and the associated RMSEs. The "Upper_Colorado_River_Basin_Boundary" folder contains geographic data including a shapefile for plotting in the UCRB_Drought_Workflow.ipynb. The "RESERVOIRS_RAW" folder contains ".csv" files for each reservoir in the UCRB with daily reservoir storage. There are also two files in the INPUTS folder that have combined reservoir storage data and reservoir metadata. The OUTPUTS folder is organized into the following major directories and sub-directories. The "RDC_WT_SC_data" folder contains a folder "Water_year" with the associated cleaned data, metadata, and data availability information in ".csv" files, a folder "Median_Relchange" with the relative change comparing drought to non-drought years in ".csv" files, and a folder "Peak95_Min5_Relchange" that has ".csv" files for the relative change in peak (95th %) and minimum (5th %) variables. The "NLCD_data" folder contains the difference in land cover from the beginning to end of the study period and the percentage of the county that is within UCRB bounds can be found in Nagamoto et al (2025)). The "MET_data" folder contains separated monthly air temperature and precipitation data and the calculated PET in ".csv" files. The "SPEI_data" folder contains ".csv" files with calculated SPEI values (one restricted to the study period and the other with information from the entire MET data period). The "Paper_Tables" folder contains two ".csv" files containing site information and data availability and information about the GAGESII trait aggregated categories. The base directory includes the file “flmd.csv” for a list and description of all files and the file “dd.csv” for data dictionaries. Scripts for preprocessing, analysis, and figure generation are located in the associated GitHub repository found at [https://github.com/iNAIADS/drought-impacts/tree/develop/UCRB-drought]. UPDATE 1: Title and code file updated to match submitted manuscript 10-15-2025. UPDATE 2: Code and data files updated to match revised manuscript 3-4-2026. UPDATE 3: Code and data files updated to match revised manuscript 6-7-2026. ** NOTE: DD and FLMD have not been updated yet. UPDATE 4: Added associated Manuscript information and DD and FLMD have been updated. To cite this code, please use the following BibTeX: @misc{nagamoto2025drought, author = {Emily Nagamoto and Fabio Ciulla and Mohammad Ombadi and Jared Willard and Rosemary Carroll and Charuleka Varadharajan}, title = {Dataset: "Widespread Drought-driven Declines in Streamflows and Water quality in the Upper Colorado River Basin (1998-2022)"}, year = {2025}, doi = {10.15485/2551894}, publisher = {ESS-DIVE Repository}, url = {https://data.ess-dive.lbl.gov/datasets/doi:10.15485/2551894} }

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Woody Feedstock 2022 State of Technology Report

The U.S. Department of Energy promotes production of advanced liquid transportation fuels from lignocellulosic biomass by funding fundamental and applied research that advances the state of technology (SOT). As part of its involvement in this mission, Idaho National Laboratory completes an annual SOT report for n th -plant and 1 st -plant woody biomass feedstock logistics. The purpose of the SOT is to provide the status of feedstock supply system technology development for woody biomass to biofuels relative to technical targets and cost goals from specific design cases, based on data and experimental results. Conventional feedstock supply systems need to be modified to meet the demands of conversion pathways, specifically to have the ability to adjust the quality of the raw biomass materials. Advanced systems incorporate innovative methods of material handling, preprocessing and supply chain configuration. In advanced designs, variability of the raw biomass can be reduced to produce feedstocks of a uniform format, moving toward biomass commoditization. Against this backdrop, the 2022 Woody SOT for low-ash woody feedstocks utilizes feedstock fractionation by incorporating technologies that can separate the biomass into its anatomical fractions (wood, bark, needle, and extrinsic ash) to reduce impurities and attempt to maximize the retention of usable fractions that satisfy downstream quality considerations. By using a series of air classification steps, this strategy can reduce the extrinsic ash in forest residues, separate out a majority of the incoming needles (which can be supplied to alternate markets), and maximize the retention of whitewood in the usable fraction. The fractionated forest residues are then mixed with clean-pine chips in a 50-50 blend to prepare the feedstock for the desired conversion pathway. The n th -plant analysis estimated the delivered cost for the feedstock at $\$$69.23/dry ton (2016$\$$) which represents a $\$$6.64/dry ton decrease compared to the cost estimate of the 2021 Woody SOT supply system for low-ash woody feedstocks. The quality requirements in the 2022 Woody SOT were identical to those of the 2021 Woody SOT at = 1.00 wt % ash and = 50.51 wt% carbon. The cost savings derive primarily from reductions in dry matter losses during air classification. The GHG emissions for the n th -plant analysis were estimated at 178.39 kg CO2e/dry ton compared to 178.71 kg CO 2 e/dry ton in the 2021 Woody SOT, a decrease of 0.32 kg CO2e/dry ton. The small change stems from an increase in emissions attributed to preprocessing and slightly larger savings in emissions from transportation. In the 1 st -plant analysis of the 2022 Woody SOT system, the average throughput was estimated to be approximately 2,128 dry tons/day or 96.51% of the name plate capacity. During the simulation the daily throughput ranged from 1,090 dry tons/day to 2,200 dry tons/day, or 49.43% to 99.75% of the daily nameplate capacity. After the year of operation 722,403 tons of processed feedstock were produced in total without regard to quality considerations (99.64% of the annual nameplate capacity). The variability in throughput was primarily caused by equipment failures in the system. Regular failures, downtime caused by routine maintenance per manufacturer guidelines, contributed to a majority 62.50% of failures and 62.60% of downtime. Failures due to wear were the other cause of disruption within the system, impacting the rotary shear and orbital screen and accounting for 37.50% of the failures and 37.40% of the total downtime. Ultimately the system was on stream for 87.84% during the simulation period, which is only 2.16 percentage points below the nth-plant assumption for on-stream time. The production cost of the system averaged $\$$71.66/dry ton. The costs ranged from a minimum of $\$$71.23/dry ton to a maximum of $\$$2,115.30/dry ton. When dry matter losses (disposed low-quality fractions as well as other losses such as in grinders) were considered the costs increased to an average of $\$$75.11/dry ton with a minimum of $\$$74.69/dry ton and a maximum of $\$$2,136.86/dry ton...

09 BIOMASS FUELS↗
DOE OSTI2026

A reproducible study design for the MIMIC-IV in-hospital mortality task

Open, tabular electronic health record (EHR) datasets such as MIMIC-III and MIMIC-IV have become critical resources for developing machine learning (ML) models addressing clinical prediction tasks, including hospital readmission, length of stay, and in-hospital mortality (IHM). While MIMIC-III has benefited from well-established preprocessing pipelines and standardized feature sets, MIMIC-IV remains comparatively challenging to work with because there are no standardized benchmarks to support reproducibility and comparability across studies. To address this limitation, we present a rigorously curated MIMIC-IV custom feature set optimized for IHM prediction, constructed through a reproducible preprocessing pipeline and feature selection strategy.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Environmental Quenching of Low-surface-brightness Galaxies Near Hosts from Large Magellanic Cloud to Milky Way Mass Scales

Low-surface-brightness galaxies (LSBGs) are excellent probes of quenching and other environmental processes near massive galaxies. We study an extensive sample of LSBGs near massive hosts in the local universe that are distributed across a diverse range of environments. The LSBGs with surface-brightness ${\mu }_{\mathrm{eff},{g}}\gt 24.2\,\mathrm{mag}\,{\mathrm{arcsec}}^{-2}$ are drawn from the Dark Energy Survey Year 3 catalog while the hosts with masses $9.0\lt \mathrm{log}({{ \mathcal M }}_{\star }/{M}_{\odot })\lt 11.0$ comparable to the Milky Way and the Large Magellanic Cloud are selected from the z0MGS sample. We study the projected radial density profiles of LSBGs as a function of their color and surface brightness around hosts in both the rich Fornax–Eridanus cluster environment and the low-density field. We detect an overdensity with respect to the background density, out to 2.5 times the virial radius for both hosts in the cluster environment and the isolated field galaxies. When the LSBG sample is split by g − i color or surface brightness μ eff, g , we find the LSBGs closer to their hosts are significantly redder and brighter, like their high-surface-brightness counterparts. The LSBGs form a clear “red sequence” in both the cluster and isolated environments that is visible beyond the virial radius of the hosts. This suggests preprocessing of infalling LSBGs and a quenched backsplash population around both host samples. More so, the relative prominence of the “blue cloud” feature implies that preprocessing is ongoing near the isolated hosts compared to the cluster environment where the LSBGs are already well processed.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

Explainable Machine Learning for Functional Data

Black-box machine learning models are recognized as useful tools for prediction applications, but the algorithmic complexity of some models causes interpretation challenges. Explainability methods have been proposed to provide insight into these models, but there is little research focused on supervised modeling with functional data inputs. We argue that, especially in applications of high consequence, it is important to explicitly model the functional dependence in a black-box analysis to not obscure or misrepresent patterns in explanations. As such, we propose the V ariable importance E xplainable E lastic S hape A nalysis (VEESA) pipeline for training supervised machine learning models with functional inputs. The pipeline is an analysis process that includes the data preprocessing, modeling, and post-hoc explanations. The preprocessing is done using elastic functional principal components analysis, which accounts for vertical and horizontal variability in functional data and, ultimately, allows for explanations in the original data space that identify the important functional variability without bias due to correlated variables. Here, we demonstrate the pipeline on two high-consequence applications: explosives classification for national security and inkjet printer identification in forensic science. The applications exhibit the VEESA pipeline’s ability to provide an understanding of the characteristics of the functional data useful for prediction. Code for implementing the pipeline is available in the veesa R package (and supplemental python code).

Elastic Shape Analysis↗
NASA NTRS1971

Layered 'recognition cone' networks that pre-process, classify, and describe.

A sequence of six types of pattern recognition system is examined. A program is described to illustrate some of the features developed. The first type (similar to many of the programs currently used) preprocesses by applying layers of local averaging and differencing transforms to smooth, fill in gaps and heighten contours, curves, and angles. It then applies a set of characterizers, each of which implies a set of names. The program chooses the single most high implied name. The second type combines the preprocessing transforms and the characterizers into a single operation of general type. Transforms build up a next representation of the input, while the characterizers imply the output name. The third type erases the distinction between a transform and an implication. Now all outputs are stored in the next transform layer. As the program averages information, its layers shrink, so that the system builds a cone of layers. When the program reaches the apex (a layer of only one cell that contains all the information), it chooses the single name with which it classifies the input. The fourth type is capable of choosing more than one name and, therefore, can both describe and classify the scene. The fifth type examines the interrelations among the set of names chosen. The sixth step can be taken to converse about the scene, developing an appropriate description in response to suggestions and queries. This allows the program to perform more computations and to look again on demand.

Uhr, L.↗
NASA NTRS1972

Layered 'recognition cone' networks that pre-process, classify, and describe.

Discussion of pattern recognition programs for input data preprocessing with simultaneous or subsequent characterization, or characterization into a 'recognition cone,' or description and naming, interrelated descriptions, and conversion. A computer program is described that transforms and characterizes the input through the successive layers of a recognition cone. The program can choose and put forth names of parts of the input scene. It combines pieces of a description into interrelated wholes by using n-tuple characterizers and conducts a simple and stylized conversation about what it has seen. The technique of combining recognition cones with preprocessing transformations and characterizations is expected to contribute to technology in this field.

Uhr, L.↗
NASA NTRS1971

Automatic Computer Mapping of Terrain

Computer processing of 17 wavelength bands of visible, reflective infrared, and thermal infrared scanner spectrometer data, and of three wavelength bands derived from color aerial film has resulted in successful automatic computer mapping of eight or more terrain classes in a Yellowstone National Park test site. The tests involved: (1) supervised and non-supervised computer programs; (2) special preprocessing of the scanner data to reduce computer processing time and cost, and improve the accuracy; and (3) studies of the effectiveness of the proposed Earth Resources Technology Satellite (ERTS) data channels in the automatic mapping of the same terrain, based on simulations, using the same set of scanner data. The following terrain classes have been mapped with greater than 80 percent accuracy in a 12-square-mile area with 1,800 feet of relief; (1) bedrock exposures, (2) vegetated rock rubble, (3) talus, (4) glacial kame meadow, (5) glacial till meadow, (6) forest, (7) bog, and (8) water. In addition, shadows of clouds and cliffs are depicted, but were greatly reduced by using preprocessing techniques.

Smedes, H. W.↗
NASA NTRS1973

Signature extension techniques applied to multispectral scanner data.

Review of a number of spectral radiance signature extension techniques based on the concept of preprocessing the data to reduce the effects due to atmospheric effects, scanner look angle, etc. One of the promising methods studied to date involves using a ratio preprocessing transformation wherein the signals generated in adjacent spectral bands are ratioed on a point-by-point basis prior to classification. This method is easily and efficiently implemented and tests to date have yielded excellent results. Signatures have been successfully extended over 100+ miles, four days, different times of day, and very different atmospheric conditions.

Nalepka, R. F.↗
NASA NTRS1973

Constrained optimization of image restoration filters.

A preprocessing method to correct for image degradation is proposed which can be thought of as a generalization and extension of previous work by Smith (1966) and Stuller (1972). This method accomodates the problem of noncircularly symmetric imaging system point-spread functions, provides for controlled extent of the preprocessing filter to minimize distortion due to transients resulting from truncation errors and edge effects, can be used with various kinds of system noise, and can be readily extended to provide constraint of other system parameters. The analysis relates to a line-scanner system, although it is applicable in principle to many other system configurations.

Riemer, T. E.↗
NASA NTRS1973

Machine processing methods for earth observational data

A brief review of the development over the last decade of earth resource information systems is presented. Machine data preprocessing and analysis methods are surveyed and illustrated. These include preprocessing steps intended to modify geometric and radiometric aspects of earth observational image data to enhance the ability of either human interpreters or machine algorithms to extract information from the data. Illustrations of processed and analyzed images from spaceborne sensors including the Earth Resources Technology Satellite are discussed.

Landgrebe, D. A.↗
NASA NTRS1976

User data dissemination concepts for earth resources: Executive summary

The impact of the future capabilities of earth-resources data sensors (both satellite and airborne) and their requirements on the data dissemination network were investigated and optimum ways of configuring this network were determined. The scope of this study was limited to the continental U.S.A. (including Alaska) and to the 1985-1995 time period. Some of the conclusions and recommendations reached were: (1) Data from satellites in sun-synchronous polar orbits (700-920 km) will generate most of the earth-resources data in the specified time period. (2) Data from aircraft and shuttle sorties cannot be readily integrated in a data-dissemination network unless already preprocessed in a digitized form to a standard geometric coordinate system. (3) Data transmission between readout stations and central preprocessing facilities, and between processing facilities and user facilities are most economically performed by domestic communication satellites. (4) The effect of the following factors should be studied: cloud cover, expanded coverage, pricing strategies, multidiscipline missions.

Davies, R.↗
NASA NTRS1976

User data dissemination concepts for earth resources

Domestic data dissemination networks for earth-resources data in the 1985-1995 time frame were evaluated. The following topics were addressed: (1) earth-resources data sources and expected data volumes, (2) future user demand in terms of data volume and timeliness, (3) space-to-space and earth point-to-point transmission link requirements and implementation, (4) preprocessing requirements and implementation, (5) network costs, and (6) technological development to support this implementation. This study was parametric in that the data input (supply) was varied by a factor of about fifteen while the user request (demand) was varied by a factor of about nineteen. Correspondingly, the time from observation to delivery to the user was varied. This parametric evaluation was performed by a computer simulation that was based on network alternatives and resulted in preliminary transmission and preprocessing requirements. The earth-resource data sources considered were: shuttle sorties, synchronous satellites (e.g., SEOS), aircraft, and satellites in polar orbits.

Davies, R.↗
NASA NTRS1977

Altimeter waveform software design

Techniques are described for preprocessing raw return waveform data from the GEOS-3 radar altimeter. Topics discussed include: (1) general altimeter data preprocessing to be done at the GEOS-3 Data Processing Center to correct altimeter waveform data for temperature calibrations, to convert between engineering and final data units and to convert telemetered parameter quantities to more appropriate final data distribution values: (2) time "tagging" of altimeter return waveform data quantities to compensate for various delays, misalignments and calculational intervals; (3) data processing procedures for use in estimating spacecraft attitude from altimeter waveform sampling gates; and (4) feasibility of use of a ground-based reflector or transponder to obtain in-flight calibration information on GEOS-3 altimeter performance.

Hayne, G. S.↗
NASA NTRS1977

Cluster compression algorithm: A joint clustering/data compression concept

The Cluster Compression Algorithm (CCA), which was developed to reduce costs associated with transmitting, storing, distributing, and interpreting LANDSAT multispectral image data is described. The CCA is a preprocessing algorithm that uses feature extraction and data compression to more efficiently represent the information in the image data. The format of the preprocessed data enables simply a look-up table decoding and direct use of the extracted features to reduce user computation for either image reconstruction, or computer interpretation of the image data. Basically, the CCA uses spatially local clustering to extract features from the image data to describe spectral characteristics of the data set. In addition, the features may be used to form a sequence of scalar numbers that define each picture element in terms of the cluster features. This sequence, called the feature map, is then efficiently represented by using source encoding concepts. Various forms of the CCA are defined and experimental results are presented to show trade-offs and characteristics of the various implementations. Examples are provided that demonstrate the application of the cluster compression concept to multi-spectral images from LANDSAT and other sources.

Hilbert, E. E.↗