Search NASA⌕ Search

SEARCH · Search NASA

Results for “Preprocessing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

{sup 12}CO(J = 1 → 0) On-the-fly Mapping Survey of the Virgo Cluster Spirals. II. Molecular Gas Properties in Different Density Environments

This study investigated the properties of the molecular gas content and star formation activity of 17 Virgo spirals, 21 Ursa Major (UMa) spirals, 13 Pisces spiral galaxies, and a comparison sample of 11 field spiral galaxies with a spatially resolved gas and stellar distribution. The H i-deficient galaxies with a def{sub H i} > 0.4 have a similar range of CO luminosity normalized by the K-band luminosity (L {sub CO}/L {sub K}) like the field spirals, although their CO content can be smaller by up to a factor of 2. The CO, H i, and stellar disk diameters are closely related to each other for both cluster and field galaxies, and the relative diameters of the CO and H i disks grow monotonically and smoothly as the H i-to-stellar disk diameter ratio decreases. Cluster galaxies have a molecular gas consumption time up to 10 times shorter than that of the field comparison sample, suggesting a significant change in the molecular gas content and star formation activity among all the cluster galaxies, even when they do not show any sign of H i stripping. The strongly H i-stripped Virgo cluster galaxies show only a modestly reduced total gas consumption time, indicating that the star formation activity and gas consumption are a highly local (rather than global) phenomenon. Our finding is that the depletion of cold gas by ram-pressure stripping and/or starvation caused by preprocessing in each cluster environment makes galaxies evolve passively.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2017

Comparison of Two Coronal Magnetic Field Models to Reconstruct a Sigmoidal Solar Active Region with Coronal Loops

Magnetic field extrapolation is an important tool to study the three-dimensional (3D) solar coronal magnetic field, which is difficult to directly measure. Various analytic models and numerical codes exist, but their results often drastically differ. Thus, a critical comparison of the modeled magnetic field lines with the observed coronal loops is strongly required to establish the credibility of the model. Here we compare two different non-potential extrapolation codes, a nonlinear force-free field code (CESE–MHD–NLFFF) and a non-force-free field (NFFF) code, in modeling a solar active region (AR) that has a sigmoidal configuration just before a major flare erupted from the region. A 2D coronal-loop tracing and fitting method is employed to study the 3D misalignment angles between the extrapolated magnetic field lines and the EUV loops as imaged by SDO /AIA. It is found that the CESE–MHD–NLFFF code with preprocessed magnetogram performs the best, outputting a field that matches the coronal loops in the AR core imaged in AIA 94 Å with a misalignment angle of ∼10°. This suggests that the CESE–MHD–NLFFF code, even without using the information of the coronal loops in constraining the magnetic field, performs as good as some coronal-loop forward-fitting models. For the loops as imaged by AIA 171 Å in the outskirts of the AR, all the codes including the potential field give comparable results of the mean misalignment angle (∼30°). Thus, further improvement of the codes is needed for a better reconstruction of the long loops enveloping the core region.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Analytical Modeling of Exoplanet Transit Spectroscopy with Dimensional Analysis and Symbolic Regression

Abstract The physical characteristics and atmospheric chemical composition of newly discovered exoplanets are often inferred from their transit spectra, which are obtained from complex numerical models of radiative transfer. Alternatively, simple analytical expressions provide insightful physical intuition into the relevant atmospheric processes. The deep-learning revolution has opened the door for deriving such analytical results directly with a computer algorithm fitting to the data. As a proof of concept, we successfully demonstrate the use of symbolic regression on synthetic data for the transit radii of generic hot-Jupiter exoplanets to derive a corresponding analytical formula. As a preprocessing step, we use dimensional analysis to identify the relevant dimensionless combinations of variables and reduce the number of independent inputs, which improves the performance of the symbolic regression. The dimensional analysis also allowed us to mathematically derive and properly parameterize the most general family of degeneracies among the input atmospheric parameters that affect the characterization of an exoplanet atmosphere through transit spectroscopy.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

H I Depletion Begins Well Beyond the Virial Radius: A FAST Stacking Study of 36 Galaxy Clusters to 5 × R 200

Abstract We present a stacking study of the neutral atomic hydrogen (H i ) content in and around 36 local galaxy clusters at z < 0.07, using a combination of the FAST All Sky H i survey (FASHI) and the extensive spectroscopic catalog mainly from the Dark Energy Spectroscopic Instrument (DESI). We employ spectral stacking techniques to probe the average H i mass and HI-to-stellar mass ratio ( M HI / M * ) for member galaxies down to stellar masses of M * ∼ 10 9 M ⊙ , spanning a projected cluster-centric distance of up to 5 R 200 . Our analysis reveals a pronounced environmental effect; both M HI and M HI / M * decrease steadily toward the cluster center, dropping by ∼0.5 dex on average from the outskirts to the core. Crucially, we find that M HI / M * of galaxies remain lower than the field galaxies even at the 5 R 200 . This provides direct, statistical evidence for substantial gas stripping and preprocessing in the cluster outskirts, likely occurring in infalling groups and large-scale filaments. By further splitting the sample by g − r color, we show that the H i deficiency persists at fixed galaxy color; even the bluest cluster members exhibit ∼0.5 dex lower M HI / M * than field galaxies of similar color, reflecting environmental effects on the cold gas reservoir prior to full optical transformation. The total H i mass within clusters and their outskirts agrees broadly with predictions from cosmological simulations. Our results underscore the critical role of the extended cluster environment in quenching galaxies by depleting their cold gas reservoirs well before they enter the dense cluster core.

Cheng, Cheng [Chinese Academy of Sciences South Am↗
DOE OSTI2022

Unsupervised Machine Learning for Exploratory Data Analysis of Exoplanet Transmission Spectra

Abstract Transit spectroscopy is a powerful tool for decoding the chemical compositions of the atmospheres of extrasolar planets. In this paper, we focus on unsupervised techniques for analyzing spectral data from transiting exoplanets. After cleaning and validating the data, we demonstrate methods for: (i) initial exploratory data analysis, based on summary statistics (estimates of location and variability); (ii) exploring and quantifying the existing correlations in the data; (iii) preprocessing and linearly transforming the data to its principal components; (iv) dimensionality reduction and manifold learning; (v) clustering and anomaly detection; and (vi) visualization and interpretation of the data. To illustrate the proposed unsupervised methodology, we use a well-known public benchmark data set of synthetic transit spectra. We show that there is a high degree of correlation in the spectral data, which calls for appropriate low-dimensional representations. We explore a number of different techniques for such dimensionality reduction and identify several suitable options in terms of summary statistics, principal components, etc. We uncover interesting structures in the principal component basis, namely well-defined branches corresponding to different chemical regimes of the underlying atmospheres. We demonstrate that those branches can be successfully recovered with a K-means clustering algorithm in a fully unsupervised fashion. We advocate for lower-dimensional representations of the spectroscopic data in terms of the main principal components, in order to reveal the existing structure in the data and quickly characterize the chemical class of a planet.

Matchev, Konstantin T. (ORCID:0000000341829096)↗
DOE OSTI2021

Inpainting radar missing data regions with deep learning

Abstract. Missing and low-quality data regions are a frequent problem for weather radars. They stem from a variety of sources: beam blockage, instrument failure, near-ground blind zones, and many others. Filling in missing data regions is often useful for estimating local atmospheric properties and the application of high-level data processing schemes without the need for preprocessing and error-handling steps – feature detection and tracking, for instance. Interpolation schemes are typically used for this task, though they tend to produce unrealistically spatially smoothed results that are not representative of the atmospheric turbulence and variability that are usually resolved by weather radars. Recently, generative adversarial networks (GANs) have achieved impressive results in the area of photo inpainting. Here, they are demonstrated as a tool for infilling radar missing data regions. These neural networks are capable of extending large-scale cloud and precipitation features that border missing data regions into the regions while hallucinating plausible small-scale variability. In other words, they can inpaint missing data with accurate large-scale features and plausible local small-scale features. This method is demonstrated on a scanning C-band and vertically pointing Ka-band radar that were deployed as part of the Cloud Aerosol and Complex Terrain Interactions (CACTI) field campaign. Three missing data scenarios are explored: infilling low-level blind zones and short outage periods for the Ka-band radar and infilling beam blockage areas for the C-band radar. Two deep-learning-based approaches are tested, a convolutional neural network (CNN) and a GAN that optimize pixel-level error or combined pixel-level error and adversarial loss respectively. Both deep-learning approaches significantly outperform traditional inpainting schemes under several pixel-level and perceptual quality metrics.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2018

The Variable Infiltration Capacity model version 5 (VIC-5): infrastructure improvements for new applications and reproducibility

The Variable Infiltration Capacity (VIC) model is a macroscale semi-distributed hydrologic model. VIC development began in the early 1990s and the model has since been used extensively for basin- to global-scale applications that include hydrologic dataset construction, trend analysis of hydrologic fluxes and states, data evaluation and assimilation, forecasting, coupled climate modeling, and climate change impact assessment. Ongoing operational applications of the VIC model include the University of Washington's drought monitoring and forecasting systems and NASA's Land Data Assimilation System. This paper documents the development of VIC version 5 (VIC-5), which includes a major reconfiguration of the legacy VIC source code to support a wider range of modern hydrologic modeling applications. The VIC source code has been moved to a public GitHub repository to encourage participation by the broader user and developer communities. The reconfiguration has separated the core physics of the model from the driver source code, whereby the latter is responsible for memory allocation, preprocessing and post-processing, and input–output (I–O). VIC-5 includes four drivers that use the same core physics modules, but which allow for different methods for accessing this core to enable different model applications. Finally, VIC-5 is distributed with robust test infrastructure, components of which routinely run during development using cloud-hosted continuous integration. The work described here provides an example to the model development community for extending the life of a legacy model that is being used extensively. The development and release of VIC-5 represents a significant step forward for the VIC user community in terms of support for existing and new model applications, reproducibility, and scientific robustness.

42 ENGINEERING↗
DOE OSTI2020

Improving climate model coupling through a complete mesh representation: a case study with E3SM (v1) and MOAB (v5.x)

One of the fundamental factors contributing to the spatiotemporal inaccuracy in climate modeling is the mapping of solution field data between different discretizations and numerical grids used in the coupled component models. The typical climate computational workflow involves evaluation and serialization of the remapping weights during the preprocessing step, which is then consumed by the coupled driver infrastructure during simulation to compute field projections. Tools like Earth System Modeling Framework (ESMF) and TempestRemap offer capability to generate conservative remapping weights, while the Model Coupling Toolkit (MCT) that is utilized in many production climate models exposes functionality to make use of the operators to solve the coupled problem. However, such multistep processes present several hurdles in terms of the scientific workflow and impede research productivity. In order to overcome these limitations, we present a fully integrated infrastructure based on the Mesh Oriented datABase (MOAB) library, which allows for a complete description of the numerical grids and solution data used in each submodel. Through a scalable advancing-front intersection algorithm, the supermesh of the source and target grids are computed, which is then used to assemble the high-order, conservative, and monotonicity-preserving remapping weights between discretization specifications. The Fortran-compatible interfaces in MOAB are utilized to directly link the submodels in the Energy Exascale Earth System Model (E3SM) to enable online remapping strategies in order to simplify the coupled workflow process. We demonstrate the superior computational efficiency of the remapping algorithms in comparison with other state-of-the-science tools and present strong scaling results on large-scale machines for computing remapping weights between the spectral element atmosphere and finite volume discretizations on the polygonal ocean grids.

58 GEOSCIENCES↗
DOE OSTI2020

Janus: A Python Package for Agent-Based Modeling of Land Use and Land Cover Change

Janus is an open source Python package for agent-based modeling (ABM) of land use and land cover change (LULCC). Many ABMs of LULCC have been created across platforms, some of which are not ideal for large scale, high resolution scenarios. This model provides a simple object-oriented framework for creating ABMs specific to LULCC. The organizational philosophy of the modeling framework is to create software objects (agents) that are associated with specific and contextual attributes which are isolated from where those agents exist in the spatial setting of the model, yet provide clear linkages between the agent, their environment, and other agents in the simulation. In this way, the framework allows for assembly of LULCC ABMs with low (programmatic) overhead, making the models extensible and providing clear mechanisms for integrating them with process-oriented biophysical models. Provided with Janus is a suite of geospatial data preprocessing tools that can use arbitrary land cover products as an input. Crop choice decisions are based on potential crop prices, these can be created synthetically, or drawn from integrated human-Earth systems models such as the GCAM. Janus is publicly accessible through GitHub and provides an example dataset for testing.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

NLR HPC Kestrel Jobs Data

Overview: Anonymized job-level records from the Kestrel HPC system at the National Laboratory of the Rockies (NLR). Each record represents a Slurm batch job with scheduling metadata, resource requests, utilization, energy estimates, and efficiency metrics. Sensitive fields (user, account, job name, submit line, working directory, submit script, and job type) are replaced with 7-character cryptographic hashes. System & Timeframe: Kestrel is located at the NLR campus. Standard compute nodes have 104 cores and 256 GB RAM; bigmem nodes have 2,000 GB. GPU nodes (gpu-h100 partition) use NVIDIA H100 GPUs. Data covers jobs submitted August 2023 through December 2025. Funding provided by the U.S. Department of Energy, EERE. Files: esif.hpc.kestrel.job-anon.zip — Anonymized job records (Hive-partitioned Parquet) datacard.md — Full dataset documentation ~11 million rows, 50 variables. Readable with PyArrow, pandas, DuckDB, Apache Spark, or any Parquet-compatible tool. Data Collection: Jobs collected via sacct with timezone-aware export (SLURM_TIME_FORMAT="%Y-%m-%dT%H:%M:%S%z"), loaded into PostgreSQL. Calculated columns updated via database triggers and batch functions. All timestamps use timestamptz and correctly handle DST transitions. Preprocessing: Anonymization of name, user, account, submit_line, work_dir, submit_script, and job_type via 7-char hex hashes Derived columns: queue_wait, cpu_eff, max/min/avg_mem_eff, energy estimates Simplified job state mapping (e.g., "CANCELLED by 132357" → "CANCELLED") Boolean flags: python_job, reframe_job Temporal decomposition: year, month, day, day_of_week, hour, minute from submit_time Shared node tracking: shared_job_count, nodes_shared, jobs_shared Key Variables: Scheduling: job_id, partition, state_simple, submit_time, start_time, end_time, queue_wait Resources: nodes_req/used, processors_req/used, memory_req, wallclock_req/used, gpus_requested Efficiency: cpu_eff, max/min/avg_mem_eff Energy: cpu_energy_tdp_estimated_max/used_watt_hours, consumed_energy_raw_joules, consumed_energy_raw_watt_hours Sharing: shared_job_count, nodes_shared, jobs_shared Partitions: short, standard, debug, gpu-h100 Job States: CANCELLED, COMPLETED, FAILED, PENDING, RUNNING QoS Levels: normal, high Important Notes: Timestamps include timezone offsets; DST transitions are handled correctly, though adding intervals across DST boundaries requires offset adjustment shared_job_count reflects physical node co-residency, not use of the shared partition Job step records and raw Slurm JSONB fields are excluded Do not attempt to re-identify individuals from hashed fields

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

NLR HPC Eagle Jobs Data and Additional Energy Metrics

Overview: Anonymized job-level records from the Eagle high-performance computing (HPC) system at the National Laboratory of the Rockies (NLR). Each record represents a Slurm batch job with scheduling metadata, resource requests, resource utilization, CPU/GPU energy consumption, and efficiency metrics. Sensitive fields (user, account, job name) are replaced with cryptographic hashes. System & Timeframe: Eagle was a 2,000-node, 8-petaflop system operated at NLR from 2019–2024. Data covers the full operational lifetime of the system. Slurm data was processed nightly; timestamps are in Mountain Time. Funding provided by the U.S. Department of Energy, EERE. Files: esif.hpc.eagle.job-anon.zip — Core anonymized job records (Hive-partitioned Parquet) esif.hpc.eagle.job-anon-energy-metrics.zip — Same records with additional iLO and Ganglia energy metrics datacard.md — Full dataset documentation ~13.8 million rows, 62 variables. Readable with PyArrow, pandas, DuckDB, Apache Spark, or any Parquet-compatible tool. Data Collection: Jobs collected via sacct through a pipeline: Eagle Jobs API → Redpanda → StreamSets → HPCMON API → PostgreSQL. Node-level power from iLO (HP Integrated Lights-Out); GPU power from Ganglia monitoring, joined to jobs via node lists and time ranges. Preprocessing: Anonymization of name, user, and account fields via cryptographic hashing Derived columns: queue_wait, cpu_eff, max_mem_eff Simplified job state mapping (e.g., "CANCELLED BY 12345" → "CANCELLED") QoS accounting rules (buy-in, standby, or Slurm QoS value) CPU energy estimated from TDP (200W, Intel Xeon Gold 6154, 18 cores) Timezone-aware columns (_tz) sourced from LEX accounting database to correctly handle DST transitions Key Variables: Scheduling: job_id, partition, state_simple, submit_time_tz, start_time_tz, end_time_tz, queue_waitResources: nodes_req/used, processors_req/used, memory_req, wallclock_req/used, gpus_requested Efficiency: cpu_eff, max_mem_eff Energy: cpu_energy_tdp_estimated_max/used_watt_hours, node_energy_total_watt_hours (iLO), gpu0/1_energy_total_watt_hours (Ganglia) Partitions: bigmem, bigmem-8600, bigscratch, csc, dav, ddn, debug, gpu, haswell, long, mono, short, standard Job States: CANCELLED, COMPLETED, FAILED, NODE_FAIL, OUT_OF_MEMORY, PENDING, RUNNING, TIMEOUT QoS Levels: Unknown, normal, buy-in, debug, penalty, high, standby Important Notes: Non-_tz timestamp columns may be off by one hour across DST boundaries; use _tz columns for time difference calculations Energy fields are null for jobs without monitoring coverage Job step records and raw Slurm JSONB fields are excluded from this extract Do not attempt to re-identify individuals from hashed fields

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

To The Biorefinery: Delivered Forestland and Agricultural Resources

It can be challenging and costly to transport biomass feedstock supplies from the roadside, or farmgate, to a biorefinery. Given the geographic dispersion and lowbulk density of cellulosic feedstocks, cost effective scaling of commercial biorefinery operations requires overcoming many challenges. The Biomass Research and Development Board’s Feedstock Logistics Interagency Working Group identified four primary barriers related to biorefinery commercialization: • Capacity and efficiency of harvest and collection equipment • High-moisture content leading to degradation of biomass • Variable biomass quality upon arrival at the biorefinery • Costly transportation options.1 Further, feedstock supply systems do not currently mitigate risks such as low crop yield, fire, or competition for resource use. Delivery and preprocessing improvements will allow for the development of a commercial-scale bioenergy industry that achieves national production and cost targets.

To The Biorefinery Factsheet↗
DOE OSTI2017

Image processing tool for automatic feature recognition and quantification

A system for defining structures within an image is described. The system includes reading of an input file, preprocessing the input file while preserving metadata such as scale information and then detecting features of the input file. In one version the detection first uses an edge detector followed by identification of features using a Hough transform. The output of the process is identified elements within the image.

36 MATERIALS SCIENCE↗
DOE OSTI2017

Method for discovering relationships in data by dynamic quantum clustering

Data clustering is provided according to a dynamical framework based on quantum mechanical time evolution of states corresponding to data points. To expedite computations, we can approximate the time-dependent Hamiltonian formalism by a truncated calculation within a set of Gaussian wave-functions (coherent states) centered around the original points. This allows for analytic evaluation of the time evolution of all such states, opening up the possibility of exploration of relationships among data-points through observation of varying dynamical-distances among points and convergence of points into clusters. This formalism may be further supplemented by preprocessing, such as dimensional reduction through singular value decomposition and/or feature filtering.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Sandia Engineering Analysis Code Access System v. 2.0.1

The Sandia Engineering Analysis Code Access System (SEACAS) is a suite of preprocessing, post processing, translation, visualization, and utility applications supporting finite element analysis software using the Exodus database file format.

Sjaardema, Gregory D.↗
DOE OSTI2016

Testing Feedstocks: INL Researcher Finds Way to Accurately Test Flowability

Here, cellulosic biofuels plants have been plagued by problems caused by the poor feeding and handling characteristics of biomass feedstocks. A new method has been developed to measure flowability of feedstocks to make sure that they will feed properly before introducing them into processes in which they may plug equipment. The new method uses an instrumented hopper that can be configured to provide real-time, in-line flowability characterization of feedstocks within supply and preprocessing operations for quality assurance and control. There is no other available method that measures flowability that is compatible with real-time, in-line data collection for quality control.

09 BIOMASS FUELS↗
DOE OSTI2018

Assessing Potential Air Pollutant Emissions from Agricultural Feedstock Production using MOVES

Biomass feedstock production is expected to grow as demand for biofuels and bioenergy increases. The change in air pollutant emissions that may result from large-scale biomass supply has implications for local air quality and human health. We developed spatially explicit emissions inventories for corn grain and six cellulosic feedstocks through the extension of the National Renewable Energy Laboratory's Feedstock Production Emissions to Air Model (FPEAM). These inventories include emissions of seven pollutants (nitrogen oxides, ammonia, volatile organic compounds, particulate matter, sulfur oxides, and carbon monoxide) generated from biomass establishment, maintenance, harvest, transportation, and biofuel preprocessing activities. By integrating the EPA's MOtor Vehicle Emissions Simulator (MOVES) into FPEAM, we created a scalable framework to execute county-level runs of the MOVES-Onroad model for representative counties (i.e., those counties with the largest amount of cellulosic feedstock production in each state) on a national scale. We used these results to estimate emissions from the on-road transportation of biomass and combined them with county-level runs of the MOVES-Nonroad model to estimate emissions from agricultural equipment. We also incorporated documented emission factors to estimate emissions from chemical application and the operation of drying equipment for feedstock processing, and used methods developed by the EPA and the California Air Resources Board to estimate fugitive dust emissions. The model developed here could be applied to custom equipment budgets and is extensible to accommodate additional feedstocks and pollutants. Future work will also extend this model to analyze spatial boundaries beyond the county-scale (e.g., regional or sub-county levels).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

URBAN-NET-TOOLKIT

Strengthening and maintaining secure, functioning, and resilient Critical Infrastructure Systems (CISs) is one of our nation's primary goals, and it requires proactive and coo0rdinated efforts among federal, state, local, public and private owners and operators of CISs. Recent worldwide extreme events have shown how interdependencies across CISs (electric grid, telecommunications, transportation, water networks, etc.) can drastically increase system vulnerability. To understand these complex interdependencies, ORNL has instantiated Urban-NET, a novel decision support system for CIS operators who need to identify vulnerabilities for minimizing the cascading impacts - saving utilities, the federal and state government, and private sector companies potentially billions by avoiding a complete CI failure and/or shutdown. The developed system includes a stack of software tools for preprocessing, modeling, interconnecting multiple CIS datasets as an integrated heterogeneous graph (i.e. network), identifying vulnerabilities through topological analysis, and predicting consequences based on simulations along with visualization.

Chinthavali, Supriya↗