Search NASA⌕ Search

SEARCH · Search NASA

Results for “Process variable”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,135 records · Page 63

PERMEABILITY OF SALTSTONE MEASUREMENT BY BEAM BENDING

One of the goals of the Saltstone variability study is to identify (and, quantify the impact of) the operational and compositional variables that control or influence the important processing and performance properties of Saltstone mixes. A performance property for Saltstone mixes that is important but not routinely measured is the liquid permeability or saturated hydraulic conductivity of the cured Saltstone mix. The value for the saturated hydraulic conductivity is an input into the Performance Assessment for the SRS Z-Area vaults. Therefore, it is important to have a method available that allows for an accurate and reproducible measurement of permeability quickly and inexpensively. One such method that could potentially meet these requirements for the measurement of saturated hydraulic conductivity is the technique of beam bending, developed by Professor George Scherer at Princeton University. In order to determine the feasibility of this technique for Saltstone mixes, a summer student, David Feliciano, was hired to work at Princeton under the direction of George Scherer. This report details the results of this study which demonstrated the feasibility and applicability of the beam bending method to measurement of permeability of Saltstone samples. This research effort used samples made at Princeton from a Modular Caustic side solvent extraction Unit based simulant (MCU) and premix at a water to premix ratio of 0.60. The saturated hydraulic conductivities for these mixes were measured by the beam bending technique and the values determined were of the order of 1.4 to 3.4 x 10{sup -9} cm/sec. These values of hydraulic conductivity are consistent with independently measured values of this property on similar MCU based mixes by Dixon and Phifer. These values are also consistent with the hydraulic conductivity of a generic Saltstone mix measured by Langton in 1985. The high water to premix ratio used for Saltstone along with the relatively low degree of hydration for MCU based mixes leads to high total and capillary porosities. These two conditions generally lead to higher permeabilities as has been well documented in the literature for typical cementitious pastes in water. Therefore, it is not unexpected that the hydraulic conductivities of these Saltstone mixes are relatively high.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

MEASUREMENT OF WASTE LOADING IN SALTSTONE

One of the goals of the Saltstone variability study is to identify the operational and compositional variables that control or influence the important processing and performance properties of Saltstone grout mixtures. One of those properties of importance is the Waste Loading (WL) of the decontaminated salt solution (DSS) in the Saltstone waste form. Waste loading is a measure of the amount of waste that can be incorporated within a waste form. The value of the Saltstone waste loading ultimately determines the number of vaults that will be required to disposition all of the DSS. In this report, the waste loading is defined as the volume in milliliters of DSS per liter of Saltstone waste form. The two most important parameters that determine waste loading for Saltstone are water to cementitious material (w/cm) ratio and the cured grout density. Data are provided that show the dependence of waste loading on the w/cm ratio for a fixed DSS composition using the current premix material (45% Blast Furnace Slag (BFS), 45% Fly Ash (FA) and 10% Ordinary Portland Cement (OPC)). The impact of cured grout density on waste loading was also demonstrated. Mixes (at 0.60 w/cm) made with a Modular Caustic side extraction Unit (MCU) simulant and either OPC or BFS have higher cured grout densities than mixes made with premix and increase the WL to 709 mL/L for the OPC mix and 689 mL/L for the BFS mix versus the value of 653 mL/L for MCU in premix at 0.60 w/cm ratio. Bleed liquid reduces the waste loading and lowers the effective w/cm ratio of Saltstone. A method is presented (and will be used in future tasks) for correcting the waste loading and the w/cm ratio of the as-batched mixes in those cases where bleed liquid is present. For example, the Deliquification, Dissolution and Adjustment (DDA) mix at an as-batched 0.60 w/cm ratio, when corrected for % bleed, gives a mix with a 0.55 w/cm ratio and a WL that has been reduced from 662 to 625 mL/L. An example is provided that demonstrated the quantitative impact of WL on the number of cells (each Saltstone vault contains two cells) required to disposition all of the {approx}100 million gallons of DSS available in the tanks. This calculation revealed that the number of cells required over the range of 0.48 to 0.62 w/cm ratio (equivalent to a WL range of 591 to 666 mL/L) varies from 65 to 57 cells (33 to 29 vaults). The intent of this oversimplified example was to show the range of variation in vaults expected due to w/cm ratio rather than to estimate the actual number of vaults required. There is a tradeoff between the waste loading and the processing and performance properties of Saltstone. The performance properties improve in general as the w/cm ratio decreases whereas the waste loading is reduced at lower w/cm ratios resulting in a larger number of Saltstone vaults. The final performance and processing requirements of Saltstone will determine the maximum waste loading achievable.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

SOLAR ATMOSPHERIC MAGNETIC ENERGY COUPLING: BROAD PLASMA CONDITIONS AND SPECTRUM REGIMES

Solar variability investigations that include magnetic energy coupling are paramount to solving many key solar/stellar physics problems, particularly for understanding the temporal variability of magnetic energy redistribution and heating processes. Using three years of observations from the Solar Dynamics Observatory ’ s Atmospheric Imaging Assembly and Heliosemic Magnetic Imager, we measured radiative and magnetic fluxes from gross features and at full-disk scales, respectively. Magnetic energy coupling analyses support radiative flux descriptions via the plasma heating connectivity of dominant (magnetic) and diffuse components, specifically of the predominantly closed-field corona. Our work shows that this relationship favors an energetic redistribution efficiency across large temperature gradients, and potentially sheds light on the long-standing issue of diffuse unresolved low corona emission. The close connection between magnetic energy redistribution and plasma conditions revealed by this work lends significant insight into the field of stellar physics, as we have provided possible means for probing distant sources in currently limited and/or undetectable radiation distributions.

79 ASTRONOMY AND ASTROPHYSICS↗

Observational ozone datasets over the global oceans and polar regions (version 2024)

Studying tropospheric ozone over the remote areas of the planet, such as the open oceans and the polar regions, is crucial to understand the role of ozone as a global climate forcer and regulator of atmospheric oxidative capacity. A focus on the pristine oceanic and polar regions complements the available land-based datasets and provides insights into key photochemical and depositional loss processes that control the concentrations and spatiotemporal variability in ozone as well as the physicochemical mechanisms driving these patterns. However, an assessment of the role of ozone over the oceanic and polar regions has been hampered by a lack of comprehensive observational datasets. Here, we present the first comprehensive collection of ozone data over the oceans and the polar regions. The overall dataset consists of 77 ship cruises/buoy-based observations and 48 aircraft-based campaigns. The dataset, consisting of more than 630 000 independent ozone measurement data points covering the period from 1977 to 2022 and an altitude range from the surface to 5000 m (with a focus on the lowest 2000 m), allows systematic analyses of the spatiotemporal distribution and long-term trends over the 11 defined ocean/polar regions. The datasets from ships, buoys, and aircraft are complemented by ozonesonde data from 29 launch sites or field campaigns and by 21 non-polar and 17 polar ground-based station datasets. The datasets contain information on how long the observed air masses were isolated from land, as estimated by backward trajectories from the individual observation points. To extract observations representative of oceanic conditions, we recommend using a subset of the data with an isolation time of 72 h or longer, from the analysis with coincident radon observations. These filtered oceanic and polar data showed typically flat diurnal cycles at high latitudes, whereas daytime decreases in ozone (11 %–16 %) were observed at lower latitudes. The ship/buoy- and aircraft-based datasets presented here will supplement the land-based ones in the TOAR-II (Tropospheric Ozone Assessment Report Phase II) database to provide a fully global assessment of tropospheric ozone. The described dataset is available at https://doi.org/10.17596/0004044 (Kanaya et al., 2025).

Kanaya, Yugo [Japan Agency for Marine-Earth Scienc↗

Deep-learning-driven simulations of boundary layer clouds over the Southern Great Plains

Abstract. Based on long-term observations at the Southern Great Plains site by the Atmospheric Radiation Measurement (ARM) program for training and validation, a deep-learning model is developed to simulate the daytime evolution of boundary layer clouds (BLCs) from the perspective of land–atmosphere coupling. The model takes ARM measurements (including early-morning soundings and diurnally varying surface meteorological conditions and heat fluxes) as inputs and predicts hourly estimates (including cloud occurrence, the positions of cloud boundaries, and the vertical profile of the cloud fraction) as outputs. The deep-learning model offers good agreement with the observed cloud fields, especially in the accuracy with which cloud occurrence and base height are reproduced. When the inputs are substituted by reanalysis data from ERA5 and MERRA-2, the outputs of the deep-learning model provide a better agreement with observation than the cloud fields extracted from ERA5 and MERRA-2 themselves. Thus, the deep-learning model shows great potential to serve as a diagnostic tool for the performance of physics-based models in simulating stratiform and cumulus clouds. By quantifying biases in clouds and attributing them to the simulated atmospheric state variables versus the model-parameterized cloud processes, this observation-based deep-learning model may offer insights into the directions needed to improve the simulation of BLCs in physics-based models for weather forecasting and climate prediction.

54 ENVIRONMENTAL SCIENCES↗

PyRAT - python radiography analysis tool (u)

PyRAT is a radiography analysis tool used to reconstruction images of unknown 1-0 objects. The tool is written in Python and developed for use on LINUX and Windows platforms. The tool is capable of performing nonlinear inversions of the images with minimal manual interaction in the optimization process. The tool utilizes the NOMAD mixed variable optimization tool to perform the optimization.

97 MATHEMATICS AND COMPUTING↗

UNCERTAINTY ANALYSIS FOR THE TECHA RIVER DOSIMETRY SYSTEM

Uncertainties in the doses estimated for the members of the Techa River Cohort (TRC) are being estimated with a two-dimensional Monte Carlo approach. In order to provide more accurate and precise estimates of individual dose (and thus more precise estimates of radiation risk) for the members of the TRC, a new dosimetric calculation system, the Techa River Dosimetry System-2009 (TRDS-2009) has been prepared. The deterministic version of the improved dosimetry system TRDS-2009D was basically completed in April 2009. Recent developments in evaluation of dose-response models in light of uncertain dose have highlighted the importance of different types of uncertainties in the development of individual dose estimates. These include uncertain parameters that may be either shared (common to some or all individuals) or unshared (a unique value for each person whose dose is to be estimated) within the dosimetric cohort. The nature of the type of uncertainty may be aleatory (random variability of true values due to stochastic processes) or epistemic (due to lack of complete knowledge about a unique quantity). Finally, there is a need to identify whether the structure of the errors is either related to measurement (the estimate differs from the true value by an error that is stochastically independent of the true value; frequently called classical uncertainty) or related to grouping (the true value varies from the estimate by an error that is random and is independent of the estimate; frequently called Berkson uncertainty). An approach has been developed that identifies the nature of the various input parameters and calculational methods incorporated in the Techa River Dosimetry System (based on the TRDS-2009D implementation), and a stochastic calculation model has been prepared to estimate the uncertainties in the dose estimates. This article reviews the concepts of uncertainty analysis, the equations, and input parameters, and then identifies the authors’ interpretations of their general nature. It presents the approach selected so that the stochastic, Monte-Carlo, implementation of the dosimetry system - TRDS-2009MC - provides useful information regarding the uncertainties of the doses.

uncertainty analysis, radiation dosimetry, Techa R↗

Occurrence and Stabilization of Colloidal and Nanoparticulate Mercury Sulfides (Final Technical Report)

Mercury contaminates the soils at several DOE sites, including facilities at Oak Ridge, Hanford, and the Savannah River site. This contamination poses a hazard to surrounding ecosystems because the metal continuously leaches to local streams and rivers and is converted to methylmercury, a neurotoxic and highly bioaccumulative form of mercury. Anaerobic microorganisms such as sulfate-reducing and iron-reducing bacteria are primarily responsible for methylmercury production in the environment. The remediation of mercury-contaminated ecosystems is difficult because we lack a poor understanding of the factors that control mercury mobilization and methylation in sediments. The objective of this project was to investigate the forms of mercury that are bioavailable to methylating bacteria. This project specifically focused on mercury-sulfide-organic matter geochemistry and the persistence of nanoparticulate HgS that are intermediates of metal sulfide precipitation reactions occurring in the presence of natural organic matter (NOM). This work involved two specific aims: 1) Investigate environmental processes that facilitate the occurrence and stabilization of nanoparticulate HgS and other metal sulfides; and (2) Investigate the occurrence of nanoparticulate HgS in sediments of the East Fork Poplar Creek, located in near a DOE facility in Oak Ridge, TN. For specific aim #1, laboratory experiments were performed to simulate metal sulfide precipitation processes in environmental systems. A key experimental variable was the type and concentration of NOM that was used in the experiments simulating metal sulfide precipitation. The characteristics of the metal sulfide nanoparticles were analyzed using a suite of analytical techniques that included X-ray based methods at DOE synchrotron facilities (National Synchrotron Light Source and Advanced Photon Source). Field sampling in Oak Ridge was performed in collaboration with colleague at the Oak Ridge National Laboratory. The experimental results demonstrated that a combination of photon scattering and spectroscopic tools can be used to characterize nanoscale structures that form during heterogeneous metal-sulfide-NOM precipitation. ZnS precipitation experiments with a low molecular weight thiol indicated the importance of specific metal-ligand binding for altering the kinetics of ZnS nanoparticle aggregation and monomer growth. However, NOM consists of macromolecular structures that may interact with nano-HgS in non-specific ways (e.g. electrosteric surface interactions. The methylation potential of nano-HgS in sediments may depend on size and crystallinity of nanoparticles and interactions with NOM. For specific aim #2, sediment core samples were collected in three separate locations along the East Fork Poplar Creek in Oak Ridge, TN in June 2011. Analysis of the sediment samples indicated that more than 90% of the Hg in the porewater is associated with colloidal material. These results highlight the importance of small particles for dominating Hg speciation in sediment porewater. Overall, this project highlights the importance of nanoscale phases for mercury speciation. Nanoparticulate HgS will be of particular importance in the anaerobic habitats of methylating microorganisms. In previous models for Hg bioavailability, nanoparticles of Hg are presumed to be unavailable for methylating bacteria. However, recent work shows that nanoparticles contribute to bioavailable Hg that greatly exceeds the contribution from larger HgS microparticles (Zhang et al., 2012). The work performed for this exploratory project lays the groundwork for further study on nanoscale mercury sulfide geochemistry and contributes to an improved understanding of Hg bioavailability and methylation potential.

54 ENVIRONMENTAL SCIENCES↗

Effects of Iris Surface Curvature on Iris Recognition

To focus on objects at various distances, the lens of the eye must change shape to adjust its refractive power. This change in lens shape causes a change in the shape of the iris surface which can be measured by examining the curvature of the iris. This work isolates the variable of iris curvature in the recognition process and shows that differences in iris curvature degrade matching ability. To our knowledge, no other work has examined the effects of varying iris curvature on matching ability. To examine this degradation, we conduct a matching experiment across pairs of images with varying degrees of iris curvature differences. The results show a statistically signi cant degradation in matching ability. Finally, the real world impact of these ndings is discussed

Thompson, Joseph T↗

UQ Toolkit v 2.0

The Uncertainty Quantification (UQ) Toolkit is a software library for the characterizaton and propagation of uncertainties in computational models. For the characterization of uncertainties, Bayesian inference tools are provided to infer uncertain model parameters, as well as Bayesian compressive sensing methods for discovering sparse representations of high-dimensional input-output response surfaces, and also Karhunen-Loève expansions for representing stochastic processes. Uncertain parameters are treated as random variables and represented with Polynomial Chaos expansions (PCEs). The library implements several spectral basis function types (e.g. Hermite basis functions in terms of Gaussian random variables or Legendre basis functions in terms of uniform random variables) that can be used to represent random variables with PCEs. For propagation of uncertainty, tools are provided to propagate PCEs that describe the input uncertainty through the computational model using either intrusive methods (Galerkin projection of equations onto basis functions) or non-intrusive methods (perform deterministic operation at sampled values of the random values and project the obtained results onto basis functions).

Source record↗

Hiereachical Bayesian Model for Combining Geochemical and Geophysical Data for Environmental Applications Software

Development of a hierarchical Bayesian model to estimate the spatiotemporal distribution of aqueous geochemical parameters associated with in-situ bioremediation using surface spectral induced polarization (SIP) data and borehole geochemical measurements collected during a bioremediation experiment at a uranium-contaminated site near Rifle, Colorado. The SIP data are first inverted for Cole-Cole parameters including chargeability, time constant, resistivity at the DC frequency and dependence factor, at each pixel of two-dimensional grids using a previously developed stochastic method. Correlations between the inverted Cole-Cole parameters and the wellbore-based groundwater chemistry measurements indicative of key metabolic processes within the aquifer (e.g. ferrous iron, sulfate, uranium) were established and used as a basis for petrophysical model development. The developed Bayesian model consists of three levels of statistical sub-models: 1) data model, providing links between geochemical and geophysical attributes, 2) process model, describing the spatial and temporal variability of geochemical properties in the subsurface system, and 3) parameter model, describing prior distributions of various parameters and initial conditions. The unknown parameters are estimated using Markov chain Monte Carlo methods. By combining the temporally distributed geochemical data with the spatially distributed geophysical data, we obtain the spatio-temporal distribution of ferrous iron, sulfate and sulfide, and their associated uncertainity information. The obtained results can be used to assess the efficacy of the bioremediation treatment over space and time and to constrain reactive transport models.

Chen, Jinsong↗

An Evaluation of OpenSHMEM Interfaces for the Variable-length Alltoallv() Collective Operation

Alltoallv() is a collective operation which allows all pro- cesses to exchange variable amounts of data with all other processes in the communication group. This means that Alltoallv() requires not only O(N2) communications, but typically also additional exchanges of the data lengths that will be transmitted in the eventual Alltoallv() call. This pre-exchange is used to calculate the proper offsets for the re- ceiving buffers on the target processes. However, we propose two new can- didate interfaces for Alltoallv() that would mitigate the need for the user to set up this extra exchange of information at the possible cost of memory efficiency. We explain the new interface variants and show how a single call can be used in place of the traditional Alltoall()/Alltoallv() pair. We then discuss the performance tradeoffs for overall communica- tion and memory costs, as well as both software and hardware-based optimizations and their applicability to the various proposed interfaces.

Lopez, Matthew Graham↗

Subaru Spectroscopy of SDSS-II Supernovae

The Sloan Digital Sky Survey II (SDSS-II) Supernova Survey discovered Type Ia supernovae (SNe Ia) in an almost unexplored intermediate redshift range of $0.05 < z < 0.4$ and provided densely sampled multi-color light curves for SN candidates. Followup spectroscopy of this survey was carried out with the Subaru telescope and spectra of 71 SN Ia candidates were obtained. One spectrum was observed per candidate except for a peculiar variable. This paper presents the method for processing these spectra. The observed wavelength ranges of our spectra are 4000 to 9000 {\AA} for Year 2005 and 3600 to 9000 {\AA} for Year 2006. Most SN Ia spectra have signal to noise ratios (S/N) between 4 and 10 per 2 {\AA} averaged over the entire wavelength region. We developed a new code to extract the SN spectral component from spectra contaminated by the host galaxy. Of 71 SN Ia candidates, 59 are identified as normal SNe Ia and 3 are peculiar SNe Ia. The range of spectral phases varies from -7 days to +30 days from maximum brightness. There are also 7 SNe II, 1 possible hypernova and 1 AGN.

79 ASTRONOMY AND ASTROPHYSICS↗

Combined Algal Processing for the Synthesis of Liquid Oleofuels and Products (CAPSLOC) - WBS 1.3.4.204

Combined Algal Processing for the Synthesis of Liquid Oleofuels and Products (CAPSLOC) aims to develop a biorefinery concept that is composition-agnostic and enables economically, environmentally, and socially viable biofuel production by maximizing the value from each fraction. To achieve this, we have employed an iterative approach between R&D and TEA/LCA to establish process targets and quantify improvements in the minimum fuel selling price (MFSP), reduce carbon intensity, and support the state of technology (SOT). We have surveyed various methods for biomass pretreatment and successfully demonstrated pre-pilot scale pretreatment in batch mode. Using techniques such as lipid modification, fermentation, mild oxidative treatment (MOT), and carbonization, we have developed biofuel precursors and a range of value-added co-products. Our project has demonstrated significant improvements in key aspects of algae processing, particularly in pretreatment technologies suitable for variable composition, co-products available from lipid and extracted solid streams, conditioning and fermenting high-protein hydrolysates, and recovering nitrogen (N) and phosphorus (P) nutrients for recycling to cultivation ponds. The success of this project will support the commercialization of a sustainable microalgal biorefinery industry.

biofuel↗

MAJOR RIVER PLUMES IN THE TROPICAL OCEAN: PHYSICAL AND BIOGEOCHEMICAL EXPRESSION

The Caribbean and Western Tropical Atlantic receive massive inputs of Orinoco and Amazon River water carrying a load of organic and inorganic materials into waters characteristically devoid of these. The magnitude of riverine impact became evident as remote sensing became an ocean color monitoring tool. These observations depict riverine plumes, containing dissolved organic matter and phytoplankton well above background concentrations, being advected into the oligotrophic ocean. Subsequent research revealed that riverine intrusions radically modulate trophic balance and activity through changes in plankton abundance, composition and size distribution and last but not least: availability of solar irradiance. Moreover, riverine influence responds to climate processes and oceanic mesoscale processes bringing about significant spatial and temporal variability at annual and interannual scales. We discuss observations of physical and biogeochemical gradients in the Orinoco River Plume in the above context.

54 ENVIRONMENTAL SCIENCES↗

Investigating the Diurnal Radiative, Turbulent, and Biophysical Processes in the Amazonian Canopy‐Atmosphere Interface by Combining LES Simulations and Observations

Abstract We investigate the atmospheric diurnal variability inside and above the Amazonian rainforest for a representative day during the dry season. To this end, we combine high‐resolution large‐eddy simulations that are constrained and evaluated against a comprehensive observation set, including CO 2 concentrations, gathered during GoAmazon2014/15. We design systematic numerical experiments to quantify whether a multilayer approach in solving the explicit canopy improves our canopy‐atmosphere representation. We particularly focus on the relationship between photosynthesis and plant transpiration, and their distribution at leaf and canopy scales. We found the variability of photosynthesis drivers like vapor pressure deficit and leaf temperature to be about 3 times larger for sunlit leaves compared to shaded leaves. This leads to a large spread on leaf stomatal conductance values with minimum and maximum values varying more than 100%. Regarding the turbulent structure, we find wind‐driven stripe‐like shapes at the canopy top and structures resembling convective cells at the canopy. Wind‐related variables provide the best spatiotemporal agreement between model and observations. The potential temperature and heat flux profiles agree with an observed decoupling near the canopy top interface, although with less variability and cold biases of up to 3 K. The increasing complexity on the biophysical processes leads to the largest disagreements for evaporation, CO 2 plant assimilation and soil efflux. The model is able to capture the correct dependences and trends with the magnitudes still differing. We finally discuss the need to revise leaf and soil models and to complete the observations at leaf and canopy levels.

54 ENVIRONMENTAL SCIENCES↗

Constraints on the anisotropy of dark energy

If the equation of state of dark energy is anisotropic there will be additional quadrupole anisotropy in the cosmic microwave background induced by the time-dependent anisotropic stress quantified in terms of {Delta}w. Assuming that the entire amplitude of the observed quadrupole is due to this anisotropy, we conservatively impose a limit of |{Delta}w|<2.1x10{sup -4} for any value of w{>=}-1 assuming that {Omega}{sub m}<0.5. This is considerably tighter than that which comes from supernovae. Stronger limits, up to a factor of 10, are possible for specific values of {Omega}{sub m} and w. Since we assume this component is uncorrelated with the stochastic component from inflation, we find that both the expectation value and the sample variance are increased. There is no improvement in the likelihood of an anomalously low quadrupole as suggested by previous work on an elliptical universe.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Hindsight logging for model training

In modern Machine Learning, model training is an iterative, experimental process that can consume enormous computation resources and developer time. To aid in that process, experienced model developers log and visualize program variables during training runs. Exhaustive logging of all variables is infeasible, so developers are left to choose between slowing down training via extensive conservative logging, or letting training run fast via minimalist optimistic logging that may omit key information. As a compromise, optimistic logging can be accompanied by program checkpoints; this allows developers to add log statements post-hoc, and "replay" desired log statements from checkpoint---a process we refer to as hindsight logging. Unfortunately, hindsight logging raises tricky problems in data management and software engineering. Done poorly, hindsight logging can waste resources and generate technical debt embodied in multiple variants of training code. In this paper, we present methodologies for efficient and effective logging practices for model training, with a focus on techniques for hindsight logging. Our goal is for experienced model developers to learn and adopt these practices. To make this easier, we provide an open-source suite of tools for Fast Low-Overhead Recovery (flor) that embodies our design across three tasks: (i) efficient background logging in Python, (ii) adaptive periodic checkpointing, and (iii) an instrumentation library that codifies hindsight logging for efficient and automatic record-replay of model-training. Model developers can use each flor tool separately as they see fit, or they can use flor in hands-free mode, entrusting it to instrument their code end-to-end for efficient record-replay. Our solutions leverage techniques from physiological transaction logs and recovery in database systems. Evaluations on modern ML benchmarks demonstrate that flor can produce fast checkpointing with small user-specifiable overheads (e.g. 7%), and still provide hindsight log replay times orders of magnitude faster than restarting training from scratch.

Computer Science↗