Consensus Multi-Image Super Resolution
Explore the source record for details and available documents.
Sun, Yufang↗
Engineering topics
Bouman, Charles A.
Publications and source records attributed to Bouman, Charles A..
Super Resolving Unrolled Neural Networks for Remote Sensing
In remote sensing systems, the capabilities of the system are constrained by the complex interactions between size, weight, and power (SWAP) of potential designs. In electro-optical (EO) systems, examples of these critical parameters include the system’s sensitivity and resolution. Those parameters can be increased by ever larger optical apertures and focal planes but at the cost of more SWAP. Multi-image super resolution (MISR) techniques allow resolution to be enhanced via computation rather than more sophisticated optical hardware. These algorithms combine multiple images together into a single, higher resolution image, trading temporal resolution and computation for spatial resolution. Fielded MISR techniques, such as Drizzle, can require several hundred images to create a single super resolved image, implying reduced temporal resolution, increased data acquisition load, and limiting mission applications. Iterative techniques, such as model-based image reconstruction and compressive sensing, have been shown to create super resolved images using fewer images than Drizzle. They do this by posing an optimization problem that balances accuracy between a highly accurate physical model and an image model. In the case of super resolution, the physical model is defined by the relation between low resolution input images and the desired high resolution output image. The image model encodes some assumptions about the super resolved image. These assumptions are meant to suppress reconstruction artifacts that arise due to deterministic physical model error, stochastic measurement noise, and potential undersampling. In practice, the performance of iterative methods are limited by imaging models compatible with optimization. Deep learning-based methods can effectively learn image models of arbitrary complexity, but lack the theoretical explainability and robustness of iterative techniques. Consensus equilibrium (CE) generalizes the iterative techniques beyond optimization, enabling blackbox algorithms such as traditional and neural image denoisers to be used as the image model. CE-based approaches retain much of the explainability and robustness of iterative techniques while allowing the expressiveness of machine learning image models to be used. Additionally, by unrolling iterations of CE with an embedded image denoiser, the image denoiser can be further trained and specialized to the specific application with potentially higher quality reconstructions. Under this project, we demonstrated the feasibility of training an unrolled neural network based upon CE. While we didn’t train one, we showed that the CE process is differentiable and its gradient can be tractably computed. We also explored the usage of a variants of CE akin to generative neural works. Most importantly, we applied the CE framework to a number of problems including non-blind deconvolution, upsampling, single-image super resolution, MISR, event-based sensing, and saturated deconvolution. Our MISR prototype creates high quality reconstructions with an order of magnitude fewer images than previous approaches and, critically, produces these reconstructions fast enough for practical usage.
47 OTHER INSTRUMENTATION↗
A machine learning decision criterion for reducing scan time for hyperspectral neutron computed tomography systems
We present the first machine learning-based autonomous hyperspectral neutron computed tomography experiment performed at the Spallation Neutron Source. Hyperspectral neutron computed tomography allows the characterization of samples by enabling the reconstruction of crystallographic information and elemental/isotopic composition of objects relevant to materials science. High quality reconstructions using traditional algorithms such as the filtered back projection require a high signal-to-noise ratio across a wide wavelength range combined with a large number of projections. This results in scan times of several days to acquire hundreds of hyperspectral projections, during which end users have minimal feedback. To address these challenges, a golden ratio scanning protocol combined with model-based image reconstruction algorithms have been proposed. This novel approach enables high quality real-time reconstructions from streaming experimental data, thus providing feedback to users, while requiring fewer yet a fixed number of projections compared to the filtered back projection method. In this paper, we propose a novel machine learning criterion that can terminate a streaming neutron tomography scan once sufficient information is obtained based on the current set of measurements. Our decision criterion uses a quality score which combines a reference-free image quality metric computed using a pre-trained deep neural network with a metric that measures differences between consecutive reconstructions. The results show that our method can reduce the measurement time by approximately a factor of five compared to a baseline method based on filtered back projection for the samples we studied while automatically terminating the scans.
97 MATHEMATICS AND COMPUTING↗
TRINIDI: Time-of-Flight Resonance Imaging With Neutrons for Isotopic Density Inference
Accurate reconstruction of 2D and 3D isotope densities is a desired capability with great potential impact in applications such as evaluation and development of next-generation nuclear fuels. Neutron time-of-flight (TOF) resonance imaging offers a potential approach by exploiting the characteristic neutron absorption spectra of each isotope. However, it is a major challenge to compute quantitatively accurate images due to a variety of confounding effects such as severe Poisson noise, background scatter, beam non-uniformity, absorption non-linearity, and extended source pulse duration. We present the TRINIDI algorithm which is based on a two-step process in which we first estimate the neutron flux and background counts, and then reconstruct the areal densities of each isotope and pixel. Both components are based on the inversion of a forward model that accounts for the highly non-linear absorption, energy-dependent emission profile, and Poisson noise, while also modeling the substantial spatio-temporal variation of the background and flux. Further, to do this, we formulate the non-linear inverse problem as two optimization problems that are solved in sequence. We demonstrate on both synthetic and measured data that TRINIDI can reconstruct quantitatively accurate 2D views of isotopic areal density that can then be reconstructed into quantitatively accurate 3D volumes of isotopic volumetric density.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
Projected Multi-Agent Consensus Equilibrium (PMACE) With Application to Ptychography
Multi-Agent Consensus Equilibrium (MACE) formulates an inverse imaging problem as a balance among multiple update agents such as data-fitting terms and denoisers. However, each such agent operates on a separate copy of the full image, leading to redundant memory use and slow convergence when each agent affects only a small subset of the full image. In this article, we extend MACE to Projected Multi-Agent Consensus Equilibrium (PMACE), in which each agent updates only a projected component of the full image, thus greatly reducing memory use for some applications. We describe PMACE in terms of an equilibrium problem and an equivalent fixed-point problem and show that in most cases the PMACE equilibrium is not the solution of an optimization problem. To demonstrate the value of PMACE, we apply it to the problem of ptychography, in which a sample is reconstructed from the diffraction patterns resulting from coherent X-ray illumination at multiple overlapping spots. In our PMACE formulation, each spot corresponds to a separate data-fitting agent, with the final solution found as an equilibrium among all the agents. In conclusion, our results demonstrate that the PMACE reconstruction algorithm generates more accurate reconstructions at a lower computational cost than existing ptychography algorithms when the spots are sparsely sampled.
97 MATHEMATICS AND COMPUTING↗
Plug-and-Play Methods for Integrating Physical and Learned Models in Computational Imaging: Theory, algorithms, and applications
Plug-and-play (PnP) priors constitute one of the most widely used frameworks for solving computational imaging problems through the integration of physical models and learned models. PnP leverages high-fidelity physical sensor models and powerful machine learning methods for prior modeling of data to provide state-of-the-art reconstruction algorithms. PnP algorithms alternate between minimizing a data fidelity term to promote data consistency and imposing a learned regularizer in the form of an image denoiser. Recent highly successful applications of PnP algorithms include biomicroscopy, computerized tomography (CT), magnetic resonance imaging (MRI), and joint ptychotomography. This article presents a unified and principled review of PnP by tracing its roots, describing its major variations, summarizing main results, and discussing applications in computational imaging. Additionally, we also point the way toward further developments by discussing recent results on equilibrium equations that formulate the problem associated with PnP algorithms.
97 MATHEMATICS AND COMPUTING↗
CodEx: A Modular Framework for Joint Temporal De-Blurring and Tomographic Reconstruction
In many computed tomography (CT) imaging applications, it is important to rapidly collect data from an object that is moving or changing with time. Tomographic acquisition is generally assumed to be step-and-shoot, where the object is rotated to each desired angle, and a view is taken. However, step-and-shoot acquisition is slow and can waste photons, so in practice fly-scanning is done where the object is continuously rotated while collecting data. However, this can result in motion-blurred views and consequently reconstructions with severe motion artifacts. In this paper, we introduce CodEx, a modular framework for joint de-blurring and tomographic reconstruction that can effectively invert the motion blur introduced in sparse view fly-scanning. The method is a synergistic combination of a novel acquisition method with a novel non-convex Bayesian reconstruction algorithm. CodEx works by encoding the acquisition with a known binary code that the reconstruction algorithm then inverts. Using a well chosen binary code to encode the measurements can improve the accuracy of the inversion process. The CodEx reconstruction method uses the alternating direction method of multipliers (ADMM) to split the inverse problem into iterative deblurring and reconstruction sub-problems, making reconstruction practical to implement. Here we present reconstruction results on both simulated and binned experimental data to demonstrate the effectiveness of our method.
42 ENGINEERING↗
High-precision inversion of dynamic radiography using hydrodynamic features
While radiography is routinely used to probe complex, evolving density fields in research areas ranging from materials science to shock physics to inertial confinement fusion and other national security applications, complications resulting from noise, scatter, complex beam dynamics, etc. prevent current methods of reconstructing density from being accurate enough to identify the underlying physics with sufficient confidence. In this work, we show that using only features that are robustly identifiable in radiographs and combining them with the underlying hydrodynamic equations of motion using a machine learning approach of a conditional generative adversarial network (cGAN) provides a new and effective approach to determine density fields from a dynamic sequence of radiographs. In particular, we demonstrate the ability of this method to outperform a traditional, direct radiograph to density reconstruction in the presence of scatter, even when relatively small amounts of scatter are present. Our experiments on synthetic data show that the approach can produce high quality, robust reconstructions. We also show that the distance (in feature space) between a testing radiograph and the training set can serve as a diagnostic of the accuracy of the reconstruction.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Algorithm-Driven Advances for Scientific CT Instruments: From model-based to deep learning-based approaches
Multiscale 3D characterization is widely used by materials scientists to further their understanding of the relationships between microscopic structure and macroscopic function. Scientific computed tomography (SCT) instruments are one of the most popular choices for 3D nondestructive characterization of materials at length scales ranging from the angstrom scale to the micron scale. These instruments typically have a source of radiation (such as electrons, X-rays, or neutrons) that interacts with the sample to be studied and a detector assembly to capture the result of this interaction (see Figure 1 ). A collection of such high-resolution measurements is made by reorienting the sample, which is mounted on a specially designed stage/holder after which reconstruction algorithms are used to produce the final 3D volume of interest. The specific choice of which instrument to use depends on the desired resolution and properties of the materials being imaged. Additionally, the end goal of SCT scans includes determining the morphology, chemical composition, or dynamic behavior of materials when subjected to external stimuli. In summary, SCT instruments are powerful tools that enable 3D characterization across multiple length scales and play a critical role in furthering the understanding of the structure–function relationships of different materials.
42 ENGINEERING↗
Physics‐based iterative reconstruction for dual‐source and flying focal spot computed tomography
Purpose For single‐source helical Computed Tomography (CT), both Filtered‐Back Projection (FBP) and statistical iterative reconstruction have been investigated. However, for dual‐source CT with flying focal spot (DS‐FFS CT), a statistical iterative reconstruction that accurately models the scanner geometry and acquisition physics remains unknown to researchers. Therefore, our purpose is to present a novel physics‐based iterative reconstruction method for DS‐FFS CT and assess its image quality. Methods Our algorithm uses precise physics models to reconstruct from the native cone‐beam geometry and interleaved dual‐source helical trajectory of a DS‐FFS CT. To do so, we construct a noise physics model to represent data acquisition noise and a prior image model to represent image noise and texture. In addition, we design forward system models to compute the locations of deflected focal spots, the dimension, and sensitivity of voxels and detector units, as well as the length of intersection between x‐rays and voxels. The forward system models further represent the coordinated movement between the dual sources by computing their x‐ray coverage gaps and overlaps at an arbitrary helical pitch. With the above models, we reconstruct images by an advanced Consensus Equilibrium (CE) numerical method to compute the maximum a posteriori estimate to a joint optimization problem that simultaneously fits all models. Results We compared our reconstruction with Siemens ADMIRE, which is the clinical standard hybrid iterative reconstruction (IR) method for DS‐FFS CT, in terms of spatial resolution, noise profile, and image artifacts through both phantoms and clinical scan datasets. Experiments show that our reconstruction has a higher spatial resolution, with a Task‐Based Modulation Transfer Function (MTF task ) consistently higher than the clinical standard hybrid IR. In addition, our reconstruction shows a reduced magnitude of image undersampling artifacts than the clinical standard. Conclusions By modeling a precise geometry and avoiding data rebinning or interpolation, our physics‐based reconstruction achieves a higher spatial resolution and fewer image artifacts with smaller magnitude than the clinical standard hybrid IR.
Wang, Xiao↗
Plug and play: a general approach for the fusion of sensor and machine learning models
Regularized or Bayesian inversion has revolutionized our ability to reconstruct images from incomplete data.
97 MATHEMATICS AND COMPUTING↗
First Examination of Irradiated Fuel with Pulsed Neutrons at LANSCE (Preliminary Results)
We present preliminary results on the characterization of an irradiated U-lOZr-lPd fuel sample that was prepared from the irradiated AFC-3A-R5A sample. U-lOZr metallic fuels are researched as host materials for potential transmutation fuels and the addition of palladium strives to bind lanthanides, thus preventing fuel-cladding chemical interactions (FCCI). These interactions limit the lifetime of metallic fuels and are caused by migration of lanthanide fission products to the periphery of the fuel slug, where they start to interact with the D9 or HT9 steel cladding, ultimately leading to failure of the mechanical integrity of the cladding. Neutrons offer bulk characterization of irradiated materials for which X-ray tomography methods are not suitable due to the immense gamma background emitted from the samples. In particular pulsed neutrons provide information from the ability to resolve the neutron energy using their time-of-flight and thus the potential to utilize neutron absorption resonance to characterize the spatial distribution of isotopes. This, in turn, may allow to characterize the distribution of fission and neutron capture products non-destructively and may ultimately be applied to the bulk of an irradiation capsule prior to destructive post-irradiation examination to identify regions of interest. To allow the characterization of entire irradiation capsules, a cask is under development in the advanced post-irradiation work package at LANL and progress on this development was reported elsewhere. In parallel, an irradiated U-lOZr-lPd sample cut from the AFC-3AR5A irradiation was shipped to LANL and will be fully characterized with an NSUF funded rapid turnaround experiment (RTE) in the 2020 LANSCE run cycle. The sample emits at a dose rate of ~3R/hr on contact and is therefore manageable with remote handling, without requiring a cask. The disk-shaped material is larger than samples prepared for analysis using electron or X-ray methods and is therefore an intermediate step towards characterization of bulk samples at LANSCE. However, since it covers the full diameter of the irradiated fuel slug, some insight on redistribution of elements, spatially resolved information on microstructure, e.g. phase composition and texture, will be possible using the pulsed neutron-based methods developed for fuel characterization at LANSCE. This report describes the development of procedures to handle the sample at LANSCE as well as preliminary data and results from tests conducted in December 2019 on the energy-resolved neutron imaging (ERNI) beam line at flight path 5 and the high pressure-preferred orientation diffractometer (HIPPO) at LANSCE. This effort is a collaboration between LANL, INL, and ORNL. To compare our capabilities with prior work, we present an overview of previously reported bulk characterization of irradiated or spent fuels. The overview addresses neutron diffraction and neutron absorption resonance spectroscopy, both of which have only few reported applications on irradiated or spent nuclear fuel, as well as neutron radiography. This literature review was already described in a previous report but is repeated here to put our current efforts in context of previous work.
11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
MBIR: A cryo-ET 3D reconstruction method that effectively minimizes missing wedge artifacts and restores missing information
Cryo-Electron Tomography (cryo-ET) has become an essential technique in revealing cellular and macromolecular assembly structures in their native states. However, due to radiation damage and the limited tilt range, cryo-ET suffers from low contrast and missing wedge artifacts, which limits the tomograms to low resolution and hinders further biological interpretation. We applied the Model-Based Iterative Reconstruction (MBIR) method to obtain tomographic 3D reconstructions of experimental cryo-ET datasets and demonstrated the advantages of MBIR in contrast improvement, missing wedge artifacts reduction, missing information restoration, and subtomogram averaging compared with other reconstruction approaches. Considering the outstanding reconstruction quality, MBIR has a great potential in the determination of high resolution biological structures with cryo-ET.
59 BASIC BIOLOGICAL SCIENCES↗
Model-Based Iterative Reconstruction of Magnetization Using Vector Field Electron Tomography
Vector field electron tomography (VFET) is extensively used for three-dimensional (3-D) imaging of magnetic materials at nanometer resolutions. The conventional approach is to reconstruct and visualize the magnetic vector potential or the magnetic field associated with the sample. There is a lack of algorithms capable of reconstructing the 3-D distribution of magnetization from VFET data. Unlike magnetic vector potential and magnetic field, magnetization is a fundamental physical property of the sample that does not extend beyond the dimensions of the sample. We present a model-based iterative reconstruction algorithm (MBIR) that reconstructs the magnetization by minimizing a cost function consisting of a forward model term and a prior model term. The forward model uses the physics of imaging to model the VFET data as a function of the magnetization, and the prior model enforces sparsity in the magnetization reconstruction. We then formulate an optimization algorithm based on the theory of alternate direction method of multipliers to minimize the resulting MBIR cost function. In conclusion, using simulated and real data, we show that our algorithm accurately reconstructs both the magnetization and the magnetic vector potential.
97 MATHEMATICS AND COMPUTING↗
Reduced electron exposure for energy-dispersive spectroscopy using dynamic sampling
Analytical electron microscopy and spectroscopy of biological specimens, polymers, and other beam sensitive materials has been a challenging area due to irradiation damage. There is a pressing need to develop novel imaging and spectroscopic imaging methods that will minimize such sample damage as well as reduce the data acquisition time. The latter is useful for high-throughput analysis of materials structure and chemistry. Here, in this work, we present a novel machine learning based method for dynamic sparse sampling of EDS data using a scanning electron microscope. Our method, based on the supervised learning approach for dynamic sampling algorithm and neural networks based classification of EDS data, allows a dramatic reduction in the total sampling of up to 90%, while maintaining the fidelity of the reconstructed elemental maps and spectroscopic data. In conclusion, we believe this approach will enable imaging and elemental mapping of materials that would otherwise be inaccessible to these analysis techniques.
36 MATERIALS SCIENCE↗
Second Harmonic Generation Guided Raman Spectroscopy for Sensitive Detection of Polymorph Transitions
Second harmonic generation (SHG) was integrated with Raman spectroscopy for the analysis of pharmaceutical materials. Particulate formulations of clopidogrel bisulfate were prepared in two crystal forms (Form I and Form II). Image analysis approaches enable automated identification of particles by bright field imaging, followed by classification by SHG. Quantitative SHG microscopy enabled discrimination of crystal form on a per particle basis with 99.95% confidence in a total measurement time of ~10 ms per particle. Complementary measurements by Raman and synchrotron XRD are in excellent agreement with the classifications made by SHG, with measurement times of ~1 min and several seconds per particle, respectively. In conclusion, coupling these capabilities with at-line monitoring may enable real-time feedback for reaction monitoring during pharmaceutical production to favor the more bioavailable but metastable Form I with limits of detection in the ppm regime.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
Dynamic X-ray diffraction sampling for protein crystal positioning
A sparse supervised learning approach for dynamic sampling (SLADS) is described for dose reduction in diffraction-based protein crystal positioning. Crystal centering is typically a prerequisite for macromolecular diffraction at synchrotron facilities, with X-ray diffraction mapping growing in popularity as a mechanism for localization. In X-ray raster scanning, diffraction is used to identify the crystal positions based on the detection of Bragg-like peaks in the scattering patterns; however, this additional X-ray exposure may result in detectable damage to the crystal prior to data collection. Dynamic sampling, in which preceding measurements inform the next most information-rich location to probe for image reconstruction, significantly reduced the X-ray dose experienced by protein crystals during positioning by diffraction raster scanning. The SLADS algorithm implemented herein is designed for single-pixel measurements and can select a new location to measure. In each step of SLADS, the algorithm selects the pixel, which, when measured, maximizes the expected reduction in distortion given previous measurements. Ground-truth diffraction data were obtained for a 5 µm-diameter beam and SLADS reconstructed the image sampling 31% of the total volume and only 9% of the interior of the crystal greatly reducing the X-ray dosage on the crystal. Furthermore, by using in situ two-photon-excited fluorescence microscopy measurements as a surrogate for diffraction imaging with a 1 µm-diameter beam, the SLADS algorithm enabled image reconstruction from a 7% sampling of the total volume and 12% sampling of the interior of the crystal. When implemented into the beamline at Argonne National Laboratory, without ground-truth images, an acceptable reconstruction was obtained with 3% of the image sampled and approximately 5% of the crystal. The incorporation of SLADS into X-ray diffraction acquisitions has the potential to significantly minimize the impact of X-ray exposure on the crystal by limiting the dose and area exposed for image reconstruction and crystal positioning using data collection hardware present in most macromolecular crystallography end-stations.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗