Engineering topics
Doty, Christina
Publications and source records attributed to Doty, Christina.
SAM-I-Am: Semantic boosting for zero-shot atomic-scale electron micrograph segmentation
Image segmentation is a critical enabler for tasks ranging from medical diagnostics to autonomous driving. However, the correct segmentation semantics — where are boundaries located? what segments are logically similar? — change depending on the domain, such that state-of-the-art foundation models can generate meaningless and incorrect results. Moreover, in certain domains, fine-tuning and retraining techniques are infeasible: obtaining labels is costly and time-consuming; domain images (micrographs) can be exponentially diverse; and data sharing (for third-party retraining) is restricted. To enable rapid adaptation of the best segmentation technology, we propose the concept of semantic boosting: given a zero-shot foundation model, guide its segmentation and adjust results to match domain expectations. Here, we apply semantic boosting to the Segment Anything Model (SAM) to obtain microstructure segmentation for transmission electron microscopy. Our booster, SAM-I-Am, serves as a post-processing engine that extracts geometric and textural features of various intermediate masks to perform mask removal and mask merging operations. We demonstrate a zero-shot performance increase of (absolute) +21.35%, +12.6%, +5.27% in mean IoU, and a -9.91%, -18.42%, -4.06% drop in mean false positive masks across images of three difficulty classes over vanilla SAM (ViT-L).
Machine learning for automated experimentation in scanning transmission electron microscopy
Abstract Machine learning (ML) has become critical for post-acquisition data analysis in (scanning) transmission electron microscopy, (S)TEM, imaging and spectroscopy. An emerging trend is the transition to real-time analysis and closed-loop microscope operation. The effective use of ML in electron microscopy now requires the development of strategies for microscopy-centric experiment workflow design and optimization. Here, we discuss the associated challenges with the transition to active ML, including sequential data analysis and out-of-distribution drift effects, the requirements for edge operation, local and cloud data storage, and theory in the loop operations. Specifically, we discuss the relative contributions of human scientists and ML agents in the ideation, orchestration, and execution of experimental workflows, as well as the need to develop universal hyper languages that can apply across multiple platforms. These considerations will collectively inform the operationalization of ML in next-generation experimentation.
Evaluating Stage Motion for Automated Electron Microscopy
Abstract Precise control is an essential and elusive quality of emerging self-driving transmission electron microscopes (TEMs). It is widely understood these instruments must be capable of performing rapid, high-volume, and arbitrary movements for practical self-driving operation. However, stage movements are difficult to automate at scale, owing to mechanical instability, hysteresis, and thermal drift. Such difficulties pose major barriers to artificial intelligence-directed microscope designs that require repeatable, precise movements. To guide design of emerging instruments, it is necessary to understand the behavior of existing mechanisms to identify rate limiting steps for full autonomy. Here, we describe a general framework to evaluate stage motion in any TEM. We define metrics to evaluate stage degrees of freedom, propose solutions to improve performance, and comment on fundamental limits to automated experimentation using present hardware.
Forecasting of in situ electron energy loss spectroscopy
Abstract Forecasting models are a central part of many control systems, where high-consequence decisions must be made on long latency control variables. These models are particularly relevant for emerging artificial intelligence (AI)-guided instrumentation, in which prescriptive knowledge is needed to guide autonomous decision-making. Here we describe the implementation of a long short-term memory model (LSTM) for forecasting in situ electron energy loss spectroscopy (EELS) data, one of the richest analytical probes of materials and chemical systems. We describe key considerations for data collection, preprocessing, training, validation, and benchmarking, showing how this approach can yield powerful predictive insight into order-disorder phase transitions. Finally, we comment on how such a model may integrate with emerging AI-guided instrumentation for powerful high-speed experimentation.
pnnl/Multimodal-Few-Shot
This project combines few-shot machine learning with a multimodal framework for electron microscope data quantification.
pnnl/EELSTM
EELSTM: Time Series Prediction on Electron Energy Loss Spectra - long short-term memory model approach to forecasting of electron energy loss spectroscopy, based on a Bayesian deep learning model
Design of a graphical user interface for few-shot machine learning classification of electron microscopy data
The recent growth in data generation by modern electron microscopes requires rapid, scalable, and flexible approaches to image segmentation and analysis. Few-shot machine learning, which can richly classify images from a handful of user-provided examples, is a promising route to high-throughput analysis. However, current command-line implementations of such approaches can be slow and unintuitive to use, lacking the real-time feedback necessary to perform effective classification. Here we report on the development of a Python-based graphical user interface that enables end users to easily conduct and visualize the output of few-shot learning models. This interface is portable and can be hosted locally or on the web, providing the opportunity to reproducibly conduct, share, and crowd-source few-shot analyses.