Abstract: In this study, we explore the possibility to use deep learning for the reconstruction of phase images from 4D scanning transmission electron microscopy (4D-STEM) data. The process can be divided into two main steps. First, the complex electron wave function is recovered for a convergent beam electron diffraction pattern (CBED) using a convolutional neural network (CNN). Subsequently, a corresponding patch of the phase object is recovered using the phase object approximation. Repeating this for each scan position in a 4D-STEM dataset and combining the patches by complex summation yields the full-phase object. Each patch is recovered from a kernel of 3x3 adjacent CBEDs only, which eliminates common, large memory requirements and enables live processing during an experiment. The machine learning pipeline, data generation, and the reconstruction algorithm are presented. We demonstrate that the CNN can retrieve phase information beyond the aperture angle, enabling super-resolution imaging. The image contrast formation is evaluated showing a dependence on the thickness and atomic column type. Columns containing light and heavy elements can be imaged simultaneously and are distinguishable. The combination of super-resolution, good noise robustness, and intuitive image contrast characteristics makes the approach unique among live imaging methods in 4D-STEM.

Abstract: Nanoscience and nanotechnologies are of immense importance across many fields of science and for numerous practical applications. In this context, scanning transmission electron microscopy (STEM) and 4D-STEM are among the most powerful characterization methods at the atomic scale. Annular dark-field (ADF)-STEM can be used to quantify atomic structures in 3D by counting atoms based on a single projection image. In 4D-STEM a full diffraction pattern is recorded at each scan step, which enables more dose efficient imaging and the utilization of various advanced imaging modalities, which can however be complex and slow. Both, STEM and 4D-STEM suffer from noise and distortions. In the first section of this work the most important of these distortions are discussed and it is shown how image restoration with a dedicated convolutional neural network (CNN) can be beneficial for atomic structure quantifications in ADF-STEM. In the second part, a new 4D-STEM imaging method real-time-integrated-centre-of-mass (riCOM) is introduced, which is a very dose-efficient and fast algorithm that enables unprecedented live-imaging capabilities for 4D-STEM. It is based on the integrated centre-of-mass approach, but is reformulated with variable integration ranges and optional filters, which allows for a tunable contrast transfer function. This enables the imaging of light and heavy elements simultaneously at very low doses. In the third part another new 4D-STEM method, coined AIRPI (AI-assisted rapid phase imaging) is introduced, which uses a CNN to retrieve a patch of the specimen's phase image for each scan position, based on the diffraction patterns in the probe's immediate surroundings. This allows also live imaging in principle and surpasses comparable state-of-the-art algorithms in terms of resolution also at low doses. Different atomic columns can be reliably distinguished over a wide range of atomic numbers, enabling a very good image interpretability. Further, AIRPI can recover low frequency image components, which preserves thickness information. This is a unique and important feature which could make quantitative 4D-STEM feasible.

Abstract: This paper investigates the possible benefits for counting atoms of different chemical nature when analysing multiple 2D scanning transmission electron microscopy (STEM) images resulting from independent annular dark field (ADF) detector regimes. To reach this goal, the principles of statistical detection theory are used to quantify the probability of error when determining the number of atoms in atomic columns consisting of multiple types of elements. In order to apply this theory, atom-counting is formulated as a statistical hypothesis test, where each hypothesis corresponds to a specific number of atoms of each atom type in an atomic column. The probability of error, which is limited by the unavoidable presence of electron counting noise, can then be computed from scattering-cross sections extracted from multiple ADF STEM images. Minimisation of the probability of error as a function of the inner and outer angles of a specified number of independent ADF collection regimes results in optimal experimental designs. Based on simulations of spherical Au@Ag and Au@Pt core–shell nanoparticles, we investigate how the combination of two non-overlapping detector regimes helps to improve the probability of error when unscrambling two types of atoms. In particular, the combination of a narrow low angle ADF detector with a detector formed by the remaining annular collection regime is found to be optimal. The benefit is more significant if the atomic number Z difference becomes larger. In
addition, we show the benefit of subdividing the detector regime into three collection areas for heterogeneous nanostructures based on a structure consisting of three types of elements, e.g., a mixture of Au, Ag and Al atoms. Finally, these results are compared with the probability of error resulting when one would ultimately use a pixelated 4D STEM detector and how this could help to further reduce the incident electron dose.