Single-shot image-based autofocus with machine learning
This shows how to reproduce the data collection aspects of the paper:
In this paper, we show how the existing transmitted light path and camera on a microscope can be used collect a single image, which can then be fed into a neural network to predict how far out of focus that image is so the Z-stage can be moved back into the correct focal plane.
In particular, this tutorial will show: 1) How to create the neccessary hardware for the system. 2) How to use Micro-Magellan to easily collect training data 3) How to use Pycro-Manager to apply a trained network to apply focus corrections during subsequent experiments. An different notebook shows the steps in between (2) and (3) of processing data and training the network.
This tutorial assumes the use of a Micro-Manager configuration with motorized XY and Z stages.
Background
Many biological experiments involve imaging samples in a microscope over long time periods or large spatial scales, making it difficult to keep the sample in focus. When observing a sample over time periods of hours or days, for example, thermal fluctuations can induce focus drift. Or, when scanning and stitching together many fields-of-view (FoV) to form a high-content, high-resolution image, a sample that is not sufficiently flat necessitates refocusing at each position. Since it is often experimentally impractical or cumbersome to manually maintain focus, an automatic focusing mechanism is essential.
Many hardware and software-based solutions have been proposed for this problem. Hardware solutions are, in general, more expensive and faster, while software methods tend to be slower. The technique we propose here aims to capture the benefits of each: A fast method for autofocusing that requires minimal, cheap hardware. The most basic software autofocus method is to simply take a focal stack of images, calculate some metric of image sharpness over the stack, and find the peak.
In our technique, we will bootstrap training data best on this basic method for generating ground truth. That is, we will collect paired Z-stacks of a sample, one with incoherent illumination (i.e. regular transmitted light with the aperture not fully closed down), and the other with coherent illumination, as provided by an off-axis LED illuminating the sample. A single plane of the latter will be used as the input to a neural network that can predict the correct focal position.
Overview
This tutorial will go through the full process in the figure below and is broken down in the following sections
Hardware setup
How to choose off-axis illumination
How to control through Micro-Manager
Collect training data (part a in figure below)
Use trained network to apply focus corrections during experiments (part b in figure below)
1) Hardware setup
This technique requires some type of off-axis, coherent illumination. We can easily achieve this by placing one or a few LED(s) off axis to illuminate the sample.
How to choose off-axis illumination
As we show in the paper, different LED positions and different patterns of LEDs can be more or less effective for this task (see figure below). The best performing networks used ~10 LEDs, all in a straight line on one side of the optical axis.
How to control through Micro-Manager
The easiest way to set up this type of illumination in Micro-Manager is with the Arduino device adapter. Essentially, you can wire up a simple circuit, load the provided firmwire onto the arduino, and then use the device API in Micro-Manager to control this circuit.
2) Collect training data
Now we’re going to collect 2-channel Z-stacks, where one channel is the coherent illumination we just set up, and the other channel is uses incoherent illumination (which gets more blurry when out of focus, and making it easier to calculate the ground truth focal position)
Calibrating XY stage for Micro-Magellan
Micro-Magellan provides a convenient graphical user interface for collecting a many of such Z stacks. In order to use its features, we need to calibrate the “affine transform” matrix, which gives the relation between the coordinate space of pixels on the camera and coordinates of the XY stage.
First go here: