Learned adaptive illumination multiphoton microscopy
In this notebook, we demonstrate how to implement Learned Adaptive Multiphoton Illumination microscopy (LAMI), using Pycro-Manager and Micro-Magellan. Throughout the tutorial, we’ll use Napari for visualizing data. This technique enables automatic sample-dependent adjustment of excitation laser power in real time while imaging a sample on a 2-photon microscope in order to compensate for attenuation of fluorescence when imaging deep into intact tissue.
This notebook outlines all the steps for implementing this technique, including microscope control software, how to run experiments, how to analyze the data and train the machine learning model, and how to apply its predictions in real time when collecting new data. Each of these steps is implemented with entirely open-source software. The tutorial is entirely in Python, with the exception of the first part that deals with the hardware setup and may require modifying C/C++ code.
The tutorial aims to be as modular as possible, so that ideas/techniques can be drawn from it and applied to other applications.
Although running through the full tutorial in sequence is how one would set up this experiment in practice, we’ve provided demo data at various steps along the way both as a way to check that things work as expected, and to allow the sections to be run independently. This demo data is available here, along with the lami_helper.py file which contains some code that will be used in various places in the notebook
Background
When imaging deep into a sample using 2-photon microscopy, excitation light focusing to different points in the sample will be subjected to different amounts of scattering, and the excitation laser power must be increased in order to maintain signal. Failing to increase sufficiently will lead to the loss of detectable fluorescence. Increasing too much subjects the sample to unnecessary photobleaching and photodamage, with the potential to disrupt or alter the biological processes under investigation.
This problem can be solved by using cells with identical fluorescent labels (i.e. standard candles) imaged in situ to train a physics-based machine learning model. After this training has been performed, the correct illumination power can be predicted and adaptively adjusted at each point in a 3D volume on subsequent samples as a function of the sample’s shape, without the need for specialized fluorescent labelling.
Requirements
This tutorial was created using the following python packages and versions (though other versions likely work as well)
as well as:
Micro-Manager 2.0gamma 20210330
Overview
This tutorial will go through the full process in the figure below and is broken down in the following sections
Setup
Required hardware: Time-realized spatial light modulator
Calibrating XY stage
How to use Micro-Magellan + surfaces to collect data
Collect and process training data
Control microscope to image standard candle cells with randomized power
Run 3D segmentation on this data to extract needed information
Process segmented cells to get features needed for neural network training
Train neural network to predict excitation laser control
Train network
Visualize predictions of trained network
Use the trained model to modulate excitation power on subsequent experiments
An overview of everything that will be done in this notebook is in the figure below 
1) Setup
Hardware: Time-realized spatial light modulator
Utilizing Learned adaptive multiphoton illumination requires the ability to modulate excitation laser power over a single field of view. A two photon microscope with an existing EOM or AOM for modulating laser power can be easily modified into a “time-realized spatial light modulator” (TR-SLM) (see paper for full description).
The TR-SLM can be controlled through the SLM device API in Micro-Manager. The TR-SLM we describe has two important software components. One is the Arduino firmware (located here), which runs on the Teensy3.2 microcontroller that is the heart of the TR-SLM. The second is a Micro-Manager device adapter (found here), which provides the glue to convert the Micro-Manager SLM API into the serial commands expected by the firmware on the Teensy. It is likely that the firmware running on the Teensy will need to be modified, since it contains parameters specific to the scanning pattern of our microscope, which has a resonant scanning mirror with a fixed period scan time on one axis and a galvo scanner on the other axis. However, the device adapter code might not need to be modified. A pre-built version of it comes with
An example Micro-Manager configuration file using this device adapter can be found here. There are many things in the configuration file that are specific to the microscope on which we used that won’t neccessarily need to be duplicated, but the parts referencing “TeensySLM” should be fairly generic. Note that this configuration corresponds to a system with two TeensySLMs (one for each EOM/Laser).
Calibrate XY stage
We need to calibrate the “affine transform” matrix, which gives the relation between the coordinate space of pixels on the camera and coordinates of the XY stage.
First go here: 
Then select “edit”. Then use “calculate” or “measure” to either manually or automatically figure this out (You need a high contrast sample that is firmly secured in place to get an accurate calibration)
Creating surfaces and collecting data with Micro-Magellan
To acquire data, we will be using a combination of Micro-Magellan and Pycro-Manager. Micro-Magellan (available in Micro-Manager plugins menu) provides the graphical user interface (GUI) for mapping samples and setting up experiments, and Pycro-Manager enables injection of customized code to run during those experiments for various purposes.
The first step when acquiring data will often be to map the top the sample with a “surface” in Micro-Magellan. To do so, we are going to mark a bunch points on the surface, which the software will use to interpolate a surface (see figure below). A walkthrough of how to do this can be found in this video.
2) Collect and process training data
In this next section, we’ll demonstrate how to set up an experiment that images each Z-stack with a random amount of power