Using an external master clock for hardware control of a stage-scanning high NA oblique plane microscope

Tutorial provided by qi2lab.

This tutorial uses Pycro-Manager to rapidly acquire terabyte-scale volumetric images using external hardware triggering of a stage scan optimized, high numerical aperture (NA) oblique plane microscope (OPM). The microscope that this notebook controls is described in detail in this preprint, under the stage scan OPM section in the methods.

This high NA OPM allows for versatile, high-resolution, and large field-of-view single molecule imaging. The main application is quantifying 3D spatial gene expression in millions of cells or large pieces of intact tissue using interative RNA-FISH (see examples here and here). Because the fluidics controller for the iterative labeling is also controlled via Python (code not provided here), using Pycro-Manager greatly simplifies controlling these complex experiments.

The tutorial highlights the use of the post_camera_hook_fn and post_hardware_hook_fn functionality to allow an external controller to synchronize the microscope acquisition (external master). This is different from the standard hardware sequencing functionality in Pycro-Manager, where the acquisition engine sets up sequencable hardware and the camera serves as the master clock.

The tutorial also discusses how to structure the events and avoid timeouts to acquire >10 million of events per acquistion.

Microscope hardware

Briefly, the stage scan high NA OPM is built around a bespoke tertiary objective designed by Alfred Millet-Sikking and Andrew York at Calico Labs. Stage scanning is performed by an ASI scan optimized XY stage, an ASI FTP Z stage, and an ASI Tiger controller with a programmable logic card. Excitation light is provided by a Coherent OBIS Laser Box. A custom Teensy based DAC synchronizes laser emission and a galvanometer mirror to the scan stage motion to eliminate motion blur. Emitted fluorescence is imaged by a Photometrics Prime BSI.

The ASI Tiger controller is the master clock in this experiment. The custom Teensy DAC is setup in a closed loop with the Photometrics camera. This controller is detailed in a previous publication on adaptive light sheet microscopy.

The code to orthogonally deskew the acquired data and place it into a BigDataViewer HDF5 file that can be read stitched and fused using BigStitcher is found at the qi2lab (www.github.com/qi2lab/OPM/).

Initial setup

Imports

[1]:
from pycromanager import Bridge, Acquisition
import numpy as np
from pathlib import Path
from time import sleep
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/tmp/ipykernel_393/742649323.py in <module>
----> 1 from pycromanager import Bridge, Acquisition
      2 import numpy as np
      3 from pathlib import Path
      4 from time import sleep

ModuleNotFoundError: No module named 'pycromanager'

Create bridge to Micro-Manager

[2]:
with Bridge() as bridge:
    core = bridge.get_core()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/tmp/ipykernel_393/4290683680.py in <module>
----> 1 with Bridge() as bridge:
      2     core = bridge.get_core()

NameError: name 'Bridge' is not defined

Define pycromanager specific hook functions for externally controlled hardware acquisition

Post camera hook function to start external controller

This is run once after the camera is put into active mode in the sequence acquisition. The stage starts moving on this command and outputs a TTL pulse to the camera when it passes the preset initial position. This TTL starts the camera running at the set exposure time using internal timing. The camera acts the master signal for the galvo/laser controller using its own “exposure out” signal.

[3]:
def post_camera_hook_(event,bridge,event_queue):

    """
    Run a set of commands after the camera is started

    :param event: current list of events, each a dictionary, to run in this hardware sequence
    :type event: list
    :param bridge: pycro-manager java bridge
    :type bridge: pycromanager.core.Bridge
    :param event_queue: thread-safe event queue
    :type event_queue: multiprocessing.Queue

    :return: event_queue
    """

    # acquire core from bridge
    core=bridge.get_core()

    # send Tiger command to start constant speed scan
    command='1SCAN'
    core.set_property('TigerCommHub','SerialCommand',command)

    return event

Post hardware setup function to make sure external controller is ready

This is run once after the acquisition engine sets up the hardware for the non-sequencable hardware, such as the height axis stage and channel.

[4]:
def post_hardware_hook(event,bridge,event_queue):

    """
    Run a set of commands after the hardware setup calls by acquisition engine are finished

    :param event: current list of events, each a dictionary, to run in this hardware sequence
    :type event: list
    :param bridge: pycro-manager java bridge
    :type bridge: pycromanager.core.Bridge
    :param event_queue: thread-safe event queue
    :type event_queue: multiprocessing.Queue

    :return: event_queue
    """

    # acquire core from bridge
    core = bridge.get_core()

    # turn on 'transmit repeated commands' for Tiger
    core.set_property('TigerCommHub','OnlySendSerialCommandOnChange','No')

    # check to make sure Tiger is not busy
    ready='B'
    while(ready!='N'):
        command = 'STATUS'
        core.set_property('TigerCommHub','SerialCommand',command)
        ready = core.get_property('TigerCommHub','SerialResponse')
        sleep(.500)

    # turn off 'transmit repeated commands' for Tiger
    core.set_property('TigerCommHub','OnlySendSerialCommandOnChange','Yes')

    return event

Acquistion parameters set by user

Select laser channels and powers

[5]:
# lasers to use
# 0 -> inactive
# 1 -> active

state_405 = 0
state_488 = 0
state_561 = 1
state_635 = 0
state_730 = 0

# laser powers (0 -> 100%)

power_405 = 0
power_488 = 0
power_561 = 0
power_635 = 0
power_730 = 0

# construct arrays for laser informaton
channel_states = [state_405,state_488,state_561,state_635,state_730]
channel_powers = [power_405,power_488,power_561,power_635,power_730]

Camera parameters

[6]:
# FOV parameters.
# x size (256) is the Rayleigh length of oblique light sheet excitation
# y size (1600) is the high quality lateral extent of the remote image system (~180 microns)
# camera is oriented so that cropping the x size limits the number of readout rows and therefore lowering readout time
ROI = [1024, 0, 256, 1600] #unit: pixels

# camera exposure
exposure_ms = 5 #unit: ms

# camera pixel size
pixel_size_um = .115 #unit: um

Stage scan parameters

The user defines these by interactively moving the XY and Z stages around the sample. At the edges of the sample, the user records the positions.

[7]:
# distance between adjacent images.
scan_axis_step_um = 0.2  #unit: um

# scan axis limits. Use stage positions reported by Micromanager
scan_axis_start_um = 0. #unit: um
scan_axis_end_um = 5000. #unit: um

# tile axis limits. Use stage positions reported by Micromanager
tile_axis_start_um = 0. #unit: um
tile_axis_end_um = 5000. #unit: um

# height axis limits. Use stage positions reported by Micromanager
height_axis_start_um = 0.#unit: um
height_axis_end_um = 30. #unit:  um

Path to save acquistion data

[8]:
save_directory = Path('/path/to/save')
save_name = 'test'
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/tmp/ipykernel_393/3548632766.py in <module>
----> 1 save_directory = Path('/path/to/save')
      2 save_name = 'test'

NameError: name 'Path' is not defined

Setup hardware for stage scanning sample through oblique digitally scanned light sheet

Calculate stage limits and speeds from user provided scan parameters

Here, the number of events along the scan (x) axis in each acquisition, the overlap between adajcent strips along the tile (y) axis, and the overlap between adajacent strips along the height (z) axis are all calculated.

[9]:
# scan axis setup
scan_axis_step_mm = scan_axis_step_um / 1000. #unit: mm
scan_axis_start_mm = scan_axis_start_um / 1000. #unit: mm
scan_axis_end_mm = scan_axis_end_um / 1000. #unit: mm
scan_axis_range_um = np.abs(scan_axis_end_um-scan_axis_start_um)  # unit: um
scan_axis_range_mm = scan_axis_range_um / 1000 #unit: mm
actual_exposure_s = actual_readout_ms / 1000. #unit: s
scan_axis_speed = np.round(scan_axis_step_mm / actual_exposure_s,2) #unit: mm/s
scan_axis_positions = np.rint(scan_axis_range_mm / scan_axis_step_mm).astype(int)  #unit: number of positions

# tile axis setup
tile_axis_overlap=0.2 #unit: percentage
tile_axis_range_um = np.abs(tile_axis_end_um - tile_axis_start_um) #unit: um
tile_axis_range_mm = tile_axis_range_um / 1000 #unit: mm
tile_axis_ROI = ROI[3]*pixel_size_um  #unit: um
tile_axis_step_um = np.round((tile_axis_ROI) * (1-tile_axis_overlap),2) #unit: um
tile_axis_step_mm = tile_axis_step_um / 1000 #unit: mm
tile_axis_positions = np.rint(tile_axis_range_mm / tile_axis_step_mm).astype(int)  #unit: number of positions

# if tile_axis_positions rounded to zero, make sure acquisition visits at least one position
if tile_axis_positions == 0:
    tile_axis_positions=1

# height axis setup
# this is more complicated, because the excitation is an oblique light sheet
# the height of the scan is the length of the ROI in the tilted direction * sin(tilt angle)
height_axis_overlap=0.2 #unit: percentage
height_axis_range_um = np.abs(height_axis_end_um-height_axis_start_um) #unit: um
height_axis_range_mm = height_axis_range_um / 1000 #unit: mm
height_axis_ROI = ROI[2]*pixel_size_um*np.sin(30*(np.pi/180.)) #unit: um
height_axis_step_um = np.round((height_axis_ROI)*(1-height_axis_overlap),2) #unit: um
height_axis_step_mm = height_axis_step_um / 1000  #unit: mm
height_axis_positions = np.rint(height_axis_range_mm / height_axis_step_mm).astype(int) #unit: number of positions

# if height_axis_positions rounded to zero, make sure acquisition visits at least one position
if height_axis_positions==0:
    height_axis_positions=1
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/tmp/ipykernel_393/1286776232.py in <module>
      3 scan_axis_start_mm = scan_axis_start_um / 1000. #unit: mm
      4 scan_axis_end_mm = scan_axis_end_um / 1000. #unit: mm
----> 5 scan_axis_range_um = np.abs(scan_axis_end_um-scan_axis_start_um)  # unit: um
      6 scan_axis_range_mm = scan_axis_range_um / 1000 #unit: mm
      7 actual_exposure_s = actual_readout_ms / 1000. #unit: s

NameError: name 'np' is not defined

Setup Coherent laser box from user provided laser parameters

[10]:
with Bridge() as bridge:
    core = bridge.get_core()
    # turn off lasers
    # this relies on a Micro-Manager configuration group that sets all lasers to "off" state
    core.set_config('Coherent-State','off')
    core.wait_for_config('Coherent-State','off')

    # set lasers to user defined power
    core.set_property('Coherent-Scientific Remote','Laser 405-100C - PowerSetpoint (%)',channel_powers[0])
    core.set_property('Coherent-Scientific Remote','Laser 488-150C - PowerSetpoint (%)',channel_powers[1])
    core.set_property('Coherent-Scientific Remote','Laser OBIS LS 561-150 - PowerSetpoint (%)',channel_powers[2])
    core.set_property('Coherent-Scientific Remote','Laser 637-140C - PowerSetpoint (%)',channel_powers[3])
    core.set_property('Coherent-Scientific Remote','Laser 730-30C - PowerSetpoint (%)',channel_powers[4])
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/tmp/ipykernel_393/2215657290.py in <module>
----> 1 with Bridge() as bridge:
      2     core = bridge.get_core()
      3     # turn off lasers
      4     # this relies on a Micro-Manager configuration group that sets all lasers to "off" state
      5     core.set_config('Coherent-State','off')

NameError: name 'Bridge' is not defined

Setup Photometrics camera for low-noise readout and triggering

The camera input trigger is set to Trigger first mode to allow for external control and the output trigger is set to Rolling Shutter mode to ensure that laser light is only delivered when the entire chip is exposed. The custom Teensy DAC waits for the signal from the camera to go HIGH and then sweeps a Gaussian pencil beam once across the field-of-view. It then rapidly resets and scans again upon the next trigger. The Teensy additionally blanks the Coherent laser box emission between frames.

[11]:
with Bridge() as bridge:
    core = bridge.get_core()
    # set camera into 16bit readout mode
    core.set_property('Camera','ReadoutRate','100MHz 16bit')
    # give camera time to change modes
    sleep(5)

    # set camera into low noise readout mode
    core.set_property('Camera','Gain','2-CMS')
    # give camera time to change modes
    sleep(5)

    # set camera to give an exposure out signal
    # this signal is used by the custom DAC to synchronize blanking and a digitally swept light sheet
    core.set_property('Camera','ExposureOut','Rolling Shutter')
    # give camera time to change modes
    sleep(5)

    # change camera timeout.
    # this is necessary because the acquisition engine can take a long time to setup with millions of events
    # on the first run
    core.set_property('Camera','Trigger Timeout (secs)',300)
    # give camera time to change modes
    sleep(5)

    # set camera to internal trigger
    core.set_property('Camera','TriggerMode','Internal Trigger')
    # give camera time to change modes
    sleep(5)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/tmp/ipykernel_393/250576116.py in <module>
----> 1 with Bridge() as bridge:
      2     core = bridge.get_core()
      3     # set camera into 16bit readout mode
      4     core.set_property('Camera','ReadoutRate','100MHz 16bit')
      5     # give camera time to change modes

NameError: name 'Bridge' is not defined

Setup ASI stage control cards and programmable logic card in the Tiger controller

Hardware is setup for a constant-speed scan along the x direction, lateral tiling along the y direction, and height tiling along the z direction. The programmable logic card sends a signal to the camera to start acquiring once the scan (x) axis reaches the desired speed and crosses the user defined start position.

Documentation for the specific commands to setup the constant speed stage scan on the Tiger controller is at the following links, - SCAN - SCANR - SCANV

Documentation for the programmable logic card is found here.

The Tiger is polled after each command to make sure that it is ready to receive another command.

[12]:
with Bridge() as bridge:
    core = bridge.get_core()
    # Setup the PLC to output external TTL when an internal signal is received from the stage scanning card
    plcName = 'PLogic:E:36'
    propPosition = 'PointerPosition'
    propCellConfig = 'EditCellConfig'
    addrOutputBNC3 = 35
    addrStageSync = 46  # TTL5 on Tiger backplane = stage sync signal
    core.set_property(plcName, propPosition, addrOutputBNC3)
    core.set_property(plcName, propCellConfig, addrStageSync)

    # turn on 'transmit repeated commands' for Tiger
    core.set_property('TigerCommHub','OnlySendSerialCommandOnChange','No')

    # set tile (y) axis speed to 25% of maximum for all moves
    command = 'SPEED Y=.25'
    core.set_property('TigerCommHub','SerialCommand',command)

    # check to make sure Tiger is not busy
    ready='B'
    while(ready!='N'):
        command = 'STATUS'
        core.set_property('TigerCommHub','SerialCommand',command)
        ready = core.get_property('TigerCommHub','SerialResponse')
        sleep(.500)

    # set scan (x) axis speed to 25% of maximum for non-sequenced moves
    command = 'SPEED X=.25'
    core.set_property('TigerCommHub','SerialCommand',command)

    # check to make sure Tiger is not busy
    ready='B'
    while(ready!='N'):
        command = 'STATUS'
        core.set_property('TigerCommHub','SerialCommand',command)
        ready = core.get_property('TigerCommHub','SerialResponse')
        sleep(.500)

    # turn off 'transmit repeated commands' for Tiger
    core.set_property('TigerCommHub','OnlySendSerialCommandOnChange','Yes')

    # turn on 'transmit repeated commands' for Tiger
    core.set_property('TigerCommHub','OnlySendSerialCommandOnChange','No')

    # set scan (x) axis speed to correct speed for constant speed movement of scan (x) axis
    # expects mm/s
    command = 'SPEED X='+str(scan_axis_speed)
    core.set_property('TigerCommHub','SerialCommand',command)

    # check to make sure Tiger is not busy
    ready='B'
    while(ready!='N'):
        command = 'STATUS'
        core.set_property('TigerCommHub','SerialCommand',command)
        ready = core.get_property('TigerCommHub','SerialResponse')
        sleep(.500)

    # set scan (x) axis to true 1D scan with no backlash
    command = '1SCAN X? Y=0 Z=9 F=0'
    core.set_property('TigerCommHub','SerialCommand',command)

    # check to make sure Tiger is not busy
    ready='B'
    while(ready!='N'):
        command = 'STATUS'
        core.set_property('TigerCommHub','SerialCommand',command)
        ready = core.get_property('TigerCommHub','SerialResponse')
        sleep(.500)

    # set range and return speed (25% of max) for constant speed movement of scan (x) axis
    # expects mm
    command = '1SCANR X='+str(scan_axis_start_mm)+' Y='+str(scan_axis_end_mm)+' R=25'
    core.set_property('TigerCommHub','SerialCommand',command)

    # check to make sure Tiger is not busy
    ready='B'
    while(ready!='N'):
        command = 'STATUS'
        core.set_property('TigerCommHub','SerialCommand',command)
        ready = core.get_property('TigerCommHub','SerialResponse')
        sleep(.500)

    # turn off 'transmit repeated commands' for Tiger
    core.set_property('TigerCommHub','OnlySendSerialCommandOnChange','Yes')
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/tmp/ipykernel_393/3626689900.py in <module>
----> 1 with Bridge() as bridge:
      2     core = bridge.get_core()
      3     # Setup the PLC to output external TTL when an internal signal is received from the stage scanning card
      4     plcName = 'PLogic:E:36'
      5     propPosition = 'PointerPosition'

NameError: name 'Bridge' is not defined

Setup and run the acquisition

Change core timeout

This is necessary because of the large, slow XY stage moves.

[13]:
with Bridge() as bridge:
    core = bridge.get_core()
    # change core timeout for long stage moves
    core.set_property('Core','TimeoutMs',20000)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/tmp/ipykernel_393/3624047695.py in <module>
----> 1 with Bridge() as bridge:
      2     core = bridge.get_core()
      3     # change core timeout for long stage moves
      4     core.set_property('Core','TimeoutMs',20000)

NameError: name 'Bridge' is not defined

Move stage hardware to initial positions

[14]:
with Bridge() as bridge:
    core = bridge.get_core()
    # move scan (x) and tile (y) stages to starting positions
    core.set_xy_position(scan_axis_start_um,tile_axis_start_um)
    core.wait_for_device(xy_stage)

    # move height (z) stage to starting position
    core.set_position(height_position_um)
    core.wait_for_device(z_stage)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/tmp/ipykernel_393/3684974007.py in <module>
----> 1 with Bridge() as bridge:
      2     core = bridge.get_core()
      3     # move scan (x) and tile (y) stages to starting positions
      4     core.set_xy_position(scan_axis_start_um,tile_axis_start_um)
      5     core.wait_for_device(xy_stage)

NameError: name 'Bridge' is not defined

Create event structure

The external controller handles all of the events in x for a given yzc position. To make sure that pycro-manager structures the acquistion this way, the value of the stage positions for x are kept constant for all events at a given yzc position. This gives the order of the loops to create the event structure as yzcx.

[15]:
# empty event dictionary
events = []

# loop over all tile (y) positions.
for y in range(tile_axis_positions):

    # update tile (y) axis position
    tile_position_um = tile_axis_start_um+(tile_axis_step_um*y)

    # loop over all height (z) positions
    for z in range(height_axis_positions):

        # update height (z) axis position
        height_position_um = height_axis_start_um+(height_axis_step_um*z)

        # loop over all channels (c)
        for c in range(len(channel_states)):

            # create events for all scan (x) axis positions.
            # The acquistion engine knows that this is a hardware triggered sequence because
            # the physical x position does not change when specifying the large number of x events
            for x in range(scan_axis_positions):

                # only create events if user sets laser to active
                # this relies on a Micromanager group 'Coherent-State' that has individual entries that correspond
                # the correct on/off state of each laser. Laser blanking and synchronization are handled by the
                # custom Teensy DAC controller.
                if channel_states[c]==1:
                    if (c==0):
                        evt = { 'axes': {'x': x, 'y':y, 'z':z}, 'x': scan_axis_start_um, 'y': tile_position_um,
                               'z': height_position_um, 'channel' : {'group': 'Coherent-State', 'config': '405nm'}}
                    elif (c==1):
                        evt = { 'axes': {'x': x, 'y':y, 'z':z}, 'x': scan_axis_start_um, 'y': tile_position_um,
                               'z': height_position_um, 'channel' : {'group': 'Coherent-State', 'config': '488nm'}}
                    elif (c==2):
                        evt = { 'axes': {'x': x, 'y':y, 'z':z}, 'x': scan_axis_start_um, 'y': tile_position_um,
                               'z': height_position_um, 'channel' : {'group': 'Coherent-State', 'config': '561nm'}}
                    elif (c==3):
                        evt = { 'axes': {'x': x, 'y':y, 'z':z}, 'x': scan_axis_start_um, 'y': tile_position_um,
                               'z': height_position_um, 'channel' : {'group': 'Coherent-State', 'config': '637nm'}}
                    elif (c==4):
                        evt = { 'axes': {'x': x, 'y':y, 'z':z}, 'x': scan_axis_start_um, 'y': tile_position_um,
                               'z': height_position_um, 'channel' : {'group': 'Coherent-State', 'config': '730nm'}}

                    events.append(evt)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/tmp/ipykernel_393/886332684.py in <module>
      3
      4 # loop over all tile (y) positions.
----> 5 for y in range(tile_axis_positions):
      6
      7     # update tile (y) axis position

NameError: name 'tile_axis_positions' is not defined

Run acquisition

  • The camera is set to Trigger first mode. In this mode, the camera waits for an external trigger and then runs using the internal timing.

  • The acquisition is setup and started. The initial acquisition setup by Pycro-manager and the Java acquisition engine takes a few minutes and requires at significant amount of RAM allocated to ImageJ. 40 GB of RAM seems acceptable. The circular buffer is only allocated 2 GB, because the computer for this experiment has an SSD array capable of writing up to 600 MBps.

  • At each yzc position, the ASI Tiger controller supplies the external master signal when the the (scan) axis has ramped up to the correct constant speed and crossed scan_axis_start_um. The speed is defined by scan_axis_speed = scan_axis_step_um / camera_exposure_ms. Acquired images are placed into the x axis of the Acquisition without Pycro-Manager interacting with the hardware.

  • Once the full acquisition is completed, all lasers are set to off and the camera is placed back in Internal Trigger mode.

[16]:
with Bridge() as bridge:
    core = bridge.get_core()
    # set camera to trigger first mode for stage synchronization
    # give camera time to change modes
    core.set_property('Camera','TriggerMode','Trigger first')
    sleep(5)

    # run acquisition
    # the acquisition needs to write data at roughly 100-500 MBps depending on frame rate and ROI
    # so the display is set to off and no multi-resolution calculations are done
    with Acquisition(directory=save_directory, name=save_name, post_hardware_hook_fn=post_hardware_hook,
                    post_camera_hook_fn=post_camera_hook, show_display=False, max_multi_res_index=0) as acq:
        acq.acquire(events)

    # turn off lasers
    core.set_config('Coherent-State','off')
    core.wait_for_config('Coherent-State','off')

    # set camera to internal trigger
    core.set_property('Camera','TriggerMode','Internal Trigger')
    # give camera time to change modes
    sleep(5)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/tmp/ipykernel_393/1384460599.py in <module>
----> 1 with Bridge() as bridge:
      2     core = bridge.get_core()
      3     # set camera to trigger first mode for stage synchronization
      4     # give camera time to change modes
      5     core.set_property('Camera','TriggerMode','Trigger first')

NameError: name 'Bridge' is not defined