Building Software Systems for Image-Guided Robot-Assisted Interventions

Workshop at 2021 ISMR

Back to Tutorial Home


In this tutorial, we will create a surgical plan using a 3D model of the spine reconstructed from tracked ultrasound images. Following steps are required to accomplish this task:

Load Slicer Scene with Data

Before loading the file, make sure to download the following files:

The scene file can be loaded to 3D Slicer as follows:

Please also check if the “Sequence brower” tool bar is visible on the window. If not, turn it on by selecting “View”->”Toolbars” -> “Sequence browser”.

Set up Input and Output for Segmentation Module

Segmentation module can be opened from the “Modules” menu by selecting “Ultrasound” -> “Segmentation UNet”. Configure the module as follows:

Initialize Prediction by Updating Input

Once the moduel is configured, click “Apply segmentation” button, and then click the “Next frame” button on the Sequence bar to fully start the AI process.

Visualize prediction

First, change the window layout to “Four-Up” layout so that the original ultrasound frame, prediction image, and 3D rendering are all visible on the window.

Configure the Green viewers as follows:

Apply the PredictionToRas transform to the “Predition” image as follows:

Finally, configure the viewers to follow the imaging plane.

Set up volume reconstruction

The volume reconstruction module creates a 3D volumetric image from a tracked image. Configure the module as follows to use the prediction image:

Once the module is configured, click the “Start” button at the bottom of the module panel.

Start ultrasound replay

From the Sequence tool bar, click “Playback” button to start ultrasound replay. The green viewer should start showing the prediction image.

Set up volume rendering to visualize spine volume

To visualize the prediction in 3D, setup volume rendering as follows:

Stop Volume Reconstruction

When the volume becomes ready, stop playback, and stop Volume Reconstruction and Segmentation UNet. Also,


First, load the spine models on 3D Slicer:

Next, setup fiducial registration: