So far, we have covered a lot of biomedical image modalities. In each one, the end result was usually an array of integers, which we call $I$. Every one of them began with a specimen and a measurement as a continuous quantity: attenuation along a ray, a voltage in a coil, pressure at a transducer face, light on a sensor. Somewhere in between, the quantity was measured at a finite set of positions and each measurement was stored as a whole number.
A digital image is the continuous field read on a grid and quantized. A detail of a T2-weighted brain slice (box in the inset) is averaged over 12 × 12 pixels, and each average is quantized to one of 8 levels; the stored image is the array of integers on the right. Scanners use finer grids and $2^{12}$ or more levels, but the two steps are the same. Image: Sean Novak, CC BY-SA 4.0, Wikimedia Commons.
In 2.1 What is a biomedical image? we talked about sampling and quantization. Sampling decides where values are measured; quantization decides which value is stored. This page covers those processes and the complications they bring.
Sampling and quantization also happen when an image is resized, resliced or cast to a different data type. The concepts we will cover therefore apply to a data loader as much as to a scanner.
In code, an image is an array: a grid of numbers of one data type, indexed by integers along each axis (NumPy calls it an ndarray). Deep learning frameworks call the same object a tensor, which also records the device it lives on (CPU or GPU) and whether gradients are tracked. In mathematics a tensor is more than an array of numbers, but in deep learning the word simply means an array with any number of axes.
The shape of an array lists its axes: rows and columns for a 2D image, one more axis for channels or slices, and a batch axis N in front when images go into a network.
An array has a shape: two axes for a radiograph, three for a CT volume or a microscopy z-stack, and extra axes for channels (fluorescence, color) and for time (cine loops, live imaging). Which axis is which, and how far apart the samples are, is written in the metadata (2.15 File formats and metadata).
In a color photo, the three channels measure the same quantity, light, in three color bands. In medical images a channel can hold a different measurement altogether: T1- and T2-weighted MRI of the same patient, several fluorescent markers, or the PET and CT of a PET-CT scan. Stacking them on one grid only makes sense if a voxel index points to the same place in the specimen in every channel, so the images must first be registered to the same geometry (2.14 Image geometry: spacing, orientation and the affine). Each channel also has its own units and range, as with HU and PET activity, so loaders normalize each channel separately.
A minibatch for a network adds one more axis, the number of examples. PyTorch orders a batch of volumes as (N, C, D, H, W), where N is the number of images in the batch, C the number of channels, and D, H and W the depth, height and width of each volume.
The data type sets which values a pixel can hold. Label maps hold class numbers, not measurements. The value axis is logarithmic on both sides of zero (a symmetric log scale).
All values in an array share one data type, which fixes how many bits each value takes and which numbers it can hold. File formats restrict the choice: PNG and JPEG, for example, cannot store negative values or fractions. Five types cover almost everything in this course, shown in the figure above. Regardless of how the values are stored in files, machine learning frameworks usually convert them and work with float32. The exception is a label map (an integer type, usually uint8), which stores class numbers, not measurements.
A digital image is not the continuous field evaluated at grid points. Each stored value comes from a small region of the specimen, not from a point. A CMOS camera pixel, for example, converts the light landing on its sensitive area into charge during the exposure and records the total as one number. The size and shape of that region are set by the measurement system. The value is then assigned to one grid position and quantized. Each of these steps loses some information.
Measuring over a region erases detail smaller than the region: two specks closer together than its width merge into one. Sampling records one value per grid position, and nothing about what lies between neighboring positions. Quantization replaces each value by the nearest of a fixed set of levels, so values closer together than one level become identical.
Sampling and quantization turn a real signal into a digital image. Measuring over a region of width $h$ merges the two thin peaks into one. Sampling keeps one value per grid point, $\Delta$ apart. Quantization sets each value to one of 8 levels, and the array stores the level numbers.
Write the continuous field as $s(\mathbf{x})$, defined on the field of view $\Omega$ of Eq. 2.1.2. The region each value comes from is described by the point spread function $h$: the image the system makes of a single bright point. It is a model, accurate when the system responds in the same way across the field of view.
The width of $h$ depends on the modality. In radiography, the focal spot, the size of one detector element and the spread of light in the scintillator; in microscopy, diffraction in the objective; in ultrasound, the width of the beam and the length of the pulse. Mathematically, measuring through $h$ is a convolution, written $sh$: each point of the measured field is a weighted sum of the true field around it, with weights given by $h$. The stored value at grid index $i$ is the measured field $sh$, read at the grid position:
$$ I[i]=Q\big[(s*h)(\mathbf{x}_0+i\,\Delta)\big]\tag{2.13.1} $$