The previous ten pages covered the modalities behind most medical and biological image data, but not all of them. This page collects six more that you are likely to meet, each with a short description: what the instrument is for and when it is used, what its images look like, what to consider when training on them, and where to find public data. Three of them (endoscopy, fundus photography, dermoscopy) are ordinary color cameras pointed at the body through a tube, a pupil or a magnifier. Their images look like photographs and the tools built for photographs transfer to them, which is both their convenience and their trap: the lamp, the device, the compression and the unknown scale still shape what a model learns. The other three are not cameras: OCT measures depth by timing reflected light, the way ultrasound times reflected sound, except that light is too fast to time directly and interference does the timing; photoacoustic imaging sends light into the tissue, the absorbed light makes a tiny pressure wave, and an ultrasound probe listens for it; and fluoroscopy is 2.2 X-ray radiography run as video.

Endoscopy and surgical video

a) A colonoscope: a flexible tube with the camera, light guide and working channel in its tip, steered from the control handle (Gilo1969, CC BY-SA 3.0). b) What it records: a polyp on a stalk, with the round field, vignetting and specular highlights typical of endoscopic video (Gilo1969, CC BY-SA 3.0). Both Wikimedia Commons.

An endoscope is a camera on a tube, put into the body through a natural opening or a small incision. Gastroenterologists use it to look at the oesophagus, stomach and colon: to screen for colorectal cancer by finding and removing polyps, to find the source of bleeding, and to take biopsies. Surgeons use the same technology as a laparoscope in keyhole surgery, where the video is their only view. A capsule endoscope, a swallowed camera, films the small bowel, which the tubes do not reach.

The image is color video of a wet, curved surface lit by a lamp that travels with the camera: a round field in a black mask, bright reflections, brightness that falls off with distance, motion blur, fluid and bubbles. Nothing in the image says where the camera is or how large things are.

Polyp detection is the main AI application. Colonoscopists miss about a quarter of adenomas (Zhao et al., 2019), and real-time detection has been tested in randomized trials; in one, the adenoma detection rate rose from 40.4% to 54.8% with the system on (Repici et al., 2020). In surgical video the tasks are recognising the phase of the operation and segmenting the instruments.

Considerations for AI models

• Consecutive video frames are near-duplicates, so split training and test data by patient and procedure, never by frame.
• The camera is not a neutral observer. Color depends on the scope, its lamp and its imaging mode, and the endoscopist moves closer to anything suspicious, so polyps tend to be filmed larger, brighter and with more reflections than normal mucosa. A model can learn that cue instead of the polyp.
• A curated dataset holds clean frames; the live stream holds blur, fluid and bubbles. Accuracy on held-out images overstates performance in the procedure room.

Public datasets. HyperKvasir (Borgli et al., 2020) is the main open set for gastrointestinal endoscopy: 110,079 images and 374 videos, 10,662 of the images labelled with findings and anatomical landmarks. Kvasir-SEG adds 1,000 polyp images with segmentation masks. For surgical video, Cholec80 (Twinanda et al., 2017) holds 80 recorded gallbladder removals with phase and instrument labels.

Fundus photography

Fundus photograph of proliferative diabetic retinopathy: optic disc, vessel tree, exudates and new vessels. National Eye Institute, public domain.

A fundus camera photographs the back of the eye through the pupil. It is the routine instrument of eye clinics and of diabetic eye screening programmes, in which every person with diabetes is photographed every one to two years so that retinopathy is caught before it threatens sight. The same photographs serve for glaucoma, read from the shape of the optic disc, and for macular degeneration.

The image is a color photograph of a disc of retina about 45° wide: the bright optic disc where the nerve and the vessels enter, the darker macula, and the vessel tree. Diabetic retinopathy shows as small red dots (microaneurysms), haemorrhages, pale exudates and, late, new vessels, and is graded from 0 to 4.

Grading these photographs is where medical AI first reached the clinic. Gulshan et al. (2016) trained a network on 128,000 graded photographs and reached about 90% sensitivity (the share of eyes with referable retinopathy that it flagged) and 98% specificity (the share of eyes without it that it correctly passed). In 2018 the FDA authorized IDx-DR (Abràmoff et al., 2018), the first AI diagnostic allowed to give a result without a clinician reading the image, for one task, one population and one camera. Poplin et al. (2018) predicted age, sex, smoking and cardiac risk from the same photographs. The attention maps pointed at the vessels and the optic disc: the model had found patterns that correlate with those targets, which no grader looks for.

Considerations for AI models

• Quality: ungradable photographs run from about 4% with a dilated pupil to about 20% without (Scanlon et al., 2005), more often in the sickest patients, and dust on the lens marks every image from one camera.
• Laterality and framing: left and right eyes are mirror images, so flips are fine for lesions and wrong for anything that depends on side; cameras and field positions change color and framing.
• Labels: graders disagree on the retinopathy grade (Krause et al., 2018), so a model scored against one grader is partly scored against that grader's errors.