An image file holds an array of numbers and a header that says what the numbers mean. We already learned that the image array alone cannot tell a 5 mm nodule from a 10 mm one, or tell a Hounsfield unit from a stored integer, or even tell left from right.
File headers with metadata can answer these questions and more. Geometry: the voxel spacing, the position of the first voxel in the patient, and the orientation of the axes. Values: how many bits are used, whether they are signed, how to turn them into physical units, and whether they were compressed. Provenance: which modality, scanner and protocol produced the image, when, and of whom.
The same pixel values can carry a lot more meaning, depending on the file format. DICOM keeps the geometry, the values, the provenance and the patient's identity, in one file per slice. NIfTI keeps the geometry and a value scaling.
Formats trade completeness for convenience. DICOM keeps all three kinds of information, plus the patient's identity. It is not a wrapper around other files: each DICOM file holds one image with its own header of hundreds of tags, so a CT scan arrives as a folder of hundreds of files (more on this below). That makes DICOM awkward to share, and most deep learning code expects one array per example rather than a folder of files, although some loaders, such as MONAI's, read DICOM directly.
Leaner formats, such as NIfTI (.nii), PNG (.png) and JPEG (.jpg), or arrays saved in NumPy's own format (.npy), hold a volume or image per file and load fast, but keep less of the header. AI pipelines therefore often convert, and keep the metadata they still need in a text or table file next to the images, such as .json, .csv or .tsv; large projects use .parquet tables or a small database. A conversion can keep everything, but it can also drop metadata, change pixel values by rescaling, casting or lossy compression, or reorient the array, so check what the converter did and keep the metadata and provenance you will need.
The common formats at a glance: where you meet each one, what one file holds, which geometry and units it keeps, how it stores pixels, and what to check after loading.
We cover some of the most important formats below. In radiology almost every image starts as a DICOM file, so we start there.
DICOM is the format that most clinical scanners write and that hospital archives (PACS) store. A DICOM file (.dcm) holds one image, such as one CT slice or one radiograph, together with a header of named tags. A CT scan is therefore not one file but a folder of several hundred. No file wraps them together. Instead, each file's header carries identifiers, generated by the scanner and unique worldwide, that say where the file belongs. All slices of one reconstruction form a series and share a SeriesInstanceUID. All series from one visit to the scanner, for example a CT before and after contrast plus the scout images, form a study and share a StudyInstanceUID. Some newer scanners instead write enhanced multi-frame files: a single file that holds all the frames of a volume, an ultrasound cine loop or a whole slide, with NumberOfFrames saying how many. We write tags by their keyword, the name that pydicom uses (ds.PixelSpacing). The tags that matter for analysis generally fall into four groups:
PixelSpacing gives the in-plane spacing in millimeters, between rows and then between columns. ImagePositionPatient gives the position of the first pixel of the slice, and ImageOrientationPatient the directions of its rows and columns, both in the patient's LPS coordinates in millimeters (2.14 Image geometry: spacing, orientation and the affine). SliceThickness is how thick a slab of tissue each slice averages over, not how far apart the slices are. That distance comes from the positions of neighboring slices, and it can be smaller than the thickness, so that slices overlap, or larger, so that there are gaps.BitsStored and PixelRepresentation say how many bits are used and whether they are signed. RescaleSlope and RescaleIntercept turn the stored integers into physical units, Hounsfield units for CT. WindowCenter and WindowWidth suggest how to display the image. PhotometricInterpretation says whether bright means high (MONOCHROME2) or low (MONOCHROME1), and the TransferSyntaxUID says whether the pixels are compressed, possibly with lossy compression.Modality (CT, MR, US and so on); Manufacturer, ManufacturerModelName and SoftwareVersions (which scanner, running which software); InstitutionName and StationName (which hospital, which machine); SeriesDescription and ProtocolName (the protocol, as free text); for CT, KVP, XRayTubeCurrent and ConvolutionKernel; for MR, MagneticFieldStrength, RepetitionTime and EchoTime; ContrastBolusAgent if contrast was given; and AcquisitionDate, AcquisitionTime and, in cine loops, FrameTime. These are the tags to group your data by when a model works at one hospital and fails at another. They also explain what a model can learn as a shortcut: a sharper reconstruction kernel or another scanner model changes the texture of every image.PatientName, PatientID, PatientBirthDate, AccessionNumber, the dates, and the UIDs, which are globally unique and so link the file back to the hospital's records. These are the tags that de-identification must remove or replace before images leave the hospital (see Trustworthy AI).
Sorting the slices of a CT series by file name (middle) scrambles the anatomy; sorting by the z value of ImagePositionPatient (right) restores it.
To build a volume, collect the files of one series, sort them by their position along the slice axis, and check that neighboring slices are equally spaced (figure). Sorting by file name instead can scramble the anatomy, because nothing makes the names follow it. SimpleITK's series reader and dcm2niix sort for you, but silently: a missing slice or two mixed series comes out wrong without an error.
NIfTI (.nii, or .nii.gz when compressed) stores a whole volume in one file: a small header followed by the voxel array. It began in brain imaging and is now the usual research format for 3D and 4D medical images in every specialty, from abdominal oncology to cardiology. Researchers convert DICOM to NIfTI because it turns a folder of hundreds of slice files into one file that loads with one line of Python, and because most 3D tools, such as nnU-Net, expect it. Almost everyone converts with dcm2niix, which writes one NIfTI file per DICOM series.