Vision & Multimodal AI

An image is a grid of pixels, each holding one value per channel at some bit depth — typically 8-bit RGB, giving 256 levels per channel. Height × width × channels is the tensor every later stage consumes.

Bit depth matters more than people expect: 8-bit is fine for display but clips highlights and quantises gradients, which is why medical and scientific imaging works in 12- or 16-bit.

Uses: numpy · cv2.imread · PIL.Image.open

RGB is how displays emit light, not how anything perceives it. HSV separates hue from saturation and brightness, making colour thresholding tractable; LAB is designed so that equal numeric distances correspond to roughly equal perceptual differences.

Choosing the right space often converts a hard problem into a trivial one — segmenting a coloured object is painful in RGB and nearly free in HSV, because illumination changes move all three RGB channels at once.

Uses: cv2.cvtColor · skimage.color

A histogram counts pixel intensities. Its shape tells you immediately whether an image is under- or over-exposed, low contrast, or clipped.

Equalization redistributes intensities to spread the histogram, and CLAHE does it in local tiles so one bright region does not wash out the rest. It is a standard preprocessing step wherever illumination varies across a dataset.

Uses: cv2.calcHist · cv2.equalizeHist · cv2.createCLAHE

Operations applied to each pixel independently of its neighbours — brightness offsets, contrast scaling, gamma correction, inversion, and lookup-table mappings.

Gamma is the one that matters most and is most often ignored: pixel values are usually stored non-linearly, so averaging or blending raw values is arithmetically wrong unless you linearise first.

Uses: numpy · cv2.LUT · cv2.convertScaleAbs

Convolution slides a small matrix over the image, replacing each pixel with a weighted sum of its neighbourhood. The kernel's values determine the operation — blur, sharpen, or edge response.

This is the single most important operation in the field. The only difference between classical image processing and a CNN is whether the kernel weights are designed or learned.

Uses: cv2.filter2D · scipy.ndimage.convolve

Gaussian blur averages with distance-weighted neighbours; median filtering replaces each pixel with the neighbourhood median and is far better against salt-and-pepper noise; bilateral filtering weights by both spatial and intensity distance.

Bilateral is the useful one when you want denoising without destroying edges — it refuses to average across a large intensity jump, which is exactly what Gaussian blur does wrong.

Uses: cv2.GaussianBlur · cv2.medianBlur · cv2.bilateralFilter

Edges are intensity discontinuities, found by approximating spatial derivatives. Sobel and Prewitt give gradient magnitude and direction; Laplacian responds to second derivatives.

Canny remains the standard because it is a pipeline rather than a filter: smooth, compute gradients, suppress non-maxima to thin edges, then hysteresis-threshold so weak edges survive only when connected to strong ones.

Uses: cv2.Canny · cv2.Sobel · cv2.Laplacian

Reducing an image to foreground and background. A global threshold works only under even illumination; Otsu's method picks it automatically by maximising between-class variance.

Adaptive thresholding computes a local threshold per region and is what makes OCR work on photographed documents, where one corner is always darker than the other.

Uses: cv2.threshold · cv2.adaptiveThreshold · skimage.filters.threshold_otsu

Shape operations on binary images using a structuring element. Erosion shrinks foreground, dilation grows it; opening (erode then dilate) removes small specks, and closing (dilate then erode) fills small holes.

These are the cheap, reliable cleanup tools after thresholding or segmentation — removing isolated noise pixels and closing gaps in contours that would otherwise break connected-component analysis.

Uses: cv2.erode · cv2.dilate · cv2.morphologyEx

A pyramid is the same image at successively lower resolutions. Gaussian pyramids downsample after blurring; Laplacian pyramids store the difference between levels, capturing detail per scale.

Scale is the recurring problem in vision — an object's size in pixels is arbitrary — and pyramids are the classical answer, echoed later by feature pyramid networks in detection.

Uses: cv2.pyrDown · cv2.pyrUp · skimage.transform.pyramid_gaussian

The Fourier transform re-expresses an image as a sum of spatial frequencies. Low frequencies carry broad structure, high frequencies carry edges and texture.

It makes some operations trivial — periodic noise becomes an isolated spike you can remove, and convolution becomes multiplication — and it explains aliasing, which is why downsampling without blurring first produces moiré artefacts.

Uses: numpy.fft · cv2.dft · scipy.fft

Translation, rotation, scaling, affine, and perspective (homography) transforms map pixels to new positions. Because the mapping lands between pixels, the value must be interpolated — nearest neighbour, bilinear, or bicubic.

A homography maps any plane to any other plane, which is what lets you rectify a photographed document to a flat rectangle. Interpolation choice is a real quality decision: nearest neighbour preserves hard labels in masks, bilinear is right for photographs.

Uses: cv2.warpAffine · cv2.warpPerspective · cv2.findHomography

JPEG divides the image into blocks, applies a discrete cosine transform, and quantises high-frequency coefficients — which is where the information is thrown away. PNG is lossless; WebP and AVIF improve on both.

Worth understanding because compression artefacts become training signal. A model trained only on high-quality JPEGs frequently degrades on heavily compressed real-world uploads, and blocking artefacts can be mistaken for texture.

Uses: cv2.imencode · PIL.Image.save · imageio

Corners and blobs are locations identifiable across viewpoints. Harris detects corners; SIFT and ORB add scale and rotation invariance plus a descriptor vector for matching between images.

These carried vision for two decades and remain in use where learned features are unnecessary or unaffordable — SLAM, panorama stitching, and registration all still run on them.

Uses: cv2.SIFT_create · cv2.ORB_create · cv2.BFMatcher

Aligning multiple images of the same scene: detect features, match them, estimate a transform robustly with RANSAC to reject bad matches, then warp and blend.

RANSAC is the key idea and generalises well beyond vision — fit a model on random minimal subsets, keep the hypothesis with the most inliers, and let outliers be ignored rather than averaged in.

Uses: cv2.findHomography (RANSAC) · cv2.Stitcher

Non-local means averages similar patches from anywhere in the image rather than only nearby pixels; wavelet methods threshold coefficients in a multi-scale basis; deconvolution attempts to invert a known blur.

The insight behind non-local means — that natural images are self-similar, so the best evidence about a patch is other patches like it — later reappears in self-attention.

Uses: cv2.fastNlMeansDenoising · skimage.restoration · PyWavelets

Recovering how a scene would look under neutral illumination. Grey-world assumes the average scene is achromatic; more careful methods estimate the illuminant explicitly.

It matters for datasets because illumination is a confound: a classifier can learn indoor tungsten lighting instead of the object, and then fail outdoors.

Uses: cv2.xphoto.createSimpleWB · colour-science

Classification assigns a label to the whole image. Detection adds localisation with bounding boxes. Semantic segmentation labels every pixel by class; instance segmentation separates individual objects; panoptic segmentation does both at once.

Choosing the weakest task that solves your problem is a real cost decision — annotation effort rises steeply from image labels to boxes to pixel masks.

Uses: torchvision.datasets · pycocotools

Convolutional layers apply learned filters, stride controls downsampling, padding preserves spatial size, and pooling aggregates locally. Stacking them grows the receptive field — how much of the input one output unit can see.

Receptive field is the concept to hold onto: if it is smaller than the object, the network structurally cannot see the whole thing, no matter how much you train it.

Uses: torch.nn.Conv2d · torch.nn.MaxPool2d

A backbone is a feature extractor — ResNet, EfficientNet, ViT — pretrained on a large dataset and reused. You attach a task-specific head and fine-tune.

This is the default starting point for almost all applied vision. Early layers learn edges and textures that transfer across essentially every visual domain, so training from scratch is rarely justified below very large data scales.

Uses: timm.create_model · torchvision.models

Synthetic variation applied during training — flips, crops, rotations, colour jitter, and stronger schemes like Mixup, CutMix, and RandAugment.

It encodes the invariances you want. The discipline is that augmentation must preserve the label: horizontal flips are fine for cats and wrong for text or for distinguishing left from right hands.

Uses: albumentations · torchvision.transforms.v2

Intersection over Union measures box overlap. Detectors emit many overlapping candidates, so non-maximum suppression keeps the highest-confidence box and discards others above an IoU threshold.

NMS is a frequent source of subtle bugs: too aggressive and genuinely overlapping objects get deleted, too permissive and duplicates survive into the output.

Uses: torchvision.ops.nms · torchvision.ops.box_iou

mAP averages precision over recall levels and IoU thresholds for detection; Dice and IoU score segmentation masks; classification uses precision, recall, and F1.

Metric choice encodes what you care about. A pedestrian detector should be scored on recall at high precision, because a missed pedestrian and a false box are not equally costly — and mAP hides that entirely.

Uses: torchmetrics · pycocotools.cocoeval

Two-stage detectors (R-CNN → Faster R-CNN) propose regions then classify them, favouring accuracy. One-stage detectors (YOLO, SSD, RetinaNet) predict directly over a grid in a single pass, favouring speed. Anchor-free methods (FCOS, CenterNet) drop preset box priors entirely.

Focal loss was the pivotal idea for one-stage models: background overwhelms foreground by orders of magnitude, and down-weighting easy negatives is what closed the accuracy gap.

Uses: ultralytics.YOLO · torchvision.models.detection

FCN made segmentation fully convolutional. U-Net added skip connections between encoder and decoder, restoring the spatial detail that downsampling destroys — which is why it dominates medical imaging. Mask R-CNN adds a mask head to detection; DeepLab uses atrous convolution to widen receptive field without losing resolution.

The recurring tension is resolution against context: you must downsample to see broadly, and you must retain detail to place boundaries precisely.

Uses: segmentation_models_pytorch · torchvision maskrcnn

ViT splits an image into patches, embeds them as tokens, and applies a standard transformer. Without convolution's built-in locality bias, it needs more data or stronger augmentation — but scales better when that data exists.

DETR reframed detection as set prediction with bipartite matching, removing anchors and NMS from the pipeline entirely and making detection end-to-end for the first time.

Uses: timm · transformers.ViTModel · DetrForObjectDetection

Maintaining identity across frames. SORT combines a Kalman filter for motion with Hungarian matching on IoU; DeepSORT adds an appearance embedding so identities survive occlusion.

The hard part is not detection but association — deciding whether the box in this frame is the same object as the box in the last one, through occlusion, crossing paths, and missed detections.

Uses: ultralytics track · filterpy (Kalman) · scipy.optimize.linear_sum_assignment

Locating structured landmarks — human joints, facial points, object keypoints. Top-down approaches detect a person then find joints within the crop; bottom-up detect all joints then group them.

Heatmap regression rather than direct coordinate regression is the standard trick: predicting a spatial probability map per keypoint is far easier to learn than predicting numbers.

Uses: ultralytics pose · mediapipe · torchvision keypointrcnn

The pinhole model relates 3D points to image pixels through intrinsics (focal length, principal point) and extrinsics (rotation, translation). Real lenses add distortion that must be estimated and corrected.

Nothing metric is possible without this. Any claim about real-world size, distance, or position depends on knowing the camera parameters — uncalibrated images support relative reasoning only.

Uses: cv2.calibrateCamera · cv2.undistort

Two cameras a known distance apart produce disparity between corresponding points, which converts directly to depth. Monocular depth estimation instead learns the mapping from a single image using learned priors.

Monocular depth is fundamentally ambiguous — a small near object and a large far one project identically — so learned models produce plausible relative depth rather than metric truth.

Uses: cv2.StereoSGBM_create · transformers DPT

Epipolar geometry constrains where a point in one view can appear in another, encoded in the fundamental and essential matrices. Structure from Motion recovers both 3D structure and camera poses from an unordered image collection; SLAM does it online.

Bundle adjustment — jointly refining all points and all poses to minimise reprojection error — is the optimisation at the centre of both.

Uses: COLMAP · pycolmap · cv2.findEssentialMat

NeRF represents a scene as a neural field mapping position and viewing direction to colour and density, rendered by volumetric ray marching. 3D Gaussian splatting instead represents the scene as explicit anisotropic Gaussians and rasterises them.

Splatting largely displaced NeRF for practical use by making rendering real-time while keeping quality, trading an implicit representation for an explicit, editable one.

Uses: nerfstudio · gsplat

Learning representations without labels. Contrastive methods (SimCLR, MoCo) pull augmented views of the same image together and push different images apart; masked autoencoders reconstruct hidden patches; DINO uses self-distillation.

The payoff is that labels become needed only for fine-tuning, and DINO features show emergent segmentation without ever being trained on masks — evidence the representation captures real structure.

Uses: lightly · transformers Dinov2Model

Traditional detectors recognise a fixed label set. Open-vocabulary models align visual features with text embeddings so any phrase becomes a query. Segment Anything is promptable — a point, box, or mask yields a segmentation without class labels at all.

This separates segmentation from recognition: SAM finds the boundary, a language model decides what the thing is, and the two compose without retraining either.

Uses: segment_anything · transformers OwlViT

Adding time. Approaches range from frame-wise CNNs with temporal pooling, through 3D convolutions and two-stream networks using optical flow, to video transformers with spatiotemporal attention.

The binding constraint is cost: attention over every frame patch is quadratic in sequence length, so nearly every practical system is a compromise on sampling rate and resolution.

Uses: decord · torchvision.io.read_video · transformers VideoMAE

VAEs encode to a probabilistic latent and decode, giving stable training but blurry samples. GANs pit generator against discriminator, giving sharp samples but unstable training and mode collapse. Normalising flows are exactly invertible but architecturally constrained.

Diffusion won because it trades a single hard problem for many easy ones: instead of learning to generate an image in one step, learn to remove a little noise — a stable regression objective, repeated.

Uses: diffusers · torch

Gaussian noise is added over many timesteps until the image becomes pure noise. This process is fixed, requires no learning, and has a closed form — you can jump to any timestep directly rather than simulating step by step.

That closed form is what makes training practical: sample a random timestep, noise the image once, and train on it.

Uses: diffusers.DDPMScheduler.add_noise

Generation runs the process backwards: start from noise and repeatedly remove a little, guided by a network trained to predict the noise present at each step.

The training objective is remarkably plain — predict the noise you added, measured by mean squared error. Nearly all of the apparent complexity in diffusion sits in sampling, not in training.

Uses: diffusers.UNet2DConditionModel

The schedule sets how much noise is added at each timestep — linear, cosine, or learned. It determines how the difficulty of the denoising task is distributed across the trajectory.

Cosine schedules improved results by not destroying the image too quickly at the start, leaving more of the trajectory spent on the steps that actually carry structure.

Uses: DDPMScheduler(beta_schedule=...)

Usually a U-Net — encoder, decoder, skip connections — conditioned on the timestep so one network handles every noise level. Attention layers at lower resolutions supply global coherence. Newer systems increasingly use transformers (DiT) instead.

Timestep conditioning is what allows a single set of weights to serve as a thousand different denoisers.

Uses: UNet2DConditionModel · DiTTransformer2DModel

DDPM sampling is stochastic and originally needed a thousand steps. DDIM reformulates it as a deterministic process that can skip steps, producing comparable images in twenty to fifty.

Determinism also gives reproducibility and meaningful latent interpolation — the same seed yields the same image, and paths between seeds become traversable.

Uses: DDIMScheduler · DDPMScheduler

Train the model both with and without conditioning, then at sampling time extrapolate away from the unconditional prediction toward the conditional one. A guidance scale controls how far.

This single trick is most of why text-to-image works as well as it does. The scale is a direct prompt-adherence against diversity dial: too low and the prompt is ignored, too high and images become oversaturated and stereotyped.

Uses: guidance_scale in any diffusers pipeline

Rather than diffusing in pixel space, compress the image with an autoencoder and diffuse in that much smaller latent space, decoding at the end. This is what Stable Diffusion is.

The saving is enormous — a 512×512 image becomes a 64×64 latent, roughly sixty-four times fewer positions — and it is the reason high-resolution generation became feasible on consumer hardware.

Uses: AutoencoderKL · StableDiffusionPipeline

A text encoder (originally CLIP, later T5-family) embeds the prompt, and cross-attention layers inside the denoiser let each spatial position attend to text tokens.

Text encoder quality is a major determinant of prompt fidelity — several generational jumps in prompt following came from changing the text encoder rather than the diffusion model.

Uses: CLIPTextModel · T5EncoderModel

Sampling is numerical integration of a differential equation, so better solvers need fewer evaluations. Euler, Heun, and DPM-Solver++ trade steps against fidelity.

This is why step count is a user-facing setting: it is literally the number of function evaluations, and the useful range depends on which solver is running.

Uses: DPMSolverMultistepScheduler · EulerDiscreteScheduler

The continuous-time view: diffusion is a stochastic differential equation, and the model learns the score — the gradient of log-density — which points toward higher-probability regions.

This unifies the discrete-step and continuous formulations, and it is the framework in which most modern sampler and distillation work is derived.

Uses: ScoreSdeVeScheduler

ControlNet adds a trainable copy of the encoder that accepts a structural input — edges, depth, pose, segmentation — and injects it into the frozen base model through zero-initialised connections.

Zero initialisation is the trick: the added branch starts as an identity operation, so the base model is not damaged at the beginning of training. It converts an uncontrollable generator into a controllable one without retraining it.

Uses: ControlNetModel · controlnet_aux

Teaching a model a new subject or style from a handful of images. Textual inversion learns a new embedding only; DreamBooth fine-tunes the model against a rare token with a preservation loss; LoRA trains small low-rank adapters.

LoRA dominates in practice because adapters are a few megabytes, compose with each other, and swap at inference — one base model can serve hundreds of styles.

Uses: peft · pipe.load_lora_weights

All three start the reverse process from something other than pure noise. img2img noises an existing image partway and denoises back; inpainting restricts changes to a mask; outpainting extends beyond the original border.

The denoising strength setting is just where in the trajectory you begin — high strength discards more of the original, low strength preserves composition and alters only surface detail.

Uses: StableDiffusionImg2ImgPipeline · ...InpaintPipeline

Consistency models, latent consistency models, and adversarial distillation compress the sampling trajectory so a student produces in one to four steps what the teacher needed fifty for.

This is what made real-time and interactive generation possible. The cost is usually some diversity and fine detail — distilled models are noticeably more deterministic.

Uses: LCMScheduler · AutoPipelineForText2Image

Instead of a noising process, learn a velocity field that transports noise to data along near-straight paths. Training is a simple regression onto that velocity.

Straighter trajectories are cheaper to integrate, so fewer steps are needed by construction. Several recent large image and video generators are built on this rather than classical diffusion.

Uses: FlowMatchEulerDiscreteScheduler

Extending generation across time, using temporal attention or 3D convolutions, often initialised from an image model and fine-tuned on video.

Temporal consistency is the hard part: independently generating frames yields flicker, and the compute cost scales with frame count, which is why most systems generate short clips at limited resolution.

Uses: StableVideoDiffusionPipeline · export_to_video

FID compares feature distributions of real and generated sets; CLIP score measures prompt alignment; human preference remains the reference standard.

All the automatic metrics are weak. FID is sensitive to sample count and preprocessing and rewards distribution matching rather than quality, and a model can score well while producing images nobody likes — which is why leaderboards in this area are dominated by human-preference arenas.

Uses: torchmetrics FID · clip_score

Train an image encoder and a text encoder so that matching image–text pairs land close together in a shared embedding space, and mismatched pairs land far apart.

Once that space exists, classification becomes retrieval: embed the image, embed candidate label phrases, and take the nearest. This is the mechanism behind zero-shot vision.

Uses: open_clip · transformers CLIPModel

CLIP trains on hundreds of millions of image–caption pairs with a contrastive objective over a large batch: each image should match its own caption against all the others in the batch as negatives.

Batch size is a first-class hyperparameter here, because it determines how many negatives each example sees. CLIP embeddings became infrastructure — used for retrieval, filtering, guidance, and as the vision encoder inside later VLMs.

Uses: open_clip_torch · CLIPProcessor

The dominant modern architecture: a pretrained vision encoder produces patch features, a projection layer maps them into the language model's embedding space, and the LLM consumes them as if they were tokens.

Its appeal is economy. Both large components are pretrained and often frozen; only the small projector must be trained, so a capable VLM can be built without training either half from scratch.

Uses: transformers AutoModelForVision2Seq

The two canonical tasks — describe this image, and answer a question about it. Both are now generative rather than classification tasks.

Their evaluation is genuinely unresolved. n-gram metrics like CIDEr reward matching reference phrasing, so a correct description written differently scores badly, which is why benchmarks increasingly use multiple-choice or model judges.

Uses: BlipForConditionalGeneration · AutoProcessor

BLIP-2 inserts a lightweight querying transformer between a frozen vision encoder and a frozen LLM. A small set of learned query tokens attends to image features and outputs a fixed-length summary for the language model.

It solves the bridging problem economically: a fixed number of query outputs regardless of image resolution, so the language model's context cost stays bounded.

Uses: Blip2ForConditionalGeneration

LLaVA showed that a simple linear or MLP projection plus instruction-tuning data is enough for strong conversational visual ability, with training data generated by prompting a text-only model with image annotations.

The lesson was that data mattered more than architecture — a minimal connector with good instruction data outperformed more elaborate designs, and this recipe became the default.

Uses: LlavaForConditionalGeneration

Flamingo interleaves gated cross-attention layers inside a frozen language model, letting text tokens attend to image features at multiple depths rather than only at the input.

It handles interleaved image–text sequences naturally and supports few-shot prompting with images, at the cost of modifying the language model's internals rather than only its input.

Uses: IdeficsForVisionText2Text

Connecting language to specific image regions — "the mug behind the laptop" must resolve to coordinates, not just a description. Models output boxes or masks alongside text.

This is what separates a model that describes a scene from one that can act in it, and it is the capability robotics and UI-automation work depends on.

Uses: OwlViTForObjectDetection · GroundingDINO

Reading text in images — signage, receipts, forms, charts, screenshots — and reasoning over layout as well as content.

It is disproportionately resolution-sensitive: small text is destroyed by the downsampling most vision encoders apply, which is why document-capable models invest heavily in high-resolution handling.

Uses: pytesseract · transformers TrOCR · docTR

Describing objects, attributes, or relationships that are not present. Object hallucination is measurable with benchmarks like POPE, which simply ask whether a given object appears.

The causes are structural: language priors overwhelm weak visual evidence, so the model completes a plausible scene rather than reporting the observed one. Mitigations include stronger visual grounding, contrastive decoding, and training on negatives.

Uses: POPE benchmark · lmms-eval

Vision encoders were trained at modest fixed resolutions, but real tasks need detail. Solutions include tiling the image into crops, dynamic aspect-ratio handling, and multi-scale encoding.

Every approach trades against context: more visual tokens means better detail and fewer tokens left for reasoning, and a single high-resolution image can consume thousands of tokens.

Uses: AutoProcessor(size=...) · dynamic tiling

Reducing how many tokens an image occupies through pruning uninformative patches, merging similar ones, or learned resamplers.

An interesting extreme is optical compression, where text is rendered as an image because visual tokens can carry it more densely than text tokens — which inverts the usual assumption about which modality is expensive.

Uses: transformers resamplers · Perceiver

Extending to video means choosing frames, since encoding every frame is infeasible. Strategies include uniform sampling, keyframe selection, and temporal pooling of features.

Sampling is where most video VLMs actually fail: an event lasting half a second is simply absent if it falls between sampled frames, and no amount of reasoning recovers it.

Uses: decord · VideoLlavaForConditionalGeneration

Answering questions about relative position, distance, size, and occlusion. Standard VLMs are notably weak here, since 2-D patch features encode little explicit geometry.

Recent work augments VLMs with reconstructed 3D structure from monocular video, moving from models that describe a scene toward models that hold a geometric representation of it.

Benchmarks such as MMMU, MMBench, and MathVista probe reasoning rather than recognition, usually as multiple choice for automatic scoring.

Two caveats. Multiple choice is gameable — some models score above chance from text alone, without the image. And contamination is pervasive, since benchmark images and questions are on the public web and end up in pretraining data.

Uses: lmms-eval · MMMU

More than one modality — text, image, audio, video, and increasingly action — handled by one model. Some accept several inputs but emit only text; genuinely multimodal systems both understand and generate across modalities.

The distinction matters when reading claims: many "multimodal" models are multimodal in and text-only out.

Transformers consume sequences of tokens, so every modality must become one. Images become patches, audio becomes spectrogram frames or learned acoustic codes, video becomes spatiotemporal patches.

Discrete tokenisers such as VQ-VAE make generation as straightforward as next-token prediction, which is why any-to-any models tend to adopt them.

Uses: torchaudio · transformers AutoFeatureExtractor

Early fusion combines modalities at the input so all layers see everything. Late fusion encodes separately and merges near the output. Intermediate fusion interleaves cross-attention at several depths.

Early fusion allows the richest interaction and costs the most; late fusion is cheap and modular but can only combine already-formed representations.

Uses: torch.nn.MultiheadAttention

Training on mixed-modality data from the start, rather than attaching a vision encoder to a finished language model.

It produces better cross-modal integration because representations co-develop, at the cost of far more expensive training and the inability to reuse an existing language model.

Uses: transformers · accelerate

Training on documents where images and text alternate naturally — web pages, papers, tutorials — rather than isolated caption pairs.

This is what teaches a model that an image can refer to text before it and be referred to by text after it. It is also the source of in-context multimodal few-shot ability.

Uses: datasets · webdataset

Either train modality-specific encoders that project into a common space, or push a single architecture to handle raw inputs from several modalities directly.

Specialised encoders exploit known structure and reuse pretrained weights; shared architectures are simpler and scale better, and the field has been drifting toward them as data and compute grow.

The mechanism that lets one modality condition another — text tokens attending to image patches, audio attending to text.

Where and how often it appears is the main architectural decision in multimodal design, and it determines both integration quality and compute cost.

Uses: torch.nn.MultiheadAttention

One model that both understands and generates across modalities — text to image, image to audio, video to text — usually by unifying everything into discrete tokens and predicting them autoregressively.

It removes the pipeline of separate specialist models and allows genuine cross-modal composition, though per-modality quality has generally trailed dedicated systems.

Uses: transformers · VQ tokenizers

When one modality is easier or more abundant, the model leans on it and underuses the others — a VQA model that answers from the question alone, ignoring the image.

It inflates benchmark scores while the model is not actually looking. The diagnostic is simple and underused: withhold a modality and see whether performance drops. If it barely moves, it was not being used.

Uses: ablation harness · lmms-eval

Reasoning over hours of video, long documents with figures, or extended interaction histories, where token counts run to hundreds of thousands.

Compression is unavoidable at this scale, and the open question is what to discard. Uniform sampling is simple and loses precisely the brief events that often matter most.

Uses: decord · frame samplers

Vision-language-action models add robot actions as an output modality, mapping observations and instructions directly to control commands.

Action differs from other modalities in a way that matters: it is closed-loop and consequential. Errors change the world and compound, so there is no equivalent of regenerating a bad caption.

Uses: lerobot · transformers

Learned models that predict how a scene evolves, allowing an agent to simulate outcomes before acting. Increasingly used inside training loops rather than only for evaluation.

Evaluating them is unresolved: visual fidelity is easy to measure and largely irrelevant, while what matters is whether the predicted dynamics support good decisions.

Safety work developed for text does not transfer cleanly. Instructions can be embedded in an image, harmful content can be split across modalities, and refusal behaviour trained on text often fails when the same request arrives visually.

The general lesson is that every added modality is an added attack surface, and guardrails have to be evaluated per modality and per combination rather than assumed to generalise.

An encoder compresses input to a low-dimensional code, a decoder reconstructs the input from it, and the loss is reconstruction error against the input itself. No labels are involved, which makes it the simplest form of representation learning.

The bottleneck is the entire mechanism. Given enough latent capacity the network learns the identity function and generalises nothing — so the constraint, not the architecture, is what forces useful structure to be discovered.

Uses: torch.nn.Sequential · torch.nn.MSELoss

Denoising autoencoders corrupt the input and ask for the clean version, which prevents identity-copying and produces features robust to exactly the corruption used. Sparse autoencoders penalise mean activation so only a few units fire per input, yielding parts-based features — and are now central to interpretability work on language models.

Contractive autoencoders penalise the Jacobian of the encoding with respect to the input, so the code changes little under small input perturbations. All three are the same idea: the useful autoencoder is the constrained one.

Uses: torch.autograd.functional.jacobian · torch.nn.L1Loss

Pixel MSE is the default and the reason autoencoder outputs blur — under uncertainty the loss-minimising output is the average of plausible reconstructions. L1 blurs slightly less. Perceptual loss compares activations of a pretrained network rather than pixels, so outputs match in feature space and look sharper.

LPIPS is the standard learned perceptual metric. Combining pixel loss with perceptual loss, and sometimes an adversarial term, is what production autoencoders actually use — including the VAE inside Stable Diffusion.

Uses: lpips · torchmetrics.image.LearnedPerceptualImagePatchSimilarity

Train only on normal data; anything the autoencoder reconstructs badly is anomalous. Attractive because it needs no labelled defects, which is usually the situation in inspection and monitoring.

Its weakness is that a sufficiently powerful autoencoder reconstructs anomalies too. Practical systems constrain capacity, use memory banks of normal features, or compare features from a pretrained network rather than pixels.

Uses: anomalib · torch

A VAE encodes to a distribution rather than a point: the encoder outputs a mean and log-variance, a sample is drawn, and the decoder reconstructs from that sample. This is what makes the latent space continuous and sampleable — a plain autoencoder's latent space has holes, so decoding a random point produces nothing meaningful.

Training maximises the evidence lower bound, which is reconstruction likelihood minus the KL divergence between the encoder's posterior and a standard normal prior. The KL term is the regulariser that pulls the latent space into a usable shape.

Uses: torch.distributions.Normal · torch.distributions.kl_divergence

Sampling is not differentiable, so gradients cannot flow through a random draw. Writing the sample as z = mu + sigma * eps with eps drawn from a fixed standard normal moves the randomness outside the computation graph, leaving mu and sigma differentiable.

It is a small algebraic move with an outsized consequence — it is what made variational inference trainable by backpropagation, and it is the single most commonly asked VAE question.

Uses: torch.randn_like

If the KL term dominates, the encoder outputs the prior regardless of input and the decoder learns to ignore the latent entirely, producing the same average image every time. This is posterior collapse, and it is the characteristic VAE failure.

Mitigations: KL annealing that ramps the weight up during training, free bits that exempt a minimum KL per dimension from the penalty, and weakening the decoder so it cannot succeed alone. Beta-VAE turns the weight into a deliberate knob — higher beta trades reconstruction quality for more disentangled factors.

Uses: torch · custom loss weighting schedules

A generator maps noise to images; a discriminator classifies real against generated. The discriminator's gradient tells the generator how to become more convincing. At the theoretical optimum the generator matches the data distribution and the discriminator is reduced to guessing.

The original minimax loss saturates when the discriminator wins early, giving the generator no gradient — so in practice the non-saturating form is used, maximising log D(G(z)) rather than minimising log(1 - D(G(z))).

Uses: torch.nn.BCEWithLogitsLoss

When real and generated distributions barely overlap, the Jensen-Shannon objective gives vanishing gradients. WGAN approximates Earth Mover's distance instead, which stays informative — but requires the critic to be Lipschitz-constrained.

Weight clipping enforced that crudely and damaged capacity; WGAN-GP replaced it with a penalty on the gradient norm at interpolated points. Hinge loss is the common modern default, used in BigGAN and StyleGAN-adjacent work, and is simpler than either.

Uses: torch.autograd.grad · torch.nn.functional.relu

DCGAN established the recipe that made GAN training reproducible: strided convolutions instead of pooling, transposed convolutions for upsampling, batch normalisation in both networks, ReLU in the generator with tanh output, LeakyReLU in the discriminator, and no fully-connected layers.

Much of it has since been revised — transposed convolution is now usually replaced by upsample-then-convolve to avoid checkerboard artefacts — but it remains the baseline any GAN discussion starts from.

Uses: torch.nn.ConvTranspose2d · torch.nn.BatchNorm2d · torch.nn.LeakyReLU

Mode collapse is the generator discovering a handful of outputs that reliably fool the discriminator and abandoning the rest of the distribution. It looks like impressive individual samples with almost no diversity across a batch — which is why you always inspect a grid, never a single image.

Causes trace to the generator optimising against a discriminator that has not seen enough variety. Mitigations: minibatch discrimination so the discriminator sees batch statistics, WGAN-GP's better-behaved objective, two-timescale update rules with different learning rates, and spectral normalisation on the discriminator.

Uses: torch.nn.utils.spectral_norm

A conditional GAN feeds a label or embedding to both networks so generation can be steered. pix2pix does paired image-to-image translation with a U-Net generator, an L1 term for structural fidelity, and a PatchGAN discriminator that judges overlapping local patches rather than the whole image — cheaper and sharper on texture.

CycleGAN removes the need for paired data with a cycle-consistency loss: translating to the other domain and back should return the original. It is the standard answer for style transfer when no aligned dataset exists, and its known failure is hiding information imperceptibly in the output to satisfy the cycle.

Uses: torch · kornia · pix2pix / CycleGAN reference implementations

There is no held-out likelihood to report, so evaluation is indirect. Inception Score rewards confident and diverse classifier predictions but never looks at the real data, so it cannot detect a model that generates convincing images from the wrong distribution.

Fréchet Inception Distance fits Gaussians to Inception features of real and generated sets and measures the distance between them — lower is better, and it is the standard. Its caveats matter: it is biased by sample count, depends on the feature extractor, and is blind to memorisation of training images. Precision and recall for generative models separate fidelity from coverage, which FID conflates into one number.

Uses: torchmetrics.image.fid.FrechetInceptionDistance · torchmetrics.image.inception · clean-fid

StyleGAN's key move is a mapping network from the sampled z to an intermediate w space, which need not be Gaussian and so can un-warp the entangled factors that z forces together. w then modulates each layer through AdaIN, giving scale-specific control — coarse layers govern pose and shape, fine layers govern colour and texture.

StyleGAN2 traced the characteristic droplet artefacts to AdaIN's normalisation and replaced it with weight demodulation. The extended W+ space, with a separate vector per layer, is what GAN inversion and editing methods actually operate in.

Uses: stylegan2-ada-pytorch · torch

Editing a real photograph with a GAN requires first finding the latent code that reproduces it — inversion, done by optimisation, by a trained encoder, or by a hybrid of the two. There is a persistent tension between reconstruction accuracy and editability: codes that reconstruct perfectly often sit outside the well-behaved region of the latent space and edit badly.

Once inverted, editing means moving along a direction in latent space. Those directions are found supervised with attribute classifiers, or unsupervised through PCA of sampled codes (GANSpace) and closed-form factorisation (SeFa).

Uses: stylegan2-ada-pytorch · lpips · encoder4editing

With a few thousand images the discriminator memorises the training set, stops providing useful gradient, and the generator degenerates. Naive augmentation makes it worse — the generator learns to reproduce the augmentation artefacts.

Adaptive discriminator augmentation applies augmentations to both real and fake inputs with a probability tuned automatically from the discriminator's overfitting signal, and the augmentations remain invisible in the generator's output. It made high-quality GAN training on small datasets practical.

Uses: stylegan2-ada-pytorch · kornia.augmentation

Instead of a continuous code, the encoder output is snapped to the nearest entry in a learned codebook. The gradient is passed through the non-differentiable lookup with a straight-through estimator, and a commitment loss keeps encoder outputs near their chosen codes.

Discretisation sidesteps posterior collapse and, more importantly, turns an image into a sequence of tokens — which is what lets an autoregressive transformer model images the way it models text. This is the mechanism behind DALL·E 1 and behind image tokenisation in multimodal LLMs.

Uses: vector-quantize-pytorch · torch

Diffusion surpassed GANs on sample quality, diversity, and controllability for text-to-image, and largely displaced them there. But a GAN generates in one forward pass while diffusion needs tens — so on latency-bound and on-device work the trade-off is not settled.

Super-resolution, face restoration, real-time style transfer, and interactive editing still commonly use GANs or GAN-distilled diffusion models. The honest position is that diffusion is the default for quality and GANs for speed, with distillation steadily narrowing the gap.

Uses: diffusers · basicsr · gfpgan

A rig is a hierarchy of joints. Skinning binds each mesh vertex to one or more joints with weights, so moving a joint deforms the surface — linear blend skinning is the standard, with its familiar collapsing artefact at strongly bent joints.

Facial animation normally uses blendshapes instead: a set of sculpted target expressions that are linearly combined by coefficients. This is the interface most ML-driven animation targets, because predicting fifty-odd coefficients per frame is a tractable regression problem.

Uses: trimesh · smplx · ARKit blendshape coefficients

The short-time Fourier transform slices a waveform into overlapping windows and takes the frequency content of each, producing a time-frequency image. Mapping frequency onto the mel scale — which is roughly logarithmic, matching human pitch perception — gives the mel spectrogram that nearly all audio models consume.

The consequence worth internalising: once audio is a mel spectrogram it is a 2D array, so convolutional and transformer architectures from vision apply unchanged. Window size sets the trade-off between time and frequency resolution, and that choice matters more than most architecture decisions.

Uses: torchaudio.transforms.MelSpectrogram · librosa.feature.melspectrogram

Locating body or hand joints in the image plane. Top-down pipelines detect a person then find joints inside the crop, which is accurate but scales with the number of people; bottom-up pipelines find all joints then group them into individuals, which is constant-time but harder.

Heatmap regression — predicting a spatial probability map per joint rather than coordinates directly — is the standard trick, because a spatial map is a far easier target to learn than raw numbers.

Uses: mediapipe · mmpose · ultralytics (pose models)

Point clouds are unordered sets, so networks over them must be permutation-invariant — PointNet's shared MLP plus max-pooling was the founding solution. Voxels are regular and convolvable but cost memory cubically. Meshes render efficiently but are awkward to predict. Implicit representations — signed distance fields, occupancy networks — are continuous and resolution-free but need querying.

The representation choice constrains everything downstream, and picking one for a stated use case is a standard interview question rather than a detail.

Uses: open3d · pytorch3d · trimesh

Lifting 2D pose to 3D, or regressing the parameters of a body model such as SMPL directly. Parametric models help because they constrain output to anatomically plausible bodies rather than arbitrary joint positions.

The persistent problems are temporal jitter, foot sliding, and depth ambiguity from a single view. Jitter is not fixed by a better per-frame model — it is fixed by predicting over a temporal window or filtering the sequence, because per-frame independence is the cause.

Uses: smplx · mmpose · pytorch3d

Any per-frame prediction driving animation needs smoothing or it reads as noise. A Kalman filter models position and velocity and is the classical answer. The One Euro filter is the practical favourite for interactive work because it adapts its cutoff to speed — heavy smoothing when still, light smoothing when moving fast.

The trade-off is always smoothing against latency: every filter that removes jitter also delays response, and in an interactive system users feel the delay.

Uses: filterpy · One Euro filter implementations

wav2vec 2.0 learns from raw audio by masking latent speech units and solving a contrastive task over quantised targets; HuBERT uses offline clustering to provide masked-prediction targets. Both give representations that fine-tune to strong recognition with a fraction of the labelled data.

Whisper took the opposite route — large-scale weakly-supervised training on diverse audio — and is the practical default for transcription because it degrades gracefully on accents, noise, and code-switching.

Uses: transformers (Wav2Vec2, HuBERT, Whisper) · torchaudio.pipelines

Mapping speech to mouth motion, either through explicit phoneme-to-viseme rules or by regressing blendshape coefficients from learned audio features. Learned approaches capture coarticulation — the way a sound's mouth shape is influenced by its neighbours — which rule-based mappings miss.

Wav2Lip and SadTalker are the reference points for video-space generation. The evaluation problem is that lip sync quality is perceptual: humans detect misalignment of a few tens of milliseconds, so any metric that ignores timing is measuring the wrong thing.

Uses: transformers · torchaudio · Wav2Lip / SadTalker reference implementations

Depth arrives from stereo matching on a calibrated pair, from structure-from-motion across many views, from monocular estimators such as MiDaS and Depth Anything, or from active sensors — LiDAR and time-of-flight cameras.

The distinction that matters: monocular depth is relative unless anchored by known scale or sensor data. Treating a monocular depth map as metric is a common and consequential mistake in AR work.

Uses: transformers (Depth Anything) · open3d · cv2.StereoSGBM

NeRF represents a scene as a function from 3D position and viewing direction to colour and density, rendered by marching rays and integrating. It produces striking view synthesis but was originally slow to both train and render, and it bakes lighting into the representation.

3D Gaussian Splatting instead represents the scene as millions of anisotropic Gaussians rasterised directly to the screen, reaching real-time frame rates at comparable quality. For interactive and AR applications it has largely displaced NeRF, which is a rendering-pipeline argument rather than a modelling one.

Uses: nerfstudio · gsplat · pytorch3d

Pose tracking through visual-inertial odometry, fusing camera features with IMU because each covers the other's weakness — the IMU drifts but is fast, vision is stable but slow and fails on blur. Then plane detection and scene meshing, anchoring virtual content into that reconstruction, occlusion so virtual objects can pass behind real ones, and lighting estimation so they are shaded consistently.

Occlusion is the piece that most often looks wrong, and it needs either a depth sensor or a per-frame segmentation mask for people — which is a real-time vision model competing for the same compute as everything else.

Uses: ARKit · RealityKit · open3d

Sixty frames per second leaves sixteen milliseconds for capture, preprocessing, inference, postprocessing, and rendering combined — and the renderer wants the GPU too. On a handheld device this sits under a thermal ceiling, so sustained performance is lower than a benchmark run suggests.

The practical consequences: measure p95 rather than mean, measure after thermal soak rather than cold, budget end to end rather than model-only, and treat model choice as subordinate to the budget. Most accuracy arguments in interactive vision are settled by latency before they are settled by metrics.

Uses: coremltools · onnxruntime · Instruments / Metal System Trace

Video diffusion extends image diffusion with temporal layers or full spatiotemporal attention, and the hard constraint is consistency — a model that generates each frame well but independently produces flicker that is immediately obvious.

Approaches include conditioning on previous frames, latent interpolation along a trajectory, and explicit motion conditioning from pose or depth sequences. Evaluation remains unresolved: per-frame quality metrics reward exactly the models that flicker.

Uses: diffusers (video pipelines) · transformers