Turning a single still photograph into an animated, anime-styled scene — by recovering depth the image never recorded, fusing it back with the colour information, and stylising the result. The depth channel is what separates this from a filter.

← All projects Report (PDF) Code

What it does

SAIND takes a static RGB image and produces an animated, stylised version of the same scene. It does this by first inferring structure the photograph does not contain — a dense depth map — then combining that geometry with the original colour image through a fusion step, and finally applying anime-style transfer to the fused representation.

Why — the motivation

Style transfer applied directly to a photograph is a 2-D operation on a 3-D scene. It has no notion of what is near and what is far, so it treats a foreground subject and a distant background as the same kind of surface. The results look like a texture pasted over the image, and they break in the specific ways you would predict: object boundaries smear, background detail receives the same stylistic weight as the subject, and any attempt at motion produces warping rather than parallax.

Animation makes the problem sharper still. Convincing motion in a scene is largely a depth phenomenon — near things move across the field of view faster than far things. Without a depth estimate there is no principled way to produce that, so single-image animation tends to be either a global pan or a warp that visibly distorts.

The motivating claim is therefore: recover the missing dimension first, and both problems become better posed. Depth gives the stylisation a sense of what belongs to which surface, and gives the animation a basis for differential motion.

Intuition behind the approach

The pipeline is built around one idea: depth estimation from a single image is easier if you first tell the model where the edges are.

Monocular depth is fundamentally ambiguous — infinitely many 3-D scenes project to the same 2-D image. What resolves it in practice is structural cues, and discontinuities in depth almost always coincide with edges in the image. An object boundary is simultaneously a colour edge and a depth step. So an explicit edge map is not redundant with the RGB input; it is a prior that tells the depth model exactly where it is permitted to be discontinuous.

That is why the pipeline runs edge detection before depth estimation and feeds both into the depth model, rather than predicting depth from RGB alone. The edge map does not add information about the scene — it makes the information already present easier for the model to use.

The second design idea is in the fusion step. RGB and depth are different kinds of signal: one is appearance, one is geometry. The project’s EnGD algorithm combines them into a joint representation before stylisation, so the style transfer operates on something that carries both — rather than styling appearance and hoping geometry survives.

How it works

The repository documents a three-stage pipeline:

1. Edge detection with LDC. The RGB image is passed through a Learned Dense Connections edge detector to produce an edge map.
2. Depth generation with TokenFusion. A transformer-based image-to-image model takes the RGB image and the edge map and produces a dense depth map. Inputs are standardised to 512 × 512.
3. Fusion and stylisation. The proposed EnGD algorithm fuses the RGB and depth information, and Scenimefy applies anime-style transfer to produce the final animated output.

Intermediate artefacts are written to dedicated directories at each stage — edge maps under LDC/result/MDBD2CLASSIC, depth maps under TokenFusion/results — which makes each stage independently inspectable. For a multi-model pipeline this matters more than it sounds: when the output is wrong, being able to look at the intermediate depth map immediately tells you whether the failure is upstream or downstream.

Architecture

Component notes:

• LDC implements 4-block and 5-block deep architectures for dense edge prediction.
• TokenFusion is a transformer for image-to-image translation whose distinguishing mechanism is token exchange between modalities — which is precisely why it suits an RGB-plus-edge input pair. It is designed to fuse two input streams rather than concatenate them.
• Scenimefy performs semi-supervised anime style transfer built on the CUT (Contrastive Unpaired Translation) framework, trained with both paired and unpaired data under a curriculum learning strategy.

The pipeline is assembled from strong published components with one novel joining step, rather than trained end to end. That is a pragmatic choice: each stage can use a model already trained on far more data than this project could collect, at the cost of no gradient flowing between stages.

Technical specifications

Optimizations and results

The optimisation story in this project is architectural rather than numerical — the design choices are where the work is:

• Edge map as an explicit prior, rather than expecting a depth model to infer boundaries from RGB alone. This is a way of injecting structure instead of demanding the model learn it from data the project does not have.
• Token exchange for fusing modalities, rather than channel concatenation. Concatenating RGB and edge channels asks the network to discover their relationship; a token exchange mechanism is built for cross-modal fusion and gets it more directly.
• A dedicated fusion algorithm before stylisation, so style transfer receives geometry and appearance jointly rather than appearance alone — which is the whole reason for computing depth in the first place.
• Fixed 512 × 512 processing to keep the transformer stage within tractable memory, an explicit resolution-versus-cost trade.

Limitations and honest caveats

• Errors compound across stages. With three separately-trained models in series and no end-to-end gradient, a bad edge map yields a bad depth map yields a bad stylisation — and nothing downstream can recover it.
• Monocular depth remains an estimate. The edge prior improves it but does not make it metric; the geometry is plausible rather than measured.
• The 512 × 512 constraint caps output detail, which matters for scenes whose interest lies in fine texture.

Back to all projects.