APPA

In 2022 I started building a pipeline to outpaint Avatar: The Last Airbender from 4:3 to 16:9, taking a show that aired in the pre-widescreen era and inventing the missing side regions using classical CV and ML. I got partway there before the scope outran my time budget. This is a retrospective on the design, what worked, what didn’t, and how I’d build it today. I’ll properly revisit this project again, when time allows.

premise

Outpainting on natural video is hard. Every new pixel you invent has to look plausible against arbitrary, textured, motion-blurred content. Animation is different in ways that actually help.

ATLA is 2D cel work: characters animate on top of largely static or slow-panning background plates. That structure is a gift. When the camera pans across a scene, the right edge of frame 1 frequently contains content that should appear in the right-outpainted region of frame 10. A meaningful fraction of “what to outpaint” isn’t hallucination at all. It’s recovery of real source content that was visible a few frames earlier or later.

That insight shaped the entire architecture.

2022 architecture

Raw episode (4:3 MP4)
        │
        ▼
[TransNetV2]  ──►  309 shot-bounded scenes (S01E01)
        │
        ▼
[RAFT optical flow]  ──►  dense per-pair flow fields
        │
        ▼
[MiVOS-STCN interactive masks]  ──►  per-scene object masks
        │
        ▼
[FGT, flow-guided transformer]  ──►  video extrapolation (4:3 → 16:9)

The picks, briefly:

• TransNetV2 for shot detection. Cuts cause temporal discontinuity, so the outpainter has to be run per-shot.
• RAFT after benchmarking against Farneback, PWCNet, and SPyNet. Animation breaks every flow method in a different way (flat color fields with no texture to anchor on, hard outlines that flow algorithms hate, and hold frames where the same drawing is held for 2-3 frames so motion is intermittently zero), but RAFT held up best.
• FGT (Flow-Guided Transformer, ECCV 2022) for the outpainting core. The architectural premise, using optical flow as guidance for spatial attention rather than as hard supervision, degrades gracefully when flow is noisy. For an animation domain with no ground truth, that mattered.

what got built

• S01E01 → 309 shots via TransNetV2 ✓
• Frame extraction, padding, train/val split ✓
• Optical flow on the pilot episode. RAFT looked plausible to the eye, though without ground truth there was no way to validate numerically.
• An auxiliary Optuna search to tune Farneback’s hyperparameters against a flow-consistency objective. Cute idea, but the objective (std_dev + smoothness) was a biased proxy that ended up rewarding over-smoothed flow.
• 4 of 309 scenes worth of interactive object masks via MiVOS-STCN.

what stalled it

Two things: one practical, one a comprehension failure.

The mask pipeline didn’t scale. MiVOS-STCN’s interactive workflow requires scribbling on a keyframe per scene, watching the propagation, fixing failures, and saving. Per-scene cost was roughly 5 to 15 minutes of focused attention. At 309 scenes that’s a several-day annotation slog, and that’s just the pilot. The full series is 61 episodes, so call it ~18,000 scenes end-to-end. I got 4 scenes in before the friction won.

The bigger one: I’d misread FGT’s API. FGT has three modes: object removal, watermark removal, video extrapolation. The README emphasizes object removal, and that mode needs hand-annotated masks per frame. Extrapolation mode, the one I actually wanted, doesn’t. The “mask” for outpainting is just the new strip outside the original frame, generated internally by the model. The entire interactive masking detour was solving a problem I didn’t have.

This is a recurring failure mode when stitching together research code: same repo, different mode, completely different data requirements. Easy to read the headline example, assume that’s the canonical interface, and build infrastructure you don’t need.

2026 rebuild

The field has moved. If I revisited APPA today, the architecture would look almost nothing like the original.

1. Shot segmentation and classification. TransNetV2 still works, but I’d add a classifier on top to bin each shot as static / horizontal-pan / tracking / action-cut based on the mean optical flow vector. Different shot types take different paths through the pipeline.

2. Multi-frame mosaicing (classical CV, no ML). For pan and tracking shots with mostly-static backgrounds, a large fraction of ATLA, stitch frames into a wider panoramic background using SIFT/ORB + RANSAC homography + multi-band blending. This recovers real source content with zero hallucination. Probably handles 30 to 50% of shots outright.

3. Foreground separation. SAM 2 or Cutie to extract characters automatically. Single click per scene, model tracks through the shot. This is what kills the old MiVOS bottleneck: minutes of scribbling becomes a click.

4. Video diffusion outpainter. Wan 2.x or CogVideoX with a style LoRA fine-tuned on ~10k ATLA stills. This replaces FGT entirely. Temporal consistency is handled natively by the U-Net’s temporal attention, so the “flow-guided” trick FGT needed is built in to the architecture.

5. Composite the original 4:3 region back over the outpainted 16:9 result. Source pixels are never touched by the model. Only the new side regions are generated. Edge feathering and per-shot histogram alignment clean the seams.

The mental shift from 2022 to 2026 is simple: don’t train a custom model from scratch. Orchestrate pretrained ones, and use classical CV where the structure of the problem allows it. The animation-specific insight, that backgrounds reveal themselves over time through camera motion, is the part that doesn’t age.

takeaways

• Stitching research repos together is mostly an exercise in reading docs carefully and writing glue. The misread on FGT’s modes cost more time than any actual algorithmic problem.
• Bottlenecks in CV pipelines are almost always the human-in-the-loop steps, not the GPU steps. The fix is to swap them out for automation the moment the automation is good enough.
• Without ground truth, hyperparameter search optimizes whatever proxy you pick. And your proxy will be biased in ways that aren’t obvious until you stare at outputs.
• The right architecture in 2022 was wrong by 2024. The right architecture in 2026 will probably be wrong by 2028. Picking a research direction is partly a bet on what stays useful.