SIGGRAPH Asia 2026

SketchFlow

Zero-Shot Vector Sketch Generation via GMM Prior Flow in CLIP Latent Space

Jin Zhou, Hongliang Yang, Pengfei Xu, Hui Huang
Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, CSSE, Shenzhen University
Paper arXiv DOI Code Model
SketchFlow generations of letters, numbers, Malaysian landmarks, characters, emotions, and poses
Local zero-shot vector sketch generation by SketchFlow. SketchFlow leverages drawing priors from the CLIP latent space to synthesize prompts beyond its 345-class QuickDraw training vocabulary. The left shows text-prompted letters, numbers, and landmarks in Malaysia; the right shows pop-culture characters, creatures, abstract emotions, and dynamic poses. These prompts have no category-specific training examples.

Abstract

Vector sketches are a concise medium for abstract human expression, yet high-quality text-to-sketch pairs are scarce. SketchFlow formulates cross-modal alignment as a continuous mapping problem in CLIP latent space. It expands discrete category text embeddings into a continuous Gaussian Mixture Model prior, then learns an Optimal Transport Conditional Flow Matching vector field that transports this prior to the rendered-sketch feature distribution. A hybrid diffusion decoder combining a 1D U-Net and Transformer decodes the transported features into vector stroke trajectories. Trained from category-only QuickDraw sketches, SketchFlow supports fast local zero-shot synthesis for concepts and semantic modifiers beyond the QuickDraw training vocabulary, together with continuous concept interpolation.

Method

SketchFlow method overview
Overview of the SketchFlow generative framework. Noise injection expands discrete text anchors into a continuous GMM prior. OT-CFM learns a deterministic transport to the rendered-sketch feature distribution. At inference, an unseen CLIP query follows the learned field to a target sketch embedding, which the Hybrid Diffusion Decoder converts into a vector stroke sequence.

01

Continuous GMM Prior

Noise around category text embeddings expands isolated semantic anchors into a continuous source distribution.

02

OT-CFM Transport

A learned vector field transports the GMM prior toward the rendered-sketch manifold in CLIP space.

03

Vector Decoding

A hybrid 1D U-Net and Transformer diffusion decoder generates ordered 256-point stroke trajectories.

Zero-Shot Results Beyond the Training Vocabulary

SketchFlow vector sketches for prompts outside the QuickDraw training vocabulary
Zero-shot generation beyond the training vocabulary. All prompts shown here lie outside the QuickDraw taxonomy, yet the model produces coherent and style-consistent vector sketches. Examples include Pikachu, Tetromino, and Goblin, together with the composite and plural prompts Sun & cloud and Two eyes.

Drawing Process

Each animation follows the generated vector trajectory from its first point to the completed sketch.

SIGGRAPH

8 generated trajectories
Generated letter S Generated letter I Generated letter G Generated letter G Generated letter R Generated letter A Generated letter P Generated letter H

ASIA 2026

8 generated trajectories
Generated letter A Generated letter S Generated letter I Generated letter A Generated digit 2 Generated digit 0 Generated digit 2 Generated digit 6

Malaysia Landmarks

6 generated trajectories
Generated Malaysia landmark 1 Generated Malaysia landmark 2 Generated Malaysia landmark 3 Generated Malaysia landmark 4 Generated Malaysia landmark 5 Generated Malaysia landmark 6

Scope and Limitations

SketchFlow is trained from the 345 discrete QuickDraw categories. It is not a general image model and does not reliably parse long, compositional prompts. Its strongest zero-shot behavior appears on concise, visually distinctive concepts that CLIP represents clearly, such as Kirby, Mickey, ghosts, rockets, emotions, symbols, and landmarks. Generation remains stochastic, so sampling multiple candidates can be useful.

BibTeX

@inproceedings{zhou2026sketchflow,
  title     = {SketchFlow: Zero-Shot Vector Sketch Generation
               via GMM Prior Flow in CLIP Latent Space},
  author    = {Zhou, Jin and Yang, Hongliang and
               Xu, Pengfei and Huang, Hui},
  booktitle = {ACM SIGGRAPH Asia 2026 Conference Papers},
  year      = {2026},
  doi       = {10.1145/3829340.3842307},
  eprint    = {2608.21659},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}