- publication
- ACM SIGGRAPH ASIA 2024
- authors
- Alexandre Binninger, Olga Sorkine-Hornung
SD-πXL specializes in creating pixel art, characterized by its intentionally low resolution and limited color palette. Our method enables varying degrees of control: the input is a text prompt, and optionally a reference (high-resolution) image for initialization or spatial control. SD-πXL's output style can be adjusted using fine-tuned diffusion models. In this example, the full prompt reads ''Embroidery of a Chinese dragon flying through the air on a dark background with smoke coming out of its mouth and tail.''. The output pixel art can be used for crafted fabrications, such as the shown cross-stitch embroidery.
abstract
Low-resolution quantized imagery, such as pixel art, is seeing a revival in modern applications ranging from video game graphics to digital design and fabrication, where creativity is often bound by a limited palette of elemental units. Despite their growing popularity, the automated generation of quantized images from raw inputs remains a significant challenge, often necessitating intensive manual input. We introduce SD-πXL, an approach for producing quantized images that employs score distillation sampling in conjunction with a differentiable image generator. Our method enables users to input a prompt and optionally an image for spatial conditioning, set any desired output size H × W, and choose a palette of n colors or elements. Each color corresponds to a distinct class for our generator, which operates on an H × W × n tensor. We adopt a softmax approach, computing a convex sum of elements, thus rendering the process differentiable and amenable to backpropagation. We show that employing Gumbel-softmax reparameterization allows for crisp pixel art effects. Unique to our method is the ability to transform input images into low-resolution, quantized versions while retaining their key semantic features. Our experiments validate SD-πXL's performance in creating visually pleasing and faithful representations, consistently outperforming the current state-of-the-art. Furthermore, we showcase SD-πXL's practical utility in fabrication through its applications in interlocking brick mosaic, beading and embroidery design.
downloads
acknowledgments
We thank the anonymous reviewers for their constructive feedback and Danielle Luterbacher for her help with setting up the embroidery machine. Ximing Xing's open-source version of VectorFusion was instrumental in the development and design of our source code. This work was supported in part by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement No. 101003104, ERC CoG MYCLOTH).