Whenever I consume art, its so-called “flaws” are what I enjoy the most. From the car hitting the camera during the chase in Bullitt to the vertex snapping of a PSX game. However, there’s always a pursuit for ridding work of its blemishes, with Disney altering movies after their release and video games pursuing ever-greater realism. This is not to say that I’m against making edits to old movies or improving graphical realism, as much as I mourn the loss of the flaws of imperfect techniques.

For myself, creative coding for generating images and text took up much of my time in high school and undergrad. For me, the art of a generation system included the peculiarities of its design and implementation, not just its output.

At present, some might say I should use existing generative AI, but that feels boring, unfulfilling, and uninspired. It’s not that I’m anti-AI in general, as neurosymbolic methods that integrate symbolic programs with learned models are very interesting. Without any symbolic hand-written portion, it eliminates a fundamental source of enjoyment for me.

Since starting my PhD, I’ve been so consumed with work that I haven’t done much creative coding, so I’ve decided to write this series of articles, “Putting Pixels Together”, as a source of motivation. We’ll implement various techniques to build a toolbox for generating, editing, and combining images. These techniques vary widely in their sources, from academic papers to game development talks to my own ideas. More importantly, they will also vary in their effectiveness and age.

A bit of a roadmap

To try and hold myself to some kind of plan, I lay out the overarching goals here.

As a programming languages (PL) person, I can’t resist the opportunity to design a domain-specific language (DSL) for generating and manipulating images.

We will then write an interpreter in Rust, which lets us easily compile our interpreter into WebAssembly, which I’ll use to embed interactive examples into the articles.

In each article, we’ll develop more primitives for our DSL, covering procedural textures (such as Perlin noise), texture synthesis (such as Wave Function Collapse), and filters/modifiers (such as Kuwahara filter/seam carving).

Finally (or incrementally), we’ll build a node graph tool to provide a nice interactive way to construct programs in our DSL.

In summary, we will:

  • Design a domain-specific language (DSL) for image manipulation.
  • Write an interpreter in Rust
    • Compile it to WebAssembly for interactive examples.
  • Explore various techniques for generating, editing, and combining images.
  • Build a node graph tool for an intuitive way to create programs in the DSL.

See you in the next article, where we’ll design our DSL!