Pixel bending with butter and crook

Going pixel deep is one of the most fascinating aspects of drawing on canvas. It's also the kind of job that lends itself nicely to using typed arrays and as such offloading data processing tasks onto a web worker. A whole range of Photoshop filters may be recreated in JavaScript this way.

And while chroma keying, blur, or thresholding are well documented and somewhat less than challenging to put together, here's the core particulars for doing displacement mapping and pixel sorting, a couple of effects I found myself eager to port, because existing solutions looked either incomplete, or bloated and out of date.

Butter for pixel sorting

In the world of data moshing, pixel sorting is king. Looking for pointers on Github, I hit upon butter.js, a very literal, almost word for word take on Kim Asendorf's often cited Processing script, a bit woolly for my taste, good enough to be branching off of still.

Some of the features I sought to improve in my fork: Too many options, too much repetition, significantly though accepting ImageData for input/output. But in the end, all I kept was the name. The main idea is to sort() each row or column of pixels in an image if they happen to fall above or below a given threshold color, along the lines of,

// Adapted from
// github.com/thewhodidthis/butter

// Note, cutoff in hex includes alpha channel,
// assuming RGBA, but may be inverted for BE systems
const threshold = 'ffffffff'

// In the context of an `Uint32Array` decked input, the two
// extremes would be: BLACK = 0, WHITE = 4294967295
const limit = parseInt(threshold.match(/.{1,2}/g).reverse().join(''), 16)

// Rotate a quarter of a circle before and
// after processing for sorting in the vertical dimension
function applySorting(pixel, i, line) {
  // Sort approved
  const bang = phase * pixel > limit

  // For tracking matches
  this.mark = 0

  // Match found, take note
  if (!this.mark && bang) {
    this.mark = i
  }

  // Time's up
  if (!bang && this.mark) {
    // Assuming typed array input
    line.subarray(this.mark, i).sort()

    // Reset
    this.mark = 0
  }
}

Crook for pixel shifting

Displacement mapping involves going through the color values in one image (color map) to rearrange the contents of another. In particular, for each pixel in the lookup image the amount of shift produced is proportional to its distance from a baseline gray across separate RGB channels. Essentially,

// Determine color distance above and below mid-gray, snippet from
// github.com/thewhodidthis/crook
function calculateColorShift(color, scale) {
  const ratio = scale * ((color - 128) / 256)

  return Math.sign(ratio) * Math.round(Math.abs(ratio))
}

Furthermore, what happens when displacement values overstep the image boundaries comes into play, with options for allowing transparent gaps, clamping, or wrapping round the edges,

// Clamping, displacement value only goes up to a certain max
const snap = (v, max) => (v >= max ? max - 1 : Math.max(v, 0))

// Wrapping, displacement value inverted past the edges
const wrap = (v, max) => (v >= max || v < 0 ? v + (max * Math.sign(v)) : v)

// Allows for using source pixel when shift out of bounds
const skip = (v, max) => (v >= max || v < 0 ? NaN : v)

Composed

Because both filters accept and return ImageData, they're chainable,

import butter from '@thewhodidthis/butter'
import crook from '@thewhodidthis/crook'

const canvas = document.createElement('canvas')
const target = canvas.getContext('2d')

const buffer = canvas.cloneNode().getContext('2d')
const shadow = canvas.cloneNode().getContext('2d')

const { width: w, height: h } = canvas

// Fill lookup with noise
for (let i = 0; i < w * h; i += 1) {
  const x = i % w
  const y = Math.floor(i / w)
  const c = Math.floor(Math.random() * 255)

  shadow.fillStyle = `rgba(${c}, ${c}, ${c}, ${c})`
  shadow.fillRect(x, y, 1, 1)
}

const sort = butter({
  // Target pixels exceeding dark gray
  threshold: '303030ff'
})

const warp = crook({
  // Use reds for horizontal, blues for vertical displacement
  channel: { x: 0, y: 2 },
  // 0: Transparency, 1: Clamping, 2: Wrapping
  mode: 1,
  // Shift five pixels in each direction
  scale: { x: 5, y: 5 }
})

const master = document.createElement('img')

master.addEventListener('load', () => {
  buffer.drawImage(master, 0, 0)

  const source = buffer.getImageData(0, 0, w, h)
  const lookup = shadow.getImageData(0, 0, w, h)

  // Process in place
  target.putImageData(sort(warp(source, lookup)), 0, 0)
})

master.setAttribute('src', 'path/to/image.png')

Reference

• Canvas based image filtering walkthrough on HTML5 Rocks
• ActionScript's own DisplacementMapFilter
• ASDFPixelSort, Kim Asendorf's original