I remember seeing a post on twitter of someone showing off an image with an old LED TV effect applied, and I wanted to achieve something similar to that on my own. After a few hours of research and coding I managed to achieve this effect, but there are a few caveats which I will explain later.

I was pretty familiar with Python's image processing library, Pillow after using it extensively in one of my projects, but since we're dealing with Javscript, that would not work. After a few rounds of google searches, I landed on sharp, an image processing library with all the tools that I needed to make this project work. I also used d4c/numjs, which is an alternative to numpy for my array reshaping needs.

The idea was pretty simple, first we have to take a look at how old LED TVs work. Each pixel can be broken down to three separate components, each representing red, green and blue. Back then LEDs were pretty limited; they could only light up in a singular color, and the only thing that could be changed is the intensity of each LED. When you take a closer look at an old LED screen, you could see three singular LEDs for each pixel.

Mixing these three LEDs at different intensities allows for different colors to be displayed, a la what we use in css rgb(red, green, blue). Using sharp, I convert an image file into a pixel data array, with every three values in the array representing red, green, and blue intensity values [r, g, b, r, g, b, ...], and plugged each of these values into a function that converts them into the three LEDs surrounded by black borders to mimic the old TV look.

Turns out that dealing with huge arrays causes a lot of memory issues, that if not dealt with can cause the Nest server to crash, even locally on my M2 Macbook Pro with 16GB memory. So a few steps had to be taken to avoid server crashes, especially after it has been deployed. First I had to limit the size of the image that my algorithm runs on. After some tests with max memory size of 1025MB, the sweet spot that I landed on is a max of 80 x 80 pixels image. Second, since each color value on an image goes up to a maximum value of 255, instead of using integer arrays, using Uint8Arrays is much more memory efficient. Javascript has some nifty garbage collection features which I had to utilize, freeing unused memory by assigning null values to variables that have outlived their usefulness. In the end, I landed on the code below:

function generatePixels([red, green, blue]: Uint8Array) {
  const blackPixel = Uint8Array.from([0, 0, 0]);
  const redPixel = Uint8Array.from([red, 0, 0]);
  const greenPixel = Uint8Array.from([0, green, 0]);
  const bluePixel = Uint8Array.from([0, 0, blue]);

  return [...Array(18).keys()].map((rowKey) => {
    if (![0, 17].includes(rowKey)) {
      return [...Array(18).keys()].map((colKey) => {
        if (![0, 17].includes(colKey)) {
          if ([1, 2, 3, 4].includes(colKey)) {
            return redPixel;
          }
          if ([7, 8, 9, 10].includes(colKey)) {
            return greenPixel;
          }
          if ([13, 14, 15, 16].includes(colKey)) {
            return bluePixel;
          }
          return blackPixel;
        } else {
          return blackPixel;
        }
      });
    } else {
      return new Array(18).fill(blackPixel);
    }
  });
}

async ledEffect(data: Buffer): Promise<string> {
    const resizedBuffer = await sharp(data)
      .resize(80, 80, {
        fit: 'inside',
      })
      .toBuffer();

    const { width: imageWidth, height: imageHeight } = await sharp(
      resizedBuffer,
    ).metadata();

    const dataBuffer = await sharp(resizedBuffer)
      .toFormat('jpeg')
      .jpeg({
        quality: 100,
        chromaSubsampling: '4:4:4',
        force: true,
      })
      .flatten()
      .raw()
      .toBuffer();

    let { data: imageData }: { type: 'Buffer'; data: number[] } =
      dataBuffer.toJSON();

    const formattedImagePixels = nj
      .array(imageData)
      .reshape(imageHeight, imageWidth, 3)
      .tolist();

    imageData = null;

    const imageArray: number[] = [];

    (formattedImagePixels as unknown as number[][][]).forEach((pixelRow) => {
      const ledRow = [];
      pixelRow.forEach((pixelCol) => {
        ledRow.push(generatePixels(Uint8Array.from(pixelCol)));
      });
      let indexArray: Uint8Array | null = Uint8Array.from([
        ...Array(18).keys(),
      ]);

      indexArray.forEach((ledPixelRow) => {
        ledRow.forEach((ledCol) => {
          imageArray.push(ledCol[ledPixelRow].flat());
        });
      });

      indexArray = null;
    });

    let tempImageArray = nj.array(imageArray).flatten().tolist();

    const newBuffer = await sharp(
      Uint8Array.from(tempImageArray as unknown as number[]),
      {
        raw: {
          width: imageWidth * 18,
          height: imageHeight * 18,
          channels: 3,
        },
      },
    )
      .toFormat('jpeg')
      .toBuffer();

    tempImageArray = null;

    return `data:image/jpeg;base64,${newBuffer.toString('base64')}`;
  }

There were other ways to improve memory efficiency since sharp supports streams, but I have no experience using them, plus the code that ChatGPT spat out did not work, I decided to abandon that idea.

I learned a lot about how Javascript handles memory, and how simple number arrays can actually cause server crashes. It was a pretty good experience, and the webapp is pretty fun to play around with.