Why we migrated opensource 😼Inboxkitten (77 million serverless request) from 🔥Firebase to ☁️ Cloudflare workers & 🐑 CommonsHost

Get Mail Nyow! 📩

Since our original launch of inboxkitten.com, a free opensource serverless project for creating a disposable email service...

The 🐈 kittens have multiplied and gotten out of hand...

• 3rd on product hunt

• February bill shock of $143

  • 221 GB of static website data (this is from Cloudflare cache miss)

  • 215 GB of email data

  • 77 Million API requests

  • 20k+ Unique Visitors (according to Cloudflare)

  • Depletion of my steam summer sale funds 😭 (that $143 got to come from somewhere).

Credit: Samantha Whitten

Ok cat team, we need a plan to protect the summer sale! 💳

One of the key limitations of existing AWS lambda or GCP/Firebase cloud functions is that it limits 1 request to an application instance at any point in time.

While that is awesome, if you are compressing images, or doing complex stuff.

Inboxkitten API's only role is to make HTTP requests to Mailgun, which holds the actual email, with the API key, and echo the result back.

As a result, our resource consumption is well below 1% of a CPU, or <10MB of ram per request. Well, below is the minimum of 128MB ram and dedicated CPU per request.

So our original plan was that once these kittens grow in a very constant load, would be to toss it onto a $5/month Linode Nanode 1GB instance.

However, the downside is that it wouldn't "scale" automatically at peak workload or if it ever goes to a gazillion requests a month, which would be beyond one instance. (Not that we would ever need to support such workload)

So in keeping to the spirit of serverless fun (which made this project popular), we would put that option aside and ask...

What if, there was a serverless platform, with an alternative billing model. What if it bills only per request, or only for the amount of CPU and ram used. What if we...

Use Cloudflare workers ☁️

Cloudflare worker, which is part of the growing trend of "edge" serverless computing. Also happens to simplify serverless billing down to a single metric.

Turning what was previously convoluted, and hard to decipher bills from GCP...

Like seriously, trying to guess how much your API will cost for a million requests in advance is kinda impossible on GCP / AWS, without historical data

Into something much easily understood, and cheaper overall per request...

Note that there is a minimum of $5/mth

😸 That's enough net savings for 7 more $9.99 games during the summer sale!

And has a bonus benefit of lower latency edge computing!

But, there are catches...

1) <5ms CPU time limit

Each request is limited to < 5ms of CPU time, this differs greatly for the request time (also known as wall clock time), in that it only counts the amount of time the CPU spent on our function itself and ignores all its sleep/wait time.

This is as opposed to how GCP or AWS measures the time taken from start to end of the function, including all sleep/wait time.

In such a setup, serverless functions fold DNA strands or cat images, with large CPU or RAM consumption. Will find Cloudflare unusable.

However, our serverless functions spend 99.99% of the time waiting for Mailgun API to respond. Making the <5ms limit purrfect.

Also if we ever need to upgrade this to <10ms, it is an optional addon.

2) Incompatibility with express.js (as it uses web workers pattern)

Another thing to note is that Cloudflare workers are based on the web workers model, where it hooks onto the "fetch" event in Cloudflare like an interceptor function.

So instead of the following in express JS (as we currently do for firebase)

const express = require('express')
const app = express()

app.get('/', (req, res) => res.send('Hello World!'))

const port = 3000
app.listen(port, () => console.log(`Example app listening on port ${port}!`))

In Cloudflare, it would be the following.

addEventListener('fetch', event => {
  event.respondWith(fetchAndApply(event.request))
})

async function fetchAndApply(request) {
  return new Response('hello world')
}

As this is a huge fundamental change in the code structure. It can be a major no-go for larger existing projects, due to the sheer amount of rewrite work involved. Despite the extremely close similarity.

Perhaps one day someone will come up with a Cloudflare to express.js adapter?

But for us, due to the simplicity of the project, it would be just a simple rewrite. You can compare the difference in the code between the express js version here and the Cloudflare version here on GitHub.

And while workers are an open standard, with Cloudflare being the only major provider, it is currently also a form of vendor lock-in.

3) One script limitation per domain (unless you are on enterprise)

While not a deal-breaker for Inboxkitten, this can be one for many commercial / production workloads.

Because Cloudflare serverless packages cannot be broken down into smaller packages for individual subdomains and/or URI routes.

This greatly complicates things, making it impossible to have testing and production code separated on a single domain, among many more complicated setups.

However in our use case, as this is a hobby project.... it doesn't matter...

The scratches in summary...

1. <5ms CPU time limit

2. Incompatible with express.js

3. one script limitation per domain

And all it took was a quick one-day rewrite, by @jmtiong, and we are done.

That's cool, but Why do I even need Inboxkitten disposable email again?

One of the key use cases currently, and why we built this project, is to perform email validations as part of our automated test scripts. Such as the following...

// Lets goto inbox kitten
I.goTo("https://inboxkitten.com");
I.see("Open-Source Disposable Email");

// Go to a random inbox inbox 
I.fill("email", SAMPLE.id(22));
I.click("Get Mail Nyow!");

// Check that its empty
I.see("There for no messages for this kitten :(");

// Testing for regular email
// (sent using a jenkins perodic build)
I.goTo("https://inboxkitten.com");
I.see("Open-Source Disposable Email");
I.fill("email", "ik-reciever-f7s1g28");
I.click("Get Mail Nyow!");

// See an email we expect, nyow
I.see("Testing inboxkitten subject");

With sharable test results such as...

Plus it's simple, cool, and fun to tinker around with.

Hey, how about the $31 firebase static file hosting?

The easiest free solution would be to throw the entire website onto Github pages.

However in the spirit of opensource...

We are gonna do a shout out for our friends from 🐑 commonshost.com, an open-source static site hosting platform, being built out of Singapore 🇸🇬

And help them push their network with a real production workload test across their 22+ servers on its global network.

As to why commons host instead of GitHub... cause its cool, and I want to support the underdog of the CDN world, being an underdog of the testing world.

Oops, did I say underdog? I mean undercat 😼

What's next?

Due to the project's rather unique simplicity (quick to rewrite), and its heavy production load. I am thinking of potentially expanding its deployment option to as many serverless options as possible, or Docker-based deployments even.

Exploring the various trade-offs with actual 24/7 production load.

Done

• GCP/Firebase: function and hosting

• Cloudflare workers

• Commonshost hosting

Todo

• Docker container deployment

  • ECS Fargate

  • Digital Ocean Kubernetes

• AWS lambda

• Others??

Let's see where this cat ship will sail next... Till then we will be taking a cat nap.

Happy shipping!