Taking the horror out of UI testing 😱

The problem isn't the broken tests. We just have very broken tools for UI testing.

UI testing sucks. It really does.

If you aren't already familiar with automating end-to-end tests yet, there are a few well-known free and open-source frameworks out there, in the order of Github stars: NightmareJS (16K), Selenium (12K), WebDriverIO (4K), CodeceptJS (1K).

Tests typically look more or less like this - take a minute to figure out what this "Hello World" example from NightmareJS does 🤔:

const Nightmare = require('nightmare')
const nightmare = Nightmare({ show: true })

nightmare
  .goto('https://duckduckgo.com')
  .type('#search_form_input_homepage', 'github nightmare')
  .click('#search_button_homepage')
  .wait('#r1-0 a.result__a')
  .evaluate(() => document.querySelector('#r1-0 a.result__a').href)
  .end()
  .then(console.log)
  .catch(error => {
    console.error('Search failed:', error)
  })

Have you figured it out?

What this does is to go to DuckDuckGo, enter "GitHub nightmare" into the search box, press the search button, wait for the first result to show up, and print the link address of the first result.

Come on people, I thought we already know that hard-coding stuff and using magic waits is a no-no. Test code is still code, and this code smells 💩. This makes things hard to read, and harder to maintain. What if the product changes the design or the front-end decides to do just a bit of spring cleaning? Damn, the tests broke. Ain't nobody got time to fix those hundred and one bloody CSS selectors!

And, what are we really trying to test anyway what are we really trying to test anyway?

The user journey or the HTML?

How about writing tests like this?

I.goTo("https://duckduckgo.com")
I.fill("Search", "Github nightmare")
I.pressEnter()
I.see("Github - segmentio/nightmare")
I.click("Github - segmentio/nightmare")

Concise. Readable. Maintainable.

And front-end agnostic. VueJS, ReactJS, Angular... does it matter?

@picocreator and I have been working building web applications since pre-jQuery times, we both have accumulated our own 2 AM horror stories in trying to make sure stuff gets tested and shipped on time or having things blow up in our face by testing in production 💣💣💣😱😱😱. We tell junior devs these horror stories every year on Halloween night. OK, getting a little side-tracked, anyway...

We disagree a lot on software architecture and often debate what maintainable code looks like, but one thing we agree is that the problem isn't the broken tests. We just have very broken tools for UI testing. Someone needs to fix it. And this is what we've dedicated our past two years working on:

Piece of cake.

But this test is too simple. You are probably thinking, yeah that's nice, but what if things get more complicated, like when there's 50 "Add to cart" buttons, or what about icon buttons?

Let's have some fun, shall we? 😎

Oh wait, and before we start, just so you know, this absolutely isn't a black box algorithm Powered by AI™, but more on that later.

Testing Dev.To

Let's start with the basics, and make sure that the one of the most critical feature - Search - works.

I.goTo("https://dev.to/")
I.fill("Search", "dev.to")
I.pressEnter()
I.click("thepracticaldev")
I.see("The hardworking team behind dev.to ") // mmhm, very hardworking indeed.

The nice thing about tests being decoupled from the implementation of the UI is that we can easily reuse the same test for testing responsive designs. Let's see if search works as expected on desktop and on mobile view.

Now, let's try clicking on DEV.to's logo to get back home. UIlicious scans for accessibility attributes and tooltips set with title and other similar attributes used by various popular frameworks. Does our home logo have something we can use?

<a href="/" class="logo-link" id="logo-link" aria-label="DEV Home"><svg ... /></a>

Oh look, this is what DEV.to's logo looks like under the hood. There's an aria-label, fantastic! Let's click on "Dev Home".

I.click("DEV Home") // We love aria-labels
I.amAt("https://dev.to/")

There we go:

Ok, let's get creative and do a bit of shopping at the Dev Shop. I'm just gonna grab a hundred of these Sticker Pack and Dev totes.

I.click("DEV Shop")
I.amAt("https://shop.dev.to/")

let shopping_list = [
  "Dev tote",
  "Sticker Pack"
]
shopping_list.forEach((item) => {	
  I.click("The DEV shop")
  I.click(item)
  I.fill("Quantity", 100) // lets' get a hundred of each
  I.click("Add to cart")
})

OK... almost done. No wait, let's just grab a few more totes. Hmm... There are a few rows of items on the card, we need to pick the correct quantity box to update. No sweat, I just need to be a little specific and tell UIlicious what I.see before updating the quantity.

I.amAt("/cart")
I.see("Dev tote") 
I.fill("Quantity", 120) // UIlicious will pick the quantity box for "Dev Tote" to fill
I.pressEnter()

And to top it off, let's just do some test-ception, just to make sure UIlicious itself works.

Yea baby. Piece of cake. 😎

Under the hood

No, it's not Powered by AI™. Not in the modern sense at least.

Warning, opinion™ ahead! Tests should be deterministic, meaning they should always produce the same result given the same input. Random unpredictable behavior isn't exactly desirable in tests, and fixing defects in an AI-driven test engine involves... throwing more "correct" sample data at it to make it more accurate.

UIlicioous works by methodologically reverse engineering intent (What you mean by I.click("Sign in") from your HTML) and what the previous steps were. It works best on accessible websites. You code doesn't have to be perfect to be testable, but it certainly helps to use semantic HTML and ARIA attributes.

(And by the way, the UIlicious IDE is completely built using VueJS. \o/)

Making testing great... for the person fixing the bug.

I think the most annoying part of getting bug reports is when they are incomplete and I need to chase after the reporter for steps to replicate the bug. While at the same time, to be honest, I get lazy reporting bugs too. That's why we try to make bug replication reports as complete and actionable (and pretty!) as possible. 👇

When should you automate UI testing?

A good guideline is: When you are testing that login flow for that user role for the n-th time.

And also 👇

Should you automate unit tests or integration tests or end-to-end tests first? Doesn't matter, just start somewhere. I usually recommend starting with unit tests for anything requiring complex conditionals and math, and end-to-end tests for critical user flows because these can also help catch errors downstream too.

Pros and cons?

• Pro: It starts at $0. And it's one less thing for DevOps to worry about.

• Con: It's not open-source, yet. (... until money falls from the sky)

Is the cake real?

Yes. It's not a lie, we always have cake where ever we go.

And Happy Halloween folks!👻👻👻