✨ New feature update: Get ready to Suit(e) up!

By Tai Shi Ling | March 24, 2021

Hello everyone!

We're very excited to announce this new feature - Test Suites!

Test suites lets you group related tests and run them together.

Admittedly, this has been a very long overdue feature. Many of you have been asking for better ways to group tests together and get an aggregated report for your tests, with some breakdown of failure rates across browsers. It makes us proud and terrified at the same time when folks show us their UIlicious Monitoring dashboard filled with nearly hundreds of scheduled jobs running across different environments. We knew that we need a proper way to organize our tests into suites, rather than chaining tests together with the TEST.run command in a main "test suite" file. While TEST.run command lets you reuse small snippets of test in larger test flows, or import utility functions, that has its own limitations when you try to use it to group entire end-to-end flows to run them together as one big test suite. Once you chain more than a dozen tests, test reports start to become really really long and hard to read. And these long chains of tests make it very likely for the test run to hit the hard 1 hour test execution time cutoff.

Now, let me introduce the new Test Suites feature! We have a lot planned for this feature, and this will be the first of many updates we're looking forward to.

You can now configure a single job to run a suite of tests:

Now you can schedule a job to run a suite of tests.

The scheduled job will run all the configured test files across all configured browsers at the scheduled time. Don't worry if you've configured more tests than the number of concurrencies available in your space, the tests will automatically be queued if there are not enough concurrencies at the scheduled time, and will be running as soon as a concurrency becomes available when a test completes.

You can view an aggregated overview of the tests that were run together as a suite, which looks like this:

View the results of the test suite in an aggregate report, with breakdown for each browser.

You'll also get a breakdown of the percentage of tests failing on each browser.

Email and webhook error notifications (to Slack, Microsoft teams, etc) will still fire for each individual test run. We do have plans in the future to allow you to configure whether you want to receive a single notification on the completion of the full test suite or to receive individual notification on each test run.

And now, on our Monitoring dashboard, instead of being overwhelmed with the status of hundreds of tests, we can get a glance of what modules (or you could make test suites grouped by priority) are generally failing using test suites:

Our monitoring dashboard feels less cluttered and easier to analyse at a glance broadly which modules are failing.

Bonus stuff - more job scheduling options:

Many of you have also requested more options to configure the job schedule. For example, to run every 2 weeks on Wednesdays, or maybe every 30 days on any day of the week. Now you can do so!

More flexible options for scheduling jobs

What's next:

You might be thinking, "Sweet! But what about running test suites manually, or via the CLI?". For now, a test suite can only be run as a scheduled job. However, we do have plans on the roadmap to incorporate the ability to configure and run a test suite in the Studio Editor (perhaps you can select a folder and click "Run" to run all tests in the folder) and via the command-line interface. We're also thinking of adding priority levels so that you can set specific tests within the suite to execute as soon as possible.

"Nice, so when will we get all these upcoming features?" Well, tell us what matters to you most by giving the features you like in our public roadmap a thumbs up, and we'll make it happen sooner. 🙂

What do you think? We love to hear your feedback!

How can we make UIlicious work better for your team's workflow?

Let us know on Twitter, or on the community forum, or ping us at [email protected]. :-)

Happy testing!

~ the UIlicious team

Let's build a better web


Cover image: Credits to https://unsplash.com/@historyhd

About Tai Shi Ling

Cofounder CEO of UIlicious. I've been coding for over a decade. Building delightful user interfaces is my favorite part of building software. I named the product UIlicious because I wanted folks to build UI that was delicious. Corny, I know.