Hello everyone!
We're very excited to announce this new feature - Test Suites!
Admittedly, this has been a very long overdue feature. Many of you have been asking for better ways to group tests together and get an aggregated report for your tests, with some breakdown of failure rates across browsers. It makes us proud and terrified at the same time when folks show us their UIlicious Monitoring dashboard filled with nearly hundreds of scheduled jobs running across different environments. We knew that we need a proper way to organize our tests into suites, rather than chaining tests together with the TEST.run command in a main "test suite" file. While TEST.run command lets you reuse small snippets of test in larger test flows, or import utility functions, that has its own limitations when you try to use it to group entire end-to-end flows to run them together as one big test suite. Once you chain more than a dozen tests, test reports start to become really really long and hard to read. And these long chains of tests make it very likely for the test run to hit the hard 1 hour test execution time cutoff.
Now, let me introduce the new Test Suites feature! We have a lot planned for this feature, and this will be the first of many updates we're looking forward to.
You can now configure a single job to run a suite of tests:
The scheduled job will run all the configured test files across all configured browsers at the scheduled time. Don't worry if you've configured more tests than the number of concurrencies available in your space, the tests will automatically be queued if there are not enough concurrencies at the scheduled time, and will be running as soon as a concurrency becomes available when a test completes.
You can view an aggregated overview of the tests that were run together as a suite, which looks like this:
You'll also get a breakdown of the percentage of tests failing on each browser.
Email and webhook error notifications (to Slack, Microsoft teams, etc) will still fire for each individual test run. We do have plans in the future to allow you to configure whether you want to receive a single notification on the completion of the full test suite or to receive individual notification on each test run.
And now, on our Monitoring dashboard, instead of being overwhelmed with the status of hundreds of tests, we can get a glance of what modules (or you could make test suites grouped by priority) are generally failing using test suites:
Bonus stuff - more job scheduling options:
Many of you have also requested more options to configure the job schedule. For example, to run every 2 weeks on Wednesdays, or maybe every 30 days on any day of the week. Now you can do so!
What's next:
You might be thinking, "Sweet! But what about running test suites manually, or via the CLI?". For now, a test suite can only be run as a scheduled job. However, we do have plans on the roadmap to incorporate the ability to configure and run a test suite in the Studio Editor (perhaps you can select a folder and click "Run" to run all tests in the folder) and via the command-line interface. We're also thinking of adding priority levels so that you can set specific tests within the suite to execute as soon as possible.
"Nice, so when will we get all these upcoming features?" Well, tell us what matters to you most by giving the features you like in our public roadmap a thumbs up, and we'll make it happen sooner. 🙂
What do you think? We love to hear your feedback!
How can we make UIlicious work better for your team's workflow?
Let us know on Twitter, or on the community forum, or ping us at [email protected]. :-)
Happy testing!
~ the UIlicious team
Let's build a better web
Cover image: Credits to https://unsplash.com/@historyhd
