When writing automated tests for a website relying heavily on a template language, how do I gain confidence while maintaining fast feedback?

The heart of your question really is about optimizing code. While there are a few general tricks, your first step is profiling your test code. Only after identifying the steps in each test that perform the worst, can you crunch numbers and determine what is worth optimizing. I'll get to the general advice, but first a short story about optimizing tests, so you don't waste the kind of time that I did.

A Hard Lesson About Optimizing Tests

(which you can skip if you've already learned this lesson)

One of my projects has an extensive test suite using Selenium. I'm not advocating that everyone should have 4,000 tests that open a web browser and perform a use case, but that's what this project has. Our full test suite was taking about 12 hours to run. Suddenly, one day, it was 10:00 AM and the test suite was still running, despite having started the tests at 5:00 PM the previous night.

I spent time looking at how we were initializing our ORM database sessions, connection sharing, how often we open and close the browser window. Nothing seemed to account for the 5+ hours of additional test execution time. The time saved optimizing things that (I guessed) were slow, saved 90 minutes or so. Not bad, but certainly didn't explain why our tests were taking 5 hours more than normal. Until I profiled our tests, and discovered we were traversing a temp directory recursively to delete file uploads before each test.

Deleting these temp files used to take mere milliseconds. Fast-forward 6 months, and I discovered it was taking 7 seconds per test to delete these temp files, because each day we added a new sub folder in the form of year/month/day. The increasing number of sub folders killed performance, even though 98% of our tests did not upload files. Just doing some naïve math, 7 seconds x 4,000 tests is 28,000 seconds — just spent deleting non-existent file uploads! (Imagine me beating my head against a wall at this point) That's nearly 7 hours and 45 minutes of test execution time over the course of the whole test run. And there was my 5+ hours of additional test execution time.

So, I added a step in our build process before executing tests to recursively delete all files and folders in this temp folder (akin to rm -rf /uploads/temp). Sure 98% of our tests didn't upload files, so we don't need to delete temp files before every test, but now at least deleting non-existent files takes mere milliseconds again.

So, lesson learned: always, always, always, always, always, always, always, always, always, always, always, always, always, always, always, always profile your application or tests before wasting time optimizing things. And why this is a lesson I continually need to learn is another question indeed.

Ok, so enough anecdotes. Let's assume you have profiled your tests and eliminated the low hanging fruit, performance-wise. You still have 1,000+ tests to execute, and they simply take time. Too much time for a quick feedback loop during development.

General Advice For Optimizing Slow UI Tests

• Execute tests in parallel. This isn't always possible and depends on the setup for your tests and application environment. I've done this before. Now instead of 1,000 tests running, you need to wait for the equivalent of 500 tests to run since maybe you can run 2 simultaneously.

• Install an SSD drive. Yup. The old "throw more hardware at the problem" trick. When tests become I/O bound, make I/O faster.

  • Or a faster CPU or more RAM.

• Don't rely on slow UI tests during development. You need tests that execute faster, and the UI, along with a server and database, can only execute so fast. This requires structuring your code for unit tests so you don't need a web server or database. You can mock any calls to external resources. This is pretty tricky for UI components, but not impossible. It is a lot of work, though, so the increase in test speed should be enough to offset the additional work to write unit tests for UI code.

Unfortunately the only true solution to your problem is "Don't rely on slow UI tests during development." You will need to structure your UI components so they can be isolated from the rest of the page, and then write unit tests for them. They still might execute asynchronously, but they shouldn't be relying on real servers or databases. All calls should be mocked so the tests execute quicker.

Otherwise, you are stuck with slow UI tests during development.

The standard solution for dealing with huge test times (or huge build times) is to run the majority of tests on a separate machine, a dedicated test server (or build server, or CI server), automatically after each checkin into your repository.

Of course, since you still want parts of your tests to be executed immediately when doing TDD, this will only work when you split your tests suit into different testing modules which correspond to the modules of your application. So when you work on a module X, run the subset of tests which correspond to module X locally (and maybe some other tests you always want to be executed). The majority of tests, however, should be executed automatically in the background, and when they fail, the test server (or build server, or CI server) sends you a message or email about the failure.

And yes, this will sacrifice "ultra fast feedback" to some degree, but the feedback should be fast enough for most practical purposes. How well this works depends a lot how well you can structure your test suit into parts which can be executed locally and quickly, and other parts where a certain time lag for the feedback is acceptable.