What I Learnt Writing my First Test Suite

— a Reflection

Abstract

I recently wrote a test suite for a basic web application, which resulted in two different types of bugs surfacing… After spending a week avoiding working on the test suite. The experience of writing that first test suite has also lead me to a simple model of how testing works. Finally I give a brief summary of assorted learning nuggets.

Backdrop

A friend and I are working on a basic web application (using Ruby on Rails) which we hope will carve out a small income stream, and it fell on me to write (the majority of) our test suite. My friend got the task of doing the CSS styling, though, so I drew the tallest straw there. Anyways, as a CS bachelor student, I haven’t needed to write any test suites before. Certainly, I have used a few test suites to verify acceptability of programming assignments, and I have heard about both test-after and TDD. Even written a couple of perfunctory tests for a group assignment. But I haven’t had a real use for them. Not before now.

The crucial aspect of the app we are working on is that it has to handle semi-daily reports over arbitrary time periods. Since it’s intended as a production app, it would be fairly annoying to have obvious bugs in the main feature. But manually testing time-tracking features would require undue amounts of arbitrary time… Automated tests to the rescue — not because of significant complexity, but because of significant time aspects that make manual testing impractical. So, instead of spending weeks on manual tests, I got to spend weeks procrastinating instead.

Whoops.

Writing the Test Suite

Actually, I started out fairly well, writing several tests the first couple of days. But then I found quite a number of conceivable excuses to avoid continuing the work and spent one and a half week doing who-knows-what instead. Turned out, I found writing tests an annoying task. 1) I had to wrestle with the rails minitest system, learning it during use (and found that some of the app code wasn’t available from the test files I had imagined). 2) The time spent working on tests felt like it was being wasted rather than pushing the project forwards – it was the Nth pass over code we had already written, after all.

Eventually, my urge to finish the task, move on, do other things and stop procrastinating kicked in and I got to work on the test suite again, finishing it after a couple of days. (As mentioned in the backdrop: a basic web application.) But, what do you think I found as I wrote the tests? Yup. That’s right: bugs. Primarily of two different categories: bugs in the application, and bugs in the correspondence between my friend’s mental model of the application and mine.

The bugs in the application would have messed up the usability, but were generally fairly easy to fix. Off-by-one errors, some conditional errors, a few function returns that were off. If we were to launch with those bugs still present, it would have been an embarrassing situation. But I think the second category of bugs was more important: finding places where the code behaved in ways I didn’t expect, but which, when I brought them up, lead to my friend arguing for the behaviour being desirable. Sometimes I agreed, sometimes I disagreed. But locating those differences in how we expected the system to work and discussing them meant that we could continue work on one system, rather than accidentally working on two-different-systems-in-one.

Reflection

The test suite I wrote is not the most perfect test suite in the history of Ruby on Rails test suites, utilizing conventions perfectly, breaking them as needed, and proven to cover every single case. Instead, it’s more of a hodgepodge of “oh, so this is where-how I can test that piece of code” and the set of test cases my beginner mind managed to cough up. Which is hopefully the majority of the relevant paths for the core features, but I would be surprised if we’ve squashed every single bug living inside our code.

We didn’t use TDD, instead writing the code first and testing (much) later. It seems to me, with my limited experience, that TDD works to make development heavier through creating “scaffolding” around the code, encoding expectations before having done the exploratory coding necessary to learn what you actually want to do. Though, perhaps that makes TDD perfectly suited to use-cases where you know the expectations in advance, like implementing some algorithm or small function.

In retrospect, writing the test suite has given me an appreciation for tests and the increased confidence they bring. It feels good to know that a hopefully-representative subset of all possible paths behave appropriately. Surprisingly, writing tests for someone else’s code was particularly useful, as it forced us to recalibrate our mental models and discuss aspects of the system. Not just aspects of the system as in how it was currently coded, but aspects of the system in how we wanted it to behave as well.

Finally, the act of writing a test suite has given me a basic model of testing that helps me make sense of the different aspects of a testing system… Unit tests, integration tests, system test. They all boil down to something simple – their main difference is just which level of the system they test. Here is my model for automated testing:

In summary:

• Some of the particulars of the Ruby on Rails minitest system.
• Some of the generals of different levels of testing (unit, integration, system…).
• The ease with which tiny bugs sneak into code.
• Time aspects make automated test suites desirable.
• The ease with which mental model mismatches sneak into the team’s heads.
• What feels like progress and what is progress doesn’t necessarily correspond.
• I have a significant problem with procrastination.
• Testing, at base, is simply doing something and checking the results.