The Unimportance of Tests
Drew Strickland on July 15th, 2015 | About 10 minutes to read.To the chase; you can't throw a rock without hitting a post about how and why and with what software you should be testing yours. In fact, according to my completely unscientific and subjective methodology, the category "testing software" is only rivaled in density and saturation by porn 1. Current analytics predict that by this time next year there will be more tutorials on how to use Karma and Mocha together, than total combined cat gifs on Reddit 2.
In short, tests are a BFD, and everyone is talking about them. Even me. This article is about tests, and how wildly, amazingly, and completely up-its-own-butt our industry is about them right now.
Now that you've come this far, it's fair to warn you, I'm going to make a strong case for why you should ignore tests, and why they do more harm than good, but I will not under any circumstances tell you to do away with them altogether. Tests have their place, and we'll get into that, but they're not the end-all-be-all of software quality.
Also, for the sake of covering my butt, the opinions expressed in this post are my own, and do not necessarily reflect the opinions or practices of other persons or companies.
The Purpose of Tests
Google it. Everyone pretty much agrees on this one.
- Find / Prevent Defects
- Ensure expected behavior hasn't changed
- Make sure the software meets business and system requirements
- To prove quality of code
Wrong. Just damn wrong. All wrong. Underwear-goes-on-the-inside-of-your-pants wrong!
Finding & Preventing Defects
If the tests are meant to identify defects in the code, then your code can only ever be as good as the tests. 90% of the time, those tests are written by the same person writing the code, so the net change in code quality is nullified by the issue of proximity. The person writing the tests and code together is simply not objective enough.
Rarely, the tests are written by a QA engineer, and even more rarely, in the case that tests literally define requirements, by a Product engineer. In those cases your tests are now objective enough to identify defects without a bias towards how a particular developer works, but the quality and completeness of your tests has suffered for it.
Not Changing Behavior
Granted, tests are really good at validating that expected behavior hasn't changed, but how about when it is supposed to? The obvious answer is to update the tests, and that's generally what people do, but this is a metaphorical interpretation of nudging facts to fit theories. The alternate solution is to create new methods, and new helpers, and brand new tests, all of which create the potential for a well-tested method elsewhere to still be using the old method for unknown reasons. Worse still, there is now potential for brand new code to be using the old methods because someone ignored a deprecation.
Both solutions have a really high probability of going wrong when things change, and let's face it, things are going to change, and change again, then change back, then go away for a while, then come back.
The only real world scenarios where having a test validate that the expected behavior of a method hasn't changed are during a refactor, during performance optimizations of a method, or possibly if you are trying to reverse-engineer a code-base from the tests.
Ensure the Software Meets the Requirements
If we look back to the section about finding and preventing defects, you can probably tell where I am going with this. Software engineers writing the tests will use their interpretation of the requirements, QA theirs, and possibly the most accurate, would be tests written by the product person who wrote the requirements. The problem here, is the requirements. The solution? Very clearly defined requirements with a high degree of communication around them.
Communication is a big deal, and it should be. This is the one thing that takes us from "working alone" to "working together". I can't stress that enough. Regardless of the size of the app / org / company / team / whatever, communication is how shit gets done right. End of line.
So, if we re-examine the thing we are trying to fix with tests, you'll notice that getting that done right, and getting the software done right without tests both require the same thing. From that perspective, the tests might as well not even be there.
Proving Quality of Code
This is probably my favorite one.
Tests are meta-code. They describe what we want to happen, run the stuff, and then check the output. Okay, good deal, we have proved that stuff is happening as expected, and that the output looks like what we think it looks like. Awesome. Drinks all around.
Now let's do that for all 1000 classes, across 17 repositories, so that we make sure all our code is covered. I know I'm being sarcastic, but you might not, because this is the metric that gets used. Coverage. What percentage of your code base has possibly crappy meta-code around it to prove that an input of 2 always results in an output of 5 where you think it should be? How many different ways might a specific function be called, and have we made sure that there is meta-code to describe what happens no matter what the input is? If User.OrderDrinks( new Bartender(), 'peanuts') is called, we have a tests that expects an exception right?
Again, if you depend on tests, then your code is only ever as good as your tests. Do you test your tests? Call me crazy for asking this, but, should you? If test coverage is the metric for quality, and tests are themselves code (which, they are, I mean there's no argument to be made there), then shouldn't your tests be tested? At what point do you call this "too much recursion" and give up? Because apparently, the answer is one "level of tests".
If tests prove quality, and your code is only as good as your tests, then you should probably be testing your tests. As far as I know, nobody does that. If they are, they aren't blogging about it.
It might sound like I am focusing on coverage, and I am, but that's the metric that gets used. If you want to talk about having very detailed, very bite-sized, and very accurate tests which prove that everything only does the bare minimum of what is expected of it, you could possibly convince me that your tests are high-quality, and therefore your code-base is. However, you have not escaped the fact that you can't objectively prove the quality of your tests without testing them.
The Actual Purpose of Tests
Short and simple; Tests prove to people unable to read your code that your code works. Note, I didn't say "other people" just ... people. Like I said, tests have their place, and, to me, that's it. Need to prove to yourself your ungodly kluge-beast does it's job? Write a test. Are you a new developer, trying to learn a new language, or writing a language from scratch? Start with tests! Writing OSS to distribute to legions of your adoring fans on multiple platforms and unknown hardware? Why not tests?!
"But Drew" I hear you say loudly with the smashing of fingers into keys, "how can we prove quality without coverage? How do we make high-quality code?" The answer is, unfortunately, complicated.
Regarding Quality
If you conducted an informal survey, asking people to compare Mercedes and Toyota as brands, in terms of the quality of their vehicles, you're going to get a lot of interesting answers. The question you should be asking here is "What is quality?". honestly, there is no right answer. There's a lot of wrong answers, because quality is obviously not a measure of "how quickly can this be turned to crap". If that's your answer, you've optimized incorrectly. Try again.
Quality means different things to different people. Trying to assign a metric to quality (for example, coverage) makes quality subject to Goodheart's Law.
Quality is subjective. In fact it really depends on what is most important to an individual, team, or organization.
A New Measure of Quality
I propose that high-quality code is simply easy to read. Now, when I talk about being easy to read, I'm not talking about tabs vs. spaces, or where the brackets go, or how many characters per line.
Self Documenting Code
The best example I can give you is something one of my former bosses use to do all the time. Back when I was a Junior Engineer, I worked under Eric. Eric was notorious for making methods named something like public String convertTheUserGroupsForDisplay(). If memory serves, he holds the world record for "longest method name". He even has a trophy, but the name plate fell off because it was too long.
The point is, Eric believed very strongly in self-documenting code. Does your code do what it says on the tin? Then change the code or change the tin. When I was a wide-eyed, fresh, eager, green developer, this struck me as the best thing since sliced bread. I didn't even have to really read the code, I could just look at the method names and make some really good assumptions about what the methods did. It's simply a great way to live.
Since then, my methodologies have evolved, my code style has matured, and I don't work in Java daily, but the one thing I have absolutely held on to is the principle of self-documenting code. I do a lot of front-end work, so naturally, 328 character methods names are not okay, but I always try to make sure the tins are labeled correctly.
Can I Read my Own Code?
function doStuffToThings(things){
var a = things.length > 2? 215 : 212
, b = a % 5 == 0? 919 : 1;
if ( (a + b)/b - (b*2) > 0){
return a;
}
return b;
}
I just wrote this, and I can't immediately tell what it might return. If I come back to this after 2 weeks or 5 drinks, screw it, time for a refactor. If you look at the code really carefully though, you'll notice the only real condition is wether or not there are more than 2 things.
function doStuffToThings(things){
if(things.length > 2){
return 215;
}
return 212;
}
The point of all that nonsense code is simply this: have you made your code as simple as you possibly can? Here's an easy test: write a method, take a shot, repeat for 3 methods. If you can still tell what your code does, congratulations, you are already doing this right. Not only is this more productive than TDD or BDD, it's a lot more fun too.
The principle here is universal and easy to understand. There's a reason we have high-level languages and don't simply write web pages in Assembly.
Naturally, you're going to run into a situation where you have to string together a bunch of bitwise operators, or where plain english variale names don't make sense, it's unavoidable, and we have a way to handle it already. In fact, we've had it all along.
Comments
What do you do when your code isn't that readable? Comment!
I can almost guarantee your favorite programming language supports comments. If it doesn't, maybe it's time to pick up a new language.
What's funny is that some people have experimented with hobby languages that do not support comments and what they found was that removing comments forces you to write literate, self-documenting code. So, are comments necessary? Maybe, maybe not. Without them, you need to make your code highly readable, but you can't do that 100% of the time, so it's a good thing you can use comments where you need to.
So, Death to Tests, Right?
Nope. Not even close. Like I said at the start, tests have their place. That place is not "everywhere". What I want to see is the industry gently, but firmly, removing it's head from its own butt with this "high coverage == high quality" rhetoric. It's pants-on-head stupid and a giant waste of everyone's time.
I write tests. Sometimes I write them because I wrote a method that is too hard to follow just by staring at it, but more often than not, it's to check a box in a list of ill-defined requirements.
What I am advocating here is better communication between the various parts of your teams, easier to read code, and clearly defined, concise requirements. These three things by themselves will improve the quality of software. If it works for you to write tests for everything, they will also improve the tests.
I have always brought a straight-forward GSD attitude to teams, and that has worked well for me. In fact, it has worked pretty well for the teams I have been on.