Goad Testing: Guaranteeing that Tests Make Distinctions
Add to calendarTest Driven Development has reached maturity. Goad Testing takes you a level deeper, examining what tests really are and using the outcome of that exercise to derive new ways of keeping software - both production and test - healthy, flexible, and on-specification.
Making distinctions is a critical aspect of a test: It allows a test to serve as an executable specification. Sometimes this ability is lost in the course of maintenance. Goad testing is a way to prevent that from happening without introducing significant extra work or complexity.
Slides can be found here.
The bulk of this talk is in demonstrating the actual effect we want to avoid and how to avoid it. The theory part can be taught in about thirty minutes.
The session will be broken up as follows:
- Establish the problem
- Debunking solutions derived from existing techniques
- Rethinking the problem from the perspective of design
- Show how to apply the solution in new tests and code
- Show how to retrofit the solution to existing tests and code
- Comparison with other techniques and technologies
- Final Q&A
Establishing the problem
In the first part of the talk, I will show the problem this is meant to address. Using code examples in slides, I will demonstrate how a small mistake in the course of a somewhat routine refactoring can neutralize a test - render it incapable of making a meaningful distinction.
Debunking solutions derived from existing techniques
I will show how certain solutions that we have thought of, including the whimsical “test your tests” solution don’t work. I will also show that we do, in fact, test our tests. We just do it manually.
Rethinking the problem in terms of design
I will quickly cover the value of a test: specifying and verifying how software works. I will apply that to a test’s ability to make a distinction while reconsidering some of the fundamental assumptions we tend to make in test-driven development.
Show how to apply the solution to new code
Using code examples in PowerPoint slides, I will take the abstract example of how tests should be redesigned and show a concrete example of how this technique should be applied when writing new tests.
Show how to apply the solution to existing code
I will also show how to take a test that has already been written and apply goad testing to it so that it can safely be refactored. There is a parallel here between legacy code - that which is not covered in automated tests - and legacy tests - those which are not verified with a goad.
Comparison to other techniques and technologies
I will spend a little time talking about some other self-verifying/automatic verification techniques that are useful. Most importantly, I will distinguish goad testing, which verifies that every test makes a distinction, from mutation testing, which verifies that all code is the product of a distinction made in some test.
Q&A
Questions are allowed throughout. However, some people prefer to ask questions at the end. We will have a little time at the end for discussion and Q&A.
- Weaknesses can be introduced into tests by (supposed) refactoring
- Why existing techniques cannot defend against such decay
- How design changes the nature of the problem
- A new technique that ensures every test makes some distinction
- How to apply that technique to new and existing tests
- What this technique does not help with (what is still your responsibility)
