Now, this seems like something that should be pretty straight forward ... testing the test and all that; what-what. But there are a few underlying considerations that make how to Unit Testing A/B Test Code a fairly remarkable question.

Common Language

While writing this article about testing test code, the language started to get a bit wonky.

Here are the definitions I will use moving forward ...

• A-TEST CODE refers to the code that should already exist when it is decided that and A/B Test is needed. The expectation should be that Unit and Integration tests already exist against this code.
• B-TEST CODE refers to the code that will be written. Unit and Integration Tests should be written against this code, as well.

This article is focused on the fact that additional tests (Unit and Integration) may need to be created to ensure these two sets of code work together properly with the existence of the other. Having said that ...

• NEGATIVE-SCENARIOS should not be confused with Negative Testing which examines ways the code can fail. Negative Scenarios are the inverse of the tests for the alternate scenario (A-TEST CODE <-> B-TEST CODE).

Overview of Considered Scenarios

Examining the scenarios this way ...

^* These are tests written against the B-TEST code, testing the new functionality.
^** These are tests that should be written as negative scenarios, testing for bleed-over of code between A-TEST and B-TEST scenarios. The tests for the alternate area should fail appropriately (B-TEST tests should include the INVERSE of A-TEST tests).

The Questions

1. There should be test code for both the A-TEST and B-TEST scenarios. The question is how much test coverage should there be for the "new" code?
2. Both scenarios need to be concerned about bleed-over from the other code. Are the negative versions of the other scenarios tests enough?
3. Is there common code between the two scenarios that is covered appropriately (and therefore not included as negative test scenarios)?