Can we say that most developers don't know how to test?

Every developer knows unit tests exist to prevent defects from reaching production. Unit testing is a way of assuring the quality of some code.

What most developers do not know are the essential ingredients of every unit test. With a failed unit test, what feature was the developer trying to test, what went wrong and why does it matter?

Code For This Blog

https://github.com/bob-fornal/machining-better-tests

State Machines

By themselves, conventional unit testing frameworks do not offer proper support for systematically testing object-oriented units. A state machine model offers a more structured system is a suggested improvement.

Exhaustively testing any non-trivial unit of code in most cases would take too long to be practical. Therefore, the tester needs to select a number of significant states and a number of significant input sequences.

Conventionally, automated unit testing is conducted by writing scripts, where the code-under-test (CUT) is started in a specific state, wherein the functionality of the CUT is tested against the specification.

Unfortunately, conventional test scripts are often written in an ad hoc manner without analyzing the true nature of the CUT. Thus, it is easy to miss, or fail to test, some aspect of the functionality.

Cost of Bugs

Bugs that reach customers cost in many ways:

They interrupt the user experience, which can impact sales, usage, and even drive customers away.
Reports must be validated by QA or developers.
Fixes are work interruptions which cause a context switch. The context switch does not count the time to fix the bug.
Diagnosis happens outside of normal feature development, sometimes by different developers.
The development team must wait for bug fixes before they can continue working on the planned development roadmap.

The cost of a bug that makes it into production is many times larger than the cost of a bug caught by an automated test suite.

Isolation

Discussion of Integration Tests here assumes these are developer level tests, not system level run by QA.

Unit and integration tests need to be isolated from each other so that they can be run easily during different phases of development. During continuous integration, tests are frequently used in two ways:

Development: For developer feedback. Unit tests are particularly helpful at this stage.
Staging Environment: To detect problems and stop the deploy process if something goes wrong. The full suite of test types are typically run at this stage.

Test Discipline

Tests are the first and best line of defense against software defects. They are more important than linting or static analysis (which can only find a layer of errors, not issues with logic).

Unit tests combine many features that will lead to success:

Design Aid: Writing tests first provides a clear perspective on the ideal design.
Feature Documentation: Test descriptions cover implemented feature requirement.
Developer Understanding: Articulation, in code, of all critical requirements.
Quality Assurance: Manual QA is error prone. It is impossible for a developer to remember all the features that need testing when refactoring, adding, or removing features.

Bug Reporting

What is in a good bug report?

What was tested?
What should the feature do?
What was the output (actual behavior)?
What was the expected output (expected behavior)?
Can it be reproduced?

Implementing State Machines

The state machine-model based unit testing approach requires that the tester develop a state machine model of the unit. The model should contain the states that are significant for testing, and state transitions. The transitions should effectively test all means of getting from one state to another.

Code Under Test (CUT)

var testableCode = {
  items: [],
  push: function(item) {
    if (testableCode.items.length >= 10) {
      return testableCode.items;
    }
    testableCode.items.push(item);
    return testableCode.items;
  },
  pop: function() {
    if (testableCode.items.length === 0) {
      return testableCode.items;
    }
    return testableCode.items.pop();
  },
  clear: function() {
    testableCode.items = [];
    return testableCode.items;
  }
};

Beginning with an example of an array with a limited maximum capacity (10 objects) and three methods: push, pop, and clear. There should be three states:

Empty: no objects in the array.
Full: max (10) objects in the array.
Loaded: not Empty or Full.

The three methods for this example, should function as follows:

push: should add an element to the end of the array.
pop: should remove the last element from the array.
clear: should remove all elements from the array.

Given the information provided, we can examine all the ways that each of the states (Empty, Full, Loaded) can be achieved.

Pattern (From / To)	Method to Achieve
Empty / Full	PUSH 10 times **
Empty / Loaded	PUSH 4 times *
Full / Empty	POP 10 times **
Full / Empty	CLEAR 1 time
Full / Loaded	POP 6 times (10 - 4)
Loaded / Empty	POP 4 times **
Loaded / Empty	CLEAR 1 time
Loaded / Full	PUSH 6 times (10 - 4) **

* 4 was simply chosen as being not empty or full. Anything from 1 to 9 items could have been used.
** Here are where test exceptions should be identified; places where the code could do something unusual.

This gives eight possible transitions:

Empty to Full: using push 10 times (10 objects).
Empty to Loaded: using push 4 times (4 objects).
Full to Empty: using pop 10 times (0 objects).
Full to Empty: using clear 1 time (0 objects).
Full to Loaded: using pop 6 times (4 objects).
Loaded to Empty: using pop 4 times (0 objects).
Loaded to Empty: using clear 1 time (0 objects).
Loaded to Full: using push 6 times (10 objects).

describe('Machining Better Tests', function() {

  beforeEach(function() {
    testableCode.items = [];
  });

  describe('Standard State Transitions', function() {
    it('expects "Empty to Full: using push 10 times (10 objects)"', function() {
      var push = 10;
      for (var i = 0, len = push; i < len; i++) {
        testableCode.push(i);
      }
      expect(testableCode.items.length).toEqual(10);
    });

    it('expects "Empty to Loaded: using push 4 times (4 objects)"', function() {
      var push = 4;
      for (var i = 0, len = push; i < len; i++) {
        testableCode.push(i);
      }
      expect(testableCode.items.length).toEqual(4);
    });

    it('expects "Full to Empty: using pop 10 times (0 objects)"', function() {
      testableCode.items = [1,2,3,4,5,6,7,8,9,10];
      var pop = 10;
      for (var i = 0, len = pop; i < len; i++) {
        testableCode.pop();
      }
      expect(testableCode.items.length).toEqual(0);
    });

    it('expects "Full to Empty: using clear 1 time (0 objects)"', function() {
      testableCode.items = [1,2,3,4,5,6,7,8,9,10];
      testableCode.clear();
      expect(testableCode.items.length).toEqual(0);
    });

    it('expects "Full to Loaded: using pop 6 times (4 objects)"', function() {
      testableCode.items = [1,2,3,4,5,6,7,8,9,10];
      var pop = 6;
      for (var i = 0, len = pop; i < len; i++) {
        testableCode.pop();
      }
      expect(testableCode.items.length).toEqual(4);
    });

    it('expects "Loaded to Empty: using pop 4 times (0 objects)"', function() {
      testableCode.items = [1,2,3,4];
      var pop = 4;
      for (var i = 0, len = pop; i < len; i++) {
        testableCode.pop();
      }
      expect(testableCode.items.length).toEqual(0);
    });

    it('expects "Loaded to Empty: using clear 1 time (0 objects)"', function() {
      testableCode.items = [1,2,3,4];
      testableCode.clear();
      expect(testableCode.items.length).toEqual(0);
    });

    it('expects "Loaded to Full: using push 6 times (10 objects)"', function() {
      testableCode.items = [1,2,3,4];
      var push = 6;
      for (var i = 0, len = push; i < len; i++) {
        testableCode.push(i);
      }
      expect(testableCode.items.length).toEqual(10);
    });  
  });
});

Examining the eight possible transitions, a few exceptions should be called out (are they handled correctly):

(see 1): Empty to Full: using push 11 times [exception] (10 objects).
(see 3): Full to Empty: using pop 11 times [exception] (0 objects).
(see 6): Loaded to Empty: using pop 5 times [exception] (0 objects).
(see 8): Loaded to Full: using push 7 times [exception] (10 objects).

describe('Machining Better Tests', function() {

  beforeEach(function() {
    testableCode.items = [];
  });

  describe('EXCEPTIONS ...', function() {
    it('expects "Empty to Full: using push 11 times (10 objects)"', function() {
      var push = 11;
      for (var i = 0, len = push; i < len; i++) {
        testableCode.push(i);
      }
      expect(testableCode.items.length).toEqual(10);  
    });

    it('expects "Full to Empty: using pop 11 times (0 objects)"', function() {
      testableCode.items = [1,2,3,4,5,6,7,8,9,10];
      var pop = 11;
      for (var i = 0, len = pop; i < len; i++) {
        testableCode.pop();
      }
      expect(testableCode.items.length).toEqual(0);
    });

    it('expects "Loaded to Empty: using pop 5 times (0 objects)"', function() {
      testableCode.items = [1,2,3,4];
      var pop = 5;
      for (var i = 0, len = pop; i < len; i++) {
        testableCode.pop();
      }
      expect(testableCode.items.length).toEqual(0);
    });

    it('expects "Loaded to Full: using push 7 times (10 objects)"', function() {
      testableCode.items = [1,2,3,4];
      var push = 7;
      for (var i = 0, len = push; i < len; i++) {
        testableCode.push(i);
      }
      expect(testableCode.items.length).toEqual(10);
    });  
  });
});

Equivalence

Whenever two separate CUT-instances are brought into the same state, they should be logically equal, even if the state is reached through different transition paths. We can see this with the various expects above ...

expect(testableCode.items.length).toEqual(10);
expect(testableCode.items.length).toEqual(4);
expect(testableCode.items.length).toEqual(0);

Conclusion