Thoughts on testing

Unit Tests

Unit tests test individual units (modules, functions, classes) in isolation from the rest of the program. Compare unit tests with integration tests, which test integrations between two or more units. If we have to mock something, then we are testing integrations between two or more units.

Functional Tests

Functional testing to check a particular feature for correctness by comparing the results for a given input against the 'spec'. Functional tests don't concern themselves with intermediate results or side-effects, just the result (they don't care that after doing x, object y has state z). They test part of the specification such as, "calling function Times(x, y) with the argument of 2, 3 returns 6".

Black Box & White Box

In general, units are tested using only the “public API” of the unit. This is also called Black box testing. Black box testing leads to less brittle tests, because the implementation details of a unit tend to change more over time than the public API of the unit.

Compared to white box testing, where tests are aware of implementation details, any change to the implementation details could break the test, even if the public API continues to function as expected. White Box testing is very thorough , but the tests can be extremely brittle leading to lots of effort fixing tests. It is better to write these when the internals of the Component are more settled.

Lets Talk Code Coverage!

Code coverage refers to the amount of code covered by test cases. Coverage reports can be created by instrumenting the code and recording which lines were exercised during a test run. Most of the time, we try to produce a high level of coverage, but code coverage starts to deliver diminishing returns as it gets closer to 100%.

An alternative to code coverage is case coverage or 'test coverage', which determines whether the test cases are covering the entire functional/case requirements: How the code will behave in the context of real world environment, with real users, and real networks. This can even include bad or malicious inputs.

Coverage reports identify code-coverage weaknesses, not case-coverage weaknesses. The same code may apply to more than one use-case, and a single use-case may depend on code outside the subject-under-test.

Code coverage: how much of the code is exercised

Case coverage: how many of the use-cases are covered by the test suites

Because use-cases involve multiple units, users, and even networking, it is really hard to test all the use-cases with a test suite that only contains unit tests. Unit tests by definition test units in isolation, not in integration, meaning that a test suite containing only unit tests will always have close to 0% case coverage for integration, but they might have a higher code coverage metric.

100% code coverage does not guarantee 100% case coverage.

100% Code coverage should not be a metric, but rather a 'litmus' test for the health of a project. Also there are some huge diminishing returns when going above 80% code coverage.