Software Testing for Dummies — exposed and ridiculed

One test execution per requirement = unsustainable cost and delay

Software engineers — have you ever honestly considered how we approach testing of fully integrated software systems? If we applied our approach to any other mature industry, we would be laughed out of existence.

Can you imagine a car manufacturer, in crash testing their vehicle, using a crash dummy with sensors for one particular body part? They set up the dummy with head sensors, crash a car head-on, then read the results to ensure compliance with regulatory standards. They then swap to torso sensors, crash a second car head-on, then read the results. Next, they refit the dummy with lower body sensors, and crash a third car head-on. Each test destroys a vehicle. Each vehicle could have been sold.

How long do you think that car manufacturer could remain in business? The material cost of destroying a sellable vehicle for each individual sensor reading would be staggering — and every one of those results could have been captured in a single test. The approach of using one sensor per test may be best practice for testing a component — an airbag, for example — pre-assembly. But it is the wrong approach when testing a post-assembly product like a vehicle. Yet this is commonly the approach used in testing fully integrated software systems, each test with verifications for a single, specific requirement. Is it any wonder that the cost of testing fully integrated systems is so high, and why there are so many delays in delivering finished product?

Now imagine applying to software development the approach used in mature industries. Given data in a unique scenario, perform a set of actions to a destination, with sensors — verifications — in place to confirm any number of result states, each responding with feedback regardless of how many indicate failures. A multitude of individual results can now be measured against any number of requirements, in a single test. No verification is dependent on the success of any other verification.

Take the analogy one step further. Every crash dummy is fitted with the same standard harness of sensors — regulatory compliance verified identically across every test, without exception, regardless of what additional sensors a specific scenario requires. The comprehensive baseline is not assembled test by test. It is built into the dummy itself.

The same principle applies to software. A single validation module — shared across every test that validates the same subject — ensures that baseline verification is comprehensive, consistent, and not dependent on the discipline of the individual engineer who wrote the test. The standard sensors are always in place. The scenario-specific verifications are added on top. Every test benefits from both, without the author having to think about either.

This is similar to setting up a crash dummy with head, torso, arm and leg sensors, submitting the vehicle to a head-on collision, then verifying the results of each and every sensor individually. Use the same fully-sensorized dummy in every crash scenario, and the number of unique test executions required is greatly reduced. Doing the same for each software scenario has the same result: a smaller, more manageable test library and greatly reduced number of unique test executions.

This principle does not change in the age of AI-assisted testing. AI generates tests based on what it understands to be important — which is almost always derived from the requirements. It will set up one sensor per test as reliably as any engineer trained to do the same. The approach does not become comprehensive simply because a tool generated it. Comprehensive validation requires deliberate design — the decision to instrument every sensor, not just the one the requirement described.

Reconciling our approach to validation of post-assembly software with that used by other mature industries makes comprehensive, resilient, effective and efficient testing of fully integrated software systems achievable. The result is higher-quality software, delivered in a sustainable release cycle, at greatly reduced cost.

Be reconciled!