Pawel KadluczkaShipping high-quality software is the responsibility of each software developer. Not only are the days when handing untested code to the QA team for validation a common practice long gone, but many companies have also moved away from QA-based testing, making developers fully own the quality of the product. This creates a problem: how can you ensure developers do their due diligence and validate their changes? At Meta (a.k.a. Facebook), we use "test plans."
What are test plans?
Test plans describe how authors tested their changes. They are an integral part of Meta's code review process: the code review tool does not allow submitting code for review if the test plan is empty.
While it is common for test plans to say: "unit tests," many are much more interesting and often include:
- screenshots showing the UI before and after the change
- videos showing the change in action
- API requests and corresponding responses (e.g., JSON payloads)
- dashboard snapshots from canary runs
- terminal printouts
- funny memes - especially common if the testing was not comprehensive or applicable (e.g., auto-generated code, changes to unit tests only, etc.)
Benefits of requiring Test Plans in commits
The most obvious benefit of requiring test plans is forcing developers to think about validating their changes. While not all test plans are comprehensive, most are decent.
Occasionally, test plans reveal that the code change does not work as the author intended. I once proudly included a graph hovering a little below 100% as my test plan, only for a reviewer to point out that my graph represented not the success rate, as I claimed, but the error rate.
However, one often overlooked benefit of requiring test plans is that they can be lifesavers when working with unfamiliar code.
Imagine you are tasked with building a new feature in a mobile app. While working on the feature, you discover that the API powering your app doesn't return all the necessary information. You can ask the API team to add it, but they may not be able to accommodate your request on short notice. Perhaps it would be faster if you implemented this change yourself. The only problem is that you are not familiar with the backend code. You don't know how to test it to ensure you didn't break anything. In these situations, test plans can come in extremely handy. You can check the commit history of the code you want to change and see how developers who regularly contribute to it test it. In addition, checking past test plans may help you discover edge cases you need to consider. I successfully used this strategy multiple times to change an unfamiliar codebase I would otherwise be afraid to touch.
“My unit test coverage is 100%”
Test plans should not be considered a replacement for unit tests. As great as unit testing is, it is not always sufficient. Additional end-to-end validation helps confirm that the changes worked as intended outside of the isolation provided by unit tests and that they didn't introduce unwanted behavior. I have seen (and caused) situations where my application wouldn't start even though all unit tests were passing.
Call To Action
Including test plans in pull requests is not a common practice, let alone a requirement, in most companies or teams. Despite that, I encourage you to follow this practice. Your team members will notice it and may start doing the same if they find it useful. And even if they don't, you will still benefit from this habit. Going the extra mile can help you find issues before they impact users. With time, it will get easier because you can reuse past test plans to validate some of your current changes. You may need to tweak them a little, but you won't have to start from scratch each time.
💙 If you liked this article...
I publish a weekly newsletter for software engineers who want to grow their careers. I share mistakes I’ve made and lessons I’ve learned over the past 20 years as a software engineer.
Sign up here to get articles like this delivered to your inbox.
