A CLI test matrix covers combinations of valid arguments, missing required options, invalid values and expected exit codes.
Understanding the Concept
A CLI test matrix covers combinations of valid arguments, missing required options, invalid values and expected exit codes. Start with a few representative cases instead of one successful manual run.
Some parse errors raise ArgumentError with exit_on_error=False while others may still trigger SystemExit; test the exact parser behavior.
How this fits into a real test suite
A reliable test starts with a specific expected behavior. Keep the fixture small, arrange the input, call the real code, and assert a result the reader can verify. Add one boundary or failure case so the test does not pass only for a convenient example. In a larger project, run the same test automatically after each change.
Tests are repeatable experiments. Use temporary resources and injected dependencies where possible; when the environment is part of the requirement, add a separate integration test rather than weakening the unit test.
Worked Example
Read the code from the first line to the last. Identify which values are inputs, which operation is being demonstrated and what should happen before running it.
import argparse
def parser():
p = argparse.ArgumentParser(exit_on_error=False)
p.add_argument("--count", type=int, default=1)
return p
for arguments, expected in [([], 1), (["--count", "4"], 4)]:
assert parser().parse_args(arguments).count == expected
print("CLI cases passed")Walkthrough: What Each Part Does
- The imports make argparse available. These are standard-library building blocks unless the example explicitly names an external tool.
- The definition parser creates reusable behavior. Its body runs when called, awaited or instantiated; defining it alone does not run the body of an ordinary function.
- The loop repeats one well-defined unit of work. Check termination, ordering and whether an item may be processed more than once.
- The final check compares actual behavior with an explicit expectation. A successful test tells you that this particular case passed, not that every possible input works.
This demonstration focuses on one practical behavior. Before changing several lines, vary one input or one configuration choice and check whether the result changes for the reason you expected.
The example uses terminal, platform, external tool or process behavior that was not exercised automatically. Run it only in a suitable environment after reviewing the shown permissions and paths.
Important Behavior and Edge Cases
Testing only the happy path and overlooking help text, invalid arguments and nonzero exit codes.
Some parse errors raise ArgumentError with exit_on_error=False while others may still trigger SystemExit; test the exact parser behavior.
Practice Exercise
Extend the matrix for a missing value, an invalid integer and the --help option.
Write down the result you expect before running your changed code. Check at least one normal case and one boundary or failure case. If your example starts workers or opens resources, verify how they are cleaned up.
Further reading: official documentation and API reference.
Common Beginner Mistakes
- Testing only the happy path and overlooking help text, invalid arguments and nonzero exit codes.
- Writing assertions that check implementation details but fail to validate the behavior that matters to the caller.
Key Takeaways
- A CLI test matrix covers combinations of valid arguments, missing required options, invalid values and expected exit codes.
- Some parse errors raise ArgumentError with exit_on_error=False while others may still trigger SystemExit; test the exact parser behavior.
- Practice: Extend the matrix for a missing value, an invalid integer and the --help option.