1. Lack of code convention, uniform style, docstrings, coding principles (DRY, SOLID, etc).
    • Detect: run linters
    • Fix: Treat autotests as a code too. Write separate code convention (eg: pylint configs). Periodic code reviews for tests should be automated.
  2. Using time.sleep() (especially in async code)
    • Detect: search code
    • Fix: wait for events, markers, flags, conditions (polling)
  3. Using print() or print-print-print-assert
    • Detect: search code
    • Fix: consider logging (delegate log setup to test runner!)
  4. Using ..except: pdb.set_trace()
    • Detect: search code
    • Fix: consider --pdb flag
  5. True == False (happy blind tests)
    • Detect: make tests fail
    • Fix: review test-plan, use pytest.mark.xfail (or skip)
  6. True != False (obvious and useless obscure results) missing value explanation
    • Detect: make tests fail, see the output
    • Fix: Use error message (assert 2nd param)
  7. Hardcoding without explanation (in test cases DO describe, DON’T define).

    Note: hardcoding here is used in broader meaining, as a source for “semantic satiation” in code.

    • Detect: There are exact values, steps, behaviour without ability to trace WHY are those chosen and what logic (specs?) are considered behind them.
    • Fix: use data generators with rules derived from specs (also helps mitigate pesticide effect a little).
    • Fix: don’t be too implementatin specific (eg: use ‘turn lights on’ instead of ‘flip switch up’).
    • Fix: comments and docstrings might help.
  8. Missing (hard to retrive) breadcrumbs (thread id, timestamp, case name, uuid, log/code line numbers… etc) during debugging/investigation/results analysis.
    • Detect: See if advertised error message can easily lead to point in test/environment WHERE unexpected behaviour happened.
    • Fix: Add bits (href-s, id-s, timesamps…) to easily track back to error point.
  9. Too many asserts
    • Detect: count asserts in tests
    • Fix: pre-defined validators, prepare results for validation, soft-asserts
  10. Failing outside expected behaviour checks
    • Detect: review how many tests fail on test run errors
    • Fix: assert only target feature (consider fixtures in pytest OR raising non AssertionError in unittest-likes)
  11. Focusing outside SUT
    • Detect: check if SUT interaction present in tests (sometimes also requires peeking into SUT code structure)
    • Fix: tests review, test-plans review, dev assistance
  12. Branching/nested (non-parameterized) logic in steps (and expected results).
    • Detect: search code (and docstrings) for if-statements
    • Fix: proper parametrize, split cases
    • Fix: prohibit If-else statements (nesting non parameterizable logic) inside test case.
  13. Workarounds for unstable run conditions
    • Detect: fuzzy/conditional handling of permanent steps
    • Fix: Abstract, narrow down
  14. Parameters deep nesting, generic variable names
    • Detect: option, value… smell-words
    • Fix: Avoid generic variable/parameters naming (eg: option, value).
    • Fix: Keep parameterise as flat as possible.
  15. Unclear docstring logic
    • Detect: Try to reproduce docstring steps manually.
    • Fix: Use <actor> <action> <options>.
  16. Missing logs BEFORE risky operations and/or at the brim of setup/steps/teardown.
    • Detect: run tests and see if You can clearly distinguish test phase boundaries
  17. Missing asserts and/or validations in test case entirely.
  18. Fixture flood (there are more fixtures than test functions)
    • Detect: compare

    cd tests/; grep -r --include='*.py' '@.*fixture' . | wc -l

    vs

    grep -r --include='*.py' 'def test_' . | wc -l

    If former is bigger than the latter, chances are fixtures got misused. * Fix: Refactor fixtures into meaningful pythonic reusable modules. Parametrize fixtures. RTFM, think hard of what fixtures are meant for.

  19. God fixture.

Short summary

Docstring should be reproducible manually (to some extent).
Avoid generic variable/parameters naming (eg: option, value).
Keep parameterise as flat as possible.
If-else statements (nesting non parameterizable logic) are prohibited inside test case.