Automatic testing based on contracts

mristin


Description:

Overview

Writing unit tests is easy. Writing thorough unit tests is hard and tedious: as the number of function arguments and the complexity of a function grow, so does the quantity of the edge cases. Oftentimes, due to busy schedules, negligence or ignorance, we simply leave out the edge cases and test only a couple of happy paths in our code. Obviously, this leaves potential big holes of untested code that eventually runs in the production lurking to lay mayhem.

We would like to introduce a possible remedy. Instead of writing more and more unit tests, we specify the behavior of a function with code contracts (e.g., using assertions or a library such as icontract) and let the tools automatically test the function for us. We will talk about the scenarios in which contracts can significantly reduce the need for unit testing, and the scenarios where they can't.

In this workshop, we will start by:

  • Briefly introducing you to programming with contracts,
  • Explaining how it slightly differs from similar approaches such as property-based testing and finally
  • Presenting our tools for automatic testing (crosshair and icontract-hypothesis).

After the introduction, we will:

  • Solve a couple of programming exercises together,
  • Help you design the contracts as you develop your code, and
  • Show you how to test your code automatically both in IDE and as part of continuous integration.

Breakdown

  1. Introduction about the contracts (15 slides, 30 min)
  2. Present a couple of problems and we work out the contracts together (30min)
  3. Break (15min)
  4. Introduction about the tools for automatic testing (very brief! 10-12 slides in total, only 30min)
  5. Implement the problems from 2) together and test using the tools (45min)

Outline

1) Introduction about the contracts (15 slides, 30 min)

  • Why contracts -- specify behavior
    • Preconditions (before)
    • Postconditions (after)
    • Assertions (within the body of the function)
  • Why are contracts better than tests
    • Verifiable documentation
    • Test for properties -- not just one data point (one test == one set of test data)
    • Deeper tests
      • You can turn on the contracts in integration testing/ end-to-end testing/ production
      • Contracts are verified for all the functions on the paths
      • ... versus unit/integration/end-to-end tests which only check the output
      • Discover bugs in the intermediate functions!
      • (Output might be correct, but the intermediate functions might be wrong. This is something your unit tests can not tell you.)
    • Indispensable for static analysis
    • Faster development
      • Test automatically while developing (one mouse click!)
      • Spot bugs earlier and more focused
    • Easier to maintain than many tests
      • Contracts are shorter than tests.
      • You write much fewer tests.
      • Live close to code -- so when you change code, you tend to change the contracts automatically
  • Contracts versus Property-based tests
    • Property-based tests can not run in integration/end-to-end tests
  • Misconceptions
    • "Contracts must be complete."
    • "You have to first write the contracts."
    • "Contracts are hard to read".
    • "Contracts are slow."
    • "Contracts are only important for airplane systems."
    • "You need to practice Design-by-Contract (DbC) as methodology."
    • "Contracts can have bugs themselves."
    • "Contracts are often longer than the body of the function."
  • Where contracts fail or fall short
    • Contracts need to be pure (though the code need not to!)
      • This is rarely a practical limitation.
    • For some functions, the complexity of the contracts can be overwhelming.
      • In such cases, unit tests in form of table-driven tests might be a better option.
    • Some contracts are simply unenforceable
      • ... and probably untestable as well.
      • For example, global unique identifier.
  • Point to further topics and literature
    • Snapshots (OLD values), invariants, inheritance, purity etc.

2) Present a couple of problems and we work out the contracts together (1h)

3) Break (15min)

4) Introduction about the tools (very brief! 10-12 slides in total, only 30min)

  • icontract-hypothesis
    • Examine the contracts
    • Infer the strategy about how to generate the inputs randomly
    • Sample the inputs (according to this strategy)
    • Execute the function with the inputs
  • crosshair
    • Based on symbolic execution
    • Monkey-patch the inputs of the code
    • Pass in the patched objects instead of real inputs
    • Infer properties based on the patched objects (and what methods were called on them)
  • When should you use one or the other?
    • icontract-hypothesis considers a function to be a black box.
      • Generate the inputs and execute the function with it.
      • The assertions are verified during the execution.
      • The post-conditions are verified after the execution.
  • crosshair works as a whitebox.
    • Works less randomly (or almost deterministically).
  • crosshair > icontract-hypothesis:
    • icontract-hypothesis depends on sampling.
    • If the bugs require very fine-grained sampling, icontract-hypothesis will miss them.
    • Instead, crosshair would catch them by analysis.
    • Example
  • icontract-hypothesis > crosshair
    • For many problems symbolic execution does not work well
      • Shortcomings of the solver
      • Large-scale tests → crosshair might sweat here where the randomized approach would have no problems.
    • External dependencies can not be monkey-patched.
      • Symbolic execution can not follow external systems: file system, database etc.
      • Randomized testing does not care about that as long as you pass in your dependencies as inputs.
    • Example
  • Point to further topics and literature
    • List of other tools, other techniques (such as fuzzying) etc.

5) Implement the problems from 2) together and test using the tools. (45min)

  • Keep testing with both tools while we go.
  • Hope to find the bugs early

Setup Instructions

  • We will use Thonny IDE and show how to install the plug-ins and the tools,
    • Thonny runs on Windows, Mac and Linux.
  • … but you are free to use your own IDE (and use the tools from the command line).
    • Plug-ins are available for PyCharm and VS Code.
    • Crosshair requires Python 3.7+.
  • We will provide the code stubs for the exercises.

Prerequisites:

Our target audience are people familiar with the lambda functions, Boolean logic (and, or), quantifiers (any, all) and related generator expressions such as:

all(
  item > 0 and item % 3 == 0
  for item in some_list
)

Video URL:

https://www.youtube.com/watch?v=8QNBRfuUdMw

Speaker Info:

(Equal contribution, in alphabetical order)

Marko Ristin is a software engineer and computer vision researcher. He worked at different software engineering and research positions, some of which included writing correct Python programs. Marko developed and maintains icontract, a design-by-contract library for Python, and icontract-hypothesis, a tool for inferring property-based tests from contracts.

Phillip Schanely is a database and program language enthusiast in New York City. He's had the opportunity to work with Python in various startups, as a technology consultant, and today, at Google. Phillip is the primary developer behind CrossHair, a tool that brings the power of modern theorem provers to help Python developers.

Speaker Links:

(Equal contribution, in alphabetical order)

Marko Ristin: https://github.com/mristin

Phillip Schanely: https://github.com/pschanely

Section: Developer tools and automation
Type: Workshop
Target Audience: Intermediate
Last Updated: