Monday, February 28, 2011

4 Types of automated tests - unit, integration, UI, and performance

A software engineer could spend their life continually improving test automation - it's a big field. While the sky is the limit, there are at least 4 types of automated tests: unit, integration, UI, and performance.
Some key ideas:
  • These build off of each other -
    • Unit --> Integration: Don't bother with complex integration tests if you can't even have a simple unit test.
    • Integration --> UI: It's going to be near impossible to do a UI test (which usually has poor APIs) if you can't at least integrate the backend (with at least has APIs - like web services, SQL, or C# calls).
    • UI --> Performance: If you can't at least functionally run the code from end-to-end, then you can't expect reliable performance measures on it. Yes, there are always exceptions, and semantics (one may consider "UI" to be fronted integration, or one may test performance on just the backend APIs and bypass the UI). But in general, these 4 tests are a very natural path to follow.
  • The higher you go, the more expensive: Unit tests (low-level) are cheapest, performance tests (high-level) are most expensive. So it's bad business to pay for an integration test to do the work of a unit test. It's like using an armani suit as a dish rag.
  • These 4 types of tests should be separated. You can call any code from a unit test (depending on security, you could even call APIs to shutdown the server), so you could mix all your tests into one test harness. But don't do this - it will burn you. For example, unit tests are generally fast (they're all in-memory), whereas UI and integration are much slower (databases, web services, IIS hosts, etc...) So you don't want them coupled together because the slow integration tests will bog down your fast unit tests, and then developers never run unit tests before check-in because "it takes too long".
  • Unit testing is but one tool. There is different types of code (algorithms, data containers, plumbing, installation scripts, UI, persistence plumbing, etc...). This requires an arsenal of developer skills, of which unit testing is one tool. With respect to unit testing and code coverage, the goal isn't N% coverage of all code, but rather N% coverage of unit-testable code. (You can get better coverage tools, like NCover, which can provide coverage when running integration, UI, and even manual tests run by QA, but that's a different story).
And it wouldn't be complete unless I had a pro/con table of each option:

Test TypeGood ForBad For
Unit
  • Units of in-memory code, like parsing, calculations, validations,
    algorithms, formatting, etc...
  • Because unit tests are in-memory, they usually run very fast, and
    hence can be run upon check-in and with each build. Therefore they provide
    the "first level of defense" to ensure code continues to functionally
    work.

  • Integration, like anything that hits networks - the tests will be too
    brittle, something will break
  • Plumbing or generated code (like designer.cs files or database
    mapping) - you'll just now have both tedious plumbing code and
    tedious unit tests.
Integration ("backend")Ensuring high-level flows work, such as you can
call a web service that loads or saves data to a database and writes
something to a file.
Anything that can be handled with a unit test
instead. For example, you likely wouldn't use an integration test to verify
every input combination for a text field.
UI ("frontend integration")Very-high level, functional tests.Anything that can be handled via backend
integration or unit tests.
PerformanceIdentifying performance problems that could be
costly to the business.
Any functional testing

4 comments:

  1. Nice post, Im suprise this hasnt had any comments so I thought Id be the first. It was a very informative and very useful, thank you.

    ReplyDelete
  2. Fantastic summary, everything important contained in such a short post!

    ReplyDelete
  3. Very helpful information, thanks. I'm working with http://www.nixsolutions.com/services/quality-assurance-services/ and they are very good. I would lke to recommend them if you need to test something.

    ReplyDelete
  4. Nice post. I learn something more challenging on different blogs everyday. It will always be stimulating to read content from other writers and practice a little. Thanks for sharing.

    ReplyDelete