Unit test framework

General

Requirements

• Tests execution should be controllable through nant
• Tests execution should be done by build server
• For writing tests it should be possible to execute separate test cases

How do we want to get test peripherals?

• Write test data into database
  • Set up new, additional test data for each test run
  • Delete database and set up a database snapshot
• Write mock ups, so that database is not needed for the test
• Mixed way of test data in database and mock up

Intuitive feeling: Test data in database by adding test data for each test run, perhaps mixed with some mock ups

kind of tests we want to write

• Tests, which are testing the server code
  • by calling the server functionality directly (in the same process)
  • by connecting as client to the server
• Tests, the client code without UI
• Tests, which are testing by interacting with the UI?

Usage in the past

• Calls a function directly
  • no database
    • Test of TFileDiffMerge
    • Test for parts of ICT.Common
    • Pseudo-Test of some controls
    • yml writer test
  • with database
    • rudimentary sequencer test and db constraint test

• Dialogs automated and asserts rudimentary only on dialogs
  • With database and server
    • reset database and Import Data through nant
      • Import/Export Gift batches
      • GL Tests
    • generates data through GUI with minimal random string
      • GL Forms
    • Expects certain data exists / does not exists
      • MFinance.server.gift
      • Partner import (assert on the number of imported partners only)
• A first concept of some dialog tester classes which can be generated automatically by the yaml files

How to write a test

1. Sit down and think, what functionality you have implemented
2. Write down, what needs to be tested. There are always following cases:
   1. Good case in middle range of the possible values
   2. Good case next to the limits of the possible values in all directions
   3. Bad Case in middle range of the not allowed values
   4. Bad Case next to the limits of the not allowed values
3. Write down, on what outcome you should assert (Remember: We are doing white box tests and look INTO the software!)
   1. in the database
   2. in the state of the server (if in the same process)
   3. in the state of the client (if client is used)
4. Write down, which test data you need and how you could generate it
5. Implement the test. Do the test in three stages:

   Testing through Client GUI should be avoided, if possible. It is better to change the client, so that this is possible

   Always ensure, that you on later stages you do not re-test earlier stages. Sometimes it is sufficient to show, that the function is correctly called!

   1. Test without DB, whenever possible
   2. Test inside the server process
   3. Test like a client without GUI
6. Get your test working

Need to look into

Interesting concepts

• http://blog.benhall.me.uk/2010/07/improving-testability-with-castle.html Dependency injection even for configuration

xUnit like frameworks

• http://www.nunit.org/
  • ChristianK uses the version of NUnit that is integrated in SharpDevelop 3.2, as well as the standalone NUnit application. The SharpDevelop-bundled version works well and is nicely integrated into the IDE, but doesn't offer the full feature set of the standalone NUnit application. Combined with the integrated NCover application it is very handy, though!
    • Caveat: The test project needs to be a Commandline or Windows Form Project for it to work (DLL projects don't work).
• http://gallio.org/ ( http://www.mbunit.com/ is part of it)
  • Mirror concept (Interface to reflection for accessing private properties and methods)
  • http://stackoverflow.com/questions/3678783/mbunit-vs-nunit
  • http://stackoverflow.com/questions/261139/nunit-vs-mbunit-vs-mstest-vs-xunit-net
• http://www.csunit.org/
• http://dotunit.sourceforge.net/
• https://launchpad.net/nunit-3.0/ (Still beta version)
• http://xunit.codeplex.com/

Mock Up

• https://sourceforge.net/projects/easymocknet/
• http://www.ayende.com/projects/rhino-mocks.aspx
• http://nmock.sourceforge.net/
• http://code.google.com/p/moq/

Validates test quality

• http://ncover.sourceforge.net/
  • ChristianK uses the version of NCover that is integrated in SharpDevelop 3.2. It works well and is nicely integrated. Combined with the integrated NUnit application it is very handy!
    • Caveat: The test project needs to be a Commandline or Windows Form Project for it to work (DLL projects don't work).
• http://nester.sourceforge.net/

Integration test framework

• http://qaliber.net/testbuilder.php
• http://blogs.msdn.com/b/ericgu/archive/2007/10/09/unit-testing-through-the-ui.aspx (Overview how to access forms)
• http://nunitforms.sourceforge.net/
• http://watin.org/ http://aleksjones.com/blog/?p=44

N/A

• http://code.google.com/p/autofac/ (Library for Inversion of Control / Dependency Injection)
• http://roaster.codeplex.com/ (nunit for compact framework)
• http://dbunit-net.sourceforge.net/ (Write data to database and remove it afterwards)
• http://nbehave.org/ (Write test in almost human English language and map the steps to C# classes)

Unsorted List of Links:

Overview pages

http://www.google.co.uk/search?q=test+case+management+system+opensource+nunit&hl=en&client=opera&hs=rNW&rls=en&channel=suggest&prmd=ivns&ei=JwCfTbyCBJG1hAeX0JWBBQ&start=10&sa=N- - a Goole search which returned rather good results

http://requirementsandtestinghub.wordpress.com/ - overview of and link to following PDF article

http://www.testingexperience.com/testingexperience12_12_10.pdf - Article 'More for less: the growth of open source testing tools' (page 6)

http://www.testingexcellence.com/15-open-source-test-management-tools/ - 15+ Open Source Test Management Tools

http://www.jayphilips.com/2009/08/04/100-open-source-free-functional-testing-tools/ - 100+ Open Source/Free Functional Testing Tools

http://www.opensourcetesting.org/ - Open source software testing tools, news and discussion website

http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks - WikiPedia article on Unit Testing Frameworks

Tools

https://sourceforge.net/projects/qamanager/ - qaManager

https://sourceforge.net/projects/rth/ - RTH - Requirements and Testing Hub

http://code.google.com/p/rth-turbo/ - RTH Turbo

https://wiki.ow2.org/salome-tmf/ - Salome-TMF

https://sourceforge.net/projects/tcw/ - Test Case Web

http://tesly.sourceforge.net/ - Tesly

http://www.majordojo.com/testitool/ - Testitool

http://testlink.sourceforge.net/docs/testLink.php - TestLink

http://www.xqual.com/ - XStudio

http://www.automatedtestinginstitute.com/home/index.php?option=com_content&view=article&id=1344:ati-automation-honors&catid=125:award-honorees&Itemid=1000#OpenSourceTools - ATI Automation Honors - NUnit came 1st, csunit 2nd in the .NET category

http://www.automatedtestinginstitute.com/home/index.php?option=com_content&task=view&id=584&Itemid=185 - Info on csUnit

http://www.csunit.org/ - csUnit