Unit test framework
General
Requirements
- Tests execution should be controllable through nant
- Tests execution should be done by build server
- For writing tests it should be possible to execute separate test cases
How do we want to get test peripherals?
- Write test data into database
- Set up new, additional test data for each test run
- Delete database and set up a database snapshot
- Write mock ups, so that database is not needed for the test
- Mixed way of test data in database and mock up
Intuitive feeling: Test data in database by adding test data for each test run, perhaps mixed with some mock ups
kind of tests we want to write
- Tests, which are testing the server code
- by calling the server functionality directly (in the same process)
- by connecting as client to the server
- Tests, the client code without UI
- Tests, which are testing by interacting with the UI?
Usage in the past
- Calls a function directly
- no database
- Test of TFileDiffMerge
- Test for parts of ICT.Common
- Pseudo-Test of some controls
- yml writer test
- with database
- rudimentary sequencer test and db constraint test
- no database
- Dialogs automated and asserts rudimentary only on dialogs
- With database and server
- reset database and Import Data through nant
- Import/Export Gift batches
- GL Tests
- generates data through GUI with minimal random string
- GL Forms
- Expects certain data exists / does not exists
- MFinance.server.gift
- Partner import (assert on the number of imported partners only)
- reset database and Import Data through nant
- With database and server
- A first concept of some dialog tester classes which can be generated automatically by the yaml files
How to write a test
- Sit down and think, what functionality you have implemented
- Write down, what needs to be tested. There are always following cases:
- Good case in middle range of the possible values
- Good case next to the limits of the possible values in all directions
- Bad Case in middle range of the not allowed values
- Bad Case next to the limits of the not allowed values
- Write down, on what outcome you should assert (Remember: We are doing white box tests and look INTO the software!)
- in the database
- in the state of the server (if in the same process)
- in the state of the client (if client is used)
- Write down, which test data you need and how you could generate it
- Implement the test. Do the test in three stages:
- Testing through Client GUI should be avoided, if possible. It is better to change the client, so that this is possible
- Always ensure, that you on later stages you do not re-test earlier stages. Sometimes it is sufficient to show, that the function is correctly called!
- Test without DB, whenever possible
- Test inside the server process
- Test like a client without GUI
- Get your test working
Need to look into
Interesting concepts
- http://blog.benhall.me.uk/2010/07/improving-testability-with-castle.html Dependency injection even for configuration
xUnit like frameworks
- http://www.nunit.org/
- ChristianK uses the version of NUnit that is integrated in SharpDevelop 3.2, as well as the standalone NUnit application. The SharpDevelop-bundled version works well and is nicely integrated into the IDE, but doesn't offer the full feature set of the standalone NUnit application. Combined with the integrated NCover application it is very handy, though!
- Caveat: The test project needs to be a Commandline or Windows Form Project for it to work (DLL projects don't work).
- ChristianK uses the version of NUnit that is integrated in SharpDevelop 3.2, as well as the standalone NUnit application. The SharpDevelop-bundled version works well and is nicely integrated into the IDE, but doesn't offer the full feature set of the standalone NUnit application. Combined with the integrated NCover application it is very handy, though!
- http://gallio.org/ ( http://www.mbunit.com/ is part of it)
- Mirror concept (Interface to reflection for accessing private properties and methods)
- http://stackoverflow.com/questions/3678783/mbunit-vs-nunit
- http://stackoverflow.com/questions/261139/nunit-vs-mbunit-vs-mstest-vs-xunit-net
- http://www.csunit.org/
- http://dotunit.sourceforge.net/
- https://launchpad.net/nunit-3.0/ (Still beta version)
- http://xunit.codeplex.com/
Mock Up
- https://sourceforge.net/projects/easymocknet/
- http://www.ayende.com/projects/rhino-mocks.aspx
- http://nmock.sourceforge.net/
- http://code.google.com/p/moq/
Validates test quality
- http://ncover.sourceforge.net/
- ChristianK uses the version of NCover that is integrated in SharpDevelop 3.2. It works well and is nicely integrated. Combined with the integrated NUnit application it is very handy!
- Caveat: The test project needs to be a Commandline or Windows Form Project for it to work (DLL projects don't work).
- ChristianK uses the version of NCover that is integrated in SharpDevelop 3.2. It works well and is nicely integrated. Combined with the integrated NUnit application it is very handy!
- http://nester.sourceforge.net/
Integration test framework
- http://qaliber.net/testbuilder.php
- http://blogs.msdn.com/b/ericgu/archive/2007/10/09/unit-testing-through-the-ui.aspx (Overview how to access forms)
- http://nunitforms.sourceforge.net/
- http://watin.org/ http://aleksjones.com/blog/?p=44
N/A
- http://code.google.com/p/autofac/ (Library for Inversion of Control / Dependency Injection)
- http://roaster.codeplex.com/ (nunit for compact framework)
- http://dbunit-net.sourceforge.net/ (Write data to database and remove it afterwards)
- http://nbehave.org/ (Write test in almost human English language and map the steps to C# classes)
Unsorted List of Links:
Overview pages
http://www.google.co.uk/search?q=test+case+management+system+opensource+nunit&hl=en&client=opera&hs=rNW&rls=en&channel=suggest&prmd=ivns&ei=JwCfTbyCBJG1hAeX0JWBBQ&start=10&sa=N- - a Goole search which returned rather good results
http://requirementsandtestinghub.wordpress.com/ - overview of and link to following PDF article
http://www.testingexperience.com/testingexperience12_12_10.pdf - Article 'More for less: the growth of open source testing tools' (page 6)
http://www.testingexcellence.com/15-open-source-test-management-tools/ - 15+ Open Source Test Management Tools
http://www.jayphilips.com/2009/08/04/100-open-source-free-functional-testing-tools/ - 100+ Open Source/Free Functional Testing Tools
http://www.opensourcetesting.org/ - Open source software testing tools, news and discussion website
http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks - WikiPedia article on Unit Testing Frameworks
Tools
https://sourceforge.net/projects/qamanager/ - qaManager
https://sourceforge.net/projects/rth/ - RTH - Requirements and Testing Hub
http://code.google.com/p/rth-turbo/ - RTH Turbo
https://wiki.ow2.org/salome-tmf/ - Salome-TMF
https://sourceforge.net/projects/tcw/ - Test Case Web
http://tesly.sourceforge.net/ - Tesly
http://www.majordojo.com/testitool/ - Testitool
http://testlink.sourceforge.net/docs/testLink.php - TestLink
http://www.xqual.com/ - XStudio
http://www.automatedtestinginstitute.com/home/index.php?option=com_content&view=article&id=1344:ati-automation-honors&catid=125:award-honorees&Itemid=1000#OpenSourceTools - ATI Automation Honors - NUnit came 1st, csunit 2nd in the .NET category
http://www.automatedtestinginstitute.com/home/index.php?option=com_content&task=view&id=584&Itemid=185 - Info on csUnit
http://www.csunit.org/ - csUnit