Unit test framework

General

Requirements

• Tests execution should be controllable through nant
• Tests execution should be done by build server
• For writing tests it should be possible to execute separate test cases

How do we want to get test peripherals?

• Write test data into database
  • Set up new, additional test data for each test run
  • Delete database and set up a database snapshot
• Write mock ups, so that database is not needed for the test
• Mixed way of test data in database and mock up

Intuitive feeling: Test data in database by adding test data for each test run, perhaps mixed with some mock ups

Look here to find out how to put data for a specific Unit Test into the database.

kind of tests we want to write

• Tests, which are testing the server code
  • by calling the server functionality directly (in the same process)
  • by connecting as client to the server
• Tests, the client code without UI
• Tests, which are testing by interacting with the UI?

Usage in the past

• Calls a function directly
  • no database
    • Test of TFileDiffMerge
    • Test for parts of ICT.Common
    • Pseudo-Test of some controls
    • yml writer test
  • with database
    • rudimentary sequencer test and db constraint test

• Dialogs automated and asserts rudimentary only on dialogs
  • With database and server
    • reset database and Import Data through nant
      • Import/Export Gift batches
      • GL Tests
    • generates data through GUI with minimal random string
      • GL Forms
    • Expects certain data exists / does not exists
      • MFinance.server.gift
      • Partner import (assert on the number of imported partners only)
• A first concept of some dialog tester classes which can be generated automatically by the yaml files

Putting data for a specific Unit Test into the database

For many Unit Tests it is a requirement to have defined test data in the database from/on which the Unit Test works.

While in some cases the resetting of the database to the base database (by issuing 'CommonNUnitFunctions.ResetDatabase' in the Unit Test) is a workable solution, in other cases specific data will need to be in the database in order to perform the Unit Test. For the latter cases the following options exist:

Option 1: Manually hard-code the program code that creates the necessary records in DataTable(s)/DataSet(s) and call a server-side Method that will create the necessary records in the DB from that DataTable/DataSet

• Hard-code ADO.NET data row creation in the Unit Test, add those data rows to DataTables, optionally add those DataTables to a DataSet and pass those as a DataTable/DataSet Argument of a public Method in a call to the PetraServer. That Method will then create the necessary records in the DB from those DataTable(s)/DataSet(s). (Obviously, the called Method needs to know about the passed-in DataTable(s)/DataSet(s) and needs to know what to do with it!)
• Depending on the nature of the data you are inserting into the database you might need to reset the database to the content of the base database beforehand. This can be done programmatically in the Unit Test by issuing 'CommonNUnitFunctions.ResetDatabase'.
• Example: Method 'TestSaveNewPartnerWithLocation' in file \csharp\ICT\Testing\lib\MPartner\server\PartnerEdit\TestPartnerEdit.cs

Option 2: Import data from a text, CVS or YAML file into the Unit Test and call a server-side Method that can process the data and which will create the necessary records in the DB

• Use 'CommonNUnitFunctions.LoadCVSFileToString' for loading of an arbitrary text file that is then passed as a string Argument of a public Method in a call to the PetraServer. That Method needs to that act as an import mechanism (e.g. TGLTransactionWebConnector.ImportGLBatches). (Obviously, the called Method needs to understand the passed-in string and needs to know what to do with it!)
• Depending on the nature of the data you are inserting into the database you might need to reset the database to the content of the base database beforehand. This can be done programmatically in the Unit Test by issuing 'CommonNUnitFunctions.ResetDatabase'.
• Example: Method 'Test_01_GL_Import' in file \csharp\ICT\Testing\lib\MFinance\GL\Test.GL.Import.cs

Option 3: Execute SQL from a file in the Unit Test to create the the necessary records directly in the DB

• Use 'CommonNUnitFunctions.LoadTestDataBase' for loading SQL INSERT queries (which can be created from within e.g. pgAdmin III) and to have those executed against a (PostgreSQL) DB.
• Depending on the nature of the data you are inserting into the database you might need to reset the database to the content of the base database beforehand. This can be done programmatically in the Unit Test by issuing 'CommonNUnitFunctions.ResetDatabase'.
• Example: Method 'PrepareTestCaseData' in file \csharp\ICT\Testing\lib\MFinance\GL\Test.GL.CommonAccountingTool.cs

How to write a test

1. Sit down and think, what functionality you have implemented
2. Write down the different use cases - each use case generates more than one test
3. Write down, what needs to be tested. There are always following cases:
   1. Good case in middle range of the possible values
   2. Good case next to the limits of the possible values in all directions
   3. Bad Case in middle range of the not allowed values
   4. Bad Case next to the limits of the not allowed values
4. Write down, on what outcome you should assert (Remember: We are doing white box tests and look INTO the software!)
   1. in the database
   2. in the state of the server (if in the same process)
   3. in the state of the client (if client is used)
5. Write down, which test data you need and how you could generate it
   1. See Test Method Parameterisation for test data that doesn't come from the database
6. Implement the test. Do the test in three stages:

   Testing through Client GUI should be avoided, if possible. It is better to change the client, so that this is possible

   Always ensure, that you on later stages you do not re-test earlier stages. Sometimes it is sufficient to show, that the function is correctly called!

   1. Test without DB, whenever possible
   2. Test inside the server process
   3. Test like a client without GUI
7. Get your test working

Need to look into

Interesting concepts

• http://blog.benhall.me.uk/2010/07/improving-testability-with-castle.html Dependency injection even for configuration

xUnit like frameworks

• http://www.nunit.org/
  • ChristianK uses the version of NUnit that is integrated in SharpDevelop 3.2, as well as the standalone NUnit application. The SharpDevelop-bundled version works well and is nicely integrated into the IDE, but doesn't offer the full feature set of the standalone NUnit application. Combined with the integrated NCover application it is very handy, though!
    • Caveat: The test project needs to be a Commandline or Windows Form Project for it to work (DLL projects don't work).
  • Caveat: https://sourceforge.net/apps/phpbb/openpetraorg/viewtopic.php?p=366 talks about issues with the setup method
• http://gallio.org/ ( http://www.mbunit.com/ is part of it)
  • Mirror concept (Interface to reflection for accessing private properties and methods)
  • http://stackoverflow.com/questions/3678783/mbunit-vs-nunit
  • http://stackoverflow.com/questions/261139/nunit-vs-mbunit-vs-mstest-vs-xunit-net
• http://www.csunit.org/
• http://dotunit.sourceforge.net/
• https://launchpad.net/nunit-3.0/ (Still beta version)
• http://xunit.codeplex.com/

Mock Up

• https://sourceforge.net/projects/easymocknet/
• http://www.ayende.com/projects/rhino-mocks.aspx
• http://nmock.sourceforge.net/
• http://code.google.com/p/moq/

(Ok, Thias a mock up can be used to implement a non existing partner module if the routines are required in the financial module. But I recommend a discussion "mock up yes or no" before we decide to select one of them. I think a mock up is a good idea if you are in the design phase of one module while you are realizing the other. But here? Wolfganguhr 06:45, 12 May 2011 (UTC))

Validates test quality

• http://ncover.sourceforge.net/
  • ChristianK uses the version of NCover that is integrated in SharpDevelop 3.2. It works well and is nicely integrated. Combined with the integrated NUnit application it is very handy!

Thias we have actually decided not to run tests with full code coverage ... Wolfganguhr 06:48, 12 May 2011 (UTC)

• • • Caveat: The test project needs to be a Commandline or Windows Form Project for it to work (DLL projects don't work).
• http://nester.sourceforge.net/

Integration test framework

• http://qaliber.net/testbuilder.php

I suggest to test this something more detailed. I've had some experience with an old test software QaRun from Compuware and this solution was not good. But if it works, it is a very nice tool that may help us to integrate other testers. Such tests can be made by experienced users too and of something works wrong they only have to create a test and to attach it to the error message. Wolfganguhr 07:20, 12 May 2011 (UTC)

• http://blogs.msdn.com/b/ericgu/archive/2007/10/09/unit-testing-through-the-ui.aspx (Overview how to access forms)
• http://nunitforms.sourceforge.net/
• http://watin.org/ http://aleksjones.com/blog/?p=44

N/A

• http://code.google.com/p/autofac/ (Library for Inversion of Control / Dependency Injection)
• http://roaster.codeplex.com/ (nunit for compact framework)
• http://dbunit-net.sourceforge.net/ (Write data to database and remove it afterwards)

May be that is something that will solve our problems. In our test cases we always have to integrate test data depending on the situation and even wrong test data in order to create an error message. Actually this is a sql file created by the data base and imported by nant. But this only works for one database and the actual realization is something tricky. Wolfganguhr 06:59, 12 May 2011 (UTC)

• http://nbehave.org/ (Write test in almost human English language and map the steps to C# classes)

Unsorted List of Links:

Overview pages

http://www.google.co.uk/search?q=test+case+management+system+opensource+nunit - a Goole search which returned rather good results

http://requirementsandtestinghub.wordpress.com/ - overview of and link to following PDF article

http://www.testingexperience.com/testingexperience12_12_10.pdf - Article 'More for less: the growth of open source testing tools' (page 6)

http://www.testingexcellence.com/15-open-source-test-management-tools/ - 15+ Open Source Test Management Tools

http://www.jayphilips.com/2009/08/04/100-open-source-free-functional-testing-tools/ - 100+ Open Source/Free Functional Testing Tools

http://www.opensourcetesting.org/ - Open source software testing tools, news and discussion website

http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks - WikiPedia article on Unit Testing Frameworks

Tools

https://sourceforge.net/projects/qamanager/ - qaManager

https://sourceforge.net/projects/rth/ - RTH - Requirements and Testing Hub

http://code.google.com/p/rth-turbo/ - RTH Turbo

https://wiki.ow2.org/salome-tmf/ - Salome-TMF

https://sourceforge.net/projects/tcw/ - Test Case Web

http://tesly.sourceforge.net/ - Tesly

http://www.majordojo.com/testitool/ - Testitool

http://testlink.sourceforge.net/docs/testLink.php - TestLink

http://www.xqual.com/ - XStudio

http://www.automatedtestinginstitute.com/home/index.php?option=com_content&view=article&id=1344:ati-automation-honors&catid=125:award-honorees&Itemid=1000#OpenSourceTools - ATI Automation Honors - NUnit came 1st, csunit 2nd in the .NET category

http://www.automatedtestinginstitute.com/home/index.php?option=com_content&task=view&id=584&Itemid=185 - Info on csUnit

http://www.csunit.org/ - csUnit