Evaluation and Comparison of different logging Systems

Based on mantis ID 62 this article contains a report of a comparison of three different logging systems in order to preprae a relevant developement decision. The three different systems are:

1. The openpetra internal logging system called TLogging
2. The Log4Net system from Apache
3. The nLog system

In the very first step only the documentation is refered because this has led to a predecison against nLog. In the next step some log4net benchmarks has been done in order to get some experience with log4net.

Comparison of nlog and log4net

First Impression

Both systems ([Apache log4net] and [nLog]) are something similar. Both systems provide an Interface to [Chainsaw] and both systems provide a message system to a remote instance. Both systems handle the log-levels DEBUG, INFO, WARN, ERROR and FATAL and both systems can handle an output to a file or a set of files, to a database or to a xml-report or else and simultaneusly.

Furthermore both systems can be configured by external xml-files and programmatically. And of course both systems can be used by mono. In Mono we cannot use the log4net.Appender.ColoredConsoleAppender but the log4net.Appender.ConsoleAppender (I think that is something we can live with), and it makes no sense to use the NetSendAppender and the OutputDebugStringAppender. The NetSendAppender writes logging events to the Windows Messenger service and this only can be used by using a .net framework and the OutputDebugStringAppender sends the message to a Debugger. This interface actually is used by Microsoft Software only.

The OutputDebugStringAppender may be an interesting option for troubleshooting. May be we can switch between frameworks ...

NLog itself does not provide such an easy feature overview. Referring [this page] you can find different versions of nlog. If you open either the [Version for mono] and [for .net 4.0] you will see the problem. You have to study the documentation in detail to see what is going on and what is not going on.

The apache solution is not a standalone solution but [one of four logging services] for java, c++, c# and php and so experience on one of thoose frameworks will help to understand the others.

Referring to the [Apache-FAQ] there is a set of requirements for a logging system.

Base Requirements

Reliability

Reliability means that a system logs something before an action is performed. Typical solutions are the log files of a web server. If the process does not have the privilege to write something to a file an error message is shown and the site is not delivered to the client.

This means that an Exception has been thrown if the logging does not work. A troubleshooting log system mustn't fulfill such an requirement. It's not a good idea to troubleshoot your software and find a problem in your logging system only.

1. log4net is not reliable. it reclaims to be a "best-effort and fail-stop logging system"
2. nlog does not provide any information about it

Thread Safetyness

Both systems are thread safe.

Example for detailed differences: Remote-Syslog

I've not investigated all differences but only three and I've described it only for one example because the following differences are typical.

Apache knows two Remote-Classes, [the RemotingAppender] and the [RemoteSyslogAppender].

The first provides a communication to a client providing a [IRemoteLoggingSink-Interface] and the second a very common and convenient udp-Interface. The documentation is reduced down to the most important facts.

The nlog provides "Targets" and among a set of targets there exists a DatabaseTarget Class and a DatabaseParameterInfo Class. But the Database parameters itself are not a target. They are used to define a specific database target. So the hierarchical code structure is something strange.

First Decision between nLog and log4net

In this case the decision is relative simple. We'll take log4net.

Log4net benchmarks

The benchmark tests

Basic idea of this test is to get some ideas about the using of the system, and the relative performance to TLogging. The basic question is: Makes it sense to change or not?

The openpetra sources contains source tree (csharp\ICT\Testing) which handles different software tests. Inside of this tree. A class is added to csharp\ICT\Testing\Common\Logging.Benchmark.Test.cs. This test is not a unit test in a convenient way because no assert is realy done. Insted of this the System.Diagnostics.Stoppwatch is used to measure the time which is necessary to log 10.000 lines one by another.

The base routine is:

stopwatch.Start();
 // Do 10.000 Logs
stopwatch.Stop();
// Log the time in Millisecons

The different tests are:

TLogging

for (int i=0; i<10000; i++)
  {
    TLogging.Log("Entry: " + i);
  }

Log4NetDirect

 for (int i=0; i<10000; i++)
   {
     log.Debug("Entry: " + i);
   }

LogOnLevelDependency

 for (int i=0; i<10000; i++)
   {
     if (log.IsDebugEnabled) log.Debug("Entry: " + i);
   }

UseConsole

 for (int i=0; i<10000; i++)
   {
     System.Console.WriteLine("Entry: " + i);
   }

First differences

TLogging only knows one Log-Level and you can use all or nothing. In log4net you have the log-commands log.Debug("...") log.Info("..."), log.Warn(..."), log.Error("..." ) and log.Fatal("..."). In our example debug is used and benchmarks are done by the levels DEBUG and INFO and so we can measure the non logging speed too. UseConsole and TLogging will define a reference level and Log4NetDirect is used for a direct measurement. If a log entry requres a complex string it may be a good idea to check if a string is required to log and so the LogOnLevelDependency test has been created.

We are testing the performance to write the data to the console. In this case the speed depends if the console is visible or not. 10.000 Log are 10.000 Screen updates and sometimes it is a new line, a scroll or a scroll bar update only. If you want to do your own tests, you shall make shure, that the conditions are allways the same.

Used Files

The log4net-files which are part of the apache software are placed in csharp\ThirdParty\log4net. Furthermore the file csharp\ICT\Testing\Common\Logging.Benchmark.Test.cs contains the tests and the file Tests.Common.xml in the same directory contains the log4net configuration. The file csharp\ICT\Testing\_bin\Debug\Tests.Common.dll is generated by Sharpdevelop and can be loaded by NUnit to run the performance tests.

Log Results

The picture shows Nant and the four "Tests". The border of the two logs lies betweeen TLogging and Log4NetDirect.

The first difference is the amount of information. The logfile contains the date and time and also the time in milliseconds (2011-01-05 15:22:52,078) the the name of the thread which is actually logging ([TestRunnerThread]), the log level (DEBUG) the namespace (Tests.Common.Logging_Benchmark_Test), a value named ndc (nested diagnostic contexts) - this value is acutally "[(null)]" and the logging text (Entry: 0).

The ndc may be deprecated and so this is acutally not investigated. First we have to decide to use log4net or not. Then we can investigate this point (or not). The most important point at all is that there is something more information in the log and this file provide a better structure so that you can use chainsaw.