OpenPETRA Technical Implementation Details

(This page has been imported from Petra Wiki in September 2010)

Client-Server Communication

Server notifications sent to the Client

• Firewall would stop the Petra Server communicating with the client when using remote Petra Clients through VPN
• therefore we cannot use .NET Remoting events and callbacks; the connection is not asynchronous, but this is what would be needed for them to work
• our solution
  • Client polls the Petra Server with the 'Keep Alive' signal anyways. On the Server side we have an out parameter in the function that is called from the Keep Alive thread of the Client. This function returns a specific shared object.
    • this shared object is empty (nil) as long as no notification needs to be passed to the Client (which is the normal case)
    • to pass notifications to the Client, the server can create such a shared object and parameterise it in a way that the Client understands
      • an example of such a notification is the refreshing of Cacheable DataTables in all Client's Caches if one Client updates the contents of a Cacheable DataTable (this happens automatically through the single instance of the CacheManager on the server side).
  • advantage
    • works fine through VPN and Firewalls because only the Client makes calls to the Server
  • disadvantage
    • immediate notification of the Client by the Server is not possible - notification time depends on the 'Keep alive' poll rate of the Client

Notification of progress while running time-consuming tasks (eg. Progress Bar)

• Client polls an instance of a special 'Progress Object' (TAsynchronousExecutionProgress) which it received back from a call to a Server Object function that executes a time-consuming task on a separate Thread
  • the 'Progress Object' holds the text and/or percentage
  • polling would usually be done on a separate Thread (not the UI Thread) to keep the UI responsive
• Server calculates progress continuously (text and/or percentage) and updates the properties of the 'Progress Object'
• Client can tell the Server Object to stop execution of time-consuming task before it finished
• the 'Progress Object' holds a state that tells wheter the operation is running, finished, stopped, etc.

Data Caching

Purposes:

• Frequently accessed data gets cached on the Client side to reduce roundtrips to the Server and speed up the Client on slower connections.
• Frequently accessed data gets cached on the Server side as well to reduce DB queries and through that speed up the Server.

Caveats

• Update Lag: Due to the implementation of the Client- and Server-side Cache Managers and the Server-to-Client notification mechanisms, there will be a delay between one Client updating data and other Clients realising this and picking up the new or changed data. In the time span between, clients could display outdated data. This is by design and any usage of data caching needs to factor this in, i.e. critical data that needs to be up-to-date always and at all times for all clients must not be cached!
• Sensitive Data must not be cached: Due to the persistence of the client-side cache on disk for increased performance, any data that is cached must not be sensitive because it could be read out of the files on the PC that contain the persisted data.

Implementation of the Client- and Server-side Cache Managers for Lookup Tables

Client side

• Client has got a Cache Manager class (TDataCache) that can hold specific, named 'cacheable' Typed DataTables.
• The Cache Manager works transparent for the caller (caller doesn't need to worry if data is already cached or not, he just gets it from the Cache Manager because the Cache Manager knows how to load data for a certain 'cacheable' DataTable)
• Client Cache Manager is contacted by some function in the Client to request a DataTable
  • Cache Manager checks whether the requested DataTable is available in memory
    • If it is available in memory, the DataTable is returned to the calling function.
    • If it isn't available in memory, it checks whether it is available in a local file (serialized DataTable)
      • If it is available in a local file, the Cache Manager contacts the Server to check whether the contents of the DataTable are still up-to-date (by sending just a Hash value of the data contained in the DataTable). If this isn't the case, the DataTable is retrieved from the Server and serialized to a local file. The Cache Manager then returns the DataTable to the calling function.
      • If it isn't available in a local file, the Cache Manager contacts the Server to retrieve the DataTable once from the Server and serializes it to a local file. The Cache Manager then returns the DataTable to the calling function.

• In case the database content of a cached DataTable changes
  • Server informs all Clients about this and tells them what DataTable needs refreshing
  • All Clients make note of that. They will refresh the specific cached DataTable in their Cache Managers and local files - but only at the point when this changed DataTable is actually requested by some function in the Client!
  • Clients are responsible to display the changed cached data - this will happen only the next time a screen is opened / a function accesses the changed data

Server side

• Server has got a Cache Manager class (TCacheableTablesManager) that can hold an arbitrary number of DataTables. Any DataTable (Typed DataTable or normal DataTable) can be cached with this class. There is only one instance of this class for all Clients.
• The Cache Manager isn't as transparent as the one on the Client side - the caller will need to know how to load the DataTable if it isn't cached yet (the Cache Manager doesn't know how to load the data for a certain datatable).
• Server Cache Manager is contacted by a function in the Server
  • can be a function that is called by a Client's CachePopulator class (which is called from the Client Cache Manager) (to populate the Client side cache or to perform an up-to-date check of a DataTable in the Client side cache)
  • can be any other function that is executed on behalf of a Client or the Server
  • Cache Manager checks whether the requested DataTable is available in memory
    • If it is available in memory, the DataTable is returned to the calling function.
    • If it isn't available in memory, the caller gets informed about that. The caller will then need to load this DataTable. It will usually want to add it to the Cache so that it is available from the Cache Manager the next time.
• In case the database content of cached DataTable changes
  • The Client that updated the database content of a cached DataTable needs to inform the Client's CachePopulator class on the Server (since we can't trigger a Server function if data is changed in a specific DB table)
  • The Client's CachePopulator class on the Server re-loads the DataTable in the Cache Manager. The Cache Manager informs all Clients (excluding the one who made that call) about this and tells them what DataTable needs refreshing.

Specific Data Caching Implementations

Each of the following implementations uses its own Cache Manager.

• Cache for Lookup Tables (Client side, Server side)
  • used eg. in ComboBoxes, other value lists
  • Client gets notified by the Server if data of a lookup table changes and needs to be reloaded
• Cache for System Defaults (Client side, Server side)
• Cache for User Defaults (Client side, Server side)
• Cache for User/Security information (Client side, Server side)

User Defaults/Preferences

• User Defaults/Preferences are stored in the Petra DB (s_user_default table).
  • not in Windows Roaming Profiles, not in Windows Registry (for OS independence)
  • except for window sizes/positions, which _are_ in local profile (as they apply to the screen size of the local computer).
• User Defaults/Preferences are cached on the Client side to reduce roundtrips to the Server and speed up the Client on slower connections
• The individual s_user_default entries need to follow a naming convention.

Server-side Multithreading and DB Access

Problem

• ADO.NET only allows one concurrent execution of a SQL command
• ADO.NET only allows one transaction per DB connection, ie. one client can have no more than one open transaction

Possible Solution

Could be solved in the DB Access library.

Either

• queue SQL command execution requests if several SQL commands should be run concurrently (simple option, in fact serial execution instead of parallel execution), or
• open several connections per AppDomain as needed if several SQL commands should be run concurrently (favourite option, real parallel execution)
  • use (self-programmed) connection pooling for reuse and closing of connections

Abortion of long-running SQL commands

• TODO: check whether long-running SQL commands can be segmented to be able to interrupt them

Data Verification

• Some verification can be done both by Petra Server and Petra Client: goes into Ict.Petra.Shared.Verification Namespace
• Petra Client has no database connection, can therefore only do verification that do not involve the database
• Petra Server has to pass data verification error information to the Petra Client
• Data Verification routines just return an object with information, the Petra Client needs to display a MessageBox

Task Oriented Petra (Proposal)

Petra would become more of a Workflow System

• Customisable per Petra site by the System Administrator
  • eg. certain steps of multi-step Tasks could be suppressed if they are not applicable to a certain Petra site

Benefits

• Automatic creation of certain Tasks upon detected expiring commitments, reminders, etc.
• Task lists would replace Shepherds
  • more dynamic than Shepherds ever can be!
    • would allow every user to enter data in his/her own preferred order
    • would allow a user to store a task half finished (if we allow this)
    • users would not need to jump over Shepherd pages that don't make sense for their Petra site if a certain step of a multi-step Task was suppressed for their Petra site by the System Administrator
• All users with according privileges would be able to see a specific Task / specific Task Lists
• Tasks could be bound to certain users

Realisation

• Either
  • run background tasks at user login (would require multi-threaded access to the DB), or
  • creation of task list for each user/user group: one 'fake' client that runs on the Petra Server (therefore non-GUI) could regularly login to the Petra Server an invoke creation of a list of tasks (for all users)
• TODO: Decision on one of the two methods must be made (Method #2 is more likely)
• TODO: Details must be worked on

Data Encryption

General

• The TSecureData Class is found in the Ict.Petra.Shared.Security Namespace. It uses a System.Image to provide a large byte array, and an array of string to give indexes into the byte array, providing the actual encryption key.

constructor Create(Text: array of string; Icon: Image);
function GetData(Data: string): string;
function PutData(Data: string): string;

• PutData takes clear text as a parameter and returns cypher text (encoded in base64 for easy storing in a database field). GetData takes the b64-encoded cypher text and returns clear text.
• This class is unforgiving of errors: the programmer must ensure that the string array contains at least 24 characters (giving a 192-bit key) and that indexing into the image by adding the ASCII codes of these characters does not overrun its byte array. The strings should all be 7-bit ASCII.

Encrypted Connection between Server and Client

See here.

Logging

• there is the client log file which is in the /usr/local/samba/dat/petra/tmp22 or c:\petra2\tmp22 directory, PetraClient.log
• the server log file is in /usr/local/petra/PetraServer.log or /usr/local/petra/db_xy/PetraServer.log or c:\petra2\PetraServer.log
• the ServerAdminConsole log file is in /usr/local/petra/PetraServerAdminConsole.log or /usr/local/petra/db_xy/PetraServerAdminConsole.log or c:\petra2\PetraServerAdminConsole.log

• The server log files are controlled via the config files
• The multidb setup will modify the path in the config file
• for compatibility reasons the paths are hardcoded in case there is no log file setting in the config file
• the housekeeping functions will rename the log files and delete old copies after some time