Specification for Mail Merge (former 'Form Letters' in Petra 2.x)

WORK IN PROGRESS

THIS PAGE IS WORK IN PROGRESS!

Introduction

The proposed 'Mail Merge' functionality in OpenPetra should allow users to create a series of documents that are a merge between layout definition and data for the purpose of creating various 'mailings' to (usually multiple) recipients. Placeholders in the layout-defining document will be substituted with data, resulting in documents which are different for each Partner.
Such documents would typically be letters, but could really be any formatted document. The documents could end up printed, as a PDF document, or could get e-mailed to the recipients.

Background Information

What we call the upcoming ‘Mail Merge’ functionality in OpenPetra is called ‘Form Letters’ in Petra 2.x, the predecessor of OpenPetra.

Also, Petra 2.x contains the 'Form Design' facility in the Finance Module, which is used for receipting and cheque printing and in principle does much the same than 'Form Letters' in the Partner Module, but it can deal with (complex) finance data.
The 'Form Design' facility also needs to be replaced by the ‘Mail Merge’ functionality in OpenPetra. Caveat: 'Form Design' in the Finance Module is done technically different than 'Form Letters' in the Partner Module (ie. different program code lies behind those features) - it might not seem so from the output, though!

Petra 2.x Implementation - 'Form Letters'

The way how the Form Letters work is powerful, flexible, supports good re-use and includes good abilities to parameterise a lot of things, but it is overly technical, cumbersome to use and not liked by most Petra 2.x users. To put it bluntly, many Petra 2.x users actually ‘fear’ the Form Letters facility because the concept behind the whole functionality is so abstract and technical! Form Letters is an area in Petra 2.x that is not easy to grasp for the average person who works on a PC in an office, and it is rather difficult to ‘master it’ even for quite advanced users in an office. (The whole concept might not look too bad when seen through software engineers' eyes, but software engineers are technicians and not average users in an office who just want ‘to get that mailing out’!)

To summarise the approach of Petra 2.x: it allows the re-use of definitions of Form Letter Bodies for various Mail Merges and allows to have them parameterised using the ‘Form Letter Design’ approach, and then further at print time with ‘Inside Address’, “Sender’s Name” and ‘Closing Text‘ and other layout options, plus it has the ability to utilise ‘Inserts’ to produce different texts or text sections for different Partners (and Inserts can be done in two ways!). That is where the flexibility of the Form Letter approach lies, but at the same time the complexity! It is suggested that one skim-reads the whole Chapter 6 of the Partner Module User Guide of Petra 2.x to get an overview of what can be done with Petra 2.x' Form Letter implementation and how it is achieved. That manual can be downloaded from here: http://www.openpetra.org/en/user-manuals

Primary Goals of the Proposed OpenPetra Implementation - ‘Mail Merge‘

With OpenPetra’s Mail Merge implementation we want to significantly reduce the complexity of what is known as Form Letters in Petra 2.x. At the same time we want to give the users more and better control over the layout, and especially take the pain out of the creation and maintenance of the layout of a mail merge document. To be able to achieve that we will need to change the approach that was taken in Petra 2.x quite drastically and quite likely remove some of the advanced functionalities which make the Petra 2.x approach that technical and complex.

The approach we want to take in OpenPetra is to be based on ‘mail merge documents’, which is the approach that is taken in standard word processing applications for PC’s (e.g. MS Word or OpenOffice Writer).

Disambiguation From Other 'Similar-sounding' Petra 2.x Features

Two of Petra 2.x's features sound as if they would be related to what will become the Mail Merge functionality of OpenPetra. However, they have a different purpose and are not to be confused with what should get implemented in OpenPetra!

Extract Mail Merge

Facility to export the relevant data in a form that one can then import into a word processor, spreadsheet, etc. in CSV format. Going that route rather than using Form Letters allows users greater flexibility in layout, etc. because they can use a word processor for the creation of the layout and the process of mail merging. That functionality is often just referred to as 'mail merge' in Petra 2.x!

Mailings

The Mailings Table contains a list of mailing events that one wishes to remember (to see the results). For example one may send out a special appeal for funds for a new building project and may wish to track the response to this particular letter. Also, a Partner Contact can be recorded for all Partners in a particular mass mailing by choosing a particular Mailing.

Implementation

The implementation will need to be done in several phases as Mail Merge represents a bigger subset of the features of OpenPetra, in fact it is seen as a Sub-Module of the Partner Module and will need to be implemented in that way.

Initial Plan for the Phases

Phase I

Investigation of Various Technologies for the Core Mail Merge Functionalities

Investigations into technologies that will be needed for supporting the biggest parts of the new implementation, namely the creation and parameterisation of the Mail Merge Document and the production of the Merged Documents for the individual recipients out of the underlying Mail Merge Data Set are to be undertaken.

Required Outcome

The required outcome of that process is that one technology will be identified that

• satisfies the general and OM-specific business requirements that will be gathered from key users;
• satisfies the layout requirements (at least to a very large and acceptable degree);
• fulfills the technical requirements that will be drawn up;
  • incl. that the technology is seen 'fitting' to OpenPetra from a technical standpoint;
• a migration path for existing Petra 2.x Form Letters can be done for;
• that can be made to work in a suitable time frame.

Establishment of the Migration Path for Existing Form Letters

A challenge in changing the approach of how mail merging is done in Petra 2.x is that we will need to have a migration path to the new approach in OpenPetra.
The migration path will need to be approved by Project Management before Phase II can be started!

The desired outcome is that the details of the migration path are identified and documented and a rough technical design for the implementation will be written.

• Basically, Form Letters that are designed in an OM office with Petra 2.x will need to be accessible in some format in OpenPetra so they are not lost to an OM office that migrates from Petra 2.x to OpenPetra.
  
• That could mean various things, such as
  • merely having the ability to be able to look at examples of combinations of various Form Bodies and settings in Petra 2.x that are 'printed' one-off into PDF documents at the time of data conversion so users in an office can see how they looked like in Petra 2.x; that would mean that
    • users would have no access anymore to the data that defined the Form Letter in Petra 2.x
    • the OM offices' OpenPetra installation would not contain any of their previous form letters as Mail Merge Documents.
  • at the time of data conversion Mail Merge Documents would be generated in OpenPetra that resemble combinations of various Form Bodies and settings in Petra 2.x; that would mean that
    • users would have no access anymore to the data that defined the Form Letter in Petra 2.x
    • the OM offices' OpenPetra installation would contain Mail Merge Documents that resemble what they had set up as Form Letters in Petra 2.x
  • etc. TODO

Assigned To: MarkP (with support of ChristianK [technology investigation] and WolfgangB [migration path])

TODO

Gathering of Requirements from Petra 2.x 'key users'

We will need to find 'key users' in OM that will help drawing up the business requirements that exist in typical OM offices today.

The desired outcome is that business requirements will be identified and documented and that advanced functionalities of Petra 2.x’s Form Letter implementation that we could afford to not take over into OpenPetra (if any) are identified.

Assigned To: WolfgangB (with support of AlwynV)

TODO

Phase II

Implementation of the OpenPetra Mail Merge Sub-Module

TODO

Creation of migration program for existing Petra 2.x Form Letters

TODO

Phase III

Testing

TODO

Approval

TODO

Requirements

Technological Requirements

Mail Merge Specification: Technological Req'mts

Technical Requirements

Mail Merge Specification: Technical Req'mts: Placeholders
Mail Merge Specification: Technical Req'mts: Mail Merge Data Set
TODO

Layout Requirements

Mail Merge Specification: Layout Requirements

Business Requirements (General)

Mail Merge Specification: Business Requirements (General)

Business Requirements of OM Offices

Mail Merge Specification: Business Requirements (OM Offices)

Terminology

Throughout this wiki page and in the wiki pages that are linked from this wiki page the following words have special meaning in the context of Mail Merge. Their meaning is explained here.

• Mail Merge Document: A document that defines the overall layout plus all of the content of the Merged Documents that isn't changed through the mail merge process. Placeholders in Mail Merge Documents will be substituted with data from the Mail Merge Data Set in the process of the Production of Mail-merged Documents, resulting in the Merge Result.

• Mail Merge Data Set: Data that is determined at the time of Production of Mail-merged Documents (some data items may be calculated). That term describes the data for all the Partners that are to be processed in the Production of Mail-merged Documents (that is, 1..n instances of Partner Mail Merge Data).

• Partner Mail Merge Data: Data that is linked with one of the Partners that are processed in the Production of Mail-merged Documents and therefore the data that is the basis for a single Merged Document.

• Placeholders: Placeholders are specially set-aside words ('Markup' or 'Fields') that define places in the Mail Merge Documents where Placeholders should be substituted with data from the Mail Merge Data Set. There are various kinds of Placeholders for different purposes.

• Parameterised Placeholders: Placeholders to which parameters can be passed. The behaviour of the Placeholder will change in some way depending on the parameter(s) passed to it, depending on the kind of placeholder and the parameter value(s). The parameter(s) will be evaluated during the Production of Mail-merged Documents.

• Production of Mail-merged Documents, Mail Merge Run: The process of taking Partner data of selected Partners to produce the Mail Merge Data Set and to replace Placeholders in a specified Mail Merge Document with Partner Mail Merge Data (once for each Partner), yielding the Merge Result. Merged Documents can be produced for many Partners at once using an Extract for the Production of Mail-merged Documents, or for a single Partner by selecting that Partner for the Production of Mail-merged Documents.

• Merge Result: The result of the Production of Mail-merged Documents: 1..n Merged Documents.

• Merged Document: A single document of the 1..n documents that make up the Merge Result. Placeholders in the Mail Merge Document that was chosen for the Production of Mail-merged Documents will have been replaced with the Partner Mail Merge Data that belongs to the single document - that makes the Merged Document the ' end product ' that the user wants to create for each Partner!

• Partner(s): In principle the same meaning as in the rest of Petra 2.x/OpenPetra. However, when this set of documentation talks about Partners it focuses on the data associated with the Partner and the use of that data in the Production of Partner Mail Merge Data.

• Extract(s): In principle the same meaning as in the rest of Petra 2.x/OpenPetra. However, when this set of documentation talks about Extracts it focuses on the 0..n Partners that are contained in the Extract (=that the Extract points to) and the use of that data in the production of the Mail Merge Data Set.