Specifications Partner Import

Purpose

There are many situations where the user needs to import Partner addresses and personal details into their instance of the OpenPetra database:

• The Personnel department receives the data of a new member of staff from the sending field.
• Data is collected on a webserver (satellite database) from partners that have entered their own details. This data must be imported into the office database (master database).
• Data from legacy databases should be easy to import when migrating to OpenPetra.
• Generated test data can be imported this way too.

Challenges

• partners might already exist in the OpenPetra database. We do not want duplicate partners after the import.
• addresses might already exist in the OpenPetra database, which should be reused.
• we need fuzzy matching, since partners might write their address slightly different, with or without spaces, Street or St., etc.
• If an address changes for a partner, should we ask if the address of other partners at that location should be changed too? What happens to the old address, will it be made invalid by setting the expiring date?
• Partners might have entered wrong data, which needs to be monitored and filtered before importing into the office database.
• Relationships between partners should be imported to. So for an applicant, you want to import reference partners as well, and there must be a connection created by the import.
• Once data has been imported, we do not want to manually review the same data again. So the web database should know the partner key in the office database, fixed data should be committed back into the web database etc.

Design

• The GUI for the Partner Import screen is able to pull all data of a partner from a file, or web service, etc.
• The user decides whether to create a new partner for this partner, attach it to an existing family, create a new address, ignore certain data etc.
• The partner is exported again from the master database, and the hash of the import is created. Both are sent back to the satellite database. The partner in the satellite database is updated, and the partner key is merged with the partner key from the master database (I am thinking of a new type of merging, with just connecting partner keys).

• web service: recognizes which partners have already been imported. They don't have a hash tag yet? the hash tag does not match the current data? The changed date is newer since the last import?

• automated import: if we do an import into a clean database, or we are sure we won't need to interact, it should be possible to just import the partners in one go. This can be useful for importing test data, or initialising a database during migration from a legacy system.