Sample data create and import

From OpenPetra Wiki
Jump to navigation Jump to search

Overview

Goal: creating sample data for the database.

The sample data should have

  • many donors
  • many recipients
  • many donations

Data creation and import is done in a two step process:

alt This diagram shows the data flow of created sample data from benedata benerator into the sample data constructor and the into OpenPetra Server

  1. The 3rd-party data creation tool databene benerator creates raw data (names, addresses, phone numbers...)
  2. The OpenPetra SampleDataConstructor-tool is specifically built to take this data, assemble, and import it into OpenPetra

Related Bugtracking-Issues

Running Sample Data Constructor

Currently, creation of raw sample data and assembling this into the database are still two separate steps which have to be called manually (not integrated via nant yet):

Creating Raw Sample Data

The following assumes that databene benerator is installed and functioning (see below on how to do this). If it is not installed, you may skip this step if you already have working sample data csv-files.

Creating Raw Sample Data with Benerator

Using your command line, please change into the directory containing the sample data constructor source code. In it you find a directory "benerator" containing a file "benerator.xml". Go into this directory.

cd whatever\SampleDataConstructor\benerator

You can run benerator with the following command:

"%BENERATOR_HOME%"\bin\benerator.bat

This should create csv files containing sample data in the directory "SampleDataConstructor\created-data": "PeopleDE","AddressesDE.csv","OrganisationsDE.csv". Please copy these into the directory "(OpenPetraBaseDirectory\tmp\Tests-Exe.SampleDataConstructor".

Inserting Pre-Created Raw Sample Data Files

If you already have files containing sample data ("PeopleDE","AddressesDE.csv","OrganisationsDE.csv"), you can just copy them into the directory "(OpenPetraBaseDirectory)\tmp\Tests-Exe.SampleDataConstructor".

Running Sample Data Constructor

The Sample Data Constructor assembles the raw data, and saves it onto the Petra Server. It requires raw sample data files ("PeopleDE","AddressesDE.csv","OrganisationsDE.csv") to be in the directory "(OpenPetraBaseDirectory)\tmp\Tests-Exe.SampleDataConstructor".

To run the Sample Data Constructor, please go into the directory "(OpenPetraBaseDirectory)\delivery\bin" and run "Tests-Exe.SampleDataConstructor.exe" from there.

After having run the sample data constructor, you can access the created data from the OpenPetra Client.

Current State

  • names and addresses are created, and assembled
  • putting these into the OpenPetra database is in progress (hopefully nearly finished)
  • creating donations is not done yet

Steps:

  1. create raw data as csv files (sufficiently complete!) -- done
  2. import and put together raw data in SampleDataConstructor (for creating relationships) -- nearly done
  3. connect to petra server from temporary program, assemble data as OpenPetra TDS, save -- in progress

Additionally:

  1. create nant job to import data

Installing third party software

Currently the only external software used is Benedata Benerator for creation of the raw data, but this in turn requires the java runtime enviroment.

Java

  • Required: Java Runtime Environment only

Abridged Instructions

PATH = ....; "C:\Program Files (x86)\Java\jre6\bin"

  • Give it a quick test by opening a new command prompt / terminal and typing:

java -version

Example output: 

java version "1.6.0_25"
Java(TM) SE Runtime Environment (build 1.6.0_25-b06)
Java HotSpot(TM) Client VM (build 20.0-b11, mixed mode, sharing)

Databene Benerator

  • Project Website: http://databene.org/databene-benerator.html
  • Note: The project is a java project
  • Used version: 0.6.6 (as of 2011-July - but it may be advisable to use the recent one, we just use simple features anyway)
  • Other requirements: Java Runtime Enviroment

These instructions are largely copied from benerator website and slightly abridged. The original instructions are more detailed and contain details about installing benerator on a Windows / Mac / Linux-System.

Abridged Instructions

  • Download the Benerator distribution from SourceForge
  • Unzip into a directory of your choice (e.g. C:\Program Files\Development\databene-benerator-0.6.6)
  • Create an environment variable BENERATOR_HOME that points to the path you extracted benerator to (example located below).
  • Append BENERATOR_HOME/bin to your environment variable PATH (examples located below)
  • Give benerator a quick test by just opening a new command prompt / terminal and typing

benerator -v

Example Output: 

Local classpath: .; ...
Benerator 0.6.6 build 1255
Java version 1.6.0_25
JVM Java HotSpot(TM) Client VM 20.0-b11 (Sun Microsystems Inc.)
OS Windows 2003 5.2 (x86)
Installed JSR 223 Script Engines:
- Mozilla Rhino[js, rhino, JavaScript, javascript, ECMAScript, ecmascript]

NOTE: I myself could not get benerator to be found from the commandline on one windows development machine (and did not find out why). As a workaround, the following was sufficient to run benerator:

"BENERATOR_HOME%"\bin\benerator -v

Examples for Environment Variables

BENERATOR_HOME = C:\Program Files\Development\databene-benerator-0.6.6

PATH = C:\WINDOWS; ....; %BENERATOR_HOME%\bin


See also

Retrieved from ‘https://wiki.openpetra.org/index.php?title=Sample_data_create_and_import&oldid=2490’