Sample data create and import: Difference between revisions

From OpenPetra Wiki
Jump to navigation Jump to search
No edit summary
 
(15 intermediate revisions by 3 users not shown)
Line 9: Line 9:
Data creation and import is done in a two step process:
Data creation and import is done in a two step process:


# The 3rd-party data creation tool [http://databene.org/databene-benerator.html benedata benerator] creates raw data (names, addresses, phone numbers...)
[[File:SampleDataCreation.png|600px|alt This diagram shows the data flow of created sample data from benedata benerator into the sample data constructor and the into OpenPetra Server]]
# The OpenPetra SampleDataConstructor-tool is specifically built to take this data, assemble, and import it into OpenPetra  


[[File:SampleDataCreation.png|800px|alt This diagram shows the data flow of created sample data from benedata benerator into the sample data constructor and the into OpenPetra Server]]
# The 3rd-party data creation tool [https://www.benerator.de/ benerator] creates raw data (names, addresses, phone numbers...)
# The OpenPetra SampleDataConstructor-tool is specifically built to take this data, assemble, and import it into OpenPetra


== Related Bugtracking-Issues ==
== Related Bugtracking-Issues ==
Line 19: Line 19:
* import this data ([https://sourceforge.net/apps/mantisbt/openpetraorg/view.php?id=220 issue #220])
* import this data ([https://sourceforge.net/apps/mantisbt/openpetraorg/view.php?id=220 issue #220])


== Current State ==
== Running Sample Data Constructor ==
Creation of raw sample data and assembling this into the database are two separate steps which can be executed with nant:
 
=== Creating Raw Sample Data ===
The following assumes that databene benerator is installed and functioning (see below on how to do this). If it is not installed, you may skip this step if you already have working sample data csv-files.
==== Creating Raw Sample Data with Benerator ====
To start the generation of data, run:
nant generateDemodata
 
This should create csv files containing sample data in the directory <code>demodata\generated</code>: "People.csv","Addresses.csv","Organisations.csv".
 
=== Running Sample Data Constructor ===
 
The Sample Data Constructor assembles the raw data, and saves it onto the Petra Server. It requires raw sample data files ("People.csv","Addresses.csv","Organisations.csv") to be in the directory <code>demodata\generated</code>.


* names and addresses are created, and assembled
To run the Sample Data Constructor, run:
* putting these into the OpenPetra database is in progress (hopefully nearly finished)
nant resetDatabase
* creating donations is not done yet
nant importDemodata


Steps:
'''Note''': It is necessary to reset the database content with <code>nant resetDatabase</code> before running <code>nant importDemodata</code> as otherwise the RDBMS's Referential Integrity checks will throw duplicate key Exceptions when Units are created!


# create raw data as csv files (sufficiently complete!) -- done
After having run the sample data constructor, you can access the created data from the OpenPetra Client.
# import and put together raw data in temporary program (for creating relationships) --done
# connect to petra server from temporary program, assemble data as OpenPetra TDS, save --in progress


Additionally:
== Current State ==


# create nant job to import data
* creating names and addresses and putting these into the OpenPetra database is completed
* creating donations is not done yet


== Installing third party software ==
== Installing third party software ==
Line 45: Line 57:
=== Abridged Instructions ===
=== Abridged Instructions ===


* Download the [http://www.java.com/de/download/|current Java Runtime environment] and install
* Download the [http://www.java.com/de/download/ current Java Runtime environment] and install
* You might need to add the bin directory to your PATH:
* You might need to add the bin directory to your PATH:


Line 66: Line 78:
=== Databene Benerator ===
=== Databene Benerator ===


* Project Website: http://databene.org/databene-benerator.html
* Project Website: https://www.benerator.de/
* Note: The project is a java project
* Note: The project is a java project
* Used version: 0.6.6 (as of 2011-July - but it may be advisable to use the recent one, we just use simple features anyway)
* Used version: 0.7.1 (as of 2011-November)
* Other requirements: Java Runtime Enviroment
* Other requirements: Java Runtime Enviroment


These instructions are largely copied from [http://databene.org/databene-benerator/installation.html|the benerator website] and slightly abridged. The original instructions are more detailed and contain details about installing benerator on a Windows / Mac / Linux-System.
==== Abridged Instructions ====


* [https://sourceforge.net/project/platformdownload.php?group_id=222964 Download the Benerator distribution] from SourceForge
* [https://sourceforge.net/project/platformdownload.php?group_id=222964 Download the Benerator distribution] from SourceForge
* Unzip into a directory of your choice (e.g. C:\Program Files\Development\databene-benerator-0.6.6)
* Unzip into Program files (e.g. C:\Program Files\databene-benerator-0.7.1)
* Create an environment variable BENERATOR_HOME that points to the path you extracted benerator (example located below).
* nant should pick that version up, otherwise you need to add to your OpenPetra.build.config:
* Append BENERATOR_HOME/bin to your environment variable PATH (examples located below)
  <property name="external.Benerator" value="${OP::GetFileInProgramDirectory('/databene-benerator-0.7.1/bin/benerator.bat')}"/>
* Give benerator a quick test by just opening a new command prompt / terminal and typing
 
<code>
benerator -v
</code>
 
Example Output:
 
Local classpath: .; ...
Benerator 0.6.6 build 1255
Java version 1.6.0_25
JVM Java HotSpot(TM) Client VM 20.0-b11 (Sun Microsystems Inc.)
OS Windows 2003 5.2 (x86)
Installed JSR 223 Script Engines:
  - Mozilla Rhino[js, rhino, JavaScript, javascript, ECMAScript, ecmascript]
 
==== Examples for Environment Variables ====
 
<code>
BENERATOR_HOME = C:\Program Files\Development\databene-benerator-0.6.6
 
PATH = C:\WINDOWS; ....; %BENERATOR_HOME%\bin
</code>
 


== See also ==
== See also ==


* List of possible 3rd party [[3rd-party Test Data Generators]]
* List of possible 3rd party [[3rd-party Test Data Generators]]

Latest revision as of 17:37, 31 May 2022

Overview

Goal: creating sample data for the database.

The sample data should have

  • many donors
  • many recipients
  • many donations

Data creation and import is done in a two step process:

alt This diagram shows the data flow of created sample data from benedata benerator into the sample data constructor and the into OpenPetra Server

  1. The 3rd-party data creation tool benerator creates raw data (names, addresses, phone numbers...)
  2. The OpenPetra SampleDataConstructor-tool is specifically built to take this data, assemble, and import it into OpenPetra

Related Bugtracking-Issues

Running Sample Data Constructor

Creation of raw sample data and assembling this into the database are two separate steps which can be executed with nant:

Creating Raw Sample Data

The following assumes that databene benerator is installed and functioning (see below on how to do this). If it is not installed, you may skip this step if you already have working sample data csv-files.

Creating Raw Sample Data with Benerator

To start the generation of data, run: 

nant generateDemodata

This should create csv files containing sample data in the directory demodata\generated: "People.csv","Addresses.csv","Organisations.csv".

Running Sample Data Constructor

The Sample Data Constructor assembles the raw data, and saves it onto the Petra Server. It requires raw sample data files ("People.csv","Addresses.csv","Organisations.csv") to be in the directory demodata\generated.

To run the Sample Data Constructor, run: 

nant resetDatabase
nant importDemodata

Note: It is necessary to reset the database content with nant resetDatabase before running nant importDemodata as otherwise the RDBMS's Referential Integrity checks will throw duplicate key Exceptions when Units are created!

After having run the sample data constructor, you can access the created data from the OpenPetra Client.

Current State

  • creating names and addresses and putting these into the OpenPetra database is completed
  • creating donations is not done yet

Installing third party software

Currently the only external software used is Benedata Benerator for creation of the raw data, but this in turn requires the java runtime enviroment.

Java

  • Required: Java Runtime Environment only

Abridged Instructions

PATH = ....; "C:\Program Files (x86)\Java\jre6\bin"

  • Give it a quick test by opening a new command prompt / terminal and typing:

java -version

Example output: 

java version "1.6.0_25"
Java(TM) SE Runtime Environment (build 1.6.0_25-b06)
Java HotSpot(TM) Client VM (build 20.0-b11, mixed mode, sharing)

Databene Benerator

  • Project Website: https://www.benerator.de/
  • Note: The project is a java project
  • Used version: 0.7.1 (as of 2011-November)
  • Other requirements: Java Runtime Enviroment


  • Download the Benerator distribution from SourceForge
  • Unzip into Program files (e.g. C:\Program Files\databene-benerator-0.7.1)
  • nant should pick that version up, otherwise you need to add to your OpenPetra.build.config:
<property name="external.Benerator" value="${OP::GetFileInProgramDirectory('/databene-benerator-0.7.1/bin/benerator.bat')}"/>

See also

Retrieved from ‘https://wiki.openpetra.org/index.php?title=Sample_data_create_and_import&oldid=4919’