Sample data create and import
Data creation and import is split into two tasks:
- create sample data (issue #29)
- import this data (issue #220)
The import is done via import file. Decision: simple for user.
Keeping the focus: the focus is creating sample data for the database, not import/export.
The above points were decided to be done this way in a phonecall with Timo.
This page aims to act as whiteboard for displaying current state and solving this task. The task itself is tracked in the two issue stated above.
Creating sample data
Goal: creating sample data for the database.
The sample data should have
- many donors
- many recipients
- many donations
Lists of test data generators:
- http://www.webresourcesdepot.com/test-sample-data-generators/
- http://databene.org/databene-benerator/similar-products.html
The majority of the software listed below was extracted from former page.
Idea to be checked: use data from generatedata / geo-database / briandunning together benerator to compile data to common format, which is then imported as shown below.
| interest? | Program | creates | area | Output | App-Type | License |
|---|---|---|---|---|---|---|
| * | benerator | creates data / transforms given data to test data | various databases, xml, csv, excel | Framework | GPL / commercial | |
| * | generatedata.com | Addresses / Cities / Countries | Netherlands, Canada, UK, US | XML, Excel, HTML, CSV, SQL | Webapp (JS,PHP,MySQL) | GPL v2 |
| * | Geographical Places Database | geographical locations (schools, universities, whitehouse, eiffel tower...) | tab delimited | website, download, libraries (various languages), webservice | creative commons attribution | |
| http://www.briandunning.com/sample-data/ | Website with real address and company data (US and Canada) but with fake names. This could be useful with testing map services as well since there are real geographic locations. | US, Canada | free | |||
| DBMonster | generates test data | SQL | Command-Line (Java) | Apache License | ||
| CSV Data generator | CSV? | (Ruby) | ||||
| Datagenerator | library / GUI | GPL | ||||
| dqMaster | text,xml,db | GUI (extensible) | ||||
| Spawner Data Generator | random proper names, terms and connectors | delimited text / SQL | apptype | license | ||
| Test Dictionary | java interface | |||||
| data at most | Fresh Trash Generator | Random Website, Email, Family and First Names, Phone Number, Company, Birthday (at least some of the resource data might be interesting) | Greek Names and Companies, German Streets | java utility package | ||
| nn | google api toolkit | nn | Web API | |||
| - | Data Science Toolkit | convert address to coordinates, vv, ip to coordinates etc | Web API / VM | |||
| - | fakenamegenerator.com | Names,Adresses from many countries | Website / Web API | proprietary for API (kostenlos, but attribution) | ||
| - (com) | GEDIS Studio for Test Data | "Realistic Test Data" (not viewed) | CSV, XML, SQL, or HTML | Windows / Scripting | community edition kostenlos / commercial | |
| - (com) | Excel random data generator | Generates sample data, somewhat acclaimed here | MS Excel Plugin | commercial | ||
| - (com) | SQL Data Generator | Generates complex sample data (addresses, companies, interaction), a business person liked it on stackoverflow. Would probably be the right thing except it is SQL Server and commercial. | Application for MS SQL Server | commercial | ||
| - (com) | Microsoft Visual Studio Database Edition | Generates sample data, and several people pointed to it on stackoverflow. | Part of Visual Studio | commercial |
List of others (not checked):
- 2011-02 sf dagen
- 2007-05 sf pharaon
- 2011-03 sf encapet
- 2010-08 sf adag
- 2009-11 sf jrando
- 2010-05 sf bbf-data-genera
Coding
Some coding has been done already: See csharp\ICT\PetraTools\GenerateSampleData for transforming sample data into family records etc.
Also see partner import module, which processes csv and yaml files. csharp\ICT\Petra\Client\lib\MPartner\gui\PartnerImport.ManualCode.cs
Importing sample data
The import is done via import file. Decision: simple for user.
Keeping the focus: the focus is creating sample data for the database, not import/export. Import/export is a simple tool - which we put effort into, to keep it nice and simple and easy to understand. But in this case, a tool for sample data only.
Make the import file as simple as possible for the user, e.g. consciously limit the scope of the import files capability (one address per person), but rather not powerful import-file.
Concider data liberation?
Not necessarily - only if useful to keep it simple and make it work quickly.
Intended location of data in OpenPetra
| Data | Table |
|---|---|
| Person | p_family |
| Address | - |
| Donations | - |
p_family will be used for all data, and p_person ignored (This is in line with the attempts to replace p_person by p_family).