====== Structured Data Plugin ======

---- plugin ----
description: Add and query structured data in your wiki
author     : Andreas Gohr
email      : andi@splitbrain.org
type       : syntax, action, helper
lastupdate : 2010-04-25
compatible : 2009-12-25, anteater
depends    : sqlite
conflicts  : newline
similar    : fields, pagemod, datatemplate
tags       : database, sqlite, data, tags, tables, listing

downloadurl: https://github.com/splitbrain/dokuwiki-plugin-data/zipball/master
sourcerepo : https://github.com/splitbrain/dokuwiki-plugin-data
bugtracker : https://github.com/splitbrain/dokuwiki-plugin-data/issues
donationurl: http://donate.dokuwiki.org/data
----

This plugin allows you to add structured data to any DokuWiki page. Think about this data as additional named attributes. Those attributes can then be queried and aggregated. The plugin is similar to what was done here for the [[plugin:repository|repository plugin]] but its internals are very different to the [[plugin:repository|repository plugin]].

**This plugin requires the SQLite extension for PHP!** (Should be included with PHP5).

===== Download and Installation =====

This plugin can be downloaded [[..plugin_installation_instructions|manually]] (rename archive dir to "data" dokuwiki/lib/plugins/data/) or through the [[plugin|Plugin Manager]] from the download link given above.

==== Changes ====

[[https://github.com/feeds/splitbrain/commits/dokuwiki-plugin-data/master|{{http://img529.imageshack.us/img529/5725/feedm.png|Recent commits to Structured Data Plugin (RSS feed)}}]]
{{rss>https://github.com/feeds/splitbrain/commits/dokuwiki-plugin-data/master date}}

==== Updating from versions prior to 2010-03-22 ====

Since 2010-03-22, the data plugin uses the [[plugin:sqlite|sqlite helper plugin]]. Therefore, the database file location and structure changed. If you want to keep your old database, you have to perform the following steps prior to upgrading the data plugin:

  - Install the [[plugin:sqlite]] plugin
  - Move the file ''data/cache/dataplugin.sqlite'' to ''data/meta/data.sqlite''
  - Perform the following SQL statements on the database (for example using the sqlite plugin’s admin page)<code sql>
CREATE TABLE opts (opt,val);
CREATE UNIQUE INDEX idx_opt ON opts(opt);
INSERT INTO opts VALUES ('dbversion', 1);</code>
  - Upgrade the data plugin

If you upgraded the data plugin before these steps, a blank ''data.sqlite'' has been created and you have to copy the ''data/cache/dataplugin.sqlite'' over the existing ''data/meta/data.sqlite''.

===== Plugin Syntax =====

This plugin depends on multiple parts, each having a similar syntax. The syntax defines a block with various key/value pairs configuring the behaviour of the plugin part.

==== Data Entry (Input) ====

This part is used to add structured data to a page. All data entered here is tied to the page. Let's start with an example:

<code>
---- dataentry projects ----
type            : web development
volume          : 1 Mrd    # how much do they pay?
employees       : Joe, Jane, Jim
customer_page   : customers:microsoft
deadline_dt     : 2009-08-17
server_pages    : servers:devel01, extern:microsoft
website_url     : http://www.microsoft.com
task_tags       : programming, coding, design, html
----
</code>

As you can see the block is defined by hyphens and the word ''dataentry''. You may add additional words after the ''dataentry'' keyword. Those will be added as additional CSS classes in the final HTML output and can be queried as ''%class%'' later. You can use this for styling how different entry types should be displayed later or limiting aggregation to certain types of pages.

You may use the ''#'' character to add comments to the block. Those will be ignored and will neither be displayed nor saved. If you need to enter ''#'' as data, escape it with a backslash (''\#''). If you need a backslash, escape it as well (''\\'').

Inside the block you see //column names// and their //values//. There are a few rules for the column names:

  * Use any name you like
  * If the name ends with the ''s'' character, you may add multiple values separated by commas (like in the employees row). This removes the last ''s'' character from the rendered column name. If you want to avoid the multiple value option and keep your column name as is, add an underscore to the end of your name (example: "thickness_ : 1cm"). If you want the multiple value option and keep the your column name as is, add a second ''s'' to your column name...
  * Special //types// can be added to the name to have the output formatted accordingly. Use an underscore to separate //identifier// and //type//. The following //types// are available currently:
    * ''dt'' -- a date in the form YYYY-MM-DD, formatted as simple text but the input is checked for correct format
    * ''page'' -- the entry is treated as Wiki [[:pagename]] and will be linked in output
    * ''title'' -- like page, but an additional display title can be given separated by a pipe
    * ''nspage'' -- like page, but the column name is treated as namespace for the link.
    * ''url'' -- the value will be treated as external link
    * ''tag'' -- the values are linked to a page named after the column name, using the value as control filter for a data table
    * ''mail'' -- the value is checked to contain a valid email address, additional text is used as name that will be linked with the given email address
    * ''img<num>'' -- the input is assumed to be a image URL or local media id. The optional <num> is the wanted width in pixels to resize the image to (defaults to 40)
    * ''wiki'' -- render the input as wikitext
    * ''pageid'' -- the input is the caption for a link to the data entry page
    * when no type is given, it's just treated as simple string
    * new types can be create by using [[#Type Aliases]]
  * When using a type, add the ''s'' for multi-values at the very end (like in the ''server_pages'' row)

==== Data Table (Output) ====

To aggregate the structured data attached to various pages in your wiki this syntax is used. It will display a configurable table with the data you want. The table can be sorted and filtered. Paging is supported as well. And just as with the dataentry, you may add additional words after the datatable keyword. Those will be added as additional CSS classes in the final HTML output. Let's start with an example again:

<code>
---- datatable ----
cols    : %pageid%, employees, deadline_dt, volume
headers : Details, Assigned Employees, Deadline, $$$
max     : 10
filter  : type=web development
sort    : ^volume
----
</code>

The above config will display a table with all web development projects, the employees assigned to the project, the deadline and the volume. The table will be sorted by the volume and will display a maximum of 10 projects.

The keyword before the colon is a configuration option and the value behind is the actual setting. To make it more fault tolerant often multiple option names are possible. Here is a list of all available options:

^ Option(s)       ^ Required? ^ Description ^
| cols\\ select | yes       | These are the attributes you want to display. These are the same //names// you used in the Data Entry part |
| head\\ header\\ headers | no | If specified, these names will be used in the table headers instead of the column names |
| max\\ limit   | no | How many rows should be displayed. If more rows are available the table will be made browsable. If not given all matching rows are shown |
| sort\\ order | no | By what column should the table be sorted initially? Prepend a ''%%^%%'' to reverse the sorting |
| filter\\ where\\ filterand\\ and | no | Filter by a column value. You may specify this more than once, multiple filters will be ANDed. |
| filteror\\ or | no | Like filter, but multiple instances will be ORed |
| dynfilters | no | Set to ''1'' to enable a row of input fields for dynamically filtering the table |
| summarize | no | FIXME |
| align | no | FIXME |

For filtering, multiple comparators are possible:

^ Comparator       ^ Meaning ^
| ''=''            | Exact match |
| ''!='' or ''<>'' | Does not exactly match |
| ''<''            | Less than |
| ''%%<=%%''       | Less or equal than |
| ''>''            | Greater than |
| ''%%>=%%''       | Greater or equal than |
| ''~''            | Wildcard match. Use a ''*'' as wildcard. Like ''Apple*'' to match ''Apple Pie'' and ''Apple Computer''; e.g. ''dessert~ *Pie''|
| ''!~''           | Negative Wildcard match. Select everything that does not match the expression. |

You may use the special word **''%user%''** in a filter to make it match against the currently logged in user. To compare with the current date, use the special word **''%now%''**.

This syntax will disable all caching for the current page!

There are a few variables available to be used as field names:

^ Variable ^ Meaning ^
| %pageid% | The whole [[:pagename]] of a page |
| %title%  | The "title" of a page. This is the first headline of a page. |
| %class%  | The class of a data entry |

==== Data List (Output) ====

The ''datalist'' works like the ''datatable'' above and accepts the same parameters. But instead of displaying a table, an unordered list is shown. Each result row is one list item, all selected columns are joined with a space. The ''header'' parameter is completely ignored. No pagination is used.

<code>
---- datalist ----
cols    : %pageid%, volume
max     : 10
filter  : type=web development
sort    : ^volume
----
</code>

==== Related Pages (Output) ====

This mode allows you to display a list of pages which are similar to the current page because they share some of the structured data. Which columns are used for similarity comparison has to be given in the ''cols'' option. Additional filters and sorting options can be set. Here is an example:

<code>
---- datarelated ----
cols  : task_tags, type
title : Similar projects
max   : 5
sort  : ^volume
----
</code>

The shown config will look for pages which share values in the columns ''task'' and ''type''. A maximum of 5 pages is shown, sorted by volume.

Refer to datatable on what options are available.

This mode will **not** disable caching for the page, so the list might not always be up to date.

==== Tag Cloud (Control) ====

This syntax will display the values of a given data name as a tag cloud. Each value will link back to the current page (unless configured otherwise). The page should also contain a Data Table - this table will then be filtered for all entries matching the selected tag.

Example:

<code>
---- datacloud ----
field: employees
min: 2
limit: 20
----
</code>

The above code would display a cloud of employees assigned to at least two different projects. A maximum of the 20 most busiest employees are shown.

These are the possible options for the cloud:

^ Option(s)       ^ Required? ^ Description ^
| field\\ select\\ col | yes  | What attribute is used to build the cloud? |
| limit\\ max | no | Maximum number of tags to display. If not given all will be displayed |
| min | no | Minimum count a tag must have. If not given all will be shown |
| page\\ target | no | Give a page which contains the Data Table to control. If not given the current page is used |

Additionally filters can be given as described in the datatable syntax.

===== Customizing the Styling =====

Position and styling of input and output boxes can be done with simple CSS. Everything given after the syntax keyword will be assigned as additional class name to the surrounding div. Additionally are the field keys used as classes for the entry box items.

Let's assume the following "dataentry" box, as specified at the beginning of the page.

<code>
---- dataentry character24 ----
name            : Jack Bauer
agency_nspage   : CTU
status_tags     : alive, active, disowned, furtive
.... more info
----
</code>

You now can style this box with CSS in your ''<dokuwiki>/conf/userstyle.css''((http://forum.dokuwiki.org/thread/4500)). Here is an example:

<code css>
div.character24 {
  /* code for the container block, such as position, layout and borders, goes here */
  float: right;
  width: 33%;
}

div.character24 dl > dt { 
  /* this will allow you to style the definition terms */
  font-variant: small-caps;
}

div.character24 dl > dt + dd {
  /* this will allow you to style to the definition values */
  font-family: sans-serif;
}

div.character24 dl dd.name{
  /* this makes the name larger than the rest */
  font-size: 120%;
}
</code>

===== Type aliases =====

Type aliases are custom data types which are defined by a base type (one of the built-in types like ''page'' or no type), and optionally a prefix, postfix or a comma-separated set of valid values. Type aliases are managed on their own page in the admin menu and stored in the database. They can be used in data entries, lists and tables like the built-in types.

For example to create new datatype ''map'' which would open [[http://maps.google.com/maps|Google Maps]]:

define [[:interwiki]] address map:
<file>
map     http://maps.google.com/maps?q={NAME}
</file>

And in Admin -> Page Aliases (''do=admin&page=data'') define:

^ Type Alias ^ Type ^ Data Prefix ^ Data Postfix ^ Valid values |
| map        | wiki | %%[[map>%%   | %%]] %%          |              | 	 	 	 	 
 	 	 	 
which you could use now as:
<file>
---- dataentry  ----
Address_map  : 51.103028,-1.780611
----
</file>

You can even enter a valid adress, e.g. ''Bahnhofsstraße 5, Regensburg, Deutschland''

==== using other plugins ====

=== iCal-Plugin ===
If you aggregate any meeting, you´d link to combine them to a downloadable link.

[[.ical|iCal-Plugin]]

^ Type Alias ^ Type ^ Data Prefix ^ Data Postfix ^ Valid values |
| ical | wiki | %%{{ical>%%   | %%}} %%          |              | 	 

=== vCard ===

similar to iCal
===== Custom entry editor =====

The data plugin is the first plugin to provide a custom editor for its data entries. With a current [[devel:develonly|devel version of DokuWiki]], data entries have an own edit button. When pressing this button, the user gets an edit form where she can edit the data entries’ content or – depending on the configuration option ''edit_content_only'' – even the structure, i. e. change a field’s name, type or append and delete fields. With a current [[plugin:bureaucracy|bureaucracy]] version, some JavaScript magic for editing fields of the page and date types are available, too.

===== Missing Features =====

  * more control options:
    * attribute list (similar to tag cloud but as a simple list)
    * search field
  * better documentation, examples
  * Add JavaScript support for the content_only edit form to the full-blown form as well
  * more data types?
  * links in aggregations and data entries don't generate backlinks
  * search data in a specific namespace
  * WHERE checks against date fields need SQL INTERVAL equivalent ("WHERE field_dt < %now% + INTERVAL 6 MONTH" equals to "field_dt before 6 months from now")
  * compare against "this page" name (in datatable)
===== Examples =====

This plugin may be seen in action in the [[:template#available_templates|templates table]] on this wiki. See the source for that page and for some [[:template:arctic|individual template page]].

===== Question and Answers =====

==== How to refresh data? ====

The data that is copied to the SQLite database //should// be refreshed automatically whenever the page containing the dataentry box is changed.

However there might be some bugs making this not always 100% reliable. Refer to the [[#open bugs]] section and [[#changes]] to see what might be broken or what was recently fixed.

When you delete the whole database, all entry pages have to be re-rendered to repopulate the database. There is no automatic way to do so. 

In case page were externally modified, or fresh database, one trick to refresh the database is to download the wiki pages, in batch, using wget, a command-line file downloader. For example: wget http://myserver/wiki/doku.php/document;presentation;37.

You can of course open the database file directly in the SQLite command line client and manipulate it through SQL queries.

==== Multiple Data Entry Boxes on one Page? ====

You can not have multiple data entry boxes on the same page, at least not with the same structure.

The page id is used as the primary key to identify a set of data. There would be no difference between the data in multiple boxes (from a database point of view). The later data will just overwrite the earlier mentioned data.

==== Which SQLite for my PHP? ====

Your Linux distribution might distribute the SQLite PHP extension in a package separate from the PHP package. E.g. a Debian System lists the following available packages: php-sqlite, php-sqlite3, php5-sqlite, php5-sqlite3 -- if you are running the php5 package for PHP then you should install the php5-sqlite package.

==== Other Questions ====

If you still have questions, search the forum and if it hasn't been asked before ask your question there.

===== Bugs, Feature Requests and Patches =====

Please submit bugs and feature requests in the issue tracker linked at the top. Patches should be sent unified diff format or as git patches against the devel branch. Or even better: fork the repository at github and send a merge request.


==== No Output if there are more than 12 hits ====

Queries with more than 12 hits won't show - actually the whole page isn´t shown. Does this depend on RAM-limitation or time-outs on rented servers or are there other limitations?
==== Alias-Bug ====
//please remove, when bug fixed// (23.5.2011, deshi)
\\ //Crossposting//: [[https://github.com/splitbrain/dokuwiki-plugin-data/issues/41]]

When following the instruction [[#type_aliases]]

I couldn´t choose the type 'wiki' only following entries in the dropdown menu are available:
<blank>,page,title,mail,url,dt

In aliases.php (101-103):
  102: array('','page','title','mail','url', 'dt', 'wiki','tag')
all options can be found

In admin.php (84-86):
  85: array('','page','title','mail','url', 'dt')
elements are missing

Update second code line with upper one and everything´s fine.

==== CSS optimisation for plugin ====

=== breaking "table" ===
In most of the cases you wish to format the output like a table, but for values (right side), which need more lines the layout breaks. Do following changes and see what happens: ((http://archivist.incutio.com/viewlist/css-discuss/100531))

style.css
<code css>
div.dataplugin_entry dl dd {
    /*float: left;*/
    /*margin-left: 0.5em;*/
	float: none;
	margin-left: 10.5em;
	width: auto;	
}
</code>

=== width ===

In style.css
<code css>
div.dataplugin_entry dl {
    border: 1px solid __border__;
    padding: 1em;
    margin: 1em;
    font-size: 90%;
    overflow: auto;
    width: 50%;
}
</code>
width controlls the width of the box.

==== Localization ====

  * [[http://my.opera.com/alexgearbox/blog/download-data-ru-zip|Russian translation]]