DokuWiki

It's better when it's simple

User Tools

Site Tools


plugin:data

Structured Data Plugin

Compatible with DokuWiki

  • 2013-12-08 "Binky" yes
  • 2013-05-10 "Weatherwax" yes
  • 2012-10-13 "Adora Belle" yes
  • 2012-01-25 "Angua" unknown

plugin Add and query structured data in your wiki

Last updated on
2013-02-06
Provides
Syntax, Helper, Action
Repository
Source
Conflicts with
authorstats, fckg, newline
Requires
sqlite

This plugin allows you to add structured data to any DokuWiki page. Think about this data as additional named attributes. Those attributes can then be queried and aggregated. The plugin is similar to what was done here for the repository plugin but its internals are very different to it.

Download and Installation

Install the plugin using the Plugin Manager and the download URL above, which points to latest version of the plugin. Refer to Plugins on how to install plugins manually.

Changes

Updating from versions prior to 2013-02-06

Be sure to update the sqlite plugin and follow its instructions on sqlite version upgrades.

Updating from versions prior to 2010-03-22

Since 2010-03-22, the data plugin uses the sqlite helper plugin. Therefore, the database file location and structure changed. If you want to keep your old database, you have to perform the following steps prior to upgrading the data plugin:

  1. Install the sqlite plugin
  2. Move the file data/cache/dataplugin.sqlite to data/meta/data.sqlite
  3. Perform the following SQL statements on the database (for example using the sqlite plugin’s admin page)
    CREATE TABLE opts (opt,val);
    CREATE UNIQUE INDEX idx_opt ON opts(opt);
    INSERT INTO opts VALUES ('dbversion', 1);
  4. Upgrade the data plugin

If you upgraded the data plugin before these steps, a blank data.sqlite has been created and you have to copy the data/cache/dataplugin.sqlite over the existing data/meta/data.sqlite.

Plugin Syntax

This plugin depends on multiple parts, each having a similar syntax. The syntax defines a block with various key/value pairs configuring the behaviour of the plugin part. Next parts are available:

  • Data Entry (Input)
  • Data Table (Output)
  • Data List (Output)
  • Related Pages (Output)
  • Tag Cloud (Control)

Data Entry (Input)

This part is used to add structured data to a page. All data entered here is tied to the page. So you can use it only once on a page1). Let's start with an example:

---- dataentry projects ----
type            : web development
volume          : 1 Mrd    # how much do they pay?
employees       : Joe, Jane, Jim
description_wiki: **Important** project
customer_page   : customers:microsoft
deadline_dt     : 2009-08-17
server_pages    : servers:devel01, extern:microsoft
website_url     : http://www.microsoft.com
task_tags       : programming, coding, design, html
----

As you can see the block is defined by hyphens and the word dataentry. You may add additional words after the dataentry keyword. Those will be added as additional CSS classes in the final HTML output and can be queried as %class% later. You can use this for styling how different entry types should be displayed later or limiting aggregation to certain types of pages.

You may use the # character to add comments to the block. Those will be ignored and will neither be displayed nor saved. If you need to enter # as data, escape it with a backslash (\#). If you need a backslash, escape it as well (\\).

Inside the block you see column names and their values. There are a few rules for the column names:

  • Use any name you like
  • If the name ends with the s character, you may add multiple values separated by commas (like in the employees row). This removes the last s character from the rendered column name.
    • If you want to avoid the multiple value option and keep your column name as is, add an underscore to the end of your name (example: thickness_ : 1cm).
    • If you want the multiple value option and keep the your column name as is, add a second s to your column name…
      e.g.:
      name : John → name: John
      names: John, Jack → name: John, Jack
      name_tag : John, Jack → name: John, Jack
      name_tags: John, Jack → name: John, Jack
      thickness : 1cm → thicknes: 1cm
      thickness_: 1cm → thickness: 1cm
      thicknesss: 1cm, 2inch → thickness: 1cm, 2inch
  • Special types can be added to the name to have the output formatted accordingly. Use an underscore to separate identifier and type. The following types are available currently:
    • dt – a date in the form YYYY-MM-DD, formatted as simple text but the input is checked for correct format
    • page – the entry is treated as Wiki pagename and will be linked in output
    • title – like page, but an additional display title can be given separated by a pipe
    • nspage – like page, but the column name is treated as namespace for the link.
    • url – the value will be treated as external link
    • tag – the values are linked to a page named after the column name, using the value as control filter for a data table (change target by a type alias)
    • mail – the value is checked to contain a valid email address, additional text is used as name that will be linked with the given email address
    • img<num> – the input is assumed to be a image URL or local media id. The optional <num> is the wanted width in pixels to resize the image to (defaults to 40)
    • wiki – render the input as wikitext (use sparse, performance wise this type is not efficient)
    • pageid – the input is the caption for a link to the data entry page
    • hidden – the input will not be shown in the data entry or the entry editor
    • when no type is given, it's just treated as simple string
    • new types can be create by using Type Aliases
  • When using a type, add the s for multi-values at the very end (like in the server_pages row)

In the values you can use two placeholders, useful for creating localization depended links.

Variable for value Meaning
%lang% Language code of your default wiki language. e.g. en
%trans% Language code of the localisation of the current page.
Requires the Translation Plugin. For page link de:test:page, it is replaced by
de. Fallback to default lang. Empty without Translation Plugin.

Data Table (Output)

To aggregate the structured data attached to various pages in your wiki this syntax is used. It will display a configurable table with the data you want. The table can be sorted and filtered. Paging is supported as well. And just as with the dataentry, you may add additional words after the datatable keyword. Those will be added as additional CSS classes in the final HTML output. Let's start with an example again:

---- datatable ----
cols    : %pageid%, employees, deadline_dt, website_url, volume
headers : Details, Assigned Employees, Deadline, Personal website, $$$
max     : 10
filter  : type=web development
sort    : ^volume
----

The above config will display a table with all web development projects, the employees assigned to the project, the deadline and the volume. The table will be sorted by the volume and will display a maximum of 10 projects.

Like the dataentry, the values in the columns can also be formatted by postfixing the column names with the typealias. Like the deadline_dt column above. Another example shows website, as an url is given label the entries in cols with e.g. _url to enable linking in output.

The keyword before the colon is a configuration option and the value behind is the actual setting. To make it more fault tolerant often multiple option names are possible. Here is a list of all available options:

Option(s) Required? Description
cols
select
yes These are the attributes you want to display. These are the same names you used in the Data Entry part
head
header
headers
no If specified, these names will be used in the table headers instead of the column names
max
limit
no How many rows should be displayed. If more rows are available the table will be made browsable. If not given all matching rows are shown
sort
order
no By what column should the table be sorted initially? Prepend a ^ to reverse the sorting
filter
where
filterand
and
no Filter by a column value. You may specify this more than once, multiple filters will be ANDed.
filteror
or
no Like filter, but multiple instances will be ORed
dynfilters no Set to 1 to enable a row of input fields for dynamically filtering the table
summarize no Set to 1 to calculate sum of columns
align no List of column alignments. The alignments can be left (l), center (c) or right (r)
rownumbers no Set to 1 to show row numbers
widths no Specifies column widths e.g. 50px, 20em, -, 10%. Skip column by dash -.

For filtering, multiple comparators are possible:

Comparator Meaning
= Exact match
!= or <> Does not exactly match
< Less than
<= Less or equal than
> Greater than
>= Greater or equal than
~ Wildcard match. Use a * as wildcard. Like Apple* to match Apple Pie and Apple Computer; e.g. dessert~ *Pie. Case insensitive.
*~ Wildcard match. Look for matches containing search term; e.g. dessert*~ Pi match Apple Pie
!~ Negative Wildcard match. Select everything that does not match the expression.

You may use the special variables in a filter to make it match against current logged in user or current date e.g. filter: username=%user%

Variable for filtervalue Meaning
%user% match against the currently logged in user
%now% compare with the current date
%lang% Language code of your default wiki language. e.g. en
%trans% Language code of the localisation of the current page.
Requires the Translation Plugin. For page link de:test:page, it is replaced by
de. Fallback to default lang. Empty without Translation Plugin.

There are a few variables available to be used as field names:

Variable for fieldname Meaning
%pageid% The whole pagename of a page
%title% The “title” of a page. This is the first headline of a page.
%class% The class of a data entry

Data List (Output)

The datalist works like the datatable above and accepts the same parameters. But instead of displaying a table, an unordered list is shown. Each result row is one list item, all selected columns are default joined with a space. When sepbyheaders is not set, the header parameter is completely ignored. No pagination is used.

Option(s) Required? Description
sepbyheaders no Set to 1 to use headers as separators between the values. In headers you can use additional spaces, when enclosed by quotes e.g. ” - ”
---- datalist ----
cols    : %pageid%, volume
max     : 10
filter  : type=web development
sort    : ^volume
----

This mode allows you to display a list of pages which are similar to the current page because they share some of the structured data. Which columns are used for similarity comparison has to be given in the cols option. Additional filters and sorting options can be set. Here is an example:

---- datarelated ----
cols  : task_tags, type
title : Similar projects
max   : 5
sort  : ^volume
----

The shown config will look for pages which share values in the columns task_tags and type. A maximum of 5 pages is shown, sorted by volume.

Refer to datatable on what options are available.

This mode will not disable caching for the page, so the list might not always be up to date.

Tag Cloud (Control)

This syntax will display the values of a given data name as a tag cloud. Each value will link back to the current page (unless configured otherwise by target option). The page should also contain a Data Table - this table will then be filtered for all entries matching the selected tag.

Example:

---- datacloud ----
field: employees
min: 2
limit: 20
----

The above code would display a cloud of employees assigned to at least two different projects. A maximum of the 20 most busiest employees are shown.

These are the possible options for the cloud:

Option(s) Required? Description
field
select
col
yes What attribute is used to build the cloud?
limit
max
no Maximum number of tags to display. If not given all will be displayed
min no Minimum count a tag must have. If not given all will be shown
page
target
no Give a page which contains the Data Table to control. If not given the current page is used

Additionally filters can be given as described in the datatable syntax.

Customizing the Styling

Position and styling of input and output boxes can be done with simple CSS. Everything given after the syntax keyword will be assigned as additional class name to the surrounding div. Additionally are the field keys used as classes for the entry box items.

Let's assume the following “dataentry” box, as specified at the beginning of the page.

---- dataentry character24 ----
name            : Jack Bauer
agency_nspage   : CTU
status_tags     : alive, active, disowned, furtive
.... more info
----

You now can style this box with CSS in your <dokuwiki>/conf/userstyle.css2). Here is an example:

div.character24 {
  /* code for the container block, such as position, layout and borders, goes here */
  float: right;
  width: 33%;
}
 
div.character24 dl > dt { 
  /* this will allow you to style the definition terms */
  font-variant: small-caps;
}
 
div.character24 dl > dt + dd {
  /* this will allow you to style to the definition values */
  font-family: sans-serif;
}
 
div.character24 dl dd.name{
  /* this makes the name larger than the rest */
  font-size: 120%;
}

Type aliases

Type aliases are custom data types which are defined by a base type (one of the built-in types like page or no type), and optionally a prefix, postfix or a comma-separated set of valid values. Type aliases are managed on their own page in the admin menu and stored in the database. They can be used in data entries, lists and tables like the built-in types.

Map type alias

For example to create new datatype map which would open Google Maps:

define interwiki address map:

map     http://maps.google.com/maps?q={NAME}

And in Admin → Page Aliases (do=admin&page=data_aliases) define:

Type Alias Type Data Prefix Data Postfix Valid values
map wiki [[map> ]]

which you could use now as:

---- dataentry  ----
Address_map  : 51.103028,-1.780611
----

You can even enter a valid adress, e.g. Bahnhofsstraße 5, Regensburg, Deutschland

Change tag target with a tag type alias

Tag links use default the column name as target page. You can change it by a type alias. Give as Prefix the url of the page that has your datatable. Clicking on taglink will filter that table by the tagvalue.

Type Alias Type Data Prefix Data Postfix Valid values
newtag tag namespace:pagewithdatatable

which you could use now as:

---- dataentry  ----
Address_newtag  : cheese
----

Clicking on this cheese tag will refer to namespace:pagewithdatatable and filter on cheese.

iCal type alias with iCal-Plugin

If you aggregate any meeting, you´d link to combine them to a downloadable link.

iCal-Plugin

Type Alias Type Data Prefix Data Postfix Valid values
ical wiki {{ical> }}

vCard type alias with vCard Plugin

Setup for the vCard plugin is similar to iCal

Custom entry editor

The data plugin is the first plugin to provide a custom editor for its data entries. Data entries have an own edit button. When pressing this button, the user gets an edit form where she can edit the data entries’ content or – depending on the configuration option edit_content_only – even the structure, i. e. change a field’s name, type or append and delete fields.

Custom editor with config "edit_content_only" off let's you edit structure and values
Custom editor: Editing structure and values

Custom editor with config "edit_content_only" on let's you edit only values
Custom editor: Editing only values by switching on config option edit_content_only

Nice fields for "edit_content_only"

When the configuration option edit_content_only is enabled, some nice fields are used:

  • Dropdown list is shown filled with the values given in the Valid values column of type aliases.
  • Datepicker for fields of type dt
  • Autocompletion suggestion of existing pages for type aliases based on the page type
    (type page itself don't work)

Requirement: The datepicker and pagesuggestions needs the bureaucracy plugin, which supplies the JavaScript magic.

Page suggestions are activated by a type alias of type page. The second alias shows only pages in plugin: namespace, demonstrated in the image too.

Type Alias Type Data Prefix Data Postfix Valid values
pagesuggest page
pagesuggestplugin page plugin:

Custom entry editor: datepicker for _dt and selects for alias type with Valid values

Custom entry editor: pagesuggestions for type aliases of page type. Suggestions show page title and in brackets the page name.

Left: custom editor with datapicker field and a single and a multiple select field. Right: a field with pagesuggestions.

Bureaucracy plugin and type aliases

The nice fields of data plugin used in the custom entry editor are also available for fields in the form of the bureaucracy plugin. These are: select of valid values, datepicker and pagesuggestions. Other types are allowed, but are a plain textbox only.

Some examples of defining form fields in the bureaucracy form, using dataplugin fields:

dataplugin Tag _fixedtag 	
dataplugin Multitags _fixedtags 

dataplugin "Date of happening" _dt	
dataplugin "Plugin Page" _pagesuggestplugin

Last line uses the type alias of previous section, the first two lines are in combination with type alias:

Type Alias Type Data Prefix Data Postfix Valid values
fixedtag tag Book, Notebook, Reader

Datepicker for dataplugin _dt field in a bureaucracy form Single and multiple select for dataplugin field using a type alias with Valid values in a bureacracy form

Left image shows a datepicker in a bureaucracy form and right a single and a multiple select, these are created with a type alias which defines some valid values.

Translation Support

If you use the translation plugin, you might want to use the same data entries in different languages. The plugin allows for translated field names - values will not be translated.

To create fieldname translations, create a file in conf/lang/<langcode>/data-plugin.php. The file needs to contain a PHP array named $lang with the fieldname in the array key and the translation in the value:

conf/lang/en/data-plugin.php
<?php
$lang['somekey']  = 'My name for the key';
$lang['otherkey'] = 'My name for the other key';

Fieldnames are to be given without their type. So it's deadline not deadline_dt.

Missing Features

  • more control options:
    • attribute list (similar to tag cloud but as a simple list)
    • search field
  • better documentation, examples
  • Add JavaScript support for the content_only edit form to the full-blown form as well
  • more data types?
  • links in aggregations and data entries don't generate backlinks
  • search data in a specific namespace
  • WHERE checks against date fields need SQL INTERVAL equivalent (“WHERE field_dt < %now% + INTERVAL 6 MONTH” equals to “field_dt before 6 months from now”)
  • compare against “this page” name (in datatable)
  • history for the type alias table (changes in the table can affect the pages, but no record and no revert is available).

Examples

Some use cases:

Question and Answers

How to refresh data?

The data that is copied to the SQLite database should be refreshed automatically whenever the page containing the dataentry box is changed. If you deleted or renamed pages outside the wiki, the database will still contain data about now non-existing pages. These can be cleaned using the Data Plugin: Clean up database entry in the Admin menu.

However there might be some bugs making this not always 100% reliable. Refer to the open bugs section and changes to see what might be broken or what was recently fixed.

When you delete the whole database, all entry pages have to be re-rendered to repopulate the database. There is no automatic way to do so.

In case page were externally modified, or fresh database, one trick to refresh the database is to download the wiki pages, in batch, using wget, a command-line file downloader.

You can of course open the database file directly in the SQLite command line client and manipulate it through SQL queries.

Other Questions

If you still have questions, search the forum and if it hasn't been asked before ask your question there. Please leave bugreports on Githubs issuetracker.

Bugs, Feature Requests and Patches

Please submit bugs and feature requests in the issue tracker on github. Patches should be sent unified diff format or as git patches against the devel branch. Or even better: fork the repository at github and send a merge request.

1) any additional dataentry on the same page would overwrite any preceding dataentry on this page and so - while many dataentry blocks on one page may be displayed - only the values of the last one would be stored in the SQLite database and could be retrieved as an output by e.g. a datatable
plugin/data.txt · Last modified: 2014/02/21 09:06 by hh lohmann