DokuWiki

It's better when it's simple

User Tools

Site Tools


plugin:data

data Plugin

Compatible with DokuWiki

  • 2024-02-06 "Kaos" unknown
  • 2023-04-04 "Jack Jackrum" unknown
  • 2022-07-31 "Igor" yes
  • 2020-07-29 "Hogfather" yes

plugin Add and query structured data in your wiki

Last updated on
2024-01-30
Provides
Syntax, Helper, Action
Repository
Source
Conflicts with
data-au, fckg, headerfooter
Requires
sqlite

This plugin allows you to add structured data to any DokuWiki page. Think about this data as additional named attributes. Those attributes can then be queried and aggregated. The plugin is similar to what was done here for the repository plugin but its internals are very different to it.

:!: You might want to consider using the spiritual successor to this plugin: struct Plugin

Download and Installation

Search and install the plugin using the Extension Manager. Refer to Plugins on how to install plugins manually.

Changes

Build Status

Updating from versions prior to 2013-02-06

Be sure to update the sqlite plugin and follow its instructions on sqlite version upgrades.

Updating from versions prior to 2010-03-22

Since 2010-03-22, the data plugin uses the sqlite helper plugin. Therefore, the database file location and structure changed. If you want to keep your old database, you have to perform the following steps prior to upgrading the data plugin:

  1. Install the sqlite plugin
  2. Move the file data/cache/dataplugin.sqlite to data/meta/data.sqlite
  3. Perform the following SQL statements on the database (for example using the sqlite plugin’s admin page)
    CREATE TABLE opts (opt,val);
    CREATE UNIQUE INDEX idx_opt ON opts(opt);
    INSERT INTO opts VALUES ('dbversion', 1);
  4. Upgrade the data plugin

If you upgraded the data plugin before these steps, a blank data.sqlite has been created and you have to copy the data/cache/dataplugin.sqlite over the existing data/meta/data.sqlite.

Plugin Syntax

This plugin depends on multiple parts, each having a similar syntax. The syntax defines a block with various key/value pairs configuring the behaviour of the plugin part. The following parts are available:

Part Function
Data Entry Input
Data Table Output
Data List Output
Related Pages Output
Tag Cloud Control

Data Entry (Input)

This part is used to add structured data to a page. All data entered here is tied to the page. So you can use it only once on a page1). Let's start with an example:

---- dataentry projects ----
type            : web development
volume          : 1 Mrd    # how much do they pay?
employees       : Joe, Jane, Jim
description_wiki: **Important** project
customer_page   : customers:microsoft
deadline_dt     : 2009-08-17
server_pages    : servers:devel01, extern:microsoft
website_url     : http://www.microsoft.com
task_tags       : programming, coding, design, html
----

As you can see the block is defined by hyphens and the word dataentry. You may add additional words after the dataentry keyword. Those will be added as additional CSS classes in the final HTML output and can be queried as %class% later. You can use this for styling how different entry types should be displayed later or limiting aggregation to certain types of pages.

You may use the # character to add comments to the block. Those will be ignored and will neither be displayed nor saved. If you need to enter # as data, escape it with a backslash (\#). If you need a backslash, escape it as well (\\).

Inside the block you see column names and their values. There are a few rules for the column names:

  • Use any name you like
  • If the name ends with the s character, you may add multiple values separated by commas (like in the employees row). This removes the last s character from the rendered column name.
    • The s suffix or “multiple values” come into play when using the ~ filter as opposed to *~ (see below)2)
    • If you want to avoid the multiple value option and keep your column name as is, add an underscore to the end of your name (example: thickness_ : 1cm).
    • If you want the multiple value option and keep the your column name as is, add a second s to your column name…
      e.g.:
      name : John → name: John
      names: John, Jack → name: John, Jack
      name_tag : John, Jack → name: John, Jack
      name_tags: John, Jack → name: John, Jack
      thickness : 1cm → thicknes: 1cm
      thickness_: 1cm → thickness: 1cm
      thicknesss: 1cm, 2inch → thickness: 1cm, 2inch
  • When using a type, add the s for multi-values at the very end (like in the server_pages row)
  • Special types can be added to the name to have the output formatted accordingly. Use an underscore to separate identifier and type. The currently available types are listed in the table below.
    • when no type is given, it's just treated as simple string
    • new types can be created by using Type Aliases
type Meaning
dt a date in the form YYYY-MM-DD, formatted as simple text but the input is checked for correct format
page the entry is treated as Wiki pagename and will be linked in output. It is always considered to be an absolute pagename starting in the root namespace. Relative pagenames are not possible.
title like page, but an additional display title can be given separated by a pipe. It is always considered to be an absolute pagename starting in the root namespace. Relative pagenames are not possible.
nspage like page, but the column name is treated as namespace for the link.
url the value will be treated as external link
tag the values are linked to a page named after the column name, using the value as control filter for a data table (change target by a type alias)
mail the value is checked to contain a valid email address, additional text is used as name that will be linked with the given email address
img<num> the input is assumed to be a image URL or local media id. The optional <num> is the wanted width in pixels to resize the image to (defaults to 40)
wiki render the input as wikitext (use sparingly as this type impacts performance)
pageid the input is the caption for a link to the data entry page
hidden the input will not be shown in the data entry or the entry editor

In the values you can use two placeholders, useful for creating localization depended links.

Variable for value Meaning
%lang% Language code of your default wiki language. e.g. en
%trans% Language code of the localisation of the current page.
Requires the Translation Plugin. For page link de:test:page, it is replaced by
de. Fallback to default lang. Empty without Translation Plugin.

Data Table (Output)

To aggregate the structured data attached to various pages in your wiki this syntax is used. It will display a configurable table with the data you want. The table can be sorted and filtered. Paging is supported as well. And just as with the dataentry, you may add additional words after the datatable keyword. Those will be added as additional CSS classes in the final HTML output. Let's start with an example again:

---- datatable ----
cols    : %pageid%, employees, deadline_dt, website_url, volume
headers : Details, Assigned Employees, Deadline, Personal website, $$$
max     : 10
filter  : type=web development
sort    : ^volume
----

The above config will display a table with all web development projects, the employees assigned to the project, the deadline and the volume. The table will be sorted by the volume and will display a maximum of 10 projects.

Like the dataentry, the values in the columns can also be formatted by postfixing the column names with the typealias. Like the deadline_dt column above. Another example shows website, as an url is given label the entries in cols with e.g. _url to enable linking in output.

The keyword before the colon is a configuration option and the value behind is the actual setting. To make it more fault tolerant often multiple option names are possible. Here is a list of all available options:

Option(s) Required? Description
cols
select
yes These are the attributes you want to display. These are the same names you used in the Data Entry part
head
header
headers
no If specified, these names will be used in the table headers instead of the column names
max
limit
no How many rows should be displayed. If more rows are available the table will be made browsable. If not given all matching rows are shown
sort
order
no By what column should the table be sorted initially? Prepend a ^ to reverse the sorting
filter
where
filterand
and
no Filter by a column value. You may specify this more than once, multiple filters will be ANDed.
filteror
or
no Like filter, but multiple instances will be ORed
dynfilters no Set to 1 to enable a row of input fields for dynamically filtering the table
summarize no Set to 1 to calculate sum of columns
align no List of column alignments. The alignments can be left (l), center (c) or right (r)
rownumbers no Set to 1 to show row numbers
widths no Specifies column widths e.g. 50px, 20em, -, 10%. Skip column by dash -. (remark: if a cell contains a long string of characters with no space, the width of the corresponding column will not be able to be smaller than the size of this string of characters. Using “widths” to specify a smaller width for this column will have no effect)

For filtering, multiple comparators are possible:

Comparator Meaning
= Exact match
!= or <> Does not exactly match
< Less than
<= Less than or equal to
> Greater than
>= Greater than or equal to
~ Wildcard match. Use a * as wildcard. Like Apple* to match Apple Pie and Apple Computer; e.g. dessert~ *Pie. Case insensitive.
*~ Wildcard match. Look for matches containing search term; e.g. dessert*~ Pi match Apple Pie
!~ Negative Wildcard match. Select everything that does not match the expression.
~~ Looks if contained in given comma separated list; e.g. dessert ~~ diner,dessert

You may use the special variables in a filter to make it match against current logged in user or current date e.g. filter: username=%user%

Variable for filtervalue Meaning
%user% match against the currently logged in user
%groups% match against the groups of the current user; use the ~~ comparator
%now% compare with the current date
%lang% Language code of your default wiki language. e.g. en
%trans% Language code of the localisation of the current page.
Requires the Translation Plugin. For page link de:test:page, it is replaced by
de. Fallback to default lang. Empty without Translation Plugin.

There are a few variables available to be used as field names e.g. filter : %pageid%~members:*:

Variable for fieldname Meaning
%pageid% The whole pagename of a page
%title% The “title” of a page. This is the first headline of a page.
%class% The class of a data entry

Additionally, the date of the last modification of found pages can be retrieved and sorted by %lastmod%:

---- datatable ----
cols    : %pageid%, %lastmod%
headers : Details, Last Update
max     : 10
filter  : type=release notes
sort    : ^%lastmod%
----

Data List (Output)

The datalist works like the datatable above and accepts the same parameters. But instead of displaying a table, an unordered list is shown. Each result row is one list item, all selected columns are default joined with a space. When sepbyheaders is not set, the header parameter is completely ignored. No pagination is used.

Option(s) Required? Description
sepbyheaders no Set to 1 to use headers as separators between the values. In headers you can use additional spaces, when enclosed by quotes e.g. " - "
---- datalist ----
cols    : %pageid%, volume
max     : 10
filter  : type=web development
sort    : ^volume
----

This mode allows you to display a list of pages which are similar to the current page because they share some of the structured data. Which columns are used for similarity comparison has to be given in the cols option. Additional filters and sorting options can be set. Here is an example:

---- datarelated ----
cols  : task_tags, type
title : Similar projects
max   : 5
sort  : ^volume
----

The shown config will look for pages which share values in the columns task_tags and type. A maximum of 5 pages is shown, sorted by volume.

Refer to datatable on what options are available.

This mode will not disable caching for the page, so the list might not always be up to date.

Tag Cloud (Control)

This syntax will display the values of a given data name as a tag cloud. Each value will link back to the current page (unless configured otherwise by target option). The page should also contain a Data Table - this table will then be filtered for all entries matching the selected tag.

Example:

---- datacloud ----
field: employees
min: 2
limit: 20
----

The above code would display a cloud of employees assigned to at least two different projects. A maximum of the 20 most busiest employees are shown.

These are the possible options for the cloud:

Option(s) Required? Description
field
select
col
yes What attribute is used to build the cloud?
limit
max
no Maximum number of tags to display. If not given all will be displayed
min no Minimum count a tag must have. If not given all will be shown
page
target
no Give a page which contains the Data Table to control. If not given the current page is used
summarize no Set to 1 to show sum of field

Additionally filters can be given as described in the datatable syntax.

Customizing the Styling

Position and styling of input and output boxes can be done with simple CSS. Everything given after the syntax keyword will be assigned as additional class name to the surrounding div. Additionally are the field keys used as classes for the entry box items.

Let's assume the following “dataentry” box, as specified at the beginning of the page.

---- dataentry character24 ----
name            : Jack Bauer
agency_nspage   : CTU
status_tags     : alive, active, disowned, furtive
.... more info
----

You now can style this box with CSS in your <dokuwiki>/conf/userstyle.css3). Here is an example:

div.character24 {
  /* code for the container block, such as position, layout and borders, goes here */
  float: right;
  width: 33%;
}
 
div.character24 dl > dt { 
  /* this will allow you to style the definition terms */
  font-variant: small-caps;
}
 
div.character24 dl > dt + dd {
  /* this will allow you to style to the definition values */
  font-family: sans-serif;
}
 
div.character24 dl dd.name{
  /* this makes the name larger than the rest */
  font-size: 120%;
}

Type aliases

Type aliases are custom data types which are defined by a base type (one of the built-in types like page or no type), and optionally a prefix, postfix or a comma-separated set of valid values. Type aliases are managed on their own page in the admin menu and stored in the database. They can be used in data entries, lists and tables like the built-in types.

Map type alias

For example to create new datatype map which would open Google Maps:

define interwiki address map:

map     http://maps.google.com/maps?q={NAME}

And in Admin → Data Plugin: Field Aliases (do=admin&page=data_aliases) define:

Type Alias Type Data Prefix Data Postfix Valid values
map wiki [[map> ]]

which you could use now as:

---- dataentry  ----
Address_map  : 51.103028,-1.780611
----

You can even enter a valid address, e.g. Bahnhofsstraße 5, Regensburg, Deutschland

Change tag target with a tag type alias

Tag links use default the column name as target page. You can change it by a type alias. Give as Prefix the url of the page that has your datatable. Clicking on taglink will filter that table by the tagvalue.

Type Alias Type Data Prefix Data Postfix Valid values
newtag tag namespace:pagewithdatatable

which you could use now as:

---- dataentry  ----
Address_newtag  : cheese
----

Clicking on this cheese tag will refer to namespace:pagewithdatatable and filter on cheese.

iCal type alias with iCal-Plugin

If you aggregate any meeting, you´d link to combine them to a downloadable link.

iCal-Plugin

Type Alias Type Data Prefix Data Postfix Valid values
ical wiki {{ical> }}

vCard type alias with vCard Plugin

Setup for the vCard plugin is similar to iCal.

Custom entry editor

The data plugin is the first plugin to provide a custom editor for its data entries. Data entries have an own edit button. When pressing this button, the user gets an edit form where she can edit the data entries’ content or – depending on the configuration option edit_content_only – even the structure, i. e. change a field’s name, type or append and delete fields.

Custom editor with config "edit_content_only" off let's you edit structure and values
Custom editor: Editing structure and values

Custom editor with config "edit_content_only" on let's you edit only values
Custom editor: Editing only values by switching on config option edit_content_only

Nice fields for "edit_content_only"

When the configuration option edit_content_only is enabled, some nice fields are used:

  • Dropdown list is shown filled with the values given in the Valid values column of type aliases.
  • Datepicker for fields of type dt
  • Autocompletion suggestion of existing pages for type aliases based on the page type
    (type page itself don't work)

Requirement: The datepicker and pagesuggestions needs the bureaucracy plugin, which supplies the JavaScript magic.

Page suggestions are activated by a type alias of type page. The second alias shows only pages in plugin: namespace, demonstrated in the image too.

Type Alias Type Data Prefix Data Postfix Valid values
pagesuggest page
pagesuggestplugin page plugin:

Custom entry editor: datepicker for _dt and selects for alias type with Valid values

Custom entry editor: pagesuggestions for type aliases of page type. Suggestions show page title and in brackets the page name.

Left: custom editor with datapicker field and a single and a multiple select field.

Right: a field with pagesuggestions.

Bureaucracy plugin and type aliases

The nice fields of data plugin used in the custom entry editor are also available for fields in the form of the Bureaucracy Plugin. These are: select of valid values, datepicker and pagesuggestions. Other types are allowed, but are a plain textbox only.

Some examples of defining form fields in the bureaucracy form, using dataplugin fields:

data_aliastextbox Tag _fixedtag 	
data_aliastextbox Multitags _fixedtags 

data_aliastextbox "Date of happening" _dt	
data_aliastextbox "Plugin Page" _pagesuggestplugin

Last line uses the type alias of previous section, the first two lines are in combination with type alias:

fixedtag tag Book, Notebook, Reader

Datepicker for dataplugin _dt field in a bureaucracy form Single and multiple select for dataplugin field using a type alias with Valid values in a bureacracy form

Left image shows a datepicker in a bureaucracy form and right a single and a multiple select, these are created with a type alias which defines some valid values.

Translation Support

If you use the translation plugin, you might want to use the same data entries in different languages. The plugin allows for translated field names – values will not be translated. If you only want to use English field names while having them displayed in your language, you can also follow these instructions.

To create fieldname translations, create a file in conf/lang/<langcode>/data-plugin.php. The file needs to contain a PHP array named $lang with the fieldname in the array key and the translation in the value:

conf/lang/en/data-plugin.php
<?php
$lang['somekey']  = 'My name for the key';
$lang['otherkey'] = 'My name for the other key';

Fieldnames are to be given without their type. So it's deadline not deadline_dt.

After creating or modifiying data-plugin.php file, please update wiki configuration to refresh cache start?do=admin&page=config, otherwise modifications are not taken into account.

Missing Features

  • more control options:
    • attribute list (similar to tag cloud but as a simple list)
    • search field – see datasearchform plugin
  • better documentation, examples
    • Please provide suggestions for which pieces need more attention or where you do not understand stuff. Thanks.
  • Add JavaScript support for the content_only edit form to the full-blown form as well
  • more data types?
  • links in aggregations and data entries don't generate backlinks
  • search data in a specific namespace
  • WHERE checks against date fields need SQL INTERVAL equivalent (“WHERE field_dt < %now% + INTERVAL 6 MONTH” equals to “field_dt before 6 months from now”)
  • compare against “this page” name (in datatable)
  • history for the type alias table (changes in the table can affect the pages, but no record and no revert is available).
  • More data-entry blocks at one page. Reason: the Data Plugin is designed as data-entry block per page. ⇒ Recommended alternative: Strata Plugin, which is equivalent to Data plugin, but bit more sophisticated.

Examples

Some use cases:

Question and Answers

How to refresh data?

The data that is copied to the SQLite database should be refreshed automatically whenever the page containing the dataentry box is changed. If you deleted or renamed pages outside the wiki, the database will still contain data about now non-existing pages. These can be cleaned using the Data Plugin: Clean up database entry in the Admin menu.

However there might be some bugs making this not always 100% reliable. Refer to the open bugs section and changes to see what might be broken or what was recently fixed.

When you delete the whole database, all entry pages have to be re-rendered to repopulate the database. There is no automatic way to do so.

In case page were externally modified, or fresh database, few tricks to refresh the database are:

  • to download the wiki pages, in batch, using wget, a command-line file downloader;
  • to install the searchindex plugin, launch the newly added SearchIndex Manager panel in Admin menu and click the Rebuild Index button.

You can of course open the database file directly in the SQLite command line client and manipulate it through SQL queries.

What is the difference between Strata plugin and Data plugin?

The Strata Plugin is in the big lines equivalent to Data plugin. However, it has more sophisticated query syntax, less and simpler 'aliastypes' system and additional features like graph or output based at an predefined template. So for handling of more complicated data the Strata Plugin has more opportunities.

How do you filter empty fields/values?

Suppose some entries contain empty fields and you want to display such entries: how do you write the filter in the query to output these?

e.g.

---- dataentry projects ----
type            : web development
volume          : 1 Mrd    # how much do they pay?
employees       : Joe, Jane, Jim
description_wiki: **Important** project
customer_page   : 
----

and if the query is:

---- datatable ----
cols    : %pageid%
headers : Details
filter  : customer_page=
----

the output does not show the above record

The solution is to change the filter to a value that will always be populated.
---- datatable ----
cols    : %pageid%
headers : Details
filter  : %class% = projects
----
SFITCS 2016-10-15 04:58

Why don't my variables/placeholders work when I create a table in a namespace template?

DokuWiki calls strftime on _template.txt when creating a new page. This replaces some combinations of %[a-z] (for example, %title% will become [TAB]itle%). When using placeholders in _template.txt you should use double percent signs (%%title%%). Strftime will treat those as literal percent characters.

Other Questions

If you still have questions, search the forum and if it hasn't been asked before ask your question there. Please leave bug reports on Github's issue tracker.


Customisations

Important! These are hacks – there is no guarantee they'll always work, and they will not survive updates.

If it breaks you get to keep the pieces. :-)

Customise the Date Format

NOTE: The configuration setting for the PHP strftime function in /conf/dokuwiki.php also need to be changed (can also be changed via Admin→Configuration)

$conf['dformat']     = '%d/%m/%Y %H:%M';

To change the default date format from yy-mm-dd to dd-mm-yy (e.g. for Australia)

Change this:

plugins/data/helper.php
    /**
     * Replace placeholders in sql
     */
    function _replacePlaceholdersInSQL(&$data) {
        global $USERINFO;
        // allow current user name in filter:
        $data['sql'] = str_replace('%user%', $_SERVER['REMOTE_USER'], $data['sql']);
        $data['sql'] = str_replace('%groups%', implode("','", (array) $USERINFO['grps']), $data['sql']$
        // allow current date in filter:
        $data['sql'] = str_replace('%now%', dformat(null, '%Y-%m-%d'), $data['sql']);
 
        // language filter
        $data['sql'] = $this->makeTranslationReplacement($data['sql']);
    }

to this:

plugins/data/helper.php
    /**
     * Replace placeholders in sql
     */
    function _replacePlaceholdersInSQL(&$data) {
        global $USERINFO;
        // allow current user name in filter:
        $data['sql'] = str_replace('%user%', $_SERVER['REMOTE_USER'], $data['sql']);
        $data['sql'] = str_replace('%groups%', implode("','", (array) $USERINFO['grps']), $data['sql']$
        // allow current date in filter:
        $data['sql'] = str_replace('%now%', dformat(null, '%d-%m-%Y'), $data['sql']);
 
        // language filter
        $data['sql'] = $this->makeTranslationReplacement($data['sql']);
    }

and change this:

plugins/data/helper.php
switch($type) {
            case 'dt':
                if(preg_match('/^(\d\d\d\d)-(\d\d?)-(\d\d?)$/', $value, $m)) {
                    return sprintf('%d-%02d-%02d', $m[1], $m[2], $m[3]);
                }

to this:

plugins/data/helper.php
switch($type) {
            case 'dt':
                if(preg_match('/^(\d\d?)-(\d\d?)-(\d\d\d\d)$/', $value, $m)) {
                    return sprintf('%02d-%02d-%d', $m[1], $m[2], $m[3]);
                }

and change this:

plugins/data/script.js
/**
 * Init datepicker for all date fields
 */
jQuery(function () {
    jQuery('.data_type_dt input').datepicker({
        dateFormat: "yy-mm-dd",
        changeMonth: true,
        changeYear: true
    });

to this:

plugins/data/script.js
/**
 * Init datepicker for all date fields
 */
jQuery(function () {
    jQuery('.data_type_dt input').datepicker({
        dateFormat: "dd-mm-yy",
        changeMonth: true,
        changeYear: true
    });

SFITCS 2016-11-26 01:24

Bugs, Feature Requests and Patches

Please submit bugs and feature requests in the issue tracker on GitHub. Patches should be sent unified diff format or as git patches against the devel branch. Or even better: fork the repository at github and send a merge request.

1)
any additional dataentry on the same page would overwrite any preceding dataentry on this page and so - while many dataentry blocks on one page may be displayed - only the values of the last one would be stored in the SQLite database and could be retrieved as an output by e.g. a datatable
2)
details: an s suffix results for e.g. “adam, eve” in a list that you can request per element with ~ or as a whole with *~, without s suffix “adam, eve” will be stored as a string that you can request only as a whole with *~ - given that, in most contexts the s suffix is rather a matter of semantic clearness than a technical requirement (for field values that humans understand as multiple or list values)
plugin/data.txt · Last modified: 2024-02-01 23:50 by andi

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki