Differences

This shows you the differences between two versions of the page.

--- plugin:referrers [2009-10-10 18:18] – tag cleanup laynee
+++ plugin:referrers [2020-09-19 10:22] (current) – [Automatic commit of referrers page versions] tjgrant@tatewake.com
@@ Line 1: / Line 1: @@
-====== Referrers for DokuWiki ======
+====== Referrers Plugin ======
 ---- plugin ----
-description: This plugin logs referrers for your site to the page :wiki:referrers, so you can easily see where your traffic comes from.
+description: This plugin logs referrers for your site to the page :wiki:referrers, so you can easily see where your traffic comes from
 author     : Terence J. Grant
 email      : tjgrant@tatewake.com
 type       : action
-lastupdate : 2009-05-26
+lastupdate : 2020-09-19
-compatible : (current version)
+compatible : 2020-07-29 "Hogfather", 2018-04-22 "Greebo", 2017-02-19 "Frusterick Manners", 2016-06-26 "Elenor Of Tsort"
 depends    :
 conflicts  :
-similar    : google_analytics
+similar    : googleanalytics
 tags       : statistics, logging
+downloadurl: https://github.com/tatewake/dokuwiki-plugin-referrers/zipball/master
+bugtracker : https://github.com/tatewake/dokuwiki-plugin-referrers/issues
+sourcerepo : https://github.com/tatewake/dokuwiki-plugin-referrers
+donationurl:
+screenshot_img:
 ----
-[[http://tatewake.com/wiki/projects:referrers_for_dokuwiki|Details and Download]]
+===== License =====
+  * **Author:** [[tjgrant@tatewake.com|Terence J. Grant]]
+  * **License:** [[http://www.opensource.org/licenses/gpl-license.php|GNU GPL v2]]
+  * **Donate:** [[https://www.paypal.com/xclick/business=tjgrant%40tatewake.com&item_name=Referrers%20for%20DokuWiki%20Donation&no_shipping=1&no_note=1&tax=0&currency_code=USD&lc=US|Click here.]]
+===== Download =====
+Before downloading or using this product, make sure you __**understand and accept**__ the terms of the [[#license]].
+After downloading, make sure to follow the [[#install]] instructions or [[#upgrading]] instructions below; trust me, they're worth reading.
+  * This is the **latest version**, compatible with **The Latest DokuWiki**:
+    * Latest Version: [[https://github.com/tatewake/dokuwiki-plugin-referrers/zipball/master]]
+    * GitHub URL: [[https://github.com/tatewake/dokuwiki-plugin-referrers]]
+  * Older downloads are available on request only.
+**Note: https://github.com/tatewake/dokuwiki-plugin-referrers/zipball/master always points to the latest stable version!**
+===== Installation =====
+Use the plugin manager to install, or...
+  - [[#download]] the tarball/zip-file
+  - unpack it into ''<dokuwiki>/lib/plugins/''
+  - login as admin and change the plugin in the configuration manager
+**using git:**
+<code>
+% cd <dokuwiki>/lib/plugins/
+% git clone git://github.com/tatewake/dokuwiki-plugin-referrers.git referrers
+</code>
+**Note: https://github.com/tatewake/dokuwiki-plugin-referrers/zipball/master always points to the latest stable version!**
+===== About =====
+This plugin logs referrers for your site, so you can see where your traffic comes from.
+Set the options for this plugin via the **Configuration Settings** menu from the DokuWiki admin menu. (It will be near the bottom of the page.)
+|{{http://imageshack.us/a/img502/7879/warningb.png?48x48&nolink |Warning}}The log will be located at **:wiki:referrers**, so you will need to create that page for this plugin to work.  |
+===== Install =====
+As a plugin all you need to do is unpack the file into the lib/plugins/ directory (you should end up with a lib/plugins/referrers folder.)
+Remember to create a page in the **wiki** namespace called **referrers**.
+===== Upgrading =====
+To upgrade, remove the original lib/plugins/referrers folder, and install the new version as instructed above. You may wish to make a note of your settings first though.
+===== What's New =====
+==== Sep 19, 2020 ====
+  * Updated release with nicer documentation and code cleanup
+==== May 26, 2009 ====
+  * Plugin is now an **action plugin** that requires no additional PHP code modification.
+  * Referrer spam blocking has been removed in order to simplify the code.
+==== April 29, 2007 ====
+  * More referrer spam blocking is in place.
+  * Better support for not logging spiders and search engines.
+  * Non-standard browser/client IDs will also get logged.
+==== September 3, 2006 ====
+  * Initial release
+====== Discussion ======
+  * Start all subtopics using H2. (Heading 2.)
+===== Adding username to referrers output =====
+For anyone finding this plug-in as useful as I am, here's a mod to code.php that will add the user's DokuWiki username as the third column of the wiki:referrers output.  Note: After this change, you will have to add a " | " between the IPs and ref urls for all the old lines in your wiki:referrers page.  Otherwise, old data will NOT display properly.  If the old data isn't important, just delete the old lines...much easier.  Here's the details:
+<code>
+function re_log_referrers_3($refignore)
+{
+...
+                                #fwrite($fp, "^ hostname ^ ip address ^ referrer ^\n");
+                                fwrite($fp, "^ hostname ^ ip address ^ username ^ referrer ^\n");
+                        }
+                        #write new ref
+                        #$rh = getenv("REMOTE_ADDR");
+                        $rh = $_SERVER[REMOTE_ADDR];
+                        $ru = $_SERVER[REMOTE_USER];
+                        if (strcmp($ru, "") == 0) { $ru = "Unknown"; }
+                        fwrite($fp, "| %%".gethostbyaddr($rh)."%% | ");
+                        #fwrite($fp, $rh);
+                        fwrite($fp, $rh . " | " . $ru);
+</code>
+// Bobby Metz 07/25/2007 //
+===== Disabling Null entries =====
+Though rare, some servers don't provide an HTTP_REFERER value and the plug-in will record NULL in the referrer column.  While advantageous for access logs, this also allows for the logging of internal POST requests made by your DokuWiki itself.  So any internal links that are clicked or users who directly browse to your DokuWiki will be recorded as a NULL.  Also, when a remote website redirects to your DokuWiki you will end up with an entry for that server followed by a NULL entry where the doku.php has internally redirected based on the GET attributes.  Since these entries do not list websites referencing your page, here's how to disable them:
+<code>
+        # HTTP_REFERRER is blank when user directly requests the page
+        # or remote server does not provide it's address
+        if (strcmp($refhi, "") > 0)
+        if (re_contains($ref,$refignore) == FALSE)
+        if (re_contains($ref,' 420source.com') == FALSE)
+</code>
+// Bobby Metz 07/25/2007 //
+===== Referrer script is slow =====
+//Moved from monobook for dokuwiki.//
+//Hi I have a quite serious issue with using the template, let me give a few information on my configuration first
+  * Fedora core 5 with Xampp
+  * I switched my website root from /opt/lampp/htdocs to /opt/lampp/htdocs/dokuwiki in httpd.conf
+The problem is that everything is working just fine with the default template but that the wiki turns very slow when using monobook except if browsing on the server using the localhost address. To be clear, with monobook :
+  * using http://localhost on the server : FAST
+  * using http://192.168.x.y on the server : VERY SLOW (3-4 seconds per page)
+  * using http://192.168.x.y on another machine of the network : VERY SLOW (3-4 seconds per page)
+Anybody got this problem before ?
+Thx in advance//
+----
+Did you for some reason include other.php in main.php? If so, this would be the reason for the slow down. (Its an experimental disabled module for the time being.) If not, the only other thing I can think of is this...
+From my experience, slow downs in php are mostly due to file system related activity; I use a fair amount of "file_exists" and "include" calls in the template. So the culprit could be that your PHP is getting choked by the amount of file calls. (Possibly your file system is a slow one?)
+Beyond this, in my opinion, there are probably a few things in the apache config that might improve your performance. (Though I don't know what it would be.) I'd suggest investingating PHP speed problems with Apache and your machine config; it'll probably boil down to that.
+ --- //[[tjgrant@tatewake.com|Terence J. Grant]] 05/27/2006 21:17//
+----
+Hi, and thank you for your answer. This morning, I debugged main.php and I found out that it was this part of the code that was slowing down the page generation :
+<code> <?php @include(dirname(__FILE__).'/referrers.php'); ?> </code>
+I just removed it this way and it's working fine now
+<code> <?php /*@include(dirname(__FILE__).'/referrers.php');*/ ?>
+</code>
+Is it safe to just comment it ? I'm not interested in the feature anyway.
+Thanks again
+----
+I had originally thought that might be the case as well, but for whatever reason I forgot to mention that possibility. If anything, I would imagine the reverse lookups in referrers.php would be the slowdown, but again it could be due to the file system calls. You can comment that out and everything will work just fine.  --- //[[tjgrant@tatewake.com|Terence J. Grant]] 05/29/2006 02:02//
+----
+The root cause of this problem is that machine with local, fake NAT IP addresses (192.168.X.Y) don't have reverse IP maps.  That means that anytime the server tries to find out the name of a system on that subnet to generate the referrer info, it's going to try to find it via DNS, which will take a few seconds to time-out before returning nothing useful.  When hitting the server from the 127.0.0.1 address, DNS knows that's always the localhost so the response is instant.  While the "right" approach to this problem is to screen out the NAT subnets at the DNS level, the reality of this situation is that you're always going to run into situations where Internet hosts don't have valid reverse IP addresses periodically; also, many people don't have access to the broken DNS server or the knowledge to fix it.
+This sort of problem is always coming up with using gethostbyaddr in any programming language.  At a minimum, consider screening out and ignoring the RFC 1918 private IP addresses 127.X.X.X, 10.X.X.X, 172.16-31.X.X, and 192.168.X.X, in your code, which will get rid of this issue altogether for most users.  Alternately or additionally, you may want to consider using a more robust implementation with a quicker timeout than the default, because this can be the root of a denial of service attack of sorts.  There is a discussion of this issue with some workarounds at http://us3.php.net/gethostbyaddr you may want to look into; just calling the function naked the way you do in referrers.php is asking for problems like this to appear at random times for users.  --- //[[gsmith@gregsmith.com|Greg Smith]] 06/21/2006 08:56 //
+----
+Thanks for the information and tips, I'll look at making this better for the next release.  --- //[[tjgrant@tatewake.com|Terence J. Grant]] 06/21/2006 11:36//
+===== Modifying for https sites =====
+I needed to block refs for my site which does https only.  This takes a simple change to code.php.  Following the $a & $b used by the author, you can do this:
+<code>
+$a = 'http://'.$refignore;
+$b = 'http://www.'.$refignore;
+$c = 'https://'.$refignore;
+...
+        if (re_beginsWith($ref,$a) == FALSE)
+        if (re_beginsWith($ref,$b) == FALSE)
+        if (re_beginsWith($ref,$c) == FALSE)
+</code>
+However, it seemed that the author added this as an afterthought and since most of the traffic will be interanl linking, I thought a couple of slight modifications to the original code was better, i.e. no more $a, $b, etc.
+<code>
+#$a = 'http://'.$refignore;
+#$b = 'http://www.'.$refignore;
+        if (re_contains($ref,$refignore) == FALSE)
+        if (re_contains($ref,'420source.com') == FALSE)
+</code>
+The first line will match any ref url including your page.  Using the re_beginsWith just makes extra comparisons, depending on how you specify your page url in the local_pref.php file.
+// Bobby Metz 07/25/2007 //
+Well, I'm going to go for $a, $b, $c rather than just do a contains. If someone does a "google translate" on your site, pops "yoursite.xxx" into a search engine, or has something like a blog entry with your site name in the url, you won't get it logged in the referrer logs if you use the "contains" function. So although $a, $b, $c, etc isn't horribly elegant, I think I'd rather keep it this way. (So I'll add the https ones for the next version.)  --- //[[tjgrant@tatewake.com|Terence J. Grant]] 08/02/2007 07:43//
+===== Manually Create Preferences File =====
+After installing this plug-in I was surprised to find that the local preferences for it are not loaded from the main DokuWiki local.php file.  There is a nice configuration page, but for high security sites which don't allow web-based admin changes this is a problem.  There is no documentation outlining where and how the preferences file should be created.  So, if anyone out there needs it, here's the skinny:
+File name: lib/tpl/referrers/local_pref.php
+Default settings to add:
+<code>
+<?php
+$re_settings['referrer-ignore'] = '';
+$re_settings['dontcountadmin'] = '';
+$re_settings['dontcountusers'] = '';
+?>
+</code>
+// Bobby Metz 07/25/2007 //
+Whoops, ok this'll be added to the next release, thanks.  --- //[[tjgrant@tatewake.com|Terence J. Grant]] 08/02/2007 07:53//
+===== Speedier Exec - Getting rid of getenv lag =====
+With dicussions of file system slowness above, I too began digging for ways to use this very useful plug-in while minimize user impact.  Here's another means to increase responsiveness of the referrers plug-in...remove all the getenv calls.  While this won't produce monumental speed increases, there is no point in making the system work harder when all the variables needed by this module are already loaded by PHP in the $_SERVER array.  On my system I can save about 4ms per getenv call I remove.  There are a total of nine such calls in the code.php file, so 36ms faster give or take on an average day.  Here's what to change in code.php:
+<code>
+function re_log_referrers_3($refignore)
+{
+...
+        #$refhi = getenv("HTTP_REFERER");
+        $refhi = $_SERVER[HTTP_REFERER];
+...
+                        #$rh = getenv("REMOTE_ADDR");
+                        $rh = $_SERVER[REMOTE_ADDR];
+...
+                        #if (re_beginsWith(getenv(HTTP_USER_AGENT), "Mozilla") == FALSE)
+                        #if (re_beginsWith(getenv(HTTP_USER_AGENT), "Opera") == FALSE)
+                        if (re_beginsWith($_SERVER[HTTP_USER_AGENT], "Mozilla") == FALSE)
+                        if (re_beginsWith($_SERVER[HTTP_USER_AGENT], "Opera") == FALSE)
+...
+function re_log_referrers_2($refignore)
+{
+        global $_SERVER;
+        #$x = gethostbyaddr(getenv("REMOTE_ADDR"));
+        $x = gethostbyaddr($_SERVER[REMOTE_ADDR]);
+...
+        #if (!re_contains(strtolower(getenv(HTTP_USER_AGENT)), "bot"))
+        #if (!re_contains(strtolower(getenv(HTTP_USER_AGENT)), "crawl"))
+        #if (!re_contains(strtolower(getenv(HTTP_USER_AGENT)), "spider"))
+        #if (!re_contains(strtolower(getenv(HTTP_USER_AGENT)), "larbin"))
+        if (!re_contains(strtolower($_SERVER[HTTP_USER_AGENT]), "bot"))
+        if (!re_contains(strtolower($_SERVER[HTTP_USER_AGENT]), "crawl"))
+        if (!re_contains(strtolower($_SERVER[HTTP_USER_AGENT]), "spider"))
+        if (!re_contains(strtolower($_SERVER[HTTP_USER_AGENT]), "larbin"))
+</code>
+// Bobby Metz 07/25/2007 //
+Ok, this will be in for the next version, thanks.  --- //[[tjgrant@tatewake.com|Terence J. Grant]] 08/02/2007 07:42//
+=====Question=====
+>The instructions say to insert a couple of php code lines in "your template... between <head> and </head>" without specifying
+a file, or even a file type. A typical template has like two dozen files in there. Are we talking about some css file?
+Typically you'll want to do this for any PHP (or HTML) files containing the string "<head>"; at a minimum you can do just main.php.
+If you only do main.php though you'll miss some hits to certain types of pages, so I suggest searching for head tags through all the php files in a template to make sure you get them all.  --- //[[tjgrant@tatewake.com|Terence J. Grant]] 08/11/2007 13:38//
+===== All referrers are NULL? =====
+Hi -\\ I had been running a fairly old version of Dokuwiki/Monobook for a long time. After I upgraded to the latest versions of Dokuwiki, Monobook software and added the referrers plugin, I noticed that **all** logged visits have a referrer value of //NULL//. I do not understand why, because my Apache "combined" log shows non-null and correct referrer fields. \\ I would have assumed that the HTTP_REFERER environment as well as $_SERVER parameters (I have tried both in the ''code.php'' script file) would carry the referrer information. Any ideas why I am getting empty values?
+--- //[[alien@slackware.com|Eric Hameleers]] 18/nov/2008 13:45//
+**//Edit://** It kept bugging me so I looked at the ''code.php'' again and saw a problem when the ''$re_settings['referrer-ignore']'' would be set to the default //empty// value. All non-null referrers will be filtered out of the resulting ''referrers.txt'' file because of the lines: <code>
+$a = 'http://'.$refignore;
+$b = 'http://www.'.$refignore;
+</code> and the checks on <code>
+        if (re_beginsWith($ref,$a) == FALSE)
+        if (re_beginsWith($ref,$b) == FALSE)
+</code>
+So I manually created ''./lib/plugins/referrers/local_pref.php'' as documented in one of the earlier sections of this page, and as a side effect, changing the defaults for the plugin in the admin section of the wiki works now (it used to spit errors at me).
+Now, with the value of ''$re_settings['referrer-ignore']'' set to my Wiki URL, real referrer values are pouring in :-) \\
+--- //[[alien@slackware.com|Eric Hameleers]] 18/nov/2008 14:54//
+===== Automatic commit of referrers page versions =====
-===== Discussion =====
+Just a tip...\\
+I thought it was a shame to see the referral information disappear at midnight, when the referrers.txt is reset. The DokuWiki has a few php scripts useful for automation, //''./bin/dwpage.php''// being one of those. So, I added the following to my crontab which causes the actual content of referrers.txt page to be committed to the attic. Now, I can recall //old revisions// of the ''wiki:referrers'' page. <code>
+# Archival of the wiki referrers page (otherwise all data is lost at day change)
+0 * * *       /usr/bin/php /xxxxxxx/htdocs/dokuwiki/bin/dwpage.php -m "Automatic archival" commit /xxx/htdocs/refempty.txt wiki:referrers 2>/dev/null
+</code> Where the external file //''refempty.txt''// only contains some placeholder text (it will be overwritten in a matter of minutes anyway).\\
+--- //[[alien@slackware.com|Eric Hameleers]] 19/nov/2008 01:28//
-Please use the [[http://tatewake.com/wiki/talk:projects:referrers_for_dokuwiki|discussion page]] for this plugin.