devel:scalability
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
devel:scalability [2015-03-24 23:57] – [Experiences] 72.19.145.242 | devel:scalability [2023-03-11 13:17] (current) – [Pages] Added link which gives guidance on the number of files which can be stored per directory on various file systems. 178.38.198.28 | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Scalability ====== | + | ====== Scalability |
- | NEEDSATTENTION | + | |
- | ** NOT EVEN A DRAFT YET ** | + | |
- | This page is for notes and thoughts on the scalability of DokuWiki with particular reference | + | This page tries to collect some information |
- | Reminder: characteristics | + | 95% of everyone considering using DokuWiki will not even need to bother reading this page. Unless you want to run a public wiki for a very popular topic or want to use some very underpowered hardware performance should not be a problem for you. |
- | * read often, updated less often | + | |
- | * sounds like LDAP entry per page, attribudes for source text, meta... | + | |
- | * mixed and unstructured content | + | |
- | A file system | + | As of June 2020, the largest known Dokuwiki installation |
- | Much of a DokuWiki' | + | ===== Limiting Factors ===== |
- | Searching a wiki or more properly, indexing a wiki to enable quick searching is an area where DBMS technology can assist a wiki. DokuWiki implements its own indexing by creating a list of every " | + | ==== Pages ==== |
- | A DBMS can make it easier to cross the single server threshold. At its simplest, the DBMS is placed on one box and the web server / application component on another. It's arguable that DokuWiki could accomplish something similar by using NFS mounts to allow multiple locally remote (i.e. LAN connected) servers to access the wiki data directory tree. I am not aware of any DokuWiki installation that has implemented anything similar. | + | The number of pages is not limited. Some people have hundreds of thousands pages in their wiki and no problems at all, but it all " |
- | For very large installations where the data will have to reside | + | Your Filesystem |
- | ====== Experiences ====== | + | The question how many files can be stored on different file systems was answered here: https:// |
- | I'd like to get some experiences with very large installations. Maybe from people who converted existing documents to [[: | + | Of course you are limited by disk space. Of course media files, pages and older revisions will take up space. However |
- | What does '' | + | === Inodes === |
+ | Unix/Linux servers may also have a limit on the number of inodes, which are metadata for each file on the server. Running out of available inodes can interrupt the normal functioning of your server. To reduce the number of inodes used, you can reduce the amount of files used (for example, it might make sense to remove attic files that are over X years old depending on the use case). | ||
- | **Note** To give People some idea how to rate your experiences, | + | ==== Mediafiles ==== |
- | * ./data/*txt = 986 items, 6.6MB and ./media/* = 853, 227MB. (Update, 17/04/08: > Now up to 1893 pages and still going strong) Intranet installation of 55 users across three locations. Running on FreeBSD. Lovely system! Our users are continually discovering new ways to put it to work. At this size, it would be great to have a more granular search system (i.e., search whole wiki, or just a particular namespace). I love that DokuWiki is not database driven - it's great to be a able to manipulate the page content using standard UNIX tools. -- //Nick Fahey, 23 Aug 2007// | + | Basically |
- | * Read [[:search]] - namespace search is possible | + | |
- | * < | + | |
- | * 159 documents (788KB), intranet, 6 users -- everything | + | |
- | * 312 documents (802KB data/ and 2079KB media/), intranet, 5 users -- runs smoothly.\\ The search | + | |
- | * 173 documents (1.2MB), here at splitbrain.org. | + | |
- | * 201 documents (1.5MB data/ 9MB media/), http:// | + | |
- | * 973 documents (7.3MB), intranet with 12 users -- works fine and no performance problems | + | |
- | * 7,733 documents (23.4MB), on single user system, searches > 5 minutes. On hosted server removed search facility and replaced with another. Even excluding 6632 documents (18.6MB) having huge problems with Web spiders such as Google, effectively creating a denial of service. | + | |
- | * 94 documents, (190k), works fine but search allocates more than the default 8MB of space allowed for PHP scripts so needed to tweak php.ini. | + | |
- | * One of my files was 253kb, and failed to display, presumably the parser timed out while reading the raw text file. No issues with the general data tree, which spans several meg over a half-dozen namespaces. | + | |
- | * 780 pages (3.6MB), working just fine. Search code in newer versions a big improvement! | + | |
- | * 295 pages (1.25MB) + 1.8MB media data, works fast and reliable. Very heavy usage of ACLs, users/ | + | |
- | * 650 pages (5.9 MB) Shared host; serves approx. 6000 pages a day on DokuWiki. The current version 20050922 works fine, with feed caching and better search engine. The older version had some performance issues on those two points. | + | |
- | * 1477 pages (13.2 MB), intranet with 1-2 users (being tested for scalability) - Dedicated dual-processor server w/5GB RAM and eAccelerator. Searches are slow, but almost manageable. Viewing namespace indexes is almost impossible, and I get frequent PHP timeouts even after increasing the timeout value to 60 seconds. | + | |
- | * 1496 pages (11.6 MB), increasing (slowly) at the rate of about 10 per day. 80 users, spread across three continents. Shared dual-processor server (Microsoft IIS). Search is problematic across namespaces, but otherwise very few problems. | + | |
- | * 325 pages (14.5 MB), 50 users, on a dual-proccessor 2.8 mhz opteron, with 1 GB RAM, Opensuse, PHP 5, no problem, work fine. | + | |
- | * Wiki: 5588 pages (31 MB wiki source) + 1712 images (196 MB media data). Hardware: 1.4GHz AMD Athlon + 256 MB RAM. DokuWiki version: 2005-05-07. Suggested search mode: download the GZIPped wiki source | + | |
- | * 1500 or so pages (10.66 MB), shared (hosted) Linux server, PHP 5.1.4, latest DokuWiki. NO problems; search is good. | + | |
- | * 2000 pages and 2.5gb media over 6 DokuWiki installs on a single server - RHEL 4, twin 32-bit intel xeon CPUS, 8gb RAM (server also runs various static web content), search disabled, caching enabled. Very fast and responsive. | + | |
- | * 18448 pages in 131M of text and 22gb of media across two servers. Both apache2 on rhel 5, shared servers with 30Gigs of ram, and 64bit ~12 cpus. Data lives on nfs mounted filer. It runs a bit slow, the search is very very slow and pretty much unusable. We have a ton of stuff writing .txt files, would love to have better search. | + | |
- | * We have about 328,467 pages on our site. Total size of pages is about 2.2GB. | + | |
- | ===== Links ===== | + | |
- | * Maybe also interesting: | + | |
+ | Since the media manager does not do paging (currently), | ||
- | ===== Discussion ===== | + | Mediafiles are resized through PHP's libGD mechanism by default. A more efficient way may be to use ImageMagick via the [[config: |
- | * Are there any hints, how to improve DokuWiki during runtime? | + | |
- | * Does somebody has an idea which elements | + | |
- | * The search is always going to be slower than SQL-based Wiki software, because DokuWiki' | + | ==== Search |
- | * A plugin giving Dokuwiki | + | |
- | | + | In the early days of DokuWiki, search used to be the limiting |
- | * Just an idea: DokuWiki pages are in server | + | |
- | * It allows helps to add additional caching. We use the [[http://www.dokuwiki.org/plugin: | + | Today, DokuWiki uses an **index based search**. That search index makes searching much faster since a term is simply looked up in the index and the results are immediately available. The index is word based, |
+ | |||
+ | In theory, | ||
+ | |||
+ | ==== Disk I/O ==== | ||
+ | |||
+ | DokuWiki relies heavily | ||
+ | |||
+ | Modern Operating Systems will cache disk access to frequently used files in memory. The more RAM your server has, the more is available to be used as file system cache. | ||
+ | |||
+ | DokuWiki | ||
+ | |||
+ | We try to avoid whole directory scans as much as possible, but sometimes it's necessary. The [[config:readdircache]] option may help mitigate the problem somewhat. | ||
+ | |||
+ | The reliance on the filesystem however makes it extremely difficult to run a DokuWiki on multiple servers. For performance reasons, it is not recommended | ||
+ | |||
+ | Disabling access time recording in your filesystem may be a good idea to decrease IO load. | ||
+ | |||
+ | ==== PHP Version ==== | ||
+ | |||
+ | Each and every new PHP version gets better and faster. There is a huge difference between PHP 5 and 7, and some smaller differences between the different minor versions of 7. Always use the best version available to you! | ||
+ | |||
+ | ==== Webserver ==== | ||
+ | |||
+ | The Webserver needs to pass requests to PHP. We recommend using mod_php or FPM for that. FastCGI is slower. Unfortunately FastCGI seems to be the only option when using IIS, so if you can use something else - do so. | ||
+ | |||
+ | |||
+ | ===== Mitigations ===== | ||
+ | |||
+ | The above already gives a few hints on where you may experience bottlenecks. Here is a list of further reading on how to mitigate problems when they occur. | ||
+ | |||
+ | |||
+ | * use better hardware - sometimes simply upgrading to a better | ||
+ | * use a dedicated search engine | ||
+ | * [[plugin: | ||
+ | * [[plugin: | ||
+ | * [[plugin: | ||
+ | * improve end user caching | ||
+ | * [[plugin: | ||
+ | |||
+ | ===== See also ===== | ||
+ | |||
+ | * [[faq: | ||
+ | * [[wpmeta> | ||
+ | |||
+ | ===== Experiences ===== | ||
+ | |||
+ | We'd like to collect some real world number of " | ||
+ | |||
+ | > **dokuwiki.org** has currently 4859 pages with about 15k pageviews per day on average | ||
+ | > still on PHP 5.6, Linux/ext4 with noatime, Apache + FPM | ||
+ | > running on a i7-6700 CPU with 64GB of RAM and SSD. That server | ||
+ | > --- [[user> | ||
+ | |||
+ | >> I am also curious if there are other " | ||
+ | >> **[Off-topic]** | ||
+ | >> --- [[user> |
devel/scalability.1427237839.txt.gz · Last modified: 2015-03-24 23:57 by 72.19.145.242