WebStatisticsPlugin

WebStatistics using a proper database

Description

This plugins gathers statistics in realtime by storing all data into an SQL database. It basically gathers data being sent to the event logger. The database can be bootstrapped from data already present in Foswiki's events.log files. All data may be anonymized optionally.

WebStatisticsPlugin keeps track of the size of each topic being stored, that is the footprint on disk of all topic data including attachments uploaded to this topic. o

The statistics database is being maintained on a regular base (using a cronjob) to prune the amount of data that amasses over time. By default the recent 3 months are stored. Anything before will be deleted from the database.

There is a %STATS macro for various types of queries and statistics covering the most common requirements:

• activity: displays an activity stream (only makes sense if data isn't anonymized, note this is still in alpha state)
• logs: raw assess to the statistics log data
• size: display the size of parts of the wiki, e.g. an ordered list of the largesd topics or webs
• top_contributors: display the most active users of the wiki (only makes sense if data isn't anonymized)
• top_topics: display topic based statistics; this is the most common type of query
• total: agregates records counting them, e.g. the number of views, edits and uploads being performed
• last_visited: find out when a user last visited a topic

Each query can be restricted to a certain time frame.

Typical use cases are:

• number of views of this topic during the recent x days
• 10 largest topics in this web
• number of views/edits/uploads during this month
• common search queries
• top news pages
• tag cloud of topics being visited during the recent x days
• topics I visited today
• topics I've contributed to today
• number of visitors today
• etc you name it

Configuration

WebStatisticsPlugin relies on DBIPlugin's database connection. So make sure that you are using a real database (not sqlite) to get decent query performance.

Enabling the webstatistics logger

To gather statistics in realtime this plugin will hook into the logging system of Foswiki by using the setting

 $Foswiki::cfg{Log}{Implementation} = 'Foswiki::Logger::WebStatisticsLogger'; 

The normal default logger can then be configured as a secondary implementation to still gather events as normal using

 $Foswiki::cfg{WebStatisticsPlugin}{SecondaryLogger} = 'Foswiki::Logger::PlainFile'; 

or in case you are using LopDispatchContrib

 $Foswiki::cfg{WebStatisticsPlugin}{SecondaryLogger} = 'Foswiki::Logger::LogDispatch'; 

Note of warning: if your system has gathered a lot of search actions in the logs - which is most probably the case if you are using SolrPlugin - then your database may grow a lot consisting of a lot of searches being performed by solr-based wiki apps, not necessarily searches entered by a user (TODO: need to find a way to distinguish these). It then makes sense to skip gathering search statistics by adding search to

 $Foswiki::cfg{WebStatisticsPlugin}{ExcludeActions} = 'attach, changepasswd, configure, deleteUserAccount, jsonrpc, rest, sql, search'; 

This will keep your database in a reasonable size thus performing any calls to %STATS in a more reasonable time.

Importing event data

The database may be bootstraped by importing logs and measuring the current size of the system using REST calls on the command line. This bootstrap procedure might take considerable time in case your system has been running for some time already and thus a lot of data is available in working/logs/events.* This data may be imported using:

 cd <foswiki-dir>/bin ./rest /WebStatisticsPlugin/importLogs <params> 

Optional parameters are:

Parameter	Description	Default
debug	boolean switch to enable debug output	off
rebuild	boolean switch to rebuild the database from scratch by deleting any preexisting data first	off
limit	number of records to import	all
include	regular expression an event record must match to be imported to the database
exclude	regular expression an event record must not match to be imported to the database
since	date in the past since when events are imported

Measuring the size of the wiki

During the normal course of operations, the size of the wiki is being measured as new data arrives. To bootstrap the database use:

 cd <foswiki-dir>/bin ./rest /WebStatisticsPlugin/measureSize <params> 

Optional parameters are:

Parameter	Description	Default
debug	boolean switch to enable debug output	off
web	restrict measuring the size of the given web
limit	number of records to import	all
include	regular expression an event record must match to be imported to the database
exclude	regular expression an event record must not match to be imported to the database

(TODO: capture a time series of the size of the system, not only the current size)

Anonymizing statistics

If you've been using WebStatisticsPlugin for a while already and later on decide to anonymize all data already present. If you used anoymized data right from the start then you don't need this configuration feature.

 cd <foswiki-dir>/bin ./rest /WebStatisticsPlugin/anonymizeUserData [debug=on} 

This will replace any personal information with hashed values.

Pruning the database on a regular base

To trim down data as configured in $Foswiki::cfg{WebStatisticsPlugin}{ExpireAfter} establish a cronjob such as

 cd <foswiki-dir>/bin ./rest /WebStatisticsPlugin/pruneData [debug=on] [checkTopics=on] 

When checkTopics is enabled records will be deleted that don't refere to an existing topic anymore.

Integration with ClassificationPlugin

Every record also captures tag and category information when it is being convered by ClassificationPlugin. This will allow you to create even more informative statistics based on this kind of meta data. See the examples below.

Syntax

The %STATS{"<type>" … params"}% takes one mandatory parameter specifying the type of query to perform. These are

1. activity
2. duration
3. last_visited
4. logs
5. size
6. top_contributor
7. top_search
8. top_topics
9. total

Common parameters:

Parameter	Description	Default
since="...", from="..."	date or epoch seconds since when records are included
until="...", to="..."	date or epoch seconds until when records are included
count="all/unique"	there are two ways to count records: count all, or unique for each user; for example, action="view" and count="unique" will filter for unique views of a topic, that is the number of times a user has viewed a topic at least once; with count="all"= all views would have counted, so even the same user viewing the same topic more than once would up the numbers	all
header="..."	header string to be prepended to the result
format="..."	format string to render each row of the statistics	depends on the query type
separator="..."	format string put between each formatted row
footer="..."	footer string to be appened to the result
limit="..."	maximum rows to render
skip="..."	skip the first rows in the output
web="..."	filter by web
includeweb="...", excludeweb="..."	filter by list of webs
topic="..."	filter by topic
includetopic="...", excludetopic="..."	filter by list of topics
user="..."	filter by user
includeuser="...", excludeuser="..."	filter by list of users
action="..."	filter by action
includeaction="...", excludeaction="..."	filter by list of actions
category="..."	filter by category
includecategory="...", excludecategory="..."	filter by list of category
tag="..."	filter by tag
includetag="...", excludetag="..."	filter by list of tags
level="..."	filter by level
includelevel="...", excludelevel="..."	filter by list of levels
mincount="..."	exclude all records that occur fewer than mincount times
maxcount="..."	exclude all records that occur more than maxcount times
title="..."	filter by title
includetitle="...", excludetitle="..."	filter by list of titles
type="..."	filter by type
includetype="...", excludetype="..."	filter by list of types

Examples

Note belowe examples use DateManipPlugin to specify the times and NumberPlugin to format numbers nicely.

Most viewed topics this month

 %STATS{"top_topics" limit="10" action="view" since="%DATETIME{"1st" format="$epoch" lang="en"}%" header="| *Topic* | *Count* |$n" format="| $percntIF{\"istopic '$web.$topic'\" then=\"[[$web.$topic]]\" else=\"<nop>$web.$topic\"}$percnt | $count |" separator="$n" excludeuser="admin, guest" excludetopic="WebHome" web="%WEB%" }% 

Views this month

 %STATS{"total" action="view" since="%DATETIME{"1st" format="$epoch" lang="en"}%" format="<b>Number of views:</b> $percntNUMBER{\"$count\"}$percnt" excludeuser="admin, guest" excludetopic="WebHome" web="%WEB%" }% 

Most edited topics this month

 %STATS{"top_topics" limit="10" action="edit" since="%DATETIME{"1st" format="$epoch" lang="en"}%" header="| *Topic* | *Count* |$n" format="| $percntIF{\"istopic '$web.$topic'\" then=\"[[$web.$topic]]\" else=\"<nop>$web.$topic\"}$percnt | $count |" separator="$n" excludeuser="admin, guest" excludetopic="WebHome" web="%WEB%" }% 

Edits this month

 %STATS{"total" action="edit" since="%DATETIME{"1st" format="$epoch" lang="en"}%" format="<b>Number of edits:</b> $percntNUMBER{\"$count\"}$percnt" excludeuser="admin, guest" excludetopic="WebHome" web="%WEB%" }% 

Top searches this month

 %STATS{"top_searches" since="%DATETIME{"1st" format="$epoch" lang="en"}%" limit="10" header="| *Term* | *Count* |$n" format="| $flatten(message) | $count |$n" excludeuser="admin, guest" excludetopic="WebHome" web="%WEB%" }% 

Largest topics

 %STATS{"size" header="| *Topic* | *Size* |$n" format="| $percntIF{\"istopic '$web.$topic'\" then=\"[[$web.$topic]]\" else=\"<nop>$web.$topic\"}$percnt | $percntNUMBER{\"$value\" type=\"bytes\"}$percnt |$n" separator="" limit="10" sort="value desc" web="%WEB%" }% 

Size of web

 %STATS{"size" sum="value" avg="value" web="%WEB%" format="<b>Size of %WEB%:</b> $percntNUMBER{\"$sum\" type=\"bytes\"}$percnt" }% 

Top 10 contributors this month

 %STATS{"top_contributors" since="%DATETIME{"1st" lang="en" format="$epoch"}%" format=" 1 $wikiUserName($user) contributed $count time(s)$n" filter="user not in ('admin', 'guest')" limit="10" action="edit" } 

Top visitors this month

 %STATS{"top_contributors" since="%DATETIME{"1st" lang="en" format="$epoch"}%" format=" 1 $wikiUserName($user) visited $count topics$n" excludeuser="..." action="view" limit="10" }% 

Page duration

 %STATS{"duration" since="%TODAY%" header="| *Topic* | *Views* | *Min* | *Avg* | *Max* |$n" format="| $percntIF{\"istopic '$web.$topic'\" then=\"[[$web.$topic]]\" else=\"<nop>$web.$topic\"}$percnt | $count | $min | $avg | $max |" separator="$n" excludetopic="..." excludeweb="..." web="..." limit="10" 

Last visited

 %STATS{ "last_visited" web="Forum" format="$percntDATETIME{\"$time\"}$percnt - [[$web.$topic]]" limit="1" user="SomeUser" }% 

Last topics visited today

 %STATS{ "last_visited" header="| *#* | *Topic* | *Visited*|$n" format="| $index | [[$web.$topic]] | $percntDATETIME{\"$time\"}$percnt |$n" since="%TODAY%" excludetopic="%TOPIC%" excludeweb="System,Applications" }% 

Tag Cloud of visited topics this month

Note this makes use of ClassificationPlugin to gather tag information, FilterPlugin, DateManipPlugin, TagCloudPlugin and SolrPlugin

 <noautolink>%TAGCLOUD{ terms="%FORMATLIST{ "%STATS{ "top_topics" action="view" limit="250" since="%DATETIME{""30 days ago" format="$epoch" lang="en"}%" format="$tag" separator=", " web="%WEB%" order="time desc" tag="_%" count="all" }%" limit="250" exclude="^(abandoned\-project|todo)$" }%" header="<div class='solrTagCloud clsTagCloud'>" format="<span style='font-size:$weightpx;line-height:110%'> <a href='$percntSOLRSCRIPTURL{ topic=\"%BASEWEB%.%SEARCHTOPIC{default="WebSearch"}%\" tag=\"$term\" web=\"%WEB%\" union=\"web\" type=\"topic\" }$percnt' style='color:$fadeRGB(%TAGCLOUD_START{default="111,169,255"}%,%TAGCLOUD_END{default="0,89,224"}%);' class='solrFacetValue tag_$term' name='tag_$term' title='$count' value='$term'>$term</a> </span>" footer="</div>" group="<strong>$group</strong>&nbsp;" split="\s*,\s*" buckets="25" offset="13" warn="off" min="%min{default="0"}%" }%</noautolink> 

Installation Instructions

You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server.

Open configure, and open the "Extensions" section. "Extensions Operation and Maintenance" Tab → "Install, Update or Remove extensions" Tab. Click the "Search for Extensions" button. Enter part of the extension name or description and press search. Select the desired extension(s) and click install. If an extension is already installed, it will not show up in the search results.

You can also install from the shell by running the extension installer as the web server user: (Be sure to run as the webserver user, not as root!)

 cd /path/to/foswiki perl tools/extension_installer <NameOfExtension> install 

If you have any problems, or if the extension isn't available in configure, then you can still install manually from the command-line. See https://foswiki.org/Support/ManuallyInstallingExtensions for more help.

Dependencies

Name	Version	Description
Digest::SHA	>=6.0	Required
Date::Manip	>=6.0	Required.
Foswiki::Plugins::DBIPlugin	>=0	Required
Foswiki::Plugins::TopicTitlePlugin	>1.00	Required

Change History

20 Oct 2025

initial release