Giving WikiApiary a kick

A few days ago I was listening to some of the talks at MUDCon (The MediaWiki conference aimed at non-Wikimedia uses of MediaWiki).

During James Hare's talk about a project to keep track of various Miraheze wikis using Wikibase (The software behind Wikidata), he briefly mentioned that WikiApiary has been down forever, and that maybe a Wikibase approach would be better instead of the previous Semanitc MediaWiki approach (Semantic MediaWiki is an extension to MediaWiki that allowing annotating links with the "relationship" the link represents and querying those relationshops).

This reminded me that WikiApiary existed, and i thought i would try to give it a kick. For those who don't know, WikiApiary is/was a site that tracked what public mediawiki isntances were out there and what extensions they had installed.

I had already gotten server access years ago (People wanted me to help but i never really did). The status of the site was sort of up but extremely flakey and timing out all the time.

Web server concurrency

If you've been paying attention to web hosting at all in the last while, you know that AI scrapper bots are the bane of everyone's existence.

Initially I assumed that was what was happening here. I may have been partially wrong on that, but i think it was a contributing factor.

One of the most common performance problems with MediaWiki is people setting up apache to use the default max 150 threads. If a large spike in traffic comes in, all the threads fight over resources and everything becomes really slow, causing even more threads to pile up, slowing everything down to a halt. Even worse if you don't have enough memory and have swap enabled, you end up with swap death (This server did not have swap enabled, but i mention it because its such a common failure case). Its often better to try and process a few requests at the time and make the other requests wait their turn than to try and do too much all at once and accomplish nothing.

 To address this, I installed varnish. Varnish is a great piece of software that lets you cache recently used pages, reducing server load significantly in general. It can also help by deduplicating requests to a certain extent, if two people request the same page at the same time, it will just send one request to the backend so the backend doesn't get overloaded processing the same page twice for two separate people

It can also set a maximum number of requests in flight to the backend. This can make sure that the backend doesn't get too overloaded.

One of the less used features of varnish is the ability to set up multiple backends. I like to use this to setup different backends for different classes of requests. For WikiApiary I setup four - likely bots, non-normal article views (e.g. diffs, history, special), normal article views, logged in users & images. For each of these I had different back-ends with a different number of max requests at one time. For bots, i set it to at most one. Mostly in case of false positives. My bot metric is just using an old version of chrome. Comparatively, normal page was given 15 and non-normal pages 3. Thus bots should not be able to take down the site, at worse they could just take down requests in the same class as them. I gave effectively no limit (actually 60) for logged in users and requests to static resources like images that are super cheap to serve.

As a special case, I outright blocked facebook's AI scrapper as it was being super aggressive. I also blocked logged out access to Special:Browse. I hate outright blocking things for logged out users; how can lurkers become contributors if you block them from everything? However, Special:Browse was using OFFSET based paging and was super expensive to render while at the same time being linked everywhere and a very common spider target.

This is what I ended up with for the varnish config:

  backend default {
    .host = "127.0.0.1";
    .port = "8088";
    .max_connections = 60;
    # Unfortunately the version of varnish i have is too old for .wait_limit
    # and .wait_timeout. However if you're doing something like this with small max connections, you definitely want a wait queue to even out spikes.
#    .wait_limit = 10;
#    .wait_timeout = 10s;
}

backend page {
    .host = "127.0.0.1";
    .port = "8088";
    .max_connections = 15;
#    .wait_timeout = 30s;
#    .wait_limit = 60;
    .first_byte_timeout = 200s;
    .between_bytes_timeout = 300s;
}

backend bot {
    .host = "127.0.0.1";
    .port = "8088";
    .max_connections = 1;
    .first_byte_timeout = 200s;
    .between_bytes_timeout = 300s;
}

backend special {
    .host = "127.0.0.1";
    .port = "8088";
    .max_connections = 3;
 #   .wait_timeout = 60s;
 #   .wait_limit = 15;
    .first_byte_timeout = 120s;
    .between_bytes_timeout = 200s;
}
# access control list for "purge": open to only localhost and other local nodes
acl purge {
    "127.0.0.1";
}

# vcl_recv is called whenever a request is received 
sub vcl_recv {
        # Serve objects up to 2 minutes past their expiry if the backend
        # is slow to respond.
        set req.grace = 500s;

        set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;

        set req.backend_hint= default;

        # This uses the ACL action called "purge". Basically if a request to
        # PURGE the cache comes from anywhere other than localhost, ignore it.
        if (req.method == "PURGE") {
            if (!client.ip ~ purge) {
                return (synth(405, "Not allowed."));
            } else {
                return (purge);
            }
        }

	# Was crawling very fast.
	if ( req.http.User-Agent ~ "^meta-externalagent" ) {
		return (synth( 403, "No crawling please" ) );
	}
        
        # Pass requests from logged-in users directly.
        # Only detect cookies with "session" and "Token" in file name, otherwise nothing get cached.
        if (req.http.Authorization || req.http.Cookie ~ "([sS]ession|Token)=") {
            return (pass);
        } /* Not cacheable by default */

	if ( req.url ~ "^/w/index.php\?(.*&t|t)itle=Property:.*&limit=\d*&offset=\d\d\d\d" ) {
		return (synth( 403, "Log in to view more" ));
	}

	# rate limit < chrome 136. MSIE. Opera. (Note samsung browser is chrome 136)
	# Also applebot, only rate limiting instead of blocking because it is well behaved.
	if (req.http.User-Agent ~ "(Chrome/[0-9][0-9]\.|Chrome/1[012][0-9]|Chrome/13[0-5]|Opera|MSIE|Applebot)" && req.url ~ "^/(wiki/|w/index.php|w/api.php)" ) {
		set req.backend_hint = bot;
	} elsif (req.method == "GET" && 
		( req.url ~ "^/w/index.php" || req.url ~ "^/wiki/Special:" || req.url ~ "/wiki/.*\?" || req.url ~ "^/w/api.php" ) &&
		!( req.url ~ "(Special:CreateAccount|Special:UserLogin|Special:RecentChanges|Special:Random)" )
	) {
		set req.backend_hint= special;
	} elsif ( req.method == "GET" && req.url ~ "^/wiki/" ) {
		set req.backend_hint= page;
	} elsif( req.method == "POST" && req.url ~ "(^/w/api.php|Special(:|%3A)Browse)" ) {
		set req.backend_hint= special;
	}

	if ( req.method != "GET" ) {
		return (pass);
	}

        # normalize Accept-Encoding to reduce vary
        if (req.http.Accept-Encoding) {
          if (req.http.User-Agent ~ "MSIE 6") {
            unset req.http.Accept-Encoding;
          } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
          } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
          } else {
            unset req.http.Accept-Encoding;
          }
        }
 
        return (hash);
}

sub vcl_pipe {
        # Note that only the first request to the backend will have
        # X-Forwarded-For set.  If you use X-Forwarded-For and want to
        # have it set for all requests, make sure to have:
        # set req.http.connection = "close";
 
        # This is otherwise not necessary if you do not do any request rewriting.
 
        set req.http.connection = "close";
}

# Called if the cache has a copy of the page.
sub vcl_hit {
        if (!obj.ttl > 0s) {
            return (pass);
        }
}

# Called after a document has been successfully retrieved from the backend.
sub vcl_backend_response {
        # Don't cache 50x responses
        if (beresp.status == 500 || beresp.status == 502 || beresp.status == 503 || beresp.status == 504) {
            set beresp.uncacheable = true;
            return (deliver);
        }   
        if (beresp.http.Set-Cookie) {
          set beresp.uncacheable = true;
          return (deliver);
        }

        if (!beresp.ttl > 0s) {
          set beresp.uncacheable = true;
          return (deliver);
        }
 
 
        if (beresp.http.Authorization && !beresp.http.Cache-Control ~ "public") {
          set beresp.uncacheable = true;
          return (deliver);
        }

	set beresp.grace = 2h;
        return (deliver);
}

  

Request Limits

However, I still saw lots of slow requests piling up. Sometimes DB queries seemed to take longer than the request stayed open, which was very pointless as the user had gone away by the time it was done.

Ideally the user would not be able to trigger super slow requests, however in a system like SemanticMediaWiki where the user is allowed to make arbitrary queries (and quite frankly a questionably optimized DB schema) its going to happen.

So the important thing is to make sure if a query that can't be answered in a reasonable amount of time happens, that we stop processing it instead of just wasting resources on it. This is extra important as slow requests can have a cascade effect; the first request is slow taking up a lot of resources making other requests at the same time slow down. Throughput falls and suddenly more requests come in also slowed by the general business of the system. Basically a traffic jam happens.

To deal with this I did two things:

• Install php-excimer package and set $wgRequestTimeLimit to 300. This allows MediaWiki to gracefully set a time limit for itself. Unlike php's execution_time, the php-excimer extension allows handling the timeout gracefully and also applies the timeout to wall clock time instead of cpu time (Important because in an overload, the CPU might be split amongst many php processes so only a little cpu time might have passed)
• Set mariaDB's max_statement_time config to 200 seconds. This is the max amount of time a query can take before it is killed. This ensures that a run away query is time limited. You have to be careful though since this config is global, you'll want to disable it before running any sort of DB maintenance.

This helped a bit to prevent things from piling up. However queries were still slow and when i looked at iotop & top it seemed like mariadb was using excessive CPU & Disk I/O but almost no memory.

Almost no memory? That doesn't seem right for a database. I'd missed the obvious thing: innodb_buffer_pool_size was set to only 128 MB! This is one of the most important settings for MariaDB/MySQL performance. It effectively determines how much RAM the DB users (while one time of ram anyways, kind of the important one). The server had 23GB of ram and the DB was limited to 128MB. No wonder queries were slow. Traditional advice is for a dedicated server this value should be 80% of the server's RAM. We have to subtract from that for the RAM MediaWiki needs as both were on the same server, but nonetheless this needed to be way higher. I upped it to 11GB and query speed increased by orders of magnitude almost instantly.

I also noticed that the temporary table in memory size was really small. This doesn't matter much for vanilla mediawiki, but for extensions like SemanticMediaWiki that do a lot of user defined queries that sort through many results this is really important, so I bumped max_heap_table_size, tmp_table_size and tmp_memory_table_size to 800MB.

The Database

As I looked through the slow query log, I saw a lot of Semantic MediaWiki queries that just seem somewhat questionable, at least at first glance. Semantic MediaWiki seems to love SELECT DISTINCT, which in some cases is much harder for MySQL to optimize. I think there is a lot of room for optimization in SemanticMediaWiki. I filed a bug about things that looked off to me, but I'm not a SemanticMediaWiki dev so maybe there are reasons things are the way they are.

 

I did do some local schema changes, removing some indexes that seemed duplicating other indexes, making the index on smw_hash only index the first 8 bytes of the sha1 hash instead of the whole thing. I don't really know if that helped or not, but it seems like for a large database like this, making indexes take less room means more likely data already in the buffer pool and less disk i/o to fetch things. After compressing page revisions and running optimize on all the tables, disk space usage dropped by about 160GB which was also good as disk was about 75% full.

 

Honestly, it seems like the SMW query model would be better served by something like InnoDB full text search. I think there is already an optional module for elasticsearch backend, but its not default and requires the external dependency. SMW searches are largely intersection queries between a bunch of properties. There are also some operators like <, > LIKE and !=, but those seem to be used much less. Its a graph model, but doesn't seem to implement any sort of path queries. I imagine the elasticsearch backend would do very well with this

I also tried to optimize some of the templates. Some of this involved moving array functions based templates to lua. Array functions is a MediaWiki extension providing an array_map like functionality in wikitext. However it tends to be very slow.

Some of the SMW queries in templates were essentially doing their own group by. They would request 5000 results and try and find all the unique answers.  This seemed slow for large properties such as the page for Extension:ParserFunctions. It seemed only part of this was on the database side (The sortkey isn't indexed, so mysql has to look at all the results no matter how many you return) but it seemed like SMW was doing an additional query per result which was also adding latency.

Instead I tried to make it get a single result and then use greater than query to get the next result. This seemed much faster, however it did not work properly for pages with properties with multiple values. Apparently in SemanticMediaWiki, multi valued properties are returned together in a single row, not like SQL where an inner join makes multiple. At the same time, queries have to match all the properties, so its essentially impossible to do any sort of condition beside = on a multivalued property. if you say [[My Property::!Foo]] to match it not having property foo, if the property's value is Foo and Bar, then the query still matches the bar part. Even if you do [[My Property::!Foo]] [[My Property::!Bar]], it still wouldn't work because each value in the multivalued property is tested separately, and Foo matches not Bar and Bar matches not Foo, so either way the page matches. This seems like a major oversight in the query model of SMW, but i guess it is what it is.

Conclusion

It works now. Not every page is super speedy. Some outliers, like the page on Extension:ParserFunctions still takes about a minute to load, but all pages at least load now. Previously even the simple pages were taking about 600 seconds to load, with almost none of them actually loading before hitting a timeout.

Of course, the update bots are not enabled yet, and I'm not sure what impact that will have. I'm not exactly sure how those worked originally or what it will take to turn those back on.

I also heard there is now a competing WikiApiary type site - https://catalogue.ai.wu.ac.at/ check them out!

How did XSSProtector do?

 A few months ago, frustrated by the lack of defensive anti-XSS measures in MediaWiki, I decided to make my own. Hence was born Extension:XSSProtector.

 This extension is a compromise - its the best I can do from an extension without changing anything in MediaWiki. However I think it does provide real safety against the most likely vulnerabilities in MediaWiki.

Yesterday, MediaWiki released a security update for core and bundled extensions, so lets see how XSSProtector fared:

[Note: Severity ratings for vulns are my own opinion as there is no official rating]

The Vulns

High

• ✅ (T401099, CVE-2025-61638) SECURITY: Sanitize data- attributes.
• Stored XSS in MediaWiki's parser.
  
• As an aside, this is an amazing find by gui-ying233. Its not often that people find stored XSS in MediaWiki's core parser. 

Moderate

• ✅ (T397232, CVE-2025-61656) SECURITY: Sanitize attributes unwrapped from data-ve-attributes.
• Basically if you can trick a user to copy and past something evil into visual editor, you can take over their account. Moderate because it requires complex user interaction, but is ultimately fairly serious.
  

 Low

• ❌ (T387478, CVE-2025-61634) SECURITY: REST: Set cache-control value of max-age=60 for redirects.
• ✅ (T394396, CVE-2025-61636) SECURITY: Escape rawElement $content.
• ✅ (T394856, CVE-2025-61637) SECURITY: Escape three system messages used by live preview.
• ❌ (T280413, CVE-2025-61639) SECURITY: Use ManualLogEntry::getDeleted in ::getRecentChange.
• Not an XSS type bug
• ❌ (T403757, CVE-2025-61643) SECURITY: Don't send suppressed recent changes to RCFeeds.
• ❌ (T398706, CVE-2025-61646) SECURITY: Prevent leaking hidden usernames in Watchlist/RecentChanges.
• ✅ (T402075, CVE-2025-61640) SECURITY: Parse messages instead of inserting them as HTML.
• ❌ (T298690, CVE-2025-61641) SECURITY: api: Disable maxsize in QueryAllPages in miser mode.
• ✅ (T402313, CVE-2025-61642) SECURITY: Escape submit button label for Codex-based HTMLForms.
• ✅ (T403761, CVE-2025-61645) SECURITY: Fix i18n XSS in CodexTablePager.
• ✅ [CheckUser] (T403408, CVE-2025-61651) SECURITY: fix XSS in tempuser-expired-link-tooltip message.
• ❌ [CheckUser] (T404805, CVE-2025-61658) SECURITY: Add config variable to exclude from GlobalContributions.
• ✅ [CheckUser] (T402077, CVE-2025-61648) SECURITY: Escape system messages before inserting them as HTML.
• ❌ [ConfirmEdit] (T355073, CVE-2025-61635) SECURITY: ApiFancyCaptchaReload: Reuse badcaptcha rate limit.
• ❌ [DiscussionTools]  (T397580, CVE-2025-61652) SECURITY: In API check user read permissions before showing PageInfo.
• I consider this low as it requires an unsupported configuration. People who have private wikis using officially supported configs are not affected
  
• ❌ [DiscussionTools] (T364910, T396248, CVE-2025-11175) SECURITY: DiscussionTools should use better regex.
  
• ❌ [OATHAuth] (T401862, T402094, CVE-2025-11173) SECURITY: Reauth for enabling 2FA can be bypassed by submitting a form.
• ❌[OATHAuth] (T396951) FreeOTP refuses to add MediaWiki's 2FA details, because "token is unsafe".
• ❌ [TextExtracts] (T397577, CVE-2025-61653) SECURITY: Add authorizeRead check for extracts endpoint.
• I'm considering this low because it requires configuring MediaWiki in an officially unsupported configuration. Normal private wikis are not affected as far as i can tell.
  
• ❌ [Thanks] (T397497, CVE-2025-61654) SECURITY: Exclude deleted entries when counting thanks.
• I think most users don't really consider this sensitive information.
  
• ✅ [VisualEditor] (T395858, CVE-2025-61655) SECURITY: Properly escape and parse system messages.
• ✅ [Vector] (T398636, CVE-2025-61657) SECURITY: Insert sticky header labels as text instead of HTML.

 In conclusion

It stopped all the XSS vulns, including the two that actually matter for your average MediaWiki setup. Overall it got 11 out of 24 or 46%. However I think its important to emphasize that most of the low vulnerabilities either can only be triggered by an admin, can only happen in rare configurations, or are DoS vulnerabilities that only matter if you've already spent significant effort doing performance hardening. XSSProtector prevented all the vulnerabilities that your average MediaWiki install should be worried about.

Preventing XSS in MediaWiki - A new extension to protect your Wiki

 Its no secret that the vast majority of serious security vulnerabilities in MediaWiki are Cross-Site-Scripting (XSS).

XSS is where an attacker can put evil javascript where they aren't supposed to in order to take over other users. For example, the typical attack would look like the attacker putting some javascript in a wiki page. The javascript would contain some instructions for the web browser, like make a specific edit. Then anyone who views the page would make the edit. Essentially it lets evil people take over other users' accounts. This is obviously quite problematic in a wiki environment.

This is the year 2025. We shouldn't have to deal with this anymore. Back in Y2K the advice was that "Web Users Should Not Engage in Promiscuous Browsing". A quarter of a century later, we have a better solution: Content-Security-Policy.

Everyone's favourite security technology: CSP

Content Security Policy (CSP) is a major web browser security technology designed to tackle this problem. Its actually a grab-bag of a lot of things, which sometimes makes it difficult to talk about, as its not one solution but a bunch of potential solutions to a bunch of different problems. Thus when people bring it up in conversation they can often talk past each other if they are talking about different parts of CSP.

 

First and foremost though CSP is designed to tackle XSS.

 

The traditional wisdom with CSP is that its easy if you start with it, but difficult to apply it afterwards in an effective way. Effective being the operative word. Since CSP has so many options and knobs, it is very easy to apply a CSP policy that does nothing but looks like it's doing something.

This isn't the first time I've tried MediaWiki and CSP. Back when I used to work for the Wikimedia Foundation in 2020, I was tasked with trying to make something work with CSP. Unfortunately it never really got finished. After I left, nobody developed it further and it was never deployed. *sads*

Part of the reason is I think the effort tried to do much all at once. From Wikimedia's perspective there are two big issues that they might want to solve: XSS and "privacy". XSS is very traditional, but privacy is somewhat unique to Wikimedia. Wikimedia sites allows users and admins to customize javascript. This is about as terrible an idea as it sounds, but here we are. There are various soft-norms around what people can do. Generally its expected that you are not allowed to send any data (even implicitly such as someone's IP address by loading an off-site resource) without their permission. CSP has the potential to enforce this, but its a more complex project then just the XSS piece. In theory the previous attempt was going to try and address both, which in retrospect was probably too much scope all at once relative to the resources dedicated to the project. In any case, after i left my job the project died.

Can we go simpler?

Recently I've been kind of curious about the idea of CSP but simple. What is the absolute minimal viable product for CSP in MediaWiki?

For starters this is just going to focus on XSS. Outside of Wikimedia, the privacy piece is not cared about very much. I don't know, maybe Miraheze care (not sure), but I doubt anyone else does. Most MediaWiki installs there is a much closer connection between the "interface-admin" group and the people running the servers, thus there is less need to restrict what interface-admin group can do. In any case, I don't work for WMF anymore, I'm not interested in dealing with all the political wrangling that would be needed to make something happen in the Wikimedia world. However, Wikimedia is not the only user of MediaWiki and perhaps there is still something useful we could easily do here.

The main insight is that the body of article and i18n messages generally should not contain javascript at all, but that is where most XSS attacks will occur. So if we can use CSP to disable all forms of javascript except <script> tags, and then use a post processing filter to filter all script tags out of the body of the article, we should be golden. At the same time, this should involve almost no changes to MediaWiki.

This is definitely not the recommended way of using CSP. Its entirely possible I'm missing something here and there is a way to bypass it. That said, I think this will work.

What exactly are we doing

So I made an Extension - XSSProtector. Here is what it does:

• Set CSP script-src-attr 'none'.
• This disables html attributes like onclick or onfocus. Code following MediaWiki conventions should never use these, but they are very common in attacks where you can bypass attribute sanitization. It is also very common in javascript based attacks, since the .innerHTML JS API ignores <script> tags but processes the on attributes.
• Look for <script tag in content added for output (i.e. in OutputPage) and replace it with &lt;script tag. MediaWiki code following coding conventions should always use ResourceLoader or at least OutputPage::addHeadItem to add scripts, so only evil stuff should match. If it is in an attribute, there should be no harm with replacing with entity
• Ditto for <meta and <base tags. Kind of a side point, but you can use <meta http-equiv="refresh" ... to redirect the page. <base can be used to adjust where resources are loaded from, and sometimes to pass data via the target attribute. We also use base-uri CSP directive to restrict this.
• Add an additional CSP tag after page load - script-src-elem *, this disables unsafe-inline after page load. MediaWiki uses dynamic inline script tags during initial load for "legacy" scripts. I don't think it needs that after page load (Though i'm honestly not sure). The primary reason to do this is to disable javascript: URIs, which would be a major method to otherwise bypass this system.
• We also try to regex out links with javascript URIs, but the regex is sketchy and i don't have great confidence in it the same way i do with the regex for <script.
• Restrict form-action targets to 'self' to reduce risk of scriptless XSS that tricks users with forms
  

The main thing this misses is <style> tags. Attackers could potentially add them to extract data from a page, either by unclosed markup loading a resource that contains the rest of the page in the url or via attacks that use attribute selectors in CSS (so-called "scriptless xss").  It also could allow the attacker to make the page display weird in an attempt to trick the user. This would be pretty hard to block, especially if TemplateStyles extension is enabled, and the risk is relatively quite low as there is not much you can do with it. In any case, I decided not to care

The way the extension hooks into the Message class is very hacky. If this turns out to be a good idea, probably the extension would need to become part of core or new hooks would have to be added to Message.

Does it work?

Seems to. Of course, the mere fact i can't hack the thing I myself came up with isn't exactly the greatest brag. Nonetheless I think it works and I haven't been able to think of any bypasses. It also seems to not break anything in my initial testing.

 Extension support is a little less clear. I think it will work for most extensions that do normal things. Some extensions probably do things that won't work. In most cases they could be fixed by following MediaWiki coding conventions. In some cases, they are intrinsically problematic, such as Extension:Widgets.

To be very clear, this hasn't been exhaustively tested, so YMMV.

How many vulns will it stop?

Lets take a look at recent vulnerabilities in MediaWiki core. Taking a look in the vulns in the MediaWiki 1.39 release series, between 1.39.0 and 1.39.13 there were 29 security vulnerabilities.

17 of these vulnerabilities were not XSS. Note that many of these are very low severity, to the point its debatable if they even are security vulnerabilities. If I was triaging the non-XSS vulnerabilities, I would say there are 6 informational (informational is code for: I don't think this is a security vulnerability but other people disagree), 9 low severity, 2 medium-low severity. None of them come close to the severity of an (unauthenticated) XSS, although some may be on par with an XSS vuln that requires admin rights to exploit.

While I haven't explicitly tested all of them, I believe the remaining 12 would be blocked by this extension. Additionally, if we are just counting by number, this is a bit of an under count, as in many cases multiple issues are being counted as a single phab ticket, if reported at the same time.

In conclusion, this extension would have stopped 41% of the security vulnerabilities found so far in the 1.39.x release series of MediaWiki, including all of the high severity ones. That's pretty good in my opinion.

Try it yourself

You can download the extension from here. I'd be very curious if you find that the extension breaks anything or otherwise causes unusual behaviour. I'd also love for people to test it to see if they can bypass any of its protections.

It should support MediaWiki 1.39 and above, but please use the REL1_XX for the version of MediaWiki you have (i.e. On 1.39 use REL1_39 branch) as the master branch is not compatible with older MediaWiki.

Drawing with CSS in MediaWiki

One thing you aren't allowed to do when writing pages in MediaWiki, is use SVG elements.

You can include SVG files, but you can't dynamically draw something based on the contents of the page.

There are two schools of thought on adding features to MediaWiki page formatting.

Some people think we should give low level tools and let the users figure it out. An example (in my opinion) of this is Scribunto (Lua support). This let editors create small scripts in the lua programming language as macros. Its fairly low level - learning to program is complicated. However it gave users ultimate flexibility and only a few users need to learn how to make lua scripts. The other users could just use them without understanding how they work.

The other school of thought is to give users high level tools. For example, the upcoming Charts extension or the older easytimeline extension. This gives the users some flexibility, but many of the details are hidden. The lack of flexibility sounds like a downside, but there are some benefits. Its easier to ensure everything is consistent when its managed by the system instead of the user. Its easier to ensure everything meets performance requirements, accessibility requirements, etc. Its easier to ensure there is alternative content where the rich content can't be supported (e.g. perhaps in some app).

I tend to lean towards the former, as I think the extra flexibility spurns creativity. Users will take the tools and use them in ways you never imagined, which I think is a good thing (not to mention more scalable), but I understand where the other side is coming from.

One of the limitations of MediaWiki, is a template author can't really draw things dynamically. Sure they can use uploaded images and fancy HTML, but they can't use true server-side dynamically generated graphics because SVG elements aren't allowed in wikitext. Sometimes people use quite ingenious html hacks to get around this - e.g. {{Pie chart}} uses CSS borders to make pie graphs. For the most part though this niche is filled with high level extensions like the former Graph extension or the upcoming Charts extension, which let users insert a graph based on some specification. This is great if the graph the extension generates is what the user needs, but what if their requirements differ slightly?

Personally I think we should just do both - let users have at it with SVG, while also providing the fancier high level extensions. See what they like best. (Some people argue against embedded svg on security grounds, but its really no different than HTML. You just need to whitelist safe elements/attributes and not allow risky stuff).

Modern CSS

Recently though, I realized that CSS has advanced a lot, and you don't really need SVG to do drawings. You can do it in pure CSS.

CSS has a property called "clip-path". This allows specifying a clipping path in SVG path syntax. Only the parts inside the path show through.

SVG path syntax is essentially how to tell the computer, go to this point, draw a line to this point, from there make a curve to some other point, and so on and so forth. Essentially instructing the computer how to draw a picture.

This allows drawing a lot of stuff you would normally need SVG for. To be sure, there are a lot of other things SVGs do, but at its core, this is the main drawing part.

Here is an example of a smiley face and the word Example in fancy writing (Taken from [[File:Example.svg]]) in pure css

In fact, i made a start on an implementation of the canvas API in lua using this as the output mode: https://en.wikipedia.org/wiki/Module:Sandbox/Bawolff/canvas

Limitations

A big limitation is lack of stroking. The clip-path thing works great for filling regions, but it doesn't work for stroking paths.

When drawing - to stroke means to draw an outline, where to fill means you draw a shape and fill everything inside of it.

This doesn't seem insurmountable though. Most actual rasterization software works by flattening curved paths to lines, we could do the same thing in Lua, although it might not be the most efficient solution in terms of outputted markup.

Much of the time, I was actually surprised how much of the Canvas spec has CSS equivalents. Even obscure stuff - what interpolation do you want for rendered images? CSS has image-rendering property for that.

There are some things with no equivalent. We can't do anything that depends on client side, can't get font metrics, can't get the pixel data of an image, etc. Centering text the way canvas does it actually seems kind of hard. Composition operators are mostly not supported. Blending operators can be, but won't work properly with images. A bunch of other small things aren't supported, mostly involving more complex aspects of text layout.

On the whole though, I'm shocked at how far I got and how much of the stuff I don't have yet seems like a simple matter of programming.

There is also the possibility of limited interactivity with this, using :hover CSS and CSS animations (although transitioning paths probably doesn't work great, it seems possible in theory). Possibly even combined with {{calculator}} to allow limited interaction via form controls.

Try out [[Module:Sandbox/Bawolff/canvas]] if you are curious.

Hashtags and implementing extensions in MediaWiki by modifying services

 Recently I decided to make a Hashtags extension for MediaWiki.

The idea is - if you add something like #foo to your edit summary, that becomes a link in RecentChanges/history/etc, which you can click on to view other edits so tagged.

I implemented this using MediaWiki's newish services framework. This was the first time I used this as an extension mechanism, so I thought I'd write a blog post on my thoughts.

Why Hashtags

The main idea is that users can self-group their edits if they are participating in some sort of campaign. There are already users doing this and external tools to gather statistics about it. For example Wikimedia hashtag search so there seems to be user demand.

MediaWiki also has a feature called "change tags". This allows edits to be tagged in certain ways and searched. Normally this is done automatically by software such as AbuseFilter. It is actually possible for users to tag their own edits, provided that tag is on an approved list. They just have to add a secret url parameter (?wpChangeTags) or edit by following a link with the appropriate url parameter. As you can imagine, this is not easily discoverable.

I've always been excited about the possibility of hashtags. I like the idea that it is a user-determined taxonomy (so-called folksonomy). You don't have to wait for some powerful gate-keeper to approve your tag. You can just be bold, and organize things in whatever way is useful. You don't have to discover some secret parameter. You can just see that other people write a specific something in their edit summaries and follow suite.

This feels much more in line with the wiki-way. You can be bold. You can learn by copying.

The extension essentially joins both methods together, by automatically adding change tags based on the contents of the edit summary.

Admittedly, the probability of my extension being installed on Wikipedia is probably pretty low. Realistically i am not even trying for that. But one can dream.

For more information about using the extension see https://www.mediawiki.org/wiki/Extension:Hashtags

How the extension works

There are a couple things the extension needs to do:

• When an edit (or log entry) is saved, look at the edit summary, add appropriate tags based on that edit summary
• If an edit summary is revision deleted or oversighted, remove the tags
• Make sure hashtag related change tags are marked as "active" and "hidden".
  
• When viewing a list of edits (history, Special:RecentChanges, etc), ensure that hashtags are linked and that clicking on them shows only edits tagged with that tag

 The first three parts work like a traditional extension, using normal hooks.

I used the RevisionFromEditCompleteHook and ManualLogEntryBeforePublishHook to run a callback anytime an edit or log entry is saved. Originally I used RecentChange_saveHook, however that didn't cover some cases where an edit was created but not published to RecentChanges, such as during page moves. ManualLogEntryBeforePublishHook covers more cases than it might appear at first glance, because it will also tag the revision associated with the log entry. All this is pretty straightforward, and allowed tagging most cases. Restricted (confidential) logs still do not get tagged. It seems difficult to do so with the hooks MediaWiki provides, but perhaps its best not to tag such log entries, lest information is leaked.

Similarly, I used the ArticleRevisionVisibilitySetHook to delete/undelete tags after revdel. Unfortunately MediaWiki does not provide a similar hook for log entries. I proposed adding one, but that is still pending code review.

Marking tags as active was also quite straightforward using normal hooks.

All that leaves is ensuring hashtags are linked. For this I leveraged MediaWiki's newish dependency injection system.

Services and dependency injection

For the last few years, MediaWiki has been in the progress of being re-architectured, with the goal to make it more modular, easier to test, and easier to understand. A core part of that effort has been to move to a dependency injection system.

If you are familiar with MediaWiki's dependency injection system, you might want to skip past this part.

Traditionally in MediaWiki, if some code needed to call some other code from a different class, it might look like this:

class MyClass { 

    function foo($bar) {

        $someObject = Whatever::singleton();

        $result = $someObject->doSomething( $bar );

        return $result;

    }

}

In this code, when functionality of a different class is needed, you either get it via global state, call some static method or create a new instance. This often results in classes that are coupled together.

In the new system we are supposed to use, the code would look like this:

class MyClass {

    private SomeObject $someObject;

    public function __construct( SomeObject $someObject ) {

         $this->someObject = $someObject;

    }

    public function foo( $bar ) {

         return $this->someObject->doSomething( $bar );

    }

}

The idea being, classes are not supposed to directly reference one another. They can reference the interfaces of objects that are passed to it (typically in the constructor), but they are not concretely referencing anything else in the system. Your class is always given the things it needs; it never gets them for itself. In the old system, the code was always referencing the Whatever class. In the new system, the code references whatever object was passed to the constructor. In a test you can easily replace that with a different object, if you wanted to test what happens when the Whatever class returns something unexpected. Similarly, if you want to reuse the code, but in a slightly different context, you can just change the constructor args for it to reference a different class as long as it implements the required interface.

This can make unit testing a lot easier, as you can isolate just the code you want to test, without worrying about the whole system. It can also make it quite easy to extend code. Imagine you have some front-end class that references a back-end class that deals with storing things in the database. If you suddenly need to use multiple storage backends, you can just substitute the implementation of the backend class, without having to implement anything special.

MediaWikiServices

All this is great, but at some point you actually need to do stuff and create some classes that have concrete implementations.

The way this works in "new" MediaWiki, is the MediaWikiServices class. This is responsible for keeping track of "services" which are essentially the top level classes.

Services are classes that:

• Have a lifetime of the entire request.
• Normally only have one instance existing at a time.
  
• Do not depend on global state (Config is not considered state, but anything about the request, like what page is being currently viewed is state. Generally these services should not depend on RequestContext)

You can register services, in a service wiring file. This also allows you to register what services your service needs as constructor arguments.

Some classes do not fit these requirement. For example, a class that represents some data we would expect to have multiple instances with shorter lifetimes, to represent the data in question. Generally the approach for such classes is to create a Factory class that is a service, which makes individual instances and passing along dependencies as appropriate.

But still the question remains, how do you get these services initially. There is an escape hatch, where you can call MediaWikiServices::getInstance()->getService( $foo ), however that is strongly discouraged. This essentially uses global state, which defeats the point of dependency injection, where the goal is that your class is passed everything it needs, but never reaches out and gets anything itself.

The preferred solution is that the top level entrypoint classes in your extension are specified in your extension's extension.json file.

Typically extensions work on the levels of hooks. This is where you register a class, which has some methods that are called when certain events in MediaWiki happen. You register these hook handlers in your extension's manifest (extension.json) file.

In old mediawiki, you would just register the name of some static methods. In new MediaWiki, you register a class (HookHandlers), along with details of what services its constructor needs. MediaWiki then initiates this class for you with appropriate instances of all it dependencies, thus handling all the bootstrapping for you.

To summarize, you tell MediaWiki you need certain services for your hook class. When the hook class is instantiated, it is constructed with the default version of the services you need (Possibly including services you defined yourselves). Those services are in turn instantiated with whatever services they need, and so on.

All this means you can write classes that never reach out to other classes, but instead are always provided with the other classes they need.

What does this have to do with Hashtags?

All this is not generally meant as an extension mechanism. The goal is to make it so classes are isolated from each other for easier testing, modification and understanding.

However, the services bootstrap process does have an idea of default services, which it provides when creating objects that have dependencies. The entire point is to be able to easily replace services with different services that implement the same interface. Sure it is primarily meant as a means of better abstraction and testability, but why not use it for extensibility? In this extension, I used this mechanism to allow extending some core functionality with my own implementation.

One of those services is called CommentParserFactory. This is responsible for parsing edit summaries (Technically, it is responsible for creating the CommentParser objects that do so). This is exactly what we need if we want to change how links are displayed in edit summaries.

In the old system of MediaWiki, we would have no hope in making this extension. In the old days, to format an edit summary, you called Linker::formatComment(). This was a static method of Linker. There would be no hope of modifying it, unless someone explicitly made a hook just for that purpose.

Here we can simply tell MediaWiki to replace the default instance of CommentParserFactory with our own. We do this by implementing the MediaWikiServicesHook. This allows us to dynamically change the default instantiation of services.

There are two ways of doing this. You can either call $services->redefineService() which allows you to create a new version of the Service from scratch. The alternative is to call $services->addServiceManipulator(). This passes you the existing version of the service, which you can change, call methods on, or return an entirely different object.

Hashtags uses the latter method. I essentially implemented it by wrapping the existing CommentParser. I wanted to just replace hashtags with links, but continue to use MediaWiki core for all the other parsing.

Using Services as an extension mechanism

How did this go?

On the bright side, i was able to make this extension, which i otherwise would not have been able to. It indeed works.

Additionally, I feel like simply replacing a class and implementing its interface, is a much cleaner abstraction layer for extensibility than simply executing a callback at random points in the code that can do anything.

There are some frustrations though with using this approach.

First of all, everything is type hinted with the concrete class. This meant i had to extend the concrete class in order for all the type hints to work even though I didn't really need or want to inherit any methods. Perhaps it would be nice to have some trait that automatically adds all the forwarding magic with __get().

I also intentionally did not call the parent constructor, which i guess is fine because i never call any of the parent's methods, however it does feel a bit odd. The way i implemented my class was to take in the constructor an instance of the "base" class that i forward calls to after wrapping them in my own stuff. Part of the reason for this is I wanted to use the old CommentParser object, because constructor functions in MediaWiki are highly unstable between versions, so I didn't want to have to manually construct anything from MW core.

The part where this really falls down is if you want to have multiple extensions layering things. This really only works if they both do so in the same way.

At one point in my extension, I wanted to access the list of tags parsed. In retrospect, perhaps this would have been better implemented as having a separate service alias that is explicitly for my class.  Instead, i implemented it by adding an additional method to return this data and just relied on the normal service name. But i ran into the problem of i can't be sure nobody else redefined this service, so how do i know that the service i got is the one i created with this method? I just tested using instaceof, and if not tried to wrap the returned service again in my class. This has the benefit of if anyone else modifies the CommentParser service in a layered way, I will hopefully run their code and the list of tags will match the result. On the other hand its icky, so maybe the better method was the aforementioned having a separate service alias explicitly for my class. However again, that has the problem of how to bootstrap, since I didn't want to ever have to instantiate the base CommentParser class, with MediaWiki's highly unstable constructors.

This whole part seemed icky, and the various options seemed bad. In practice it maybe doesn't matter as its unlikely two extensions will wrap this service, but it seems like there should be a better way. I guess wrapping the service is fine, its extracting additional information that's the problem.

I think documenting best practices on how to "decorate" services would go a long way here to making this better. Right now it is very unclear how to do this in a way that is composable and extensible.

I've also seen examples of people using reflection when trying to "extend" services, but luckily i didn't need to resort to that.

Testability

The last thing I wanted to mention, is i do think this general system has had dividends. The path to get here was a little rocky, but I do think this style of code is much clearer.

I used to absolutely hate writing tests for mediawiki code. It was such a frustrating developer experience. Now it is much less painful. For this extension I have 100% test coverage (excluding maintenance scripts).  Admittedly it is a relatively small extension, but five years ago, I couldn't imagine bothering to do that for any MediaWiki code.

Conclusion

I had fun writing this extension and am proud of the result. It was interesting to experiment with using MediaWikiServices as an extension mechanism

Consider trying it out on your wiki, and let me know what you think! The extension should be compatible with MediaWiki 1.39 and higher (Using appropriate REL branch)

More information about it can be found at https://www.mediawiki.org/wiki/Extension:Hashtags

MediaWiki Users and Developers Conference Spring 2024

 Last week I went to Portland for the MediaWiki Users and Developers conference (nee EMWCon). This is primarily a conference for people doing stuff with MediaWiki outside of Wikimedia. I had a blast.

I always enjoy conferences on the smaller side. They feel so much more personal. This year's conference had Ward Cunningham as the guest of honour. Ward was a fascinating person to meet and get to talk to.

I also must say hats off to the organizers - conference ran smoothly, venue was great, food was amazing. Seriously some of the best food I've ever had at any Wikimedia conference.

This was also my first time in Portland. Portland is a beautiful city. I didn't have a huge amount of time to explore the city, but I did manage to go to the Chinese garden, which was absolutely stunning. I also loved how many interesting murals there were in the city. Even the graffiti seemed prettier than normal.

 

 

While listening to the talks, I realized that a good talk is very similar to a good design doc. Perhaps this is an obvious comparison, but I never really noticed before how similar the two things are. In both cases, you want to give the reader/viewer context about the problem you want to solve, what solution you chose, why you chose it and how it worked out. At the same time you want to avoid the temptation to go too far into implementation details.

I think my favourite talk was Jeffery's. He demo'd using LLMs to answer questions based on the content of the Wiki. The demo deities weren't fully in his favour, but I think it also demonstrated an important point that LLMs are cutting edge technologies that don't always give the expected answer 100% of the time. In any case, he did a great job presenting.

I did get the sense that I think some participants were disappointed that there was very little representation of WMF management (whether "real" management or product management) at the conference. Birgit did give a remote talk and Selena did come to a happy hour event after the conference, but neither really participated.

I don't think the participants necessarily wanted anything from WMF management, but there is a little bit of a feeling of being unseen. Many of the conference goers use MediaWiki for their own purposes and are interested to know what WMFs plans are for the future and how it will affect them (as do we all).

 

 

I think some participants were hoping to maybe make some connections for better mutual understanding and just reduce uncertainty about what is on the roadmap for MediaWiki. In theory Birgit's talk was about the plans for MediaWiki, but I suspect it was too laden with annual planning corporate buzzwords for anyone to figure out what it actually meant concretely.

The flip side of that of course is that open source is a do-orcracy. The corporate MediaWiki users as a general rule do not contribute back to MediaWiki core all that often, which is the price of admission to the various power structures of MediaWiki.

Create Camp

At the create camp, I had a long chat with Mark about what parts of the documentation are unclear to users new to MediaWiki. While I think all of will admit that our documentation is sub-par (bug 1), it was great to get a fresh perspective on it.

 

I think adding screencasts in addition to the written documentation can help with the problem of assumed knowledge and missing implied steps.

I also heard a bit about SemanticMediaWiki (SMW) bug 5392. This is a bug where sometimes SMW drops properties associated with a page. It seems like there is a lot of frustration among the SMW community over this bug. At the same time, it doesn't seem like anyone has seriously tried to debug it. The bug does look a bit annoying to track down. It appears to be some sort of race condition, appearing somewhat randomly and more often when there are multiple things going on at the same time (e.g. the job queue is being run with more threads seems to make it more common) but nobody really knows so hence there are no steps to reproduce. Additionally there has been no attempts to create a minimal test case (e.g. What extensions are needed for the bug to appear) nor has anyone posted any debug logs from the parses in question. No one has even determined if the properties are missing at parse time or if they are being overridden at a later time. Anyways, I suspect its going no where unless people post a lot more information on the task or they hand over a server experiencing the bug to someone good at debugging.

Conclusion

I had a great time. Hopefully I'll be able to come again next year.

MediaWiki edit summary XSS write-up

 Back in January, I discovered a stored XSS vulnerability in core MediaWiki (T355538; CVE-2024-34507). Essentially by setting a specific edit summary when editing a page, you could run javascript (And take over the account of anyone viewing the edit summary, for example on the history page or recentchanges)

MediaWiki core is generally pretty good when it comes to security. There are many sketchy extensions, and sometimes there are issues where an admin might be able to run javascript, but by and large unauthenticated XSS vulns are fairly rare. I think the last one was CVE-2021-44858 from back in 2021. The next one before that was CVE-2017-8815 in 2017, which only applied to wikis configured to have a site language of certain languages (e.g. Serbian and Chinese). At least, those were the ones I found when looking through the list. Hopefully I didn't miss any. In any case, finding XSS triggerable by an unprivleged attacker in MediaWiki core is pretty hard.

So what is the bug? The proof of concept looks like this - Create an edit with the following edit summary:

[[Special:RecentChanges#%1b0000000|link1]] [[PageThatExists#/autofocus/onfocus=alert("xss\n"+document.domain)//|link2]]

This feels a bit random at first glance. How does it work?

The edit summary parser

Whenever you edit a page on MediaWiki, there is a box for your edit summary. This is essentially MediaWiki's version of a commit message.

Very little formatting is allowed in this summary. A major exception is links. You can link to other pages by enclosing the link in [[ and ]].

So this explains a little bit about the proof-of-concept - it involves 2 links. But why 2? It doesn't work with just 1. What is with the weird link targets? They are clearly abnormal, but they also don't look like typical XSS. There are no < or >, there aren't even any unclosed quotes.

Lets take a deeper look at how MediaWiki applies formatting to these edit summaries. The code where all this happens is includes/CommentFormatter/CommentParser.php.

The first thing we might notice is the following line in CommentParser::preprocessInternal: "// \x1b needs to be stripped because it is used for link markers"

In the proof of concept, the first part is [[Special:RecentChanges#%1b0000000|link1]], where %1b appears. This is a good hint that it has something to do with link markers, whatever those are.

Link markers

But what are link markers?

When MediaWiki makes a link, it needs to know whether the page being linked to exists or not, since missing pages use a red colour. The most natural way of doing this is, when encountering a link, to check in the DB whether or not the page exists.

However, there is a problem. When rendering a long page with a lot of links, we have to do a lot of DB lookups. The lookups are simple, but still on a separate (albeit nearby server). Each page to lookup involves a local network request to fetch the page status. While that is happening, MW just sits and waits. This is all very fast, but even still it adds up a little bit if you have say 500 links on a page.

The solution to this problem was to batch the queries. Instead of immediately looking up the page, MW would put a small link marker in the page at that point and carry on. Once it is finished, it would look up all the links all at once, and then do another pass to replace all the link markers.

So this is what a link marker is, just a little marker to tell MW to come back to this spot later after it figured out if all the links exist. The format of this marker is \x1B<number> (So \x1B0000000 for the first one, \x1B0000001 for the second, and so on). \x1B is the ASCII escape character.

Back to the PoC

This explains the first part of the proof of concept: [[Special:RecentChanges#%1b0000000|link1]] - the link target is a link marker. The code has a line:

                                // Fix up urlencoded title texts (copied from Parser::replaceInternalLinks)
                                if ( strpos( $match[1], '%' ) !== false ) {
                                        $match[1] = strtr(
                                                rawurldecode( $match[1] ),
                                                [ '<' => '&lt;', '>' => '&gt;' ]
                                        );
                                }

Which normalizes titles using percent encoding to use the real characters. Thus the %1B gets replaced with an actual 0x1B byte sequence. The code did try and strip 0x1B characters earlier, but at that point, it was still just %1b and did not match the check.

We now have a link with a link marker inside of it. An important note here is that Special:RecentChanges is not a normal page. It is a special page. MediaWiki knows it exists without having to consult the database, so it does not get the link marker treatment. This is important because we cannot use it as a fake link marker if it gets replaced by a real link marker.

At this stage after inserting link markers, the proof of concept becomes the following string:

<a href="/w/index.php/Special:RecentChanges#\x1B000000" title="Special:RecentChanges">link1</a> \x1B0000000

A link with a link marker inside it!

The second link

The \x1B0000000 is a stand in for [[PageThatExists#/autofocus/onfocus=alert("xss\n"+document.domain)//|link2]].

The replacement at the end is a normal replacement, and everything is fine. However there are now two replacements - there is also the replacement inside the link: href="/w/index.php/Special:RecentChanges#\x1B000000"

This is the fake link marker that we contrived to get inserted. Unlike the normal link markers, this is inside an attribute. The replacement text assumes it is being inserted as normal HTML, not as an attribute. Since it is a full link that also has quotes inside it, the two layers of quotes will interfere with each other.

Once the replacements happen we get the following mangled HTML for our proof of concept:

<a href="/w/index.php/Special:RecentChanges#<a href="/w/index.php/Test#/autofocus/onfocus=alert(&quot;xss\n&quot;+document.domain)//" title="Test">link2</a>" title="Special:RecentChanges">link1</a> <a href="/w/index.php/Test#/autofocus/onfocus=alert(&quot;xss\n&quot;+document.domain)//" title="Test">link2</a>

This obviously looks wrong, but its a bit unclear how browsers interpret it. A little known fact about HTML - /'s can separate attributes so long as no equal signs have been encountered yet. After the browser hits the second " mark, it thinks the href attribute is closed and that the remaing is some additional attributes. The browser essentially parses the above html as if it was:

<a href="/w/index.php/Special:RecentChanges#<a href=" w="" index.php="" Test#="" autofocus onfocus="alert(&quot;xss\n&quot;+document.domain)//&quot;" title="Test">link2</a>" title="Special:RecentChanges"&gt;link1</a> <a href="/w/index.php/Test#/autofocus/onfocus=alert(&quot;xss\n&quot;+document.domain)//" title="Test">link2</a>

In other words, an <a> tag, that has an attribute named autofocus and an onfocus event handler. On page load, the link is automatically focused, which triggers the javascript in the onfocus attribute to run, allowing the attacker to do what they want.

Take aways

I think the major take aways is that running Regexes over partially parsed HTML is always scary. We've had similar issues in the past, for example T110143.

The general pattern we've used to fix this and similar issues, is make sure the replacement token has special characters that would be mangled if it appeared in an unexpected context. Concretely, we added " and ' to the token, which would get escaped if placed in an attribute, and thus no longer matching and no longer being replaced.

More generally though, I think this is a good example of why even a minimal CSP policy would be helpful.

CSP is a complex standard, that can do a lot of things and has a lot of pieces. One of the things it can do, is disable "unsafe-inline" javascript. This means javascript from attributes (like onfocus) and javascript URLs. Usually this also includes inline <script> tags without a nonce, but that part is optional. A key point here, is this also generally means you cannot execute javascript via .innerHTML anymore, which is a fairly common vector for XSS via javascript.

Normally disabling unsafe-inline would be part of a broader effort to secure javascript, however its possible to take things a step at a time. This vulnerability would have been stopped just by disabling event attributes. A surprising portion of MediaWiki & extension XSS vulns [Excluding boring - an admin can change something in an unsafe way issues] involve just html attributes (or javascript: urls), which is a web feature that nobody really needs for legit reasons and is generally considered bad practise in normal usage. Even the most minimal CSP policy might really help MediaWiki's overall security posture against XSS vulns.

For more info about the vulnerability, please see the original report at https://phabricator.wikimedia.org/T355538.

Imagining Future MediaWiki

 As we roll into 2024, I thought I'd do something a little different on this blog.

A common product vision exercise is to ask someone, imagine it is 20 years from now, what would the product look like? What missing features would it have? What small (or large) annoyances would it no longer have?

I wanted to do that exercise with MediaWiki. Sometimes it feels like MediaWiki is a little static. Most of the core ideas were implemented a long time ago. Sure there is a constant stream of improvements, some quite important, but the core product has been fixed for quite some time now. People largely interact with MediaWiki the same way they always have. When I think of new fundamental features to MediaWiki, I think of things like Echo, Lua and VisualEditor, which can hardly be considered new at this point (In fairness, maybe DiscussionTools should count as a new fundamental feature, which is quite recent). Alternatively, I might think of things that are on the edges. Wikidata is a pretty big shift, but its a separate thing from the main experience and also over a decade old at this point.

I thought it would be fun to brainstorm some crazy ideas for new features of MediaWiki, primarily in the context of large sites like Wikipedia. I'd love to hear feedback on if these ideas are just so crazy they might work, or just crazy. Hopefully it inspires others to come up with their own crazy ideas.

What is MediaWiki to me?

Before I start, I suppose I should talk about what I think the goals of the MediaWiki platform is. What is the value that should be provided by MediaWiki as a product, particularly in the context of Wikimedia-type projects?

Often I hear Wikipedia described as a top 10 document hosting website combined with a medium scale social network. While I think there is some truth to that, I would divide it differently.

I see MediaWiki as aiming to serve 4 separate goals:

• A document authoring platform
• A document viewing platform (i.e. Some people just want to read the articles).
  
• A community management tool
• A tool to collect and disseminate knowledge
  

The first two are pretty obvious. MediaWiki has to support writing Wikipedia articles. MediaWiki has to support people reading Wikipedia articles. While I often think the difference between readers and editors is overstated (or perhaps counter-productive as hiding editing features from readers reduces our recruitment pool), it is true they are different audiences with different needs.

What I think is a bit under-appreciated sometimes but just as important, is that MediaWiki is not just about creating individual articles, it is about creating a place where a community of people dedicated to writing articles can thrive. This doesn't just happen at the scale of tens of thousands of people, all sorts of processes and bureaucracy is needed for such a large group to effectively work together. While not all of that is in MediaWiki, the bulk of it is.

One of my favourite things about the wiki-world, is it is a socio-technical system. The software does not prescribe specific ways of working, but gives users the tools to create community processes themselves. I think this is one of our biggest strengths, which we must not lose sight of. However we also shouldn't totally ignore this sector and assume the community is fine on its own - we should still be on the look out for better tools to allow the community to make better processes.

Last of all, MediaWiki aims to be a tool to aid in the collection and dissemination of knowledge¹. Wikimedia's mission statement is: "Imagine a world in which every single human being can freely share in the sum of all knowledge." No one site can do that alone, not even Wikipedia. We should aim to make it easy to transfer content between sites. If a 10 billion page treatise on Pokemon is inappropriate for Wikipedia, it should be easy for an interested party to set up there own site that can house knowledge that does not fit in existing sites. We should aim to empower people to do their own thing if Wikimedia is not the right venue. We do not have a monopoly on knowledge nor should we.

As anyone who has ever tried to copy a template from Wikipedia can tell you, making forks or splits from Wikipedia is easy in theory but hard in practice. In many ways I feel this is the area where we have most failed to meet the potential of MediaWiki.

With that in mind, here are my ideas for new fundamental features in MediaWiki:

As a document authoring/viewing platform

Interactivity

Detractors of Wikipedia have often criticized how text based it is. While there are certainly plenty of pictures to illustrate, Wikipedia has typically been pretty limited when it comes to more complex multimedia. This is especially true of interactive multimedia. While I don't have first hand experience, in the early days it was often negatively compared to Microsoft Encarta on that front.

We do have certain types of interactive content, such as videos, slippy maps and 3D models, but we don't really have any options for truly interactive content. For example, physics concepts might be better illustrated with "interactive" experiments, e.g. where you can push a pendulum with a mouse and watch what happens.

One of my favourite illustrations on the web is this one of an Enigma machine. The Enigma machine for those not familiar was a mechanical device used in world war 2 to encrypt secret messages. The interactive illustration shows how an inputted message goes through various wires and rotates various disks to give the scrambled output. I think this illustrates what an Enigma machine fundamentally is better than any static picture or even video would ever be able to.

Right now there are no satisfactory solutions on Wikipedia to make this kind of content. There was a previous effort to do something in the vein of interactive content in the graph extension, which allowed using the Vega domain specific language to make interactive graphs. I've previously wrote on how I think that was a good effort but ultimately missed the mark. In short, I believe it was too high level which caused it to lack the flexibility neccessarily to meet the needs of users, while also being difficult to build simplifying abstractions overtop.

I am a big believer that instead of making complicated projects that prescribe certain ways of doing things, it is better to make simpler, lower level tools that can be combined together in complex ways, as well as abstracted over so that users can make simple interfaces (Essentially the unix philosophy). On Wiki, I think this has been borne out by the success of using Lua scripting in templates. Lua is low level (relative to other wiki interfaces), but the users were able to use that to accomplish their goals without MediaWiki developers having to think about every possible thing they might want to do. Users were than able to make abstractions that hid the low level details in every day use.

To that end, what I'd like to see, is to extend Lua to the client side. Allow special lua interfaces that allow calling other lua functions on the client side (run by JS), in order to make parts of the wiki page scriptable while being viewed instead of just while being generated.

I did make some early proof-of-concepts in this direction, see https://bawolff.net/monstranto/index.php/Main_Page for a Demo of Extension:Monstranto. See also a longer piece I wrote, as well as an essay by Yurik on the subject I found inspiring.

Mobile editing

This is one where I don't really know what the answer is, but if I imagine MW in 20 years, I certainly hope this is better.

Its not just MediaWiki, I don't think any website really has authoring long text documents on mobile figured out.

That said, I have seen some interesting ideas around, that I think are worth exploring (None of these are my own ideas)

Paragraph or sentence level editing

This idea was originally proposed about 13 years ago by Jan Paul Posma. In fact, he write a whole bachelor's thesis on it.

In essence, Mobile gets more frustrating the longer the text you are editing is. MediaWiki often works on editing at the granularity of a section, but what about editing at the granularity of a paragraph or a sentence instead? Especially if you just want to fix a typo on mobile, I feel it would be much easier if you could just hit the edit button on a sentence instead of the entire section.

Even better, I suspect that parsoid makes this a lot easier to implement now than it would have been back in the day.

Better text editing UI (e.g. Eloquent)

A while ago I was linked to a very interesting article by Scott Jenson about the problems with text editing on mobile. I think he articulated the reasons it is frustrating very well, and also proposed a better UI which he called Eloquent. I highly recommend reading the article and seeing if it makes sense to you.

In many ways, we can't really do this, as this is an android level UI not something we control in the web app. Even if we did manage to make it in a web app somehow, it would probably be a hard sell to ordinary users not used to the new UI. Nonetheless, I think it would be incredibly beneficial to experiment with alternate UIs like these, and see how far we can get. The world is increasingly going mobile, and Wikipedia is increasingly getting left behind.

Alternative editing interfaces (e.g. voice)

Maybe traditional text editing is not the way of the future. Can we do something with voice control?

It seems like voice controlled IDEs are increasingly becoming a thing. For example, here is a blog post about someone who programs with a voice programming software called Talon. It seems like there are a couple other options out there. I see Serenade mentioned quite a bit.

A project in this space that looks especially interesting is cursorless. The demo looked really cool, and i could imagine that a power user would find it easier to use a system like this to edit large blobs of WikiText than the normal text editing interface on mobile. Anyways, i reccomend watching the demo video to see what you think.

All this is to say, I think we should look really hard at the possibilities in this space for editing MediaWiki from a phone. On screen keyboards are always going to suck, might as well look to other options.

As a community building platform

Extensibility

I think it would be really cool if we had "lua" extensions. Instead of normal php extensions, a user would be able to register/upload some lua code, that gets subscribed to hooks, and do stuff. In this vision, these extension types would not be able to do anything unsafe like raw html, but would be able to do all sorts of stuff that users normally use javascript for.

This could be per user or also global. Perhaps could be integrated with a permission system to control what they can and cannot do.

I'd also like to see a super stable API abstraction layer for these (and normal extensions). Right now our extension API is fairly unstable. I would love to see a simple abstraction layer with hard stability guarantees. It wouldn't replace the normal API entirely, but would allow simpler extensions to be written in such a way that they retain stability in the long term.

Workflows

I think we could do more to support user-created workflows. The Wiki is full of user created workflows and processes. Some are quite complex others simple. For example nominating an article for deletion or !voting in an RFC.

Sometimes the more complicated ones get turned into javascript wizards, but i think that's the wrong approach. As I side earlier, I am a fan of simpler tools that can be used by ordinary users, not complex tools that do a specific task but can only be edited by developers and exist "outside" the wiki.

There's already an extension in this area (not used by Wikimedia) called PageForms. This is in the vein of what I am imagining, but I think still too heavy. Another option in this space is the PageProperties extension which also doesn't really do what I am thinking of.

What I would really want to see is an extension of the existing InputBox/preload feature.

As it stands right now, when starting a new page or section, you can give a url parameter to preload some text as well as parameters to that text to replace $1 markers.

We also have the InputBox extension to provide a text box where you can put in the name of an article to create with specific text pre-loaded.

I'd like to extend this idea, to allow users to add arbitrary widgets² (form elements) to a page, and bind those widgets to specific parameters to be preloaded.

If further processing or complex logic is needed, perhaps an option to allow the new preloaded text to be pre-processed by a lua module. This would allow complex logic in how the page is edited based on the user's inputs. If there is one theme in this blog post, it is I wish lua could be used for more things on wiki.

I still imagine the user would be presented with a diff view and have to press save, in order to prevent shenanigans where users are tricked into doing something they don't intend to.

I believe this is a very light-weight solution that also gives the community a lot of flexibility to create custom workflows in the wiki that are simple for editors to participate in.

Querying, reporting and custom metadata

This is the big controversial one.

I believe that there should be a way for users to attach custom metadata to pages and do complex queries over that metadata (including aggregation). This is important both for organizing articles as well as organizing behind the scenes workflows.

In the broader MediaWiki ecosystem, this is usually provided by either the SemanticMediaWiki or Cargo extensions. Often in third party wikis this is considered MediaWiki's killer feature. People use them to create complex workflows including things like task trackers. In essence it turns MediaWiki into a no-code/low-code user programmable workflow designer.

Unfortunately, these extensions all scale poorly, preventing their use on Wikimedia. Essentially I dream of seeing the features provided by these extensions on Wikipedia.

The existing approaches are as follows:

• Vanilla MediaWiki: Category pages, and some query pages.
• This is extremely limited. Category pages allow an alphabetical list. Query pages allow some limited pre-defined maintenance lists like list of double redirects or longest articles. Despite these limitations, Wikipedia makes great use out of categories.
  
• Vanilla mediawiki + bots:
• This is essentially Wikipedia's approach to solving this problems. Have programs do queries offsite and put the results on a page. I find this to be a really unsatisfying solution. A Wikipedian once told me that every bot is just a hacky workaround to MediaWiki failing to meet its users' needs, and I tend to agree. Less ideologically, the main issue here is its very brittle - when bots break often nobody knows who has access to the code or how it can be fixed. Additionally, they often have significant latency for updates (If they run once a week, then the latency is 7 days) and ordinary users are not really empowered to create their own queries.
  
• Wikidata (including the WDQS SPARQL endpoint)
  
• Wikidata is adjacent to this problem, but not quite trying to solve it. It is more meant as a central clearinghouse for facts, not a way to do querying inside Wikipedia. That said Wikidata does have very powerful query features in the form of SPARQL. Sometimes these are copied into Wikipedia via bots. SPARQL of course has difficult to quantify performance characteristics that make it unsuitable for direct embedding into Wikipedia articles in the MediaWiki architecture. Perhaps it could be iframed, but that is far from being a full solution.
• SemanticMediaWiki
•  This allows adding Semantic annotations to articles (i.e. Subject-verb-object type relations). It then allows querying using a custom semantic query language. The complexity of the query language make performance hard to reason about and it often scales poorly.
• Cargo
• This is very similar to SemanticMediaWiki, except it uses a relational paradigm instead of a semantic paradigm. Essentially users can define DB tables. Typically the workflow is template based, where a template is attached to a table, and specific parameters to the template are populated into the database. Users can then use (Sanitized) SQL queries to query these tables. The system uses an indexing strategy of adding one index for every attribute in the relation.
• DPL
• DPL is an extension to do complex querying and display using MediaWiki's built in metadata like categories. There are many different versions of this extension, but all of them have potential queries that scale linearly with the number of pages in the database, and sometimes even worse.

I believe none of these approaches really work for Wikipedia. They either do not support complex queries or allow too complex queries with unpredictable performance. I think the requirements are as follows:

• Good read scalability (By read, I mean scalability when generating pages (during "parse" in mediawiki speak). On Wikipedia, pages are read and regenerated a lot more often than they are edited.
  
• We want any sort of queries to have very low read latency. Having long pauses waiting for I/O during page parsing is bad in the MediaWiki architecture
• Queries should scale consistetly. They should at worse be roughly O(log n) in the number of pages on the wiki. If using a relational style database, we would want the number of rows the DBMS have to look at be no more than a fixed max number
• Eventual write consistency
• It is ok if it takes a few minutes for things using the custom metadata to update after it is written. Templates already have a delay for updating.
  
• That said, it should still be relatively quick. On the order of minutes ideally. If it takes a day or scales badly in terms of the size of the database, that would also be unacceptable.
  
• write performance does not have to scale quite as well as read performance, but should still scale reasonably well. 
  
• Predictable performance.
• Users should not be able to do anything that negatively impacts site performance
• Users should not have to be an expert (or have any knowledge) in DB performance or SQL optimization.
  
• Limits should be predictable. Timeouts suck, they can vary depending on how much load the site is under and other factors. Queries should either work or not work. Their validity should not be run-time dependent. It should be obvious to the user if their query is an acceptable query before they try and run it. There should be clear rules about what the limits of the system are.
• Results should be usable for futher processing
• e.g. You should be able to use the result inside a lua module and format it in arbitrary ways
• [Ideally] Able to be isolated from the main database, shardable, etc.
• Be able to query for a specific page, a range of pages, or aggregates of pages (e.g. Count how many pages are in a range, average of some property, etc)
• Essentially we want just enough complexity to do interesting user defined queries, but not enough that the user is able to take any action that affects performance.
  
• There are some other query types that are more obscure but maybe harder. For example geographic related queries. I don't think we need to support that.
• Intersection queries are an interesting case, as they are often useful on wiki. Ideally we would support that too.
  

 

Given these constraints I think the CouchDB model might be the best match for on-wiki querying and reporting.

Much of the CouchDB marketing material is aimed around their local data eventual consistency replication story. Which is cool and all but not what I'm interested in here. A good starting point for how their data model works is their documentation on views. To be clear, I'm not neccesarily suggesting using CouchDB, just that its data model seems like a good match to the requirements.

CouchDB is essentially a document database based around the ideas of map-reduce. You can make views which are similar to an index on a virtual column in mysql. You can also make reduce functions which calculate some function over the view. The interesting part is that the reduce function is indexed in a tree fashion, so you can efficiently get the value of the function applied to any contiguous range of the rows in logrithmic time. This allows computing aggregations of the data very efficiently. Essentially all the read queries are very efficient. Potentially write queries can be less so but it is easy to build controls around that. Creating or editing reduce functions is expensive because it requires regenerating the index, but that is expected to be a rare operation and users can be informed that results may be unreliable until it completes.

In short, the way the CouchDB data model works as applied to MediaWiki could be as follows:

• There is an emit( relationName, key, data) function added to lua. In many ways this is very similar to adding a page to a category named relationName with a sortkey specificed by key. data is optional extra data associated with this item. For performance reason, there may be a (high) limit to the max number of emit() on a page to prevent DB size from exploding.
  
• Lua gets a function query( relationName, startKey, endKey ). This returns all pages between startKey and endKey and their associated data. If there are more than X (e.g. 200) number of pages, only return the first X.
• Lua gets a queryReduced( relationName, reducerName, startKey, endKey ) which returns the reduction function over the specified range. (Main limitation here is the reduce function output must be small in size in order to make this efficient)
  
• A way is added to associate a lua module as a reduce function. Adding or modifying these functions is potentially an expensive operation. However it is probably acceptable to the user that this takes some time

All the query types here are efficient. It is not as powerful as arbitrary SQL or semantic queries, but it is still quite powerful. It allows computing fairly arbitrary aggregation queries as well as returning results in a user-specified order. The main slow parts is when a reduction function is edited or added, which is similar to how a template used on very many pages can take a while to update. Emiting a new item may also be a little slower than reading since the reducers have to be updated up the tree (With possibly contention on the root node), however that is a much rarer operation, and users would likely see it as similar to current delays in updating templates.

I suspect such a system could also potentially support intersection queries with reasonable efficiency subject to a bunch of limitations.

All performance limitations are pretty easy for the user to understand. There is some max number of items that can be emit() from a page to prevent someone from emit()ing 1000 things per page. There is a max number of results that can be returned from a query to prevent querying the entire database, and a max number of queries allowed to be made from a page. The queries involve reading a limited number of rows, often sequential. The system could probably be sharded pretty easily if a lot of data ends up in the database.

I really do think this sort of query model provides the sweet spot of complex querying but predictable good performance and would be ideal for a MediaWiki site running at scale that wanted SMW style features.

As a knowledge collection tool

Wikipedia can't do everything. One thing I'd love to see is better integration between different MediaWiki servers to allow people to go to different places if their content doesn't fit in Wikipedia.

Template Modularity/packaging

Anyone who has ever tried to use Wikipedia templates on another wiki knows it is a painful process. Trying to find all the dependencies is a complex process, not to mention if it relies on WikiData or JsonConfig (Commons data: namespace)

The templates on a Wiki are not just user content, but complex technical systems. I wish we had a better systems for packaging and distributing them.

Even within the Wikimedia movement, there is often a call for global templates. A good idea certainly, but would be less critical if templates could be bundled up and shared. Even still, having distinct boundries around templates would probably make global templates easier than the current mess of dependencies.

I should note, that there are extensions already in this vein. For example Extension:Page_import and Extension:Data_transfer. All of them are nice and all, but I think it would maybe be cooler to have the concept of discrete template/module units on wiki, so that different components are organized together in a way that is easier to follow.

Easy forking

Freedom to fork is the freedom from which all others flow. In addition to providing an avenue for people who disagree with the status quo a way to do their own thing, easy forking/mirroring is critical when censorship is at play and people want to mirror Wikipedia somewhere we cannot normally reach. However running a wiki the size of english wikipedia is quite hard, even if you don't have any traffic. Simply importing an xml dump into a mysql DB can be a struggle at the sizes we are talking about.

I think it would be cool if we made ready to go sqlite db dumps. Perhaps possibly packaged as a phar archive with MediaWiki, so you could essentially just download a huge 100 GB file, plop it somewhere, and have a mirror/fork

Even better if it could integrate with EventStream to automatically keep things up to date.

Conclusion

So those are my crazy ideas for what I think is missing in MediaWiki (With an emphasis on the Wikipedia usecase and not the third party use-case). Agree? Disagree? Hate it? I'd love to know. Maybe you have your own crazy ideas. You should post them, after all, your crazy idea cannot become reality if you keep it to yourself!

Notes:

¹ I left out "Free", because as much as I believe in "Free Culture" I believe the free part is Wikimedia's mission but not MediaWiki's.

² To clarify, by widgets i mean buttons and text boxes. I do not mean widgets in the sense of the MediaWiki extension named "Widgets".