Elixir/HTML dump scraper

A new and wondrous data source has become available to Wikimedia researchers and hobbyists: semantic HTML dumps of all articles on wiki. Previously, archived wiki content was only available as raw wikitext, which is notoriously difficult to parse for information, or even to render. MediaWiki's wikitext depends on a number of extensions and is usually tied to a specific site's configuration and user-generated templates. This recursively parsed content can never expanded exactly as it was intended when written.

HTML dumps are an improvement in every way: content, structure and information are available in a form that can be read by ordinary tools—and the original wikitext is still available as RDFa annotations which makes the new format something like a superset.

At my day job doing Technical Wishes for Wikimedia Germany, we found a reason to parse these new dumps: it's the only reliable way to count the footnotes on each article. I'll go into some detail about why other data sources wouldn't have sufficed, and also why we're counting footnotes in the first place.

Project link ((in progress)):
https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump

Reference parsing

What are references?

References are the little footnotes all over Wikipedia articles: These footnotes ground the writing in sources, and are a key aspect of the intellectual culture of Wikipedia since all claims are supposed to be paraphrased from existing secondary sources.

Challenges in wikitext

A raw reference is straightforward in wikitext and looks like: <ref>This footnote.</ref>. If this were the end of the story, it would be simple to parse references. What makes it more complicated is that many references are produced using reusable templates, for example: {{sfn|Hacker|Grimwood|2011|p=290}}.

If "{{sfn}}" were the only template then we could search for "ref" tags and "sfn" templates in wikitext. But a search for reference-producing templates unveils over 12,000 different templates on English Wikipedia alone, and these will be different on every wiki and language edition.

Simplicity of HTML

Once the wikitext is fully rendered to HTML, we can finally see all of the footnotes which were produced. They appear something like this, <div typeof="mw:Extension/ref">Footnote text.</div>

Since the rendering is complete, we know exactly which references are visible, which is better than the potential references that we might have been able to determine from a static analysis of each template.

Template expansion also maps to HTML hierarchical structure, which makes it possible to tell when a reference was produced by templates or when a reference contains templates. Both of these cases are interesting to our research.

Resumability

The scraping job is extremely slow—our first run took two months. If the job crashes for any reason, it's crucial that we can resume again at roughly the same place it was stopped.

We've implemented two levels of resumability and idempotence:

The processing is broken down into small units which each write a single file. If a file already exists, then we skip the corresponding calculation. This general caching technique is known as memoization.

During each step of processing, we also write to a separate checkpoint file after every 100 articles. The output file is written in chunks immediately after writing to the checkpoint file, which reduces the window of time in which a crash results in inconsistency. When resuming, if a checkpoint file is present then the job will skip articles until it catches up with the number given there. The overall behavior of the job is therefore "at least once" processing, meaning that the only type of irregularity that can happen is that some articles might be duplicated in the output.

Concurrency

There are hundreds of separate wikis, so splitting up the work by wiki and processing these concurrently is a natural first implementation.

When splitting by wiki, we ran into an interesting problem where the partitioning function was using :erlang.phash2 to hash an object which contained the wiki ID so we assumed that it would give different results for each wiki, but as it turns out the Flow.partition function needed explicit clues to correctly split by wiki.

The next obvious fork point would be in the phase which makes external API requests, but this is trickier because we want to limit total concurrency across all wikis as well, to avoid overwhelming the service. This should be implemented with a connection pool, ideally one which reuses a small number of connections according to HTTP/1.1 .