Editing Elixir/HTML dump scraper (section)

== Resumability ==
The scraping job is extremely slow—our first run took two months.  If the job crashes for any reason, it's crucial that we can resume again at roughly the same place it was stopped.

We've implemented two levels of resumability and idempotence:

The processing is broken down into small units which each write a single file.  If a file already exists, then we skip the corresponding calculation.  This general caching technique is known as [[w:Memoization|memoization]].

During each unit file, also write to a checkpoint file at multiples of 100 articles processed.  The output file is written in chunks immediately after writing to the checkpoint file, which reduces the window of time in which a crash can result in inconsistency.  When resuming, if a checkpoint file is present then the job will skip articles without processing them, until the total count catches up with the count given in the checkpoint file.  The overall behavior of the job is therefore "at least once" processing, meaning that the only type of irregularity that can happen is to duplicate work for some articles.