Editing Elixir/HTML dump scraper (section)

== Concurrency ==
There are hundreds of separate wikis, so splitting up the work by wiki and processing these concurrently is a natural first implementation.

When splitting by wiki, we ran into an interesting problem where the partitioning function was using <code>:erlang.phash2</code> to hash an object which contained the wiki ID so we assumed that it would give different results for each wiki, but as it turns out the <code>Flow.partition</code> function needed explicit clues to correctly split by wiki.

The next obvious fork point would be in the phase which makes external API requests, but this is trickier because we want to limit total concurrency across all wikis as well, to avoid overwhelming the service.  This should be implemented with a connection pool, ideally one which reuses a small number of connections according to HTTP/1.1 .