Draft:Elixir/bzip2-ex

A chronicle of my first Erlang/Elixir library binding (NIF).

Adam Wight, Sept 2022

Background

Phap Nang Ngam Nai Wannakhadi (1964, p 60)

One common way to analyze Wikipedia content is to process its database backup dumps, provided as bzip2-compressed XML. The unpacked files are too large to manipulate locally and unwieldly even when compressed, so a common practice is to stream the content, decompress in memory, and analyze in a single pass. Perhaps this step extracts a smaller subset of the data and saves it to disk.

Of course, one would normally search for a proven library written in a solid data-sciencey language such as Python, run pip install mwxml and go on with the day.

That wouldn't be as interesting as trying to do exactly the same thing in an esoteric young language, in service of a pet project with zero adoption.

"But imagine how much better a truly concurrent wiki dump processor could be!" —~~~

Problem statement

Unfortunately, there was no BEAM (Elixir- and Erlang-compatible) library for reading bzip2 files, so we could either fetch and decompress the data externally through an separate process, or write a new library.

How hard could it be to write a binding...

Is libbzip2 maintained?

Um, sure... The official bzip2 pages might be hosted on a Red Hat site nobody has heard of since 2005, very likely written as static HTML, the project has no edits to its changelog since 2019 (don't worry, the code has been changed even if the changelog has not), but yes bzip2 1.0 is the current version dammit!

What isn't mentioned on the official page, but is well documented in the Wikipedia article, is that ownership has peacefully moved to a new author (while slipping through another would-be maintainer's fingertips) who will kindly take us into the future with bzip2 1.1 and beyond—once this is released and mainstream, please edit my comments here.

Don't be confused just because I am.

The bzip2 file format has no specification

That's a bigger problem and we will ignore until it comes back angry—which it does.

High- or low-level integration?

We already chose the "hard landing" by writing a new library in the first place, but there's still an opportunity for making life sort of easy again, by choosing one of two APIs provided by libbzip2, either the high-level interface which opens a file and returns the decompressed contents, or the low-level interface for maximum interactivity, which is called with tiny bites of data, shares its memory with the caller, exposes an internal state machine, and offers your humble developer many forms of additional stimulation.

There were a few considerations here, and in theory the high-level interface would have made a perfectly acceptable binding. However, this would be a single call to decompress the world, and control flow wouldn't return to Elixir for several hours. This pretty much defeats the purpose of writing a binding library for a concurrent virtual machine.

Native implemented function (NIF)

This is where I began to feel like I was learning stuff.

Elixir owes its existence to Erlang and the virtual machine that runs them both, BEAM. Extending the machine is done by writing a NIF library which consists of compiled C/C++ and a small Erlang stub providing the interface. Actually, the stub can be written in Elixir just as well, which was a nice surprise.