Draft:Elixir/bzip2-ex - Revision history

Adamw: Found another library, bzip2_decomp

2025-02-14T12:48:06Z

Found another library, bzip2_decomp

← Older revision		Revision as of 12:48, 14 February 2025
Line 1:		Line 1:
	~~A chronicle~~ of my first Erlang/Elixir library binding (NIF).		An adventure story of my first Erlang/Elixir library binding (NIF).

	''Adam Wight, Sept 2022''		''Adam Wight, Sept 2022''
Line 6:		Line 6:

	== Problem statement ==		== Problem statement ==
	[[File:Phap Nang Ngam Nai Wannakhadi (1964, p 60).jpg\|thumb\|Phap Nang Ngam Nai Wannakhadi (1964, p 60). [This painting is not titled, "Picking the low-hanging fruit". -AW]]I wanted to process some large, compressed files containing Wikipedia content<ref>https://dumps.wikimedia.org/backup-index.html</ref>, which couldn't be expanded in place. The typical approach to this problem is to stream the decompressed data through the desired analysis in memory and then throw it away.		[[File:Phap Nang Ngam Nai Wannakhadi (1964, p 60).jpg\|thumb\|Phap Nang Ngam Nai Wannakhadi (1964, p 60). [This painting is not titled, "Picking the low-hanging fruit". -AW]]I wanted to process some large, compressed files containing Wikipedia content<ref>https://dumps.wikimedia.org/backup-index.html</ref>, which couldn't be expanded in-place. The typical approach to this problem is to stream the decompressed data through the desired analysis in memory and then throw it away.

	Decompression can be accomplished by piping through an external, command-line tool or by reading the file using a native Elixir codec. In my case, I chose to mix these approaches by untarring through tar using a Port, but ~~use~~ a native bzip2 library to perform the decompression.		Decompression can be accomplished by piping through an external, command-line tool or by reading the file using a native Elixir codec. In my case, I chose to mix these approaches by untarring through tar using a Port, but writing a native bzip2 library to perform the decompression, since none existed at the time.

	In hindsight, it would have been much simpler to use command-line bunzip2. The native library should make it possible to use backpressure and concurrency. But mostly I just got excited about a small gap in the BEAM ecosystem and wanted to teach myself how to write an Erlang native implemented function, or NIF<ref>https://www.erlang.org/doc/apps/erts/erl_nif</ref>.		In hindsight, it would have been much simpler to use command-line bunzip2. The native library should make it possible to use backpressure and concurrency. But mostly I just got excited about a small gap in the BEAM ecosystem and wanted to teach myself how to write an Erlang native implemented function, or NIF<ref>https://www.erlang.org/doc/apps/erts/erl_nif</ref>.
Line 24:		Line 24:

	Here I learned the most important requirement of a NIF binding: it does work within the BEAM memory and process space but it must return control to the Elixir scheduler within a very short time period, less than 100ms or so. Low-level it is, then!		Here I learned the most important requirement of a NIF binding: it does work within the BEAM memory and process space but it must return control to the Elixir scheduler within a very short time period, less than 100ms or so. Low-level it is, then!

			If you want to look into yet another approach, Moosieus<ref>https://github.com/Moosieus/bzip2_decomp</ref> has written an Elixir binding for pure Rust bzip2-rs<ref>https://github.com/paolobarbolini/bzip2-rs</ref>. This looks good for decompression, but executes in a single run rather than streaming.

	==Native implemented function (NIF)==		==Native implemented function (NIF)==

Adamw: rewrite

2025-02-14T08:26:22Z

rewrite

← Older revision		Revision as of 08:26, 14 February 2025
Line 5:		Line 5:
	{{Project\|url=https://gitlab.com/adamwight/bzip2-ex}}		{{Project\|url=https://gitlab.com/adamwight/bzip2-ex}}

	== ~~Background~~ ==		== Problem statement ==
	[[File:Phap Nang Ngam Nai Wannakhadi (1964, p 60).jpg\|thumb\|Phap Nang Ngam Nai Wannakhadi (1964, p 60). [This painting is not titled, "Picking the low-hanging fruit". -AW]]~~One common way~~ to ~~analyze~~ Wikipedia content ~~is to mung its database backup dumps~~<ref>https://dumps.wikimedia.org/backup-index.html</ref>, which ~~are provided as [[W:bzip2\|bzip2]]-compressed XML~~. ~~These files are too large~~ to ~~use unpacked~~ and ~~unwieldy even when compressed, so are best served streaming~~.		[[File:Phap Nang Ngam Nai Wannakhadi (1964, p 60).jpg\|thumb\|Phap Nang Ngam Nai Wannakhadi (1964, p 60). [This painting is not titled, "Picking the low-hanging fruit". -AW]]I wanted to process some large, compressed files containing Wikipedia content<ref>https://dumps.wikimedia.org/backup-index.html</ref>, which couldn't be expanded in place. The typical approach to this problem is to stream the decompressed data through the desired analysis in memory and then throw it away.

	~~== Problem statement==~~		Decompression can be accomplished by piping through an external, command-line tool or by reading the file using a native Elixir codec. In my case, I chose to mix these approaches by untarring through tar using a Port, but use a native bzip2 library to perform the decompression.
	~~Unfortunately~~, ~~there was no BEAM (Elixir~~- ~~and Erlang-compatible) library for~~ reading bzip2 ~~files~~, ~~so we could either fetch~~ and ~~decompress~~ the ~~data externally through~~ an ~~separate process~~, or ~~write a glue library exposing libbzip2 functions in Elixir~~.
			In hindsight, it would have been much simpler to use command-line bunzip2. The native library should make it possible to use backpressure and concurrency. But mostly I just got excited about a small gap in the BEAM ecosystem and wanted to teach myself how to write an Erlang native implemented function, or NIF<ref>https://www.erlang.org/doc/apps/erts/erl_nif</ref>.

	How hard could it be to write a little binding...		How hard could it be to write a little binding...

	==Is libbzip2 ~~maintained?~~==		==Mysteries of libbzip2==

	Development of bzip2 has stopped at the last stable release, with v1.0.x in 2019.<ref>The project page for bzip2 v1.0 is https://sourceware.org/bzip2/.</ref> A new group has been working towards an initial fork<ref>https://gitlab.com/bzip2/bzip2/</ref> that they're calling version "1.1" but hopefully will avoid breaking changes to the programming interface.

	== The bzip2 ~~file format~~ has ~~no specification ==~~		The first interesting obstacle was that development of official bzip2 has stopped at the last stable release, with v1.0.x in 2019.<ref>The project page for bzip2 v1.0 is https://sourceware.org/bzip2/.</ref> A new group of people has been working towards a fork<ref>https://gitlab.com/bzip2/bzip2/</ref> that they're calling version "1.1" but hopefully will avoid breaking changes to the programming interface. This is still unreleased as of 2025.
	~~I can't complain~~, ~~there's a brilliant reverse-engineering effort~~<ref>https://~~github~~.~~com~~/~~dsnet~~/~~compress~~/~~blob~~/~~master~~/~~doc~~/bzip2~~-format.pdf~~</ref>.

	<~~br clear="all"~~>		The second point worth mentioning is that the bzip2 file format has no formal specification. This situation is pretty common and I can't complain, because there's a brilliant reverse-engineering<ref>https://github.com/dsnet/compress/blob/master/doc/bzip2-format.pdf</ref> effort which included the details I needed.

	==High- or low-level integration?==		==High- or low-level integration?==
	We already ~~chose~~ the ~~"hard landing" by writing~~ a ~~new library in the first place, but~~ there~~'s still an opportunity for making life sort of easy again~~, ~~by choosing one of two APIs provided by~~ libbzip2~~, either the~~ high-level<ref>https://sourceware.org/bzip2/manual/manual.html#hl-interface</ref> ~~interface~~ which ~~opens a~~ file and ~~returns the~~ decompressed ~~contents~~, or the low-level<ref>https://sourceware.org/bzip2/manual/manual.html#low-level</ref> ~~interface for maximum interactivity,~~ which ~~is called~~ with ~~tiny bites~~ of data~~, shares its memory with the caller, exposes an internal state machine, and offers your humble developer many forms of additional stimulation~~.		As I mentioned, my own project was already in hard mode due to the decision to write a NIF at all. But there was a second choice, between a the libbzip2 high-level interface<ref>https://sourceware.org/bzip2/manual/manual.html#hl-interface</ref> which does everything for you: open the file and return its contents decompressed, or the low-level interface<ref>https://sourceware.org/bzip2/manual/manual.html#low-level</ref> which works with block or even sub-block chunks of data.

	~~There were a few considerations here, and in theory~~ the ~~high-level interface would have made~~ a ~~perfectly acceptable~~ binding~~. However, this would be a single call to decompress~~ the ~~world,~~ and control ~~flow wouldn't return~~ to Elixir ~~for several hours~~. ~~This pretty much defeats the purpose of writing a binding library for a concurrent virtual machine.~~		Here I learned the most important requirement of a NIF binding: it does work within the BEAM memory and process space but it must return control to the Elixir scheduler within a very short time period, less than 100ms or so. Low-level it is, then!

	==Native implemented function (NIF)==		==Native implemented function (NIF)==
	[[File:Potato_sprout,_January_23,_2006.jpg\|right\|267x267px]]		[[File:Potato_sprout,_January_23,_2006.jpg\|right\|267x267px]]TODO...
	~~This is where I began to feel like I was learning stuff.~~

	Elixir owes its existence to Erlang and the virtual machine that runs them both, BEAM<ref>https://blog.stenmans.org/theBeamBook/#_the_erlang_virtual_machine_beam</ref>. Extending the machine is done by writing a NIF library<ref>https://www.erlang.org/doc/man/erl_nif.html</ref> which consists of a C/C++ to adapt the library, and a small Erlang or Elixir stub to define an interface.

	TODO ...

	== Parallel processing ==		== Parallel processing ==

Adamw: Link to project source

2023-03-25T19:41:55Z

Link to project source

← Older revision		Revision as of 19:41, 25 March 2023
Line 2:		Line 2:

	''Adam Wight, Sept 2022''		''Adam Wight, Sept 2022''

			{{Project\|url=https://gitlab.com/adamwight/bzip2-ex}}

	== Background ==		== Background ==
	[[File:Phap Nang Ngam Nai Wannakhadi (1964, p 60).jpg\|thumb\|Phap Nang Ngam Nai Wannakhadi (1964, p 60). [This painting is not titled, "Picking the low-hanging fruit". -AW]]]		[[File:Phap Nang Ngam Nai Wannakhadi (1964, p 60).jpg\|thumb\|Phap Nang Ngam Nai Wannakhadi (1964, p 60). [This painting is not titled, "Picking the low-hanging fruit". -AW]]One common way to analyze Wikipedia content is to mung its database backup dumps<ref>https://dumps.wikimedia.org/backup-index.html</ref>, which are provided as [[W:bzip2\|bzip2]]-compressed XML. These files are too large to use unpacked and unwieldy even when compressed, so are best served streaming.

	One common way to analyze Wikipedia content is to mung its database backup dumps<ref>https://dumps.wikimedia.org/backup-index.html</ref>, which are provided as [[W:bzip2\|bzip2]]-compressed XML. These files are too large to use unpacked and unwieldy even when compressed, so are best served streaming.

	== Problem statement==		== Problem statement==

Adamw: Adamw moved page Draft:bzip2-ex to Draft:Elixir/bzip2-ex

2022-09-13T08:24:46Z

Adamw moved page Draft:bzip2-ex to Draft:Elixir/bzip2-ex

← Older revision		Revision as of 08:24, 13 September 2022
(No difference)

Adamw: /* Is libbzip2 maintained? */ Cut some flowers

2022-09-09T20:34:57Z

Is libbzip2 maintained?: Cut some flowers

← Older revision		Revision as of 20:34, 9 September 2022
Line 4:		Line 4:

	== Background ==		== Background ==
	[[File:Phap Nang Ngam Nai Wannakhadi (1964, p 60).jpg\|thumb\|Phap Nang Ngam Nai Wannakhadi (1964, p 60)]]		[[File:Phap Nang Ngam Nai Wannakhadi (1964, p 60).jpg\|thumb\|Phap Nang Ngam Nai Wannakhadi (1964, p 60). [This painting is not titled, "Picking the low-hanging fruit". -AW]]]

	One common way to analyze Wikipedia content is to ~~process~~ its database backup dumps<ref>https://dumps.wikimedia.org/backup-index.html</ref>, provided as [[W:bzip2\|bzip2]]-compressed XML. ~~The unpacked~~ files are too large to ~~manipulate locally~~ and ~~unwieldly~~ even when compressed, so ~~a common practice is to stream the content, decompress in memory, and analyze in a single pass. Perhaps this step extracts a smaller subset of the data and saves it to disk~~.		One common way to analyze Wikipedia content is to mung its database backup dumps<ref>https://dumps.wikimedia.org/backup-index.html</ref>, which are provided as [[W:bzip2\|bzip2]]-compressed XML. These files are too large to use unpacked and unwieldy even when compressed, so are best served streaming.

	Of course, one would normally search for a proven library written in a solid data-sciencey language such as Python, run pip install mwxml<ref>https://pythonhosted.org/mwxml/</ref> and go on with the day.

	That wouldn't be as interesting as trying to do exactly the same thing in an esoteric young language, in service of a pet project<ref>[https://gitlab.com/adamwight/mediawiki_client_ex mediawiki_client_ex]</ref> with zero adoption.

	~~<nowiki>"But imagine how much better a truly concurrent wiki dump processor could be!" —~~~</nowiki>~~

	~~<br clear="all">~~

	== Problem statement==		== Problem statement==
	Unfortunately, there was no BEAM (Elixir- and Erlang-compatible) library for reading bzip2 files, so we could either fetch and decompress the data externally through an separate process, or write a ~~new~~ library.		Unfortunately, there was no BEAM (Elixir- and Erlang-compatible) library for reading bzip2 files, so we could either fetch and decompress the data externally through an separate process, or write a glue library exposing libbzip2 functions in Elixir.

	How hard could it be to write a binding...		How hard could it be to write a little binding...

	==Is libbzip2 maintained?==		==Is libbzip2 maintained?==
	~~[[File:Potato_sprout,_January_23,_2006.jpg\|right\|267x267px]]~~

			Development of bzip2 has stopped at the last stable release, with v1.0.x in 2019.<ref>The project page for bzip2 v1.0 is https://sourceware.org/bzip2/.</ref> A new group has been working towards an initial fork<ref>https://gitlab.com/bzip2/bzip2/</ref> that they're calling version "1.1" but hopefully will avoid breaking changes to the programming interface.

			== The bzip2 file format has no specification ==
			I can't complain, there's a brilliant reverse-engineering effort<ref>https://github.com/dsnet/compress/blob/master/doc/bzip2-format.pdf</ref>.

	~~Um, sure... The official bzip2 pages~~<ref>https://sourceware.org/bzip2/</ref> might be hosted on a Red Hat site nobody has heard of since 2005, very likely written as static HTML, the project has no edits to its changelog<ref>https://sourceware.org/bzip2/CHANGES</ref> since 2019 (don't worry, the code has been changed<ref>https://sourceware.org/git/?p=~~bzip2.git</ref~~> ~~even if the changelog has not), but yes bzip2 1.0 is the current version dammit!~~		<br clear="all">

	What isn't mentioned on the official page, but is well documented in the Wikipedia article, is that ownership has peacefully moved to a new author (while slipping through another would-be maintainer's fingertips) who will kindly take us into the future with bzip2 1.1 and beyond<ref>https://gitlab.com/bzip2/bzip2/</ref>—once this is released and mainstream, please edit my comments here.		==High- or low-level integration?==

	~~Don't be confused just because I am.~~

	~~==The bzip2 file format has no specification==~~
	~~[[File:Zombie-156055.svg\|frameless\|82x82px]]That's a bigger problem and we will ignore until it comes back angry—which it does.<br clear="all">~~

	== High- or low-level integration? ==
	We already chose the "hard landing" by writing a new library in the first place, but there's still an opportunity for making life sort of easy again, by choosing one of two APIs provided by libbzip2, either the high-level<ref>https://sourceware.org/bzip2/manual/manual.html#hl-interface</ref> interface which opens a file and returns the decompressed contents, or the low-level<ref>https://sourceware.org/bzip2/manual/manual.html#low-level</ref> interface for maximum interactivity, which is called with tiny bites of data, shares its memory with the caller, exposes an internal state machine, and offers your humble developer many forms of additional stimulation.		We already chose the "hard landing" by writing a new library in the first place, but there's still an opportunity for making life sort of easy again, by choosing one of two APIs provided by libbzip2, either the high-level<ref>https://sourceware.org/bzip2/manual/manual.html#hl-interface</ref> interface which opens a file and returns the decompressed contents, or the low-level<ref>https://sourceware.org/bzip2/manual/manual.html#low-level</ref> interface for maximum interactivity, which is called with tiny bites of data, shares its memory with the caller, exposes an internal state machine, and offers your humble developer many forms of additional stimulation.

	There were a few considerations here, and in theory the high-level interface would have made a perfectly acceptable binding. However, this would be a single call to decompress the world, and control flow wouldn't return to Elixir for several hours. This pretty much defeats the purpose of writing a binding library for a concurrent virtual machine.		There were a few considerations here, and in theory the high-level interface would have made a perfectly acceptable binding. However, this would be a single call to decompress the world, and control flow wouldn't return to Elixir for several hours. This pretty much defeats the purpose of writing a binding library for a concurrent virtual machine.

	== Native implemented function (NIF) ==		==Native implemented function (NIF)==
			[[File:Potato_sprout,_January_23,_2006.jpg\|right\|267x267px]]
	This is where I began to feel like I was learning stuff.		This is where I began to feel like I was learning stuff.

	Elixir owes its existence to Erlang and the virtual machine that runs them both, BEAM<ref>https://blog.stenmans.org/theBeamBook/#_the_erlang_virtual_machine_beam</ref>. Extending the machine is done by writing a NIF library<ref>https://www.erlang.org/doc/man/erl_nif.html</ref> which consists of ~~compiled~~ C/C++ and a small Erlang stub ~~providing the~~ interface. ~~Actually~~, ~~the stub~~ can ~~be written in Elixir just as well~~, ~~which was a nice surprise~~.		Elixir owes its existence to Erlang and the virtual machine that runs them both, BEAM<ref>https://blog.stenmans.org/theBeamBook/#_the_erlang_virtual_machine_beam</ref>. Extending the machine is done by writing a NIF library<ref>https://www.erlang.org/doc/man/erl_nif.html</ref> which consists of a C/C++ to adapt the library, and a small Erlang or Elixir stub to define an interface.

			TODO ...

			== Parallel processing ==
			TODO: Stream vs block, what can write multiple streams, what are the challenges of detecting blocks...

	== References ==		==References==
	<references />		<references />

Adamw at 22:32, 7 September 2022

2022-09-07T22:32:31Z

← Older revision		Revision as of 22:32, 7 September 2022
Line 10:		Line 10:
	Of course, one would normally search for a proven library written in a solid data-sciencey language such as Python, run pip install mwxml<ref>https://pythonhosted.org/mwxml/</ref> and go on with the day.		Of course, one would normally search for a proven library written in a solid data-sciencey language such as Python, run pip install mwxml<ref>https://pythonhosted.org/mwxml/</ref> and go on with the day.

	That wouldn't be as interesting as trying to do exactly the same thing in an esoteric young language, in service of a pet project<ref>https://gitlab.com/adamwight/mediawiki_client_ex</ref> with zero adoption.		That wouldn't be as interesting as trying to do exactly the same thing in an esoteric young language, in service of a pet project<ref>[https://gitlab.com/adamwight/mediawiki_client_ex mediawiki_client_ex]</ref> with zero adoption.

	<nowiki>"But imagine how much better a truly concurrent wiki dump processor could be!" —~~~</nowiki>		<nowiki>"But imagine how much better a truly concurrent wiki dump processor could be!" —~~~</nowiki>

Adamw: C/e

2022-09-07T22:28:31Z

C/e

← Older revision		Revision as of 22:28, 7 September 2022
Line 17:		Line 17:

	== Problem statement==		== Problem statement==
	Unfortunately, there's no BEAM (Elixir- and Erlang-compatible) library for reading bzip2 files, so ~~the options are~~ either to fetch and ~~run~~ the data through an ~~external~~ process ~~and bidirectional pipes~~, or write a new library.		Unfortunately, there was no BEAM (Elixir- and Erlang-compatible) library for reading bzip2 files, so we could either fetch and decompress the data externally through an separate process, or write a new library.

	How hard could it be to write a binding...		How hard could it be to write a binding...

Adamw: C/e

2022-09-07T22:26:18Z

C/e

← Older revision		Revision as of 22:26, 7 September 2022
Line 6:		Line 6:
	[[File:Phap Nang Ngam Nai Wannakhadi (1964, p 60).jpg\|thumb\|Phap Nang Ngam Nai Wannakhadi (1964, p 60)]]		[[File:Phap Nang Ngam Nai Wannakhadi (1964, p 60).jpg\|thumb\|Phap Nang Ngam Nai Wannakhadi (1964, p 60)]]

	One common way to analyze Wikipedia content is to process its database backup dumps<ref>https://dumps.wikimedia.org/backup-index.html</ref>, ~~which are~~ provided ~~XML compressed using~~ [[W:bzip2\|bzip2]]. The unpacked files are too large to manipulate locally, and even ~~the~~ compressed ~~files can be unwieldly~~, so a common practice is to stream the content, decompress in memory, and analyze in a single pass. ~~The output might be to extract~~ a smaller subset of the data.		One common way to analyze Wikipedia content is to process its database backup dumps<ref>https://dumps.wikimedia.org/backup-index.html</ref>, provided as [[W:bzip2\|bzip2]]-compressed XML. The unpacked files are too large to manipulate locally and unwieldly even when compressed, so a common practice is to stream the content, decompress in memory, and analyze in a single pass. Perhaps this step extracts a smaller subset of the data and saves it to disk.

	Of course, ~~the normal approach~~ would ~~be to~~ search for a proven library in a solid data-sciencey language such as Python, pip install mwxml<ref>https://pythonhosted.org/mwxml/</ref> and go on with the day. That wouldn't be as interesting as trying to do exactly the same thing in an esoteric young language, in service of a pet project<ref>https://gitlab.com/adamwight/mediawiki_client_ex</ref> with zero adoption.		Of course, one would normally search for a proven library written in a solid data-sciencey language such as Python, run pip install mwxml<ref>https://pythonhosted.org/mwxml/</ref> and go on with the day.

			That wouldn't be as interesting as trying to do exactly the same thing in an esoteric young language, in service of a pet project<ref>https://gitlab.com/adamwight/mediawiki_client_ex</ref> with zero adoption.

	<nowiki>"But imagine how much better a truly concurrent wiki dump processor could be!" —~~~</nowiki>		<nowiki>"But imagine how much better a truly concurrent wiki dump processor could be!" —~~~</nowiki>

Adamw at 22:02, 7 September 2022

2022-09-07T22:02:04Z

← Older revision		Revision as of 22:02, 7 September 2022
Line 42:		Line 42:

	Elixir owes its existence to Erlang and the virtual machine that runs them both, BEAM<ref>https://blog.stenmans.org/theBeamBook/#_the_erlang_virtual_machine_beam</ref>. Extending the machine is done by writing a NIF library<ref>https://www.erlang.org/doc/man/erl_nif.html</ref> which consists of compiled C/C++ and a small Erlang stub providing the interface. Actually, the stub can be written in Elixir just as well, which was a nice surprise.		Elixir owes its existence to Erlang and the virtual machine that runs them both, BEAM<ref>https://blog.stenmans.org/theBeamBook/#_the_erlang_virtual_machine_beam</ref>. Extending the machine is done by writing a NIF library<ref>https://www.erlang.org/doc/man/erl_nif.html</ref> which consists of compiled C/C++ and a small Erlang stub providing the interface. Actually, the stub can be written in Elixir just as well, which was a nice surprise.

			== References ==
			<references />

Adamw: Convert direct links to references. Donno if this will turn out being the right choice.

2022-09-07T22:01:28Z

Convert direct links to references. Donno if this will turn out being the right choice.

← Older revision		Revision as of 22:01, 7 September 2022
Line 6:		Line 6:
	[[File:Phap Nang Ngam Nai Wannakhadi (1964, p 60).jpg\|thumb\|Phap Nang Ngam Nai Wannakhadi (1964, p 60)]]		[[File:Phap Nang Ngam Nai Wannakhadi (1964, p 60).jpg\|thumb\|Phap Nang Ngam Nai Wannakhadi (1964, p 60)]]

	One common way to analyze Wikipedia content is to process its [https://dumps.wikimedia.org/backup-index.html ~~database backup dumps]~~, which are provided XML compressed using [[W:bzip2\|bzip2]]. The unpacked files are too large to manipulate locally, and even the compressed files can be unwieldly, so a common practice is to stream the content, decompress in memory, and analyze in a single pass. The output might be to extract a smaller subset of the data.		One common way to analyze Wikipedia content is to process its database backup dumps<ref>https://dumps.wikimedia.org/backup-index.html</ref>, which are provided XML compressed using [[W:bzip2\|bzip2]]. The unpacked files are too large to manipulate locally, and even the compressed files can be unwieldly, so a common practice is to stream the content, decompress in memory, and analyze in a single pass. The output might be to extract a smaller subset of the data.

	Of course, the normal approach would be to search for a proven library in a solid data-sciencey language such as Python, pip install [https://pythonhosted.org/mwxml/ ~~mwxml]~~ and go on with the day. That wouldn't be as interesting as trying to do exactly the same thing in an esoteric young language, in service of a [https://gitlab.com/adamwight/mediawiki_client_ex ~~pet project]~~ with zero adoption.		Of course, the normal approach would be to search for a proven library in a solid data-sciencey language such as Python, pip install mwxml<ref>https://pythonhosted.org/mwxml/</ref> and go on with the day. That wouldn't be as interesting as trying to do exactly the same thing in an esoteric young language, in service of a pet project<ref>https://gitlab.com/adamwight/mediawiki_client_ex</ref> with zero adoption.

	<nowiki>"But imagine how much better a truly concurrent wiki dump processor could be!" —~~~</nowiki>		<nowiki>"But imagine how much better a truly concurrent wiki dump processor could be!" —~~~</nowiki>
Line 23:		Line 23:


	Um, sure... The [https://sourceware.org/bzip2/ official bzip2 pages] might be hosted on a Red Hat site nobody has heard of since 2005, very likely written as static HTML, the project has no edits to its [https://sourceware.org/bzip2/CHANGES changelog] since 2019 (don't worry, the code [https://sourceware.org/git/?p=bzip2.git has been changed] even if the changelog has not), but yes bzip2 1.0 is the current version dammit!

	What isn't mentioned on the official page, but is well documented in the Wikipedia article, is that ownership has peacefully moved to a new author (while slipping through another would-be maintainer's fingertips) who will kindly take us into the future with [https://gitlab.com/bzip2/bzip2/ ~~bzip2 1.1] and beyond—once~~ this is released and mainstream, please edit my comments here.		Um, sure... The official bzip2 pages<ref>https://sourceware.org/bzip2/</ref> might be hosted on a Red Hat site nobody has heard of since 2005, very likely written as static HTML, the project has no edits to its changelog<ref>https://sourceware.org/bzip2/CHANGES</ref> since 2019 (don't worry, the code has been changed<ref>https://sourceware.org/git/?p=bzip2.git</ref> even if the changelog has not), but yes bzip2 1.0 is the current version dammit!

			What isn't mentioned on the official page, but is well documented in the Wikipedia article, is that ownership has peacefully moved to a new author (while slipping through another would-be maintainer's fingertips) who will kindly take us into the future with bzip2 1.1 and beyond<ref>https://gitlab.com/bzip2/bzip2/</ref>—once this is released and mainstream, please edit my comments here.

	Don't be confused just because I am.		Don't be confused just because I am.
Line 33:		Line 34:

	== High- or low-level integration? ==		== High- or low-level integration? ==
	We already chose the "hard landing" by writing a new library in the first place, but there's still an opportunity for making life sort of easy again, by choosing one of two APIs provided by libbzip2, either the [https://sourceware.org/bzip2/manual/manual.html#hl-interface ~~high-level~~ interface] which opens a file and returns the decompressed contents, or the [https://sourceware.org/bzip2/manual/manual.html#low-level ~~low-level~~ interface] for maximum interactivity, which is called with tiny bites of data, shares its memory with the caller, exposes an internal state machine, and offers your humble developer many forms of additional stimulation.		We already chose the "hard landing" by writing a new library in the first place, but there's still an opportunity for making life sort of easy again, by choosing one of two APIs provided by libbzip2, either the high-level<ref>https://sourceware.org/bzip2/manual/manual.html#hl-interface</ref> interface which opens a file and returns the decompressed contents, or the low-level<ref>https://sourceware.org/bzip2/manual/manual.html#low-level</ref> interface for maximum interactivity, which is called with tiny bites of data, shares its memory with the caller, exposes an internal state machine, and offers your humble developer many forms of additional stimulation.

	There were a few considerations here, and in theory the high-level interface would have made a perfectly acceptable binding. However, this would be a single call to decompress the world, and control flow wouldn't return to Elixir for several hours. This pretty much defeats the purpose of writing a binding library for a concurrent virtual machine.		There were a few considerations here, and in theory the high-level interface would have made a perfectly acceptable binding. However, this would be a single call to decompress the world, and control flow wouldn't return to Elixir for several hours. This pretty much defeats the purpose of writing a binding library for a concurrent virtual machine.
Line 40:		Line 41:
	This is where I began to feel like I was learning stuff.		This is where I began to feel like I was learning stuff.

	Elixir owes its existence to Erlang and the virtual machine that runs them both, [https://blog.stenmans.org/theBeamBook/#_the_erlang_virtual_machine_beam ~~BEAM]~~. Extending the machine is done by writing a [https://www.erlang.org/doc/man/erl_nif.html ~~NIF library]~~ which consists of compiled C/C++ and a small Erlang stub providing the interface. Actually, the stub can be written in Elixir just as well, which was a nice surprise.		Elixir owes its existence to Erlang and the virtual machine that runs them both, BEAM<ref>https://blog.stenmans.org/theBeamBook/#_the_erlang_virtual_machine_beam</ref>. Extending the machine is done by writing a NIF library<ref>https://www.erlang.org/doc/man/erl_nif.html</ref> which consists of compiled C/C++ and a small Erlang stub providing the interface. Actually, the stub can be written in Elixir just as well, which was a nice surprise.