Draft:Elixir/bzip2-ex: Difference between revisions
InstantCommons letting me down |
*shrug* adding with the visual editor works |
||
| Line 7: | Line 7: | ||
Of course, the normal approach would be to search for a proven library in a solid data-sciencey language such as Python, pip install [https://pythonhosted.org/mwxml/ mwxml] and go on with the day. But that wouldn't be as interesting as trying to do exactly the same thing in an esoteric young language, in service of a [https://gitlab.com/adamwight/mediawiki_client_ex pet project] with zero adoption. | Of course, the normal approach would be to search for a proven library in a solid data-sciencey language such as Python, pip install [https://pythonhosted.org/mwxml/ mwxml] and go on with the day. But that wouldn't be as interesting as trying to do exactly the same thing in an esoteric young language, in service of a [https://gitlab.com/adamwight/mediawiki_client_ex pet project] with zero adoption. | ||
[[File:Phap Nang Ngam Nai Wannakhadi (1964, p 60).jpg|thumb|Phap Nang Ngam Nai Wannakhadi (1964, p 60)]] | |||
[[File:Phap Nang Ngam Nai Wannakhadi (1964, p 60).jpg|thumb|Phap Nang Ngam Nai Wannakhadi (1964, p 60)]]But imagine how much better a truly concurrent processor could be! | But imagine how much better a truly concurrent processor could be! | ||
== Problem statement == | == Problem statement == | ||
Revision as of 20:55, 7 September 2022
A chronicle of my first Erlang/Elixir library binding (NIF).
Adam Wight, Sept 2022
Background
One common way to analyze Wikipedia content is to process its database backup dumps, which are provided XML compressed using bzip2. The unpacked files are too large to manipulate locally, and even the compressed files can be unwieldly, so a common practice is to stream the content, decompress in memory, and analyze in a single pass. The output might be to extract a smaller subset of the data.
Of course, the normal approach would be to search for a proven library in a solid data-sciencey language such as Python, pip install mwxml and go on with the day. But that wouldn't be as interesting as trying to do exactly the same thing in an esoteric young language, in service of a pet project with zero adoption.
.jpg)
But imagine how much better a truly concurrent processor could be!
Problem statement
Unfortunately, there's no BEAM (Elixir- and Erlang-compatible) library for reading bzip2 files, so the options would be to fetch and run the data through an external, bidirectional pipe, or write the bindings.
How hard could it be to write a binding...