ludd

Elixir/Ports and external process wiring: Difference between revisions

← Older editNewer edit →
Adamw (talk | contribs)
201 edits
VisualWikitext
Adamw (talk | contribs)
201 edits
No edit summary
Adamw (talk | contribs)
201 edits
memed.
Line 8: Line 8:
I was excited to learn how to interface with long-lived external processes—and this project offered more than I hoped for.
I was excited to learn how to interface with long-lived external processes—and this project offered more than I hoped for.


{{Aside|text=<p>[[w:rsync|Rsync]] is the standard utility for file transfers, locally or over a network.  It can resume incomplete transfers and synchronize directories efficiently, and after almost 30 years of usage it can be trusted to handle any edge case.</p>
{{Aside|text=<p>[[w:rsync|Rsync]] is the standard utility for file transfers, locally or over a network.  It can resume incomplete transfers and synchronize directories efficiently, and after almost 30 years of usage rsync can be trusted to handle any edge case.</p>
<p>BEAM is a fairly unique ecosystem in which it's not considered deviant to reinvent a rounder wheel: an external dependency like "cron" would often be ported into native Erlang—but the complexity of rsync and its dependence on a matching remote daemon makes it unlikely that it will be rewritten any time soon, which is why I've decided to wrap external command execution in a library.</p>}}
<p>BEAM<ref>The virtual machine shared by Erlang, Elixir, Gleam, Ash, and so on: [https://blog.stenmans.org/theBeamBook/ the BEAM Book]</ref> is a fairly unique ecosystem in which it's not considered deviant to reinvent a rounder wheel: an external dependency like "cron" will often be ported into native Erlang—but the complexity of rsync and its dependence on a matching remote daemon makes it unlikely that it will be rewritten any time soon, which is why I've decided to wrap external command execution in a library.</p>}}


[[File:Monkey eating.jpg|alt=A Toque macaque (Macaca radiata) Monkey eating peanuts. Pictured in Bangalore, India|right|300x300px]]
[[File:Monkey eating.jpg|alt=A Toque macaque (Macaca radiata) Monkey eating peanuts. Pictured in Bangalore, India|right|300x300px]]


=== Naive shelling ===
=== Naïve shelling ===


Starting rsync should be as easy as calling out to a shell:<syntaxhighlight lang="elixir">
Starting rsync should be as easy as calling out to a shell:<syntaxhighlight lang="elixir">
System.shell("rsync -a source target")
System.shell("rsync -a source target")
</syntaxhighlight>
</syntaxhighlight>
This has a few shortcomings, starting with how we pass the filenames.  It would be possible to pass a dynamic path using string interpolation like <code>#{source}</code> but this is risky: consider what happens if the filenames include whitespace or even special shell characters such as ";".
This has a few shortcomings, starting with how one would pass it dynamic paths.  It's unsafe to use string interpolation (<code>"#{source}"</code> ): consider what could happen if the filenames include unescaped whitespace or special shell characters such as ";".


=== Safe path handling ===
=== Safe path handling ===
Line 27: Line 27:


=== Asynchronous call and communication ===
=== Asynchronous call and communication ===
To run a external process asynchronously we reach for Elixir's low-level <code>Port.open</code>, nothing but a one-line wrapper<ref>See the [https://github.com/elixir-lang/elixir/blob/809b035dccf046b7b7b4422f42cfb6d075df71d2/lib/elixir/lib/port.ex#L232 port.ex source code]</ref> which passes its parameters directly to ERTS <code>open_port</code><ref>[https://www.erlang.org/doc/apps/erts/erlang.html#open_port/2 Erlang <code>open_port</code> docs]</ref>.  This function is tremendously flexible, here we turn a few knobs:<syntaxhighlight lang="elixir">
To run a external process asynchronously we reach for Elixir's low-level <code>Port.open</code>, nothing but a one-line wrapper<ref>See the [https://github.com/elixir-lang/elixir/blob/809b035dccf046b7b7b4422f42cfb6d075df71d2/lib/elixir/lib/port.ex#L232 port.ex source code]</ref> passing its parameters directly to ERTS <code>open_port</code><ref>[https://www.erlang.org/doc/apps/erts/erlang.html#open_port/2 Erlang <code>open_port</code> docs]</ref>.  This function is tremendously flexible, here we turn a few knobs:<syntaxhighlight lang="elixir">
Port.open(
Port.open(
   {:spawn_executable, rsync_path},
   {:spawn_executable, rsync_path},
Line 52: Line 52:


; <code>-v</code> : list each filename as it's transferred
; <code>-v</code> : list each filename as it's transferred
; <code>--progress</code> : report statistics per file


; <code>--info=progress2</code> : report overall progress
; <code>--info=progress2</code> : report overall progress
; <code>--progress</code> : report statistics per file


; <code>--itemize-changes</code> : list the operations taken on each file
; <code>--itemize-changes</code> : list the operations taken on each file


; <code>--out-format=FORMAT</code> : any format using parameters from rsyncd.conf's <code>log format</code><ref>https://man.freebsd.org/cgi/man.cgi?query=rsyncd.conf</ref>
; <code>--out-format=FORMAT</code> : any custom format string following rsyncd.conf's <code>log format</code><ref>https://man.freebsd.org/cgi/man.cgi?query=rsyncd.conf</ref>
}}
}}


We've chosen <code>--info=progress2</code> , so the meaning of the reported percentage is "overall percent complete".  Rsync outputs these progress lines in a fairly self-explanatory columnar format:<syntaxhighlight lang="text">
Rsync outputs <code>--info=progress2</code> lines like so:<syntaxhighlight lang="text">
          percent complete       time remaining
      overall percent complete   time remaining
bytes transferred |  transfer speed    |
bytes transferred |  transfer speed    |
         |        |        |          |
         |        |        |          |
Line 69: Line 69:
</syntaxhighlight>
</syntaxhighlight>


Our Port captures output and each line is sent to the library's <code>handle_info</code> callback as <code>{:data, line}</code>.  After the transfer is finished we receive a conclusive <code>{:exit_status, status_code}</code> message.
The controlling Port captures these lines is sent to the library's <code>handle_info</code> callback as <code>{:data, line}</code>.  After the transfer is finished we receive a conclusive <code>{:exit_status, status_code}</code> message.


As a first step, we extract the percent_done column and flag any unrecognized output:
As a first step, we extract the overall_percent_done column and flag any unrecognized output:
<syntaxhighlight lang="elixir">
<syntaxhighlight lang="elixir">
with terms when terms != [] <- String.split(line, ~r"\s", trim: true),
with terms when terms != [] <- String.split(line, ~r"\s", trim: true),
Line 81: Line 81:
         {:unknown, line}
         {:unknown, line}
     end
     end
</syntaxhighlight>The <code>trim</code> is lifting more than its weight here: it lets us completely ignore spacing and newline trickery—even skipping the leading carriage return that can be seen in the rsync source code,<ref>[https://github.com/RsyncProject/rsync/blob/797e17fc4a6f15e3b1756538a9f812b63942686f/progress.c#L129 rsync/progress.c] source code</ref>
</syntaxhighlight>The <code>trim</code> is lifting more than its weight here: it lets us completely ignore spacing and newline trickery—and ignores the leading carriage return before each line, seen in the rsync source code:<ref>[https://github.com/RsyncProject/rsync/blob/797e17fc4a6f15e3b1756538a9f812b63942686f/progress.c#L129 rsync/progress.c] source code</ref>
<syntaxhighlight lang="c">
<syntaxhighlight lang="c">
rprintf(FCLIENT, "\r%15s %3d%% %7.2f%s %s%s", ...);
rprintf(FCLIENT, "\r%15s %3d%% %7.2f%s %s%s", ...);
</syntaxhighlight>Carriage return <code>\r</code> deserves special mention: this "control" character is just a byte in the binary data coming over the pipe from rsync, but its normal role is to control the terminal emulator, rewinding the cursor so that the current line can be overwritten!
</syntaxhighlight>Carriage return <code>\r</code> deserves special mention: this is the first "control" character we come across and it looks the same as an ordinary byte in the binary data coming over the pipe from rsync, similar to newline <code>\n</code>.  Its normal role is to control the terminal emulator, rewinding the cursor so that the current line can be overwritten! And like newline, carriage return can be ignored.  Control signaling is exactly what goes haywire about this project, and the leaky category distinction between data and control seems to be a repeated theme in inter-process communication.  The reality is not so much data vs. control, as it seems to be a sequence of layers like with [[w:OSI model|networking]].
 
A repeated theme in inter-process communication is that data and control are leaky categories.  We come to the more formal control side channels later.


{{Aside|text=
{{Aside|text=
Line 97: Line 95:
You'll have to use <control>-v <control>-m to type a literal carriage return, copy-and-paste won't work.
You'll have to use <control>-v <control>-m to type a literal carriage return, copy-and-paste won't work.


The character seems to be named after pushing the physical paper carriage of a typewriter back to the beginning of the line without feeding the roller.
The character is named after the pushing of a physical typewriter carriage to return to the beginning of the current line without feeding the roller to a new line.


[[File:Nilgais fighting, Lakeshwari, Gwalior district, India.jpg|left|200x200px]]
[[File:Baboons Playing in Chobe National Park-crlf.jpg|left|300x300px|Three young baboons playing on a rock ledge.  Two are on the ridge and one below, grabbing the tail of another.  A meme font shows "\r", "\n", and "\r\n" personified as each baboon.]]
[[w:https://en.wikipedia.org/wiki/Newline#Issues_with_different_newline_formats|Disagreement about carriage return]] vs. line feed has caused eye-rolling since the dawn of personal computing.
[[w:https://en.wikipedia.org/wiki/Newline#Issues_with_different_newline_formats|Disagreement about carriage return]] vs. line feed has caused eye-rolling since the dawn of personal computing.
}}
}}
Line 145: Line 143:
{{Project|status=in review|url=https://erlangforums.com/t/open-port-and-zombie-processes|source=https://github.com/erlang/otp/pull/9453}}
{{Project|status=in review|url=https://erlangforums.com/t/open-port-and-zombie-processes|source=https://github.com/erlang/otp/pull/9453}}
It's always a pleasure to ask questions in the BEAM communities, they deserve their reputation for being friendly and open.  The first big tip was to look at the third-party library <code>erlexec</code><ref name=":0" />, which demonstrates emerging best practices which could be backported into the language itself.  Everyone speaking on the problem generally agrees that the fragile clean up of external processes is a bug, and supports the idea that some flavor of "terminate" signal should be sent to spawned programs when the port is closed.
It's always a pleasure to ask questions in the BEAM communities, they deserve their reputation for being friendly and open.  The first big tip was to look at the third-party library <code>erlexec</code><ref name=":0" />, which demonstrates emerging best practices which could be backported into the language itself.  Everyone speaking on the problem generally agrees that the fragile clean up of external processes is a bug, and supports the idea that some flavor of "terminate" signal should be sent to spawned programs when the port is closed.
 
[[File:Itinerant glassworker exhibition with spinning wheel and steam engine.jpg|thumb]]
I would be lying to hide my disappointment that the required core changes are mostly in an auxiliary C program and not written in Erlang or even in the BEAM itself, but it was still fascinating to open such an elegant black box and find the technological equivalent of a steam engine inside.  All of the futuristic, high-level features we've come to know actually map closely to a few scraps of wizardry with ordinary pipes<ref>[https://man.archlinux.org/man/pipe.7.en Overview of unix pipes]</ref>, using libc's pipe<ref>[https://man.archlinux.org/man/pipe.2.en Docs for the <code>pipe</code> syscall]</ref>, read, write, and select<ref>[https://man.archlinux.org/man/select.2.en libc <code>select</code> docs]</ref>.
I would be lying to hide my disappointment that the required core changes are mostly in an auxiliary C program and not written in Erlang or even in the BEAM itself, but it was still fascinating to open such an elegant black box and find the technological equivalent of a steam engine inside.  All of the futuristic, high-level features we've come to know actually map closely to a few scraps of wizardry with ordinary pipes<ref>[https://man.archlinux.org/man/pipe.7.en Overview of unix pipes]</ref>, using libc's pipe<ref>[https://man.archlinux.org/man/pipe.2.en Docs for the <code>pipe</code> syscall]</ref>, read, write, and select<ref>[https://man.archlinux.org/man/select.2.en libc <code>select</code> docs]</ref>.


Retrieved from "https://mw.ludd.net/wiki/Elixir/Ports_and_external_process_wiring"