ludd

Elixir/Ports and external process wiring: Difference between revisions

← Older editNewer edit →
Adamw (talk | contribs)
201 edits
VisualWikitext
Adamw (talk | contribs)
201 edits
clarify
Adamw (talk | contribs)
201 edits
c/e to the end
Line 6: Line 6:
My exploration begins while writing a beta-quality rsync library for Elixir which transfers files in the background and can monitor progress.  I hoped to learn better how to interface with long-lived external processes—and I got more than I wished for.
My exploration begins while writing a beta-quality rsync library for Elixir which transfers files in the background and can monitor progress.  I hoped to learn better how to interface with long-lived external processes—and I got more than I wished for.


[[File:Monkey eating.jpg|alt=A Toque macaque (Macaca radiata) Monkey eating peanuts. Pictured in Bangalore, India|right|400x400px]]
[[File:Monkey eating.jpg|alt=A Toque macaque (Macaca radiata) Monkey eating peanuts. Pictured in Bangalore, India|right|300x300px]]


Starting rsync should be as easy as calling out to a shell:<syntaxhighlight lang="elixir">
Starting rsync should be as easy as calling out to a shell:<syntaxhighlight lang="elixir">
Line 67: Line 67:
<syntaxhighlight lang="c">
<syntaxhighlight lang="c">
rprintf(FCLIENT, "\r%15s %3d%% %7.2f%s %s%s", ...);
rprintf(FCLIENT, "\r%15s %3d%% %7.2f%s %s%s", ...);
</syntaxhighlight>
</syntaxhighlight>The carriage return <code>\r</code> deserves a special mention: this "control" character is just a byte in the binary data coming over the pipe from rsync, but it plays a control function because of how the tty interprets it.  On the terminal the effect is to overwrite the current line!


A repeated theme is that data and control are leaky categories.  We come to the more formal control side channels later.
{{Aside|text=
{{Aside|text=
On the terminal, rsync progress lines are updated in place by emitting a [[w:Carriage return|carriage return]] control character <code>0x0d</code> or <code>\r</code> as you see above.  The character seems to be named after pushing the physical paper carriage of a typewriter backwards without feeding a new line. On the terminal this overwrites the current line!
[[File:Chinese typewriter 03.jpg|right|200x200px]]
 
On the terminal, rsync progress lines are updated in place by emitting a [[w:Carriage return|carriage return]] control character, <code>\r</code>, <code>0x0d</code> sometimes rendered as <code>^M</code>.  The character seems to be named after pushing the physical paper carriage of a typewriter back to the beginning of the line without feeding the roller.
 
[[w:https://en.wikipedia.org/wiki/Newline#Issues_with_different_newline_formats|Disagreement about carriage return]] vs. newline has caused eye-rolling since the dawn of personal computing.


[[w:https://en.wikipedia.org/wiki/Newline#Issues_with_different_newline_formats|Disagreements about carriage return]] vs. newline have caused eye-rolling since the dawn of personal computing.
[[File:Nilgais fighting, Lakeshwari, Gwalior district, India.jpg|left|200x200px]]
}}
}}


One more comment about this carriage return: the "control" character is just a byte in the binary data coming over the pipe from rsync, but it plays a control function because of how the tty interprets it.  Still, a repeated theme is that data and control are leaky categories.  We come to the more formal control side channels later.  
== OTP generic server ==
This is where Erlang/OTP really starts to shine: our rsync library wraps the Port calls under a gen_server<ref>https://www.erlang.org/doc/apps/stdlib/gen_server.html</ref> module and this gives us some special properties for free: a dedicated thread which coordinates with rsync independently from anything else, receiving and sending asynchronous messages.  It has an internal state including the latest percent done and this can be probed by calling code, or it can be set up to push updates to a listener.


This is where Erlang/OTP really starts to shine: by opening the port inside of a dedicated gen_server<ref>https://www.erlang.org/doc/apps/stdlib/gen_server.html</ref> we have a separate thread communicating with rsync, which receives an asynchronous message like <code>{:data, text_line}</code> for each progress line.  It's easy to parse the line, update some internal state and optionally send a progress summary to the code calling the library.
A gen_server should also be able to run under a [https://adoptingerlang.org/docs/development/supervision_trees/ OTP supervision tree] as well but our module has a major flaw: it can correctly detect and report when rsync crashes or completes, but if our module is stopped by its supervisor it cannot stop its external child process in turn.


== Problem: runaway processes ==
== Problem: runaway processes ==
This would have been the end of the story, but I'm a very flat-footed and iterative developer and as I was calling my rsync library from my application under development, I would often kill the program abruptly by crashing or by typing <control>-C in the terminal. Dozens of times.  What I found is that the rsync transfers would continue to run in the background even after Elixir had completely shut down.
[[File:CargoNet Di 12 Euro 4000 Lønsdal - Bolna.jpg|thumb]]
 
What this means is that rsync transfers would continue to run in the background even after Elixir had completely shut down, because the BEAM had no way of stopping the process.
That would have to change—leaving overlapping file transfers running unmonitored is exactly what I wanted to avoid by having Elixir control the process in the first place.  Once the BEAM stops there was no way to clearly identify and kill the sketchy rsyncing.


In fact, killing the lower-level threads when a higher-level supervising process dies is central to the BEAM concept of supervisors<ref>https://www.erlang.org/doc/system/sup_princ.html</ref> which has earned the virtual machine its reputation for being legendarily robust.  Why would some external processes stop and others not?  There seemed to be no way to send a signal or close the port to stop the process, either.
To check whether this was something specific to rsync, I tried the same thing with <code>sleep 60</code> and I found that it behaves exactly the same way, hanging until the sleep ends naturally regardless of what happened in Elixir or whether its pipes are still open.


== Bad assumption: pipe-like processes ==
== Bad assumption: pipe-like processes ==
Line 93: Line 98:
</syntaxhighlight>The manual for read<ref>https://man.archlinux.org/man/read.2</ref> explains that reading 0 bytes indicates the end of file, and a negative number indicates an error such as the input file descriptor already being closed.  If you think this sounds weird, I would agree: how do we tell the difference between a stream which is stalled and one which has ended?  Does the calling process yield control until input arrives?  How do we know if more than bufsize bytes are available?  If that word salad excites you, read more about <code>O_NONBLOCK</code><ref>https://man.archlinux.org/man/open.2.en#O_NONBLOCK</ref> and unix pipes<ref>https://man.archlinux.org/man/pipe.7.en</ref>.
</syntaxhighlight>The manual for read<ref>https://man.archlinux.org/man/read.2</ref> explains that reading 0 bytes indicates the end of file, and a negative number indicates an error such as the input file descriptor already being closed.  If you think this sounds weird, I would agree: how do we tell the difference between a stream which is stalled and one which has ended?  Does the calling process yield control until input arrives?  How do we know if more than bufsize bytes are available?  If that word salad excites you, read more about <code>O_NONBLOCK</code><ref>https://man.archlinux.org/man/open.2.en#O_NONBLOCK</ref> and unix pipes<ref>https://man.archlinux.org/man/pipe.7.en</ref>.


But here we'll focus on how processes affect each other through pipes.  Surprising answer: not very much!  Try opening a "cat" in the terminal and then type <control>-d to "send" an end-of-file.  Oh no, you killed it!  You didn't actually send anything, instead the <control>-d is interpreted by bash and it responds by closing the pipe to the child process.  This is similar to how <control>-c is not sending a character but is interpreted by the terminal, trapped by the shell and forwarded as an interrupt signal to the child process, completely independently of the data pipe.  My entry point to learning more is this stty webzine<ref>https://wizardzines.com/comics/stty/</ref> by Julia Evans.  Go ahead, try it: <code>stty -a</code>
But here we'll focus on how processes affect each other through pipes.  Surprising answer: they don't affect very much!  Try opening a "cat" in the terminal and then type <control>-d to "send" an end-of-file.  Oh no, you killed it!  You didn't actually send anything, though—the <control>-d is interpreted by bash and it responds by closing its pipe connected to "[[w:Standard streams|standard input]]" of the child process.  This is similar to how <control>-c is not sending a character but is interpreted by the terminal, trapped by the shell and forwarded as an interrupt signal to the child process, completely independently of the data pipe.  My entry point to learning more is this stty webzine<ref>https://wizardzines.com/comics/stty/</ref> by Julia Evans.  Go ahead and try this command, what could go wrong: <code>stty -a</code>


Any special behavior at the other end of a pipe is the result of intentional programming decisions and "end of file" (EOF) is more a convention than a real thing.  Now try opening "watch ls" or "sleep 60" and try <control>-d all you want—no effect.  You did close its stdin but nobody cares because it wasn't listening anway.
Any special behavior at the other end of a pipe is the result of intentional programming decisions and "end of file" (EOF) is more a convention than a hard reality.  You could even reopen stdin from the application, to the great surprise of your friends and neighbors.  For example, try opening "watch ls" or "sleep 60" and try <control>-d all you want—no effect.  You did close its stdin but nobody cared, it wasn't listening to you anyway.


Back to the problem at hand, as it turns out "rsync" is in this latter category of programs which sees itself as a daemon which should continue even when input is closed.  This makes sense enough, since rsync expects no user input and its output is just a side-effect of its main purpose.
Back to the problem at hand, "rsync" is in this latter category of "daemon-like" programs which will carry on even after standard input is closed.  This makes sense enough, since rsync isn't interactive and any output is just a side effect of its main purpose.


BEAM assumes the connected process behaves like this, so nothing needs to be done to clean up a dangling external process because it will end itself as soon as the Port is closed or the BEAM exits.  If the external process is known to not behave this way, the recommendation is to wrap it in a shell script which converts a closed stdin into a kill signal.<ref>https://hexdocs.pm/elixir/main/Port.html#module-orphan-operating-system-processes</ref>
== Shimming can kill ==
It's possible to write a small adapter which is sensitive to stdin closing, then converts this into a stronger signal like SIGTERM which it forwards to its own child.  This is the idea behind a suggested shell script<ref>https://hexdocs.pm/elixir/1.19.0/Port.html#module-orphan-operating-system-processes</ref> for Elixir and the erlexec<ref>[https://hexdocs.pm/erlexec/readme.html https://hexdocs.pm/erlexec/]</ref> library.  The opposite adapter is also found in the [[w:nohup|nohup]] shell command and the grimsby<ref>https://github.com/shortishly/grimsby</ref> library: these will keep standard in or out open for the child process even after the parent exits.


== BEAM internal and external processes ==
I took this approach with my rsync library and included a small C program<ref>https://gitlab.com/adamwight/rsync_ex/-/blob/main/src/main.c?ref_type=heads</ref> which wraps rsync and makes it sensitive to the BEAM port_close.  It's featherweight, leaving pipes unchanged as it passes control to rsync—its only real effect is to convert SIGHUP to SIGKILL (see the sidebar discussion of different signals below).
[[W:BEAM (Erlang virtual machine)|BEAM]] applications are built out of supervision trees and excel at managing huge numbers of parallel actor processes, all scheduled internally.  Although the communities' mostly share a philosophy of running as much as possible inside of the VM because it builds on this strength, and simplifies away much interface glue and context switching, on many occasions it will still start an external OS process.  There are some straightforward ways to simply run a command line, which might be familiar to programmers coming from another language: <code>[https://www.erlang.org/doc/apps/kernel/os.html#cmd/2 os:cmd]</code> takes a string and runs the thing.  At a lower level, external programs are managed through a [https://www.erlang.org/doc/system/ports.html Port] which is a flexible abstraction allowing a backend driver to communicate data in and out, and to send some control signals such as reporting an external process's exit and exit status.


When it comes to internal processes, BEAM is among the most mature and robust, achieved by good isolation and by its hierarchical [https://www.erlang.org/doc/system/sup_princ supervisors] liberally pruning entire subprocess trees at the first sign of going out of specification.  But for external processes, results are mixed.  Some programs are twitchy and crash easily, for example <code>cat</code>, but others like the BEAM itself or a long-running server are built to survive any ordinary I/O glitch or accidental mashing of the keyboard.  Furthermore, this will usually be a fundamental assumption of that program and there will be no configuration to make the program behave differently depending on stimulus.
== Reliable clean up ==
{{Project|status=in review|url=https://erlangforums.com/t/open-port-and-zombie-processes|source=https://github.com/erlang/otp/pull/9453}}
It's always a pleasure to ask questions in the BEAM communities, they have earned their reputation for being friendly and open.  The first big tip was to look at the third-party library [https://hexdocs.pm/erlexec/ erlexec], which demonstrates best practices that can be backported into the language itself.  Everyone speaking on the problem has generally agreed that the fragile clean up of external processes is a bug, and supported the idea that some flavor of "terminate" signal should be sent to spawned programs.


== Reliable clean up ==
I would be lying to hide my disappointment that the required core changes are mostly to a C program and not actually in Erlang, but it was fascinating to open such an elegant black box and find the technological equivalent of a steam engine inside.  All of the futuristic, high-level features we've come to know actually map closely to a few scraps of wizardry with ordinary pipes, using stdlib read, write, and select<ref>https://man.archlinux.org/man/select.2.en</ref>.
What I discovered is that the BEAM external process library assumes that its spawned processes will respond to standard input and output shutting down or so called end of file, for example what happens when <control>-d is typed into the shell. This works very well for a subprocess like <code>bash</code> but has no effect on a program like <code>sleep</code> or <code>rsync</code>.
 
Port drivers<ref>https://www.erlang.org/doc/system/ports.html</ref> are fundamental to ERTS and external processes are launched through several levels of wiring: the spawn driver starts a forker driver which sends a control message to <code>erl_child_setup</code> to execute your external command.  Each BEAM has a single erl_child_setup process to watch over all children.
 
Letting a child process outlive the one that spawned leaves it in a state called an "orphaned process" in POSIX, and the standard recommends that when this happens the process should be adopted by the top-level system process "init" if it exists.  This can be seen as undesirable because unix itself has a paradigm similar to OTP's Supervisors, in which each parent is responsible for its children.  Without supervision, a process could potentially run forever or do naughty things.  The system <code>init</code> process starts and tracks its own children, and can restart them in response to service commands. But init will know nothing about adopted, orphan processes or how to monitor and restart them.
 
The patch [https://github.com/erlang/otp/pull/9453 PR#9453] adapting port_close to SIGTERM is waiting for review and responses look generally positive so far.
 
{{Aside|text='''Which signal?'''


The hole created by this mismatch is interestingly solved by something shaped like the BEAM's supervisor itself.  I would expect the VM to spawn many processes as necessary, but I wouldn't expect the child process to outlive the VM, just because it happens to be insensitive to end of file.  Instead, I was hoping that the VM would try harder to kill these processes as the Port is closed, or if the VM halts.
Which signal to use is still an open question:


In fact, letting a child process outlive the one that spawned it is unusual enough that the condition is called an "orphan process".  The POSIX standard recommends that when this happens the process should be adopted by the top-level system process "init" if it exists, but this is a "should have" and not a must.  The reason it can be undesirable to allow this to happen at all is that the orphan process becomes entirely responsible for itself, potentially running forever without any more intervention according to the purpose of the process.  Even the system init process tracks its children, and can restart them in response to service commands.  Init will know nothing about its adopted, orphan processes.
; <code>HUP</code> : the softest "Goodbye!" that a program is free to interpret as it wishes


When I ran into this issue, I found the suggested workaround of writing a [https://hexdocs.pm/elixir/1.18.3/Port.html#module-zombie-operating-system-processes wrapper script] to track its child (the program originally intended to run), listen for the end of file from BEAM, and kill the external program.  How much simpler it would be if this workaround were already built into the Erlang Port module!
; <code>TERM</code> : has a clear intention of "kill this thing" but still possible to trap at the target and handle in a customized way


It's always a pleasure to ask questions in the BEAM communities, they have earned a reputation as being friendly and open.  The first big tip was to look at the third-party library [https://hexdocs.pm/erlexec/ erlexec], which demonstrates some best practices that might be backported into the language itself.  Everyone speaking on the problem has generally agreed that the fragile clean up of external processes is a bug, and supported the idea that one of the "terminate" signals should be sent to spawned programs.
; <code>KILL</code> : bursting with destructive potential, this signal cannot be stopped and you may not clean up


Which signal to use is still an open issue, there's a softer version <code>HUP</code> which says "Goodbye!" and the program is free to interpret as it will, the mid-level <code>TERM</code> that I prefer because it makes the intention explicit but can still be blocked or handled gracefully if needed, and <code>KILL</code> which is bursting with destructive potential.  The world of unix signals is a wild and scary place, on which there's a refreshing diversity of opinion around the Internet.
There is a refreshing diversity of opinion, so it could be worthwhile to make the signal configurable for each port.
}}


== Inside the BEAM ==
Discussion threads also included some notable grumbling about the Port API in general, it seems this part of ERTS is overdue for a larger redesign.  There's a good opportunity to unify the different platform implementations: Windows lacks the erl_child_setup layer entirely.
Despite its retro-futuristic appearance of being one of the most time-tested yet forward-facing programming environments, I was brought back to Earth by digging around inside the VM to find that it's just a C program like any other.  There's nothing holy about the BEAM emulator, there are some good and some great ideas about functional languages and they're buried in a mass of ancient procedural ifdefs, with unnerving memory management and typedefs wrapping the size of an integer on various platforms, just like you might find in other relics from the dark ages of computing, next to the Firefox or linux kernel source code.


Tantalizingly, message-passing is at the core of the VM, but is not a first-class concept when reaching out to external processes.  There's some fancy footwork with [[W:Anonymous pipe|pipes]] and [[W:Dup (system call)|dup]], but communication is done with enums, unions, and bit-rattling stdlib.  I love it, but... it might something to look at on another rainy day.
== References ==
Retrieved from "https://mw.ludd.net/wiki/Elixir/Ports_and_external_process_wiring"