Elixir/Ports and external process wiring: Difference between revisions

I was excited to learn how to interface with long-lived external processes—and this project offered more than I hoped for.

[[File:Monkey eating.jpg|alt=A Toque macaque (Macaca radiata) Monkey eating peanuts. Pictured in Bangalore, India|right|300x300px]]

Starting rsync should be as easy as calling out to a shell:<syntaxhighlight lang="elixir">

System.shell("rsync -a source target")

</syntaxhighlight>

=== Safe path handling ===

=== Asynchronous call and communication ===

Port.open(

   {:spawn_executable, rsync_path},

; <code>-v</code> : list each filename as it's transferred

; <code>--info=progress2</code> : report overall progress

; <code>--itemize-changes</code> : list the operations taken on each file

}}

bytes transferred |  transfer speed    |

         |        |        |          |

</syntaxhighlight>

<syntaxhighlight lang="elixir">

with terms when terms != [] <- String.split(line, ~r"\s", trim: true),

         {:unknown, line}

     end

<syntaxhighlight lang="c">

rprintf(FCLIENT, "\r%15s %3d%% %7.2f%s %s%s", ...);

{{Aside|text=

You'll have to use <control>-v <control>-m to type a literal carriage return, copy-and-paste won't work.

[[w:https://en.wikipedia.org/wiki/Newline#Issues_with_different_newline_formats|Disagreement about carriage return]] vs. line feed has caused eye-rolling since the dawn of personal computing.

}}

The unpleasant real-world consequence is that rsync transfers will continue to run in the background even after Elixir kills our gen_server or shuts down, because the BEAM has no way of stopping the external process.

== Bad assumption: pipe-like processes ==

A pipeline like <code>gzip</code> or <code>cat</code> it built to read from its input and write to its output.  We can roughly group the different styles of command-line application into "pipeline" programs which read and write, "interactive" programs which require user input, and "daemon" programs which are designed to run in the background.  Some programs support multiple modes depending on the arguments given at launch, or by detecting the terminal using <code>isatty</code><ref>[https://man.archlinux.org/man/isatty.3.en docs for <code>isatty</code>]</ref>.  The BEAM is currently optimized to interface with pipeline programs and it assumes that the external process will stop when its "standard input" is closed.

cat < /dev/null

A small shim can adapt a daemon-like program to behave more like a pipeline.  The shim is sensitive to stdin closing or SIGHUP, and when this is detected it converts this into a stronger signal like SIGTERM which it forwards to its own child.  This is the idea behind a suggested shell script<ref>[https://hexdocs.pm/elixir/1.19.0/Port.html#module-orphan-operating-system-processes Elixir Port docs showing a shim script]</ref> for Elixir, and the <code>erlexec</code><ref name=":0">[https://hexdocs.pm/erlexec/readme.html <code>erlexec</code> library]</ref> library.  The opposite adapter can be found in the [[w:nohup|nohup]] shell command and the grimsby<ref>[https://github.com/shortishly/grimsby <code>grimsby</code> library]</ref> library: these will keep standard in and/or standard out open for the child process even after the parent exits, so that a pipe-like program can behave more like a daemon.

== Reliable clean up ==

{{Project|status=in review|url=https://erlangforums.com/t/open-port-and-zombie-processes|source=https://github.com/erlang/otp/pull/9453}}

It's always a pleasure to ask questions in the BEAM communities, they deserve their reputation for being friendly and open.  The first big tip was to look at the third-party library <code>erlexec</code><ref name=":0" />, which demonstrates emerging best practices which could be backported into the language itself.  Everyone speaking on the problem generally agrees that the fragile clean up of external processes is a bug, and supports the idea that some flavor of "terminate" signal should be sent to spawned programs when the port is closed.

I would be lying to hide my disappointment that the required core changes are mostly in an auxiliary C program and not written in Erlang or even in the BEAM itself, but it was still fascinating to open such an elegant black box and find the technological equivalent of a steam engine inside.  All of the futuristic, high-level features we've come to know actually map closely to a few scraps of wizardry with ordinary pipes<ref>[https://man.archlinux.org/man/pipe.7.en Overview of unix pipes]</ref>, using libc's pipe<ref>[https://man.archlinux.org/man/pipe.2.en Docs for the <code>pipe</code> syscall]</ref>, read, write, and select<ref>[https://man.archlinux.org/man/select.2.en libc <code>select</code> docs]</ref>.

Which signal to use is still an open question:

; <code>TERM</code> : has a clear intention of "kill this thing" but still possible to trap at the target and handle in a customized way

There is a refreshing diversity of opinion, so it could be worthwhile to make the signal configurable for each port.

}}

== Future directions ==