Elixir/Ports and external process wiring

A deceivingly simple programming adventure veers unexpectedly into piping and signaling between unix processes.

Context: controlling "rsync"

Project link (beta):
https://hexdocs.pm/rsync/Rsync.html
Source code:
https://gitlab.com/adamwight/rsync_ex/

My exploration begins while writing a beta-quality library for Elixir to transfer files in the background and monitor progress using rsync.

I was excited to learn how to interface with long-lived external processes—and this project offered more than I hoped for.

Naïve shelling

Starting rsync should be as easy as calling out to a shell:

System.shell("rsync -a source target")

This has a few shortcomings, starting with how one would pass it dynamic paths. It's unsafe to use string interpolation ("#{source}" ): consider what could happen if the filenames include unescaped whitespace or special shell characters such as ";".

Safe path handling

We turn next to System.cmd, which takes a raw argv and can't be fooled special characters in the path arguments:

System.find_executable(rsync_path)
|> System.cmd([~w(-a), source, target])

For a short job this is perfect, but for longer transfers our program loses control and observability, waiting indefinitely for a monolithic command to return.

Asynchronous call and communication

To run a external process asynchronously we reach for Elixir's low-level Port.open, nothing but a one-line wrapper passing its parameters directly to ERTS open_port. This function is tremendously flexible, here we turn a few knobs:

Port.open(
  {:spawn_executable, rsync_path},
  [
    :binary,
    :exit_status,
    :hide,
    :use_stdio,
    :stderr_to_stdout,
    args:
      ~w(-a --info=progress2) ++
        rsync_args ++
        sources ++
        [args[:target]],
    env: env
  ]
)

Rsync outputs --info=progress2 lines like so:

       overall percent complete   time remaining
bytes transferred |  transfer speed    |
         |        |        |           |
      3,342,336  33%    3.14MB/s    0:00:02

The controlling Port captures these lines is sent to the library's handle_info callback as {:data, line}. After the transfer is finished we receive a conclusive {:exit_status, status_code} message.

As a first step, we extract the overall_percent_done column and flag any unrecognized output:

with terms when terms != [] <- String.split(line, ~r"\s", trim: true),
         percent_done_text when percent_done_text != nil <- Enum.at(terms, 1),
         {percent_done, "%"} <- Float.parse(percent_done_text) do
      percent_done
    else
      _ ->
        {:unknown, line}
    end

The trim is lifting more than its weight here: it lets us completely ignore spacing and newline trickery—and ignores the leading carriage return before each line, seen in the rsync source code:

rprintf(FCLIENT, "\r%15s %3d%% %7.2f%s %s%s", ...);

Carriage return \r deserves special mention: this is the first "control" character we come across and it looks the same as an ordinary byte in the binary data coming over the pipe from rsync, similar to newline \n. Its normal role is to control the terminal emulator, rewinding the cursor so that the current line can be overwritten! And like newline, carriage return can be ignored. Control signaling is exactly what goes haywire about this project, and the leaky category distinction between data and control seems to be a repeated theme in inter-process communication. The reality is not so much data vs. control, as it seems to be a sequence of layers like with networking.

OTP generic server

The Port API is convenient enough so far, but Erlang/OTP really starts to shine once we wrap each Port connection under a gen_server module, giving us several properties for free: A dedicated application thread coordinates with its rsync process independent of anything else. Input and output are asynchronous and buffered, but handled sequentially in a thread-safe way. The gen_server holds internal state including the up-to-date completion percentage. And the caller can request updates as needed, or it can listen for push messages with the parsed statistics.

This gen_server is also expected to run safely under an OTP supervision tree but this is where our dream falls apart for the moment. The Port already watches for rsync completion or failure and reports upwards to its caller, but we fail at the critical property of being able to propagate a termination downwards to shut down rsync if the calling code or our library module crashes.

Problem: runaway processes

The unpleasant real-world consequence is that rsync transfers will continue to run in the background even after Elixir kills our gen_server or shuts down, because the BEAM has no way of stopping the external process.

It's possible to find the operating system PID of the child process with Port.info(port, :os_pid) and send it a signal by shelling out to unix kill PID, but BEAM doesn't include built-in functions to send a signal to an OS process, and there is an ugly race condition between closing the port and sending this signal. We'll keep looking for another way to "link" the processes.

To debug what happens during port_close and to eliminate variables, I tried spawning sleep 60 instead of rsync and I found that it behaves in exactly the same way: hanging until sleep ends naturally regardless of what happened in Elixir or whether its pipes are still open. This happens to have been a lucky choice as I learned later: "sleep" is daemon-like so similar to rsync, but its behavior is much simpler to reason about.

Bad assumption: pipe-like processes

A pipeline like gzip or cat it built to read from its input and write to its output. We can roughly group the different styles of command-line application into "pipeline" programs which read and write, "interactive" programs which require user input, and "daemon" programs which are designed to run in the background. Some programs support multiple modes depending on the arguments given at launch, or by detecting the terminal using isatty. The BEAM is currently optimized to interface with pipeline programs and it assumes that the external process will stop when its "standard input" is closed.

A typical pipeline program will stop once it detects that input has ended, for example by calling read in a loop:

size_read = read (input_desc, buf, bufsize);
if (size_read < 0) { error... }
if (size_read == 0) { end of file... }

If the program does blocking I/O, then a zero-byte read indicates the end of file condition. A program which does asynchronous I/O with O_NONBLOCK might instead detect EOF by listening for the HUP hang-up signal which is can be arranged (TODO: document how this can be done with prctl, and on which platforms).

But here we'll focus on how processes can more generally affect each other through pipes. Surprising answer: without much effect! You can experiment with the /dev/null device which behaves like a closed pipe, for example compare these two commands:

cat < /dev/null

sleep 10 < /dev/null

cat exits immediately, but sleep does its thing as usual.

You could do the same experiment by opening a "cat" in the terminal and then type <control>-d to "send" an end-of-file. Interestingly, what happened here is that <control>-d is interpreted by bash which responds by closing its pipe connected to standard input of the child process. This is similar to how <control>-c is not sending a character but is interpreted by the terminal, trapped by the shell and forwarded as an interrupt signal to the child process, completely independently of the data pipe. My entry point to learning more is this stty webzine by Julia Evans. Dump information about your own terminal emulator: stty -a

Any special behavior at the other end of a pipe is the result of intentional programming decisions and "end of file" (EOF) is more a convention than a hard reality. A program with a chaotic disposition could even reopen stdin after it was closed and connect it to something else, to the great surprise of friends and neighbors.

Back to the problem at hand, "rsync" is in the category of "daemon-like" programs which will carry on even after standard input is closed. This makes sense enough, since rsync isn't interactive and any output is just a side effect of its main purpose.

Shimming can kill

A small shim can adapt a daemon-like program to behave more like a pipeline. The shim is sensitive to stdin closing or SIGHUP, and when this is detected it converts this into a stronger signal like SIGTERM which it forwards to its own child. This is the idea behind a suggested shell script for Elixir, and the erlexec library. The opposite adapter can be found in the nohup shell command and the grimsby library: these will keep standard in and/or standard out open for the child process even after the parent exits, so that a pipe-like program can behave more like a daemon.

I used the shim approach in my rsync library and it includes a small C program which wraps rsync and makes it sensitive to BEAM port_close. It's featherweight, leaving pipes unchanged as it passes control to rsync, here are the business parts:

// Set up a fail-safe to self-signal with HUP if the controlling process dies.
prctl(PR_SET_PDEATHSIG, SIGHUP);

void handle_signal(int signum) {
  if (signum == SIGHUP && child_pid > 0) {
    // Send the child TERM so that rsync can perform clean-up such as shutting down a remote server.
    kill(child_pid, SIGTERM);
  }
}

Reliable clean up

Project link (in review):
https://erlangforums.com/t/open-port-and-zombie-processes
Source code:
https://github.com/erlang/otp/pull/9453

It's always a pleasure to ask questions in the BEAM communities, they deserve their reputation for being friendly and open. The first big tip was to look at the third-party library erlexec, which demonstrates emerging best practices which could be backported into the language itself. Everyone speaking on the problem generally agrees that the fragile clean up of external processes is a bug, and supports the idea that some flavor of "terminate" signal should be sent to spawned programs when the port is closed.

I would be lying to hide my disappointment that the required core changes are mostly in an auxiliary C program and not written in Erlang or even in the BEAM itself, but it was still fascinating to open such an elegant black box and find the technological equivalent of a steam engine inside. All of the futuristic, high-level features we've come to know actually map closely to a few scraps of wizardry with ordinary pipes, using libc's pipe, read, write, and select.

Port drivers are fundamental to ERTS, and several levels of port wiring are involved in launching external processes: the spawn driver starts a forker driver which sends a control message to erl_child_setup to execute your external command. Each BEAM has a single erl_child_setup process to watch over all children. This architecture reflects the Supervisor paradigm and we can leverage it to produce some of the same properties: the subprocess can buffer reads and writes asynchronously and handle them sequentially; and if the BEAM crashes then erl_child_setup can detect the condition and do its own cleanup.

Letting a child process outlive its controlling process leaves the child in a state called "orphaned" in POSIX, and the standard recommends that when this happens the process should be adopted by the top-level system process "init" if it exists. This can be seen as undesirable because unix itself has a paradigm similar to OTP's Supervisors, in which each parent is responsible for its children. Without supervision, a process could potentially run forever or do naughty things. The system init process starts and tracks its own children, and can restart them in response to service commands. But init will know nothing about adopted, orphan processes or how to monitor and restart them.

The patch PR#9453 adapting port_close to SIGTERM is waiting for review and responses look generally positive so far.

TODO: consistency with unix process groups

... there is something fun here about how unix already has process tree behaviors which are close analogues to a BEAM supervisor tree.

Future directions

Discussion threads also included some notable grumbling about the Port API in general, it seems this part of ERTS is overdue for a larger redesign.

There's a good opportunity to unify the different platform implementations: Windows lacks the erl_child_setup layer entirely, for example.

Another idea to borrow from the erlexec library is to have an option to kill the entire process group of a child, which is shared by any descendants that haven't explicitly broken out of its original group. This would be useful for managing deep trees of external processes launched by a forked command.