Elixir/Ports and external process wiring: Difference between revisions

<p>BEAM is a fairly unique ecosystem in which it's not considered deviant to reinvent a rounder wheel: it's common to port external dependencies into native Erlang—but the complexity of rsync and its dependence on a matching remote daemon makes it unlikely that it will be rewritten any time soon, which is why I've decided to wrap external command execution in a library.</p>}}

As a first step, we extract the percent_done column and log any unrecognized output:

         percent_done_text when is_binary(percent_done_text) <- Enum.at(terms, 1),

</syntaxhighlight>The <code>trim</code> is lifting more than its weight here: it lets us completely ignore spacing and newline trickery—and even a leading carriage return that we can see in the rsync source code,<ref>[https://github.com/RsyncProject/rsync/blob/797e17fc4a6f15e3b1756538a9f812b63942686f/progress.c#L129 rsync/progress.c] source code</ref>

</syntaxhighlight>Carriage return <code>\r</code> deserves a special mention: this "control" character is just a byte in the binary data coming over the pipe from rsync, but its normal role is playing a control function because of how the terminal emulator responds to it.  On a terminal the effect is to rewind the cursor so that the current line can be overwritten!

The Port API is convenient enough so far, but where Erlang/OTP really starts to shine is when we wrap each Port connection under a gen_server<ref>[https://www.erlang.org/doc/apps/stdlib/gen_server.html Erlang gen_server docs]</ref> module, giving us some properties for free: A dedicated thread coordinates with its rsync independent of anything else.  Input and output are asynchronous and buffered, but handled sequentially in a thread-safe way.  It holds internal state including the up-to-date completion percentage.  And the caller can either request updates manually, or it can listen for pushed statistics.

This gen_server should also be able to run under an [https://adoptingerlang.org/docs/development/supervision_trees/ OTP supervision tree] but this is where the dream falls apart, for the moment.  The Port can watch for rsync completion or failure and report this to its caller, but we fail at the second critical property of being able to shut down rsync if the calling code or our library module crashes.

The unpleasant real-world consequence of this limitation is that rsync transfers would continue to run in the background even after Elixir had completely shut down, because the BEAM has no way of stopping the process.

It might be possible to send a signal using unix "kill", but BEAM doesn't expose the child process ID and it doesn't include any built-in commands to send a signal.  Clearly we're expected to do this another way.  Another problem with "kill" is that we want the external process to stop no matter how badly the BEAM is damaged so we can't rely on stored data and on making a few last calls before crashing.

To eliminate variable and to understand whether the failure to stop was specific to rsync, I tried the same Port command but spawning a <code>sleep 60</code>, and I found that it behaves exactly the same way, hanging until the sleep ends naturally regardless of what happened in Elixir or whether its pipes are still open.  This happens to have been a lucky choice, as I learned later that "sleep" is also unusual but its behavior is much simpler to reason about.