Elixir/Ports and external process wiring - Revision history

Adamw: /* Asynchronous call and communication */ link to doc heading

2025-10-24T11:52:38Z

Asynchronous call and communication: link to doc heading

← Older revision		Revision as of 11:52, 24 October 2025
Line 59:		Line 59:
	; <code>--itemize-changes</code> : list the operations taken on each file		; <code>--itemize-changes</code> : list the operations taken on each file

	; <code>--out-format=FORMAT</code> : any custom format string following rsyncd.conf's <code>log format</code><ref>https://man.~~freebsd~~.org/~~cgi~~/~~man~~.~~cgi?query=~~rsyncd.conf</ref>		; <code>--out-format=FORMAT</code> : any custom format string following rsyncd.conf's <code>log format</code><ref>[https://man.archlinux.org/man/rsyncd.conf.5#log~2 rsyncd.conf log format] docs</ref>
	}}		}}

Adamw: /* Reliable clean up */ fix link syntax

2025-10-24T11:50:00Z

Reliable clean up: fix link syntax

← Older revision		Revision as of 11:50, 24 October 2025
Line 168:		Line 168:
	Which signal to use is still an open question:		Which signal to use is still an open question:

	; <code>HUP</code> : sent to a process when its standard input stream is closed<ref>[https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap11.html#tag_11_01_10 POSIX standard "General Terminal Interface: Modem Disconnect"</ref>		; <code>HUP</code> : sent to a process when its standard input stream is closed<ref>[https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap11.html#tag_11_01_10 POSIX standard "General Terminal Interface: Modem Disconnect"]</ref>

	; <code>TERM</code> : has a clear intention of "kill this thing" but still possible to trap at the target and handle in a customized way		; <code>TERM</code> : has a clear intention of "kill this thing" but still possible to trap at the target and handle in a customized way

Adamw: /* Future directions */ teaser about process groups

2025-10-23T15:22:58Z

Future directions: teaser about process groups

← Older revision		Revision as of 15:22, 23 October 2025
Line 176:		Line 176:
	There is a refreshing diversity of opinion, so it could be worthwhile to make the signal configurable for each port.		There is a refreshing diversity of opinion, so it could be worthwhile to make the signal configurable for each port.
	}}		}}

			== TODO: consistency with unix process groups ==

			... there is something fun here about how unix already has process tree behaviors which are close analogues to a BEAM supervisor tree.

	== Future directions ==		== Future directions ==

Adamw: /* Bad assumption: pipe-like processes */ correction about when HUP is sent

2025-10-23T15:21:16Z

Bad assumption: pipe-like processes: correction about when HUP is sent

← Older revision		Revision as of 15:21, 23 October 2025
Line 123:		Line 123:
	</syntaxhighlight>		</syntaxhighlight>

	If the program does blocking I/O, then a zero-byte <code>read</code> indicates the end of file condition. A program which does asynchronous I/O with <code>O_NONBLOCK</code><ref>[https://man.archlinux.org/man/open.2.en#O_NONBLOCK O_NONBLOCK docs]</ref> might instead detect EOF by listening for the <code>HUP</code> hang-up signal which is ~~normally sent when input is closed~~.		If the program does blocking I/O, then a zero-byte <code>read</code> indicates the end of file condition. A program which does asynchronous I/O with <code>O_NONBLOCK</code><ref>[https://man.archlinux.org/man/open.2.en#O_NONBLOCK O_NONBLOCK docs]</ref> might instead detect EOF by listening for the <code>HUP</code> hang-up signal which is can be arranged (TODO: document how this can be done with <code>prctl</code>, and on which platforms).

	But here we'll focus on how processes can more generally affect each other through pipes. Surprising answer: without much effect! You can experiment with the <code>/dev/null</code> device which behaves like a closed pipe, for example compare these two commands:		But here we'll focus on how processes can more generally affect each other through pipes. Surprising answer: without much effect! You can experiment with the <code>/dev/null</code> device which behaves like a closed pipe, for example compare these two commands:

Adamw: /* Problem: runaway processes */ Correct some bad information about port os_pid. Special thank you to akash-akya's post https://elixirforum.com/t/any-interest-in-a-library-that-wraps-rsync/69297/10

2025-10-23T15:20:00Z

Problem: runaway processes: Correct some bad information about port os_pid. Special thank you to akash-akya's post https://elixirforum.com/t/any-interest-in-a-library-that-wraps-rsync/69297/10

← Older revision		Revision as of 15:20, 23 October 2025
Line 110:		Line 110:
	The unpleasant real-world consequence is that rsync transfers will continue to run in the background even after Elixir kills our gen_server or shuts down, because the BEAM has no way of stopping the external process.		The unpleasant real-world consequence is that rsync transfers will continue to run in the background even after Elixir kills our gen_server or shuts down, because the BEAM has no way of stopping the external process.

	It's possible to send a signal by shelling out to unix <code>kill PID</code>, but BEAM ~~doesn't expose the child process ID and~~ doesn't include ~~any~~ built-in functions to send a signal to an OS process. ~~Clearly we~~'~~re expected to do this~~ another way~~. Another problem with~~ "~~kill~~" ~~is that we want the external process to stop no matter how badly~~ the ~~BEAM is damaged, so we shouldn't rely on stored data or on running final clean-up logic before exiting~~.		It's possible to find the operating system PID of the child process with <code>Port.info(port, :os_pid)</code> and send it a signal by shelling out to unix <code>kill PID</code>, but BEAM doesn't include built-in functions to send a signal to an OS process, and there is an ugly race condition between closing the port and sending this signal. We'll keep looking for another way to "link" the processes.

	To debug what happens during <code>port_close</code> and to eliminate variables, I tried spawning <code>sleep 60</code> instead of rsync and I found that it behaves in exactly the same way: hanging until <code>sleep</code> ends naturally regardless of what happened in Elixir or whether its pipes are still open. This happens to have been a lucky choice as I learned later: "sleep" is daemon-like so similar to rsync, but its behavior is much simpler to reason about.		To debug what happens during <code>port_close</code> and to eliminate variables, I tried spawning <code>sleep 60</code> instead of rsync and I found that it behaves in exactly the same way: hanging until <code>sleep</code> ends naturally regardless of what happened in Elixir or whether its pipes are still open. This happens to have been a lucky choice as I learned later: "sleep" is daemon-like so similar to rsync, but its behavior is much simpler to reason about.

Adamw: light edits

2025-10-21T15:49:33Z

light edits

← Older revision		Revision as of 15:49, 21 October 2025
Line 112:		Line 112:
	It's possible to send a signal by shelling out to unix <code>kill PID</code>, but BEAM doesn't expose the child process ID and doesn't include any built-in functions to send a signal to an OS process. Clearly we're expected to do this another way. Another problem with "kill" is that we want the external process to stop no matter how badly the BEAM is damaged, so we shouldn't rely on stored data or on running final clean-up logic before exiting.		It's possible to send a signal by shelling out to unix <code>kill PID</code>, but BEAM doesn't expose the child process ID and doesn't include any built-in functions to send a signal to an OS process. Clearly we're expected to do this another way. Another problem with "kill" is that we want the external process to stop no matter how badly the BEAM is damaged, so we shouldn't rely on stored data or on running final clean-up logic before exiting.

	To debug what happens during <code>port_close</code> and to eliminate variables, I tried ~~to spawn~~ <code>sleep 60</code> ~~using the same Port command,~~ and I found that it behaves exactly the same way, hanging until ~~the~~ sleep ends naturally regardless of what happened in Elixir or whether its pipes are still open. This happens to have been a lucky choice as I learned later: "sleep" is ~~unusual in the same way as~~ rsync but its behavior is much simpler to reason about.		To debug what happens during <code>port_close</code> and to eliminate variables, I tried spawning <code>sleep 60</code> instead of rsync and I found that it behaves in exactly the same way: hanging until <code>sleep</code> ends naturally regardless of what happened in Elixir or whether its pipes are still open. This happens to have been a lucky choice as I learned later: "sleep" is daemon-like so similar to rsync, but its behavior is much simpler to reason about.

	== Bad assumption: pipe-like processes ==		== Bad assumption: pipe-like processes ==
	A pipeline like <code>gzip</code> or <code>cat</code> it built to read from its input and write to its output. We can roughly group the different styles of command-line application into "pipeline" programs which read and write, "interactive" programs which require user input, and "daemon" programs which are designed to run in the background. Some programs support multiple modes depending on the arguments given at launch, or by detecting the terminal using <code>isatty</code><ref>[https://man.archlinux.org/man/isatty.3.en docs for <code>isatty</code>]</ref>. The BEAM is currently optimized to interface with pipeline programs and it assumes that the external process will stop when its "standard input" is closed.		A pipeline like <code>gzip</code> or <code>cat</code> it built to read from its input and write to its output. We can roughly group the different styles of command-line application into "pipeline" programs which read and write, "interactive" programs which require user input, and "daemon" programs which are designed to run in the background. Some programs support multiple modes depending on the arguments given at launch, or by detecting the terminal using <code>isatty</code><ref>[https://man.archlinux.org/man/isatty.3.en docs for <code>isatty</code>]</ref>. The BEAM is currently optimized to interface with pipeline programs and it assumes that the external process will stop when its "standard input" is closed.

	A typical pipeline program will stop once it detects that input has ended, by ~~making regular C system calls to~~ <code>read</code><ref>[https://man.archlinux.org/man/read.2 libc <code>read</code> docs]</ref>:<syntaxhighlight lang="c">		A typical pipeline program will stop once it detects that input has ended, for example by calling <code>read</code><ref>[https://man.archlinux.org/man/read.2 libc <code>read</code> docs]</ref> in a loop:<syntaxhighlight lang="c">
	~~ssize_t n_read~~ = read (input_desc, buf, bufsize);		size_read = read (input_desc, buf, bufsize);
	if (~~n_read~~ < 0) { error... }		if (size_read < 0) { error... }
	if (~~n_read~~ == 0) { end of file... }		if (size_read == 0) { end of file... }
	</syntaxhighlight>When the program uses blocking I/O, reading zero bytes indicates the end of file. There are also programs which do asynchronous I/O using <code>O_NONBLOCK</code><ref>[https://man.archlinux.org/man/open.2.en#O_NONBLOCK O_NONBLOCK docs]</ref>, and these might rely on the <code>HUP</code> hang-up signal which is normally sent when input is closed.		</syntaxhighlight>

	But here we'll focus on how processes can more generally affect each other through pipes. Surprising answer: without much effect! You can experiment with the <code>/dev/null</code> device which behaves like a closed pipe, for example compare these two commands:<syntaxhighlight lang="shell">		If the program does blocking I/O, then a zero-byte <code>read</code> indicates the end of file condition. A program which does asynchronous I/O with <code>O_NONBLOCK</code><ref>[https://man.archlinux.org/man/open.2.en#O_NONBLOCK O_NONBLOCK docs]</ref> might instead detect EOF by listening for the <code>HUP</code> hang-up signal which is normally sent when input is closed.

			But here we'll focus on how processes can more generally affect each other through pipes. Surprising answer: without much effect! You can experiment with the <code>/dev/null</code> device which behaves like a closed pipe, for example compare these two commands:

			<syntaxhighlight lang="shell">
	cat < /dev/null		cat < /dev/null

Line 138:		Line 142:
	A small shim can adapt a daemon-like program to behave more like a pipeline. The shim is sensitive to stdin closing or SIGHUP, and when this is detected it converts this into a stronger signal like SIGTERM which it forwards to its own child. This is the idea behind a suggested shell script<ref>[https://hexdocs.pm/elixir/1.19.0/Port.html#module-orphan-operating-system-processes Elixir Port docs showing a shim script]</ref> for Elixir, and the <code>erlexec</code><ref name=":0">[https://hexdocs.pm/erlexec/readme.html <code>erlexec</code> library]</ref> library. The opposite adapter can be found in the [[w:nohup\|nohup]] shell command and the grimsby<ref>[https://github.com/shortishly/grimsby <code>grimsby</code> library]</ref> library: these will keep standard in and/or standard out open for the child process even after the parent exits, so that a pipe-like program can behave more like a daemon.		A small shim can adapt a daemon-like program to behave more like a pipeline. The shim is sensitive to stdin closing or SIGHUP, and when this is detected it converts this into a stronger signal like SIGTERM which it forwards to its own child. This is the idea behind a suggested shell script<ref>[https://hexdocs.pm/elixir/1.19.0/Port.html#module-orphan-operating-system-processes Elixir Port docs showing a shim script]</ref> for Elixir, and the <code>erlexec</code><ref name=":0">[https://hexdocs.pm/erlexec/readme.html <code>erlexec</code> library]</ref> library. The opposite adapter can be found in the [[w:nohup\|nohup]] shell command and the grimsby<ref>[https://github.com/shortishly/grimsby <code>grimsby</code> library]</ref> library: these will keep standard in and/or standard out open for the child process even after the parent exits, so that a pipe-like program can behave more like a daemon.

	I used the shim approach in my rsync library and it includes a small C program<ref>[https://gitlab.com/adamwight/rsync_ex/-/blob/main/src/main.c?ref_type=heads rsync_ex C shim program]</ref> which wraps rsync and makes it sensitive to BEAM <code>port_close</code>. It's featherweight, leaving pipes unchanged as it passes control to ~~rsync—its only real effect is~~ to ~~convert~~ SIGHUP ~~to SIGKILL~~ (~~but should have been~~ SIGTERM~~, see the sidebar discussion of different signals below~~).		I used the shim approach in my rsync library and it includes a small C program<ref>[https://gitlab.com/adamwight/rsync_ex/-/blob/main/src/main.c?ref_type=heads rsync_ex C shim program]</ref> which wraps rsync and makes it sensitive to BEAM <code>port_close</code>. It's featherweight, leaving pipes unchanged as it passes control to rsync, here are the business parts:<syntaxhighlight lang="c">// Set up a fail-safe to self-signal with HUP if the controlling process dies.
			prctl(PR_SET_PDEATHSIG, SIGHUP);</syntaxhighlight><syntaxhighlight lang="c">
			void handle_signal(int signum) {
			if (signum == SIGHUP && child_pid > 0) {
			// Send the child TERM so that rsync can perform clean-up such as shutting down a remote server.
			kill(child_pid, SIGTERM);
			}
			}
			</syntaxhighlight>

	== Reliable clean up ==		== Reliable clean up ==

Adamw: memed.

2025-10-21T13:51:49Z

memed.

← Older revision		Revision as of 13:51, 21 October 2025
Line 8:		Line 8:
	I was excited to learn how to interface with long-lived external processes—and this project offered more than I hoped for.		I was excited to learn how to interface with long-lived external processes—and this project offered more than I hoped for.

	{{Aside\|text=<p>[[w:rsync\|Rsync]] is the standard utility for file transfers, locally or over a network. It can resume incomplete transfers and synchronize directories efficiently, and after almost 30 years of usage it can be trusted to handle any edge case.</p>		{{Aside\|text=<p>[[w:rsync\|Rsync]] is the standard utility for file transfers, locally or over a network. It can resume incomplete transfers and synchronize directories efficiently, and after almost 30 years of usage rsync can be trusted to handle any edge case.</p>
	<p>BEAM is a fairly unique ecosystem in which it's not considered deviant to reinvent a rounder wheel: an external dependency like "cron" ~~would~~ often be ported into native Erlang—but the complexity of rsync and its dependence on a matching remote daemon makes it unlikely that it will be rewritten any time soon, which is why I've decided to wrap external command execution in a library.</p>}}		<p>BEAM<ref>The virtual machine shared by Erlang, Elixir, Gleam, Ash, and so on: [https://blog.stenmans.org/theBeamBook/ the BEAM Book]</ref> is a fairly unique ecosystem in which it's not considered deviant to reinvent a rounder wheel: an external dependency like "cron" will often be ported into native Erlang—but the complexity of rsync and its dependence on a matching remote daemon makes it unlikely that it will be rewritten any time soon, which is why I've decided to wrap external command execution in a library.</p>}}

	[[File:Monkey eating.jpg\|alt=A Toque macaque (Macaca radiata) Monkey eating peanuts. Pictured in Bangalore, India\|right\|300x300px]]		[[File:Monkey eating.jpg\|alt=A Toque macaque (Macaca radiata) Monkey eating peanuts. Pictured in Bangalore, India\|right\|300x300px]]

	=== ~~Naive~~ shelling ===		=== Naïve shelling ===

	Starting rsync should be as easy as calling out to a shell:<syntaxhighlight lang="elixir">		Starting rsync should be as easy as calling out to a shell:<syntaxhighlight lang="elixir">
	System.shell("rsync -a source target")		System.shell("rsync -a source target")
	</syntaxhighlight>		</syntaxhighlight>
	This has a few shortcomings, starting with how we pass ~~the filenames~~. It ~~would be possible~~ to ~~pass a dynamic path using~~ string interpolation ~~like~~ <code>#{source}</code> ~~but this is risky~~: consider what ~~happens~~ if the filenames include whitespace or ~~even~~ special shell characters such as ";".		This has a few shortcomings, starting with how one would pass it dynamic paths. It's unsafe to use string interpolation (<code>"#{source}"</code> ): consider what could happen if the filenames include unescaped whitespace or special shell characters such as ";".

	=== Safe path handling ===		=== Safe path handling ===
Line 27:		Line 27:

	=== Asynchronous call and communication ===		=== Asynchronous call and communication ===
	To run a external process asynchronously we reach for Elixir's low-level <code>Port.open</code>, nothing but a one-line wrapper<ref>See the [https://github.com/elixir-lang/elixir/blob/809b035dccf046b7b7b4422f42cfb6d075df71d2/lib/elixir/lib/port.ex#L232 port.ex source code]</ref> ~~which passes~~ its parameters directly to ERTS <code>open_port</code><ref>[https://www.erlang.org/doc/apps/erts/erlang.html#open_port/2 Erlang <code>open_port</code> docs]</ref>. This function is tremendously flexible, here we turn a few knobs:<syntaxhighlight lang="elixir">		To run a external process asynchronously we reach for Elixir's low-level <code>Port.open</code>, nothing but a one-line wrapper<ref>See the [https://github.com/elixir-lang/elixir/blob/809b035dccf046b7b7b4422f42cfb6d075df71d2/lib/elixir/lib/port.ex#L232 port.ex source code]</ref> passing its parameters directly to ERTS <code>open_port</code><ref>[https://www.erlang.org/doc/apps/erts/erlang.html#open_port/2 Erlang <code>open_port</code> docs]</ref>. This function is tremendously flexible, here we turn a few knobs:<syntaxhighlight lang="elixir">
	Port.open(		Port.open(
	{:spawn_executable, rsync_path},		{:spawn_executable, rsync_path},
Line 52:		Line 52:

	; <code>-v</code> : list each filename as it's transferred		; <code>-v</code> : list each filename as it's transferred

			; <code>--progress</code> : report statistics per file

	; <code>--info=progress2</code> : report overall progress		; <code>--info=progress2</code> : report overall progress

	~~; <code>--progress</code> : report statistics per file~~

	; <code>--itemize-changes</code> : list the operations taken on each file		; <code>--itemize-changes</code> : list the operations taken on each file

	; <code>--out-format=FORMAT</code> : any format ~~using parameters from~~ rsyncd.conf's <code>log format</code><ref>https://man.freebsd.org/cgi/man.cgi?query=rsyncd.conf</ref>		; <code>--out-format=FORMAT</code> : any custom format string following rsyncd.conf's <code>log format</code><ref>https://man.freebsd.org/cgi/man.cgi?query=rsyncd.conf</ref>
	}}		}}

	~~We've chosen~~ <code>--info=progress2</code> , so ~~the meaning of the reported percentage is "overall percent complete". Rsync outputs these progress lines in a fairly self-explanatory columnar format~~:<syntaxhighlight lang="text">		Rsync outputs <code>--info=progress2</code> lines like so:<syntaxhighlight lang="text">
	percent complete time remaining		overall percent complete time remaining
	bytes transferred \| transfer speed \|		bytes transferred \| transfer speed \|
	\| \| \| \|		\| \| \| \|
Line 69:		Line 69:
	</syntaxhighlight>		</syntaxhighlight>

	~~Our~~ Port captures ~~output and each line~~ is sent to the library's <code>handle_info</code> callback as <code>{:data, line}</code>. After the transfer is finished we receive a conclusive <code>{:exit_status, status_code}</code> message.		The controlling Port captures these lines is sent to the library's <code>handle_info</code> callback as <code>{:data, line}</code>. After the transfer is finished we receive a conclusive <code>{:exit_status, status_code}</code> message.

	As a first step, we extract the ~~percent_done~~ column and flag any unrecognized output:		As a first step, we extract the overall_percent_done column and flag any unrecognized output:
	<syntaxhighlight lang="elixir">		<syntaxhighlight lang="elixir">
	with terms when terms != [] <- String.split(line, ~r"\s", trim: true),		with terms when terms != [] <- String.split(line, ~r"\s", trim: true),
Line 81:		Line 81:
	{:unknown, line}		{:unknown, line}
	end		end
	</syntaxhighlight>The <code>trim</code> is lifting more than its weight here: it lets us completely ignore spacing and newline ~~trickery—even skipping~~ the leading carriage return ~~that can be~~ seen in the rsync source code,<ref>[https://github.com/RsyncProject/rsync/blob/797e17fc4a6f15e3b1756538a9f812b63942686f/progress.c#L129 rsync/progress.c] source code</ref>		</syntaxhighlight>The <code>trim</code> is lifting more than its weight here: it lets us completely ignore spacing and newline trickery—and ignores the leading carriage return before each line, seen in the rsync source code:<ref>[https://github.com/RsyncProject/rsync/blob/797e17fc4a6f15e3b1756538a9f812b63942686f/progress.c#L129 rsync/progress.c] source code</ref>
	<syntaxhighlight lang="c">		<syntaxhighlight lang="c">
	rprintf(FCLIENT, "\r%15s %3d%% %7.2f%s %s%s", ...);		rprintf(FCLIENT, "\r%15s %3d%% %7.2f%s %s%s", ...);
	</syntaxhighlight>Carriage return <code>\r</code> deserves special mention: this "control" character ~~is just a~~ byte in the binary data coming over the pipe from rsync, ~~but its~~ normal role is to control the terminal emulator, rewinding the cursor so that the current line can be overwritten!		</syntaxhighlight>Carriage return <code>\r</code> deserves special mention: this is the first "control" character we come across and it looks the same as an ordinary byte in the binary data coming over the pipe from rsync, similar to newline <code>\n</code>. Its normal role is to control the terminal emulator, rewinding the cursor so that the current line can be overwritten! And like newline, carriage return can be ignored. Control signaling is exactly what goes haywire about this project, and the leaky category distinction between data and control seems to be a repeated theme in inter-process communication. The reality is not so much data vs. control, as it seems to be a sequence of layers like with [[w:OSI model\|networking]].

	A repeated theme in inter-process communication is ~~that~~ data ~~and~~ control ~~are leaky categories. We come~~ to ~~the more formal control side channels later~~.

	{{Aside\|text=		{{Aside\|text=
Line 97:		Line 95:
	You'll have to use <control>-v <control>-m to type a literal carriage return, copy-and-paste won't work.		You'll have to use <control>-v <control>-m to type a literal carriage return, copy-and-paste won't work.

	The character ~~seems to be~~ named after pushing ~~the physical paper carriage~~ of a typewriter ~~back~~ to the beginning of the line without feeding the roller.		The character is named after the pushing of a physical typewriter carriage to return to the beginning of the current line without feeding the roller to a new line.

	[[File:~~Nilgais fighting, Lakeshwari, Gwalior district, India~~.jpg\|left\|~~200x200px~~]]		[[File:Baboons Playing in Chobe National Park-crlf.jpg\|left\|300x300px\|Three young baboons playing on a rock ledge. Two are on the ridge and one below, grabbing the tail of another. A meme font shows "\r", "\n", and "\r\n" personified as each baboon.]]
	[[w:https://en.wikipedia.org/wiki/Newline#Issues_with_different_newline_formats\|Disagreement about carriage return]] vs. line feed has caused eye-rolling since the dawn of personal computing.		[[w:https://en.wikipedia.org/wiki/Newline#Issues_with_different_newline_formats\|Disagreement about carriage return]] vs. line feed has caused eye-rolling since the dawn of personal computing.
	}}		}}
Line 145:		Line 143:
	{{Project\|status=in review\|url=https://erlangforums.com/t/open-port-and-zombie-processes\|source=https://github.com/erlang/otp/pull/9453}}		{{Project\|status=in review\|url=https://erlangforums.com/t/open-port-and-zombie-processes\|source=https://github.com/erlang/otp/pull/9453}}
	It's always a pleasure to ask questions in the BEAM communities, they deserve their reputation for being friendly and open. The first big tip was to look at the third-party library <code>erlexec</code><ref name=":0" />, which demonstrates emerging best practices which could be backported into the language itself. Everyone speaking on the problem generally agrees that the fragile clean up of external processes is a bug, and supports the idea that some flavor of "terminate" signal should be sent to spawned programs when the port is closed.		It's always a pleasure to ask questions in the BEAM communities, they deserve their reputation for being friendly and open. The first big tip was to look at the third-party library <code>erlexec</code><ref name=":0" />, which demonstrates emerging best practices which could be backported into the language itself. Everyone speaking on the problem generally agrees that the fragile clean up of external processes is a bug, and supports the idea that some flavor of "terminate" signal should be sent to spawned programs when the port is closed.
			[[File:Itinerant glassworker exhibition with spinning wheel and steam engine.jpg\|thumb]]
	I would be lying to hide my disappointment that the required core changes are mostly in an auxiliary C program and not written in Erlang or even in the BEAM itself, but it was still fascinating to open such an elegant black box and find the technological equivalent of a steam engine inside. All of the futuristic, high-level features we've come to know actually map closely to a few scraps of wizardry with ordinary pipes<ref>[https://man.archlinux.org/man/pipe.7.en Overview of unix pipes]</ref>, using libc's pipe<ref>[https://man.archlinux.org/man/pipe.2.en Docs for the <code>pipe</code> syscall]</ref>, read, write, and select<ref>[https://man.archlinux.org/man/select.2.en libc <code>select</code> docs]</ref>.		I would be lying to hide my disappointment that the required core changes are mostly in an auxiliary C program and not written in Erlang or even in the BEAM itself, but it was still fascinating to open such an elegant black box and find the technological equivalent of a steam engine inside. All of the futuristic, high-level features we've come to know actually map closely to a few scraps of wizardry with ordinary pipes<ref>[https://man.archlinux.org/man/pipe.7.en Overview of unix pipes]</ref>, using libc's pipe<ref>[https://man.archlinux.org/man/pipe.2.en Docs for the <code>pipe</code> syscall]</ref>, read, write, and select<ref>[https://man.archlinux.org/man/select.2.en libc <code>select</code> docs]</ref>.

Adamw at 14:41, 20 October 2025

2025-10-20T14:41:31Z

Show changes

Adamw: c/e

2025-10-20T12:26:14Z

c/e

Adamw: light c/e

2025-10-20T10:08:04Z

light c/e

Show changes

← Older revision		Revision as of 12:26, 20 October 2025
Line 9:		Line 9:

	{{Aside\|text=<p>[[w:rsync\|Rsync]] is the standard utility for file transfers, locally or over a network. It can resume incomplete transfers and synchronize directories efficiently, and after almost 30 years of usage it can be trusted to handle any edge case.</p>		{{Aside\|text=<p>[[w:rsync\|Rsync]] is the standard utility for file transfers, locally or over a network. It can resume incomplete transfers and synchronize directories efficiently, and after almost 30 years of usage it can be trusted to handle any edge case.</p>
	<p>BEAM is a fairly unique ecosystem in which it's not considered deviant to reinvent a rounder wheel: ~~it's common to port~~ external ~~dependencies~~ into native Erlang—but the complexity of rsync and its dependence on a matching remote daemon makes it unlikely that it will be rewritten any time soon, which is why I've decided to wrap external command execution in a library.</p>}}		<p>BEAM is a fairly unique ecosystem in which it's not considered deviant to reinvent a rounder wheel: an external dependency like "cron" would often be ported into native Erlang—but the complexity of rsync and its dependence on a matching remote daemon makes it unlikely that it will be rewritten any time soon, which is why I've decided to wrap external command execution in a library.</p>}}

	[[File:Monkey eating.jpg\|alt=A Toque macaque (Macaca radiata) Monkey eating peanuts. Pictured in Bangalore, India\|right\|300x300px]]		[[File:Monkey eating.jpg\|alt=A Toque macaque (Macaca radiata) Monkey eating peanuts. Pictured in Bangalore, India\|right\|300x300px]]
Line 71:		Line 71:
	Our Port captures output and each line is sent to the library's <code>handle_info</code> callback as <code>{:data, line}</code>. After the transfer is finished we receive a conclusive <code>{:exit_status, status_code}</code> message.		Our Port captures output and each line is sent to the library's <code>handle_info</code> callback as <code>{:data, line}</code>. After the transfer is finished we receive a conclusive <code>{:exit_status, status_code}</code> message.

	As a first step, we extract the percent_done column and ~~log~~ any unrecognized output:		As a first step, we extract the percent_done column and flag any unrecognized output:
	<syntaxhighlight lang="elixir">		<syntaxhighlight lang="elixir">
	with terms when terms != [] <- String.split(line, ~r"\s", trim: true),		with terms when terms != [] <- String.split(line, ~r"\s", trim: true),
	percent_done_text when ~~is_binary(~~percent_done_text) <- Enum.at(terms, 1),		percent_done_text when percent_done_text != nil <- Enum.at(terms, 1),
	{percent_done, "%"} <- Float.parse(percent_done_text) do		{percent_done, "%"} <- Float.parse(percent_done_text) do
	percent_done		percent_done
Line 81:		Line 81:
	{:unknown, line}		{:unknown, line}
	end		end
	</syntaxhighlight>The <code>trim</code> is lifting more than its weight here: it lets us completely ignore spacing and newline ~~trickery—and even a~~ leading carriage return that we can ~~see~~ in the rsync source code,<ref>[https://github.com/RsyncProject/rsync/blob/797e17fc4a6f15e3b1756538a9f812b63942686f/progress.c#L129 rsync/progress.c] source code</ref>		</syntaxhighlight>The <code>trim</code> is lifting more than its weight here: it lets us completely ignore spacing and newline trickery—even skipping the leading carriage return that can be seen in the rsync source code,<ref>[https://github.com/RsyncProject/rsync/blob/797e17fc4a6f15e3b1756538a9f812b63942686f/progress.c#L129 rsync/progress.c] source code</ref>
	<syntaxhighlight lang="c">		<syntaxhighlight lang="c">
	rprintf(FCLIENT, "\r%15s %3d%% %7.2f%s %s%s", ...);		rprintf(FCLIENT, "\r%15s %3d%% %7.2f%s %s%s", ...);
	</syntaxhighlight>Carriage return <code>\r</code> deserves a special mention: this "control" character is just a byte in the binary data coming over the pipe from rsync, but its normal role is ~~playing a~~ control ~~function because of how~~ the terminal emulator ~~responds to it. On a terminal the effect is to rewind~~ the cursor so that the current line can be overwritten!		</syntaxhighlight>Carriage return <code>\r</code> deserves special mention: this "control" character is just a byte in the binary data coming over the pipe from rsync, but its normal role is to control the terminal emulator, rewinding the cursor so that the current line can be overwritten!

	A repeated theme in inter-process communication is that data and control are leaky categories. We come to the more formal control side channels later.		A repeated theme in inter-process communication is that data and control are leaky categories. We come to the more formal control side channels later.
Line 103:		Line 103:

	== OTP generic server ==		== OTP generic server ==
	The Port API is convenient enough so far, but ~~where~~ Erlang/OTP really starts to shine ~~is when~~ we wrap each Port connection under a gen_server<ref>[https://www.erlang.org/doc/apps/stdlib/gen_server.html Erlang gen_server docs]</ref> module, giving us ~~some~~ properties for free: A dedicated thread coordinates with its rsync independent of anything else. Input and output are asynchronous and buffered, but handled sequentially in a thread-safe way. It holds internal state including the up-to-date completion percentage. And the caller can ~~either~~ request updates ~~manually~~, or it can listen for ~~pushed~~ statistics.		The Port API is convenient enough so far, but Erlang/OTP really starts to shine once we wrap each Port connection under a <code>gen_server</code><ref>[https://www.erlang.org/doc/apps/stdlib/gen_server.html Erlang gen_server docs]</ref> module, giving us several properties for free: A dedicated application thread coordinates with its rsync process independent of anything else. Input and output are asynchronous and buffered, but handled sequentially in a thread-safe way. The gen_server holds internal state including the up-to-date completion percentage. And the caller can request updates as needed, or it can listen for push messages with the parsed statistics.

	This gen_server ~~should~~ also ~~be able~~ to run under an [https://adoptingerlang.org/docs/development/supervision_trees/ ~~OTP supervision tree~~] but this is where ~~the~~ dream falls apart, for the moment. The Port ~~can watch~~ for rsync completion or failure and ~~report this~~ to its caller, but we fail at the ~~second~~ critical property of being able to shut down rsync if the calling code or our library module crashes.		This gen_server is also expected to run safely under an OTP supervision tree<ref>[https://adoptingerlang.org/docs/development/supervision_trees/ "Supervision Trees"] chapter from [https://adoptingerlang.org/ Adopting Erlang]</ref> but this is where our dream falls apart for the moment. The Port already watches for rsync completion or failure and reports upwards to its caller, but we fail at the critical property of being able to propagate a termination downwards to shut down rsync if the calling code or our library module crashes.

	== Problem: runaway processes ==		== Problem: runaway processes ==
	[[File:CargoNet Di 12 Euro 4000 Lønsdal - Bolna.jpg\|thumb]]		[[File:CargoNet Di 12 Euro 4000 Lønsdal - Bolna.jpg\|thumb]]
	The unpleasant real-world consequence ~~of this limitation~~ is that rsync transfers ~~would~~ continue to run in the background even after Elixir ~~had completely shut~~ down, because the BEAM has no way of stopping the process.		The unpleasant real-world consequence is that rsync transfers will continue to run in the background even after Elixir kills our gen_server or shuts down, because the BEAM has no way of stopping the external process.

	It ~~might be~~ possible to send a signal ~~using~~ unix "kill", but BEAM doesn't expose the child process ID and it doesn't include any built-in ~~commands~~ to send a signal. Clearly we're expected to do this another way. Another problem with "kill" is that we want the external process to stop no matter how badly the BEAM is damaged so we ~~can~~'t rely on stored data ~~and~~ on ~~making a few last calls~~ before ~~crashing~~.		It's possible to send a signal by shelling out to unix <code>kill PID</code>, but BEAM doesn't expose the child process ID and doesn't include any built-in functions to send a signal to an OS process. Clearly we're expected to do this another way. Another problem with "kill" is that we want the external process to stop no matter how badly the BEAM is damaged, so we shouldn't rely on stored data or on running final clean-up logic before exiting.

	To ~~eliminate variable~~ and to ~~understand whether the failure to stop was specific to rsync~~, I tried ~~the same Port command but spawning a~~ <code>sleep 60</code>, and I found that it behaves exactly the same way, hanging until the sleep ends naturally regardless of what happened in Elixir or whether its pipes are still open. This happens to have been a lucky choice, as I learned later ~~that~~ "sleep" is ~~also~~ unusual but its behavior is much simpler to reason about.		To debug what happens during <code>port_close</code> and to eliminate variables, I tried to spawn <code>sleep 60</code> using the same Port command, and I found that it behaves exactly the same way, hanging until the sleep ends naturally regardless of what happened in Elixir or whether its pipes are still open. This happens to have been a lucky choice as I learned later: "sleep" is unusual in the same way as rsync but its behavior is much simpler to reason about.

	== Bad assumption: pipe-like processes ==		== Bad assumption: pipe-like processes ==