/bin/sh: Using Named Pipes to Talk to Your Main Process

A shell application with the main thread providing output and status display for the worker processes

You want to fork off a couple of subshells and have them talk back to your main Process? Then this post is for you.

What is a Named Pipe?

A named pipe is a pipe with a file system node. This allows arbitrary numbers of processes to read and write from the pipe. Which in turn makes multiple usage scenarios possible. his post just covers one of them, others may be covered in future posts.

The Shell

The following examples should work in any Bourne Shell clone, such as the Almquist Shell (/bin/sh on FreeBSD) or the Bourne-Again Shell (bash).

HowTo

The first step is to create a Named Pipe. This can be done with the mkfifo(1) command:

# Get a temporary file name
node="$(mktemp -u)" || exit
# Create a named pipe
mkfifo -m0600 "$node" || exit

Running that code should produce a Named Pipe in /tmp.

The next step is to open a file descriptor. In this example a file descriptor is used for reading and writing, this avoids a number of pitfalls like deadlocking the script:

# Attach the pipe to file descriptor 3
exec 3<> "$node"
# Remove file system node
rm "$node"

Note how the file system node of the named pipe is removed immediately after assigning a file descriptor. The exec 3<> "$node" command has opened a permanent file descriptor, which remains open until manually closed or until the process terminates. So deleting the file system node will cause the system to remove the Named Pipe as soon as the process terminates, even when it is terminated by a signal like SIGINT (user presses CTRL-C).

Forking and Writing into the Named Pipe

From this point on the subshells can be forked using the & operator:

# This function does something
do_something() {
    echo "do_something() to stdout"
    echo "do_something() to named pipe" >&3
}

# Fork do_something()
do_something &
# Fork do_something(), attach stdout to the named pipe
do_something >&3 &

# Fork inline
(
    echo "inline to pipe" >&3
) &
# Fork inline, attach stdout to the named pipe
(
    echo "inline to stdout"
) >&3 &

Whether output is redirected per command or for the entire subshell is a matter of personal taste. Either way the processes inherit the file descriptor to the Named Pipe. It is also possible to redirect stderr as well, or redirect it into a different named pipe.

The Named Pipe is buffered, so all the subshells can start writing into it immediately. Once the buffer is full, processes trying to write into the pipe will block, so sooner or later the data needs to be read from the pipe.

Reading from the Named Pipe

To read from the pipe the shell-builtin command read is used.

Using non-builtin commands like head(1) usually leads to problems, because they may read more data from a pipe than they output, causing it to be lost.

# Make sure white space does not get mangled by read (IFS only contains the newline character)
IFS='
'

# Blocking read, this will halt the process until data is available
read line <&3

# Non-blocking read that reads as much data as is currently available
line_count=0
lines=
while read -t0 line <&3; do
    line_count=$((line_count + 1))
    lines="$lines$line$IFS"
done

Using a blocking read causes the process to sleep until data is available. The process does not require any CPU time, the kernel takes care of waking the process.

That's all that is required to establish ongoing communication between your processes.

The direction of communication can be reversed to use the pipe as a job queue for forked processes. Or a second pipe can be used to establish 2-way communications. With just two processes a single pipe might suffice for two way communications. A named pipe can be connected to an ssh(1) session or nc(1).

Basically named pipes are a way to establish a pipe to background processes or completely independent processes, which do not even have to run on the same machine. So, happy hacking!