Remember the different Linux manual sections.
man 1 -> user commands
man 2 -> Linux system calls
man 3 -> C library functions (but not necessarily C Standard Library functions)
man 4 -> special files
man 5 -> file formats
man 8 -> administration and privileged commands
Note: These are the current manual sections. Our textbook uses an
older numbering for manual sections. For example, the book considers
Section 4 to be file formats, but that is now Section 5.
You can get an introduction to section n with the command
$ man n intro
--------------------------------------------------------------------------
Chapter 10
The programs in this chapter are used in the following order.
watch.sh
listargs.c
stdinredir1.c
stdinredir2.c
whotofile.c
pipedemo.c
pipedemo2.c
pipe.c
--------------------------------------------------------------------------
Section 10.2
watch.sh is a shell "program"
who | sort > prev # get initial user list
while true # true is a program
do
sleep 30 # wait a while
who | sort > curr # get current user list
echo "Logged out:" # print header
comm -23 prev curr # and results
echo "Logged in:" # header
comm -13 prev curr # and results
mv curr prev # make now past
done
Here are the commands used in this shell program.
man 1 who
man 1 sort
man 1 true
man 1 sleep
man 1 comm
man 1 mv
The I/O redirection operator > is part of bash.
The Pipe operator | is part of bash.
The while-loop
while list-1; do list-2; done
is part of bash and so is echo.
--------------------------------------------------------------------------
Section 10.3
I/O Redirection
The three standard streams that every process has open (when the process
is created) are "standard input", "standard output", and "standard error".
I/O redirection of these three streams, when using the shell, is documented here.
man 1 bash redirection
Besides using I/O redirection with the shell, we want to study how
I/O redirection is implemented by C programs. In particular, we will
look at how the shell sets up I/O redirection for its child processes
(the commands the shell is asked to run). The programming techniques
used by a shell to implement I/O redirection are used by many other
programs. For example, web servers use I/O redirection when they run
processes that create dynamic web pages.
The three standard streams have two ways of being identified in a C
program, a "unix way" and a "C way".
In unix, the standard streams are identified by their "file descriptors",
STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO
which are symbols for the integers 0, 1, and 2 respectively.
In C, the standard streams are identified by their "FILE objects"
stdin, stdout, stderr
which are pointers to FILE data structures.
When you do I/O using the C Standard Library, you use FILE objects.
man 3 stdio
man 3 stdin, stdout, stderr
When you do I/O using unix system calls, you use file descriptors.
man 2 read
man 2 write
man 2 open
man 2 close
man 2 lseek
man 2 fstat
man 2 fcntl
The functions that implement I/O redirection (and also pipes) are system
calls and therefore they use file descriptors (not FILE objects).
man 2 dup
man 2 dup2
man 2 pipe
But there is one FILE object function that can be used to do I/O redirection
man 3 freopen
--------------------------------------------------------------------------
Section 10.4
This section describes three ways that a process can redirect its own
standard input stream to a file.
Here is how we can illustrate this idea. A process might start like this,
with its standard input stream tied to the console keyboard.
process
+------------+
| |
keyboard --->>0 1>>--->stdout
| |
| 2>>
| |
+------------+
Then, after the process redirects it standard input to the file "data",
the picture would look like this.
process
+------------+
| |
"data" --->>0 1>>--->stdout
| |
| 2>>
| |
+------------+
The key idea needed for this section is a fact mentioned on page 327.
Every time you ask the operating system for a new file descriptor, the OS
*always* gives you the lowest available descriptor. An example of how you
ask for a new file descriptor is when you open() a file. In addition to
open(), the functions dup() and pipe() also ask the operating system for
new file descriptors. (The close() function frees up a file descriptor.)
Be sure to carefully compare Figures 10.7 & 10.8 (the "close then open" method)
with Figure 10.9 (the "open, close, dup, close" method). The book does not have
a picture for the ""open, dup2, close" method, so draw your own.
You should create your own examples of a process redirecting its own standard
out stream.
Also, try to create an example of a process that redirects both it own standard
input and standard output streams.
These techniques are a stepping stone towards learning how a parent process can
redirect the standard streams of a child process.
Here is a good question. Why do we need three methods? Method 1 is straight
forward and easy to understand and use. Why not always use it? The answer is
found in Section 10.5. When we want a parent process to redirect the streams
of a child process, we need more versatility than what Method 1 provides.
Sometimes methods 2 or 3 are needed.
--------------------------------------------------------------------------
Section 10.4.1 Method 1: close() then open()
Process starts like this.
process
+------------+
| |
keyboard --->>0 1>>--->stdout
| |
| 2>>
| |
+------------+
The process calls close(0);
process
+------------+
| |
| 1>>--->stdout
| |
| 2>>
| |
+------------+
The process calls open("data") and gets the lowest possible file descriptor.
process
+------------+
| |
"data" --->>0 1>>--->stdout
| |
| 2>>
| |
+------------+
--------------------------------------------------------------------------
Section 10.4.2 Method 2: open(), close(), dup(), close()
Process starts like this.
process
+------------+
| |
keyboard --->>0 1>>--->stdout
| |
| 2>>
| |
+------------+
The process calls open("data") and gets the lowest possible file descriptor.
process
+------------+
| |
keyboard --->>0 1>>--->stdout
| |
"data" --->>3 2>>
| |
+------------+
The process calls close(0).
process
+------------+
| |
| 1>>--->stdout
| |
"data" --->>3 2>>
| |
+------------+
The process calls dup(3) and gets the lowest possible file descriptor.
process
+------------+
| |
+-->>0 1>>--->stdout
| | |
"data"---+-->>3 2>>
| |
+------------+
The process calls close(3).
process
+------------+
| |
"data" --->>0 1>>--->stdout
| |
| 2>>
| |
+------------+
--------------------------------------------------------------------------
Section 10.4.4 Method 3: open(), dup2(), close()
Process starts like this.
process
+------------+
| |
keyboard --->>0 1>>--->stdout
| |
| 2>>
| |
+------------+
The process calls open("data") and gets the lowest possible file descriptor.
process
+------------+
| |
keyboard --->>0 1>>--->stdout
| |
"data" --->>3 2>>
| |
+------------+
The process calls dup2(3,0) (this is like calling close(0); dup(3))
process
+------------+
| |
+-->>0 1>>--->stdout
| | |
"data"-- +-->>3 2>>
| |
+------------+
The process calls close(3).
process
+------------+
| |
"data" --->>0 1>>--->stdout
| |
| 2>>
| |
+------------+
--------------------------------------------------------------------------
Section 10.5
This section describes how a parent process can redirect the standard streams
of a child process.
When a parent creates a child process, there are four possible configurations
that the parent might want the child to run with. It may be that the parent
and child processes should share stdin and stdout (this is the default
configuration after the child is created).
parent
+------------+
| |
+--------->>0 1>>-----+
| | | |
| | 2>> |
| | | |
in-stream --+ +------------+ +--> out-stream
(shared) | | (shared)
| child |
| +------------+ |
| | | |
+-->>0 1>>-----------+
| |
| 2>>
| |
+------------+
It may be that the parent wants the child processes to share its standard input
but have its standard output redirected to a file.
parent
+------------+
| |
+--------->>0 1>>-------> out-stream
| | |
| | 2>>
| | |
in-stream --+ +------------+
(shared) |
| child
| +------------+
| | |
+-->>0 1>>----> "out_file"
| |
| 2>>
| |
+------------+
It may be that the parent wants the child processes to share its standard output
but have its standard input redirected to a file.
parent
+------------+
| |
in-stream ----------->>0 1>>-----+
| | |
| 2>> |
| | |
+------------+ +--> out-stream
| (shared)
child |
+------------+ |
| | |
"in_file" --->>0 1>>-----------+
| |
| 2>>
| |
+------------+
It may be that the parent wants the child processes to have both its
standard input and standard output redirected to files.
parent
+------------+
| |
in-stream ----------->>0 1>>-------> out-stream
| |
| 2>>
| |
+------------+
child
+------------+
| |
"in_file" --->>0 1>>----> "out_file"
| |
| 2>>
| |
+------------+
Its important to realize that this redirection of streams is usually done
in preparation for the exec() of a new program in the child process. It
is the standard streams of the new program that we really want redirected.
The key idea is that the new program need not even know that its standard
streams have been redirected.
After the parent forks the child, the child inherits all the parent's open
file descriptors (so the parent and child "share" every one of these open
file descriptors). The child can then use the ideas from Section 10.5 to
redirect its own stdin and stdout. (And the parent can close() any file
descriptors that it no longer has any use for.) After the child sets up
the I/O redirection, the child calls exec(), and the new program runs (in
the child process) with magically redirected streams
Let us look at a detailed example of how a parent and child can set up
the fourth situation above.
parent
+------------+
| |
in-stream ----------->>0 1>>-------> out-stream
| |
| 2>>
| |
+------------+
child
+------------+
| |
"in_file" --->>0 1>>----> "out_file"
| |
| 2>>
| |
+------------+
The parent starts like this.
parent
+------------+
| |
in-stream -------->>0 1>>-------> out-stream
| |
| 2>>
| |
+------------+
The parent calls
fd1 = open("in_file");
fd2 = open("out_file");
and gets the two lowest possible file descriptors. At this point
the parent knows for sure that the two files are available. If
they are not, the parent issues an error and doesn't do the
following steps.
parent
+------------+
| |
in-stream ----------->>0 1>>-------> out-stream
| |
"in_file" ----->>3=fd1 2>>
| |
| fd2=4>>-----> "out_file"
| |
+------------+
The parent calls fork(), and the child shares all open file descriptors.
parent
+------------+
| |
in-stream -+--------->>0 1>>-------+--> out-stream
(shared) | | | | (shared)
| +------>>3=fd1 2>> |
| | | | |
| | | fd2=4>>---+ |
| | | | | |
| | +------------+ | |
| | | |
| | child | |
| | +------------+ | |
| | | | | |
+---->>0 1>>-----------+
| | | |
"in_file" ----+-->>3=fd1 2>> |
(shared) | | |
| fd2=4>>-------+--> "out_file"
| | (shared)
+------------+
The parent calls close(fd1) and close(fd2).
parent
+------------+
| |
in-stream -+--------->>0 1>>-------+--> out-stream
(shared) | | | | (shared)
| | 2>> |
| | | |
| +------------+ |
| |
| child |
| +------------+ |
| | | |
+---->>0 1>>-----------+
| |
"in_file" ------->>3=fd1 2>>
| |
| fd2=4>>----------> "out_file"
| |
+------------+
The child calls close(STDIN_FILENO) and close(STDOUT_FILENO).
parent
+------------+
| |
in-stream -+--------->>0 1>>----------> out-stream
| |
| 2>>
| |
+------------+
child
+------------+
| |
| |
| |
"in_file" ------->>3=fd1 2>>
| |
| fd2=4>>----------> "out_file"
| |
+------------+
The child calls dup(fd1) and dup(fd2) IN THAT ORDER (why?).
parent
+------------+
| |
in-stream -+--------->>0 1>>----------> out-stream
| |
| 2>>
| |
+------------+
child
+------------+
| |
+---->>0 1>>----+
| | | |
"in_file" --+---->>3=fd1 2>> |
| | |
| fd2=4>>----+-----> "out_file"
| |
+------------+
The child calls close(fd1) and close(fd2).
parent
+------------+
| |
in-stream ----------->>0 1>>-------> out-stream
| |
| 2>>
| |
+------------+
child
+------------+
| |
"in_file" --->>0 1>>----> "out_file"
| |
| 2>>
| |
+------------+
Notice that we used Method 2 from Section 10.4.2 (not the simpler
Method 1 from Section 10.4.1). The reason is that we wanted the
parent to open the files first, before forking the child, so that
the parent could make sure that the files existed. If we used the
simpler close-open method in the child, and the open() call failed,
then we would have needlessly created the child and, worse, it will
be difficult for the child to let the parent know what went wrong.
--------------------------------------------------------------------------
Section 10.6
Pipes
This section describes how a parent process creates a pipe and then
redirects the standard streams of a child process to the pipe.
man 2 pipe http://man7.org/linux/man-pages/man2/pipe.2.html
man 7 pipe http://man7.org/linux/man-pages/man7/pipe.7.html
A parent process can create a pipe, fork once, then set up a
pipeline between itself and its child process two ways. Like this,
parent child
+------------+ +------------+
| | pipe | |
in-stream-->>0 1>>--0========0-->>0 1>>--->out-stream
| | | |
| 2>> | 2>>
| | | |
+------------+ +------------+
or, like this
child parent
+------------+ +------------+
| | pipe | |
in-stream-->>0 1>>--0========0-->>0 1>>--->out-stream
| | | |
| 2>> | 2>>
| | | |
+------------+ +------------+
Or, a parent can create a pipe, fork twice, then set up a
pipeline between its two child processes (this is what a shell
does to implement the '|' operator).
parent
+------------+
| |
+--------->>0 1>>----------------------+
| | | |
| | 2>> |
| | | |
in-stream --+ +------------+ +-->out-stream
(shared) | | (shared)
| child1 child2 |
| +------------+ +------------+ |
| | | pipe | | |
+-->>0 1>>-0======0->>0 1>>---+
| | | |
| 2>> | 2>>
| | | |
+------------+ +------------+
See the folder
create-a-pipeline-examples
for more examples of how to create a pipeline.
Everything is based on the following four facts (from the bottom
of page 327, the middle of page 333, and the bottom of page 336).
1) Every process is created with (at least) three open
file descriptors (0, 1, and 2).
2) Every time you ask the operating system for a new file
descriptor, you are given the lowest available descriptor.
3) When a process forks a child, the parent and child
processes share all open file descriptors.
4) A call to exec() does not change any of the file descriptors
(unless a file descriptor has the close-on-exec, FD_CLOEXEC,
flag set).