How Bash Actually Interprets Your Script
Bash scripts are easy to start writing.
#!/usr/bin/env bash
name="admin"
echo "Hello $name"
The trouble usually starts when the script gets a little more complicated.
Bash doesn't simply take a line and pass it to the operating system. It first interprets the line, expands variables and commands, possibly splits text into words, expands wildcards, sets up redirections, and then executes something.
That explains a surprising number of Bash behaviours.
This article follows that path.
1. How Bash Reads a Script
Before getting into the individual stages, it helps to understand what Bash is dealing with.
A Bash script contains commands, keywords, operators, assignments, functions, conditionals, loops, and so on.
For example:
if [[ -f "$file" ]]; then
echo "Found"
fi
Bash has to recognise if, [[, -f, the variable expansion, then, and fi as parts of the shell language.
This is different from a program such as Java, where the source code is compiled into another form before the program runs.
Bash is an interpreter. It reads shell syntax and executes commands as it goes.
Bash isn't the same as sh
This matters when writing scripts.
If you write:
#!/usr/bin/env bash
you're explicitly asking for Bash.
If you write:
#!/bin/sh
you're asking for the system's sh implementation.
Bash has features that aren't part of portable POSIX sh, such as:
[[ ... ]]
arrays:
items=(one two three)
and several Bash-specific parameter expansions.
So this is a bad combination:
#!/usr/bin/env bash
followed by:
sh script.sh
The shebang doesn't matter when you explicitly invoke another interpreter.
2. Parsing
Parsing is where Bash figures out the structure of what you typed.
You don't need to become a compiler expert to understand this. Think of parsing as Bash answering questions such as:
Is this an assignment?
Is this a command?
Where does this command end?
Is this an
ifstatement?Is this a pipeline?
Is this a function?
Which words belong together syntactically?
Small changes can completely change the result.
Assignment syntax
This is an assignment:
name="John"
This isn't:
name = "John"
The second line contains three separate words:
name
=
"John"
Bash therefore treats name as a command.
The same idea explains why this is wrong:
if[ "$name" = "John" ]; then
Bash doesn't see if followed by a condition. It sees a different token.
Spaces in shell scripts aren't merely formatting.
[ ... ] is a command
This often looks like special syntax:
[ "$name" = "John" ]
But [ is actually a Bash builtin command. Historically, an external [ command also exists on Unix systems.
You can inspect what Bash uses:
type [
The closing ] is required because it is part of the command's argument list.
That's why the spaces matter:
[ "$name" = "John" ]
not:
["$name" = "John"]
The latter changes the first word Bash sees.
[[ ... ]] is different
Bash also provides:
[[ "$name" == "John" ]]
[[ ... ]] is a Bash conditional construct rather than just another spelling of [ ... ].
It has some useful behaviour around quoting, pattern matching and operators.
For example:
if [[ "$name" == J* ]]; then
echo "Starts with J"
fi
The * here is being used as a pattern.
If you're writing Bash specifically, [[ ... ]] is generally preferable to [ ... ] for conditions.
;, &&, || and |
Bash also uses operators to connect commands:
command1; command2
Run both.
command1 && command2
Run command2 only if command1 succeeds.
command1 || command2
Run command2 if command1 fails.
And:
command1 | command2
Connect the output of one command to the input of another.
These aren't ordinary characters passed to a program. Bash interprets them.
3. Expansion
Once Bash has understood the structure of the command, it starts replacing pieces of it with values.
This is called expansion.
For example:
name="John"
echo "Hello $name"
The shell expands $name before echo runs.
There are several types of expansion.
Parameter expansion
The common form is:
$name
There are many more:
${name}
${name:-default}
${name#prefix}
${name%suffix}
For example:
name="application.log"
echo "${name%.log}"
produces:
application
The syntax is compact, but it is often much cleaner than starting another command such as sed just to remove a suffix.
Command substitution
This:
today=$(date)
runs date and uses its output as the value of today.
Command substitution is extremely useful, but remember that you're running another command.
For example:
files=$(find /tmp -type f)
may look convenient, but storing arbitrary filenames in a string can become problematic because filenames can contain spaces, newlines, and other characters.
For file processing, this is often safer:
while IFS= read -r file; do
...
done < <(find /tmp -type f -print0)
with appropriate null-delimited handling when filenames genuinely need to be arbitrary.
Arithmetic expansion
Bash can perform arithmetic directly:
total=$((10 + 20))
and:
((count++))
is an arithmetic command.
This is different from treating the variable as an integer type. Bash variables don't work like Java or C variables with declared numeric types.
Brace expansion
Bash can generate text using braces:
echo file{1,2,3}.txt
produces:
file1.txt file2.txt file3.txt
It can also generate ranges:
echo {1..5}
produces:
1 2 3 4 5
Brace expansion is performed by Bash before the command runs.
It isn't the same thing as wildcard expansion.
Command substitution removes trailing newlines
One easy detail to miss:
value=$(printf 'hello\n\n')
The trailing newlines are removed from the command-substitution result.
So:
printf '<%s>\n' "$value"
prints:
<hello>
If you're using command substitution to capture structured output where trailing whitespace matters, this can matter.
4. Word Splitting
After expansion, Bash may perform word splitting.
Word splitting means Bash can take the result of an unquoted expansion and turn it into multiple words.
Consider:
name="John Smith"
printf '<%s>\n' $name
The command receives two arguments:
<John>
<Smith>
But:
printf '<%s>\n' "$name"
receives one:
<John Smith>
This is one of the reasons quoting variables is such a common Bash rule.
IFS
Bash uses a variable called IFS, or Internal Field Separator, when performing word splitting.
Its default value includes spaces, tabs and newlines.
You can change it:
IFS=,
Now an unquoted expansion such as:
value="one,two,three"
for item in $value; do
echo "$item"
done
can produce:
one
two
three
Changing IFS globally, however, is often a recipe for confusing code. When you need a different separator, limit its scope:
while IFS= read -r line; do
...
done < file.txt
This is a common pattern for reading lines without letting read strip or split them unexpectedly.
5. Pathname Expansion
After word splitting, Bash can perform pathname expansion, commonly called globbing.
Globbing is simply wildcard matching against filenames.
Suppose the directory contains:
app.log
error.log
notes.txt
Then:
echo *.log
may effectively become:
echo app.log error.log
echo never sees *.log.
Bash expanded it first.
Common patterns include:
* any number of characters
? one character
[abc] one character from the set
An unmatched wildcard
Suppose there are no .log files.
Depending on Bash's settings:
echo *.log
may literally produce:
*.log
That can be surprising in scripts.
Bash provides nullglob:
shopt -s nullglob
Now an unmatched wildcard expands to nothing.
There is also failglob:
shopt -s failglob
which makes an unmatched pattern an error.
These options can be useful in scripts that process file groups.
6. Command Lookup
Eventually Bash has to answer another question:
What exactly does this command name refer to?
Suppose you type:
ls
Bash doesn't blindly search /usr/bin.
There can be aliases, functions, builtins and executables.
Try:
type -a ls
You may discover that there is an alias, a function, or multiple executables with that name.
PATH
For external commands, Bash searches directories listed in:
echo "$PATH"
A typical value might look like:
/usr/local/bin:/usr/bin:/bin
The directories are searched in order.
So if you have:
/usr/local/bin/mytool
/usr/bin/mytool
and both are executable, the first matching one in PATH wins.
This is one reason production scripts should be careful about relying on whatever happens to be first in someone's interactive shell environment.
type vs which
For Bash debugging, I usually reach for:
type -a command
rather than:
which command
type understands shell functions, aliases and builtins.
For example:
type cd
will tell you that cd is a shell builtin.
You can't find that by looking for /usr/bin/cd, because cd needs to change the current shell's directory. An external process couldn't change its parent's working directory.
7. Redirection
Before executing the command, Bash also sets up input and output redirections.
The usual streams are:
0 stdin
1 stdout
2 stderr
These numbers are called file descriptors.
A file descriptor is just a number representing an open input/output stream.
Standard output
command > output.txt
redirects stdout to the file.
Append instead:
command >> output.txt
Standard error
command 2> error.txt
redirects stderr.
You can redirect both:
command > output.txt 2>&1
The important part is the order.
Compare:
command > output.txt 2>&1
with:
command 2>&1 > output.txt
They don't mean the same thing.
In the first command, stdout is sent to the file first, and then stderr is made to point to the same destination.
In the second, stderr is first connected to the original stdout, and only afterwards is stdout redirected to the file.
The order is significant because redirections are processed by the shell.
Here-documents
Bash can provide multi-line input directly:
cat <<EOF
Hello
This is multiple lines
EOF
This is called a here-document.
It's useful for configuration files, SQL, scripts and other blocks of text.
Quoting the delimiter changes expansion:
cat <<'EOF'
$HOME
$(date)
EOF
Here Bash leaves those expressions untouched.
Without the quotes, Bash can expand them.
8. Execution
Now Bash actually starts executing commands.
This is where processes and subshells become important.
A process is a running program. Bash itself is a process, and when it launches an external command, that command normally runs as another process.
Subshells
Parentheses explicitly create a subshell:
name="John"
(
name="Jane"
echo "$name"
)
echo "$name"
Output:
Jane
John
The change happened inside the subshell and didn't modify the parent shell's variable.
Command substitution also runs in a separate shell execution environment:
result=$(some-command)
This matters when you're expecting changes to shell variables to survive.
Pipelines
A pipeline:
producer | consumer
connects the producer's stdout to the consumer's stdin.
Pipeline components may run in separate processes.
This can cause a classic surprise:
count=0
cat file.txt | while read -r line; do
((count++))
done
echo "$count"
You might expect count to contain the number of lines.
But the loop can run in a subshell, leaving the parent shell's count unchanged.
A simple alternative is:
while read -r line; do
((count++))
done < file.txt
Now the loop runs in the current shell.
Background commands
Appending & starts a command asynchronously:
long-running-command &
Bash continues immediately.
The process ID of the most recently started background command is available in:
$!
You can wait for it:
long-running-command &
pid=$!
wait "$pid"
wait returns the exit status of the process it waited for.
This becomes useful when a script launches several independent operations and needs to collect their results later.
9. Exit Status
Commands don't just produce output. They also return an exit status.
By convention:
0 success
non-zero failure
The shell makes this status available through:
$?
For example:
ls /tmp
echo $?
A successful command normally returns 0.
A failed command returns some non-zero value. The exact value depends on the command.
if checks exit status
This is why you can write:
if grep -q "ERROR" application.log; then
echo "Errors found"
fi
if isn't asking Bash whether grep printed anything.
It's checking whether grep returned a successful status.
This is an important Bash habit:
if command; then
...
fi
is perfectly normal.
&& and ||
These operators also use exit status:
mkdir backup && cp report.txt backup/
The copy happens only if mkdir succeeds.
Likewise:
cp report.txt backup/ || echo "Copy failed"
runs the echo when cp returns non-zero.
10. Error Handling
Bash provides a few options that make error handling easier.
The commonly seen combination is:
set -euo pipefail
Each part does something different.
set -e
-e tells Bash to exit when a command fails in certain contexts.
The phrase "exit on any error" is a useful shorthand, but it isn't literally what Bash does.
For example:
if grep -q "ERROR" application.log; then
echo "found"
fi
If grep doesn't find the text, it returns non-zero.
Bash doesn't exit simply because of that.
There are several other contexts where set -e behaves differently, including commands used with &&, ||, while, until, and some pipeline situations.
So:
set -e
doesn't replace deliberate error handling.
set -u
This makes Bash complain about unset variables in many parameter-expansion contexts.
Without it:
echo "$USER_NAME"
can silently produce an empty value if USER_NAME was never set.
With:
set -u
that mistake is much easier to notice.
pipefail
Without pipefail, the exit status of a pipeline normally comes from its last command.
Consider:
false | true
The first command fails, but the last command succeeds.
Without pipefail, the pipeline can therefore appear successful.
With:
set -o pipefail
the pipeline reflects a failure from an earlier command as well.
This is why:
set -euo pipefail
is a reasonable starting point for many Bash scripts.
It still isn't a substitute for understanding the commands being run.
11. Variables and Scope
Bash variables are simpler than variables in most programming languages, but there are a few behaviours worth knowing.
No declared type
This is perfectly valid:
value="hello"
value=123
value="123"
The variable doesn't have a declared type.
Arithmetic contexts are handled separately:
count=10
if (( count > 5 )); then
echo "large"
fi
Function-local variables
Without local:
process() {
result="done"
}
result belongs to the current shell environment.
Prefer:
process() {
local result="done"
}
when the variable is only needed by the function.
Dynamic scoping
Bash functions have an unusual property called dynamic scoping.
In simple terms, a function can see a caller's local variable.
outer() {
local message="hello"
inner
}
inner() {
echo "$message"
}
outer
This prints:
hello
This isn't how local variables normally work in languages such as Java.
It can be useful, but it can also make large Bash programs harder to reason about. Passing values explicitly is usually clearer.
export
export doesn't make a variable global.
It makes the variable available in the environment inherited by child processes:
export APP_ENV=production
./application
The application can read APP_ENV.
The child gets its own copy of the environment.
12. Debugging
When Bash does something unexpected, guessing is usually slower than asking Bash what it thinks is happening.
set -x
set -x
makes Bash print commands as they are executed.
For example:
name="John Smith"
echo "$name"
The trace can show the expanded command.
Turn it off with:
set +x
Be careful with this in production. Tracing can expose passwords, tokens and other secrets.
declare -p
To inspect a variable:
declare -p name
This can tell you how Bash has actually stored it.
It's particularly useful with arrays and exported variables.
type
When a command isn't behaving as expected:
type -a command
is often the first thing worth checking.
It tells you whether you're dealing with an alias, function, builtin or external command.
printf beats echo for debugging
Instead of:
echo "$value"
I often use:
printf '<%s>\n' "$value"
The delimiters make invisible whitespace easier to spot:
<hello >
is obviously different from:
<hello>
This is especially useful when debugging variables containing newlines or spaces.
13. Putting the Processing Order Together
At this point, the individual behaviours start fitting together.
Consider:
files="*.log"
grep ERROR $files > errors.txt 2>&1
There is quite a lot happening here.
Bash first parses the command.
Then it expands:
$files
which produces:
*.log
Because the result is unquoted, word splitting can take place.
Then pathname expansion can turn:
*.log
into something like:
app.log error.log server.log
Bash sets up:
> errors.txt
and:
2>&1
Then it runs:
grep ERROR app.log error.log server.log
The important thing is that grep never received:
$files
or:
*.log
It received the final arguments produced by Bash.
That's the part that tends to get lost when learning shell scripting as a collection of commands.
A More Useful Way to Think About Bash
You don't need to memorise every Bash rule.
When a command behaves strangely, ask what Bash did to it before the command ran.
A rough model is:
Shell input
↓
Parsing
↓
Expansion
↓
Word splitting
↓
Pathname expansion
↓
Redirection setup
↓
Command lookup
↓
Execution
↓
Exit status
Not every command exercises every stage, and the exact details are more nuanced than this diagram suggests.
But it is a useful debugging model.
For example:
"Why did my filename become two arguments?"
Look at quoting and word splitting.
"Why did *.log reach the program literally?"
Look at pathname expansion and nullglob.
"Why did my variable change disappear?"
Look for a subshell or pipeline.
"Why did my error message go somewhere unexpected?"
Look at file descriptors and redirection order.
"Why did this command run instead of the one I expected?"
Check aliases, functions, builtins and PATH with type.
Once you start looking at Bash this way, many of its oddities become much easier to diagnose.