Configuring your shell
Optional self-study content for week 7
1 Introduction
This page covers how the shell finds the programs you run (the $PATH), and how you can make lasting changes to your shell’s settings using configuration files like ~/.bashrc. Understanding the $PATH helps you make sense of command not found errors, and of which version of a program runs when several are installed. And with a few lines in your ~/.bashrc, you can make your shell more convenient to work in, e.g. with shortcuts for commands you use often and a more informative prompt.
1.1 Setting up
To run the commands on this page yourself, start a VS Code session at https://ondemand.osc.edu in /fs/ess/PAS3493/people/$USER.
2 $PATH: where the shell looks for programs
2.1 What is the $PATH?
$PATH is an environment variable like $HOME and $USER, which you learned about in week 3. It contains a list of directories, and whenever you type the name of a program, the shell searches these directories for an executable file with that name. This is why you can run programs without specifying the full path to their executable.
To see where the executable (binary) for a command is located, you can use the which command:
which git/usr/bin/git
To check which directories are in your $PATH, you can simply echo it:
echo $PATH/apps/xalt/xalt/bin:/opt/mvapich2/intel/19.0/2.3.3/bin:/apps/gnu/8.4.0/bin:/opt/intel/19.0.5/itac_latest/bin:/opt/intel/19.0.5/advisor/bin64:/opt/intel/19.0.5/vtune_amplifier/bin64:/opt/intel/19.0.5/inspector_2019/bin64:/opt/intel/19.0.5/compilers_and_libraries_2019/linux/bin/intel64:/apps/software_usage:/usr/lib64/qt-3.3/bin:/opt/osc/bin:/opt/moab/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/opt/ddn/ime/bin:/opt/puppetlabs/bin
That output is hard to read, because the directories are separated by colons :. Let’s make it a bit more readable by replacing the colons with newlines (tr ":" "\n"), and numbering the lines (nl):
echo $PATH | tr ":" "\n" | nl 1 /apps/xalt/xalt/bin
2 /opt/mvapich2/intel/19.0/2.3.3/bin
3 /apps/gnu/8.4.0/bin
4 /opt/intel/19.0.5/itac_latest/bin
5 /opt/intel/19.0.5/advisor/bin64
6 /opt/intel/19.0.5/vtune_amplifier/bin64
7 /opt/intel/19.0.5/inspector_2019/bin64
8 /opt/intel/19.0.5/compilers_and_libraries_2019/linux/bin/intel64
9 /apps/software_usage
10 /usr/lib64/qt-3.3/bin
11 /opt/osc/bin
12 /opt/moab/bin
13 /usr/local/bin
14 /usr/bin
15 /usr/local/sbin
16 /usr/sbin
17 /opt/ibutils/bin
18 /opt/ddn/ime/bin
19 /opt/puppetlabs/bin
The order of the directories matters: the shell searches them from first to last, and runs the first matching executable that it finds.
$PATH come from?
You never set any of these directories yourself, so where do they come from? They were added by different sources when your shell started:
- Standard Unix directories like
/usr/binand/usr/local/bin(lines 13–16) are part of the default$PATHon any Linux system, and contain basic commands likelsandgit. - System-wide configuration files that OSC has set up (in
/etc/, which run for every user when their shell starts) add OSC-specific directories like/opt/osc/bin(line 11). - Modules that OSC loads by default add the directories at the start (lines 1–8), like those for the Intel compilers. You can see which modules are currently loaded with
module list. - Your own configuration files like
~/.bashrccan add directories as well (more on these below).
2.2 Modules (and Conda) change your $PATH
In week 6, you saw that the fastqc command is not available until you load its module. Now you can better understand what module load does: it adds the directory with FastQC’s executable to the start of your $PATH.
Before loading the module, which fastqc shows that the shell cannot find it:
which fastqc/usr/bin/which: no fastqc in (/users/PAS0471/jelmer/micromamba/condabin:/apps/xalt/xalt/bin:/opt/mvapich2/intel/19.0/2.3.3/bin:/apps/gnu/8.4.0/bin:/opt/intel/19.0.5/itac_latest/bin:/opt/intel/19.0.5/advisor/bin64:/opt/intel/19.0.5/vtune_amplifier/bin64:/opt/intel/19.0.5/inspector_2019/bin64:/opt/intel/19.0.5/compilers_and_libraries_2019/linux/bin/intel64:/apps/software_usage:/usr/lib64/qt-3.3/bin:/opt/osc/bin:/opt/moab/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/opt/ddn/ime/bin:/opt/puppetlabs/bin:/users/PAS0471/jelmer/software/bin/:/users/PAS0471/jelmer/.local/bin:/users/PAS0471/jelmer/software/quarto/quarto-1.10.15/bin)
After loading the module, which fastqc shows the path to the executable:
module load fastqc/0.12.1
which fastqc/apps/spack/0.21/cardinal/linux-rhel9-sapphirerapids/fastqc/gcc/11.3.1/0.12.1-ct5vuye/bin/fastqc
(Activating a Conda environment works the same way.)
Say you have your own, older fastqc in ~/software/bin, and your ~/.bashrc adds that dir to the start of your $PATH. If you then run module load fastqc/0.12.1, which fastqc will run?
The module’s version. Your ~/.bashrc added your dir to the start of the $PATH when the shell started, but module load runs later and adds its own dir in front of that. When in doubt, which fastqc tells you which one will run.
2.3 Changing your $PATH
You can add a directory to your $PATH by assigning a new value to the variable that consists of the old value ($PATH), a colon, and the new directory:
# Add a dir to the _end_ of the $PATH:
export PATH="$PATH:$HOME/software/bin"
# Or add a dir to the _start_ of the $PATH:
export PATH="$HOME/software/bin:$PATH"Like any variable you set in the shell, this change only affects your current shell session. To make it permanent, you’ll need to add the line to a configuration file — see below.
exportmakes a variable available to programs that you run from your shell.$PATHis already exported, soexportis not strictly needed here, but it is customary to include it.We use
$HOMErather than~because the shell does not expand~inside double quotes.Don’t forget the
$PATHpart! If you were to runPATH="$HOME/software/bin", the shell would no longer find basic commands likels. If this happens, simply open a new terminal to start over.
In w7a, you saw that a script in your working dir needs to be run as ./script.sh when you don’t use bash script.sh. If you put scripts with execute permissions in a directory that is in your $PATH, you can run them from anywhere just by typing their name, e.g.:
mkdir -p ~/software/bin
cp scripts/fastqc.sh ~/software/bin
chmod +x ~/software/bin/fastqc.sh
export PATH="$PATH:$HOME/software/bin"
# You can now run the script without specifying its path:
fastqc.sh ../garrigos/fastq/S63_R1.fastq.gz results/fastqcFor scripts that belong to a specific research project, though, it is better to keep them within the project dir and run them from there, so that the project is self-contained and others can rerun your analysis.
3 Shell configuration files
3.1 The ~/.bashrc file
Your Bash shell is configured by hidden text files in your Home dir, the most important of which is ~/.bashrc. The commands in this file are run every time you start a new shell, so it is the place to store settings that you want to keep.
~/.bash_profile? (Click to expand)
Strictly speaking, Bash reads different configuration files depending on the type of shell:
| File | Read by | Example |
|---|---|---|
~/.bash_profile |
Login shells | The shell you get when you first log in. |
~/.bashrc |
Non-login interactive shells | Shells you start when you’re already logged in, e.g. by opening another terminal. |
To avoid having to deal with this odd distinction, it is common to put all your settings in ~/.bashrc, and to have ~/.bash_profile run ~/.bashrc. At OSC, this is set up by default:
cat ~/.bash_profile# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
This code checks whether ~/.bashrc exists (-f), and if so, runs it (. is a shorthand for the source command, which is explained below). So, you only need to edit your ~/.bashrc.
3.2 What to put in your ~/.bashrc
To see what’s currently in your ~/.bashrc, use cat or open the file in VS Code:
cat ~/.bashrc# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
# Path
PATH=/apps/xalt/xalt/bin:/opt/mvapich2/intel/19.0/2.3.3/bin:/apps/gnu/8.4.0/bin:/opt/intel/19.0.5/itac_latest/bin:/opt/intel/19.0.5/advisor/bin64:/opt/intel/19.0.5/vtune_amplifier/bin64:/opt/intel/19.0.5/inspector_2019/bin64:/opt/intel/19.0.5/compilers_and_libraries_2019/linux/bin/intel64:/apps/software_usage:/usr/lib64/qt-3.3/bin:/opt/osc/bin:/opt/moab/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/opt/ddn/ime/bin:/opt/puppetlabs/bin
Some things that you can store in this file:
- Changes to your
$PATH, like the one in the previous section. - Other environment variables, like the
$APPTAINER_CACHEDIRvariable from week 6. - Aliases: your own shortcuts for commands — see below.
- The look of your prompt.
To make a change, open the file in VS Code (e.g. with File > Open File... and typing ~/.bashrc), and add a line at the end. For example, to permanently add a directory to your $PATH:
~/.bashrc (excerpt)
# Add my own software dir to the $PATH:
export PATH="$PATH:$HOME/software/bin"Changes to your ~/.bashrc take effect in new shells, because the file is read when the shell starts. To apply changes you just made to your current shell as well, run:
source ~/.bashrcsource command? (Click to expand)
~/.bashrc is really a shell script, so why not run it with bash ~/.bashrc instead of source ~/.bashrc?
bash ~/.bashrc would run the file in a new shell, and any variables or aliases that it sets would disappear as soon as that shell exits. In contrast, source runs the commands in your current shell, so that its changes stick.
~/.bashrc with echo instead (Click to expand)
You can also add a line to your ~/.bashrc without opening the file, by appending it with echo and >>:
echo 'export PATH="$PATH:$HOME/software/bin"' >> ~/.bashrcTwo things to watch out for:
Use
>>to append, and not>, which would overwrite the entire file!Use single quotes: with double quotes, the shell would replace
$PATHwith its current value before the line is written to the file. Single quotes make sure that the line is written exactly as shown, so that$PATHis expanded each time the file is read instead.
~/.bashrc (Click to expand)
Make a backup before making changes:
cp ~/.bashrc ~/.bashrc_backupA mistake in your
~/.bashrccan cause errors every time you open a shell. If that happens, fix or remove the problematic line, or restore your backup. If your shell won’t even start anymore, you can still edit the file through the OnDemandFilesmenu (checkShow Dotfilesto see hidden files).Be careful with loading software in your
~/.bashrc, e.g. withmodule loadorconda activate. This can be convenient for software that you use all the time, but that software will then be loaded in every shell, which can cause conflicts that are hard to track down. Also, make sure that your scripts don’t rely on this: they should still load the software they need themselves, so that they also work for others.Don’t add commands that print output, like
echo. This can break tools that transfer files over SSH, likescp,sftp, andrsync.
4 Aliases
An alias is a custom shortcut for a command. Aliases save typing for commands you run often, especially ones with options you always use, and can help when you have trouble remembering long or complex commands.
You create one with the alias command, followed by the alias name, an equals sign, and the full command in quotes:
alias l='ls -lh'
ltotal 32K
drwxrwx---+ 5 jelmer PAS0471 4.0K Oct 1 12:37 GA3
drwxrwx---+ 6 jelmer PAS0471 4.0K Sep 14 14:41 garrigos
drwxr-x---+ 3 jelmer PAS0471 4.0K Sep 11 15:04 week02
drwxr-x---+ 2 jelmer PAS0471 4.0K Sep 11 15:32 week03
drwxrwx---+ 6 jelmer PAS0471 4.0K Sep 25 15:20 week04
drwxrwx---+ 4 jelmer PAS0471 4.0K Sep 28 15:44 week05
drwxrwx---+ 2 jelmer PAS0471 4.0K Sep 30 14:14 week06
drwxrwx---+ 4 jelmer PAS0471 4.0K Oct 1 12:09 week07
From now on, typing l will run ls -lh.
You can also add further options and arguments when you use an alias:
# Add the alias
alias ae='apptainer exec'
# This will now do the same as `apptainer exec <image> <command>`:
container="oras://community.wave.seqera.io/library/fastqc:0.12.1--104d26ddd9519960"
ae "$container" fastqc --versionINFO: Downloading oras image
384.0MiB / 384.0MiB [=========================================================================================] 100 % 37.5 MiB/s 0s
INFO: gocryptfs not found, will not be able to use gocryptfs
FastQC v0.12.1
Or:
# Add the alias
alias gcm='git commit -m'
# This will now do the same as `git commit -m "Add new script"`:
gcm "Add new script"To see all aliases that are currently defined, run alias without arguments. OSC has defined a bunch of aliases for you by default, mostly related to coloring the output of commands like ls and grep:
aliasalias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'
alias grep='grep --color=auto'
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto'
alias mc='. /usr/libexec/mc/mc-wrapper.sh'
alias xzegrep='xzegrep --color=auto'
alias xzfgrep='xzfgrep --color=auto'
alias xzgrep='xzgrep --color=auto'
alias zegrep='zegrep --color=auto'
alias zfgrep='zfgrep --color=auto'
alias zgrep='zgrep --color=auto'
As with changes to your $PATH, aliases only last for the current shell session, unless you add them to your ~/.bashrc:
~/.bashrc (excerpt)
alias l='ls -lh'
You add alias l='ls -lh' to your ~/.bashrc and save the file. Then, in a terminal that was already open, you type l. What happens?
You’ll get bash: l: command not found (unless you had already typed the alias command in that shell). The ~/.bashrc file is only read when a shell starts, so your already-open shell doesn’t know about the new alias. Run source ~/.bashrc or open a new terminal.
Aliases are a convenience for working interactively. They are not available in scripts by default, and even if they were, a script that relies on your personal aliases would not work for anyone else. So, always write out the full commands in scripts.
5 Customizing your prompt
The look of your prompt is controlled by the variable $PS1. Let’s see what it contains by default at OSC:
echo "$PS1"[\u@\h \W]\$
The backslash codes in there are replaced by the shell each time the prompt is shown. Here are some useful ones:
| Code | Meaning |
|---|---|
\u |
Your user name |
\h |
The name of the node you are on |
\W |
The name of the working dir (without the rest of its path) |
\w |
The full path to the working dir (with ~ for your Home dir) |
\t |
The current time |
\n |
A new line |
\$ |
A $ sign |
For example, a potentially useful change is to show the full path to your working dir, so you always know exactly where you are:
PS1='[\u@\h \w]\$ '[jelmer@p0362 /fs/ess/PAS3493/people/jelmer]$
With a long path, there won’t be much room left for your command on the same line. You can fix that with \n, so you type your commands on the next line:
PS1='[\u@\h \w]\n\$ '[jelmer@p0362 /fs/ess/PAS3493/people/jelmer]
$
You can also add color, which makes it easier to spot where the output of one command ends and the next command starts. For example, to make everything except the $ bold and blue:
PS1='\[\e[1;34m\][\u@\h \w]\[\e[0m\]\n\$ '[jelmer@p0362 /fs/ess/PAS3493/people/jelmer]
$
Here, \[\e[1;34m\] switches to bold (1) blue (34) text, and \[\e[0m\] switches back to the default. Other color codes include 31 (red), 32 (green), 33 (yellow), and 35 (magenta). The \[ and \] around each color code tell the shell that these characters take up no space on screen. If you leave them out, the prompt still looks fine, but long commands will wrap onto the next line in strange ways.
Finally, you may not care much about seeing your user name and node name all the time. If so, you can leave those out and only show the full path to your working dir. In the example below, the path is bold and blue (1;34), the $ is bold and magenta (1;35), and the brackets are left in the default color:
PS1='[\[\e[1;34m\]\w\[\e[0m\]]\[\e[1;35m\]\$\[\e[0m\] '[/fs/ess/PAS3493/people/jelmer]$
Like aliases, a change to $PS1 only lasts for the current shell session. This makes it easy to experiment: if you don’t like the result, just open a new shell. Once you’re happy, add the line to your ~/.bashrc:
~/.bashrc (excerpt)
PS1='\[\e[1;34m\][\u@\h \w]\[\e[0m\]\n\$ 'Inside single quotes, nothing is interpreted by the shell, so $PS1 stores exactly what you typed, and the shell fills it in anew each time the prompt is shown. This matters for anything that should be updated every time. For example, the variable $PWD contains the path to your working dir:
- With
PS1='$PWD \$ ', the prompt always shows your current working dir. - With
PS1="$PWD \$ ",$PWDis replaced by its value right away, so the prompt keeps showing the dir you were in when you set it, even after youcdelsewhere.
(This is also why you should use single quotes when adding a line to your ~/.bashrc with echo, as shown above.)
The quotes also preserve the space after \$, which separates the prompt from the command you type.