Shell scripts and Slurm

Part IV of the pre-workshop learning session

Author

Affiliation

Jelmer Poelstra

CFAES Bioinformatics Core, Ohio State University

Published

July 1, 2026

Modified

July 10, 2026

1 Introduction

In this part, you will learn:

The basics of shell scripts to (e.g.) run command-line bioinformatics programs
Submitting shell scripts as batch jobs with Slurm

1.1 What are shell scripts and why use them?

Previously, we ran a first bioinformatics program, FastQC, from the command line. However, we ran it interactively: typing the command in the shell and then pressing Enter, waiting for the program to finish before moving on the the next command.

But when you run bioinformatics tools, it is often a much better idea to do so using shell scripts, plain-text files with shell code — because, first of all:

It is a good way to save and organize your code.
You can easily rerun scripts and re-use them in similar contexts.

And at OSC, we can submit scripts as “batch jobs” to the job scheduling program Slurm, which allows us to:

Run scripts remotely without needing to stay connected to the running process: we can submit a script, log out from OSC and shut down our computer, and it will still run.
Run a script many times simultaneously, such as for different files/samples.
Easily perform long-running analyses that take many hours or even multiple days.

2 A first shell script — to run FastQC

2.1 A single-line script

Create your first script, fastqc.sh (shell scripts usually have the extension .sh) as follows:

# Create an empty file with the touch command:
touch fastqc.sh

Find the fastqc.sh file in the VS Code file explorer (you may need to refresh the explorer if you don’t see it immediately) and click on it to open it in the editor pane. Then, copy the following into the script, and save the script (File > Save or Ctrl/Cmd+S):

echo "This script will run FastQC"

Shell scripts mostly contain the same Unix shell code you have become familiar with. As such, the fastqc.sh file with a single echo command constitutes a functional shell script!

One way of running the script is by typing bash followed by the path to the script¹:

bash fastqc.sh

This script will run FastQC

That worked! The script doesn’t yet run FastQC like it “promises” to do, but we will add that functionality.

2.2 Add a shebang line

A so-called “shebang” line is commonly used as the first line of a script to indicate which computer language the script uses. While not always strictly necessary, adding a shebang line to every shell script is good practice, especially when you submit your script with Slurm, as we’ll do later.

For the sake of time, we won’t go into the details of how this weird-looking line is structured, but you can go ahead and add it to the top of your fastqc.sh script, so it now looks like this:

#!/bin/bash

echo "This script will run FastQC"

2.3 Make the script run FastQC

Exercise

Add a line to the end of the fastqc.sh script that runs FastQC on the first FASTQ file (data/SRR31869590_1.fastq.gz) and writes the output to the results/fastqc directory.
Run the script again.

Click for the solution

The full script should now look like this:

#!/bin/bash

echo "This script will run FastQC"

fastqc --outdir results/fastqc data/SRR31869590_1.fastq.gz

Run the script again:

bash fastqc.sh

This script will run FastQC
application/gzip
Started analysis of SRR31869590_1.fastq.gz
Approx 5% complete for SRR31869590_1.fastq.gz
Approx 10% complete for SRR31869590_1.fastq.gz
Approx 15% complete for SRR31869590_1.fastq.gz
# [...output truncated...]

3 Submitting scripts as batch jobs with Slurm

3.1 Batch jobs with Slurm

Automated scheduling software allows many with different requirements to effectively and fairly access compute nodes at supercomputers. OSC uses Slurm (Simple Linux Utility for Resource Management) for this.

Scripts can be submitted with the sbatch command to the Slurm scheduler, which will request and then run a compute job on a compute node.

3.2 The `sbatch` command

First, recall how we’ve run shell scripts so far:

# Don't run this
bash fastqc.sh

The above command ran the script on whatever node you are on, and printed output to the screen. To instead submit the script to the Slurm queue, start by simply replacing bash with sbatch:

sbatch fastqc.sh

srun: error: ERROR: Job invalid: Must specify account for job  
srun: error: Unable to allocate resources: Unspecified error

However, as the above error message –“Must specify account for job”– informs us, you need to indicate which OSC Project (AKA “account”) you want to use for this compute job.

We will do so by adding this to the shell script itself using a type of special comment line. These line(s) should be located at the top of the script, right below the shebang line.

Let’s add the sbatch option to specify the account in the fastqc.sh script, such that the first few lines read:

#!/bin/bash
#SBATCH --account=PAS3454

Now, resubmit the script with sbatch:

sbatch fastqc.sh

Submitted batch job 12275177

This output line means your job was successfully submitted, and you can see the job’s unique identifier that you can use to monitor and manage it.

After submitting a batch job, you immediately get your prompt back. The job will run outside of your immediate view, and you can continue doing other things in the shell while it does, or even log off.

This behavior allows you to submit many jobs at the same time: you don’t have to wait for other jobs to finish!

3.3 Where does the script’s output go?

Above, when you ran bash fastqc.sh, the script’s output was printed to screen — but when you submitted it as a batch job, the only thing printed to screen was Submitted batch job <job-number>. So, for batch jobs, where does this output go?

It ends up in a file called slurm-<job-number>.out (e.g., slurm-12431942.out; since each job number is unique, each file has a different number). We’ll call this type of file a Slurm log file.

If you run ls, you should see a Slurm log file for the job you just submitted:

ls

data  fastqc.sh  results  slurm-12275177.out

Let’s take a look at its contents – you can click on the file in VS Code or use the cat command to print the contents of a file to screen:

# Replace 12275177 with your job number -- or better yet, USE AUTO-COMPLETION!
cat slurm-12275177.out

This script will run FastQC
application/gzip
Started analysis of SRR31869590_1.fastq.gz
Approx 5% complete for SRR31869590_1.fastq.gz
Approx 10% complete for SRR31869590_1.fastq.gz
Approx 15% complete for SRR31869590_1.fastq.gz
# [...output truncated...]

Adding other sbatch options

The --account option is just one of many you can use for a batch job, but is the only required one. Defaults exist for all other options, such as the amount of time (1 hour) and number of cores (1 core).

For example, we could add a few more options to the fastqc.sh script to request:

More time: a maximum of 3 hours of compute time²
More computing power: 4 cores³

#!/bin/bash
#SBATCH --account=PAS3454
#SBATCH --time=3:00:00
#SBATCH --cpus-per-task=4

See this page for more about these and other Slurm options.

3.4 Monitoring batch jobs

It’s important to know how you can monitor your batch jobs, because real-life batch jobs may run for a while, and/or you may submit many jobs at once. Additionally, after you submit a job, it may initially be queued, waiting to get resources allocated to it.

The `squeue` command

You can check the status of your batch jobs using the squeue command — try the following:

# -l gives more verbose output; replace `jelmer` with your OSC username
squeue -l -u jelmer

Wed Jul 08 13:56:33 2026
        JOBID PARTITION     NAME     USER    STATE       TIME TIME_LIMI  NODES NODELIST(REASON)
     23640814 condo-osu ondemand   jelmer  RUNNING       6:34   2:00:00      1 p0133

In the squeue output, following a line with the date & time and a header line, you should see information about a single compute job, as shown above: this is the Interactive App job that runs VS Code — that’s not a batch job, but it is a compute job.

The key pieces of information are:

The STATE column: whether a job is PENDING (queued) or RUNNING (completed jobs will not appear)
The TIME column: for how long has the job has been running
The TIME_LIMIT: for how long is the job allowed to run

Information about each column of the squeue output (Click to expand)

The following pieces of information about each job are listed:

Column	Explanation
`JOBID`	The job ID number
`PARTITION`	The type of queue (usually auto-assigned and not of interest)
`NAME`	The name of the job (by default the name of the script when submitting a script)
`USER`	The username of the person who submitted the job
`STATE`	The job’s state, usually either `PENDING` or `RUNNING` – Finished jobs do not appear
`TIME`	For how long the job has been running (here in “minutes:seconds” format)
`TIME_LIMIT`	The amount of time you reserved for the job (here in “hours:minutes:seconds” format)
`NODES`	The number of nodes reserved for the job
`NODELIST(REASON)`	- When running: the ID of the node on which it is running. - When pending: why the job is pending

A fuller squeue example (Click to expand)

We’ll use another short shell script to practice monitoring and managing batch jobs. First create a new file:

touch sleep.sh

Open the file in the VS Code editor and copy the following into it:

#!/bin/bash
#SBATCH --account=PAS3454

echo "I will sleep for 30 seconds" > sleep.txt
sleep 30s
echo "I'm awake! Successfully finished script sleep.sh"

Now, let’s see a batch job in the squeue listing. Start by submitting the sleep.sh script as a batch job:

sbatch sleep.sh

Submitted batch job 12431945

If you’re quick enough, you may be able to catch the job’s STATE as PENDING before it starts:

squeue -u $USER -l

Wed Jul 08 13:58:57 2026
         JOBID PARTITION     NAME     USER    STATE       TIME TIME_LIMI  NODES NODELIST(REASON)
      12520046 serial-40 sleep.sh   jelmer  PENDING       0:00   1:00:00      1 (None)
      23640814 condo-osu ondemand   jelmer  RUNNING       7:12   2:00:00      1 p0133

But soon enough it should read RUNNING:

squeue -u $USER -l

Wed Jul 08 13:59:59 2026
         JOBID PARTITION     NAME     USER    STATE       TIME TIME_LIMI  NODES NODELIST(REASON)
      12520046 condo-osu sleep.sh   jelmer  RUNNING       0:12   1:00:00      1 p0133
      23640814 condo-osu ondemand   jelmer  RUNNING       8:15   2:00:00      1 p0133

The script should finish after 30 seconds (because your command was sleep 30s), after which the job will immediately disappear from the squeue listing, because only pending and running jobs are shown:

squeue -u $USER -l

Wed Jul 08 14:00:48 2026
         JOBID PARTITION     NAME     USER    STATE       TIME TIME_LIMI  NODES NODELIST(REASON)
      23640814 condo-osu ondemand   jelmer  RUNNING       9:02   2:00:00      1 p0133

Checking a job’s output files

Whenever you’ve ran a script as a batch job, you should make sure it ran successfully. You can do this by checking the output file(s) — you’ll usually have two types of output from a batch job:

A Slurm log file that typically contains logging-type output and potential errors
Output file(s) created by commands in the script (in the case, the FastQC output files)

Footnotes

Bash is the name of the specific Unix shell program we are using, and that will run the script.↩︎
If the script is still running after 3 hours, it will be automatically stopped by Slurm.↩︎
Many bioinformatics tools, including FastQC, can use multiple CPU cores to speed up their computations.↩︎

1 Introduction

1.1 What are shell scripts and why use them?

2 A first shell script — to run FastQC

2.1 A single-line script

2.2 Add a shebang line

2.3 Make the script run FastQC

Exercise

3 Submitting scripts as batch jobs with Slurm

3.1 Batch jobs with Slurm

3.2 The sbatch command

3.3 Where does the script’s output go?

3.4 Monitoring batch jobs

The squeue command

Checking a job’s output files

Footnotes

3.2 The `sbatch` command

The `squeue` command