Working with files in the Unix shell II:
Viewing, summarizing, and manipulating files

Week 4 – Lecture C

Author
Affiliations

Jelmer Poelstra

CFAES Bioinformatics Core, Ohio State University

Dept. of Plant Pathology, Ohio State University

Presented on

September 16, 2026

Page last modified

September 16, 2026


1 Introduction

1.1 Context & overview

The Unix shell is great for many tasks other than the file browser-like ones you learned about in the previous lecture. One of those is performing basic viewing, querying, and editing operations on text files.

Recall from the Project File Organization lecture that we’ll almost exclusively use “plain-text” files in this course. For example:

  • Sequence and annotation file formats such as FASTA, FASTQ, and GTF
  • Spreadsheet-like/tabular files stored as TSV/CSV
  • Scripts and code notebooks
  • Documentation files such as Markdown

The commands we’ll discuss here can be used with plain-text files but not with “binary” formats like Excel or Word files. (The one exception is compressed plain-text files, like our .fastq.gz files: as you’ll see, some tools can read those directly.)

1.2 Learning goals

In this lecture, you will learn to use the Unix shell to:

  • View the contents of text files in various ways
  • Use redirection and the pipe to flexibly process the output of commands
  • Search within, manipulate, and extract information from text files

1.3 Getting ready

Navigate back to your top-level personal dir from week04/sandbox:

cd ../..
pwd
/fs/ess/PAS3493/people/jelmer

(Or use cd /fs/ess/PAS3493/people/$USER if you’ve gotten lost.)

  1. Log in to OSC’s OnDemand portal at https://ondemand.osc.edu.

  2. In the blue top bar, select Interactive Apps and near the bottom, click Code Server.

  3. Fill out the form as follows:

    • Cluster: pitzer
    • Account: PAS3493
    • Number of hours: 2
    • Working Directory: /fs/ess/PAS3493/people/<user> (replace <user> with your user name)
    • App Code Server version: 4.129.0

  4. Click Launch, and in the next page, click the Connect to VS Code button once it appears.

  5. In VS Code, open a terminal by either:

    • Clicking → Terminal → New Terminal
    • Using the keyboard shortcut Ctrl+`

  6. Check that you are in /fs/ess/PAS3493/people/$USER by typing pwd in the terminal.

    Recall that $USER is a variable that represents your username. If you’re not in /fs/ess/PAS3493/people/$USER, it may be listed under Recents in the Get Started document — if so, click on that entry. Otherwise, click File > Open Folder and type/select that path.

  • Optional: create and open a Markdown file for notes:

    # After this command, click on the file name in the file explorer to open it:
touch week04/lectureC.md

  • Most of today’s commands use files from the Garrigós et al. (2025) dataset. For convenience, copy the handful of files you’ll need into the sandbox dir you created in the previous lecture:

    # (Recall from the previous lecture: the '*' is a wildcard, so the second
#  command copies both the _R1 and the _R2 file for this sample.)
cp garrigos/meta/metadata.tsv week04/sandbox/
cp garrigos/fastq/S63_R*.fastq.gz week04/sandbox/
cp garrigos/ref/annot.gtf.gz week04/sandbox/

    Copying data files like this is only for convenience while practicing. In a real project, you would instead organize your data in a dedicated data dir inside the project, as discussed in the Project File Organization lecture.

  • Move into the sandbox dir:

    cd week04/sandbox

    Check with ls that these four files (including two FASTQ files) are now in your working dir. (You will likely also see leftover dirs and files from the previous lecture — that’s fine, just ignore those.)

2 Viewing the contents of text files

Several commands can view all or part of one or more text files, and we’ll discuss the most common ones below.

2.1 cat

The cat command prints the entire contents of one or more files to screen:

cat metadata.tsv
sample_id       time    treatment
S82     10dpi   cathemerium
S75     10dpi   cathemerium
S79     10dpi   cathemerium
S83     10dpi   cathemerium
S78     10dpi   control
S84     10dpi   control
S77     10dpi   control
S81     10dpi   control
S76     10dpi   relictum
S80     10dpi   relictum
S85     10dpi   relictum
S86     10dpi   relictum
S64     24hpi   cathemerium
S67     24hpi   cathemerium
S70     24hpi   cathemerium
S66     24hpi   control
S69     24hpi   control
S63     24hpi   control
S71     24hpi   relictum
S74     24hpi   relictum
S65     24hpi   relictum
S68     24hpi   relictum

For reference, let’s also open this file in the editor. Either:

  • Find it in the file explorer in the sidebar and click on it, or
  • Hover over the file name in the terminal and press Cmd (Mac) or Ctrl (Windows/Linux), and you should see it become underlined and clickable. Click it to open the file in the editor.

Especially when working with omics data, files can be huge, and this is one reason we use commands instead of a text editor to work with them. But cat is not always the best tool for viewing large files either: printing a whole large file can take a while, and makes it hard to see the beginning of the output. We’ll explore some other options below.

2.2 head and tail

The twin commands head and tail print only the first (head) or last (tail) lines of a file. By default, they print 10 lines:

head metadata.tsv
sample_id       time    treatment
S82     10dpi   cathemerium
S75     10dpi   cathemerium
S79     10dpi   cathemerium
S83     10dpi   cathemerium
S78     10dpi   control
S84     10dpi   control
S77     10dpi   control
S81     10dpi   control
S76     10dpi   relictum

Use the -n option to specify the number of lines to print:

head -n 3 metadata.tsv
sample_id       time    treatment
S82     10dpi   cathemerium
S75     10dpi   cathemerium
NoteOptions that take an argument (🤯)

This is the first option we’ve seen that needs a value: unlike the ON/OFF flags you saw so far, it has to be followed by a number.

  • In -n 3, 3 is an argument of the option -n (cf. metadata.tsv is an argument of the command head).
  • You don’t need a space between the option and its argument: head -n3 metadata.tsv works the same as head -n 3 metadata.tsv.

Let’s try to take a peek inside a FASTQ file. To print the first eight lines (two reads), use -n 8 with head:

head -n 8 S63_R1.fastq.gz
�
Խے�8�E��_1f�"�QD�J��D�fs{����Yk����d��*��
|��x���l޴�j�N������?������ٔ�bUs�Ng�Ǭ���i;_��������������|<�v����3��������|���ۧ��3ĐHyƕ�bIΟD�%����Sr#~��7��ν��1y�Ai,4
w\]"b�#Q����8��+[e�3d�4H���̒�l�9LVMX��U*�M����_?���\["��7�s\<_���:�$���N��v�}^����sw�|�n;<�<�oP����
i��k��q�ְ(G�ϫ��L�^��=��<���K��j�_/�[ۭV�ns:��U��G�z�ݎ�j����&��~�F��٤ZN�'��r2z}�f\#��:�9$�����H�݂�"�@M����H�C�
�0�pp���1�O��I�H�P됄�.Ȣe��Q�>���
�'�;@D8���#��St�7k�g��|�A䉻���_���d�_c������a\�|�_�mn�]�9N������l�٢ZN�c�9u�����n��n�`��
"gͺ�
    ���H�?2@�FC�S$n���Ԓh�       nԙj��望��f      �?N@�CzUlT�&�h�Pt!�r|��9~)���e�A�77�h{��~��     ��
# [...output truncated...]
Ouch! 😳 What went wrong here? (Click to see the answer) We were presented with the contents of the compressed file, which isn’t human-readable.

To get around the problem you just encountered with head, you might be inclined to decompress these files (e.g. with the gunzip command, which you’ll use in next week’s assignment). However, at least when it comes to FASTQ files, it is better to keep them compressed:

  • Uncompressed files take up much more disk storage space than compressed ones.
  • Almost any bioinformatics program accepts compressed FASTQ files.
  • You can view these files in compressed form, as shown below with less. Additionally, several commands including cat have a counterpart for compressed files (zcat in the case of cat).

Other genomics files, like assembly FASTA files and annotation GTF/GFF files, are often also compressed when you download them. These types of files, though, are more commonly decompressed before usage — e.g. because they don’t take up nearly as much space as a set of FASTQ files.

2.3 less to page through files

The previous commands printed file contents to the screen and then returned your shell prompt. The less command is different, as it opens the file in a way that allows you to browse through it nicely. And it will display gzip-compressed files in human-readable form!1 🥳

less S63_R1.fastq.gz
@ERR10802863.8435456 8435456 length=74
CAACGAATACATCATGTTTGCGAAACTACTCCTCCTCGCCTTGGTGGGGATCAGTACTGCGTACCAGTATGAGT
+
AAAAAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
@ERR10802863.27637245 27637245 length=74
GCCACACTTTTGAAGAACAGCGTCATTGTTCTTAATTTTGTCGGCAACGCCTGCACGAGCCTTCCACGTAAGTT
+
AAAAAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEAEE<EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
@ERR10802863.10009244 10009244 length=73
CTCGGCGTTAACTTCATCACGCAGATCATTCCGTTCCAGCAGCTGAAGCAAGACTACCGTCAGTACGAGATGA
+
AAAAAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE

When you’re inside the less pager, you can move around in the file in several ways — for example:

  • By scrolling with your mouse.
  • With up ↑ and down ↓ arrows: move line-by-line.
  • With Space (down) and b (back up): move a full page at a time.
  • By pressing G to go to the end of the file, and g to go (back) to the top.

To exit/quit less and get your shell prompt back, press q. (When you do that, no output from the file will remain on the screen.)

Exercise 1: less

Open the FASTQ file with less S63_R1.fastq.gz, then:

  1. Move around a bit using the keys listed above.
  2. Among the first reads, you should already spot some unusual-looking ones. What is unusual about them?
  3. Press q to quit less and get your prompt back.
Click for the solution to question b. Some reads are much shorter than the others, and only consist of Ns, i.e. uncalled bases. These reads therefore don’t contain any useful information, and are usually filtered out before downstream analysis.
  1. Bonus: Another useful feature of less is the ability to search through a file. Press /, type a search pattern, and press Enter to jump to the first match (then n for the next and N for the previous match). Open the file back up with less, then use this feature to find more reads like the ones you found in part b. What pattern did you search for?
Click for the solution.

A number of reads are much shorter than the others, and only consist of Ns, i.e. uncalled bases. For example:

@ERR10802863.11918285 11918285 length=35
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
+
###################################
To jump to more of these reads, search for a stretch of Ns: type /NNNNN, press Enter, and then press n to go to each next match.

When lines are too long to fit on your computer screen, less will by default “wrap” them onto the next line on the screen. That’s convenient for reading, but can be confusing for tabular formats and formats like FASTQ, because it means that one line on the screen no longer corresponds to one line in the file.

With the -S option to less, lines will not be wrapped but will “run out of the screen” on the right-hand side (press the right arrow key → to see that part):

less -S S63_R1.fastq.gz
# [output not shown]

2.4 wc -l to count lines

(This command doesn’t display file contents, but you’ll need it in the next section.)

The wc command by default counts the number of lines, words, and bytes in its input — but is most commonly used to only count lines, with the -l option:

wc -l metadata.tsv
23 metadata.tsv

This simple operation can be surprisingly useful: the number of lines reflects the number of entries in many file types. For example, recall that FASTQ files have four lines per read.

3 Redirection and the pipe

3.1 Redirection

The regular output that commands print to the screen (like a list of files by ls, or a number of lines by wc -l) is technically called “standard out” or in short “stdout”. Sometimes, you may want to do something else with this output, like storing it in a file. Luckily, this is easy to do with something called “redirection”.

First, a reminder of what echo does without redirection — it simply prints the text we provide to the screen:

echo "plum"
plum

Now, redirect echo’s standard out to a new file fruits.txt using the > operator:

echo "plum" > fruits.txt

No output was printed to the screen — it went into the file instead:

cat fruits.txt
plum

The > operator will redirect output to a file with the following rules:

  • If the file doesn’t exist, it will be created.
  • If the file does exist, any contents will be overwritten.

Redirect another line into that same file:

echo "fig" > fruits.txt
cat fruits.txt
fig

That may not have been what we intended! As explained above, the earlier file content was overwritten. With >>, however, output is appended (added) to a file:

echo "banana" >> fruits.txt
cat fruits.txt
fig
banana

  • Never redirect output back to the command’s own input file. Because > empties an existing file before the command even runs, you shouldn’t do this even if you want to edit a file in place. For example, the command head metadata.tsv > metadata.tsv would leave you with an empty metadata.tsv. (More on this in the self-study section at the end.)

  • Error messages are not redirected by >. Commands print error messages to a separate “stream” called standard error (stderr). Because > only redirects standard out, errors are still shown on the screen:

    # [We won't run this in class, but you can try it out yourself.]
ls fruits.txt nonexistent.txt > ls-out.txt
    ls: cannot access 'nonexistent.txt': No such file or directory
    cat ls-out.txt
    fruits.txt

    That is usually a good thing, since you’ll notice errors right away. You’ll run into standard error again later in the course.

3.2 The pipe

Say that you want to count the number of FASTQ files in the garrigos/fastq dir. You could do that as follows:

# [Hypothetical example, don't run this]
# First, you'd redirect 'ls' output to a file:
ls ../../garrigos/fastq > fastq-files.txt

# Then you'd count the number of lines, i.e. the number of files in that dir:
wc -l fastq-files.txt
44 fastq-files.txt
# Finally, you'd remove the file you no longer need:
rm fastq-files.txt

(Wait, doesn’t ls print several entries per line? Only when printing straight to the screen: when its output is redirected, ls prints one entry per line, so this count is correct.)

Above, we essentially wrote an “intermediate result” (fastq-files.txt) to file, which we later removed. What if you could avoid having to write this intermediate file? You can do so with a so-called pipe:

ls ../../garrigos/fastq | wc -l
44

With the pipe, the output of the command on the left-hand side (a file listing, in this case) is redirected into the command on the right-hand side (wc in this case). Like many others, the wc command will gladly accept input that way (i.e., via standard in, or stdin) instead of via a file name argument.

Compare this output with the earlier wc -l output. What’s different, and why? (Click to see the answer) The output no longer includes a file name: wc now reads from standard in rather than from a file, so there is no file name to report.

To recap, pipes are useful because they avoid having to write/read intermediate files — this saves typing, makes the operation quicker, and reduces file clutter. In the example above, you don’t need to make a fastq-files.txt file to count the number of files. We’ll see pipes in action a bunch more in the next section.

Exercise 2: the pipe

  1. What do you think the command below does? Could you divide the resulting count by 4 to get the total number of reads for sample S63? Why or why not?

    cat S63_R1.fastq.gz S63_R2.fastq.gz | wc -l

    Click for the solution.

    Like many shell commands, cat accepts multiple arguments, i.e. it can operate on multiple files. In this case, cat would simply print the contents of the two files back-to-back (concatenate them!). Therefore, wc -l will count the total number of lines across the two files.

    cat S63_R1.fastq.gz S63_R2.fastq.gz | wc -l
    193612

    However, you can’t get the total number of reads for sample S63 by dividing this count by 4 (recall: one FASTQ read covers 4 lines), since the files are compressed: compressed and uncompressed line counts for a file are not the same.

  2. A classmate wants to pipe the output of ls into wc, but accidentally types > instead of |:

    ls > wc

    What do you think will happen? Then run it, and check what happened, starting with ls.

    Click for the solution.

    Nothing is printed, and no counting happens. Instead, the shell treats the word right after > as the name of the output file, so it creates a file called wc that contains the file listing (which even includes wc itself!). Remove the file with rm wc.

4 Unix data tools

We’ll now turn to some commands that may be described as “Unix data tools”. These are especially good for relatively simple data processing and summarizing steps, and are excellent at dealing with very large files such as genomics files. We will cover the following commands:

  • grep to search for text in files
  • cut to select one or more columns from tabular data
  • sort to sort lines, or tabular data by column
  • uniq to remove duplicates

But we’ll start by revisiting one of the example data files, and discussing tabular plain-text files.

4.1 Tabular plain-text files and file extensions

Let’s take another look at the first few lines of the now-familiar metadata.tsv file:

head -n 3 metadata.tsv
sample_id       time    treatment
S82     10dpi   cathemerium
S75     10dpi   cathemerium

“Tabular” files like this one contain data that is arranged in a rows-and-columns format, i.e. as in a table or an Excel spreadsheet. Because plain-text files do not have an intrinsic way to define columns, certain characters are used as column delimiters in plain-text tabular files. Most commonly, these are:

  • A Tab (Tab), and such files are often stored with a .tsv extension for Tab-Separated Values (“TSV file”).
  • A comma (,), and such files are often stored with a .csv extension for Comma-Separated Values (“CSV file”).

From the head output above, you can’t tell whether the columns are separated by Tabs or by a bunch of spaces. The -A option of cat makes normally invisible characters visible: Tabs are shown as ^I, and the end of each line as $:

cat -A metadata.tsv | head -n 3
sample_id^Itime^Itreatment$
S82^I10dpi^Icathemerium$
S75^I10dpi^Icathemerium$

So, the columns in this file are separated by single Tab characters.

cat -A is also useful for spotting Windows-style line endings, which are common in files created on Windows computers (e.g., exported from Excel). These show up as ^M$ instead of $ and can make commands behave unexpectedly.

WarningPlain-text file extensions are flexible and mostly for human readability

All plain-text files are fundamentally the same, i.e. “just plain-text files”. Therefore, changing the file extension does not change the file itself. Instead, different file extensions are used primarily to make clear to humans, and to some extent to the computer, what the file contains.

  • This applies to all types of plain-text files, like .txt and .tsv, but also genomics files like .fastq and script files like .R.
  • The same is true for compressed files: removing .gz from a file name (e.g., renaming annot.gtf.gz to annot.gtf) does not decompress the file — it only makes its name misleading.

4.2 grep to print lines that match a pattern

The grep command is an extremely useful command that searches for specific text (or text patterns) in a file. By default, it prints each line that contains a “match” in full.

Its basic syntax is grep "<search-pattern>" <file-path>. For example, to print all lines from metadata.tsv that contain “relictum”:

grep "relictum" metadata.tsv
S76     10dpi   relictum
S80     10dpi   relictum
S85     10dpi   relictum
S86     10dpi   relictum
S71     24hpi   relictum
S74     24hpi   relictum
S65     24hpi   relictum
S68     24hpi   relictum

Two notes about the above command and output:

  1. While not always necessary, it’s good practice to use quotes ("...") around the search pattern like above2.
  2. In most terminals, including at OSC, grep will highlight the matching part of a line.

Instead of printing matching lines, you can also count them with the -c option. For example, how many control samples are in the dataset?

grep -c "control" metadata.tsv
7

Exercise 3: grep

  1. What output do you expect from the command below? Check your answer by executing the command.

    grep -c "contro" metadata.tsv

    Click for the solution.

    The command gives the same output as the previous example: the shorter pattern contro still matches all the control lines:

    grep -c "contro" metadata.tsv
    7

  2. Based on the output of the above command, how does grep’s matching behavior seem to differ from globbing with the shell wildcard *?

    Click for the solution.

    While the initial examples also implied this, the above example should make it abundantly clear that grep does not need to match entire lines or words, etc.

    This behavior is very different from globbing with the * wildcard, where the pattern has to match the entire file name!

Self-study exercise: more grep

  1. A classmate wants to count the 24hpi samples. Reasoning that 24hpi contains a 4 and 10dpi does not, they run:

    grep -c "4" metadata.tsv

    Will this give them the correct count? Why or why not?

    Click for the solution.

    No: it reports 11, but there are only 10 24hpi samples. grep matches the pattern anywhere in a line, so it also matches the 10dpi sample S84. With tabular files, use a pattern specific enough to only match the column you have in mind, like "24hpi".

  2. Count the number of relictum samples at the 24hpi time point with a single line of code.

    Click here for a hint.

    You can pipe the output of one grep command into another grep command!

    Click for the solution.

    You can pipe the output of one grep command into another: the first selects the 24hpi lines, and the second counts the relictum lines among those.

    grep "24hpi" metadata.tsv | grep -c "relictum"
    4
NoteAdditional useful grep options

grep has many other useful options, such as:

  • -v to invert the matching, i.e. print all lines that do not match the pattern.
  • -i to ignore case (uppercase vs. lowercase) when searching.
  • -n to print the line number for each matching line.
  • -A <n> and -B <n> to print n lines after and before each match.
  • -w to make a pattern match whole “words” only.
  • -r to search files in directories recursively3.

4.3 Operating on compressed files

The zcat command is the counterpart of cat for gzip-compressed files: it prints their contents in human-readable form. By piping zcat’s output into grep or into any other command, you can therefore use all the tools you’ve learned about even with compressed files — let’s try that with the Culex pipiens genome annotation GTF file:

# Search for cecropin-related lines in the GTF file:
zcat annot.gtf.gz | grep "cecropin" | head -n 3
NC_068937.1     Gnomon  gene    43201550        43201918        .       +       .       gene_id "LOC120426347"; transcript_id ""; db_xref "GeneID:120426347"; description "cecropin-A2-like"; gbkey "Gene"; gene "LOC120426347"; gene_biotype "protein_coding";
NC_068937.1     Gnomon  transcript      43201550        43201918        .       +       .       gene_id "LOC120426347"; transcript_id "XM_039591134.2"; db_xref "GeneID:120426347"; experiment "COORDINATES: polyA evidence [ECO:0006239]"; gbkey "mRNA"; gene "LOC120426347"; model_evidence "Supporting evidence includes similarity to: 26 ESTs, 18 Proteins"; product "cecropin-A2-like"; transcript_biotype "mRNA";
NC_068937.1     Gnomon  exon    43201550        43201708        .       +       .       gene_id "LOC120426347"; transcript_id "XM_039591134.2"; db_xref "GeneID:120426347"; experiment "COORDINATES: polyA evidence [ECO:0006239]"; gene "LOC120426347"; model_evidence "Supporting evidence includes similarity to: 26 ESTs, 18 Proteins"; product "cecropin-A2-like"; transcript_biotype "mRNA"; exon_number "1";

In this week’s exercises and next week’s assignment (GA2), you’ll further explore this GTF file.

For very large files like this one: if you forget to pipe zcat (or cat) output into another command and your screen floods with text, press Ctrl+C to stop it.

4.4 Combining cut, sort, and uniq to create a list

Say you want an alphabetically sorted list of distinct treatments in metadata.tsv (that’s trivial for this file, because it’s small enough to see that information at a glance — but the same approach works on a huge GTF file). To do this, you’ll need to learn about three new commands:

  • cut to select, or “cut out”, one or more columns from a tabular file.
  • sort to sort/order/arrange rows — by default alphabetically.
  • uniq to remove duplicates (i.e., keep all distinct/unique entries) from a sorted file/list.

We’ll build up a small “pipeline” to do this step by step, piping the output into head at every step. Piping output into head is a handy way to quickly check whether a command works, without printing a large number of lines.

First, we need to get rid of the header line — otherwise, the column name treatment would end up in our list as if it were one of the treatments. We can do this with a neat tail trick that tells it to start at a specific line using -n +<starting-line>:

# '-n +2' tells tail to start at line 2:
tail -n +2 metadata.tsv | head
S82     10dpi   cathemerium
S75     10dpi   cathemerium
S79     10dpi   cathemerium
S83     10dpi   cathemerium
S78     10dpi   control
S84     10dpi   control
S77     10dpi   control
S81     10dpi   control
S76     10dpi   relictum
S80     10dpi   relictum

Don’t forget to make use of the command history here! Press the up arrow key ↑ to recall the previous command, then edit it to create the next command.

Second, select the column of interest with cut, using its -f (“field”) option to specify the desired column(s):

tail -n +2 metadata.tsv | cut -f 3 | head
cathemerium
cathemerium
cathemerium
cathemerium
control
control
control
control
relictum
relictum

To select multiple columns, use a comma-separated list, or a range with -:

cut -f 1,3 metadata.tsv | head -n 3
sample_id       treatment
S82     cathemerium
S75     cathemerium
cut -f 1-2 metadata.tsv | head -n 3
sample_id       time
S82     10dpi
S75     10dpi

Note that it is not possible to change the order of columns with cut!

The default column delimiter that cut expects is a Tab. Therefore, we didn’t have to specify the delimiter in the above examples. But when a file has a different column delimiter, use the -d option — e.g. -d , for a CSV file:

# [Don't run this - for illustration only]
# Select the second column in a comma-delimited (CSV) file:
cut -d , -f 2 my-data.csv

Third, use sort to alphabetically sort the result:

tail -n +2 metadata.tsv | cut -f 3 | sort | head
cathemerium
cathemerium
cathemerium
cathemerium
cathemerium
cathemerium
cathemerium
control
control
control

Finally, use uniq to keep only distinct values — and get rid of head, since the command is now complete:

tail -n +2 metadata.tsv | cut -f 3 | sort | uniq
cathemerium
control
relictum

Generating a frequency table

With a very small modification to this pipeline, you can generate a “frequency table” instead of a simple list: a table with the number of times each distinct value occurs. You only have to add uniq’s -c option (for count):

tail -n +2 metadata.tsv | cut -f 3 | sort | uniq -c
      7 cathemerium
      7 control
      8 relictum

These frequency tables can in turn be sorted by most frequent occurrence.

By default, sort sorts alphabetically, even when lines start with numbers: it compares text character by character, so 10 would come before 9 (because 1 comes before 9). To sort numbers by their value instead, use sort’s -n option. Here, combine it with -r for reverse sorting so the largest numbers go first:

tail -n +2 metadata.tsv | cut -f 3 | sort | uniq -c | sort -nr
      8 relictum
      7 control
      7 cathemerium

Above, we used sort to simply sort a list. More generally, sort will by default perform sorting based on the entire line. To sort based on one or more columns, the way you’ve probably done in Excel, use the -k option — for example:

# Sort based on the third column:
tail -n +2 metadata.tsv | sort -k3

However, -k3 actually means “sort based on everything from column 3 to the end of the line.” To sort based on a single column only, give both a start and a stop column:

# Explicitly indicate to sort based on the third column only:
tail -n +2 metadata.tsv | sort -k3,3

To sort first by one column and then by another, use -k multiple times:

# Sort first by the third column, then break ties using the second column:
tail -n +2 metadata.tsv | sort -k3,3 -k2,2

Self-study exercise: cut, sort, and uniq

  1. The four pieces below, when combined with pipes (|) in the right order, produce a frequency table of the number of samples per time point. In which order should they go? Write out the full command and run it — how many 10dpi and how many 24hpi samples are there?

    sort
cut -f 2
uniq -c
tail -n +2 metadata.tsv

    Click for the solution.

    The file comes first, then we select the column, then sort, then count:

    tail -n +2 metadata.tsv | cut -f 2 | sort | uniq -c
         12 10dpi
     10 24hpi

    So the two time points do not have the same number of samples: there are 12 10dpi samples and 10 24hpi samples.

  2. What if you forget sort? Say you run the treatment frequency table pipeline again, but leave out sort like so:

    tail -n +2 metadata.tsv | cut -f 3 | uniq -c

    What do you expect to happen?

    1. The same output as before; sort does not affect the counts.
    2. An error message, because uniq needs sorted input.
    3. More than three lines, with each treatment appearing several times.
    4. Three lines, but in a different order.

    Click for the solution.

    The correct answer is C.

    uniq only compares adjacent lines, so without sorting first, it counts each separate “run” of identical values — here, one run per treatment per time point:

          4 cathemerium
      4 control
      4 relictum
      3 cathemerium
      3 control
      4 relictum

    This is why sort almost always comes right before uniq.

  3. Using a pipeline similar to those above to create another frequency table, check whether any duplicate sample IDs are present in metadata.tsv.

    Click for the solution.

    To check for duplicate sample IDs, create a frequency table of the first column (sample IDs):

    tail -n +2 metadata.tsv | cut -f 1 | sort | uniq -c

    This produces one line per sample (22 in total), and every single one shows a count of 1. That means there are no duplicate sample IDs: if any sample ID were duplicated, its count would be 2 or higher.

    Even better/more elegant: instead of scanning all 22 lines yourself, add sort -nr | head -n 1 to the end of the pipeline. This sorts the counts from high to low and shows only the top line. If that single count is 1, you immediately know there are no duplicates:

    tail -n +2 metadata.tsv | cut -f 1 | sort | uniq -c | sort -nr | head -n 1
          1 S86

5 Self-study: The Unix philosophy and data streaming

In this lecture, and in particular with these Unix data tool examples, we saw the Unix philosophy in action:

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
— Doug McIlroy

Some advantages of a modular approach are that it’s easier to spot errors, to swap out components, and to learn.

5.1 Text/data streaming

What about those text “streams” mentioned in the quote above? Rather than loading entire files into memory, most Unix tools process them one line at a time (a notable exception is sort, which has to see all lines before it can output anything). This is very useful when working with large files in particular! For example, head prints results instantly even for a 10 TB file that could never be loaded into memory. By contrast, try to take a peek at such a file with a GUI text editor, and it will freeze or refuse to open the file.

Here is one other example, combining the cat command with globbing and redirection. This command would concatenate (combine) all your R1 FASTQ files into a single file. This code is elegantly short, and will run quickly without using much computer memory!

# [Don't run this - for illustration only]
cat ../../garrigos/fastq/*R1.fastq.gz > all_R1.fastq.gz
TipHow does the above command work?

  1. cat is presented with a list of all R1 FASTQ files in the garrigos/fastq dir: the shell expands the glob pattern *R1.fastq.gz into a list of matching files, and passes that list to cat as arguments.

  2. cat then reads each file in turn, and prints its contents: When it prints multiple files, it simply prints them one after the other, i.e. concatenates them.

  3. The concatenated output is redirected to the file all_R1.fastq.gz. That is in fact a valid compressed FASTQ file, because:

    • FASTQ files can be concatenated freely since every read is a “stand-alone” unit of 4 lines.
    • Gzipped (.gz) files can also be concatenated freely!

5.2 Don’t redirect back to the input file!

One thing you should never do is redirect the output of a command “back” to its own input file. This doesn’t just fail — it destroys the input file, because the shell empties the file to prepare it for the output before the command has had a chance to read it.

# [Don't run this - for illustration only]
# You should NEVER run something like this:
sort metadata.tsv > metadata.tsv

If you really want to edit the original file instead of creating a separate edited copy, you will need two steps:

  1. Redirect to a new, edited copy of the input file.
  2. Rename the copy to overwrite the original file.
# [Don't run this - for illustration only]
# Step 1 - redirect to a new file:
sort metadata.tsv > metadata_sorted.tsv

# Step 2 - rename the new file to overwrite the original (if really needed!)
mv metadata_sorted.tsv metadata.tsv


Back to top

References

Garrigós, Marta, Guillem Ylla, Josué Martínez-de la Puente, Jordi Figuerola, and María José Ruiz-López. 2025. “Two Avian Plasmodium Species Trigger Different Transcriptional Responses on Their Vector Culex pipiens.” Molecular Ecology 34 (15): e17240. https://doi.org/10.1111/mec.17240.

Footnotes

  1. This works because OSC, like most Linux systems, has a helper program called lesspipe set up — on some other computers, less may show you the same gibberish that head did.↩︎

  2. Use double quotes by default, like in the examples. Single quotes generally work too, but they behave differently in some situations. There are some scenarios where omitting the quotes can lead to rather unexpected behavior.↩︎

  3. Note that even without -r, grep can operate on multiple files: you can give multiple file names as arguments or use a wildcard.↩︎