Week 4 exercises: Files in the Unix shell
Introduction
Overview
In these exercises, you’ll practice with shell commands, especially those that operate on files. In exercise 4, you’ll start exploring the GTF annotation file of the example dataset, which you will continue doing in next week’s Graded Assignment 2.
Setting up
- Start a VS Code session through OSC OnDemand in the folder
/fs/ess/PAS3493/people/<user>(see the w4b lecture for instructions). - Optional: Create and use a Markdown file to keep notes on what you’re doing.
Make sure to do any of the exercises on this week’s lecture pages that we did not do in class. Also, if you struggled with any of the exercises we did do in class, go over those again.
Exercise 1: Nuances of cp and mv behavior
You should start in /fs/ess/PAS3493/people/$USER. Use shell commands to do the following:
Create a dir
week04/ex1within/fs/ess/PAS3493/people/$USER, and navigate there. Use this as the working dir for all following questions in this exercise, so avoid navigating anywhere else.Create a new file called
newfile.txtin your working dir.
cp and mv behavior
For both cp and mv, when operating on files (and this works equivalently for dirs, though cp needs -r for those):
- If the destination path is an existing dir, the file will go into that dir and keep its original name.
- If the destination path is not an existing dir, the (last bit of the) destination specifies the new file name.
- A trailing slash in the destination path makes explicit that you are referring to a dir and not a file.
With the above information in mind, try to answer the questions below about the following command:
cp newfile.txt more_data/What do you think the command would do or attempt to do?
Do you think the command will succeed? Run the command and see whether you were right.
What would the command have done if you had omitted the trailing slash?
Exercise 2: Copying, removing, and moving dummy FASTQ files
Create a dir
week04/ex2within/fs/ess/PAS3493/people/$USER, and navigate there. Like in the previous exercise, use this as the working dir for all following questions in this exercise.Create a new dir
seqs, and use the command below to create some empty dummy files with FASTQ file names in that dir. Then, check the result withlsand use a command to get a count of the number of files you created.touch seqs/sample{01..25}_R{1,2}.fastqThis uses a trick called ‘brace expansion’ to create many files at once. You don’t need to understand how this works.
You should have found that the
seqsdir contains 50 files. Without running it, how many files do you thinkls seqs/*2*will list?- 8
- 25
- 33
- 50
Then check your answer by adding
| wc -lto that command.Copy the dummy FASTQ files for samples
01through09into your working dir with a single command.Remove the files you just copied, and make the command report what it’s doing.
Move all R1 files into a dir
seqs_R1, and then renameseqs, which should now only contain R2 files, toseqs_R2.
Exercise 3: More copying, and rmdir
Create a dir
week04/ex3within/fs/ess/PAS3493/people/$USER, and navigate there. Use this as the working dir for all following questions in this exercise, so avoid navigating anywhere else.This week, you downloaded the Garrigós et al. (2025) dataset (
garrigosfolder). Copy its entirerefsubdir into you current working dir, giving the new copy the nameref_copy.Run the following command (
rmdirwas only mentioned briefly on the lecture page!):rmdir ref_copyWhat is this trying to do and why does it fail? (Hint: check the callout box in the
rmsection of the lecture page)Move the GTF file in
ref_copyto your current working dir, keeping the same file name.Do you think you will now be able to successfully run
rmdir ref_copy? Try it.Does
rmdirseem like a slightly silly, incompetent command? Why do you think it exists at all?
Exercise 4: Exploring the GTF annotation file
GTF file info
This exercise will work with the GTF file in the example dataset, so we’ll start with some more info about GTF files. As mentioned in the genomics file types lecture, GTF files typically start with header lines that contain metadata. Then, the main part of a GTF file is essentially a table, with:
- One row for each annotated “genomic feature” (gene, exon, transcript, etc.)
- Columns with information like the genomic coordinates of the features
Here is a snippet of our Culex pipiens GTF table, again with column names added for clarity (they aren’t in the file!):
seqname source feature start end score strand frame attributes
NC_068937.1 Gnomon gene 2046 110808 . + . gene_id "LOC120427725"; transcript_id ""; db_xref "GeneID:120427725"; description "homeotic protein deformed"; gbkey "Gene"; gene "LOC120427725"; gene_biotype "protein_coding";
NC_068937.1 Gnomon transcript 2046 110808 . + . gene_id "LOC120427725"; transcript_id "XM_052707445.1"; db_xref "GeneID:120427725"; gbkey "mRNA"; gene "LOC120427725"; model_evidence "Supporting evidence includes similarity to: 25 Proteins"; product "homeotic protein deformed, transcript variant X3"; transcript_biotype "mRNA";
NC_068937.1 Gnomon exon 2046 2531 . + . gene_id "LOC120427725"; transcript_id "XM_052707445.1"; db_xref "GeneID:120427725"; gene "LOC120427725"; model_evidence "Supporting evidence includes similarity to: 25 Proteins"; product "homeotic protein deformed, transcript variant X3"; transcript_biotype "mRNA"; exon_number "1";
NC_068937.1 Gnomon exon 52113 52136 . + . gene_id "LOC120427725"; transcript_id "XM_052707445.1"; db_xref "GeneID:120427725"; gene "LOC120427725"; model_evidence "Supporting evidence includes similarity to: 25 Proteins"; product "homeotic protein deformed, transcript variant X3"; transcript_biotype "mRNA"; exon_number "2";
Some details on what’s in each column:
| Column # | Column ID | Explanation |
|---|---|---|
| 1 | seqname | Name of the chromosome, scaffold, or contig. |
| 2 | source | Name of the program that generated this feature, or the data source (e.g. database). |
| 3 | feature | Name of the genomic feature type, e.g. “gene”, “transcript”, “exon”, “CDS”. |
| 4 | start | Start position of the feature. |
| 5 | end | End position of the feature. |
| 6 | score | A confidence score for the feature, often absent (in which case it is .). |
| 7 | strand | Whether the feature is on the + (forward) or - (reverse) strand1. |
| 8 | frame | The “frame” for coding features, either 0, 1, or 2 (or . for no frame). E.g. 0 means the first base of the feature is the first base of a codon. |
| 9 | attributes | A semicolon-separated list of key-value pairs with additional information. Can be very wide! |
Your turn
You should still be in your ex3 dir with the GTF file and can stay there.
Explore
annot.gtf.gzwithless, using its-Soption to prevent line wrapping (see the box just above this point). Just look around a bit and try to navigate with the keybindings shown in the lecture, then consider the following questions:- How many header lines does the file have?
- How can you tell where the header ends and the table part begins?
- Is the
-Soption useful here? If you’re not sure, you can compare the output with and without it to see why.
Recall that
zcatis the counterpart ofcatfor gzip-compressed files: it prints their contents in human-readable form. Pipe the output ofzcatto another command to print only the first 15 lines ofannot.gtf.gz.Count the number of lines in the GTF file both in compressed and uncompressed form. Are the counts the same?
Click to show a hint.
Use the
zcat-then-pipe trick from the previous question to get the “uncompressed count”. The “compressed count”, on the other hand, is simply the number of lines inannot.gtf.gz.Count the number of lines in the GTF file that contain the text “carbonic anhydrase”. Based on this count, what can you conclude about the number of carbonic anhydrase genes in the genome?
- Exactly as many as the line count, because each line in a GTF file is one gene.
- Fewer than the line count, because each gene is described by multiple lines.
- More than the line count, because
grep -ccounts at most one match per line. - You can’t tell, because line counts of compressed files are meaningless.
Build on your command from the previous question to create a frequency table of feature types among the “carbonic anhydrase” lines. Then answer these questions:
- How many carbonic anhydrase genes are there?
- Why do you think there are more transcripts than genes?
Click to show a hint.
The feature type is in column 3 of the GTF file. To create the frequency table, use the
cut-sort-uniqcombination from the lecture.
Solutions
Exercise 1
1. Create a new dir and navigate there.
This assumes you start in /fs/ess/PAS3493/people/$USER:
mkdir -p week04/ex1
cd week04/ex1Or, with absolute paths:
mkdir -p /fs/ess/PAS3493/people/$USER/week04/ex1
cd /fs/ess/PAS3493/people/$USER/week04/ex1
2. Create a new file called newfile.txt in your working dir.
touch newfile.txt3. What do you think the command would do or attempt to do?
Because we added a trailing forward slash to more_data/, we are making clear that we are referring to a directory. So the command will attempt to copy the file into a dir more_data, with the file keeping the same name.
4. Do you think the command will succeed?
The more_data/ dir does not exist, and cp will not create a dir on the fly — so it will fail:
cp newfile.txt more_data/cp: cannot create regular file 'more_data/': Not a directory
5. What would the command have done if you had omitted the trailing slash?
If you had omitted the trailing forward slash, it would have created a copy of the file with file name more_data (if that seems confusing: note that in Unix, files don’t need to have a file extension).
P.S.: To copy the file according to the original intention, first create the destination dir and then copy:
mkdir more_data
cp newfile.txt more_data/Note also that once the more_data dir exists, it does not make a difference whether or not you use a trailing slash.
Exercise 2
1. Create a new dir and navigate there.
Assuming you’re doing this right after Exercise 1, you should be in /fs/ess/PAS3493/people/$USER/week04/ex1, in which case the following commands work:
mkdir ../ex2
cd ../ex2Or, with absolute paths:
mkdir /fs/ess/PAS3493/people/$USER/week04/ex2
cd /fs/ess/PAS3493/people/$USER/week04/ex22. Create a dir with dummy sequence files.
Create the dir:
mkdir seqsCreate the dummy files:
touch seqs/sample{01..25}_R{1,2}.fastqCheck the result:
ls seqssample01_R1.fastq sample03_R1.fastq sample05_R1.fastq sample07_R1.fastq sample09_R1.fastq sample11_R1.fastq sample13_R1.fastq sample15_R1.fastq sample17_R1.fastq sample19_R1.fastq sample21_R1.fastq sample23_R1.fastq sample25_R1.fastq sample01_R2.fastq sample03_R2.fastq sample05_R2.fastq sample07_R2.fastq sample09_R2.fastq sample11_R2.fastq sample13_R2.fastq sample15_R2.fastq sample17_R2.fastq sample19_R2.fastq sample21_R2.fastq sample23_R2.fastq sample25_R2.fastq sample02_R1.fastq sample04_R1.fastq sample06_R1.fastq sample08_R1.fastq sample10_R1.fastq sample12_R1.fastq sample14_R1.fastq sample16_R1.fastq sample18_R1.fastq sample20_R1.fastq sample22_R1.fastq sample24_R1.fastq sample02_R2.fastq sample04_R2.fastq sample06_R2.fastq sample08_R2.fastq sample10_R2.fastq sample12_R2.fastq sample14_R2.fastq sample16_R2.fastq sample18_R2.fastq sample20_R2.fastq sample22_R2.fastq sample24_R2.fastqCount the number of files:
ls seqs | wc -l50Note that this gives the correct count even though when printed to screen,
lsoutput typically uses multiple columns and therefore shows more than one file per line.In contrast, the count from
ls -lis not correct, since its output includes atotalline:ls -l seqs | wc -l51
3. How many files does ls seqs/*2* list?
The correct answer is c) 33.
Because * matches any number of any characters (including none), *2* matches every file name that contains a “2” somewhere:
- All 25
_R2files — because “_R2” itself contains a 2! - Plus the 8 R1 files of the samples whose number contains a 2:
02,12,20,21,22,23,24, and25.
ls seqs/*2* | wc -l33
4. Copy the files for samples 01 through 09 into your working dir with a single command.
Copy the files — remember to use
.to indicate your current working dir as the destination path.cp seqs/sample0* .Let’s check the result:
lssample01_R1.fastq sample02_R1.fastq sample03_R1.fastq sample04_R1.fastq sample05_R1.fastq sample06_R1.fastq sample07_R1.fastq sample08_R1.fastq sample09_R1.fastq seqs sample01_R2.fastq sample02_R2.fastq sample03_R2.fastq sample04_R2.fastq sample05_R2.fastq sample06_R2.fastq sample07_R2.fastq sample08_R2.fastq sample09_R2.fastq
5. Remove the files you just copied.
rm -v sample0*removed 'sample01_R1.fastq'
removed 'sample01_R2.fastq'
removed 'sample02_R1.fastq'
removed 'sample02_R2.fastq'
removed 'sample03_R1.fastq'
removed 'sample03_R2.fastq'
removed 'sample04_R1.fastq'
removed 'sample04_R2.fastq'
removed 'sample05_R1.fastq'
removed 'sample05_R2.fastq'
removed 'sample06_R1.fastq'
removed 'sample06_R2.fastq'
removed 'sample07_R1.fastq'
removed 'sample07_R2.fastq'
removed 'sample08_R1.fastq'
removed 'sample08_R2.fastq'
removed 'sample09_R1.fastq'
removed 'sample09_R2.fastq'
6. Move all R1 files into a dir seqs_R1 and rename seqs.
Create the new dir:
mkdir seqs_R1This step is needed: when you move multiple files, the destination must be an existing dir. Without it, you’d get an error:
mv: target 'seqs_R1' is not a directory.Move the R1 files into the new dir:
mv seqs/*R1.fastq seqs_R1Rename the old dir from
seqstoseqs_R2:mv seqs seqs_R2Check the result:
tree. ├── seqs_R1 │ ├── sample01_R1.fastq │ ├── sample02_R1.fastq │ ├── sample03_R1.fastq │ ├── sample04_R1.fastq │ ├── sample05_R1.fastq │ ├── sample06_R1.fastq │ ├── sample07_R1.fastq │ ├── sample08_R1.fastq │ ├── sample09_R1.fastq │ ├── sample10_R1.fastq │ ├── sample11_R1.fastq │ ├── sample12_R1.fastq │ ├── sample13_R1.fastq │ ├── sample14_R1.fastq │ ├── sample15_R1.fastq │ ├── sample16_R1.fastq │ ├── sample17_R1.fastq │ ├── sample18_R1.fastq │ ├── sample19_R1.fastq │ ├── sample20_R1.fastq │ ├── sample21_R1.fastq │ ├── sample22_R1.fastq │ ├── sample23_R1.fastq │ ├── sample24_R1.fastq │ └── sample25_R1.fastq └── seqs_R2 ├── sample01_R2.fastq ├── sample02_R2.fastq ├── sample03_R2.fastq ├── sample04_R2.fastq ├── sample05_R2.fastq ├── sample06_R2.fastq ├── sample07_R2.fastq ├── sample08_R2.fastq ├── sample09_R2.fastq ├── sample10_R2.fastq ├── sample11_R2.fastq ├── sample12_R2.fastq ├── sample13_R2.fastq ├── sample14_R2.fastq ├── sample15_R2.fastq ├── sample16_R2.fastq ├── sample17_R2.fastq ├── sample18_R2.fastq ├── sample19_R2.fastq ├── sample20_R2.fastq ├── sample21_R2.fastq ├── sample22_R2.fastq ├── sample23_R2.fastq ├── sample24_R2.fastq └── sample25_R2.fastq 2 directories, 50 files
Exercise 3
1. Create a new dir and navigate there.
Assuming you’re doing this right after Exercise 2, you should be in /fs/ess/PAS3493/people/$USER/week04/ex2, in which case the following commands work:
mkdir ../ex3
cd ../ex3Or, with absolute paths:
mkdir /fs/ess/PAS3493/people/$USER/week04/ex3
cd /fs/ess/PAS3493/people/$USER/week04/ex3
2. Copy the ref dir to ref_copy.
Because you are copying a dir, you will need the -r (recursive) option:
cp -r ../../garrigos/ref ref_copy
3. Run the rmdir command.
rmdir tries to remove a directory, but unlike rm -r, it only works on empty directories. Here it fails because ref_copy still contains the GTF file:
rmdir ref_copyrmdir: failed to remove 'ref_copy': Directory not empty
4. Move the GTF file in ref_copy to your current working dir.
To keep the same file name, remember to simply use the . shortcut for the current working dir as the destination:
mv ref_copy/annot.gtf.gz .
5. Do you think you will now be able to successfully run rmdir ref_copy?
Yes, because the ref_copy dir is now empty, this will work:
# [No output means it succeeds!]
rmdir ref_copyLet’s check — indeed, the ref_copy dir is no longer there:
lsannot.gtf.gz
6. Why does rmdir exist if it’s so limited?
rmdir exists as a safety measure: because it refuses to remove non-empty dirs, you can’t accidentally delete a dir with important files inside, which is easy to do with rm -r.
Exercise 4
1. Explore annot.gtf.gz with less -S.
The file has 4 header lines:
less -S annot.gtf.gz#gtf-version 2.2 #!genome-build TS_CPP_V2 #!genome-build-accession NCBI_Assembly:GCF_016801865.2 #!annotation-source NCBI RefSeq GCF_016801865.2-RS_2022_12 NC_068937.1 Gnomon gene 2046 110808 . + . gene_id "LOC120427725"; transcript_id ""; db_xref "GeneID:120427725"; description "homeotic protein deformed"; gbkey "Gene"; gene "LOC120427725"; gene_biotype "protein_coding"; NC_068937.1 Gnomon transcript 2046 110808 . + . gene_id "LOC120427725"; transcript_id "XM_052707445.1"; db_xref "GeneID:120427725"; gbkey "mRNA"; gene "LOC120427725"; model_evidence "Supporting evidence includes similarity to: 25 Proteins"; product "homeotic protein deformed, transcript variant X3"; transcript_biotype "mRNA";You can tell the header apart because all header lines start with a
#: the first line that doesn’t start with a#is the first line of the table.The
-Soption is very useful here: each line in the table part is one feature, and with the line-wrappinglesswill do by default (i.e., without the-Soption), it’s hard to see where one line ends and the next begins.
2. Pipe the output of zcat to another command to print the file’s first 15 lines.
zcat annot.gtf.gz | head -n 15#gtf-version 2.2
#!genome-build TS_CPP_V2
#!genome-build-accession NCBI_Assembly:GCF_016801865.2
#!annotation-source NCBI RefSeq GCF_016801865.2-RS_2022_12
NC_068937.1 Gnomon gene 2046 110808 . + . gene_id "LOC120427725"; transcript_id ""; db_xref "GeneID:120427725"; description "homeotic protein deformed"; gbkey "Gene"; gene "LOC120427725"; gene_biotype "protein_coding";
NC_068937.1 Gnomon transcript 2046 110808 . + . gene_id "LOC120427725"; transcript_id "XM_052707445.1"; db_xref "GeneID:120427725"; gbkey "mRNA"; gene "LOC120427725"; model_evidence "Supporting evidence includes similarity to: 25 Proteins"; product "homeotic protein deformed, transcript variant X3"; transcript_biotype "mRNA";
NC_068937.1 Gnomon exon 2046 2531 . + . gene_id "LOC120427725"; transcript_id "XM_052707445.1"; db_xref "GeneID:120427725"; gene "LOC120427725"; model_evidence "Supporting evidence includes similarity to: 25 Proteins"; product "homeotic protein deformed, transcript variant X3"; transcript_biotype "mRNA"; exon_number "1";
NC_068937.1 Gnomon exon 52113 52136 . + . gene_id "LOC120427725"; transcript_id "XM_052707445.1"; db_xref "GeneID:120427725"; gene "LOC120427725"; model_evidence "Supporting evidence includes similarity to: 25 Proteins"; product "homeotic protein deformed, transcript variant X3"; transcript_biotype "mRNA"; exon_number "2";
NC_068937.1 Gnomon exon 70113 70962 . + . gene_id "LOC120427725"; transcript_id "XM_052707445.1"; db_xref "GeneID:120427725"; gene "LOC120427725"; model_evidence "Supporting evidence includes similarity to: 25 Proteins"; product "homeotic protein deformed, transcript variant X3"; transcript_biotype "mRNA"; exon_number "3";
NC_068937.1 Gnomon exon 105987 106087 . + . gene_id "LOC120427725"; transcript_id "XM_052707445.1"; db_xref "GeneID:120427725"; gene "LOC120427725"; model_evidence "Supporting evidence includes similarity to: 25 Proteins"; product "homeotic protein deformed, transcript variant X3"; transcript_biotype "mRNA"; exon_number "4";
NC_068937.1 Gnomon exon 106551 106734 . + . gene_id "LOC120427725"; transcript_id "XM_052707445.1"; db_xref "GeneID:120427725"; gene "LOC120427725"; model_evidence "Supporting evidence includes similarity to: 25 Proteins"; product "homeotic protein deformed, transcript variant X3"; transcript_biotype "mRNA"; exon_number "5";
NC_068937.1 Gnomon exon 109296 109660 . + . gene_id "LOC120427725"; transcript_id "XM_052707445.1"; db_xref "GeneID:120427725"; gene "LOC120427725"; model_evidence "Supporting evidence includes similarity to: 25 Proteins"; product "homeotic protein deformed, transcript variant X3"; transcript_biotype "mRNA"; exon_number "6";
NC_068937.1 Gnomon exon 109726 110808 . + . gene_id "LOC120427725"; transcript_id "XM_052707445.1"; db_xref "GeneID:120427725"; gene "LOC120427725"; model_evidence "Supporting evidence includes similarity to: 25 Proteins"; product "homeotic protein deformed, transcript variant X3"; transcript_biotype "mRNA"; exon_number "7";
NC_068937.1 Gnomon CDS 70143 70962 . + 0 gene_id "LOC120427725"; transcript_id "XM_052707445.1"; db_xref "GeneID:120427725"; gbkey "CDS"; gene "LOC120427725"; product "homeotic protein deformed"; protein_id "XP_052563405.1"; exon_number "3";
NC_068937.1 Gnomon CDS 105987 106087 . + 2 gene_id "LOC120427725"; transcript_id "XM_052707445.1"; db_xref "GeneID:120427725"; gbkey "CDS"; gene "LOC120427725"; product "homeotic protein deformed"; protein_id "XP_052563405.1"; exon_number "4";
3. Count the number of lines in the GTF file both in compressed and uncompressed form.
Uncompressed count:
zcat annot.gtf.gz | wc -l408402Compressed count:
wc -l annot.gtf.gz22934 annot.gtf.gz
So: no, the counts are not the same! The lesson: line counts of compressed files are meaningless — to count lines in a compressed file, count the lines of its decompressed output, e.g. with zcat <file> | wc -l.
4. What can you conclude about the number of carbonic anhydrase genes?
Count the lines that contain “carbonic anhydrase”:
zcat annot.gtf.gz | grep -c "carbonic anhydrase"243The correct answer is b): fewer genes than lines. Recall from the GTF snippet at the top of this exercise that a single gene is described by multiple lines: one
geneline, but alsotranscript,exon, and (for protein-coding genes)CDSlines.Regarding answer d): because we used
zcat,grepoperated on the decompressed contents, so this count is meaningful.
5. Create a frequency table of feature types among the “carbonic anhydrase” lines.
Select column 3 with
cut, thensortand count withuniq -c. Unlike in the lecture, you don’t need something along the lines oftail -n +2here: the header lines don’t contain “carbonic anhydrase”, sogrephas already removed them.zcat annot.gtf.gz | grep "carbonic anhydrase" | cut -f 3 | sort | uniq -c81 CDS 89 exon 16 gene 19 start_codon 19 stop_codon 19 transcriptSo, there are 16 carbonic anhydrase genes.
Besides the feature types we’ve discussed, you also see
start_codonandstop_codonlines, which give the positions of the first and last codon of each coding sequence.There are more transcripts (19) than genes (16) because some genes have multiple transcripts — different versions of the gene’s mRNA, mostly due to alternative splicing. In the GTF snippet at the top of this exercise, you can see a hint of this in the
productattribute:transcript variant X3.
References
Footnotes
Because double-stranded DNA is represented as a single sequence, the
strandcolumn effectively indicates in which direction a feature should be read along this sequence.↩︎