Graded Assignment 2:
Unix shell data tools and version control

Deadline

September 27, 2026

Page last modified

September 21, 2026

Overview

In this graded assignment, you will practice with Unix shell commands, especially the “data tools” commands, and with Git commands.

Directions and grading

Submission expectations

  • Deadline: Sunday September 27th at 11:59 pm.
  • Submission: Same as for GA1. That is, you don’t need to submit anything or notify the instructors: we will look for files, which you’ll create as part of the assignment, in the appropriate location at OSC. (We can also see when the files were last modified.)

Academic integrity

ImportantGen-AI check

  • For this assignment, you are allowed to search the internet, consult books, and so on, but you are not allowed to use generative AI tools. As such, using Google AI Mode output is not allowed either, but unfortunately, it may pop up even when you’re trying to use Google for regular web searches — in case that helps, please note that one way to prevent AI mode output from showing up at all is adding -ai to your Google search queries.

  • Along these lines, if you use commands or code constructs we didn’t cover in class, you must provide the source, such as a specific webpage or book page. Alternatively, you can argue you already knew that command from previous self-study or experience. But if you provide neither a source nor an explanation for commands not covered in class, your answer will be considered wrong and will not earn any points for the relevant question.

Use of generative AI tools is not permitted.
Open-book research for the assignment is permitted.
Getting help on the assignment is not permitted.
Collaborating, or completing the assignment with others, is not permitted.
Copying or reusing previous work is not permitted.
APA Citations and/or formatting for this assignment are not required.

Rubric

You can earn a maximum of 0.5 points for each of the 20 numbered questions/items below.

There are two bonus questions (21 and 22) that can also earn you 0.5 points each, but only in case you dropped points elsewhere.

Detailed steps

General

  • Execute Unix shell and Git commands to answer all questions unless explicitly instructed otherwise.

  • When you issue a command in the terminal, include the command in a Markdown code block, and the command’s output in another code block — same as you did for GA1. However, this isn’t needed for some Git commands, because we can check what you did by looking at your repository, and because you’re also allowed to use VS Code functionality instead of commands.

    To avoid confusion and make life easier for you, we provide you with a template Markdown file where you can add your answers (see question 1), which contains code blocks wherever needed.

    Assignment questions:

    1. Print your current working directory to screen.

    Your answers as part of your Markdown document:

    ### 1: Print your current working directory to screen

```bash
pwd
```

The output of the command was:

```
/fs/ess/PAS3493/people/jelmer
```

  • Keep a Markdown preview open to the side while you work, like you did for GA1, so you can check that your answers render as you intended — especially the code blocks.

Setting up

  • Start a VS Code (Code Server) session at OSC OnDemand as per usual, and open a terminal in that session.

  • Create a new dir for this assignment, /fs/ess/PAS3493/people/$USER/GA2, and navigate there. This will be your working dir for the entire assignment.

Part A: Starting a Git repository

  1. Copy the template file for this assignment, /fs/ess/PAS3493/share/GA2/GA2_template.md, into your working dir as README.md. Open it in the VS Code editor and modify its title (level 1 header) to include your name.

    You’ll use this file throughout the assignment to add your answers (remove the instructional comments inside the file as you go: these are Markdown comments, i.e. text between <!-- and -->, which is hidden in the Markdown preview).

  2. Initialize a Git repository inside your new directory.

  3. Add & stage and then commit the README file with an appropriate commit message.

Part B: Exploring the GTF file with Unix data tools

  1. Create a dir data and a dir results inside your working dir. Then, copy the genome annotation GTF file annot.gtf.gz that is in your copy of the Garrigós et al. (2025) data into your data dir, keeping the same file name.

  2. Check the file size of annot.gtf.gz.

  3. Decompress the GTF file with the gunzip command below, and check the size of the resulting annot.gtf file. Roughly how many times larger is the uncompressed file than the compressed file?

    gunzip data/annot.gtf.gz

  4. A GTF file starts with a few “header lines” (metadata), followed by its main/table section, in which each line describes one “genomic feature” (a gene, exon, etc.). Use a command to count the total number of lines in annot.gtf.

  5. Now, look at the GTF file contents to count the number of header lines by eye, like in last week’s exercises. How many lines does that leave for the table section?

  6. Now, count the number of lines in the table section in a more automated way: based on what you know about what GTF header lines look like, combine grep -v with another option or command to count the number of lines that are not header lines. What count do you get?

    Click here for some hints.

    • The -v option inverts grep’s behavior, so that it prints all lines that do not match the pattern (see the “Additional useful grep options” box in the section on grep). Use this option with a search pattern that matches the header lines.

    • Then, to get a line count, add another grep option to your command, or pipe the output of the grep command into wc -l.

    • You should encounter a minor difference in the number of lines counted in this question vs. the previous one. If you want to, you can try to figure out why that is as part of the bonus question at the end.

  7. How many distinct “sequences” (chromosomes/scaffolds/contigs) are represented in the annotation file? Store a list of these distinct sequences in a file results/scaffolds.txt.

    Click here for some context and a hint.

    • Context: The number of sequences gives a rough idea of the quality of the genome assembly, because a high-quality assembly will have only a few sequences, ideally one per chromosome. (Note, though, that any sequences in the genome assembly FASTA file that don’t have any annotated features would not be represented in the GTF file.)

    • Hint: In this and the following GTF questions, you’ll only want to consider the table part of the file, so start your commands with the same (or a similar) grep -v command as above.

  8. Print a “frequency table” of feature types in the annotation: this should have a count of the number of times each feature type occurs.

    Click here for some context and a hint.
    • Context: A frequency table of feature types will for example show how many genes have been annotated in this genome — that’s quite useful!
    • Hint: Feature types are in the 3rd column of the table part of the GTF file.

  9. A labmate wants to know how many genes have been annotated in this genome, and runs the command below:

    grep -c "gene" data/annot.gtf

    Before running it, predict which of the following statements is true:

    1. This gives the correct number of genes.
    2. This overcounts the number of genes, because the word “gene” also occurs in lines for other feature types.
    3. This undercounts the number of genes, because grep only matches whole words.
    4. This command gives an error, because the file is tab-delimited.

    Then, run the command and compare its output with the number of genes in your frequency table from question 11. Was your prediction correct, and what does this comparison tell you? (No points will be subtracted if you didn’t predict correctly.)

Part C: Updating your Git repository

  1. Check the status of your Git repository. You should see that your README.md has changed but also that Git has detected the files in the data and results dirs.

  2. You don’t want to commit any files in the data and results dirs, so create a .gitignore file that makes Git ignore them. Check the status of your repository again — is your .gitignore file working as intended?

  3. Stage and commit the .gitignore file. Then, in a separate commit, stage and commit the README.md file again.

Part D: Exploring the FASTQ files with Unix data tools

  1. Copy the FASTQ file S63_R1.fastq.gz of the Garrigós et al. (2025) data into your data dir, keeping the original file name.

  2. Check the status of your Git repository. Does the FASTQ file show up? Why/why not?

  3. How many reads does the FASTQ file contain?

    Click here for hints.

    Recall from the week 4 exercises that a direct line count of the compressed file will not give you the correct answer. Also, don’t forget to convert the number of lines to the number of reads.

  4. How many reads in the FASTQ file contain at least 10 consecutive Ns (i.e., uncalled bases)? (You can assume that stretches of Ns do not occur in the quality score lines.)

  5. Stage and commit the README.md file again.

Bonus

(You don’t need to commit your answer to these questions with Git, but you can if you want to.)

  1. In questions 8 and 9, you counted the number of lines in the table section of the GTF file in two different ways, but got different results. Can you figure out what accounts for the difference?

    Click here for some hints.

    • The file contains a line that is not a header line, but that also does not describe any genomic feature.

    • So the question boils down to: which line(s) other than the header lines contain the pattern you used with grep in question 9?

    • Review the “Additional useful grep options” box in the section on grep to find an option that might help you make sense of that.

  2. Name at least two concepts or commands that we have gone over in the course so far, but which you feel you do not (fully) understand. It helps to be specific about what you struggle with.

Solutions

The commands below assume that your working dir is /fs/ess/PAS3493/people/<user>/GA2 and that your copy of the Garrigós et al. (2025) data is in /fs/ess/PAS3493/people/<user>/garrigos. In the example output, <user> stands for your user name, and file dates, commit IDs, etc. will differ from yours.

Part A: Starting a Git repository

1. Copy the template file and add your name. 
cp /fs/ess/PAS3493/share/GA2/GA2_template.md README.md

Then, in the VS Code editor, change the level 1 header at the top of README.md to something like:

# GA2 answers - Jane Doe
2. Initialize a Git repository. 
git init
Initialized empty Git repository in /fs/ess/PAS3493/people/<user>/GA2/.git/
3. Stage and commit the README file. 
git add README.md
git commit -m "Add README"
  • Any commit message that describes the change is fine.
  • You could also have done this with the VS Code Source Control panel.

Part B: Exploring the GTF file with Unix data tools

4. Create data and results dirs and copy the GTF file. 
mkdir data results
cp ../garrigos/ref/annot.gtf.gz data/

Neither command prints any output. Using the absolute path to the GTF file (/fs/ess/PAS3493/people/<user>/garrigos/ref/annot.gtf.gz) is fine too.

5. Check the file size of annot.gtf.gz. 
ls -lh data/annot.gtf.gz
-rw-r--r-- 1 <user> PAS3493 5.1M Sep 25 14:02 data/annot.gtf.gz

The compressed file is 5.1 MB.

6. Decompress the GTF file and compare file sizes. 
gunzip data/annot.gtf.gz
ls -lh data/annot.gtf
-rw-r--r-- 1 <user> PAS3493 123M Sep 25 14:02 data/annot.gtf

The uncompressed file is 123 MB, which is roughly 24 times larger than the compressed file (123 / 5.1 ≈ 24). That’s a big difference, and the reason that large genomic files are typically stored in compressed form.

7. Count the total number of lines in annot.gtf. 
wc -l data/annot.gtf
408402 data/annot.gtf

The file has 408,402 lines in total.

8. Count the header lines by eye. 
head -n 5 data/annot.gtf
#gtf-version 2.2
#!genome-build TS_CPP_V2
#!genome-build-accession NCBI_Assembly:GCF_016801865.2
#!annotation-source NCBI RefSeq GCF_016801865.2-RS_2022_12
NC_068937.1     Gnomon  gene    2046    110808  .       +       .       gene_id "LOC120427725"; transcript_id ""; db_xref "GeneID:120427725"; description "homeotic protein deformed"; gbkey "Gene"; gene "LOC120427725"; gene_biotype "protein_coding";
  • There are 4 header lines, all starting with a #.
  • That leaves 408,402 - 4 = 408,398 lines for the table section.
  • Other commands to view the file, like less -S data/annot.gtf, are fine too.
9. Count the number of lines in the table section with grep -v. 
grep -v "#" data/annot.gtf | wc -l
408397
  • This gives 408,397 lines: one less than in the previous question — see bonus question 21 for the explanation.
  • Instead of piping to wc -l, you can use grep’s -c option: grep -vc "#" data/annot.gtf.
  • The quotes around # are needed: without them, the shell interprets everything from the # onwards as a comment, so grep wouldn’t get a search pattern at all.
10. Count the distinct sequences and save them to results/scaffolds.txt.
  • First, exclude the header lines with grep -v.
  • Then, select the first column, which contains the sequence names, with cut -f 1.
  • To only keep distinct sequences, use sort | uniq, and write the output to a file.
  • Finally, count the number of lines in the resulting file with wc -l.
grep -v "#" data/annot.gtf | cut -f 1 | sort | uniq > results/scaffolds.txt
wc -l results/scaffolds.txt
167 results/scaffolds.txt

  • There are 167 distinct sequences.

  • If you forgot grep -v "#", you got 172: the # lines don’t contain any tabs, so cut prints them in full, and each of these 5 lines is counted as an additional “sequence”.

  • The file results/scaffolds.txt should start like this:

    head -n 4 results/scaffolds.txt
    NC_015079.1
NC_068937.1
NC_068938.1
NC_068939.1
11. Print a frequency table of feature types.

The feature types are in the 3rd column:

grep -v "#" data/annot.gtf | cut -f 3 | sort | uniq -c
 141444 CDS
 166177 exon
  19673 gene
  25864 start_codon
  25892 stop_codon
  29347 transcript
  • So, for example, 19,673 genes have been annotated in this genome.
  • Adding | sort -nr to sort the table by count is a nice touch.
12. Predict, then run grep -c "gene". 
grep -c "gene" data/annot.gtf
408397
  • The correct answer is b): this overcounts the number of genes. The command returns 408,397, which is far more than the 19,673 genes in the frequency table from question 11.
  • That’s because nearly every line in the GTF file contains “gene” in its 9th (attributes) column, in gene_id "..." and gene "LOC...". In fact, 408,397 is exactly the number of table lines from question 9: every line in the table section matches.
  • The lesson: grep matches a pattern anywhere in a line, so to count genes, you need to check the feature type column specifically, as you did in question 11.
  • Regarding the other answers: grep does not only match whole words (unless you use its -w option), and it doesn’t care whether a file is tab-delimited.

Part C: Updating your Git repository

13. Check the status of your Git repository. 
git status
On branch main
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   README.md

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        data/
        results/

no changes added to commit (use "git add" and/or "git commit -a")

README.md has been modified since you committed it, and data/ and results/ are listed as untracked.

14. Create a .gitignore file and check the status again. 
echo "data/" > .gitignore
echo "results/" >> .gitignore
git status
On branch main
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   README.md

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        .gitignore

no changes added to commit (use "git add" and/or "git commit -a")
  • Yes, the .gitignore file is working: data/ and results/ are no longer listed. Instead, the new .gitignore file itself shows up as untracked.
  • Creating the .gitignore file in the VS Code editor is fine too. Patterns like data and results, or data/* and results/*, also work.
15. Commit .gitignore and then README.md in separate commits. 
git add .gitignore
git commit -m "Add .gitignore"

git add README.md
git commit -m "Add answers for Part B"

You can check that these are two separate commits with git log.

Part D: Exploring the FASTQ files with Unix data tools

16. Copy the FASTQ file. 
cp ../garrigos/fastq/S63_R1.fastq.gz data/

This command doesn’t print any output.

17. Does the FASTQ file show up in git status? 
git status
On branch main
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   README.md

no changes added to commit (use "git add" and/or "git commit -a")
  • No, the FASTQ file doesn’t show up, because it is in the data dir, which is listed in .gitignore.
  • Whether README.md is listed as modified depends on whether you had already added answers to it after your last commit. If not, you’d see “nothing to commit, working tree clean”.
18. How many reads does the FASTQ file contain? 
zcat data/S63_R1.fastq.gz | wc -l
2000000
  • Each read takes up 4 lines in a FASTQ file, so the file contains 2,000,000 / 4 = 500,000 reads.
  • Running wc -l data/S63_R1.fastq.gz directly on the compressed file returns 101,139, which is meaningless: you need to count the lines of the decompressed output.
19. How many reads contain at least 10 consecutive Ns? 
zcat data/S63_R1.fastq.gz | grep -c "NNNNNNNNNN"
32612
  • 32,612 reads contain at least 10 consecutive Ns.
  • A read with more than 10 consecutive Ns also matches, and is still counted only once, since grep -c counts matching lines.
20. Stage and commit the README file again. 
git add README.md
git commit -m "Add answers for Parts C and D"

Bonus

21. Why do the counts from questions 8 and 9 differ?

Use grep’s -n option to print line numbers along with each matching line:

grep -n "#" data/annot.gtf
1:#gtf-version 2.2
2:#!genome-build TS_CPP_V2
3:#!genome-build-accession NCBI_Assembly:GCF_016801865.2
4:#!annotation-source NCBI RefSeq GCF_016801865.2-RS_2022_12
408402:###
  • Besides the 4 header lines, the last line of the file, ###, also contains a #. This line seems to function as an end-of-file marker.
  • It is not a header line, so you didn’t count it as one in question 8, but grep -v "#" did exclude it in question 9: 408,402 - 5 = 408,397.
  • You could also have found this line by looking at the end of the file with tail data/annot.gtf.
22. Concepts or commands you don’t (fully) understand.

There is no single correct answer here.


Back to top

References

Garrigós, Marta, Guillem Ylla, Josué Martínez-de la Puente, Jordi Figuerola, and María José Ruiz-López. 2025. “Two Avian Plasmodium Species Trigger Different Transcriptional Responses on Their Vector Culex pipiens.” Molecular Ecology 34 (15): e17240. https://doi.org/10.1111/mec.17240.