Week 6 Exercises: GitHub, OSC, and software
Introduction
Overview
In Exercise 1, you’ll practice with the GitHub material from last week that wasn’t part of Graded Assignment 2. In Exercises 2 and 3, you’ll practice with this week’s material: you’ll explore OSC’s login and compute nodes, and find and run software with modules and containers.
Make sure to do any of the exercises on this week’s lecture pages that we did not do in class. Also, if you struggled with any of the exercises we did do in class, go over those again.
Setting up
Start a VS Code session through OSC OnDemand in the folder /fs/ess/PAS3493/people/$USER.
Exercise 1: GitHub
Go to the GitHub website, log in, and create a new GitHub repository. This will be the remote counterpart of the Git repo you made as part of last week’s assignment.
- You can give the GitHub repo any name you want.
- Like we did in class, keep the repo Public (the default), because that’s needed for the instructor tagging exercise later on1.
- Also like we did in class, don’t let GitHub add a README or any other files when you create the repo.
Navigate to your dir for last week’s assignment in the VS Code terminal (you can also use the “Open Folder” option in VS Code to open that folder if you want).
Set up a connection between the Git repo and the GitHub counterpart using Git commands.
Push your local repo to the online counterpart.
Explore your repo on GitHub. Does your
README.mdrender well? Can you see all your commits? Are yourdataandresultsdirs there? Why or why not?You will hand in the remaining assignments for this course by “tagging” the instructors in a GitHub Issue. To make sure you know how this works, let’s do a test run. On GitHub, open a GitHub Issue for your repo: click the Issues tab and then “New Issue”.
Under “Add a title”, give the issue a title like “Testing whether tagging works”.
Under “Add a description”, tag both instructors using
@<username>syntax:Click the green “Create” button to post your Issue. If the tagging worked, the
@jelmerpand@kstarr791should have turned into links to the instructors’ GitHub profiles.
Click here to see screenshots of the tagging.
- When you type the tags, it will look like regular plain text:
- But once you create the issue, the tags, if they are valid, will turn bold and underlined, and are links to the instructors’ GitHub profiles. (You shouldn’t see a yellow background, though: that is only because I tagged myself to get this screenshot.)
Back in VS Code, add a line to your
README.mdto describe that you created a GitHub repo on today’s date (include the repo’s URL in the note).Commit the change to
README.mdto your Git repo.Push to the remote and on GitHub, check that the changes were applied in the online repo.
Exercise 2: Login versus compute nodes
In GA1, you found out that your VS Code session runs on a compute node. Let’s explore that a bit more, and compare with a login node.
In OnDemand, open a shell on a login node by clicking “Clusters” in the top bar and then “Cardinal Shell Access”. You should see many lines of text printed to the terminal, which is standard information that OSC prints on login (but not compute) nodes, including details about your Project’s usage of its storage quota2.
In the Cardinal shell, run the command
hostname, which prints the name of the node that you’re on (likeecho $HOSTNAMEthat you used in GA1). Then, run the same command in your VS Code terminal, and compare the outputs. What in each node name tells you whether it is a login node or a compute node? And which cluster is each node part of?Why does the Cardinal shell put you on a login node, whereas your VS Code session is on a compute node? And what does this mean for what you should (not) do in the Cardinal shell? For example, would it be OK to run FastQC on all FASTQ files of the Garrigós data there?
In the Cardinal shell, navigate to
/fs/ess/PAS3493/people/$USERand list its contents. Do you see the same files as in VS Code, even though your VS Code session is running on a different cluster?In your VS Code terminal, load the FastQC module and check that
fastqc -vworks. Predict whetherfastqc -vwill also work in the following two scenarios, and then check your predictions and interpret the results:- In the Cardinal shell.
- In a second VS Code terminal (click the
+icon in the terminal panel or the downward arrow next to it — see the screenshots below).
Click here for screenshots on the second VS Code terminal.
If you click the downward arrow next to the
+icon in the terminal panel, you get several options:I like the “Split Terminal” option best, which allows you to see both terminals at once:
If you just click the
+icon, or select “New Terminal” from the dropdown menu, you get a new terminal that takes up the entire panel, and you can switch between terminals using the panel on the side:Either way, having multiple terminals can be quite handy!
In the lecture’s self-study exercise on modules,
module spider salmonin VS Code said there was no Salmon module. Run the same command in the Cardinal shell. What do you get, and why?
Exercise 3: A specific software version
Say that you want to compare your results with those of a published study that used FastQC version 0.11.9, so you want to use that exact version as well.
Is FastQC version 0.11.9 available as an OSC module?
Get the URI of a Seqera container that contains FastQC version 0.11.9.
Check the container’s FastQC version.
Load the
fastqc/0.12.1module. You now have access to two different versions of FastQC. Before running anything, predict which versionfastqc -vwill print when you run it with and without the container prefix. Then, run both commands to check.
Solutions
Exercise 1
2. Navigate to your dir for last week’s assignment
cd GA23. Set up a connection to the online repo
# (replace <SSH-URL-to-repo> with your actual SSH (not HTTPS!) URL,
# which looks like 'git@github.com:<github-username>/<repo-name>.git')
git remote add origin <SSH-URL-to-repo>4. Push your local repo to the online one you just created
When you push to a remote repo for the first time, you need to add the -u origin main part:
git push -u origin main
5. Are data and results on GitHub?
No: you listed these dirs in your .gitignore file, so their files were never committed, and git push only uploads commits. (Your .gitignore file itself is on GitHub, since you did commit it.)
8. Commit the change you made to README.md
# Stage the change to README.md
git add README.md
# Make a commit with a message describing the change
git commit -m "Update README with GitHub repo information"9. Push to the remote again
Since you’ve pushed to this remote before, a simple git push works:
git pushExercise 2
2. Node names
In the Cardinal shell:
hostnamecardinal-login01.hpc.osc.edu
In VS Code:
hostnamep0231.ten.osc.edu
(Your exact node names will differ.)
- The Cardinal shell is on a login node, as you can tell from the
loginin its name, and it is on the Cardinal cluster, as the name also says. - VS Code is on a compute node, whose name is a letter followed by digits: the
pmeans it is part of the Pitzer cluster.
3. Why a login node versus a compute node
The “Clusters” shell connects you directly to the cluster, without requesting any resources, so you end up on a login node.
By contrast, when you start a Code Server session, you request a compute job, and the job scheduler assigns your session to a compute node. This is because Code Server itself needs resources to run, and if many users were to run Code Server on a login node, it could become overloaded.
In the Cardinal shell, you share a login node with many other users, so you should only run light tasks there, like navigating, listing files, or checking which modules are available. Running FastQC on dozens of FASTQ files is a real analysis that would take a while and use considerable resources: do that on a compute node instead.
4. Files across clusters
cd /fs/ess/PAS3493/people/$USER
lsYou see the same files, because all of OSC’s nodes and clusters share the same file systems.
5. Loaded modules in other shells
# In the first VS Code terminal:
module load fastqc/0.12.1
fastqc -vFastQC v0.12.1
# In the Cardinal shell, and in a second VS Code terminal:
fastqc -vbash: fastqc: command not found...
Failed to search for file: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Could not activate remote peer.
It doesn’t work in either of them: loading a module only affects the shell session you loaded it in. The second VS Code terminal is on the same compute node as the first one, but it’s a separate shell session, so you would need to load the module there as well.
6. Salmon on Cardinal
In the Cardinal shell, module spider salmon finds a Salmon module, whereas in VS Code (on Pitzer) it didn’t. The clusters have mostly, but not entirely, the same modules.
Therefore, if you want to use modules, you should check that the software you need is available on the cluster you’ll run your analysis on. If that seems like a pain, I agree, and it’s one reason I prefer containers. Also note that this is quite an old version of Salmon (1.4.0, from 2020), whereas a container gives you the latest version.
module spider salmon-------------------------------------------------------------------------------------------
salmon: salmon/1.4.0
-------------------------------------------------------------------------------------------
This module can be loaded directly: module load salmon/1.4.0
Help:
This module loads salmon
Configured and installed with modules:
No modules loaded
Exercise 3
1. Is FastQC 0.11.9 available as a module?
No, only version 0.12.1 is available:
module spider fastqc--------------------------------------------------------------------------------
fastqc: fastqc/0.12.1
--------------------------------------------------------------------------------
This module can be loaded directly: module load fastqc/0.12.1
2. Get a container with FastQC 0.11.9
At https://seqera.io/containers, search for fastqc, select version 0.11.9 in the dropdown menu next to the bioconda::fastqc hit, and then proceed as shown in the lecture (don’t forget to select Singularity).
Your URI will look like oras://community.wave.seqera.io/library/fastqc:0.11.9--<hash>.
3. Check the container’s FastQC version
apptainer exec oras://community.wave.seqera.io/library/fastqc:0.11.9--1cc469c72218f8fc \
fastqc -vINFO: Downloading oras image
352.3MiB / 352.3MiB [====================================================================================================================================================================================================] 100 % 43.1 MiB/s 0s
INFO: gocryptfs not found, will not be able to use gocryptfs
FastQC v0.11.9
4. Compare the module and container versions
module load fastqc/0.12.1
apptainer exec oras://community.wave.seqera.io/library/fastqc:0.11.9--1cc469c72218f8fc \
fastqc -vINFO: gocryptfs not found, will not be able to use gocryptfs
FastQC v0.11.9
fastqc -vFastQC v0.12.1
With the container, you always get version 0.11.9, regardless of which modules are loaded: the container uses the software installed inside it. Without the container, you get the version from the loaded module.
Footnotes
You can also make it private later if you want. Or you can make it private from the start, but then you need to add the instructors as collaborators in the repo’s settings.↩︎
But the corresponding information on the OnDemand dashboard is much more user-friendly, so I wouldn’t worry too much about the quota info printed in the shell.↩︎