<aside> 🖥️
Pipelines that tap the computational power of remote servers — larger datasets, more complex processing. Currently image processing, sequencing, and comparative genomics.
</aside>
Our lab has access to powerful computing resources and support through the Center for High-Throughput Computing (CHTC). Our core bioinformatics and image processing pipelines are deployed through CHTC servers. All pipelines are maintained on GitHub and associated with Docker environments to ensure reproducibility. Many of our pipelines use Nextflow. Consult official CHTC documentation before getting started.
Register for an account using this form to access CHTC and our prioritized lab server node.
The CHTC has an extensive set of execute nodes. Our lab has a prioritized node that can be accessed on-demand using a designated flag.
Jobs on the CHTC are deployed from submit servers. You can ssh into our assigned submit server using your UW net-id and password. If you've added the lab aliases to your ~/.zshrc (see Local Mac Environment), you can just use submit.
# Log into submit server
ssh {net-id}@ap2002.chtc.wisc.edu
# Or, with the lab alias set up
submit
While movement of smaller files and scripts to CHTC can be handled a number of ways, the transfer of large files to CHTC Staging should use the dedicated transfer server.
# Log into transfer server
ssh {net-id}@transfer.chtc.wisc.edu
# Or, with the lab alias set up
transfer
Each net-id is associated with a home folder, where we manage job submission scripts. We have a shared staging folder for transfer of large files in and out of the CHTC system.
├── home/{net-id}/ [quota: 20 GB, submit script dir]
└── staging/groups/zamanian_group/ [quota: 1 TB | 100k files]
└── input/ [input dir: unprocessed (raw) data]
└── metadata/ [metadata dir: metadata associated with input data]
└── output/ [output dir: processed job outputs]
In general, pipelines run in three steps: