<aside> 🖥️
Pipelines that tap the computational power of remote servers — larger datasets, more complex processing. Currently image processing, sequencing, and comparative genomics.
</aside>
Our lab has access to powerful computing resources and support through the Center for High-Throughput Computing (CHTC). Our core bioinformatics and image processing pipelines are deployed through CHTC servers. All pipelines are maintained on GitHub and associated with Docker environments to ensure reproducibility. Many of our pipelines use Nextflow. Consult official CHTC documentation before getting started.
Register for an account using this form to access CHTC and our prioritized lab server node.
The CHTC has an extensive set of execute nodes. Our lab has a prioritized node, parabyte2000.chtc.wisc.edu, reached by setting Accounting_Group = PathobiologicalSciences_Zamanian in the submit file.
Jobs on the CHTC are deployed from submit servers. While movement of smaller files and scripts to CHTC can be handled a number of ways, the transfer of large files to CHTC Staging should use the dedicated transfer server.
You can ssh into our assigned submit server or the transfer server using your UW net-id and password. If you've added the lab aliases to your ~/.zshrc (see Local Mac Environment), you can just use submit. To avoid constant log-ins and two-factor authentication via ssh, use this guide to set up a persistent connection for defined periods of time.
# Log into submit server
ssh {net-id}@ap2002.chtc.wisc.edu
# Or, with the lab alias set up
submit
# Log into transfer server
ssh {net-id}@transfer.chtc.wisc.edu
# Or, with the lab alias set up
transfer
Each net-id is associated with a home folder, where we manage job submission scripts. We have a shared staging folder for transfer of large files in and out of the CHTC system. Files can also be directly transferred into jobs from Research Drive.
├── home/{net-id}/ [quota: 20 GB, submit script dir]
└── staging/groups/zamanian_group/ [quota: 1 TB | 100k files]
└── input/ [input dir: unprocessed (raw) data]
└── metadata/ [metadata dir: metadata associated with input data]
└── output/ [output dir: processed job outputs]
In general, pipelines run in three steps: