O queue¶

/store holds the project partitions on the NAS, a large and inexpensive near-line system. It holds data waiting to be processed or already processed, and it is not backed up. [email protected] sets up the groups that have a /store partition.

/store/public is being phased out

/store/public is no longer available. Use /data/public. A cloud-based cold-storage service is being evaluated as the replacement.

/store is mounted on the login nodes, the head node, the interactive nodes and the RStudio server node.

/store is not mounted on the compute nodes

A job that opens a path under /store fails. Copy the data to /scratch or /data first, on a login node or with an I/O job, and copy results back afterwards.

For a small copy, run cp or rsync on a login node, or under qrsh for anything that takes more than a few minutes:

$ rsync -a /store/PROJECT/USERNAME/run-12/ /scratch/genomics/USERNAME/run-12/

The login-node limits apply (see Warning emails), and a qrsh session ends after 48 hours.

Copy data as an I/O job¶

Large or repeated copies run as jobs in the I/O queue, lTIO.sq, which runs on the interactive nodes. An I/O job can run for 72 hours, and use 48 hours of CPU and 8 GB per slot. One user may run eight I/O jobs at once, with up to 8 slots in total.

Write a job file that requests the queue with -q lTIO.sq -l ioq:

getData.job

#$ -S /bin/sh
#$ -N getData -o getData.log -cwd -j y
#$ -q lTIO.sq -l ioq
#
echo + `date` job $JOB_NAME started in $QUEUE with jobID=$JOB_ID on $HOSTNAME
rsync -a /store/PROJECT/USERNAME/run-12/ /scratch/genomics/USERNAME/run-12/
echo = `date` job $JOB_NAME done

Submit it:

$ qsub getData.job
Your job 15503728 ("getData") has been submitted

Chain the analysis and the copy back so they start when the previous step finishes:

$ qsub -hold_jid 15503728 analyze.job
Your job 15503729 ("analyze") has been submitted
$ qsub -hold_jid 15503729 saveNClean.job
Your job 15503730 ("saveNClean") has been submitted
$ qstat+ +a%
Total running (PEs/jobs) = 1/1, 2 queued (jobs) for user 'USERNAME'.
     jobID name                          stat     age nPEs      cpu% queue     node taskID
  15503728 getData                          r   00:01    1           lTIO.sq  64-14
  15503729 analyze                        hqw   00:01    1
  15503730 saveNClean                     hqw   00:01    1           lTIO.sq

hqw is a job waiting on a hold. qchain getData.job analyze.job saveNClean.job submits the three with the holds set. See Run jobs in sequence.

/store snapshots and quotas are on the Filesystems page. quota+ shows your /store usage, which the Linux quota command does not. Recovering a file from a /store snapshot is under Recover a file from a snapshot.

Use /store and the I/O queue¶

Copy data on a login node or in a qrsh session¶

Copy data as an I/O job¶

Further reading¶