Releasing GPUs Promptly¶
Your lab's GPUs are shared with your labmates, not queued campus-wide — but that only helps if allocations come back when you're done with them. This mostly matters for VS Code (code-server) sessions, where it's easy to forget a GPU is still reserved.
The problem¶
An interactive session holds its GPU for the entire walltime you
asked for, whether or not anything is running. Launch VS Code, start
python train.py, and walk away, and the GPU sits reserved until the
script finishes and you come back to end the session — or until
walltime runs out.
Self-cancel when the script ends¶
In the VS Code terminal, chain a scancel onto the command:
When train.py exits, scancel $SLURM_JOB_ID ends the Slurm job the
session is running under, tearing down the session — GPU included —
the moment the script is done.
Use ;, not &&. && only runs scancel on a zero exit status, so
a crash or Ctrl-C would leave the session (and GPU) reserved and
idle. ; runs it regardless.
This kills the whole session, including the editor — that's the point. Save your work first. If you need to inspect results interactively afterward, don't self-cancel; end the session manually from My Interactive Sessions instead.
Why batch is the better fit for long runs¶
For anything long — an overnight run, a sweep — submit as a
batch job (sbatch, see slurm-cli.md) instead:
- Batch jobs queue when your lab's GPUs are busy and start automatically the moment one frees up, including nights and weekends.
- The walltime you request becomes an upper bound, not a reservation — the job releases its GPU as soon as it finishes.
Net effect: batch jobs leave GPUs free for interactive use during the day and use the machines overnight when nobody's waiting.
hpcrun — the easy way to do this¶
hpcrun is a command-line tool that makes submitting a batch job
from a VS Code terminal as easy as running the command directly:
hpcrun python train.py # submit as a batch job; partition
# inherited from your session
hpcrun --gpu shared python train.py # fractional GPU (MPS), if your lab has it
hpcrun --gpu h200 --hours 24 python train.py --epochs 100
hpcrun status # your hpcrun jobs and their state
hpcrun logs -f # follow the current job's output
hpcrun cancel # cancel it
See hpcrun.md for the full reference — all options, GPU request syntax, working-directory rules, and NAS access.
Working with lab storage from a batch job¶
A batch job can land on a different node than your interactive
session. Since $HOME is local to each node (see
your-lab-storage.md), don't rely on files
under ~ being visible to a job on another node — read and write
under your lab's NAS paths instead (~/nas/..., ~/lab-research/...,
/mnt/...):