Support
Please get in touch with the PLGrid Helpdesk: https://helpdesk.plgrid.pl/ regarding any difficulties in using the cluster.
For important information and announcements, please follow this page and the messages displayed in the login message.
Machine description
Available login nodes:
- ssh <login>@ares.cyfronet.pl
Ares is built with Infiniband EDR interconnect and nodes of the following specification:
| Partition | Number of nodes | CPU | RAM | Accelerator |
|---|---|---|---|---|
| plgrid and plgrid-* | 532 | 48 cores, Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz | 192GB | |
| plgrid-bigmem | 256 | 48 cores, Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz | 384GB | |
| plgrid-gpu-v100 | 9 | 32 cores, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz | 384GB | Tesla V100-SXM2 |
Job submission
Ares is using Slurm resource manager, jobs should be submitted to the following partitions:
| Name | Timelimit | Remarks |
|---|---|---|
| plgrid | 72h | Standard partition. |
| plgrid-long | 168h | Used for jobs with extended runtime. |
| plgrid-testing | 1h | High priority, testing jobs, limited to 3 jobs. |
| plgrid-bigmem | 72h | Jobs using an extended amount of memory. |
| plgrid-now | 12h | The highest priority, interactive jobs, limited to 1 running job. |
| plgrid-gpu-v100 | 72h | GPU partition. |
Accounts and computing grants
Ares uses a new scheme of naming accounts for CPU and GPU computing grants. CPU only grants are named: grantname-cpu, while GPU accounts use grantname-gpu appropriate suffix. Please mind that sbatch -A grantname won't work on its own, you need to add the -cpu or -gpu suffix! Available computing grants, with respective account names (allocations), can be viewed by using the hpc-grants command.
Storage
Available storage spaces are described in the following table:
| Location | Location in the filesystem | Purpose |
|---|---|---|
| $HOME | /net/people/<login> | Storing own applications, configuration files |
| $SCRATCH | /net/pr2/scratch/people/<login> | High-speed storage used for short-lived data heavily used in computations. Data present for more than 30 days can be deleted without notice. |
| group storage | /net/pr2/projects/plgrid/<group name> | Long term storage, used for data living for the period of computing grant. |
Current usage, capacity and other storage attributes can be checked by issuing the hpc-fs command.
System utilities
Please use the following commands for interacting with the account and storage management system:
hpc-grants-shows available grants, resource allocationshpc-fs- shows available storagehpc-jobs- shows currently pending/running jobshpc-jobs-history- shows information about past jobs
Software
Applications and libraries are available through the modules system, list of available modules can be obtained by issuing the command:
module avail
a module can be loaded by the add command:
module add openmpi/4.1.1-gcc-11.2.0
and the environment can be purged by:
module purge
More information
Ares is following Prometheus' configuration and usage patterns. Prometheus documentation can be found here: https://kdm.cyfronet.pl/portal/Prometheus:Basics