| Note | ||
|---|---|---|
| ||
Ares is still under development, and even despite our best efforts, Ares might experience unscheduled outages or even data loss. |
Content
| Table of Contents | ||
|---|---|---|
|
...
Ares is built with Infiniband EDR interconnect and nodes of the following specification:
| Partition | Number of nodes | CPU | RAM | RAM available for job allocations | Proportional RAM for one CPU | Proportional RAM for one GPU | Proportional CPUs for one GPU | Accelerator |
|---|---|---|---|---|---|---|---|---|
| plgrid (includes plgrid-long) | 532 + 256 (if not used by plgrid-bigmem) | 48 cores, Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz | 192GB | 184800MB | 3850MB | n/a | n/a | |
| plgrid-bigmem | 256 | 48 cores, Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz | 384GB | 369600MB | 7700MB | n/a | n/a | |
| plgrid-gpu-v100 | 9 | 32 cores, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz | 384GB | 368000MB | n/a | 46000M | 4 | 8x Tesla V100-SXM2 |
Job submission
Ares is using Slurm resource manager, jobs should be submitted to the following partitions:
| Name | Timelimit | Resource type (account suffix) | Access requirements | Description |
|---|---|---|---|---|
| plgrid | 72h | -cpu | Generally available. | Standard partition. |
| plgrid-testing | 1h | -cpu | Generally available. | High priority, Only for testing jobs, limited to 1 running or queued job and , 2 nodes maximum, high priority. |
| plgrid-now | 12h | -cpu | Generally available. | The highest priority, Intended for interactive jobs, limited to 1 running or queued job, and 1 node maximum, the highest priority. |
| plgrid-long | 168h | -cpu | Requires a grant with a maximum job runtime of 168h. | Used for jobs with extended runtime. |
| plgrid-bigmem | 72h | -cpu-bigmem | Requires a grant with CPU-BIGMEM resources. | Resources used for jobs requiring an extended amount of memory. |
| plgrid-gpu-v100 | 48h | -gpu | Requires a grant with GPGPU resources. | GPU partition. |
...
| Location | Location in the filesystem | Purpose |
|---|---|---|
| $HOME | /net/people/plgrid/<login> | Storing own applications, and configuration files. Limited to 10GB. |
| $SCRATCH | /net/ascratch/people/<login> | High-speed storage for short-lived data used in computations. Data older than 30 days can be deleted without notice. It is best to rely on the $SCRATCH environment variable. Uses transparent compression. |
| $PLG_GROUPS_STORAGE/<group name> | /net/pr2/projects/plgrid/<group name> | Long-term storage for data living for the period of computing grant. Should be used for storing significant amounts of data. Uses transparent compression. |
Current usage, capacity and other storage attributes can be checked by issuing the hpc-fs command.
Please note that if a storage space uses transparent compression, the data stored on physical disks is compressed on the fly. The compression might manifest as different sizes reported by utilities like ls and du, where ls by default reports apparent file size (the amount of data in the file), and du shows actual disk usage. In most cases, the apparent file size is larger than the disk usage. The actual disk usage (as reported by du) is used for quota calculation.
System Utilities
Please use the following commands to interact with the account and storage management system:
...