Main Content

2022 Resource Allocations Competition Results

Summary

Canada’s national advanced research computing (ARC) platform has been delivered through the Compute Canada Federation (CCF), which was a partnership of Compute Canada, regional organizations (WestGrid, Compute Ontario, Calcul Québec and ACENET) and institutions across Canada. Providing researchers with access to the infrastructure and expertise they need to accomplish globally competitive, data-driven, transformative research, this national ARC platform serves the needs of more than 19,000 users, including over 5,100 faculty based at Canadian institutions as of January 1, 2022. 

Compute Canada (CC) is in its final year of operations. As of April 1, 2022, the Digital Research Alliance of Canada (the Alliance), will take over the coordination and funding activities for Canada’s national ARC platform. CC, its regional and institutional partners, and the Alliance are committed to minimizing the disruption of service to researchers during this period of transition. 

The way users access the resources described in this report will remain the same. And, while CC will no longer exist, the members of the Compute Canada Federation (CCF) team will continue to be in place; therefore, users can also expect the same high level of support that they have always received. You may also notice that a number of key resources, such as the Documentation Wiki, will remain branded as Compute Canada. These resources are still valid and will be rebranded over time.

For RAC 2022, the national ARC platform provided approximately 293,312 CPU cores, 62,000 virtual CPUs, 3,062GPUs and 181 PB of storage on Arbutus (University of Victoria), Cedar (Simon Fraser University), Graham (University of Waterloo), Niagara (University of Toronto), and Béluga and Narval (Calcul Québec). 

Ongoing growth in researcher demand for resources means that demand continues to outstrip supply. The 2022 RAC competition received the highest number of applications in its history with 716 projects submitted — 10% more applications than 2021. This year’s RAC was able to award 54% of the total compute requested, 94% of the total storage requested, and 24% of the total GPUs requested. This year’s RAC was able to allocate 79% of the total vCPUs (virtual CPUs) requested on the Arbutus, Béluga, Cedar and Graham clouds.

While close to 80% (on average) of the resources available are allocated through the RAC, we reserve a target of 20% for researchers to use through the Rapid Access Service (RAS), which grants all users access to modest quantities of compute, storage, and cloud resources as soon as they have an account.

If you have questions about the terminology used in this page, please consult the Technical Glossary. If you have any questions about the overall report, contact allocations@tech.alliancecan.ca

Table 1: Applications submitted to the Resource Allocation Competitions

Year Applications submitted Year-over- year increase
2022 716 10%
2021 651 10%
2020 590 16%
2019 507 8%
2018 469 15%
2017 409 12%
2016 366 5%
2015 350 20%
2014 291 38%
2013 211 33%
Applications Submitted to the RAC (Chart)

*Note: This report is based on the information available as of February 9, 2022.

Computational Resources

CPU Allocations

Based on available computing resources, RAC 2022 was able to meet 54% of all of the CPU (core year) requested — this is the same allocation rate as last year. Béluga, Cedar, Graham, Narval and Niagara provide 100% of the available capacity or approximately 293,312 cores, of which close to 80% are available for RAC allocations.

Table 2: 2022 Compute Allocations per System

  Allocatable Core Years (100% capacity) Total CYrequested Total CY allocated % of CPU capacity allocated
Béluga 28,960 36,844 20,006 69%
Cedar 91,968 137,207 74,467 81%
Graham 34,784 49,387 25,556 73%
Narval 61,760 76,842 44,133 71%
Niagara 75,840 136,500 70,113 92%
Total 293,312 436,780 234,275 80%

Table 3: Historical Compute Ask vs. Allocation

  Allocatable CPU CY Total CYRequested Total CYallocated Difference
% of the demand awarded
2022 293,312 436,780 234,275 202,505 54%
2021 232,704 468,498 188,925 279,573 40%
2020 232,704 455,892 181,502 274,390 40%
2019 201,320 390,352 157,262 233,089 40%
2018 211,020 284,347 158,612 129,325 56%
2017 182,760 255,638 148,100 107,538 58%
2016 155,952 237,862 128,463 109,399 54%
2015 161,888 191,690 123,699 67,991 65%
2014 190,466 172,989 133,508 39,481 77%
2013 187,227 142,106 126,677 15,429 89%
CPU Allocation Trends (Chart)

Scaling CPU Requests

As described above, there were insufficient ARC resources to fully meet the CPU demand through RAC 2022.

As a result, a scaling function was applied to the 2022 competition to provide a means by which decisions on RACallocations, in a context of insufficient capacity, could be made. This function, which is endorsed by the chairs of the review committees, was established so that only applications with a science score of 3.0 or higher (out of 5) received an allocation. Applicants who did not receive a compute allocation can still make opportunistic use of system resources via the Rapid Access Service. The average score of all of the applications submitted to the RAC2022 was 3.7.

CPU requests are scaled based on the overall score of the application and the size of the request. Details and examples of the scaling function are available here. For further questions, contact allocations@tech.alliancecan.ca.

Minimum size of RAC requests and opportunistic compute access

All researchers and their sponsored users with an active account can, as described above, automatically make opportunistic use of CPU and GPU resources on any system. (This ability to compute without an allocation is also referred to as the Rapid Access Service). There is no guarantee on how much CPU or GPU can be consumed by non-RAC holders, as their use of the systems is purely opportunistic.

The RAC application process requires a minimum ask amount for compute resources (currently set at 50 core-years for CPU and 10 GPU-years for GPUs). These minimum values are set in part to control the number of applications requiring peer-review. As the number of applications increases every year, the workload of the expert review committees increases proportionally. Therefore, a RAC award made at the minimum level will ensure higher job priorities than for any non-RAC awardee.

While usage by non-RAC awardees is opportunistic, historical utilization data shows that many groups are able to reach (or even exceed) the RAC minimums specified above. Non-RAC users who want to maximize their compute usage need to consider strategies that ensure:

  • they regularly have jobs in the queue;

  • are able to tolerate longer wait times for jobs to start; and,

  • submit jobs with ​“optimal” characteristics.

For example, opportunistic jobs with short time limits that request a few cores on a general purpose (GP) system will generally run much sooner than those requesting dozens of cores.

Please read this useful documentation about allocation scheduling priorities and job scheduling policies or contact support@tech.alliancecan.ca for advice on how to maximize usage for a non-RAC awardee.

GPU Allocations

The demand for GPU resources continues to be more competitive than for CPU resources. As Tables 4 and 5 show, requests for GPUs have increased more than 3 fold in the last 5 years. The allocation rate in 2022 was 24%.

GPU allocations are determined by the following factors: the overall score of the RAC application, the technical justification provided, evidence of previous GPU utilization, the research area of application for which GPUs are requested (e.g., Artificial Intelligence, machine learning, etc.), and the size of the research group.

Keep in mind the following:

  1. In general, RAC applicants find it difficult to estimate their GPU needs, which in most cases are over-requested (and underutilized). We strongly encourage future RAC applicants to do two things before applying: start using the GPUs in order to get a better understanding of their needs and consult with our technical staff. Our staff can provide advice on how to benchmark your codes and calculate your GPU needs as accurately as possible.

  2. GPU allocations are constrained, among other things, by the type of GPU (Pascal, Volta, T4) requested and available in each system.

  3. The demand for GPUs for AI applications has increased considerably.

Table 4: 2022 GPU Year Allocations per System

  Allocatable GPUYears (100% capacity) Total GPU Years requested Total GPU Years allocated % of GPU capacity allocated
Béluga 688 2,108 426 62%
Cedar 1,352 3,964 1,072 79%
Graham 498 1,634 268 54%
Narval 524 1,364 395 75%
Total 3,062 9,070 2,161 71%

Table 5: Historical GPU demand vs. supply (GPU years)

  Allocatable GPU Years GPU Years requested Total GPUYears allocated Difference % of the GPUneed awarded
2022

3,062

9,070 2,161 6,909 24%
2021 2,610 9,980 2,187 7,370 22%
2020 2,552 12,885 1,936 11,165 15%
2019 1,664 6,555 1,331 5,224 20%
2018 976 4,092 840 3,252 20%
2017 1,420 2,790 1,047 1,743 38.5%
2016 373 1,357 269 1,088 20%
2015 482 608 300 308 49%
2014 NA 420 308 112 73%
2013 NA 390 259 131 66%
GPU Allocation Trends (Chart)

Cloud Allocations

The Arbutus cluster at the University of Victoria has 41,920 allocatable vCPUs (virtual CPUs). These are available via RAC and RAS and are also utilized for internal services such as software development and hosting. Relatively small cloud offerings are also implemented on Cedar, Graham and Béluga. For RAC 2022, the request for vCPUs overall increased by 14%: a 14% increase in compute vCPUs and a 21% increase in persistent vCPUs.

Overprovisioning of persistent vCPUs at a 10:1 ratio has allowed a significant expansion in persistent allocations in the previous few years. Demand for real cores has continued to keep pace with hardware expansions with utilization numbers well over 80%, a threshold beyond which it becomes more difficult to schedule larger compute instances.

Between Arbutus and the additional nodes on Cedar, Graham, and Béluga, this year’s RAC was able to allocate 79% of the total virtual CPUs requested.

Table 6: Historical cloud VCPU demand vs supply

  Allocatable VCPUYears(100% capacity) Total VCPUYears requested Total VCPUYears allocated Difference % of capacity allocated
2022

62,549

34,536 27,444 7,092 79%
2021 62,549 30,323 24,443 5,880 81%
2020 50,501 18,330 18,229 101 99%
2019 29,147 19,479 18,511 968 95%
2018 24,854 12,480 11,829 651 95%
Cloud Allocation Trends (Chart)

Storage Allocations

Storage integrated with Arbutus, Béluga, Cedar, Graham, Narval and Niagara provided approximately 181 PB of storage capacity for 2022. This meant that, across all types of storage, a total of 94% of available storage capacity was allocated.

Table 7: 2022 Storage Need vs. Supply by Storage Type (TB)

Category Type Supply (TB) Storage requested (TB) Storage allocated (TB) % of the demand awarded
HPC Project 66,230 61,715 51,571 84%
HPC Nearline 87,800 72,077 73,620 102%
HPC dCache 13,680 13,091 13,090 100%
Cloud Volumes and snapshot storage 4,980 3,562 2,948 83%
Cloud Object storage 7,000 7,325 7,315 100%
Cloud Shared cloud storage 2,000 1,628 1,618 99%
  Total 181,690 159,398 150,162 94%

Assessment Process

Compute Canada established the annual Resource Allocation Competition (RAC) in response to the Canada Foundation for Innovation’s (CFI) requirement to award advanced research computing (ARC) resources based on the scientific merits of the research proposed. In addition to this requirement, we must evaluate the technical merits of allocation requests. 

To meet these two requirements, RAC involves two review processes each year:

  • a scientific review, which is a peer-review process involving more than 100 discipline-specific experts from Canadian academic institutions. These volunteers assess and rate the merits of the computational research projects submitted. The scientific review results in a single score that provides a critical and objective measure to guide allocation decisions; and 

  • a technical review that is undertaken by staff who are responsible for verifying the accuracy of the computational resources needed for each project based on the technical requirements outlined in the application and for making recommendations about the national system to which the resources should be allocated to meet the project’s needs.

The overall process is overseen by the Resource Access Program Administrative Committee, which includes representatives from each region and national system host sites.

Note that while new applications receive both scientific and technical reviews, applications submitted via the Fast Track process and Research Platforms and Portals (RPP) with a multi-year award receive only a technical review.

Guiding Principles

RAC is guided by the following principles:

  • all applications are given fair consideration through both a scientific and technical review process;
  • resources are awarded based on the merits of the computational research project presented, rather than the merits of the overall research program;
  • there is no direct correlation between the amount of computational resources needed and the quality (excellence) of the research outcomes of a project – important research can be done with a small amount of computational resources; and
  • the challenges arising from the shortage of resources and other constraints within the system are shared among all applicants.

Technical Review

The technical review is conducted by technical experts who:

  • ensure the appropriate system is requested by the PI and the required software is available;
  • evaluate application efficiency and scalability;
  • identify groups that may need help with application and workflow optimization;
  • identify discrepancies between the online request and the complete description of the project;
  • identify special software requirements; and,
  • provide a technical opinion on the reasonableness of the request.

Technical reviewers are required to sign a Non-Disclosure Agreement prior to accessing any RAC application.

Science Review

New applications submitted to the RAC are peer-reviewed and scored. Scientific reviewers are required to sign a Non-Disclosure Agreement and accept the Conflict of Interest Policy prior to accessing any RAC application.

The final RAC score is based on the following:

  • the scientific excellence of the specific research project for which computational resources are being requested;
  • the scientific and technical feasibility of the proposed research project;
  • the appropriateness of the resources requested to achieve the project’s objectives; and,
  • the likelihood that the resources requested will be efficiently used.

Applications are reviewed in one of the committees below:

  • Astronomy, Astrophysics and Cosmology
  • Bioinformatics
  • Chemistry, Biochemistry and Biophysics
  • Computer Sciences and Mathematics
  • Engineering
  • Environmental and Earth Sciences
  • Humanities and Social Sciences
  • Nano, Materials and Condensed Matter
  • Neurosciences, Medical Imaging and Medical Physics
  • Subatomic Physics, Nuclear Physics and Space Physics

Monetary Value of the 2022 Allocations

These values represent an average across the national ARC platform’s facilities and include total capital and operational costs incurred to deliver the resources and associated services. These are not commercial or market values. For the 2022 competition, the value of the resources allocated was calculated using the following rates:

Table 8: Financial Value of RAC Awards

Resources 2022
1 core year $124
1 GPU year $2,460
1 TB of project storage / year $56
1 TB of nearline / year $26
1 VCPU $43
1 VGPU year $615
1 TB of volume and snapshot storage (Ceph) / year $49
1 TB of object storage $49
1 TB of shared filesystem storage $49