List of Resource Allocation Competition 2024 Awards

• Summary
• Computational Resources
  • Minimum Size of RAC Requests and Opportunistic Compute Access
  • CPU Allocations
    • Scaling CPU Requests
  • GPU Allocations
  • Storage Allocations
  • Cloud Allocations
• Assessment Process
  • Guiding Principles
  • Technical Review
  • Science Review
• Monetary Value of the 2024 Allocations

The Alliance Federation delivers Canada’s national advanced research computing (ARC) platform in partnership with regional digital research infrastructure (DRI) organizations (Compute Ontario, Calcul Québec, ACENET, the BC DRI Group and the Prairie DRI Group) and institutions across Canada. Providing researchers with access to the infrastructure and expertise they need to accomplish globally competitive, data-driven and transformative research, this national ARC platform serves the needs of more than 20,000 researchers, including over 5,483 faculty based at Canadian institutions as of January 1, 2024. 

For the 2024-2025 allocation period, the total available capacity of the national ARC platform for the Resource Allocation Competition is 232,560 CPUs, 56,405 vCPUs (virtual CPUs), 4,237 Reference GPU units (RGU-years) and 210.7 PB of storage on Arbutus (University of Victoria), Cedar (Simon Fraser University), Graham (University of Waterloo), Niagara (University of Toronto), and Béluga and Narval (Calcul Québec). 

Ongoing growth in researcher demand for resources means that demand continues to outstrip supply. This year’s RAC was able to award, on average, 42% of the total compute requested, 76% of the total storage requested, 21% of the total RGU-years requested and 70% of the total vCPUs requested on the Arbutus, Béluga, Cedar and Graham clouds. The 2024 RAC received 670 projects submitted, 77 less applications than in the previous year mainly due to the increase in the minimum amount of resources that are required to be eligible to apply for the competition.

Note: The Alliance national systems will be replaced by spring 2025 as a result of $225M in capital funding received from federal, provincial and institutional sources. The transition process from current to new systems will necessitate service disruptions during the winter of 2025, although the Alliance will endeavour to minimise the effects on researchers. This transition may make it difficult for projects to achieve their full allocation over the year. Researchers should monitor the status pages (https://status.alliancecan.ca) for updates as we start receiving the new infrastructure and begin the transition process from the current systems to the new ones. The new systems will result in a significant increase of compute and storage resources available to researchers, as well as increased reliability and availability.

Table 1: Applications Submitted to the Resource Allocation Competition 

If you have questions about the terminology used in this page, please consult the Technical Glossary. If you have any questions about the overall report, contact allocations@tech.alliancecan.ca 

Minimum Size of RAC Requests and Opportunistic Compute Access

A minimum of compute resources (currently set at 200 core-years for CPU and 25 RGU-years for GPUs) is required to be eligible to submit a RAC application. These minimum values are set in part to control the number of applications requiring peer-review. A minimum RAC award will also ensure higher job priorities than for non-RAC awardees.

All researchers and their sponsored users with an active account can automatically make opportunistic use of CPU and GPU resources on any system. There is no guarantee on how much CPU or GPU can be consumed by non-RAC holders, as their use of the systems is purely opportunistic. 

Historical utilization data shows that many groups are able to reach (or even exceed) the RAC minimums specified above. Non-RAC users who want to maximize their compute usage need to consider strategies that ensure: 

• they regularly have jobs in the queue;
• are able to tolerate longer wait times for jobs to start; and,  
• submit jobs with “optimal” characteristics.

For example, opportunistic jobs with short time limits that request a few cores

on a general purpose (GP) system will generally run much sooner than those requesting dozens of cores. 

Please read this useful documentation about allocation scheduling priorities and job scheduling policies or contact support@tech.alliancecan.ca for advice on how to maximize usage for non-RAC awardees.

CPU Allocations

RAC 2024 was able to meet 43% of all of the CPU resources requested, 5% less than last year. Béluga, Cedar, Graham, Narval and Niagara provide approximately 232,560 cores, of which around 80% (on average) is allocated through the RAC.   

Table 2: 2024 CPU allocations per cluster

*The amount of resources allocated include resources that will be unavailable during the downtime needed to replace the old infrastructure. For example, an allocation of 100 cores for the 2024-2025 allocation period on a cluster that will have total cluster shutdown for infrastructure upgrades is reported as 100 core-years, even when it is only expected to work for 11 months. The unallocated portion (share) of the cluster made available to researchers for opportunistic use without a RAC award (that is, the difference between available and allocated) includes resources that will likely break because they have no support or due to planned or unforeseen outages, so this portion of the resources available is in practice much smaller, intermittent and unreliable than the numbers in the table may suggest. 

Table 3: Historical CPU ask vs. allocation

Scaling CPU Requests

As previously stated, there were insufficient ARC resources to fully meet the CPU demand through RAC 2024.  

As a result, a scaling function was applied to the 2024 competition to provide a means by which decisions on RAC allocations, in a context of insufficient capacity, could be made. This function, which is endorsed by the chairs of the review committees, was established so that only applications with a science score greater than 2.0 (out of 5) received an allocation. Applicants who did not receive a CPU allocation can still make opportunistic use of system resources via the Rapid Access Service. The average score of all of the applications submitted to the RAC 2024 was 3.9.

CPU requests are scaled based on the overall score of the application and the size of the request. Details and examples of the scaling function are available here. For further questions, contact allocations@tech.alliancecan.ca 
 

GPU Allocations

The Reference GPU Unit (RGU) was introduced as of RAC 2024 to request, allocate and measure the amount of GPU resources that are used. It represents the "cost" of utilizing a particular GPU model, whose RGU value varies based on performance. GPU allocations are from now on set in, and usage charged on, RGU-years and not in GPU-years. For more information about RGU, visit this page. Since this is a new unit, we are not able to provide historical numbers for GPU capacity, demand and supply, and that is why Table 4 uses RGU-years and Table 5 uses GPU-years.

The demand for GPU resources continues to be more competitive than for CPU resources. As Table 5 shows, the request for GPU resources has been stable in the last 4 years, but the gap between demand and capacity remains quite large. In 2024, the RGU allocation rate compared to the demand was 21%.

Table 4: 2024 GPU allocations per cluster (in RGU-years)

*The amount of resources allocated include resources that will be unavailable during the downtime needed to replace the old infrastructure. For example, an allocation of 10 GPU-years or 26 RGU-years for the 2024-2025 allocation period on a cluster that will have total cluster shutdown for infrastructure upgrades is reported as 10 GPU-years or 26 RGU-years, even when it is only expected to work for 11 months. The unallocated portion (share) of the cluster made available to researchers for opportunistic use without a RAC award (that is, the difference between available and allocated) includes resources that will likely break because they have no support or due to planned or unforeseen outages, so this portion of the resources available is in practice much smaller, intermittent and unreliable than the numbers in the table may suggest. 
 

Table 5: Historical GPU demand vs. supply (in GPU-years)

Scaling GPU requests

GPU allocations are determined by the following factors: 

• the overall score of the RAC application,
• the technical justification provided, 
• evidence of previous GPU utilization, 
• the research area of application for which GPUs are requested (e.g., Artificial Intelligence, machine learning, etc.), 
• the size of the research group.

Keep in mind the following:

• GPU allocations are constrained, among other things, by the type of GPU requested and available in each system.
• The demand for GPUs for AI applications has increased considerably.
• In general, RAC applicants find it difficult to estimate their GPU needs, which in most cases are over-requested (and underutilized). We strongly encourage future RAC applicants to do two things before applying: start using the GPUs in order to get a better understanding of your needs and consult with our technical staff before submitting a RAC application. Our staff can provide advice on how to benchmark your codes and calculate your GPU needs as accurately as possible.

Storage Allocations

Storage integrated with Arbutus, Béluga, Cedar, Graham, Narval and Niagara provided approximately 210.7 PB of storage capacity for 2024. This meant that, across all types of storage, 76% of the total storage capacity was allocated.

Table 6: Historical storage demand vs. supply

Table 7: 2024 Storage demand vs. supply by storage type

Cloud Allocations

The Arbutus cluster at the University of Victoria has 41,920 allocatable vCPUs (virtual CPUs). These are available via RAC and RAS and are also utilized for internal services such as software development and hosting.  Relatively small cloud offerings are also implemented on Cedar, Graham and Béluga. For RAC 2024, the request for compute vCPUs overall increased on average by 30%: a 30% increase in the request for compute vCPUs, and a 43% increase for persistent vCPUs. 

Overprovisioning of persistent vCPUs at a 10:1 ratio has allowed a significant expansion in persistent allocations in the previous few years. Demand for real cores has continued to increase, and aging equipment has begun to reduce available capacity; these pressures have pushed utilization numbers well over 80%, a threshold beyond which it becomes more difficult to schedule larger compute instances.   

Between Arbutus and the additional nodes on Cedar, Graham, and Béluga, this year’s RAC was able to allocate 70% of the total virtual CPUs requested, 7% less than in the previous year. 

Table 8: Historical Cloud vCPU Demand vs. Supply

The RAC involves two review processes each year:

• a scientific review, which is a peer-review process involving more than 100 discipline-specific experts from Canadian academic institutions. These volunteers assess and rate the merits of the computational research projects submitted. The scientific review results in a single score that provides a critical and objective measure to guide allocation decisions; and 
• a technical review that is undertaken by  staff who are responsible for verifying the accuracy of the computational resources needed for each project based on the technical requirements outlined in the application and for making recommendations about the national system to which the resources should be allocated to meet the project's needs.

The overall process is overseen by the Resource Access Program Administrative Committee, which includes representatives from each region and national system host sites.

Note that while new applications receive both scientific and technical reviews, applications submitted via the Fast Track process and Research Platforms and Portals (RPP) with a multi-year award receive only a technical review.

Guiding Principles

RAC is guided by the following principles:

• all applications are given fair consideration through both a scientific and technical review process;
• resources are awarded based on the merits of the computational research project presented, rather than the merits of the overall research program;
• there is no direct correlation between the amount of computational resources needed and the quality (excellence) of the research outcomes of a project - important research can be done with a small amount of computational resources; and 
• the challenges arising from the shortage of resources and other constraints within the system are shared among all applicants.

Technical Review

The technical review is conducted by technical experts who:

• ensure the appropriate system is requested by the PI;
• ensure that the required software is available;
• evaluate application efficiency and scalability;
• identify groups that may need help with application and workflow optimization;
• identify discrepancies between the online request and the complete description of the project;
• identify special software requirements; and,
• provide a technical opinion on the reasonableness of the request.

Technical reviewers are required to sign a Non-Disclosure Agreement prior to accessing any RAC application.

Science Review

New applications submitted to the RAC are peer-reviewed and scored. Scientific reviewers are required to sign a Non-Disclosure Agreement and accept the Conflict of Interest Policy prior to accessing any RAC application.

The final RAC score is based on the following: 

• the scientific excellence of the specific research project for which computational resources are being requested;
• the scientific and technical feasibility of the proposed research project;
• the appropriateness of the resources requested to achieve the project’s objectives; and,
• the likelihood that the resources requested will be efficiently used.

Applications are reviewed in one of the committees below: 

• Astronomy, Astrophysics and Cosmology
• Bioinformatics
• Chemistry, Biochemistry and Biophysics
• Computer Sciences and Mathematics
• Engineering
• Environmental and Earth Sciences
• Humanities and Social Sciences
• Nano, Materials and Condensed Matter
• Neurosciences, Medical Imaging and Medical Physics
• Subatomic Physics, Nuclear Physics and Space Physics

These values represent an average across the national ARC platform’s facilities and include total capital and operational costs incurred to deliver the resources and associated services. These are not commercial or market values. For the 2024 competition, the value of the resources allocated was calculated using the following rates:

Table 9: Financial Value of RAC Awards