Proxmox Task Scheduling and Tuning

When NETLAB+ drives a Proxmox VE cluster, a single classroom action can launch many hypervisor operations at once: cloning a pod duplicates every virtual machine it contains, and the start of a large reservation can power on dozens of VMs together. Proxmox will accept only so much concurrent work before it begins rejecting operations with an overload error.

As of version 25.3.0, NETLAB+ includes a task scheduler that paces the work it sends to each Proxmox cluster so the cluster is kept busy without being overwhelmed. This section explains how the scheduler works and how to inspect and tune it.

Note

Task scheduling and tuning apply only to Proxmox VE datacenters. The pages described here are not shown for VMware vSphere datacenters.

How Task Scheduling Works

Every Proxmox operation NETLAB+ performs (a clone, a migration, a power action, and so on) is assigned a weight, a small whole number that estimates how demanding that operation is on the cluster. A full disk clone is expensive and carries a high weight; powering a VM on is cheap and carries the minimum weight of 1.

NETLAB+ tracks the combined weight of all the operations it currently has running and admits new ones only while there is room under two limits:

Host Capacity:

The maximum combined weight of tasks NETLAB+ will run at once on a single Proxmox host.

Datacenter Capacity:

The maximum combined weight of tasks NETLAB+ will run at once across all hosts in the datacenter.

When admitting the next operation would push either limit over its cap, NETLAB+ holds the operation in a queue and starts it as soon as capacity frees up. This happens automatically and requires no configuration; the default weights and capacities are tuned to work well on typical clusters.

Note

NETLAB+ also counts operations it did not start. A clone or backup launched directly from the Proxmox web interface consumes the same cluster resources, so it is tracked and counted against these limits; the Task Load dashboard lists such work with a source of External.

One kind of task is deliberately exempt: the open console session. A student viewing a virtual machine’s console appears on the cluster as a long-running task, but it places almost no load on the cluster, so NETLAB+ assigns console sessions a weight of zero. A classroom full of open consoles therefore cannot crowd out real work such as clones and power actions.

Auto-Tuning

The default weights are estimates. A given cluster’s storage, networking, and host hardware determine how much work it can really absorb, so NETLAB+ can adjust the weights to fit the cluster it is managing.

When Proxmox Auto-Tuning is enabled, NETLAB+ watches for overload errors from the cluster. Each time an operation is rejected for overload, NETLAB+ raises that operation’s weight slightly and retries, learning a higher, more realistic cost over time. Learned increases are bounded: a weight will not climb beyond four times its configured value, nor beyond what a single host can admit. If a weight reaches that ceiling, it is a strong signal that the cluster has a real capacity problem worth investigating.

Auto-tuning is off by default and is enabled per datacenter.

Note

Auto-tuning only ever raises weights, and only for the specific operations that overload the cluster. It never lowers a weight on its own. To return to your configured values, use the tuning form’s restore actions described below.

Enabling Auto-Tuning

Click on > Virtual Machine Infrastructure > Virtual Datacenters and Management Agents.

Click the datacenter you want to configure, then click Edit.

Select the Proxmox Auto-Tuning checkbox and click Submit.

Proxmox Auto-Tuning checkbox

Note

Changing the check box here on the datacenter Edit page does not discard what NETLAB+ has already learned; the learned values are simply frozen. Disabling auto-tuning stops further adjustments, and re-enabling it resumes from where it left off. The same check box also appears on the Task Tuning edit form, but saving that form commits a new weight baseline and clears the learned values whether or not you changed the check box; see Restoring Defaults.

Viewing Task Load

The Task Load dashboard shows NETLAB+’s live view of how busy a Proxmox cluster is: an overall datacenter load, a load figure for each host, and the list of tasks currently consuming scheduler capacity. It is a useful first stop when clones or pod operations seem slow, to see whether NETLAB+ is pacing work against a busy cluster.

Click on > Virtual Machine Infrastructure > Virtual Datacenters and Management Agents.

Click the datacenter you want to inspect, then click View Tasks.

View Tasks button on the datacenter page

The Task Load page is displayed. The load figures and task lists refresh automatically, about once per second. Refreshing pauses while the browser tab is in the background and slows down temporarily if the cluster stops responding.

Task Load dashboard

The dashboard is organized as follows.

Datacenter Load:

A bar and percentage at the top showing the combined weight of all running tasks across the datacenter relative to the datacenter capacity.

Host Tabs:

One tab per Proxmox host. Each tab shows that host’s load relative to the host capacity, and lists the tasks NETLAB+ is currently counting on it.

Open console sessions do not appear as individual rows; each host tab collapses them into a single summary line, for example 3 console session(s) open (weight 0).

Column Descriptions - Task Load

Task Type:

The Proxmox operation, such as qmclone (clone a virtual machine) or qmstart (power one on).

UPID:

The identifier Proxmox assigned the task, useful when correlating with the cluster’s own task log. A task still waiting in the queue has not started on the cluster yet, so this column shows -- for it.

Source:

NETLAB+ for operations NETLAB+ initiated; External for operations NETLAB+ discovered running on the cluster but did not start.

Status:

The task’s current state. A task NETLAB+ has admitted but the cluster has not yet reported shows scheduled; a task the cluster is executing shows running; a task held in the queue shows pending along with how long it has been waiting, for example pending (35s). If clones seem slow, pending rows are the first thing to look for: they show the scheduler pacing work against a busy cluster.

Weight:

The weight the task was admitted with (see How Task Scheduling Works).

% Host:

The share of the host capacity this task consumes. Shown as -- for queued tasks, which do not consume capacity yet.

% Datacenter:

The share of the datacenter capacity this task consumes. Also shown as -- for queued tasks.

A load near 100% means NETLAB+ is holding new operations in the queue until running ones finish. This is the scheduler working as intended; it is not an error. Persistently high load is a sign the cluster is a bottleneck for the work being asked of it.

To adjust the weights or capacity limits for this cluster, click Tuning. To return to the datacenter, click Dismiss.

Tuning Task Weights and Capacity

The Task Tuning page lets you inspect and override the weights and capacity limits for a Proxmox datacenter. Most sites never need to change these; adjust them only when you have a specific reason, such as a cluster that is unusually powerful or unusually constrained.

From the Task Load dashboard, click Tuning.

Task Tuning page

Click Edit to change the values.

Adjust the capacity limits and task weights as needed (see the field descriptions below), then click Submit.

Editing task tuning values

Each field also carries an on-screen help tip; click the help button on the form to show the tips inline.

Field Descriptions - Task Tuning

Proxmox Auto-Tuning:

Enables the automatic weight-learning feedback loop described in Auto-Tuning. The weights and capacities below apply whether or not this box is checked; the checkbox controls only the automatic adjustments.

Datacenter Capacity:

The maximum combined weight of concurrent tasks across all hosts in the datacenter. Must be greater than or equal to the host capacity. The shipped default is 100.

Host Capacity:

The maximum combined weight of concurrent tasks on a single host. Must not exceed the datacenter capacity, and no single task weight may exceed it. The shipped default is 40.

Task weights:

One field per Proxmox operation type, giving that operation’s weight. Higher weights make an operation count for more, so fewer of them run at once. Every weight must be a whole number of at least 1 and no greater than the host capacity.

About the Clone Weights

Three of the weight fields describe virtual machine cloning, which is usually the heaviest work NETLAB+ asks of a cluster and the most common reason to tune.

Full clone:

The weight for a full clone that NETLAB+ initiates (shown as qmclone:full). Proxmox copies every virtual disk into a fully independent VM. This is the most disk-intensive clone, so it carries the heaviest clone weight.

Linked clone:

The weight for a linked clone that NETLAB+ initiates (shown as qmclone:linked). The new VM shares its parent’s disks and stores only the blocks it changes. Almost no data is copied, so it is far cheaper than a full clone.

External clone:

The weight for a clone NETLAB+ found running on the cluster but did not start itself, for example one launched from the Proxmox web interface (shown as qmclone). Because NETLAB+ cannot tell whether such a clone is full or linked, this weight assumes the worst case, a full clone.

Note

The remaining weight fields correspond to other Proxmox operations (migrate, snapshot, template, power actions, and so on). The on-screen help tip for each field describes what the operation does and why it is weighted as it is.

Restoring Defaults

The Edit page provides two ways to reset the fields. Neither takes effect until you click Submit, so you can preview the values first, and Cancel discards them.

Restore User Defaults:

Clicking Restore User Defaults fills the fields with the values you last saved, discarding any adjustments auto-tuning has learned since. Use this to return the cluster to your own configured baseline.

Restore System Defaults:

Clicking Restore System Defaults fills the fields with the built-in values NETLAB+ ships with. Use this to start over from the factory settings.

Note

Saving the tuning form commits the values shown as your new baseline and clears all learned auto-tuning adjustments for the datacenter, not only the fields you changed. The new values apply to operations admitted after you save; work already running keeps the weight it was admitted with.