Version 25

Version 25 is a beta release of NETLAB+ VE that introduces support for Proxmox VE as a virtual machine backend. It will also support VMware as a backend, but VMware will not be the primary focus of development going forward.

Note

We recommend that clients who are not impacted by VMware license expirations continue to use the stable NETLAB+ version 24. We appreciate your patience and understanding during this transition.

Known Issues (Unresolved and/or Workarounds)

Note

Proxmox 9 launched in mid-August 2025, just as many institutions are switching to Proxmox and starting their Fall semester. We are delaying support for Proxmox VE 9 until our support staff can thoroughly test it and ensure stability for our users. Please use Proxmox VE8.4 for NETLAB+ deployments. When NDG does support Proxmox VE 9, there is an update process for upgrading Proxmox VE 8.4 to Proxmox VE 9 without reinstalling. We anticipate supporting Proxmox VE 9 prior to the Spring 2026 semester.

These are issues that have been identified in version 25. We are actively working to resolve these issues.

• Before submitting a bug report, please check the known issues here.

• Please send feedback and bug reports to support@netdevgroup.com.

Ref#	Severity	Feature	Description
D1411	High	Proxmox	Please run the NDG PVE Post Install Script to version 0.7 or later. This update creates new Proxmox CPU types to support virtual machines running “nested” hypervisors such VMware, Promxox, and Hyper-V running, on top of Proxmox bare metal hosts. NDG will begin updating its distributed virtual machines as required to make use of these new CPU types. Failure to re-run this script may result in unknown CPU compatibility issues with virtual machines that use nested hypervisors. This is a one-time operation that only needs to be performed on each Proxmox cluster.
D1324	Medium	Proxmox	Error service shim XXX not initialized or provide connection is nil may occur shortly after system starts. It will usually resolve as dependencies are initialized.
D1352	Medium	Proxmox	The cloning of the Master to the Template on the local server seems to work fine, but when it tries to migrate to one of the clusters hosts, it always fails with the error storage migration for 'NETLAB1:XXX/vm-XXX-disk-0.qcow2' to storage 'NETLAB1' failed - failed to run insecure migration: command /usr/bin/ssh
D1360	Medium	Proxmox	Individual VMs in a Proxmox pod sometimes fail to start. Logs indicate a timeout and error can't lock file /var/lock/qemu-server/lock-XXX.conf. May require a reboot of the host server and rollback of snapshots to ensure the VM starts clean next time the pod is started. The root root cause is not yet known, but may be load related. See also D1372.
D1364	Medium	Proxmox	Cloning a Proxmox pod does using a the PySDK (Python) does not automatically create snapshots as it does with VMware. Pending will fix.
D1372	Medium	Proxmox	Proxmox cluster operations may fail under high load and/or when cloning many virtual machines at the same time. This may result in various errors and lock issues that do not present consistently. NDG is working to identify and rectify these issues as they are encountered and reported. In the interim, please monitor your Proxmox cluster resources and avoid full cloning of multiple pods at the same time. Update: we have increased the Proxmox client timeout to help with this issue, but it may still occur (see #D1380).
D1373	Medium	Proxmox	Pod cloning error could not update datastore selection for vm XXX (index out of bounds) This was marked solved in 25.0.1 rc4, but may have resurfaced. Resolution pending.
D1383	Medium	PC Viewer	When viewing a Proxmox VM, switching from Actual Size to Scale Display viewer does not scale properly. Workaround: undock and dock the virtual machine.
D1391	Medium	Proxmox	Persistent VMs may not manage network bridges properly on Proxmox hosts. Pending, will fix.
D1400	Medium	Proxmox	VMs occasionally fail to start with error pc failed to configure vm, 500 got no worked upid - start worked failed. Multiple reports, investigating, resolution pending. #D1400, #D1172
D1401	Medium	Proxmox	Time skew between NETLAB+ and Proxmox may be reported in the server host view, event though NTP is properly configured and working. Investigating.
D1407	Medium	Proxmox	Suspend/Resume operations are not yet supported for Proxmox VMs. Pending, will fix.
D984	Advisory	Admin	The administrator home pages display that a beta software update is available to systems that are not participating in the beta program.
D1257	Advisory	VM Inventory	The NETLAB+ user interface does not currently allow snapshots to be taken on persistent VMs. This will be rectified in a future release. The workaround is to use the Proxmox or VMware directly to create snapshots for persistent VMs.
D1326	Advisory	Console	Warning “system.journal limits reached or header out-of-date” appears on the system console or in logs. This issue is self correcting and does not impact system functionality.
D1363	Advisory	Proxmox	Multiple NETLAB+ systems cannot share the same Proxmox cluster due to overlapping Proxmox VM IDs. A feature to control the ranges of VM IDs for each datacenter is planned.
D1369	Advisory	VMware	VMware backends may encounter new issues as the NETLAB+ API must now must route virtual infrastructure requests to either Proxmox or VMware based on the backend type.
D1370	Advisory	Logging	External log servers have not been well tested and may encounter issues.
D1371	Advisory	Prometheus	Prometheus exports have not been well tested and may encounter issues.
D1366	Advisory	Proxmox	The newly release Proxmox 9 Beta (Debian 13) is not yet supported. Proxmox 8.4 is the current recommended version for NETLAB+ 25. Preliminary testing indicates that in-place migration from Proxmox 8.4 to Proxmox 9 will be possible without reinstalling Proxmox on the host servers (so no need to rush).
D1368	Advisory	Installer	The NETLAB+ 25 installer image is not the latest release candidate and contains bugs that were subsequently fixed. Please update your system to the latest release candidate after installation from scratch.
D1374	Advisory	Proxmox	NETLAB+ does not currently track virtual machine power state changes made directly in Proxmox. In VMware this was handled by event notifications. Proxmox does not provide similar event notifications, so NETLAB+ will have to periodically poll Proxmox to determine the changes to vm state. Pending, will implement.
D1387	Advisory	Proxmox	NETLAB+ and other virtual machines running under Proxmox report high memory usage (> 90%) in the Proxmox console, even when idle. Proxmox uses a memory management technique called “ballooning” to optimize memory usage across all VMs. This is normal behavior for Proxmox and does not indicate a problem with NETLAB+ or the virtual machines. Linux VMs will use all available memory for caching to improve performance. The Proxmox host will start to reclaim memory from VMs (through ballooning) when the overall host memory usage reaches 80%. Below this amount, Proxmox does not attempt to reclaim memory. Ensure that your virtual machines have the QEMU guest agent installed and that balooning is enabled in the VM settings. Your NETLAB+ VM should already be configured for this. As far as we know, it is not necessary to lower the “minimum” memory setting for Proxmox VMs as the balloing driver will still reclaim memory as needed, however we will be investigating further for pod VMs.
D1393	Advisory	Proxmox	cannot import esxi storage process create storage failed: ‘NoneType’ object has no attribute ‘extraConfig’ (500)
D1403	Advisory	Proxmox	Proxmox VE VM names have the following restrictions: they must be a maximum of 63 characters, must start with a letter, can only contain letters (a-z, A-Z), numbers (0-9), hyphens (-), and cannot contain spaces or underscores. They must also adhere to DNS naming conventions for compatibility with internal and external services.
D1363	Advisory	Proxmox	Starting with version 25.0.1 rc5, NETLAB+ no longer uses the Proxmox next free VM_ID as this does not provide an atomic lock operation and can lead to VM ID collisions. The default range is 100 to 1000000. You may change this range by editing the VM Datacenter. This range does not apply to VMware datacenters. It is not recommended to use multiple NETLAB+ systems with a single Proxmox cluster, however, this feature could be used to ensure that the VM ID ranges do not overap; such deployments are considered experimental at this time.
D1389	Investigating	Pod Deletion	When deleting pods, received a warning received TERM signal... exiting followed by an error The task XXXX-XXXX-XXXX failed before it could return a taqk result, instead returned fatal error: failed, context canceled, did not signal completion.
D1392	Investigating	Pod Deletion	When deleting pods, NETLAB+ does not delete them from host with errors. Received error E_VM_HOST_NOT_FOUND.

Upgrading from Version 24

Please review Deployment Scenarios for information on how to upgrade your NETLAB+ VE system from version 24 to version 25.

25.0.1 rc6 (2025-09-09)

• Resolved fixed a bug that caused large, but valid, vm id ranges for datacenters to be declared invalid. #D1404

• Resolved clone operations ignoring datastore parameter when cloning to same host #D1406

• Resolved pod clone failures no longer result in orphaned VMs on the datacenter #D1402

• Resolved added proxmox targets to NDG Remote Support Service #D1398

25.0.1 rc5 (2025-09-05)

• Feature added Proxmox VM ID range settings to the VM Datacenter edit page #D1363

• Resolved NDG support proxy fails when installed fresh from 25.0.0 image #D1394

• Resolved VMs can now be cloned to a different datastore on the same host #1385

• Resolved limited pve client usage to one when performing ISO checks prior to running pod clone tasks #1381

• Resolved VM discovery task may return error Can't use an undefined value as an ARRAY reference at /opt/ndg/netlab-ve/web/vminventory.cgi line 578. #D1390

• Resolved Pod clone task will now reference the pod advanced settings if the provided PC clone spec does not define copy_bios_uuid #1395

• Resolved Pod cloning interface now updates to show completion on failure #1397

25.0.1 rc4 (2025-08-19)

• Feature display special build number along side version number on admin home page #D1377

• Feature added NLCONTROL as placeholder text for the Inside vSwitch Name field of the Add and Edit Host interfaces to suggest the correct value for Proxmox hosts #D1356

• Resolved fixed a bug preventing hosts from being added when an existing host in the same cluster is down #1351

• Resolved fixed a bug preventing the importation of VMs from a cluster with a downed host #D1358

• Resolved remove vCenter language in Virtual Datacenters and Management Agents description #D1376

• Resolved fixed intermittent error Can't use an undefined value as an ARRAY reference at /opt/ndg/netlab-ve/web/vminventory.cgi line 578 #D1361

• Resolved pod and vm cloning now produces an error if source vms have optical media and require migration #D1359

• Resolved fixed datastore issues related to pod cloning where target host datastores may be misrepresented or falsely invalid #1357

• Resolved the VM Inventory Clone interface will now update the datastore selection based on the selected host #1375

• Resolved software updater from rc3 hangs on kernel purges #D1378, #D1344

• Resolved mimimize client use during ISO pod cloning precheck #D1381

• Workaround increased promox client creation timeout to 30 seconds to help with cloning timeouts #D1380

25.0.1 rc3 (2025-08-01)

• Resolved fixed a bug that could cause progress to be reported to the wrong, sometimes inactive, task. #1170

• Resolved inactive NETLAB+ systems will not attempt to request catalog update from the CDN #1348

• Resolved the reservation manager now refreshes it’s list in response to RESERVATION.COMPLETED events #1114

• Resolved correct error in software version which may impact api calls #D1349

• Resolved content updates with a test status are no longer presented to non-testers. #1152

• Resolved the Screenshot PC tab menu option is now disabled if the feature is disabled at the community level #1167

• Resolved remove old kernels after software update #D1344

• Resolved replace references to mbusd process logic with netlabd in the software updater #D1343

25.0.1 rc2 (2025-07-28)

• Resolved datacenter clients and properties are now updated when fetched from the PVE monitor if they are more than 10 minutes old. #1345

• Resolved fix proxmox clone operations result in an “not authorized to access endpoint” error after some time. #1345

• Resolved fixed a logic bug that causes all VNIC adapters as type virtio. #1346

25.0.1 rc1 (2025-07-23)

• Resolved removed references to the word master in snapshots #1292

• Resolved corrected an error that occurred when cloning a VM from its current snapshot and other snapshot related improvements #1329

• Resolved corrected an ndg online authentication error #1335

• Resolved datacenter password input is no longer visible by default #1331

• Resolved measures have been taken to prevent Proxmox vmid collisions when cloning pods and VMs #1336

• Resolved when viewing VMs, disable buttons that are not applicable to the VM or its datacenter #1337

• Resolved apply latest security updates to o/s

• Resolved add host performance events for proxmox, disable recording to table #D1338

25.0.0 final (2025-07-07)

• Feature Support for Proxmox VE 8.4 for hosting NETLAB+ VE and virtual appliance and infrastructure.

25.0.0 rc15 (2025-07-07)

• Feature added a TEMPLATE pod category make it easier to work with pods consisting of template VMs #1272

• Resolved deleting pods that have attached VMs, with dependencies, now throws an error instead of deleting the pod and leaving the VMs in the inventory and on the datacenter #1305

• Resolved fixed an error caused by cloning pod ACLs when cloning pods #1323

• updated pod related VM deletion options to work with Proxmox #1320

• Resolved VM clone and remove tasks now update the parent VM’s child count accordingly #1327

• Resolved corrected some missed cases in VM name validations #1328

25.0.0 rc14 (2025-06-30)

• Resolved pve client pool should use vdc_id, not vdc_name #D1312

• Resolved vmproxy not updating with datacenter changes #D1293

• Resolved cannot not remove datacenter #D1304

• Resolved pve gost discover does not return errors #D1318

• Resolved vm host table is too congested #D1315

• Resolved clarify wording on inbound DNS test #D1310

• Resolved fix errors in vmproxy logs #D1156

• Resolved changes to datacenter are not resetting the client pool #D1222

• Resolved prometheus task authentication failure #D1322

• Resolved fix critical sysspd shim race condition / lock #D1321

• Resolved fixed a bug that prevented NETLAB+ PVE clients from being properly initialized #1317

• Resolved add ux pseudo link on host table

• Resolved change ndg remote support service name

25.0.0 rc13 (2025-06-23)

• Resolved Fixed a bug that caused the source VM to be the target of the configure task during pod cloning. #1306

• Resolved Clone pod VM datastore parameter is now being used. #1307

• Resolved When cloning pods, error message E_VM_NAME_INVALID now includes the invalid name. #1309

25.0.0 rc12 (2025-06-20)

• Feature add Proxmox to console setting verbiage #D1296

• Resolved qemu guest agent error should be warning #D1294

• Resolved Fixed a panic when pod cloning if a VM to be cloned had the same name as a VM already in the inventory #D1298

• Resolved Pod cloning now employs two different wait conditions to throttle the execution of the next VM clone routine #D1299

• Resolved Changed the pvemon VDC map key to the more reliable VDCID instead of VDCName. #D1300

• Resolved Resolved some compile warnings related to unused context Cancel functions in some pve test files. #D1302

• Resolved Pod Cloning progress reporting has been significantly improved. #D1279

• Resolved Console IP mislabeled as IPv6 #D1303

25.0.0 rc11 (2025-06-19)

• Resolved fix missing update of #D1278

25.0.0 rc10 (2025-06-18)

• Resolved vm_snapshots table not created in netlab installer image #D1278

• Resolved cannot not remove datacenter #D1285

• Resolved vmdist tag persists to children #D1284

• Resolved prevent vdcmon startup if not vsphere #D1290

• Resolved panic interface conversion: interface {} is *hdr.Event, not ntype.KeyVal #D1248

• Resolved type conversion error during host teardown step of pod delete operation #D1288

• Resolved safeguard to prevent VM ID collisions when cloning pods on Proxmox datacenters #D1283

25.0.0 rc9 (2025-06-17)

• Feature updated the VM Inventory Change Role feature to support Proxmox. #1253

• Resolved netlabd not starting after fresh install #D1273

• Resolved eliminated error condition that could occur when cloning a pod without a pod type ID. #1227

• Resolved setting IP Address at console shows errors #D1247

• Resolved network test failure on new netlab image deployment #D1275

• Resolved VM, cloning, templating, and role changing now update Proxmox tags correctly. #1270

• Resolved cannot change vm role #D1256

• Resolved remove unsupported options from pod removal for proxmox backends #D1263

• Resolved internal connections should allow authentication by acc_id #D1273

• Resolved disable data transfer in 25.0.0 #D1278

• Resolved reduced the severity of messages logged from Task Monitor. #1268

• Resolved update console disk resize verbiage #D1276

• Resolved allow template vm to attach to pod #D1271

25.0.0 rc8 (2025-06-16)

• Resolved netlabd.log not rotating #D1259

• Resolved unregister called on unknown proxy_id warning #D1261

• Resolved do not license check proxmox #D1254

• Resolved team reservation error when no team members have been assigned in class #D1250

• Resolved search filter missing from attach vm page #D1255

• Resolved host status is disconnected after adding host #D1221

• Resolved filter attachable vms based on pod’s common host or datacenter #D1199

• Resolved vm_datacenter_test_task fails for vsphere #D1258

• Resolved implement PROXY protocol and enforce ip addresses permitted by api token #D1192

• Resolved pysdk will not connect unless 127.0.0.1 is added to token #D1193

• Resolved double authentication in lab.cgi #D1236

• Resolved suppress kick admin error in log #D1220

• Resolved pod.pc.screenshot.count request throws and logs E_COMMUNITY_DENIED #D1265

25.0.0 rc7 (2025-06-15)

• Resolved missing local sig die handler in labspd tasks #D1235

• Resolved cannot unmarshal array into Go struct field TaskReportErrorParams.error of type string #D1246

• Resolved pve monitor lock functions assume initialized #D1247

• Resolved race condition in PVEMonitor initialization #D1219

• Resolved vm clone operations now remove distribution tags and add type and role tags #1241

• Resolved host teardown has an empty string in network list #D1251

• Resolved host teardown tearing down network before starting vm #D1200

• Resolved error during power off while deleting template pod #D1249

• Resolved not all vms start on pod init #D1252

25.0.0 rc6 (2025-06-14)

• Resolved reimplement TOTP validation in user.login.task #D1218

• Resolved change log “new search” button to “modify search” #D1203

• Resolved ensure loggers and progress reporters are set after acquiring client #D1234

• Resolved various fixes and improvements to the pod cloning interface for proxmox #D1194

• Resolved remove button_app_copy.png #D1191

• Resolved vm inventory UI should allow full clone of templates to other hosts #D1225

• Resolved fix various UI problems in vm inventory clone #D1225

• Resolved fix some bad type references in log output. #D1195

• Resolved corrections to trusted and external authorization #D1232

25.0.0 rc5 (2025-06-13)

• Resolved fix password check #D1214

• Resolved refactor user.login.task #D1214

• Resolved fixes to JSONRPC error handling #D1217

• Resolved incorrect tips for proxmox vdc edit state #1180

• Resolved add boot delay value to log messages #D1211

• Resolved teardown of inside vswitch should be a no-op #D1206

• Resolved do not delete linux bridges created by other systems #D1209

• Resolved check return values on host network lock #D1207

• Resolved Import of proxmox vm sometimes does not include node in vm_uuid #D1153

• Resolved update power off dialog for proxmox support #D1178

• Resolved change vmware specific error messages to generic #D1203

• Resolved use pvc.Host over pvc.Node when able #D1205

• Resolved fixed a bug that was causing errors when running tests under pve #D1208

• Resolved nil deref when cloning pods #D1196

25.0.0 rc4 (2025-06-12)

ndg-ve-netlab (25.0.0-4) ve-deb-12; urgency=medium

• Resolved E_METHOD_NOT_FOUND when cloning VMware pods. #D1200

25.0.0 rc3 (2025-06-11)

• Feature add proxmox vm id to ui views #1189

• Feature new support methods for proxmox virtual machines

• Resolved proxmox helper functions for perl and js

• Resolved vm inventory import is not importing template vms correctly for pve #D1198

• Resolved fix vm_uuid and vm_path in vm discovery and import #D1197

• Resolved fixes to netlab project top level make

25.0.0 rc2 (2025-06-10)

• Resolved improved PVE monitor initialization and handling of datacenter (cluster) connection state changes. #1185

• Resolved ix internal.vmspd.ping

• Resolved shim request should return jsonrpc 2.0 errors directly #D1190

• Resolved ntype.timestamp precision not honored in db valuer #D1186

• Resolved proxmox client timeout too long #D1183

• Resolved added vm.datacenter.test.task to netlabd #1179

• Resolved fix sdn reset

• Resolved the pc_os_id is now being set when cloning pod or inventory VMs

• Resolved add backing log files for netlabd and vmproxy

25.0.0 rc1 (2025-06-01)

• Feature Proxmox support!