Version 25¶
Version 25 is a beta release of NETLAB+ VE that introduces support for Proxmox VE as a virtual machine backend. It will also support VMware as a backend, but VMware will not be the primary focus of development going forward.
Note
We recommend that clients who are not impacted by VMware license expirations continue to use the stable NETLAB+ version 24. We appreciate your patience and understanding during this transition.
Known Issues (Unresolved and/or Workarounds)¶
Note
Proxmox 9 launched in mid-August 2025, just as many institutions are switching to Proxmox and starting their Fall semester. We are delaying support for Proxmox VE 9 until our support staff can thoroughly test it and ensure stability for our users. Please use Proxmox VE8.4 for NETLAB+ deployments. When NDG does support Proxmox VE 9, there is an update process for upgrading Proxmox VE 8.4 to Proxmox VE 9 without reinstalling. Proxmox VE 9 support is currently in testing and expected in an upcoming release.
These are issues that have been identified in version 25. We are actively working to resolve these issues.
Before submitting a bug report, please check the known issues here.
Please send feedback and bug reports to
support@netdevgroup.com.
Ref# |
Severity |
Feature |
Description |
|---|---|---|---|
D1352 |
|
Proxmox |
The cloning of the Master to the Template on the local server seems to work
fine, but when it tries to migrate to one of the clusters hosts, it always
fails with the error |
D1360 |
|
Proxmox |
Individual VMs in a Proxmox pod sometimes fail to start. Logs indicate a timeout
and error |
D1373 |
|
Proxmox |
Pod cloning error |
D1401 |
|
Proxmox |
Time skew between NETLAB+ and Proxmox may be reported in the server host view, event though NTP is properly configured and working. Investigating. |
D1453 |
|
Proxmox |
Proxmox developers have noted that attempting to hibernate (or snapshot
with RAM) a VM that has the |
D1411 |
|
Proxmox |
Please run the NDG PVE Post Install Script to version 0.7 or later. This update creates new Proxmox CPU types to support virtual machines running “nested” hypervisors such VMware, Promxox, and Hyper-V running, on top of Proxmox bare metal hosts. NDG will begin updating its distributed virtual machines as required to make use of these new CPU types. Failure to re-run this script may result in unknown CPU compatibility issues with virtual machines that use nested hypervisors. This is a one-time operation that only needs to be performed on each Proxmox cluster. |
D984 |
|
Admin |
The administrator home pages display that a beta software update is available to systems that are not participating in the beta program. |
D1257 |
|
VM Inventory |
The NETLAB+ user interface now supports snapshot management (add, remove, revert) on persistent VMs as of 25.1.0. Previously required direct access to Proxmox or VMware. |
D1326 |
|
Console |
Warning “system.journal limits reached or header out-of-date” appears on the system console or in logs. This issue is self correcting and does not impact system functionality. |
D1363 |
|
Proxmox |
Multiple NETLAB+ systems cannot share the same Proxmox cluster due to overlapping Proxmox VM IDs. A feature to control the ranges of VM IDs for each datacenter is planned. |
D1369 |
|
VMware |
VMware backends may encounter new issues as the NETLAB+ API must now must route virtual infrastructure requests to either Proxmox or VMware based on the backend type. |
D1370 |
|
Logging |
External log servers have not been well tested and may encounter issues. |
D1371 |
|
Prometheus |
Prometheus exports have not been well tested and may encounter issues. |
D1366 |
|
Proxmox |
Proxmox VE 9 (Debian 13) is not yet supported. Proxmox 8.4 is the current recommended version for NETLAB+ 25. Support is in testing and expected in an upcoming release. In-place migration from Proxmox 8.4 to Proxmox 9 will be possible without reinstalling Proxmox on the host servers (so no need to rush). |
D1368 |
|
Installer |
The NETLAB+ 25 installer image is not the latest release candidate and contains bugs that were subsequently fixed. Please update your system to the latest release candidate after installation from scratch. |
D1374 |
|
Proxmox |
NETLAB+ does not currently track virtual machine power state changes made directly in Proxmox. In VMware this was handled by event notifications. Proxmox does not provide similar event notifications, so NETLAB+ will have to periodically poll Proxmox to determine the changes to vm state. Pending, will implement. |
D1387 |
|
Proxmox |
NETLAB+ and other virtual machines running under Proxmox report high memory usage (> 90%) in the Proxmox console, even when idle. Proxmox uses a memory management technique called “ballooning” to optimize memory usage across all VMs. This is normal behavior for Proxmox and does not indicate a problem with NETLAB+ or the virtual machines. Linux VMs will use all available memory for caching to improve performance. The Proxmox host will start to reclaim memory from VMs (through ballooning) when the overall host memory usage reaches 80%. Below this amount, Proxmox does not attempt to reclaim memory. Ensure that your virtual machines have the QEMU guest agent installed and that balooning is enabled in the VM settings. Your NETLAB+ VM should already be configured for this. As far as we know, it is not necessary to lower the “minimum” memory setting for Proxmox VMs as the ballooning driver will still reclaim memory as needed, however we will be investigating further for pod VMs. |
D1393 |
|
Proxmox |
cannot import esxi storage process create storage failed: ‘NoneType’ object has no attribute ‘extraConfig’ (500) |
D1403 |
|
Proxmox |
Proxmox VE VM names have the following restrictions: they must be a maximum of 63 characters, must start with a letter, can only contain letters (a-z, A-Z), numbers (0-9), hyphens (-), and cannot contain spaces or underscores. They must also adhere to DNS naming conventions for compatibility with internal and external services. |
D1363 |
|
Proxmox |
Starting with version 25.0.1 rc5, NETLAB+ no longer uses the Proxmox next free VM_ID as this does not provide an atomic lock operation and can lead to VM ID collisions. The default range is 100 to 1000000. You may change this range by editing the VM Datacenter. This range does not apply to VMware datacenters. It is not recommended to use multiple NETLAB+ systems with a single Proxmox cluster, however, this feature could be used to ensure that the VM ID ranges do not overap; such deployments are considered experimental at this time. |
D1389 |
|
Pod Deletion |
When deleting pods, received a warning |
D1392 |
|
Pod Deletion |
When deleting pods, NETLAB+ does not delete them from host with errors.
Received error |
Upgrading from Version 24¶
Please review Deployment Scenarios for information on how to upgrade your NETLAB+ VE system from version 24 to version 25.
25.1.0 RC1 (beta) (2026-03-23)¶
Includes all changes from 25.0.2 release candidates.
FeatureProxmox host network visibility — pod and host admin pages now display information about NETLAB+-created Proxmox host networks. Administrators can inspect network topology without SSH access to the host. #1471FeaturePersistent VM snapshot management — the snapshot manager now supports add, remove, and revert operations on persistent VMs. Previously, persistent VM snapshots required direct access to Proxmox or VMware. #1257FeatureRemote PC revert reconfiguration — reverting a remote PC during a lab reservation now reconfigures the underlying VM for the current lab. #1470FeatureTask failure messaging — added alternate HDR messaging to report task startup failures for improved diagnostics. #1439ResolvedCloud-init drives are no longer incorrectly flagged as CD-ROMs during optical media testing. #1476ResolvedFixed VM cloning failures with certain storage configurations. #1477ResolvedFixed device interface translation that incorrectly mapped higher-order router interfaces. #1479ResolvedFixed host CPU brand detection issues. #1484ResolvedEliminated unnecessary Let’s Encrypt HTTP challenge redirects. #1485ResolvedScrolling restored in the Add Reservation interface. #1478ResolvedVM names are now included in optical media cloning error messages for better diagnostics. #1473ResolvedImproved event subscription authorization for lab reservations. #1483ResolvedApplied latest O/S and 3rd party package security updates. #1486
25.0.2 RC4 (beta) (2025-12-14)¶
Includes all changes from release candidates RC1 through RC3.
ResolvedFixed conflicting snapshot settings for remote PCs. #D1452ResolvedIncreased Diffie-Hellman key size to 4096 bits for improved security. #D1467ResolvedApplied latest O/S and 3rd party package security updates.ResolvedCloning a Proxmox pod using the PySDK (Python) does not automatically create snapshots as it does with VMware. Please upgrade PySDK environment to version 23.2.0 or later for this fix. #D1364
25.0.2 RC3¶
ResolvedFixed a bug that caused some lab tab icons to change incorrectly. #D1460ResolvedConnected PC viewers now poll for changes to remote display size. #D1457ResolvedReverting a proxmox remote pc no longer results in duplicate framing elements. #D1463ResolvedLog viewer time ranges now respect local time setting. #D1419FeatureThe lab topology now occupies the full height of the tab body. #1459
25.0.2 RC2¶
FeatureImplemented custom CPU types for Proxmox hosts:ndg-host,ndg-intel-host, andndg-amd-host. These CPU types pin processor settings that have been tested with virtual machines provided by NDG. NETLAB+ will now set virtual machines distgributed with typendg-hostto eitherndg-intel-hostorndg-amd-hostbased on the virtual machine’s host. Please be sure to run the NDG PVE Post Install Script to version 0.7 (or later) on each host. This will create the new Proxmox CPU types. #D1411AdvisoryThe custom pod directivenested_vm_support=1will now use CPU typendg-intel-hostorndg-amd-host. Previously, this the CPU type was set tohost. #D1411ResolvedDisplay Issue when going fromActualback toScale. #D1383ResolvedThe class reservation length can no longer be set to a value that exceeds the one set for the community. #D1231ResolvedThesuspendVM setting now works for Proxmox VMs. #D1407AdvisoryThesuspendsettings is currently not compatible withclipboard=vncand cannot be used together; NETLAB+ will perform agracedful shutdownwhenclipboard=vncis enabled. #D1453ResolvedFixed a bug that allowed a pod to start even when one or more of its VMs failed to start. Now, if any VM in a pod fails to start, the entire pod is suspended and the user will be given the option to retry. This is now consistent with previous VMware behavior. #D1450ResolvedContent tab sometimes fails to display content. #D1449ResolvedVM may occasionally fail to start with errorpc failed to configure vm, 500 got no worked upid - start worked failed. Multiple reports. #D1172, #D1400, #D1360
25.0.2 RC1 (2025-09-18)¶
FeatureUpdated the remote PC client for Proxmox datacenters. #D1410FeatureThe remote PC viewer now supports clipboard cut/paste for Proxmox virtual machines. #D1461AdvisoryTo overcome several limitations, NETLAB+ now manages networks on Proxmox hosts directly and no longer uses the Proxmox network management APIs. As a result, NETLAB+ managed networks will no longer appear in Promxox networking, but they can be inspected in/etc/network/interfaces.d/nlb*. This change relies on SSH. Please ensure that SSH is not blocked between the NETLAB+ virtual machine and Proxmox hosts. #D1428ResolvedPromiscous mode on virtual networks initially works when a pod starts but is lost when other pods come online and networking changes are applied on the host. NETLAB+ now manages networking directly on Proxmox hosts to resolve this issue. Please disable any workarounds that may try to manage this issue before upgrading to 25.0.2 as NETLAB+ network management no longer uses Proxmox networks and APIs. #D1429ResolvedProxmoxifreload -a failswith concurrent modifications causing some VMs not to start under load #D1412.ResolvedFixed an issue which prevented persitent VMs from binding to their network bridges during pod startup when using Proxmox hosts. #D1391AdvisoryProxmox developers have noted that attempting to hibernate (or snapshot with RAM) a VM that has theclipboard=vncoption enabled can “break” the clipboard upon resume. The current Proxmox API may actively prevent this combination. NETLAB+ will nowshutdownremote PCs configured to suspend at the end of a reservation if their VM’s display utilizes the VNC clipboard feature.ResolvedFocus lost indicator no longer causes display size changes. #D1410ResolvedFixed some display and resizing issues related to changing resize modes. #D1410ResolvedMenu options can now respond to power VM power state changes. #D1410ResolvedProxmox polyfills leave lingering ssh connections. #D1432ResolvedFixed a bug that caused Proxmox pod cloning to fail when the source pod contained VMs with no disks attached. #D1435ResolvedFixed a bug:encountered CODE(0x555dfee5e658), but JSON can only represent references to arrays or hashes at /opt/ndg/netlab-ve/inc/Netlab/Fluent.pm line 140. #D1420Resolveddo not log errorE_ALREADY_MEMBERwhen adding user to roster if already a member. #D1417ResolvedFixed a warning thatvdcmon_rescan() has no task#D1437ResolvedFixed Axios (JS) DOS vulnerability. #D1418ResolvedFixed jQuery (JS) vulnerabilities. #D1441ResolvedPod PC settings now enforces a revert to snapshot value of none for pcs with persistent or template VMs. #1427ResolvedThe revert to snapshot feature has been restored in the lab environment. #1433
25.0.1 RC6 (2025-09-09)¶
ResolvedFixed a bug that caused large, but valid, vm id ranges for datacenters to be declared invalid. #D1404ResolvedClone operations ignoring datastore parameter when cloning to same host #D1406ResolvedPod clone failures no longer result in orphaned VMs on the datacenter #D1402ResolvedAdded proxmox targets to NDG Remote Support Service #D1398
25.0.1 RC5 (2025-09-05)¶
Featureadded Proxmox VM ID range settings to the VM Datacenter edit page #D1363ResolvedNDG support proxy fails when installed fresh from 25.0.0 image #D1394ResolvedVMs can now be cloned to a different datastore on the same host #1385Resolvedlimited pve client usage to one when performing ISO checks prior to running pod clone tasks #1381ResolvedVM discovery task may return errorCan't use an undefined value as an ARRAY reference at /opt/ndg/netlab-ve/web/vminventory.cgi line 578.#D1390ResolvedPod clone task will now reference the pod advanced settings if the provided PC clone spec does not definecopy_bios_uuid#1395ResolvedPod cloning interface now updates to show completion on failure #1397
25.0.1 RC4 (2025-08-19)¶
Featuredisplay special build number along side version number on admin home page #D1377FeatureaddedNLCONTROLas placeholder text for the Inside vSwitch Name field of the Add and Edit Host interfaces to suggest the correct value for Proxmox hosts #D1356Resolvedfixed a bug preventing hosts from being added when an existing host in the same cluster is down #1351Resolvedfixed a bug preventing the importation of VMs from a cluster with a downed host #D1358Resolvedremove vCenter language in Virtual Datacenters and Management Agents description #D1376Resolvedfixed intermittent errorCan't use an undefined value as an ARRAY reference at /opt/ndg/netlab-ve/web/vminventory.cgi line 578#D1361Resolvedpod and vm cloning now produces an error if source vms have optical media and require migration #D1359Resolvedfixed datastore issues related to pod cloning where target host datastores may be misrepresented or falsely invalid #1357Resolvedthe VM Inventory Clone interface will now update the datastore selection based on the selected host #1375Resolvedsoftware updater from rc3 hangs on kernel purges #D1378, #D1344Resolvedmimimize client use during ISO pod cloning precheck #D1381Workaroundincreased promox client creation timeout to 30 seconds to help with cloning timeouts #D1380
25.0.1 RC3 (2025-08-01)¶
Resolvedfixed a bug that could cause progress to be reported to the wrong, sometimes inactive, task. #1170Resolvedinactive NETLAB+ systems will not attempt to request catalog update from the CDN #1348Resolvedthe reservation manager now refreshes it’s list in response to RESERVATION.COMPLETED events #1114Resolvedcorrect error in software version which may impact api calls #D1349Resolvedcontent updates with a test status are no longer presented to non-testers. #1152Resolvedthe Screenshot PC tab menu option is now disabled if the feature is disabled at the community level #1167Resolvedremove old kernels after software update #D1344Resolvedreplace references to mbusd process logic with netlabd in the software updater #D1343
25.0.1 RC2 (2025-07-28)¶
Resolveddatacenter clients and properties are now updated when fetched from the PVE monitor if they are more than 10 minutes old. #1345Resolvedfix proxmox clone operations result in an “not authorized to access endpoint” error after some time. #1345Resolvedfixed a logic bug that causes all VNIC adapters as type virtio. #1346
25.0.1 RC1 (2025-07-23)¶
Resolvedremoved references to the word master in snapshots #1292Resolvedcorrected an error that occurred when cloning a VM from its current snapshot and other snapshot related improvements #1329Resolvedcorrected an ndg online authentication error #1335Resolveddatacenter password input is no longer visible by default #1331Resolvedmeasures have been taken to prevent Proxmox vmid collisions when cloning pods and VMs #1336Resolvedwhen viewing VMs, disable buttons that are not applicable to the VM or its datacenter #1337Resolvedapply latest security updates to o/sResolvedadd host performance events for proxmox, disable recording to table #D1338
25.0.0 final (2025-07-07)¶
FeatureSupport for Proxmox VE 8.4 for hosting NETLAB+ VE and virtual appliance and infrastructure.
25.0.0 RC15 (2025-07-07)¶
Featureadded a TEMPLATE pod category make it easier to work with pods consisting of template VMs #1272Resolveddeleting pods that have attached VMs, with dependencies, now throws an error instead of deleting the pod and leaving the VMs in the inventory and on the datacenter #1305Resolvedfixed an error caused by cloning pod ACLs when cloning pods #1323updated pod related VM deletion options to work with Proxmox #1320
ResolvedVM clone and remove tasks now update the parent VM’s child count accordingly #1327Resolvedcorrected some missed cases in VM name validations #1328
25.0.0 RC14 (2025-06-30)¶
Resolvedpve client pool should use vdc_id, not vdc_name #D1312Resolvedvmproxy not updating with datacenter changes #D1293Resolvedcannot not remove datacenter #D1304Resolvedpve gost discover does not return errors #D1318Resolvedvm host table is too congested #D1315Resolvedclarify wording on inbound DNS test #D1310Resolvedfix errors in vmproxy logs #D1156Resolvedchanges to datacenter are not resetting the client pool #D1222Resolvedprometheus task authentication failure #D1322Resolvedfix critical sysspd shim race condition / lock #D1321Resolvedfixed a bug that prevented NETLAB+ PVE clients from being properly initialized #1317Resolvedadd ux pseudo link on host tableResolvedchange ndg remote support service name
25.0.0 RC13 (2025-06-23)¶
ResolvedFixed a bug that caused the source VM to be the target of the configure task during pod cloning. #1306ResolvedClone pod VM datastore parameter is now being used. #1307ResolvedWhen cloning pods, error message E_VM_NAME_INVALID now includes the invalid name. #1309
25.0.0 RC12 (2025-06-20)¶
Featureadd Proxmox to console setting verbiage #D1296Resolvedqemu guest agent error should be warning #D1294ResolvedFixed a panic when pod cloning if a VM to be cloned had the same name as a VM already in the inventory #D1298ResolvedPod cloning now employs two different wait conditions to throttle the execution of the next VM clone routine #D1299ResolvedChanged the pvemon VDC map key to the more reliable VDCID instead of VDCName. #D1300ResolvedResolved some compile warnings related to unused context Cancel functions in some pve test files. #D1302ResolvedPod Cloning progress reporting has been significantly improved. #D1279ResolvedConsole IP mislabeled as IPv6 #D1303
25.0.0 RC11 (2025-06-19)¶
Resolvedfix missing update of #D1278
25.0.0 RC10 (2025-06-18)¶
Resolvedvm_snapshots table not created in netlab installer image #D1278Resolvedcannot not remove datacenter #D1285Resolvedvmdist tag persists to children #D1284Resolvedprevent vdcmon startup if not vsphere #D1290Resolvedpanic interface conversion: interface {} is *hdr.Event, not ntype.KeyVal #D1248Resolvedtype conversion error during host teardown step of pod delete operation #D1288Resolvedsafeguard to prevent VM ID collisions when cloning pods on Proxmox datacenters #D1283
25.0.0 RC9 (2025-06-17)¶
Featureupdated the VM Inventory Change Role feature to support Proxmox. #1253Resolvednetlabd not starting after fresh install #D1273Resolvedeliminated error condition that could occur when cloning a pod without a pod type ID. #1227Resolvedsetting IP Address at console shows errors #D1247Resolvednetwork test failure on new netlab image deployment #D1275ResolvedVM, cloning, templating, and role changing now update Proxmox tags correctly. #1270Resolvedcannot change vm role #D1256Resolvedremove unsupported options from pod removal for proxmox backends #D1263Resolvedinternal connections should allow authentication by acc_id #D1273Resolveddisable data transfer in 25.0.0 #D1278Resolvedreduced the severity of messages logged from Task Monitor. #1268Resolvedupdate console disk resize verbiage #D1276Resolvedallow template vm to attach to pod #D1271
25.0.0 RC8 (2025-06-16)¶
Resolvednetlabd.log not rotating #D1259Resolvedunregister called on unknown proxy_id warning #D1261Resolveddo not license check proxmox #D1254Resolvedteam reservation error when no team members have been assigned in class #D1250Resolvedsearch filter missing from attach vm page #D1255Resolvedhost status is disconnected after adding host #D1221Resolvedfilter attachable vms based on pod’s common host or datacenter #D1199Resolvedvm_datacenter_test_task fails for vsphere #D1258Resolvedimplement PROXY protocol and enforce ip addresses permitted by api token #D1192Resolvedpysdk will not connect unless 127.0.0.1 is added to token #D1193Resolveddouble authentication in lab.cgi #D1236Resolvedsuppress kick admin error in log #D1220Resolvedpod.pc.screenshot.count request throws and logs E_COMMUNITY_DENIED #D1265
25.0.0 RC7 (2025-06-15)¶
Resolvedmissing local sig die handler in labspd tasks #D1235Resolvedcannot unmarshal array into Go struct field TaskReportErrorParams.error of type string #D1246Resolvedpve monitor lock functions assume initialized #D1247Resolvedrace condition in PVEMonitor initialization #D1219Resolvedvm clone operations now remove distribution tags and add type and role tags #1241Resolvedhost teardown has an empty string in network list #D1251Resolvedhost teardown tearing down network before starting vm #D1200Resolvederror during power off while deleting template pod #D1249Resolvednot all vms start on pod init #D1252
25.0.0 RC6 (2025-06-14)¶
Resolvedreimplement TOTP validation in user.login.task #D1218Resolvedchange log “new search” button to “modify search” #D1203Resolvedensure loggers and progress reporters are set after acquiring client #D1234Resolvedvarious fixes and improvements to the pod cloning interface for proxmox #D1194Resolvedremove button_app_copy.png #D1191Resolvedvm inventory UI should allow full clone of templates to other hosts #D1225Resolvedfix various UI problems in vm inventory clone #D1225Resolvedfix some bad type references in log output. #D1195Resolvedcorrections to trusted and external authorization #D1232
25.0.0 RC5 (2025-06-13)¶
Resolvedfix password check #D1214Resolvedrefactor user.login.task #D1214Resolvedfixes to JSONRPC error handling #D1217Resolvedincorrect tips for proxmox vdc edit state #1180Resolvedadd boot delay value to log messages #D1211Resolvedteardown of inside vswitch should be a no-op #D1206Resolveddo not delete linux bridges created by other systems #D1209Resolvedcheck return values on host network lock #D1207ResolvedImport of proxmox vm sometimes does not include node in vm_uuid #D1153Resolvedupdate power off dialog for proxmox support #D1178Resolvedchange vmware specific error messages to generic #D1203Resolveduse pvc.Host over pvc.Node when able #D1205Resolvedfixed a bug that was causing errors when running tests under pve #D1208Resolvednil deref when cloning pods #D1196
25.0.0 RC4 (2025-06-12)¶
ndg-ve-netlab (25.0.0-4) ve-deb-12; urgency=medium
ResolvedE_METHOD_NOT_FOUND when cloning VMware pods. #D1200
25.0.0 RC3 (2025-06-11)¶
Featureadd proxmox vm id to ui views #1189Featurenew support methods for proxmox virtual machinesResolvedproxmox helper functions for perl and jsResolvedvm inventory import is not importing template vms correctly for pve #D1198Resolvedfix vm_uuid and vm_path in vm discovery and import #D1197Resolvedfixes to netlab project top level make
25.0.0 RC2 (2025-06-10)¶
Resolvedimproved PVE monitor initialization and handling of datacenter (cluster) connection state changes. #1185Resolvedix internal.vmspd.pingResolvedshim request should return jsonrpc 2.0 errors directly #D1190Resolvedntype.timestamp precision not honored in db valuer #D1186Resolvedproxmox client timeout too long #D1183Resolvedadded vm.datacenter.test.task to netlabd #1179Resolvedfix sdn resetResolvedthe pc_os_id is now being set when cloning pod or inventory VMsResolvedadd backing log files for netlabd and vmproxy
25.0.0 RC1 (2025-06-01)¶
FeatureProxmox support!