System configuration for GPU operation

To run AI models on a GPU under acceleration, the following requirements and configurations must be met.

Requirements

Use a Beckhoff image for IPCs with an ex factory installed GPU to ensure that all necessary requirements for the operation of the TwinCAT Machine Learning Server are already fulfilled.

The Beckhoff image already contains all the necessary components for operating the NVIDIA^® GPU
TCC mode is already activated (please note the restriction in the following sentence)

Note that TCC mode is only enabled on C6043 devices with GPU options without DisplayPort. Example 1: The C9900-A205 option is an NVIDIA RTX™ 2000 GPU without DisplayPort. TCC mode is already activated here. Example 2: The C9900-A206 option is an NVIDIA RTX™ 2000 GPU with DisplayPort. TCC mode is not activated here because the graphics output does not work when TCC mode is activated.

If you are operating a third-party PC, your own image or a Beckhoff IPC with a third-party GPU, please refer to

the section Installation of the NVIDIA® components
the section Configuration of the TCC mode

Installing the NVIDIA^®components

If no Beckhoff hardware with the associated Beckhoff image is used for the GPU-accelerated AI model execution, the customer is independently responsible for creating the necessary framework conditions on his system for operating the graphics card.

System configuration for GPU operation 1:

If you are using a Beckhoff IPC with GPU and associated Beckhoff image, you can skip this section.

System Requirements

System requirements are specific to NVIDIA^® components. The software versions listed below should not be newer than the specified versions (e. g. do not install CUDA^® 12.6), otherwise incompatibilities may occur.

NVIDIA^®Drivers

Normal NVIDIA^® GPU: Version 560.67
Quadro/RTX GPU: Version 522.86

CUDA^® (Compute Unified Device Architecture)

CUDA^® Version 12.5.1

cuDNN

cuDNN Version 9.2.1.18 (as zip file, do not use as Windows Installer)
Installation instructions can be found at NVIDIA^®: Tarball Installation.
It is necessary to add the path to the cuDNN libraries to the PATH variable of the system, not that of the user.

In general, the above measures meet the following system requirements:

Availability of the cuda.dll library
Availability of the nvml.dll library

Runtime behavior

When using third-party GPUs, considerable runtime fluctuations may occur during model execution, depending on the specific GPU and its settings.

The TwinCAT Machine Learning Server informs about possible challenges of the used graphics card in the log file immediately after starting the server.

Configuration of the GPU operation mode

You can set NVIDIA^® GPUs to what is known as TCC (Tesla Compute Cluster) mode, thereby reserving all of the GPU’s computing resources for computational tasks, such as the inference of an AI model.

System configuration for GPU operation 2:

Beckhoff IPCs with an ex factory installed GPU without DisplayPort are delivered in TCC mode.

What is the TCC mode?

The NVIDIA^® TCC mode is a special operation mode for NVIDIA^® GPUs in which the graphics output is disabled in order to optimize the GPU exclusively for computing tasks.

Note: Many CPUs are equipped with an integrated GPU (iGPU) so that a connected monitor can still be used despite TCC mode being enabled. This applies, for example, to the IPCs of the C6043 series.

Please note that TCC mode is not available for all GPUs. You can find information on the compatibility of your GPU directly from NVIDIA^®. All GPUs offered by Beckhoff support the TCC mode.

What are the advantages of TCC mode?

The entire memory and all computing units are available for AI calculations
No loss of time due to competing tasks such as rendering or display interrupts.
Improved response times and reduced jitter
More usable vRAM (especially important for large models or several models on one GPU).

How do I set the TCC mode?

Open the Windows Command Prompt with administrator rights. Enable the TCC mode with the command

nvidia-smi -g <GPU-Index> -dm 1

-g stands for the GPU ID (e.g. 0, 1, 2 …)

-dm 1 means switch on TCC mode

Examples

Enable TCC mode for the first GPU

nvidia-smi -g 0 -dm 1

Enable TCC mode for all compatible GPUs

nvidia-smi -dm 1

A restart of the system is recommended.