Inference Devices and Modes#

The OpenVINO™ Runtime offers several inference modes to optimize hardware usage. You can run inference on a single device or use automated modes that manage multiple devices:

single-device inference
This mode runs all inference on one selected device. The OpenVINO Runtime includes built-in plugins that support the following devices:
CPU
GPU
NPU
automated inference modes
These modes automate device selection and workload distribution, potentially increasing performance and portability:
Automatic Device Selection (AUTO)
Heterogeneous Execution (HETERO) across different device types
Automatic Batching Execution (Auto-batching): automatically groups inference requests to improve throughput

Learn how to configure devices in the Query device properties article.

Enumerating Available Devices#

The OpenVINO Runtime API provides methods to list available devices and their details. When there are multiple instances of a device, they get specific names like GPU.0 for iGPU. Here is an example of the output with device names, including two GPUs:

./hello_query_device Available devices:  Device: CPU ...  Device: GPU.0 ...  Device: GPU.1

See the Hello Query Device Sample for more details.

Below is an example showing how to list available devices and use them with multi-device mode:

ov::Core core; std::shared_ptr<ov::Model> model = core.read_model("sample.xml"); std::vector<std::string> availableDevices = core.get_available_devices(); std::string all_devices; for (auto && device : availableDevices) {  all_devices += device;  all_devices += ((device == availableDevices[availableDevices.size()-1]) ? "" : ","); } ov::CompiledModel compileModel = core.compile_model(model, "MULTI",  ov::device::priorities(all_devices));

If you have two GPU devices, you can specify them explicitly as “MULTI:GPU.1,GPU.0”. Here is how to list and use all available GPU devices:

ov::Core core; std::vector<std::string> GPUDevices = core.get_property("GPU", ov::available_devices); std::string all_devices; for (size_t i = 0; i < GPUDevices.size(); ++i) {  all_devices += std::string("GPU.")  + GPUDevices[i]  + std::string(i < (GPUDevices.size() -1) ? "," : ""); } ov::CompiledModel compileModel = core.compile_model("sample.xml", "MULTI",  ov::device::priorities(all_devices));

Additional Resources#