[DRAFT] Allow EP to provide additional HW devices #26234
Conversation
… metadata entry to indicate who created the device instance
adrianlizarraga
changed the title
| * \param[in] metadata Optional OrtKeyValuePairs instance for hardware device metadata that may be queried by | ||
| * applications via OrtApi::GetEpDevices() or the EP factory that receives this hardware device | ||
| * instance as input to OrtEpFactory::GetSupportedDevices(). | ||
| * Refer to onnxruntime_ep_device_ep_metadata_keys.h for common OrtHardwareDevice metadata keys. |
There was a problem hiding this comment.
nit: does ORT take ownership and EP does not need to release metadata KVP instance if provided?
| * \note New additional devices created by this EP factory are not provided to other EP factories. Only this | ||
| * EP factory receives the new additional hardware devices via OrtEpFactory::GetSupportedDevices(). | ||
| * Any OrtEpDevice instances that this EP factory creates with an additional hardware device are visible to | ||
| * applications that call OrtApi::GetEpDevices(). |
There was a problem hiding this comment.
Do we need this complexity?
Alternatively, if we're only going to show the OrtEpDevice to the EP that created it, can the EP create it inside of GetSupportedDevices?
e.g. inside GetSupportedDevices
- EP determines a virtual device is required
- EP creates OrtHardwareDevice.
- EP uses that to create OrtEpDevice.
EP could register the OrtHardwareDevice with ORT so we can release it when the EP is unregistered, or EP could own it if we provided a ReleaseHardwareDevice function. Only place the OrtHardwareDevice shows up is indirectly in the OrtEpDevice.
There was a problem hiding this comment.
I think that helps with complexity, but it lacks a signal for the EP to know it should create a virtual device.
GetAdditionalHardwareDevices is better, but that doesn't have a signal for device type (e.g. create a GPU or NPU or both).
Or do we expect it creates all possible virtual devices by default?
| cpu_device.vendor_id = cpuid_info.GetCPUVendorId(); | ||
| cpu_device.device_id = 0; | ||
| cpu_device.type = OrtHardwareDeviceType_CPU; | ||
| cpu_device.metadata.Add(kOrtHardwareDevice_MetadataKey_DiscoveredBy, "ONNX Runtime"); |
There was a problem hiding this comment.
Do we need this as well as IsVirtual? Wary of having too much stuff in metadata, and possibly the IsVirtual implies it was not discovered by ORT unless there's some other feature that will rely on an explicit 'discovered by' value.
There was a problem hiding this comment.
My thought is that:
DiscoveredBytells the application who discovered the device. May be useful if the app only wants stuff discovered by ORT.IsVirtualis "1" if it does not correspond to actual hardware. It's the negation ofIsHardwareused by dxcore api. This is a separate concept fromDiscoveredBy. This would be useful for off-target compilation, for example.
There was a problem hiding this comment.
If we only show the virtual device to the EP that created it, is there value in DiscoveredBy?
|
How does an EP know what sort of device it should create? e.g. an arm64 vs x64 virtual device. Would it create a generic virtual device so we have an OrtEpDevice for it, that gets explicitly selected using SessionOptionsAppendExecutionProvider_V2, and the EP options that can be provided there are used to specify things like the target architecture etc. if this is a cross compiling scenario? |
2 participants
Description
DRAFT. Not ready for detailed review.
Motivation and Context