Topic/distribute according to NUMA #2
Conversation
|
It's running without errors from what I see but please check again. |
| private var executerLocalIdMap = new TrieMap[InetSocketAddress, Int] | ||
|
|
||
| private def getExecuterLocalId(key: InetSocketAddress): Int = { | ||
| val executerLocalId = executerLocalIdMap.replace(key, executerLocalIdMap(key)+1) |
There was a problem hiding this comment.
If my understanding is correct, this function increases the "local id" each time it is called. Meaning, a getter function is not idempotent, but instead modifies data structs.
There was a problem hiding this comment.
Indeed name is confusing,getAndSet or other name implies that this function also change the value in the map will be better
| executorToDpuAddress += executorId -> dpuSockAddress | ||
|
|
||
| nvkvLock.getOrElseUpdate(SerializationUtils.deserializeInetAddress(dpuSockAddress.value), {0}) | ||
| executerLocalIdMap.getOrElseUpdate(SerializationUtils.deserializeInetAddress(dpuSockAddress.value), {0}) |
There was a problem hiding this comment.
Here the key is DPU sock addr. but in other places above seems the key is the executor address (?)
There was a problem hiding this comment.
Actually I always use the local DPU address to detect executers from the same node because it's already passed to the driver EP for advertising purposes.
I'm not using the host address at all, Check receiveAndReply, But you are right, in some place I declared vars with the name execAdd and it's wrong, need to change it to execDpuAdd or similar
2 participants
Distribute cores evenly between SPDK process