Network isolation creation error

[Hossein4185]

problem

I get an error when I try to create an isolated network.

2025-09-23 10:03:48,650 DEBUG [c.c.n.r.NetworkHelperImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) Allocating the VR with id=5337 in datacenter Zone {"id": "3", "name": "PVCL.TEH.IR", "uuid": "d04f1ed7-09e9-4beb-853b-b4f4adbce9bd"} with the hypervisor type VMware
2025-09-23 10:03:48,679 DEBUG [c.c.n.r.NetworkHelperImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) VMware won't support system vm, skip it
2025-09-23 10:03:48,690 DEBUG [o.a.c.n.r.d.RouterDeploymentDefinition] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) Lock is released for network [id: 210] (Network {"id": 210, "name": "amin-test", "uuid": "ae4e93f1-0536-4d65-a0a0-dccfa2a3ed9f", "networkofferingid": 115}) as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(3)-Pod(null)-Cluster(null)-Host(null)-Storage()]
2025-09-23 10:03:48,698 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) Failed to implement network Network {"id": 210, "name": "amin-test", "uuid": "ae4e93f1-0536-4d65-a0a0-dccfa2a3ed9f", "networkofferingid": 115} elements and resources as a part of network restart due to com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:3] is unreachable: Can't find all necessary running routers!
at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:250)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1753)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1678)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:3719)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2955)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2977)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy200.restartNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:97)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

2025-09-23 10:03:48,699 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) Network Network {"id": 210, "name": "amin-test", "uuid": "ae4e93f1-0536-4d65-a0a0-dccfa2a3ed9f", "networkofferingid": 115} failed to restart.
2025-09-23 10:03:48,733 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Complete async job-52623, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to restart network"}
2025-09-23 10:03:48,735 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Publish async job-52623 complete on message bus
2025-09-23 10:03:48,735 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Wake up jobs related to job-52623
2025-09-23 10:03:48,735 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Update db status for job-52623
2025-09-23 10:03:48,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Wake up jobs joined with job-52623 and disjoin all subjobs created from job- 52623
2025-09-23 10:03:48,751 DEBUG [c.c.a.ApiServer] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Retrieved cmdEventType from job info: NETWORK.RESTART
2025-09-23 10:03:48,775 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Done executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for job-52623
2025-09-23 10:03:48,775 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Remove job-52623 from job monitoring
2025-09-23 10:03:49,217 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-38f7c6c5]) (logid:c3bcc01d) Found 0 routers to update status.
2025-09-23 10:03:49,222 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-38f7c6c5]) (logid:c3bcc01d) Found 1 VPC's to update Redundant State.
2025-09-23 10:03:49,224 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-38f7c6c5]) (logid:c3bcc01d) Found 0 networks to update RvR status.
2025-09-23 10:03:49,459 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-deea6087]) (logid:40ac84bb) DbCollector is running...
2025-09-23 10:03:49,580 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-82351737]) (logid:070d357e) HA health check task is running...
2025-09-23 10:03:49,802 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8e48fac]) (logid:c375191b) No inactive management server node found
2025-09-23 10:03:49,802 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8e48fac]) (logid:c375191b) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms

versions

Cloudstack Management 4.21
Vmware 8.0.3

[kiranchavala]

@Hossein4185 Looks like cloudstack was not able to reach the virtual router on vmware

Could check if the virtual router is running on vcenter and is reachable from management server

[Hossein4185]

@kiranchavala
Running on host but unreachable on cloudstack.

2025-09-29 17:43:27,305 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-0415c95b, ctx-61551cf1]) (logid:de53d393) ===END=== 172.17.0.247 -- GET jobId=e73a213c-53bd-4f8c-a05f-3d771e7f9ee3&command=queryAsyncJobResult&response=json&
2025-09-29 17:43:27,479 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-040112ca]) (logid:3a198344) ===START=== 172.17.0.247 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:27,480 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-040112ca]) (logid:3a198344) Two factor authentication is already verified for the user 12, so skipping
2025-09-29 17:43:27,497 DEBUG [c.c.a.ApiServer] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) CIDRs from which account 'Account [{"accountName":"ho.karami","id":9,"uuid":"aa8fee67-15f6-41c8-a5e6-4cf64bd0878e"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-09-29 17:43:27,503 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) Account for user id 13a153b4-c31b-4a3a-833f-d895c29928c3 is Root Admin or Domain Admin, all APIs are allowed.
2025-09-29 17:43:27,504 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-09-29 17:43:27,504 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-09-29 17:43:27,516 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) ===END=== 172.17.0.247 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:27,775 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7297205]) (logid:e90196b4) Found 1 inactive management server node based on timestamp
2025-09-29 17:43:27,775 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7297205]) (logid:e90196b4) management server node msid: 345050417779, name: clst02.pvcl.afracloud.local, service ip: 172.24.48.134, version: 4.21.0.0
2025-09-29 17:43:27,775 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7297205]) (logid:e90196b4) Peer scan is finished. profiler: Done. Duration: 7ms , profilerQueryActiveList: Done. Duration: 4ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms
2025-09-29 17:43:28,158 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-d2786022]) (logid:e4d9f092) HA health check task is running...
2025-09-29 17:43:29,276 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e055b2aa]) (logid:8d29f66a) Found 1 inactive management server node based on timestamp
2025-09-29 17:43:29,276 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e055b2aa]) (logid:8d29f66a) management server node msid: 345050417779, name: clst02.pvcl.afracloud.local, service ip: 172.24.48.134, version: 4.21.0.0
2025-09-29 17:43:29,276 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e055b2aa]) (logid:8d29f66a) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 2ms
2025-09-29 17:43:29,420 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-b990b648]) (logid:e83af821) ===START=== 172.17.2.60 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:29,420 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-b990b648]) (logid:e83af821) Two factor authentication is already verified for the user 9, so skipping
2025-09-29 17:43:29,435 DEBUG [c.c.a.ApiServer] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) CIDRs from which account 'Account [{"accountName":"a.bahri","id":15,"uuid":"95be6584-640d-434d-8a1f-7df726a918ae"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-09-29 17:43:29,443 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) Account for user id e33a25d0-28c7-48df-b19f-131956f501a4 is Root Admin or Domain Admin, all APIs are allowed.
2025-09-29 17:43:29,444 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-09-29 17:43:29,444 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-09-29 17:43:29,458 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) ===END=== 172.17.2.60 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:29,961 DEBUG [c.c.a.ApiServlet] (qtp1404565079-13333:[ctx-d1bd9e73]) (logid:e197f43d) ===START=== 172.17.2.60 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:29,961 DEBUG [c.c.a.ApiServlet] (qtp1404565079-13333:[ctx-d1bd9e73]) (logid:e197f43d) Two factor authentication is already verified for the user 9, so skipping
2025-09-29 17:43:29,977 DEBUG [c.c.a.ApiServer] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) CIDRs from which account 'Account [{"accountName":"a.bahri","id":15,"uuid":"95be6584-640d-434d-8a1f-7df726a918ae"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-09-29 17:43:29,985 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) Account for user id e33a25d0-28c7-48df-b19f-131956f501a4 is Root Admin or Domain Admin, all APIs are allowed.
2025-09-29 17:43:29,985 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-09-29 17:43:29,985 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-09-29 17:43:29,997 DEBUG [c.c.a.ApiServlet] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) ===END=== 172.17.2.60 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:30,376 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-3615110c]) (logid:76b7253c) ===START=== 172.17.0.247 -- GET jobId=e73a213c-53bd-4f8c-a05f-3d771e7f9ee3&command=queryAsyncJobResult&response=json&
2025-09-29 17:43:30,376 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-3615110c]) (logid:76b7253c) Two factor authentication is already verified for the user 12, so skipping
2025-09-29 17:43:30,381 DEBUG [c.c.a.m.D.Task] (DirectAgent-458:[ctx-b9d84a2f]) (logid:e73a213c) Seq 16-4644900065679250305: Response Received:
2025-09-29 17:43:30,381 DEBUG [c.c.a.t.Request] (DirectAgent-458:[ctx-b9d84a2f]) (logid:e73a213c) Seq 16-4644900065679250305: Processing: { Ans: , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","details":"Stop VM r-5387-VM Succeed","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-29 17:43:30,382 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250305: Received: { Ans: , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 10, { StopAnswer } }
2025-09-29 17:43:30,382 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Details from executing class com.cloud.agent.api.StopCommand: Stop VM r-5387-VM Succeed
2025-09-29 17:43:30,382 ERROR [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Failed to start instance VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} com.cloud.utils.exception.ExecutionException: Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1601)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5827)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5951)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

2025-09-29 17:43:30,392 DEBUG [c.c.a.ApiServer] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) CIDRs from which account 'Account [{"accountName":"ho.karami","id":9,"uuid":"aa8fee67-15f6-41c8-a5e6-4cf64bd0878e"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-09-29 17:43:30,394 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Cleaning up resources for the vm VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} in Starting state
2025-09-29 17:43:30,401 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) Account for user id 13a153b4-c31b-4a3a-833f-d895c29928c3 is Root Admin or Domain Admin, all APIs are allowed.
2025-09-29 17:43:30,401 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-09-29 17:43:30,401 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-09-29 17:43:30,433 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) ===END=== 172.17.0.247 -- GET jobId=e73a213c-53bd-4f8c-a05f-3d771e7f9ee3&command=queryAsyncJobResult&response=json&
2025-09-29 17:43:30,439 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Wait time setting on com.cloud.agent.api.StopCommand is 1800 seconds
2025-09-29 17:43:30,445 DEBUG [c.c.a.m.ClusteredDirectAgentAttache] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250306: Routed from 345050399512
2025-09-29 17:43:30,445 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250306: Sending { Cmd , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"172.17.26.167","forceStop":"false","vlanToPersistenceMap":{"1000":"true"},"expungeVM":"false","volumesToDisconnect":[],"vmName":"r-5387-VM","executeInSequence":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-29 17:43:30,445 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250306: Executing: { Cmd , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"172.17.26.167","forceStop":"false","vlanToPersistenceMap":{"1000":"true"},"expungeVM":"false","volumesToDisconnect":[],"vmName":"r-5387-VM","executeInSequence":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-29 17:43:30,446 DEBUG [c.c.a.m.D.Task] (DirectAgent-59:[ctx-b1ed20e4]) (logid:6f34189c) Seq 16-4644900065679250306: Executing request
2025-09-29 17:43:30,446 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-59:[ctx-b1ed20e4]) (logid:e73a213c) Executing resource command StopCommand: [{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"172.17.26.167","forceStop":false,"vlanToPersistenceMap":{"1000":true},"expungeVM":false,"volumesToDisconnect":[],"vmName":"r-5387-VM","executeInSequence":true,"wait":0,"bypassHostMaintenance":false}].
2025-09-29 17:43:30,467 DEBUG [c.c.h.v.m.BaseMO] (DirectAgent-59:[ctx-b1ed20e4, esxi03.pvcl.afracloud.local, job-53846/job-53849, cmd: StopCommand]) (logid:e73a213c) find VM r-5387-VM on host
2025-09-29 17:43:30,467 INFO [c.c.h.v.m.BaseMO] (DirectAgent-59:[ctx-b1ed20e4, esxi03.pvcl.afracloud.local, job-53846/job-53849, cmd: StopCommand]) (logid:e73a213c) VM r-5387-VM not found in host cache
2025-09-29 17:43:30,467 DEBUG [c.c.h.v.m.BaseMO] (DirectAgent-59:[ctx-b1ed20e4, esxi03.pvcl.afracloud.local, job-53846/job-53849, cmd: StopCommand]) (logid:e73a213c) load VM cache on host
2025-09-29 17:43:30,565 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-59:[ctx-b1ed20e4, esxi03.pvcl.afracloud.local, job-53846/job-53849, cmd: StopCommand]) (logid:e73a213c) VM r-5387-VM is already in stopped state
2025-09-29 17:43:30,566 DEBUG [c.c.a.m.D.Task] (DirectAgent-59:[ctx-b1ed20e4]) (logid:e73a213c) Seq 16-4644900065679250306: Response Received:
2025-09-29 17:43:30,566 DEBUG [c.c.a.t.Request] (DirectAgent-59:[ctx-b1ed20e4]) (logid:e73a213c) Seq 16-4644900065679250306: Processing: { Ans: , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","details":"VM r-5387-VM is already in stopped state","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-29 17:43:30,566 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250306: Received: { Ans: , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 10, { StopAnswer } }
2025-09-29 17:43:30,704 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Released nic: NicProfile {"broadcastUri":null,"deviceId":null,"iPv4Address":null,"id":55319,"reservationId":null,"uuid":"7d4b2a25-d59c-4844-85cc-1965c520337b","vmId":5387} for vm VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"}
2025-09-29 17:43:30,704 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Released nic: NicProfile {"broadcastUri":null,"deviceId":null,"iPv4Address":null,"id":55319,"reservationId":null,"uuid":"7d4b2a25-d59c-4844-85cc-1965c520337b","vmId":5387} for vm VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"}
2025-09-29 17:43:30,704 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) The nic Nic {"broadcastUri":null,"deviceId":1,"iPv4Address":"172.17.26.101","id":55319,"instanceId":5387,"reservationId":"1772dbf2-0235-4112-99ac-e84ae568fb8d","uuid":"7d4b2a25-d59c-4844-85cc-1965c520337b"} on NicProfile {"broadcastUri":null,"deviceId":null,"iPv4Address":null,"id":55319,"reservationId":null,"uuid":"7d4b2a25-d59c-4844-85cc-1965c520337b","vmId":5387} was released according to VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} by guru com.cloud.network.guru.ControlNetworkGuru@1909b900, now updating record.
2025-09-29 17:43:30,728 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Successfully released network resources for the VM VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} in Starting state
2025-09-29 17:43:30,749 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Successfully released storage resources for the VM VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} in Starting state
2025-09-29 17:43:30,750 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Successfully cleaned up resources for the VM VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} in Starting state
2025-09-29 17:43:30,771 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b0964fe]) (logid:7567fe59) Found 1 inactive management server node based on timestamp
2025-09-29 17:43:30,771 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b0964fe]) (logid:7567fe59) management server node msid: 345050417779, name: clst02.pvcl.afracloud.local, service ip: 172.24.48.134, version: 4.21.0.0
2025-09-29 17:43:30,771 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b0964fe]) (logid:7567fe59) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms
2025-09-29 17:43:30,787 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Stopped","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} state transited from [Starting] to [Stopped] with event [OperationFailed]. VM's original host: null, new host: null, host before state transition: Host {"id":16,"name":"esxi03.pvcl.afracloud.local","type":"Routing","uuid":"766179c0-1126-4ff1-9aa9-1542d965cacd"}
2025-09-29 17:43:30,808 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Hosts's actual total CPU: 301728 and CPU after applying overprovisioning: 301728
2025-09-29 17:43:30,808 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Hosts's actual total RAM: (1023.66 GB) 1099147345920 and RAM after applying overprovisioning: (1023.66 GB) 1099147378688
2025-09-29 17:43:30,808 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) release cpu from host: Host {"id":16,"name":"esxi03.pvcl.afracloud.local","type":"Routing","uuid":"766179c0-1126-4ff1-9aa9-1542d965cacd"}, old used: 6500, reserved: 0, actual total: 301728, total with overprovisioning: 301728; new used: 6000,reserved:0; movedfromreserved: false,moveToReservered: false
2025-09-29 17:43:30,810 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) release mem from host: Host {"id":16,"name":"esxi03.pvcl.afracloud.local","type":"Routing","uuid":"766179c0-1126-4ff1-9aa9-1542d965cacd"}, old used: (4.50 GB) 4831838208, reserved: (0 bytes) 0, total: (1023.66 GB) 1099147378688; new used: (4.00 GB) 4294967296, reserved: (0 bytes) 0; movedfromreserved: false, moveToReservered: false
2025-09-29 17:43:30,820 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Invocation exception, caused by: com.cloud.exception.AgentUnavailableException: Resource [Host:16] is unreachable: Host 16: Unable to start instance due to Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
2025-09-29 17:43:30,820 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Rethrow exception com.cloud.exception.AgentUnavailableException: Resource [Host:16] is unreachable: Host 16: Unable to start instance due to Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
2025-09-29 17:43:30,820 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 5387, job origin: 53846
2025-09-29 17:43:30,820 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Unable to complete AsyncJob {"accountId":9,"cmd":"com.cloud.vm.VmWorkStart","cmdInfo":"rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAJAAAAAAAAAAwAAAAAAAAVC3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc3IAEWphdmEudXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA_QAAAAAAADHcIAAAAEAAAAAF0AA5SZXN0YXJ0TmV0d29ya3QAP3JPMEFCWE55QUJGcVlYWmhMbXhoYm1jdVFtOXZiR1ZoYnMwZ2NvRFZuUHJ1QWdBQldnQUZkbUZzZFdWNGNBRXhw","cmdVersion":0,"completeMsid":null,"created":"Mon Sep 29 17:40:32 IRST 2025","id":53849,"initMsid":345050399512,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":12,"uuid":"cd1036aa-8b98-4906-b35f-f6b3503f4ace"}, job origin: 53846 com.cloud.exception.AgentUnavailableException: Resource [Host:16] is unreachable: Host 16: Unable to start instance due to Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1637)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5827)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5951)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.cloud.utils.exception.ExecutionException: Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1601)
... 20 more

2025-09-29 17:43:30,827 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Complete async job-53849, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC1jb20uY2xvdWQuZXhjZXB0aW9uLkFnZW50VW5hdmFpbGFibGVFeGNlcHRpb25WTU9wAAAAFwIAAHhyADBjb20uY2xvdWQuZXhjZXB0aW9uLlJlc291cmNlVW5hdmFpbGFibGVFeGNlcHRpb25WTU9wAAAAHwIAAkoAA19pZEwABl9zY29wZXQAEUxqYXZhL2xhbmcvQ2xhc3M7eHIAImNvbS5jbG91ZC5leGNlcHRpb24uQ2xvdWRFeGNlcHRpb2556I4b8eRExwIAAkwAC2NzRXJyb3JDb2RldAATTGphdmEvbGFuZy9JbnRlZ2VyO0wABmlkTGlzdHQAFUxqYXZhL3V0aWwvQXJyYXlMaXN0O3hyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwc3IALGNvbS5jbG91ZC51dGlscy5leGNlcHRpb24uRXhlY3V0aW9uRXhjZXB0aW9uVk1PcAAAAA4CAAFJAAtjc0Vycm9yQ29kZXhxAH4ABnEAfgAOdABkVW5hYmxlIHRvIHN0YXJ0ICBWTTpjNWZjNTRlZC04MWY5LTQ3NDMtOGJkYi00OGRjNjM3OWY5ZjggZHVlIHRvIGVycm9yIGluIGZpbmFsaXplU3RhcnQsIG5vdCByZXRyeWluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAFXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ACUwADmRlY2xhcmluZ0NsYXNzcQB-AAlMAAhmaWxlTmFtZXEAfgAJTAAKbWV0aG9kTmFtZXEAfgAJTAAKbW9kdWxlTmFtZXEAfgAJTAANbW9kdWxlVmVyc2lvbnEAfgAJeHABAAAGQXQAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgASAQAAFsNxAH4AFHEAfgAVcQB-ABZxAH4AF3Bwc3EAfgASAv____5wdAAtamRrLmludGVybmFsLnJlZmxlY3QuTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsdAAdTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsLmphdmF0AAdpbnZva2UwdAAJamF2YS5iYXNldAAHMTcuMC4xNnNxAH4AEgIAAABNcHEAfgAacQB-ABt0AAZpbnZva2VxAH4AHXEAfgAec3EAfgASAgAAACtwdAAxamRrLmludGVybmFsLnJlZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbHQAIURlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuamF2YXEAfgAgcQB-AB1xAH4AHnNxAH4AEgIAAAI5cHQAGGphdmEubGFuZy5yZWZsZWN0Lk1ldGhvZHQAC01ldGhvZC5qYXZhcQB-ACBxAH4AHXEAfgAec3EAfgASAQAAAGZxAH4AFHQAImNvbS5jbG91ZC52bS5WbVdvcmtKb2JIYW5kbGVyUHJveHl0ABpWbVdvcmtKb2JIYW5kbGVyUHJveHkuamF2YXQAD2hhbmRsZVZtV29ya0pvYnBwc3EAfgASAQAAFz9xAH4AFHEAfgAVcQB-ABZxAH4AKnBwc3EAfgASAQAAAGNxAH4AFHQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB-ABIBAAACsXEAfgAUdAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0cHBzcQB-ABIBAAAAMXEAfgAUdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5wcHNxAH4AEgEAAAA4cQB-ABR0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4AEgEAAABncQB-ABR0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-ADp0AA9jYWxsV2l0aENvbnRleHRwcHNxAH4AEgEAAAA1cQB-ABRxAH4APXEAfgA6dAAOcnVuV2l0aENvbnRleHRwcHNxAH4AEgEAAAAucQB-ABR0ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4ANnEAfgA3cHBzcQB-ABIBAAACfXEAfgAUcQB-ADFxAH4AMnEAfgA3cHBzcQB-ABICAAACG3B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AO3EAfgAdcQB-AB5zcQB-ABICAAABCHB0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ADdxAH4AHXEAfgAec3EAfgASAgAABHBwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AHXEAfgAec3EAfgASAgAAAntwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgBMcQB-ADdxAH4AHXEAfgAec3EAfgASAgAAA0hwdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ADdxAH4AHXEAfgAec3IAH2phdmEudXRpbC5Db2xsZWN0aW9ucyRFbXB0eUxpc3R6uBe0PKee3gIAAHhweAAAAAB0ALBSZXNvdXJjZSBbSG9zdDoxNl0gaXMgdW5yZWFjaGFibGU6IEhvc3QgMTY6IFVuYWJsZSB0byBzdGFydCBpbnN0YW5jZSBkdWUgdG8gVW5hYmxlIHRvIHN0YXJ0ICBWTTpjNWZjNTRlZC04MWY5LTQ3NDMtOGJkYi00OGRjNjM3OWY5ZjggZHVlIHRvIGVycm9yIGluIGZpbmFsaXplU3RhcnQsIG5vdCByZXRyeWluZ3VxAH4AEAAAABVzcQB-ABIBAAAGZXEAfgAUcQB-ABVxAH4AFnEAfgAXcHBzcQB-ABIBAAAWw3EAfgAUcQB-ABVxAH4AFnEAfgAXcHBzcQB-ABIC_____nBxAH4AGnEAfgAbcQB-ABxxAH4AHXEAfgAec3EAfgASAgAAAE1wcQB-ABpxAH4AG3EAfgAgcQB-AB1xAH4AHnNxAH4AEgIAAAArcHEAfgAicQB-ACNxAH4AIHEAfgAdcQB-AB5zcQB-ABICAAACOXBxAH4AJXEAfgAmcQB-ACBxAH4AHXEAfgAec3EAfgASAQAAAGZxAH4AFHEAfgAocQB-AClxAH4AKnBwc3EAfgASAQAAFz9xAH4AFHEAfgAVcQB-ABZxAH4AKnBwc3EAfgASAQAAAGNxAH4AFHEAfgAtcQB-AC5xAH4AL3Bwc3EAfgASAQAAArFxAH4AFHEAfgAxcQB-ADJxAH4AM3Bwc3EAfgASAQAAADFxAH4AFHEAfgA1cQB-ADZxAH4AN3Bwc3EAfgASAQAAADhxAH4AFHEAfgA5cQB-ADpxAH4AO3Bwc3EAfgASAQAAAGdxAH4AFHEAfgA9cQB-ADpxAH4APnBwc3EAfgASAQAAADVxAH4AFHEAfgA9cQB-ADpxAH4AQHBwc3EAfgASAQAAAC5xAH4AFHEAfgBCcQB-ADZxAH4AN3Bwc3EAfgASAQAAAn1xAH4AFHEAfgAxcQB-ADJxAH4AN3Bwc3EAfgASAgAAAhtwcQB-AEVxAH4ARnEAfgA7cQB-AB1xAH4AHnNxAH4AEgIAAAEIcHEAfgBIcQB-AElxAH4AN3EAfgAdcQB-AB5zcQB-ABICAAAEcHBxAH4AS3EAfgBMcQB-AE1xAH4AHXEAfgAec3EAfgASAgAAAntwcQB-AE9xAH4ATHEAfgA3cQB-AB1xAH4AHnNxAH4AEgIAAANIcHEAfgBRcQB-AFJxAH4AN3EAfgAdcQB-AB5xAH4AVHhzcgARamF2YS5sYW5nLkludGVnZXIS4qCk94GHOAIAAUkABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAQvXNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeAAAAAAAAAAQdnIAE2NvbS5jbG91ZC5ob3N0Lkhvc3S4fD08IdIeQAIAAHhw
2025-09-29 17:43:30,829 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Publish async job-53849 complete on message bus
2025-09-29 17:43:30,829 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Wake up jobs related to job-53849
2025-09-29 17:43:30,829 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Update db status for job-53849

[weizhouapache]

I suspect if the systemvm template is Ready
please also check if the global setting router.template.vmware is set correctly

[Hossein4185]

@weizhouapache
The global settings are as follows:

I can use the shared network as shown in the image below, but I am having problems with Isoleted Network.

[prashanthr2]

@Hossein4185 In the latest logs we see that the Virtual Router associated with the network of the VM is failing to start , we see a stop was issued as below

2025-09-29 17:43:30,381 DEBUG [c.c.a.m.D.Task] (DirectAgent-458:[ctx-b9d84a2f]) (logid:e73a213c) Seq 16-4644900065679250305: Response Received:
2025-09-29 17:43:30,381 DEBUG [c.c.a.t.Request] (DirectAgent-458:[ctx-b9d84a2f]) (logid:e73a213c) Seq 16-4644900065679250305: Processing: { Ans: , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","details":"Stop VM r-5387-VM Succeed","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-29 17:43:30,382 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250305: Received: { Ans: , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 10, { StopAnswer } }

This can happen due to various reasons. Please check the status of the host esxi03.pvcl.afracloud.local to see if it is UP or has any issues.

Also, send the full managemenet server log as we need this to trace the complete logs for the VM start job. Full log will help us determine why VR start is failing

[Hossein4185]

@prashanthr2

management-server.zip

[weizhouapache]

@prashanthr2

management-server.zip

I noticed some timeout errors, not sure if it is a networking issue (or storage issue)
you can try with the 4.20.2 systemvm template on https://download.cloudstack.org/systemvm/4.20/

2025-10-01 12:50:20,146 WARN [c.c.h.v.r.VmwareResource] (DirectAgent-36:[ctx-53b9e07b, esxi02.pvcl.afracloud.local, job-53859/job-53862, cmd: AggregationControlCommand]) (logid:092f13da) Fail to create file /var/cache/cloud/VR-31a57a72-7364-47c5-b86f-aa8017a455db.cfg in VR 172.17.26.238 java.net.SocketTimeoutException: The kexTimeout (60000 ms) expired.

2025-10-01 12:58:07,450 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-0:[ctx-08be63f6]) (logid:d530d0bb) Task (job-53871) has been pending for 596 seconds
2025-10-01 12:58:58,942 ERROR [c.c.u.s.SshHelper] (DirectAgent-2:[ctx-bbf1acd4, esxi02.pvcl.afracloud.local, job-53871/job-53874, cmd: GetDomRVersionCmd]) (logid:fb638096) Timed out in waiting for SSH execution exit status
2025-10-01 12:58:58,944 INFO [c.c.h.v.u.VmwareHelper] (DirectAgent-2:[ctx-bbf1acd4, esxi02.pvcl.afracloud.local, job-53871/job-53874, cmd: GetDomRVersionCmd]) (logid:fb638096) [ignored]failed to get message for exception: Timed out in waiting for SSH execution exit status
2025-10-01 12:58:58,945 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-2:[ctx-bbf1acd4, esxi02.pvcl.afracloud.local, job-53871/job-53874, cmd: GetDomRVersionCmd]) (logid:fb638096) Command failed due to Exception: com.cloud.utils.ssh.SshException

[Hossein4185]

@weizhouapache
Currently, I am using version 4.20.2 according to Global but no resolved issue.

[weizhouapache]

@weizhouapache Currently, I am using version 4.20.2 according to Global but no resolved issue.

ok, I suspect it is a networking issue or SSH configuration issue (algorithms, ciphers, etc)

you can try to ssh into a VR in Starting state (before it is stopped in ACS) from mgmt server

ssh -vvv -p 3922 -i ~cloud/.ssh/id_cloud <private IP of VR>

[Hossein4185]

@weizhouapache
No, we can access shared networks, but not isolated ones.

[Hossein4185]

@weizhouapache
@prashanthr2
We don't have any problems with the shared network, we just have problems with the isolated. We probably have a problem with the provisioning of this service. What are the differences between the shared network and isolated settings?

[weizhouapache]

@weizhouapache @prashanthr2 We don't have any problems with the shared network, we just have problems with the isolated. We probably have a problem with the provisioning of this service. What are the differences between the shared network and isolated settings?

there are big differences. however, for both types of networks, you should be able to ssh into the VR from mgmt server

ssh -vvv -p 3922 -i ~cloud/.ssh/id_cloud <private IP of VR>

If this does not work, there might be something wrong with the network configurations (bridges, network cidr, etc)
it would be good to share more details of the shared/isolated/physical/pod networks

[Hossein4185]

The problem was solved by checking the mentioned path.

[weizhouapache]

The problem was solved by checking the mentioned path.

thanks @Hossein4185 for the update.

what's the root cause and how you fixed it ?

[Hossein4185]

The problem was that the isolated caused problems due to the presence of a third interface if you did not enter the network settings accurately. I logged into the router through the console and noticed this problem, but the problem is that CloudStack does not provide a correct log in the management about this.

[weizhouapache]

The problem was that the isolated caused problems due to the presence of a third interface if you did not enter the network settings accurately. I logged into the router through the console and noticed this problem, but the problem is that CloudStack does not provide a correct log in the management about this.

is it due to the the cidr of isolated network ? overlap with management network or so ?