internal/kubernetes/kubernetes.go: lazy K8S API calls

[andmat900]

Applicable Issues

• SSE: client rate limiter Wait returned an error: context canceled etos#423

Description of the Change

This change implements lazy K8S API calling for getting jobs/pods in internal/kubernetes/kubernetes.go.

Alternate Designs

Possible Drawbacks

Sign-off

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or

(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or

(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.

(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

Signed-off-by: Andrei Matveyeu, [email protected]

[t-persson]

Why not use the cache already in client-go?

https://pkg.go.dev/k8s.io/client-go/tools/cache

[andmat900]

Why not use the cache already in client-go?

https://pkg.go.dev/k8s.io/client-go/tools/cache

We've done some load testing and looked at the client-go cache. For now, we'd like to continue with the current implementation and file your suggestion as a future improvement.

The trade-off is simplicity vs event-driven/real-time implementation.

[t-persson]

Why not use the cache already in client-go?

https://pkg.go.dev/k8s.io/client-go/tools/cache

We've done some load testing and looked at the client-go cache. For now, we'd like to continue with the current implementation and file your suggestion as a future improvement.

The trade-off is simplicity vs event-driven/real-time implementation.

Looking at this example it feels more simple than running our own implementation, but I guess it wasnt performant enough (If I understood your comment correctly)?

https://github.com/Karthik-K-N/goCache/blob/main/main.go

[andmat900]

Why not use the cache already in client-go?
https://pkg.go.dev/k8s.io/client-go/tools/cache

We've done some load testing and looked at the client-go cache. For now, we'd like to continue with the current implementation and file your suggestion as a future improvement.
The trade-off is simplicity vs event-driven/real-time implementation.

Looking at this example it feels more simple than running our own implementation, but I guess it wasnt performant enough (If I understood your comment correctly)?

https://github.com/Karthik-K-N/goCache/blob/main/main.go

I was referring to the example from your first URL.

[t-persson]

Why not use the cache already in client-go?
https://pkg.go.dev/k8s.io/client-go/tools/cache

We've done some load testing and looked at the client-go cache. For now, we'd like to continue with the current implementation and file your suggestion as a future improvement.
The trade-off is simplicity vs event-driven/real-time implementation.

Looking at this example it feels more simple than running our own implementation, but I guess it wasnt performant enough (If I understood your comment correctly)?

https://github.com/Karthik-K-N/goCache/blob/main/main.go

I was referring to the example from your first URL.

Ah, I see. That was merely a link to the cache package in client-go.

[andmat900]

The simplified implementation in commit dcaea5c did not work as expected: it would require additional mutexes. Without them there were OOMKilled errors. Reverting to the previous implementation.