Add nightly scale tests for self-hosted runners #23

quoctruong · 2024-08-27T20:13:00Z

The test will run on all supported Linux self-hosted runners at 11pm each night. It will spin up 5 different runner instances.

MichaelHudgins · 2024-08-28T18:54:05Z

.github/workflows/stress-test.yml

+name: Self-hosted Runners Nightly Stress Test
+on:
+  schedule:
+    # Triggers at 11pm every night.


Small nit, can you leave the timezone in the comment to make it obvious for anyone reviewing

Added the timezone comment. It's in UST so I changed the time to make it run at 11pm PST.

MichaelHudgins · 2024-08-28T18:54:42Z

.github/workflows/stress-test.yml

+on:
+  schedule:
+    # Triggers at 11pm every night.
+    - cron: '0 23 * * *'


I recommend adding a workflow_dispatch to this as well for testing and manual invocation

MichaelHudgins · 2024-08-28T18:57:40Z

.github/workflows/stress-test.yml

@@ -0,0 +1,35 @@
+# Nightly Stress Test for self-hosted runners
+name: Self-hosted Runners Nightly Stress Test


Instead of "stress" maybe we should call this a scale test. We don't actually do anything with the runners which makes me lean a little away from calling it stress long term.

Changed to scale.

MichaelHudgins · 2024-08-28T18:58:24Z

.github/workflows/stress-test.yml

+        shell: bash -ex {0}
+    steps:
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # ratchet:actions/checkout@v4
+      - name: Install JAX test requirements


Do you see value in leaving the install of test requirements if we don't end up using them?

I removed it since it's not being used.

MichaelHudgins · 2024-08-28T18:58:45Z

.github/workflows/stress-test.yml

+    runs-on: ${{ matrix.runners }}
+    container:
+      image: ${{ (contains(matrix.runners, 't2a') && 'us-central1-docker.pkg.dev/tensorflow-sigs/tensorflow/build-arm64:jax-latest-multi-python') || 'index.docker.io/tensorflow/build@sha256:7fb38f0319bda36393cad7f40670aa22352b44421bb906f5cf34d543acd8e1d2' }}
+    timeout-minutes: 10


Do you know, does this timeout include the initialization of the runner?

I think it doesn't because it's supposed to be the execution time.

In that case its fine

quoctruong · 2024-08-28T19:48:23Z

Thanks @MichaelHudgins. PTAL.

When run under an optimized build and Python 3.13.2t, I saw the following high probability crash in lax_control_flow_test: ``` Stack trace of thread 3526917: #0 0x00007f0898c4bf91 dump_frame (libpython3.13t.so.1.0 + 0x24bf91) #1 0x00007f0898c4b73f dump_traceback (libpython3.13t.so.1.0 + 0x24b73f) #2 0x00007f0898c4b86f _Py_DumpTracebackThreads (libpython3.13t.so.1.0 + 0x24b86f) #3 0x00007f0898cd4fe0 faulthandler_dump_traceback (libpython3.13t.so.1.0 + 0x2d4fe0) #4 0x00007f0898cd4f44 faulthandler_fatal_error (libpython3.13t.so.1.0 + 0x2d4f44) #5 0x00007f0898849e20 __restore_rt (libc.so.6 + 0x3fe20) #6 0x00007f07eb80e493 _ZNSt8__detail16_Hashtable_allocISaINS_10_Hash_nodeISt4pairIKN3jax15WeakrefLRUCache15WeakrefCacheKeyENS4_17WeakrefCacheValueEELb1EEEEE18_M_deallocate_nodeEPS9_ (libjax_common.so + 0x2c0e493) #7 0x00007f07eb80e13e _ZN3jax15WeakrefLRUCache5ClearEv (libjax_common.so + 0x2c0e13e) #8 0x00007f07eb812e37 _ZZN8nanobind6detail11func_createILb0ELb1EZNS_16cpp_function_defIN3jax15WeakrefLRUCacheEvS4_JEJNS_5scopeENS_4nameENS_9is_methodENS_9lock_selfEEEEvMT1_FT0_DpT2_EDpRKT3_EUlPS4_E_vJSJ_EJLm0EEJS5_S6_S7_S8_EEEP> #9 0x00007f07eb7fff70 _ZN8nanobind6detailL25nb_func_vectorcall_simpleEP7_objectPKS2_mS2_ (libjax_common.so + 0x2bfff70) #10 0x00007f0898dbbdee _PyObject_VectorcallTstate (libpython3.13t.so.1.0 + 0x3bbdee) #11 0x00007f0898d1d4db _PyEval_EvalFrame (libpython3.13t.so.1.0 + 0x31d4db) #12 0x00007f0898d1ee78 _PyObject_VectorcallTstate (libpython3.13t.so.1.0 + 0x31ee78) #13 0x00007f0898dc0054 _PyVectorcall_Call (libpython3.13t.so.1.0 + 0x3c0054) #14 0x00007f0898d1d4db _PyEval_EvalFrame (libpython3.13t.so.1.0 + 0x31d4db) #15 0x00007f0898d1e02c _PyObject_VectorcallDictTstate (libpython3.13t.so.1.0 + 0x31e02c) #16 0x00007f0898ed8e35 slot_tp_call (libpython3.13t.so.1.0 + 0x4d8e35) #17 0x00007f0898dbc312 _PyObject_MakeTpCall (libpython3.13t.so.1.0 + 0x3bc312) #18 0x00007f0898d1d4db _PyEval_EvalFrame (libpython3.13t.so.1.0 + 0x31d4db) #19 0x00007f0898d1ef54 _PyObject_VectorcallTstate (libpython3.13t.so.1.0 + 0x31ef54) #20 0x00007f0899094c1f thread_run (libpython3.13t.so.1.0 + 0x694c1f) #21 0x00007f0898fa0c58 pythread_wrapper (libpython3.13t.so.1.0 + 0x5a0c58) #22 0x00007f089889c103 start_thread (libc.so.6 + 0x92103) #23 0x00007f089891a7b8 __clone3 (libc.so.6 + 0x1107b8) ``` It appears that this is due to freeing Python objects during unordered_map::clear(), which may release the enclosing critical section (`nb::lock_self()` on the method). Fix this by deferring destruction of the both the keys and the values to after the map's destruction.

quoctruong added 7 commits August 27, 2024 11:55

Run stress tests

a0699f7

Run stress tests

1fce65a

Add logging

dace222

Make stress test run

d9e4c47

Fix stress test

eb6df84

Fix stress test to run at 11pm every night

1b7ab90

Fix formatting

c0cd0c5

quoctruong requested a review from MichaelHudgins August 27, 2024 20:13

MichaelHudgins reviewed Aug 28, 2024

View reviewed changes

Address comments

766a39c

Update name for runners

eb58999

MichaelHudgins changed the title ~~Add nightly stress tests for self-hosted runners~~ Add nightly scale tests for self-hosted runners Aug 30, 2024

Fix wait-for-connection-test workflow

5132a5e

		@@ -0,0 +1,35 @@
		# Nightly Stress Test for self-hosted runners
		name: Self-hosted Runners Nightly Stress Test

Add nightly scale tests for self-hosted runners #23

Are you sure you want to change the base?

Add nightly scale tests for self-hosted runners #23

Uh oh!

Conversation

quoctruong commented Aug 27, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quoctruong Aug 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quoctruong commented Aug 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

quoctruong Aug 28, 2024 •

edited

Loading