enable topic slo

Author: vgvoleg

.github/workflows/slo.yml

@@ -57,6 +57,26 @@ jobs:
               --read-timeout 1000 \
               --write-timeout 1000
             cleanup-args: grpc://localhost:2135 /Root/testdb
+          - prefix: topic
+            workload: topic-basic
+            create-args: |
+              grpc://localhost:2135 /Root/testdb \
+              --path /Root/testdb/slo_topic \
+              --partitions-count 10
+            run-args: |
+              grpc://localhost:2135 /Root/testdb \
+              --path /Root/testdb/slo_topic \
+              --prom-pgw localhost:9091 \
+              --partitions-count 10 \
+              --read-threads 10 \
+              --write-threads 10 \
+              --report-period 250 \
+              --time ${{inputs.slo_workload_duration_seconds || 600}} \
+              --read-rps ${{inputs.slo_workload_read_max_rps || 100}} \
+              --write-rps ${{inputs.slo_workload_write_max_rps || 100}} \
+              --read-timeout 5000 \
+              --write-timeout 5000
+            cleanup-args: grpc://localhost:2135 /Root/testdb --path /Root/testdb/slo_topic
 
 
     concurrency:

tests/slo/README.md

@@ -3,42 +3,72 @@
 SLO is the type of test where app based on ydb-sdk is tested against falling YDB cluster nodes, tablets, network
 (that is possible situations for distributed DBs with hundreds of nodes)
 
-### Implementations:
+### Workload types:
+
+There are two workload types:
+
+- **Table SLO** - tests table operations (read/write)
+- **Topic SLO** - tests topic operations (publish/consume)
 
-There are two implementations:
+### Implementations:
 
 - `sync`
 - `async` (now unimplemented)
 
 ### Usage:
 
-It has 3 commands:
+Each workload type has 3 commands:
+
+**Table commands:**
+- `table-create`  - creates table in database
+- `table-cleanup` - drops table in database
+- `table-run`     - runs table workload (read and write to table with set RPS)
 
-- `create`  - creates table in database
-- `cleanup` - drops table in database
-- `run`     - runs workload (read and write to table with sets RPS)
+**Topic commands:**
+- `topic-create`  - creates topic with consumer in database
+- `topic-cleanup` - drops topic in database
+- `topic-run`     - runs topic workload (publish and consume messages with set RPS)
 
 ### Run examples with all arguments:
 
-create:
-`python tests/slo/src/ create localhost:2136 /local -t tableName
+**Table examples:**
+
+table-create:
+`python tests/slo/src/ table-create localhost:2136 /local -t tableName
 --min-partitions-count 6 --max-partitions-count 1000 --partition-size 1 -с 1000
 --write-timeout 10000`
 
-cleanup:
-`python tests/slo/src/ cleanup localhost:2136 /local -t tableName`
+table-cleanup:
+`python tests/slo/src/ table-cleanup localhost:2136 /local -t tableName`
 
-run:
-`python tests/slo/src/ run localhost:2136 /local -t tableName
---prom-pgw http://prometheus-pushgateway:9091 -report-period 250
+table-run:
+`python tests/slo/src/ table-run localhost:2136 /local -t tableName
+--prom-pgw http://prometheus-pushgateway:9091 --report-period 250
 --read-rps 1000 --read-timeout 10000
 --write-rps 100 --write-timeout 10000
 --time 600 --shutdown-time 30`
 
+**Topic examples:**
+
+topic-create:
+`python tests/slo/src/ topic-create localhost:2136 /local
+--topic-path /local/slo_topic --topic-consumer slo_consumer`
+
+topic-cleanup:
+`python tests/slo/src/ topic-cleanup localhost:2136 /local --topic-path /local/slo_topic`
+
+topic-run:
+`python tests/slo/src/ topic-run localhost:2136 /local
+--topic-path /local/slo_topic --topic-consumer slo_consumer
+--prom-pgw http://prometheus-pushgateway:9091 --report-period 250
+--topic-write-rps 50 --topic-read-rps 100
+--topic-write-timeout 5000 --topic-read-timeout 3000
+--time 600 --shutdown-time 30`
+
 ## Arguments for commands:
 
-### create
-`python tests/slo/src/ create <endpoint> <db> [options]`
+### table-create
+`python tests/slo/src/ table-create <endpoint> <db> [options]`
 
 ```
 Arguments:
@@ -61,8 +91,8 @@ Options:
 
 ```
 
-### cleanup
-`python tests/slo/src/ cleanup <endpoint> <db> [options]`
+### table-cleanup
+`python tests/slo/src/ table-cleanup <endpoint> <db> [options]`
 
 ```
 Arguments:
@@ -73,8 +103,8 @@ Options:
   -t --table-name                  <string> table name to create
 ```
 
-### run
-`python tests/slo/src/ run <endpoint> <db> [options]`
+### table-run
+`python tests/slo/src/ table-run <endpoint> <db> [options]`
 
 ```
 Arguments:
@@ -100,12 +130,70 @@ Options:
   --write-threads         <int>    number of threads to use for read requests
 ```
 
+### topic-create
+`python tests/slo/src/ topic-create <endpoint> <db> [options]`
+
+```
+Arguments:
+  endpoint                        YDB endpoint to connect to
+  db                              YDB database to connect to
+
+Options:
+  --topic-path                    <string> topic path to create
+  --topic-consumer                <string> consumer name
+  --topic-min-partitions          <int>    minimum active partitions
+  --topic-max-partitions          <int>    maximum active partitions
+  --topic-retention-hours         <int>    retention period in hours
+```
+
+### topic-cleanup
+`python tests/slo/src/ topic-cleanup <endpoint> <db> [options]`
+
+```
+Arguments:
+  endpoint                        YDB endpoint to connect to
+  db                              YDB database to connect to
+
+Options:
+  --topic-path                    <string> topic path to drop
+```
+
+### topic-run
+`python tests/slo/src/ topic-run <endpoint> <db> [options]`
+
+```
+Arguments:
+  endpoint                        YDB endpoint to connect to
+  db                              YDB database to connect to
+
+Options:
+  --topic-path                    <string> topic path
+  --topic-consumer                <string> consumer name
+
+  --prom-pgw                      <string> prometheus push gateway
+  --report-period                 <int>    prometheus push period in milliseconds
+
+  --topic-read-rps                <int>    read RPS for topics
+  --topic-read-timeout            <int>    read timeout milliseconds for topics
+  --topic-write-rps               <int>    write RPS for topics
+  --topic-write-timeout           <int>    write timeout milliseconds for topics
+
+  --topic-message-size            <int>    message size in bytes
+  --topic-read-threads            <int>    number of threads to use for read requests
+  --topic-write-threads           <int>    number of threads to use for write requests
+
+  --time                          <int>    run time in seconds
+  --shutdown-time                 <int>    graceful shutdown time in seconds
+```
+
 ## Authentication
 
 Workload using [auth-env](https://ydb.yandex-team.ru/docs/reference/ydb-sdk/recipes/auth-env) for authentication.
 
 ## What's inside
-When running `run` command, the program creates three jobs: `readJob`, `writeJob`, `metricsJob`.
+
+### Table workload
+When running `table-run` command, the program creates three jobs: `readJob`, `writeJob`, `metricsJob`.
 
 - `readJob`    reads rows from the table one by one with random identifiers generated by writeJob
 - `writeJob`   generates and inserts rows
@@ -120,13 +208,27 @@ Table have these fields:
 
 Primary key: `("object_hash", "object_id")`
 
+### Topic workload
+When running `topic-run` command, the program creates three jobs: `readJob`, `writeJob`, `metricsJob`.
+
+- `readJob`    reads messages from topic using TopicReader and commits offsets
+- `writeJob`   generates and publishes messages to topic using TopicWriter
+- `metricsJob` periodically sends metrics to Prometheus
+
+Messages contain:
+- Sequential message ID
+- Thread identifier
+- Configurable payload size (padded with 'x' characters)
+
 ## Collected metrics
 - `oks`      - amount of OK requests
 - `not_oks`  - amount of not OK requests
 - `inflight` - amount of requests in flight
 - `latency`  - summary of latencies in ms
 - `attempts` - summary of amount for request
 
+Metrics are collected for both table operations (`read`, `write`) and topic operations (`read`, `write`).
+
 > You must reset metrics to keep them `0` in prometheus and grafana before beginning and after ending of jobs
 
 ## Look at metrics in grafana