Compressa Performance Measurement Tool

This tool is designed to measure the performance of Compressa models.
It uses the OpenAI API to run inference tasks and stores the results in a SQLite database.

Installation

git clone https://github.com/compressa-ai/compressa-perf.git
cd compressa-perf
poetry install
$(poetry env activate)

Install with Pip

pip install compressa-perf

Usage

1. Run experiment with prompts from a file

❯ compressa-perf measure \
    --db some_db.sqlite \
    --openai_url https://some-api-url.ru/ \
    --api_key "${OPENAI_API_KEY}" \
    --model_name Compressa-Qwen2.5-14B-Instruct \
    --experiment_name "File Prompts Run" \
    --prompts_file resources/prompts.csv \
    --num_tasks 1000 \
    --num_runners 100

2. Run experiment with generated prompts

❯ compressa-perf measure \
    --db some_db.sqlite \
    --openai_url https://some-api-url.ru/chat-2/v1/ \
    --api_key "${OPENAI_API_KEY}" \
    --model_name Compressa-Qwen2.5-14B-Instruct \
    --experiment_name "Generated Prompts Run" \
    --num_tasks 2 \
    --num_runners 2 \
    --generate_prompts \
    --num_prompts 1000 \
    --prompt_length 5000

Full parameter list can be obtained with compressa-perf measure -h.

3. Run set of experiments from YAML file

You can describe set of experiments in YAML file and run them on different services in one command:

❯ compressa-perf measure-from-yaml experiments.yaml \
    --db some_db.sqlite \

Example of YAML file:

- openai_url: http://localhost:5000/v1/
  api_key: ${OPENAI_API_KEY}
  model_name: Compressa-LLM
  experiment_name: "File Prompts Run 1"
  description: "Experiment using prompts from a file with 500 tasks and 5 runners"
  prompts_file: resources/prompts.csv
  num_tasks: 500
  num_runners: 5
  generate_prompts: false
  num_prompts: 0
  prompt_length: 0
  max_tokens: 1000 

- openai_url: https://some-api-url/v1/
  api_key: ${OPENAI_API_KEY}
  model_name: Compressa-LLM
  experiment_name: "File Prompts Run 2"
  description: "Experiment using prompts from a file with 20 tasks and 10 runners"
  prompts_file: resources/prompts.csv
  num_tasks: 20
  num_runners: 10
  generate_prompts: true
  num_prompts: 10
  prompt_length: 10000
  max_tokens: 100

List of Parameters

• openai_url - url to chat completion endpoint - REQUIRED
• serv_api_url - url to service handlers of the Compressa platform - default is http://localhost:5100/v1/ (if None - the inference only will run)
• api_key - API key - REQUIRED
• model_name - served model name - REQUIRED
• experiment_name - REQUIRED
• description
• prompts_file - path to the file with prompts
• report_file - path to the report file - default is results/experiment
• report_mode - report file extension (.csv, .md, .pdf) - default is .pdf
• num_tasks
• num_runners
• generate_prompts - true or false
• num_prompts
• prompt_length
• max_tokens

4. List experiments

You can select experiments by name, parameters or metrics (or substrings in these fields) via compressa-perf list command.

For example:

❯ compressa-perf list \
    --show-metrics \
    --param-filter openai_url=chat-2 \
    --param-filter avg_n_input=30

List of Experiments:
+----+----------------------------------------------------------------------------+---------------------+--------+-----------------------+
|    | ID                                                                         | Name                | Date   | Description           |
+====+============================================================================+=====================+========+=======================+
| 25 | Compressa-Qwen2.5-14B-Instruct-Int4 Long Input / Short Output | 5 runners  | 2024-10-03 06:21:45 |        | ttft: 25.0960         |
|    |                                                                            |                     |        | latency: 52.5916      |
|    |                                                                            |                     |        | tpot: 0.5530          |
|    |                                                                            |                     |        | throughput: 2891.0323 |
+----+----------------------------------------------------------------------------+---------------------+--------+-----------------------+
| 23 | Compressa-Qwen2.5-14B-Instruct-Int4 Long Input / Short Output | 4 runners  | 2024-10-03 06:14:57 |        | ttft: 17.1862         |
|    |                                                                            |                     |        | latency: 37.9612      |
|    |                                                                            |                     |        | tpot: 0.3954          |
|    |                                                                            |                     |        | throughput: 3230.8769 |
+----+----------------------------------------------------------------------------+---------------------+--------+-----------------------+

Full parameter list:

❯ compressa-perf list -h
usage: compressa-perf list [-h] [--db DB] [--show-parameters] [--show-metrics] [--name-filter NAME_FILTER] [--param-filter PARAM_FILTER]

options:
  -h, --help            show this help message and exit
  --db DB               Path to the SQLite database
  --show-parameters     Show all parameters for each experiment
  --show-metrics        Show metrics for each experiment
  --name-filter NAME_FILTER
                        Filter experiments by substring in the name
  --param-filter PARAM_FILTER
                        Filter experiments by parameter value (e.g., paramkey=value_substring)

5. Generate a report for an experiment

In addition to the .pdf, .csv or .md reports the text reports also can be generated with the command:

❯ compressa-perf report <EXPERIMENT_ID>

Output example:

❯ compressa-perf report 3

Experiment Details:
ID: 3
Name: My First Run
Date: 2024-09-24 07:10:39
Description: None

Experiment Parameters:
╒══════════════╤═══════════════════════════════════════════╕
│    Parameter │                                     Value │
╞══════════════╪═══════════════════════════════════════════╡
│  num_workers │                                         2 │
├──────────────┼───────────────────────────────────────────┤
│    num_tasks │                                         2 │
├──────────────┼───────────────────────────────────────────┤
│   openai_url │ https://some-api-url.ru/chat-2/v1/        │
├──────────────┼───────────────────────────────────────────┤
│  max_tokens  │                                      1000 │
├──────────────┼───────────────────────────────────────────┤
│  model_name  │                             Compressa-LLM │
├──────────────┼───────────────────────────────────────────┤
│  avg_n_input │                                        32 │
├──────────────┼───────────────────────────────────────────┤
│  std_n_input │                                    2.8284 │
├──────────────┼───────────────────────────────────────────┤
│ avg_n_output │                                  748.5000 │
├──────────────┼───────────────────────────────────────────┤
│ std_n_output │                                    2.1213 │
╘══════════════╧═══════════════════════════════════════════╛

Experiment Metrics:

╒══════════════════════════╤══════════╕
│ Metric                   │    Value │
╞══════════════════════════╪══════════╡
│ TTFT                     │   0.0622 │
├──────────────────────────┼──────────┤
│ TTFT_95                  │   0.0693 │
├──────────────────────────┼──────────┤
│ TOP_5_TTFT               │   0.0757 │
├──────────────────────────┼──────────┤
│ LATENCY                  │   0.4642 │
├──────────────────────────┼──────────┤
│ LATENCY_95               │   0.6452 │
├──────────────────────────┼──────────┤
│ TOP_5_LATENCY            │   0.7156 │
├──────────────────────────┼──────────┤
│ TPOT                     │   0.0265 │
├──────────────────────────┼──────────┤
│ THROUGHPUT               │ 100.162  │
├──────────────────────────┼──────────┤
│ THROUGHPUT_INPUT_TOKENS  │  62.4664 │
├──────────────────────────┼──────────┤
│ THROUGHPUT_OUTPUT_TOKENS │  37.6953 │
├──────────────────────────┼──────────┤
│ RPS                      │   2.154  │
├──────────────────────────┼──────────┤
│ LONGER_THAN_60_LATENCY   │   0      │
├──────────────────────────┼──────────┤
│ LONGER_THAN_120_LATENCY  │   0      │
├──────────────────────────┼──────────┤
│ LONGER_THAN_180_LATENCY  │   0      │
├──────────────────────────┼──────────┤
│ FAILED_REQUESTS          │   0      │
├──────────────────────────┼──────────┤
│ FAILED_REQUESTS_PER_HOUR │   0      │
╘══════════════════════════╧══════════╛

For more information on available commands and options, run:

compressa-perf --help

License

This project is licensed under the MIT License. See the LICENSE file for more details.