A tool to validate Kiwix Hotspot's feature set and performances
- Identify behavioral and performance regressions before release
- Document our core feature set
- Document capacity/performance for identified scenarios
The testbench is a suite of integration and load tests that are run from a physical device, targetting a Kiwix Hotspot.
- A running Kiwix Hotspot as target
- A Raspberry Pi 5 4GB as test host
- Connected to Internet via Ethernet or with an hardware clock (for time accuracy)
- Running RaspiOS on an NVMe disk for better performance (you dont want the host to slow down the tests)
- Powered, external USB hub. We use 2 x i-tec USB3 16 port charging Hub
- Several USB WiFi dongles. We use 32 x TP-Link TL-WN725N
Because of limitations in the 802.11 chipset firmwares used and their compatibility with OS, we specifically tests
- How many concurrent WiFi clients can connect? its result configures the integration tests.
- Max WiFi throughtput
- Can connect to specific SSID (
Kiwix Hotspot). - Receives an IPv4 in specified range (
192.168.2.128/25) - Receives a specific IPv4 gateway (
192.168.2.1) - Receives a specific IPv4 DNS server (
192.168.2.1) - Can ping gateway
- Can resolve specific hotspot domains (
kiwix.hotspot,goto.kiwix.hotspot) - Can reach gateway via HTTP (
http://192.168.2.1) - Captive portal is running on specified ports (
http://192.168.2.1:2080andhttps://192.168.2.1:2443) - HTTP request to both
kiwix.hotspotand public domain is captured - Can register to the captive-portal (
GET /register-hotspot/) - Further requests (
http://kiwix.hotspot/) are not captured kiwix-serve- Served on
browse.kiwix.hotspot - Can get a suggestion
- Can get full-text search result
- Can query random article
- Served on
- Additional per-service tests
These behavior test can be tested from a single interface, serving as integration tests but also concurrently to validate that the number of concurrent client means concurrent working clients.
Those consists in an orchestration of parallel tests simulating real traffic from pre-defined scenarios.
Assuming a bookworm raspbian host
# you'll most likely be using it over SSH
systemctl enable --now ssh
# update packages
apt update && apt upgrade
# setup/fix locale (en_US.UTF-8)
dpkg-reconfigure locales
# set a country for WiFi regulations to apply (mandatory!)
raspi-config nonint do_wifi_country FR
# disable power saving on network interfaces (/!\ important)
# https://gist.github.com/jcberthon/ea8cfe278998968ba7c5a95344bc8b55
printf '[connection]\nwifi.powersave = 2\n' > /etc/NetworkManager/conf.d/wifi-powersave-off.conf
systemctl restart NetworkManager
# install ffprobe (to get durations of videos)
curl -L -O https://johnvansickle.com/ffmpeg/releases/ffmpeg-7.0.2-arm64-static.tar.xz
tar xf ffmpeg-7.0.2-arm64-static.tar.xz
mv ffmpeg-7.0.2-arm64-static/ffprobe /usr/local/bin/
rm -rf ./ffmpeg-7.0.2-arm64-static*
# Install Apache JMeter
curl -L -o /tmp/jmeter.tgz https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.6.3.tgz
tar --strip-components 1 -C /usr/local -x -f /tmp/jmeter.tgz
rm -rf /tmp/jmeter.tgzInstall testbench
# you'll want to install in isolated virtual env
python3 -m .venv
source .venv/bin/activate# should you want a release
pip install offspot-testbench# using the trunk
git clone https://github.com/offspot/testbench.git
pip install -e testbenchDownload words list (used in search queries)
# assuming code in /src
mkdir -p /src/src/data/
curl -L -O /src/src/data/words.txt https://raw.githubusercontent.com/dwyl/english-words/refs/heads/master/words_alpha.txtRun the testbench
testbench --helpThere are three sub-commands to the testbench program, serving different needs:
| Command | Description |
|---|---|
status |
Lists the available 802.11 (WiFi) devices available on the host |
integration |
Runs the integration test-suite in parallel over all requested devices |
jmeter |
Runs JMeter Test Plan with all requested devices |
Use this only to find out about your setup. You'll get the list of all WiFi devices connected.
You can use it to tweak the exclude filters to get a list of devices you'll use with next commands.
status.mp4
Always run this when testing a new Hotspot/version to ensure the Hotspot behaves properly with one client or more. Those tests are simple and a failing test is easier to diagnose than JMeter results.
This command is also your way to find out how many concurrent WiFi clients the Hotspot can accept (and sustain to some extent).
integration.mp4
Use this to find out the performance of your Hotspot regarding concurrent access.
The tool connects each requested devices, then runs JMeter and provides very basic statistics. It's up to you to dig into the JMeter results CSV.
perf.mp4
makes a couple HTTP requests to kiwix-serve and dashboard.
Downloads a large (100MB) file
See contrib/dlspeed_results.py to extract results.
Streams a youtube2zim video.
See contrib/stream_results.py to extract results.
Regular Hotspot usage scenario: potential video stream (30% chance), ZIM reading, suggestion, search
While the goal of this tool is to know whether the system runs within predictable performances, the randomness introduced everywhere means it's unpredictable.
Environment plays a huge role as well.
Make sure you change the SSID and protect the network. Unless you're in a faraday cage, chances that random people attempt to connect to the network are significant.
Be aware that not-connected but trying clients may cost you a slot.
There is no default timeout anywhere because some actions leads to incredibly high duration. It's up to you to decide whether it's worth continuing or not.
Consider checking on the target that network traffic is still happenning:
❯ iftop -i wlan0You don't want to be in an unknown state, wondering why stuff are not working the way they should.
- After a JMeter crash, reboot host.
- After a JMeter hang, reboot target.
When you overwhelm an Hotspot, it can freeze due to lack of memory (there's no swap). In this case, even though the Pi light is green, the Pi is not responding and even the ACPI power button is not working. Unplug-replug the target Pi. If the JMX is not timed-out properly, JMeter can hang forever.
The summary tables post-JMeter are built by reading the results CSV file.
When using your own JMX, make sure not to generate Test Status message that spans multiple lines as the CSV parsing is quite primitive.
Obviously, it's easier to work off a real, running device.
# switch to root
❯ sudo su -
# install sshfs and uv
❯ apt update && apt install -y sshfs curl && curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env && mkdir -p /src
# mount you dev's repo on the Pi [CHANGE THIS]
❯ sshfs USER@DEVHOST:PATHONDEVHOST /src && cd /src && uv run testbench --help
# ex: sshfs [email protected]:src/testbench /src && cd /src && uv run testbench --help
# then start any of the entrypoints
# similar to provisionOS, restarts when requested
❯ uv run testbench status
❯ uv run testbench --max-devices 1 integration
❯ uv run testbench --max-devices 1 jmeter --helptestbench adheres to openZIM's Contribution Guidelines.
testbench has implemented openZIM's Python bootstrap, conventions and policies v1.0.0.