Skip to content

Conversation

waghanza
Copy link
Collaborator

@waghanza waghanza commented Sep 3, 2025

Missing frameworks

  • yada
  • grip
  • hunt
  • chubbyts
  • chubbyts-uwebsockets
  • nestjs-fastify
  • scorper
  • comet
  • hamlet
  • ice
  • phalcon
  • argan

Replace #8577

@waghanza waghanza merged commit 3c20ceb into master Sep 3, 2025
1 of 257 checks passed
@waghanza waghanza deleted the update branch September 3, 2025 06:20
@waghanza waghanza changed the title update New releaseupdat Sep 3, 2025
@waghanza waghanza changed the title New releaseupdat New release Sep 3, 2025
@waghanza
Copy link
Collaborator Author

waghanza commented Sep 3, 2025

@kanarus @trikko @cyrusmsk The results were updated

@trikko
Copy link
Contributor

trikko commented Sep 3, 2025

So now you're running with connection: close, aren't you?

@waghanza
Copy link
Collaborator Author

waghanza commented Sep 3, 2025

No, I use disable-keepalive from https://github.com/hatoo/oha#usage. You can ping @hatoo for more info

PS : This option prevent TCP connection reuse tho

@cyrusmsk
Copy link
Contributor

cyrusmsk commented Sep 3, 2025

Though 2 solutions somehow still seems to find a way to cache or reuse connections?
Kemal(crystal) and Silent(rust)

@waghanza
Copy link
Collaborator Author

waghanza commented Sep 3, 2025

Thanks for your investigation @cyrusmsk.

How have you figured this ?

Just to find a propre solution, maybe we can ping @hubertshelley @sdogruyol.

The idea is to ensure TSP connection is not reused

@cyrusmsk
Copy link
Contributor

cyrusmsk commented Sep 3, 2025

I mean only 2 solutions that are showing 3 times higher values than anything else

Also Latency data is broken in this release

@trikko
Copy link
Contributor

trikko commented Sep 3, 2025

Of course, --disable-keepalive adds Connection: close, since in HTTP/1.1 keep-alive is enabled by default unless explicitly disabled.

With plain oha:

GET / HTTP/1.1
accept: */*
accept-encoding: gzip, compress, deflate, br
user-agent: oha/1.9.0
host: localhost:8123

With oha --disable-keepalive:

GET / HTTP/1.1
accept: */*
accept-encoding: gzip, compress, deflate, br
user-agent: oha/1.9.0
host: localhost:8123
connection: close

I always hope you also check how closely different clients adhere to the HTTP standard. For example, the spec requires all servers to include a Date header in their responses, but generating it involves an extra syscall that slightly hurts performance, so many implementations simply skip it (that’s just one example).

@waghanza
Copy link
Collaborator Author

waghanza commented Sep 3, 2025

I mean only 2 solutions that are showing 3 times higher values than anything else

So, you are not sure of what causing higher results

Also Latency data is broken in this release

Indeed, thanks for catch

@trikko
Copy link
Contributor

trikko commented Sep 3, 2025

I mean only 2 solutions that are showing 3 times higher values than anything else

So, you are not sure of what causing higher results

I have a suspicion. When a client sends Connection: close, the server should respond with Connection: close and close the connection.

I tried sending a test request to a Crystal Kemal server, and it not only does not respond with Connection: close but keeps the connection open. Maybe then oha seeing that it remains open, tries to send further requests?

Example:

curl -H "Connection: close" -v http://localhost:3000
* Host localhost:3000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:3000...
* connect to ::1 port 3000 from ::1 port 47742 failed: Connection refused
*   Trying 127.0.0.1:3000...
* Connected to localhost (127.0.0.1) port 3000
* using HTTP/1.x
> GET / HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/8.12.1
> Accept: */*
> Connection: close
> 
* Request completely sent off
< HTTP/1.1 200 OK
< X-Powered-By: Kemal
< Content-Type: text/html
< Date: Wed, 03 Sep 2025 09:00:55 GMT
< Content-Length: 12
< 
* Connection #0 to host localhost left intact

The response is missing Connection: close, and curl reports the connection as “intact,” just like when keep-alive is active.

Serverino:

* Host localhost:8080 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8080...
* connect to ::1 port 8080 from ::1 port 42542 failed: Connessione rifiutata
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080
* using HTTP/1.x
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.12.1
> Accept: */*
> connection: close
> 
* Request completely sent off
< HTTP/1.1 200 OK
< date: Wed, 03 Sep 2025 09:02:53 GMT
< connection: close
< content-length: 0
< content-type: text/html;charset=utf-8
< 
* shutting down connection #0

@trikko
Copy link
Contributor

trikko commented Sep 3, 2025

(it was just one more case where we need some checks on implementation)

@kanarus
Copy link
Contributor

kanarus commented Sep 3, 2025

I locally tried silent's benchmark code at current main (https://github.com/the-benchmarker/web-frameworks/blob/3c20cebf7b3358a573fcff4fc328b9aeb50a60b7/rust/silent/src/main.rs):

RESULT

  1. silent's original benchmark code uses wrong port 30001.
  2. So all oha's resuests for silent's implementation fail and shows Success rate: 0.00% and other meaningless stats, but oha reports very high Requests/sec (see my repo's silent/.oha-original)
    It may be just something like limit value of oha's request rate at each condition?
  3. I patched the silent's code to fix port to 3000 and did my benchmark. The result silent/.oha seems reasonable.
  4. As a reference: ohkami-nio and ohkami-tokio's results ohkami-nio/.oha ohkami-tokio/.oha show that they correctly work at generally better performance than silent as expected.

CONCLUSION

If a framework shows non-100.00% Success rate for oha's workload, its Request/sec value will be completely meaningless and we should remove it from the final results.

@trikko
Copy link
Contributor

trikko commented Sep 3, 2025

Nice catch!

@kanarus
Copy link
Contributor

kanarus commented Sep 3, 2025

But Kemal's code seems correctly binding to port 3000 and I have no idea for reason of the result now

@waghanza
Copy link
Collaborator Author

waghanza commented Sep 3, 2025

I'm aware that checking http compliance is realy a must have.

If you have any idea of industry level tools to check that, feel free to open a discussion about this. It is clearly on the roadmap but not so easy to implement with the bandwidth I have

@trikko
Copy link
Contributor

trikko commented Sep 3, 2025

If you need an idea, try to run this against the server you tests. I use the same paths you define for tests.
https://github.com/trikko/http-validator

Just checkout that repo and run dub.

@waghanza
Copy link
Collaborator Author

waghanza commented Sep 3, 2025

Yeah, I plan to do that but I mean I'm quite sure that it existe an app to valide http compliance, like https://smithy.io/2.0/additional-specs/http-protocol-compliance-tests.html

@waghanza
Copy link
Collaborator Author

waghanza commented Sep 3, 2025

The long term goal is to make some score for each frameworks with a mix of indicators, and http compliance is one of them

@karlivory
Copy link
Contributor

karlivory commented Sep 5, 2025

I mean only 2 solutions that are showing 3 times higher values than anything else

So, you are not sure of what causing higher results

I have a suspicion. When a client sends Connection: close, the server should respond with Connection: close and close the connection.

I tried sending a test request to a Crystal Kemal server, and it not only does not respond with Connection: close but keeps the connection open. Maybe then oha seeing that it remains open, tries to send further requests?

Example:

curl -H "Connection: close" -v http://localhost:3000
* Host localhost:3000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:3000...
* connect to ::1 port 3000 from ::1 port 47742 failed: Connection refused
*   Trying 127.0.0.1:3000...
* Connected to localhost (127.0.0.1) port 3000
* using HTTP/1.x
> GET / HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/8.12.1
> Accept: */*
> Connection: close
> 
* Request completely sent off
< HTTP/1.1 200 OK
< X-Powered-By: Kemal
< Content-Type: text/html
< Date: Wed, 03 Sep 2025 09:00:55 GMT
< Content-Length: 12
< 
* Connection #0 to host localhost left intact

The response is missing Connection: close, and curl reports the connection as “intact,” just like when keep-alive is active.

Serverino:

* Host localhost:8080 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8080...
* connect to ::1 port 8080 from ::1 port 42542 failed: Connessione rifiutata
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080
* using HTTP/1.x
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.12.1
> Accept: */*
> connection: close
> 
* Request completely sent off
< HTTP/1.1 200 OK
< date: Wed, 03 Sep 2025 09:02:53 GMT
< connection: close
< content-length: 0
< content-type: text/html;charset=utf-8
< 
* shutting down connection #0

If the client sends a request with header "connection: close", the spec requires that the server closes the socket but it does not require the server to echo back "connection: close" (it's only recommended).

Hence, checking for the curl debug line

Connection #0 to host localhost left intact

is not a valid diagnostic for http compliance - it just says that curl itself has not closed the socket due to not receiving "connection: close" in response headers. For example, with rust/iron framework...

curl -vH "connection: close" 127.0.0.1:3000 127.0.0.1:3000
*   Trying 127.0.0.1:3000...
* Connected to 127.0.0.1 (127.0.0.1) port 3000
> GET / HTTP/1.1
> Host: 127.0.0.1:3000
> User-Agent: curl/8.5.0
> Accept: */*
> connection: close
>
< HTTP/1.1 200 OK
< Content-Length: 0
< Date: Fri, 05 Sep 2025 05:37:19 GMT
<
* Connection #0 to host 127.0.0.1 left intact
* Found bundle for host: 0x5a731cd1ab60 [serially]
* Can not multiplex, even if we wanted to
* Connection 0 seems to be dead
* Closing connection
* Hostname 127.0.0.1 was found in DNS cache
*   Trying 127.0.0.1:3000...
* Connected to 127.0.0.1 (127.0.0.1) port 3000
> GET / HTTP/1.1
> Host: 127.0.0.1:3000
> User-Agent: curl/8.5.0
> Accept: */*
> connection: close
>
< HTTP/1.1 200 OK
< Content-Length: 0
< Date: Fri, 05 Sep 2025 05:37:19 GMT
<
* Connection #1 to host 127.0.0.1 left intact

...the server did not reply with connection: close but it did close the socket (i.e. it is still compliant).

@trikko
Copy link
Contributor

trikko commented Sep 5, 2025

Good point, you’re right.

@waghanza
Copy link
Collaborator Author

waghanza commented Sep 7, 2025

@trikko @kanarus @cyrusmsk Maybe the fact that kemal use reuse_port explains hight results

@trikko
Copy link
Contributor

trikko commented Sep 7, 2025

If this is the reason, from the next test you should see a spike for serverino, since I implemented it some days ago, adding a .setDaemonInstances() to improve parallelization

@waghanza
Copy link
Collaborator Author

waghanza commented Sep 7, 2025

We will see after #8646 merged then @trikko, should be tomorrow 😛

@trikko
Copy link
Contributor

trikko commented Sep 7, 2025

Wait, code it is not updated here and I'm out of home 😃

@kanarus
Copy link
Contributor

kanarus commented Sep 7, 2025

I found the same thing as slient happening on kemal.

I added kemal to my repo and did the benchmark. results: kemal/.oha

As you see, every thing is failing, showing Success rate: 0.00% with other meaningless stats, and oha reports extremely higher Request/sec (may be something like limit rate of oha). Same situation as silent's!

The reason of the failure seems some bug of kemal:

crystal run src/server.cr --release
In src/server.cr:22:11

 22 | Process.fork do
              ^---
Warning: Deprecated Process.fork. Fork is no longer supported.

A total of 1 warnings were found.
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???

The heading part to A total of 1 warnings were found. just displays an warning, but all others tell critical errors to start serving:

curl -v 'http://localhost:3000/user/12345678987654321'
*   Trying 127.0.0.1:3000...
* connect to 127.0.0.1 port 3000 failed: Connection refused
*   Trying ::1:3000...
* connect to ::1 port 3000 failed: Connection refused
* Failed to connect to localhost port 3000 after 0 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 3000 after 0 ms: Connection refused

By the way

require "kemal"

Kemal.config do |cfg|
  cfg.env = "production"
  cfg.serve_static = false
  cfg.logging = false
end

get "/" do |env|
  nil
end

get "/user/:id" do |env|
  env.params.url["id"]
end

post "/user" do |env|
  nil
end

- System.cpu_count.times do |i|
-   Process.fork do
-     Kemal.run do |config|
-       server = config.server.not_nil!
-       server.bind_tcp "0.0.0.0", 3000, reuse_port: true
-     end
-   end
- end
- 
- sleep
+ Kemal.run

successfully works:

crystal run src/server.cr
2025-09-07T11:24:13.945462Z   INFO - kemal: [production] Kemal is ready to lead at http://0.0.0.0:3000
^C2025-09-07T11:24:39.295384Z   INFO - kemal: Kemal is going to take a rest!
curl -v 'http://localhost:3000/user/1234567890987654321'
*   Trying 127.0.0.1:3000...
* Connected to localhost (127.0.0.1) port 3000 (#0)
> GET /user/1234567890987654321 HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: keep-alive
< X-Powered-By: Kemal
< Content-Type: text/html
< Date: Sun, 07 Sep 2025 11:24:35 GMT
< Content-Length: 19
<
* Connection #0 to host localhost left intact
1234567890987654321⏎

cf. original code: https://github.com/the-benchmarker/web-frameworks/blob/3c20cebf7b3358a573fcff4fc328b9aeb50a60b7/crystal/kemal/src/server.cr

CONCLUSION (AGAIN)

If a framework shows non-100.00% Success rate for oha's workload, its Request/sec value will be completely meaningless and we should remove it from the final results.

@kanarus
Copy link
Contributor

kanarus commented Sep 7, 2025

additionally, here

System.cpu_count.times do |i|
  Process.fork do

doesn't seem an error factor: just

- System.cpu_count.times do |i|
-   Process.fork do
    Kemal.run do |config|
      server = config.server.not_nil!
      server.bind_tcp "0.0.0.0", 3000, reuse_port: true
    end
-   end
- end
- 
- sleep

causes the same error:

crystal run src/server.cr
Unhandled exception: Could not bind to '0.0.0.0:3000': Address already in use (Socket::BindError)
  from /usr/share/crystal/src/socket/addrinfo.cr:94:9 in 'initialize:reuse_port'
  from /usr/share/crystal/src/socket/tcp_server.cr:36:3 in 'new:reuse_port'
  from /usr/share/crystal/src/http/server.cr:213:18 in 'bind_tcp'
  from lib/kemal/src/kemal.cr:55:7 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:129:5 in 'main_user_code'
  from /usr/share/crystal/src/crystal/main.cr:115:7 in 'main'
  from /usr/share/crystal/src/crystal/system/unix/main.cr:9:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/kanarus/.cache/crystal/crystal-run-server.tmp in '_start'
  from ???

@waghanza
Copy link
Collaborator Author

waghanza commented Sep 7, 2025

Wait, code it is not updated here and I'm out of home 😃

The latest version is always taken since container are distroyed on each run, but I can expliclty set version condition 😛

@waghanza
Copy link
Collaborator Author

waghanza commented Sep 7, 2025

System.cpu_count.times do |i|
  Process.fork do

Not sure why this code is here. Since official example use only

Kemal.run

https://github.com/kemalcr/kemal/blob/e684d2ec9fac727bf63865c738d6db7529a6924c/examples/hello-world/app.cr#L7
I switch on it

@trikko
Copy link
Contributor

trikko commented Sep 7, 2025

Wait, code it is not updated here and I'm out of home 😃

The latest version is always taken since container are distroyed on each run, but I can expliclty set version condition 😛

But I need to update the code here to enable the parallelism 😃

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants