Update (July 2024)
The book is getting really hard to understand.
I'm on Chapter 7 and I don't really know what's going on.
Promises, timer handles, libcurl...
The book doesn't really explain what we're doing and why.
It's probably written for much more experienced people
who strongly internalized low-level stuff.
Anyway, please keep on reading. Thanks for dropping by!
Updating the code in Modern Systems Programming with Scala Native to
- Scala 3.5.0+,
- Scala Native version 0.5.0+,
scala-cliversion 1.4.0+, and- not using the Docker container provided by the book's website.
So I...
- changed the syntax to Scala 3 syntax:
- removed all the optional braces, added the fewer braces syntax,
- replaced
if (...) {...} else {...}withif ... then ... else ...everywhere, using Python-style indentation, - removed the
returnkeyword, replacedNonLocalReturnsusage with the newutil.boundaryandboundary.break, - changed all the
snake_casenames tocamelCase, - got rid of unnecessary
mainobject wrappings and used@mainannotations instead, - changed Bash scripts to Scala-cli scripts,
- and so on.
We are using Scala-cli, so SBT (or Mill, or any other build tool) is not needed.
For Scala Native, you'll need the requirements such as Clang / LLVM stuff as listed on Scala Native page.
You can compile and run @main methods in VS Code with Metals by clicking the run button above them:
There are 35+ @main methods in the project. To compile a specific one to a binary, you can use inside the root directory, for example:
scala-cli package . --main-class ch08.simplePipe.runThis will place the binary executable in the project root directory:
Wrote /home/spam/Projects/modern-systems-scala-native/ch08.simplePipe.run, run it with
./ch08.simplePipe.runHere the class is the import path to the method: ch08 and simplePipe are package names, and run is the name of the @main method:
package ch08
package simplePipe
// ...
@main
def run: Unit = ??? // so this is ch08.simplePipe.runIf in doubt, you can use the --interactive mode, which lets you pick the @main method you want:
$ scala-cli package . --interactive
Found several main classes. Which would you like to run?
[0] ch01.helloWorld.run
[1] ch06.asyncTimer.run
[2] ch09.jsonSimple.run
[3] ch05.httpServer.run
[4] ch08.fileOutputPipe.run
[5] ch01.helloNative.run
[6] ch06.asyncHttp.run
[7] ch09.lmdbSimple.run
[8] ch08.fileInputPipe.run
[9] ch01.testNullTermination.run
[10] ch01.cStringExperiment1.run
[11] ch01.sscanfIntExample.run
[12] ch01.testingBadStuff.run
[13] ch08.filePipeOut.run
[14] ch02.aggregateAndCount.run
[15] ch01.goodSscanfStringParse.run
[16] ch01.badSscanfStringParse.run
[17] ch04.badExec.run
[18] ch07.simpleAsync.run
[19] ch06.asyncTcp.run
[20] ch03.httpClient.run
[21] ch08.simplePipe.run
[22] ch01.maxNgramFast.run
[23] ch07.curlAsync.run
[24] ch02.sortByCount.run
[25] ch08.filePipe.run
[26] ch03.tcpClient.run
[27] ch01.maxNgramNaive.run
[28] ch04.nativePipeTwo.run
[29] ch01.moreTesting.run
[30] ch01.cStringExperiment2.run
[31] ch04.nativeFork.run
[32] ch04.nativePipe.run
[33] ch10.libUvService.run
[34] ch01.bug.run
[35] ch07.timerAsync.run
21
[info] Linking (multithreadingEnabled=true, disable if not used) (2353 ms)
[info] Discovered 1119 classes and 7040 methods after classloading
[info] Checking intermediate code (quick) (76 ms)
[info] Discovered 1050 classes and 5504 methods after optimization
[info] Optimizing (debug mode) (2199 ms)
[info] Produced 9 LLVM IR files
[info] Generating intermediate code (1689 ms)
[info] Compiling to native code (3083 ms)
[info] Linking with [pthread, dl, uv]
[info] Linking native code (immix gc, none lto) (239 ms)
[info] Postprocessing (0 ms)
[info] Total (9395 ms)
Wrote /home/spam/Projects/modern-systems-scala-native/ch08.simplePipe.run, run it with
./ch08.simplePipe.runThe book uses @link and = extern constructs of Scala Native to link with libraries such as libuv, libcurl and liblmdb. For example:
@link("lmdb")
@extern
object LmdbImpl:
def mdb_env_create(env: Ptr[Env]): Int = extern
def mdb_env_open(env: Env, path: CString, flags: Int, mode: Int): Int = externOn Ubuntu I had to install these (I think libcurl might have been pre-installed already?):
sudo apt install clang libuv1-dev libcurl4-gnutls-dev liblmdb-dev libhttp-parser-devThe author did all of this work. But if we wanted to do this on our own,
it would be difficult to get right the type signatures of the functions.
Scala Native main contributor's advice is to directly take
the header file of such a library,
and use sn-bindgen to generate the bindings:
I haven't tried that myself, but that's the way to go.
The http-parse library of Chapter 10 is no longer maintained.
It was ported to llhttp.
It is possible to install this on Ubuntu with
sudo apt install node-llhttpBut I don't know how to link it with Scala Native. It's written in Typescript, which generates C output. The output then has to be compiled, and then linked to SN.
I modified the install_gatling.sh script from the book, now it's a Scala-cli
script scripts/installGatling.sc with Gatling bundle version 3.10.5+.
From the root directory, run
./scripts/installGatling.scThis will download into a folder gatling in the root directory,
and copy the simulation file from chapter 5 into the relevant subdirectory.
You need to compile and run the HTTP server from chapter 5. (Also on chapter 7.) Read the compilation message for the name of the binary executable:
scala-cli package . --main-class ch05.httpServer.run
...
Wrote /home/spam/Projects/modern-systems-scala-native/project, run it with
./projectThen run that to start the server. This starts the server listening on port 8080.
I wrote another script scripts/runGatling.sc that sets up the needed environment variables
then handles the interactive simulation for you by providing necessary inputs.
This will compile the simulation file under gatling/user-files/simulations/.
These files have to be written in Scala 2.13 unfortunately!
Gatling cannot handle Scala 3. So... run the simulation with:
./scripts/runGatling.scHere's what the Terminal output looks like:
$ ./scripts/runGatling.sc
Finished setting up environment variables for Gatling simulation.
Now running the Gatling binary:
GATLING_HOME is set to /home/spam/Projects/modern-systems-scala-native/gatling
Do you want to run the simulation locally, on Gatling Enterprise, or just package it?
Type the number corresponding to your choice and press enter
[0] <Quit>
[1] Run the Simulation locally
[2] Package and upload the Simulation to Gatling Enterprise Cloud, and run it there
[3] Package the Simulation for Gatling Enterprise
[4] Show help and exit
>>>>> Choosing option [1] to run locally!
Gatling 3.11.1 is available! (you're using 3.10.5)
ch05.loadSimulation.GenericSimulation is the only simulation, executing it.
Select run description (optional)
>>>>> Providing optional name: testSim
Simulation ch05.loadSimulation.GenericSimulation started...
================================================================================
2024-04-28 18:21:04 GMT 2s elapsed
---- Requests ------------------------------------------------------------------
> Global (OK=5000 KO=0 )
> Web Server (OK=5000 KO=0 )
---- Test scenario -------------------------------------------------------------
[##########################################################################]100%
waiting: 0 / active: 0 / done: 100
================================================================================
Simulation ch05.loadSimulation.GenericSimulation completed in 2 seconds
Parsing log file(s)...
Parsing log file(s) done in 0s.
Generating reports...
================================================================================
---- Global Information --------------------------------------------------------
> request count 5000 (OK=5000 KO=0 )
> min response time 5 (OK=5 KO=- )
> max response time 116 (OK=116 KO=- )
> mean response time 38 (OK=38 KO=- )
> std deviation 15 (OK=15 KO=- )
> response time 50th percentile 35 (OK=35 KO=- )
> response time 75th percentile 50 (OK=50 KO=- )
> response time 95th percentile 64 (OK=64 KO=- )
> response time 99th percentile 72 (OK=72 KO=- )
> mean requests/sec 2500 (OK=2500 KO=- )
---- Response Time Distribution ------------------------------------------------
> t < 800 ms 5000 (100%)
> 800 ms <= t < 1200 ms 0 ( 0%)
> t >= 1200 ms 0 ( 0%)
> failed 0 ( 0%)
================================================================================
Reports generated, please open the following file: file:///home/spam/Projects/modern-systems-scala-native/gatling/results/genericsimulation-20240428182101491/index.htmlThe graphical results are in gatling/results/.../index.html.
With 1000 users and 50000 requests, I got 1% failure rate
(connection timeouts), and 300ms average response time.
Quite amazing!
If I use the async server using libuv and the event loop in chapter 7,
then again with 1000 users and 50000 requests,
I get 100% success with 231ms mean response time! Great!
I noticed many things have changed.
There are lines of code in the zip file provided on the book's website. Some of these are also printed in the book!
For example, in Chapter 4's nativeFork there is
for (j <- (0 to count)) {
}which does nothing. There is also
val pid = unistd.getpid()which is never used. There are also illegal things like:
val p = SyncPipe(0)
val p = FilePipe(c"./data.txt")There are lots of other examples. There are also many unused / unnecessary imports in the files. Whenever I ran into these, I removed them.
There is also a lot of code duplication, I suppose, to make each individual file "runnable" by itself. I removed redundant code by adding package declarations, then importing the duplicated code from other files instead.
For example, Chapter 4's badExec.scala duplicates a lot of code from nativeFork.scala. I solved it by separating duplicate code into a file, and adding package declarations:
// this is common.scala
package ch04
// ...// this is nativeFork.scala
package ch04
package nativeFork
// ...
// then use code from common.scala here// this is badExec.scala
package ch04
package badExec
// ...
// then use code from common.scala hereThere is a lot of this duplication in later chapters. I fixed them.
The book uses Ints for a lot of calculations such as string length, how much memory should be allocated, etc. But the current version of Scala Native is using CSize for these now. So the Ints have to be converted. CSize / USize are actually ULong, so we need .toCSize, or .toUSize, or .toULong conversion. For this, we need to import:
import scalanative.unsigned.UnsignedRichLongThis also works:
import scalanative.unsigned.UnsignedRichIntMoreover, we are now able to use direct comparison between CSize / USize types and Int. For example:
// here strlen returns CSize, normally we would have to do 5.toULong
if string.strlen(myCString) != 5 then ???There are many function calls in the book that only take type arguments and no value arguments, such as stackalloc[Int] etc. This is because there is a default argument n with value 1 if none is provided, and in Scala 2 we can drop empty parentheses: stackalloc[Int] instead of stackalloc[Int]().
In Scala 3, we need to provide the empty parentheses for the default parameter of 1, or just provide 1 as an argument:
stackalloc[Int] // does not work in Scala 3
stackalloc[Int]() // this defaults to n = 1
stackalloc[Int](1) // same as previousThe book uses things like
val server_sockaddr = server_address.cast[Ptr[sockaddr]].cast is no longer available; we use .asInstanceOf[...] instead.
Function pointer classes now have different syntax. The book overrides classes like CFuncPtr2 by providing a custom apply method like so:
val by_count = new CFuncPtr2[Ptr[Byte],Ptr[Byte],Int] {
def apply(p1:Ptr[Byte], p2:Ptr[Byte]):Int = {
val ngram_ptr_1 = p1.asInstanceOf[Ptr[NGramData]]
val ngram_ptr_2 = p2.asInstanceOf[Ptr[NGramData]]
val count_1 = ngram_ptr_1._2
val count_2 = ngram_ptr_2._2
return count_2 - count_1
}
}We can no longer do this, as these classes are declared final. We must use the companion object's fromScalaFunction[...] method instead (which is nicer, since we don't have to remember that we have to implement def apply):
val byCount = CFuncPtr2.fromScalaFunction[Ptr[Byte], Ptr[Byte], Int]:
(p1: Ptr[Byte], p2: Ptr[Byte]) =>
val ngramPtr1 = p1.asInstanceOf[Ptr[NGramData]]
val ngramPtr2 = p2.asInstanceOf[Ptr[NGramData]]
ngramPtr2._2 - ngramPtr1._2The book and the code have some inconsistencies. There are sometimes two different names for the same thing, and the types are also different: For example:
// these are supposed to be the same thing.
type Timer = Ptr[Ptr[Byte]] // book, ch06
type TimerHandle = Ptr[Byte] // book, later in the same chapter
type TimerHandle = Ptr[Byte] // code, in ch06
type TimerHandle = Ptr[Ptr[Byte]] // code, in other chaptersThe book clearly says, in a "warning box":
There are many more issues. For example, given:
type TCPHandle = Ptr[Ptr[Byte]] // book and code
type ClientState = CStruct3[Ptr[Byte], CSize, CSize]but then:
val closeCB = CFuncPtr1.fromScalaFunction[TCPHandle, Unit]:
(client: TCPHandle) =>
// ...
val clientStatePtr = (!client).asInstanceOf[Ptr[ClientState]]Since client is TCPHandle = Ptr[Ptr[Byte]], !client is Ptr[Byte].
So we are casting a Ptr[Byte] into a Ptr[CStruct3[Ptr[Byte], CSize, CSize]]!
Does this imply Byte = CStruct3[Ptr[Byte], CSize, CSize]?
No, it does not work that way I think... 😕
There are many more instances of this. For example, given
type TCPHandle = Ptr[Ptr[Byte]]
type ShutdownReq = Ptr[Ptr[Byte]]we have:
def shutdown(client: TCPHandle): Unit =
val shutdownReq = malloc(uv_req_size(UV_SHUTDOWN_REQ_T)).asInstanceOf[ShutdownReq]
!shutdownReq = client.asInstanceOf[Ptr[Byte]]Again, here !shutdownReq is a Ptr[Byte], but client is a Ptr[Ptr[Byte]].
So we are trying to squeeze a Ptr[Ptr[Byte]] into a Ptr[Byte]!
We do this by pretending that the nested pointer does not exist with asInstanceOf[].
OK fine, we can trick the compiler this way, but can we later actually use the inner
nested pointer of client correctly?
Because later these are passed to actual libuv functions...
😕 😕 😕
Big brain moment: basically, pretty much anything can be cast to Ptr[Byte]...
Since "everything is a byte", the "beginning of a block of anything" is a Ptr[Byte]!
🧠 🧠 🧠 🎉 🎊 🥳
Not sure how to handle this, it will be guesswork. If compilation fails during linking phase then I'll know the types are wrong. But if linking does not fail, then I'll have to figure it out from the execution.
The book uses the usual C idiom of allocating memory that is 1 more than the length of a string, copying it, then manually null-terminating the new copy:
val string_ptr = toCString(arg) // prepare pointer for malloc
val string_len = string.strlen(string_ptr) // calculate length of string to be copied
val dest_str = stdlib.malloc(string_len + 1).asInstanceOf[Ptr[Byte]] // alloc 1 more
string.strncpy(dest_str, string_ptr, arg.size + 1) // copy
dest_str(string_len) = 0 // manually null-terminate the new copyIf you do this you'll get errors: first is the CSize errors:
arg.size + 1when you are trying to add 1, which is Int, to string_len, which is CSize, for which you have to use .toUSize.
The second is none of the overloaded alternatives for method update of Ptr[Byte]... which complains when we are trying to manually null-terminate the new copy of the string:
dest_str(string_len) = 0It has to be Byte instead.
Fixing all these problems and rewriting in Scala 3 style, we get:
val stringPtr = toCString(arg) // prepare pointer for malloc
val strLen = string.strlen(stringPtr) // calculate length of string to be copied
val destStr = stdlib.malloc(strLen + 1.toUSize) // alloc 1 more
string.strncpy(destStr, stringPtr, strLen) // copy JUST the string, not \0
destStr(strLen) = 0.toByte // manually null-terminate the new copyor we can simply copy the string, including the null-terminator:
val stringPtr = toCString(arg) // prepare pointer for malloc
val strLen = string.strlen(stringPtr) // calculate length of string to be copied
val destStr = stdlib.malloc(strLen + 1.toUSize) // alloc 1 more
string.strncpy(destStr, stringPtr, strLen + 1.toUSize) // copy, including \0If we for some reason don't trust strncpy and want extra super-duper safety, we can do both:
val stringPtr = toCString(arg) // prepare pointer for malloc
val strLen = string.strlen(stringPtr) // calculate length of string to be copied
val destStr = stdlib.malloc(strLen + 1.toUSize) // alloc 1 more
string.strncpy(destStr, stringPtr, strLen + 1) // copy, including \0
destStr(strLen) = 0.toByte // null-terminate the new copy, JUST IN CASE!Now it's null terminated twice: once with the copying, then again manually.
The book uses the old-school C-style "argv" approach to command-line arguments from Scala 2:
object Main {
def main(args: Array[String]): Unit = {
???
}
}This does not work with Scala 3 @main annotations, as it will complain about no given instance of type scala.util.CommandLineParser.fromString[Array[String]]... Things have changed in Scala 3 when it comes to main methods, command line arguments and code-running. They have been greatly simplified, the main method no longer has to be named "main", and now there is greater capability to use any user-defined type for the command-line arguments, but the compiler has to be "taught" how to do it.
We could do that by providing the given instance... but instead we fall back on the "arbitrary number of parameters of the same type" approach (and rename the method while we're at it):
@main
def nativePipeTwo(args: String*): Unit = ???In Scala 3.4.1+, Native 0.5.0+, the bad_sscanf_string_parse example given in the book does not cause a segfault like it does in the book. Or rather, we have to use a very long string to get a segfault, like > 100 characters. If we use the author's version (Scala 2.11, Native 0.4.0, and some old SBT version) then it works; we get a segfault immediately with as few as 8 characters every time. It won't segfault even with stackalloc[CString](1).
So I'm gonna drop down into C to see some reliable, reproducible segfault examples.
Well... that produced the same result, only for large string inputs (around 30 characters but not reliably).
./segfault
dddddddddddddddddddddddddd
scan results: dddddddddddddddddddddddd
ddddddddddddddddddddddddddd
scan results: ddddddddddddddddddddddddddd
dddddddddddddddddddddddddddd
malloc(): corrupted top size
Aborted (core dumped)