Memory mapped files (POSIX API) -- what is good example to start from

[vladp]

I am evaluating whether I can build a Janet-native (that is no C-modules) package to interact with lmdb database files (and eventually be able to implement LMDB-compatible key/value store APIs in Janet).

Part of the puzzle -- is that I will need to use memory mapped files facilities as these LMDB files can be very large (more than RAM memory).

Which is why I wanted to ask for a recommendation of existing packages that may deal with OS-specific APIs (for windows and for POSIX oses, as an example).

[CosmicToast]

Are you planning to sync against a buffer?
Generally speaking, memory mapped files help more when a char[] is a native exchange format for a given language, but that's not really the case in Janet.
The user experience with memory-mapped files inside of Janet will be fairly close to the user experience with regular files.
As such, can you elaborate a bit on how specifically you plan to implement things such that the implementation detail matters?
Maybe a mock-up of what the API for mmap/ vs file/ would ideally be like?

[kamisori]

this could be used to implement something like having a janet implementation of hdf5py depending on a binding to some numerical library (like numpy in case of h5py), stricly hypothetically speaking.

i think, you cant really say char[] isnt a native exchange format, opening a file in binary mode and reading it into a char buffer was always an option too. also now with cgen you can think of janet kind of like a c scripting language, and with ffi and i also saw jit somwhere.. wild stuff

[vladp]

thank you for the follow ups!

The file opening mockup/example (well in C as in orginal implementation) is
line 3712 (windows) and 3731 (posix)
https://github.com/minad/lmdb/blob/master/ext/lmdb_ext/liblmdb/mdb.c

[bakpakin]

The ffi module has facilities for wrapping pointers returned from c functions with buffers that won’t be realloced. This would let you interact with the memory without copying, although I haven’t thought about all the implications.