Skip to content

Conversation

@jyeshe
Copy link

@jyeshe jyeshe commented Sep 11, 2025

The elmdb-rs will now depend on lmdb-master-rs, a fork of lmdb-rs that uses lmdb-master-sys.

It has the FOR-141 merged on it to avoid loosing data of huge hydrations after restarting HB frequently.

@jyeshe
Copy link
Author

jyeshe commented Sep 15, 2025

Validation steps

  1. Write 100k KVs
1> LmdbStore = #{~"store-module" => hb_store_lmdb, ~"name" => ~"cache-mainnet/mylmdb"}.
2> Ids = lists:map(fun(Int) -> {ok, Id} = dev_message:id(integer_to_binary(Int)), Id end, lists:seq(1, 100000)).
[<<"a4ayc_80_OGda4BO_1o_V0etpOqiLx1JwB5S3beHW0s">>,
 <<"1HNeOiZeFu7gP1lxi5tdAwGcB9i2xR-Q2jpmbuwTqzU">>,
 <<"TgdAhWK-24tgzgXB3s_jrRa3IjCWfeAfZAt-Rym0n84">>
 <<...>>|...]
3> lists:foreach(fun(Id) -> hb_store_lmdb:write(LmdbStore, Id, <<Id/binary, Id/binary>>) end, Ids).
ok
  1. Restart HyperBEAM
  2. Read and Compare the 100 KVs
1> LmdbStore = #{~"store-module" => hb_store_lmdb, ~"name" => ~"cache-mainnet/mylmdb"}.
#{<<"name">> => <<"cache-mainnet/mylmdb">>,
  <<"store-module">> => hb_store_lmdb}
2> Ids = lists:map(fun(Int) -> {ok, Id} = dev_message:id(integer_to_binary(Int)), Id end, lists:seq(1, 100000)).
[<<"a4ayc_80_OGda4BO_1o_V0etpOqiLx1JwB5S3beHW0s">>,
 <<"1HNeOiZeFu7gP1lxi5tdAwGcB9i2xR-Q2jpmbuwTqzU">>,
 <<"TgdAhWK-24tgzgXB3s_jrRa3IjCWfeAfZAt-Rym0n84">>
 <<...>>|...]
3> lists:dropwhile(fun(Id) -> V = <<Id/binary, Id/binary>>, maybe {ok, V} ?= hb_store_lmdb:read(LmdbStore, Id), true end end, Ids).

@jyeshe jyeshe changed the title impr: bump LMDB and increase LMDB capacity impr: bump LMDB by adopting lmdb-master-sys Sep 26, 2025
@jyeshe
Copy link
Author

jyeshe commented Sep 26, 2025

@twilson63 it's reverted here to use the DEFAULT_SIZE.

jyeshe and others added 4 commits October 13, 2025 16:03
* chore: allow LMDB to flush to disk

This is important as the database is opened with NO_SYNC

* chore: handle app stop callbacks to flush and close LMDB
@jyeshe
Copy link
Author

jyeshe commented Oct 13, 2025

@samcamwilliams this is rebased and tested again with one process.

@jyeshe jyeshe requested a review from samcamwilliams October 13, 2025 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants