Skip to content

Conversation

osmaczko
Copy link
Contributor

@osmaczko osmaczko commented Oct 13, 2025

  • modularize messaging Persistence with wakuPersistence,
    encryptionPersistence, etc.
  • separate migration table for encryption migrations
  • inject encryption sqlite persistence through the interface

iterates: #6792

@status-im-auto
Copy link
Member

status-im-auto commented Oct 13, 2025

Jenkins Builds

Click to see older builds (34)
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ 2a376aa #1 2025-10-13 17:54:35 ~4 min linux/status-go 📦zip
✖️ 2a376aa #1 2025-10-13 17:55:31 ~5 min tests 📄log
✔️ 2a376aa #1 2025-10-13 17:56:37 ~6 min windows/status-go 📦zip
✔️ 2a376aa #1 2025-10-13 17:57:19 ~7 min macos/status-go 📦zip
✔️ 2a376aa #1 2025-10-13 18:03:05 ~13 min linux/nwaku 📦zip
✖️ 2a376aa #1 2025-10-13 18:07:20 ~17 min tests-rpc 📄log
✔️ da7328e #2 2025-10-15 19:07:01 ~4 min linux/status-go 📦zip
✔️ da7328e #2 2025-10-15 19:09:03 ~6 min windows/status-go 📦zip
✖️ da7328e #2 2025-10-15 19:11:57 ~9 min tests 📄log
✔️ da7328e #2 2025-10-15 19:16:15 ~13 min linux/nwaku 📦zip
✔️ da7328e #2 2025-10-15 19:17:20 ~14 min macos/status-go 📦zip
✖️ da7328e #2 2025-10-15 19:22:15 ~19 min tests-rpc 📄log
✔️ b2ed375 #3 2025-10-16 07:52:24 ~7 min windows/status-go 📦zip
✔️ ad98019 #4 2025-10-16 08:01:06 ~9 min linux/status-go 📦zip
✔️ ad98019 #4 2025-10-16 08:01:09 ~8 min windows/status-go 📦zip
✖️ ad98019 #4 2025-10-16 08:07:25 ~15 min tests 📄log
✔️ ad98019 #4 2025-10-16 08:10:05 ~18 min linux/nwaku 📦zip
✔️ ad98019 #4 2025-10-16 08:11:26 ~19 min tests-rpc 📄log
3576e19 #5 2025-10-16 08:57:09 ~4 min windows/status-go 📄log
3576e19 #5 2025-10-16 09:03:42 ~11 min linux/status-go 📄log
✖️ 3576e19 #5 2025-10-16 09:07:09 ~14 min tests 📄log
✖️ 3576e19 #5 2025-10-16 09:09:03 ~16 min tests-rpc 📄log
3576e19 #5 2025-10-16 09:11:32 ~19 min linux/nwaku 📄log
✔️ 53eb2e7 #6 2025-10-16 09:38:06 ~4 min linux/status-go 📦zip
✔️ 53eb2e7 #6 2025-10-16 09:40:11 ~6 min windows/status-go 📦zip
✖️ 53eb2e7 #6 2025-10-16 09:41:02 ~7 min tests 📄log
✔️ 53eb2e7 #6 2025-10-16 09:46:21 ~12 min linux/nwaku 📦zip
✖️ 53eb2e7 #6 2025-10-16 09:48:00 ~14 min tests-rpc 📄log
✔️ 0ee8ad6 #7 2025-10-16 10:54:40 ~8 min windows/status-go 📦zip
✔️ 0ee8ad6 #7 2025-10-16 10:57:20 ~11 min linux/status-go 📦zip
✖️ 0ee8ad6 #7 2025-10-16 11:03:42 ~17 min tests 📄log
✔️ 0ee8ad6 #7 2025-10-16 11:06:23 ~20 min linux/nwaku 📦zip
✔️ 0ee8ad6 #7 2025-10-16 11:08:52 ~22 min tests-rpc 📄log
0ee8ad6 #7 2025-10-16 11:58:50 ~49 min macos/status-go 📄log
Commit #️⃣ Finished (UTC) Duration Platform Result
459e2ec #8 2025-10-16 12:32:00 ~2 min macos/status-go 📄log
✔️ 459e2ec #8 2025-10-16 12:35:22 ~5 min linux/status-go 📦zip
✔️ 459e2ec #8 2025-10-16 12:36:18 ~6 min windows/status-go 📦zip
✔️ 459e2ec #8 2025-10-16 12:44:09 ~14 min linux/nwaku 📦zip
✔️ 459e2ec #8 2025-10-16 12:48:58 ~18 min tests-rpc 📄log
✔️ 459e2ec #8 2025-10-16 13:08:07 ~38 min tests 📄log
459e2ec #9 2025-10-16 13:10:54 ~11 min macos/status-go 📄log
459e2ec #10 2025-10-16 13:50:48 ~1 min macos/status-go 📄log
✔️ 78d6e84 #9 2025-10-16 13:58:57 ~4 min linux/status-go 📦zip
✔️ 78d6e84 #9 2025-10-16 14:02:15 ~7 min windows/status-go 📦zip
✔️ 78d6e84 #11 2025-10-16 14:04:27 ~9 min macos/status-go 📦zip
✔️ 78d6e84 #9 2025-10-16 14:09:48 ~14 min linux/nwaku 📦zip
✖️ 78d6e84 #9 2025-10-16 14:10:50 ~15 min tests-rpc 📄log
✔️ 78d6e84 #9 2025-10-16 14:26:51 ~31 min tests 📄log
✔️ 78d6e84 #10 2025-10-16 18:57:07 ~10 min tests-rpc 📄log

@osmaczko osmaczko marked this pull request as draft October 13, 2025 17:54
@osmaczko osmaczko force-pushed the refactor/messaging-persistence-interfaces branch 7 times, most recently from 0ee8ad6 to 459e2ec Compare October 16, 2025 12:29
@osmaczko osmaczko marked this pull request as ready for review October 16, 2025 12:59
Copy link

codecov bot commented Oct 16, 2025

Codecov Report

❌ Patch coverage is 73.60515% with 369 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.49%. Comparing base (0ac383b) to head (78d6e84).
⚠️ Report is 4 commits behind head on develop.

Files with missing lines Patch % Lines
messaging/layers/encryption/sqlite_persistence.go 78.13% 121 Missing and 43 partials ⚠️
...ayers/encryption/multidevice/sqlite_persistence.go 70.79% 48 Missing and 18 partials ⚠️
protocol/messaging_persistence.go 73.50% 30 Missing and 10 partials ⚠️
...yers/encryption/sharedsecret/sqlite_persistence.go 64.36% 22 Missing and 9 partials ⚠️
sqlite/migrate.go 65.78% 9 Missing and 4 partials ⚠️
...essaging/internal/sqlite_encryption_persistence.go 0.00% 12 Missing ⚠️
...ayers/encryption/hash_ratchet_key_compatibility.go 70.73% 8 Missing and 4 partials ⚠️
messaging/sqlite_persistence.go 52.00% 8 Missing and 4 partials ⚠️
messaging/adapters/persistence.go 60.00% 4 Missing and 2 partials ⚠️
t/helpers/db.go 57.14% 4 Missing and 2 partials ⚠️
... and 3 more
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #6998      +/-   ##
===========================================
+ Coverage    55.04%   59.49%   +4.45%     
===========================================
  Files          821      826       +5     
  Lines       116496   116617     +121     
===========================================
+ Hits         64120    69377    +5257     
+ Misses       45609    40153    -5456     
- Partials      6767     7087     +320     
Flag Coverage Δ
functional 35.29% <51.35%> (?)
unit 55.03% <73.46%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
appdatabase/database.go 73.23% <100.00%> (ø)
messaging/core.go 56.04% <100.00%> (+2.05%) ⬆️
messaging/layers/encryption/encryptor.go 72.25% <100.00%> (-1.12%) ⬇️
messaging/layers/encryption/migrations/migrate.go 100.00% <100.00%> (ø)
...aging/layers/encryption/multidevice/multidevice.go 86.44% <100.00%> (ø)
messaging/layers/encryption/protocol.go 70.48% <100.00%> (-0.68%) ⬇️
protocol/sqlite/prepare_migrations.go 100.00% <ø> (ø)
...ing/layers/encryption/sharedsecret/sharedsecret.go 55.55% <75.00%> (ø)
messaging/common/message_segmentation.go 67.48% <66.66%> (+1.47%) ⬆️
protocol/sqlite/db.go 42.85% <33.33%> (-7.15%) ⬇️
... and 10 more

... and 212 files with indirect coverage changes

- modularize messaging Persistence with wakuPersistence,
  encryptionPersistence, etc.
- separate migration table for encryption migrations
- inject encryption sqlite persistence through the interface

iterates: #6792
@osmaczko osmaczko force-pushed the refactor/messaging-persistence-interfaces branch from 459e2ec to 78d6e84 Compare October 16, 2025 13:54
@osmaczko
Copy link
Contributor Author

@igor-sirotin @dlipicar it is now ready for review 🙏

Copy link
Collaborator

@igor-sirotin igor-sirotin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't look the whole code yet, this is the first part of comments

if _, err := db.Exec(createIndex); err != nil {
return err
}
insertVersion := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (?, ?)`, name) // nolint:gosec
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about this to avoid // nolint?

Suggested change
insertVersion := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (?, ?)`, name) // nolint:gosec
insertVersion := fmt.Sprintf(`INSERT INTO %s (version, dirty)`, name) + "VALUES (?, ?)"

Comment on lines 7 to +10
type Persistence interface {
wakuPersistence
WakuStorage() WakuPersistence
SegmentationStorage() SegmentationPersistence
EncryptionStorage() EncryptionPersistence
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it not be simpler to compose instead?

Suggested change
type Persistence interface {
wakuPersistence
WakuStorage() WakuPersistence
SegmentationStorage() SegmentationPersistence
EncryptionStorage() EncryptionPersistence
type Persistence interface {
WakuPersistence
SegmentationPersistence
EncryptionPersistence

In case of status-go it is going to be a single implementation, right?
And in case they're separate, one could still compose all implementations into one

Comment on lines +59 to +62
migrations.AssetNames(),
func(name string) ([]byte, error) {
return migrations.Asset(name)
},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
migrations.AssetNames(),
func(name string) ([]byte, error) {
return migrations.Asset(name)
},
migrations.AssetNames(),
migrations.Asset,

)

const (
uuid = "7676fd0c"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this magic string coming from?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it's just a random value, but please add some comment clarifying it

Comment on lines +25 to +26
if version > 0 {
err := createMigrationTables(database, version)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic with version seem to be quite convoluted here and in LatestMigrationUpToVersion 🤔

But I see GetLastMigrationVersion returns migrationTableExists here, which we ignore:

version, _, err := sqlite.GetLastMigrationVersion(database, migrationsTable)

Maybe just check it and create the tables if they don't exist?

Comment on lines +79 to +89
func EncryptionPersistence(p types.Persistence) encryption.Persistence {
encryptionStorage := p.EncryptionStorage()
if encryptionStorage == nil {
return nil
}
internal, ok := encryptionStorage.(*internal.SQLiteEncryptionPersistence)
if !ok {
panic("custom EncryptionPersistence implementations are not supported yet")
}
return internal.SQLitePersistence
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part confuses me.

Sorry I had to write it down to consume, you probably know all of this 😄

Part 1. Interfaces

  1. In newCore we get a persistence pointer from the user as part of params:

    func newCore(waku wakutypes.Waku, params CoreParams, config *config) (*Core, error) {
    Persistence types.Persistence

  2. types.Persistence is expected to provide EncryptionPersistence:

    EncryptionStorage() EncryptionPersistence

  3. Yet the encryption package expects it's own internal Persistence interface:

    type Persistence interface {
    KeysStorage() dr.KeysStorage
    SessionStorage() dr.SessionStorage
    SharedSecretStorage() sharedsecret.Persistence
    MultideviceStorage() multidevice.Persistence

  4. That's why we use this EncryptionPersistence adapter here

    encryptor := encryption.New(
    adapters.EncryptionPersistence(params.Persistence),

Part 2. Adapter.

  1. The adapter works, because status-go uses the default sqlite encryption persistence:

    func (m *messagingPersistence) EncryptionStorage() messagingtypes.EncryptionPersistence {
    return messaging.NewEncryptionSQLitePersistence(m.db)
    }

    ... which, by the way is marked as "internal implementation" 🤔

    // Returns internal implementation of EncryptionPersistence.
    // Clients should not use this interface directly.
    func NewEncryptionSQLitePersistence(db *sql.DB) types.EncryptionPersistence {
    return &internal.SQLiteEncryptionPersistence{
    SQLitePersistence: encryption.NewSQLitePersistence(db),
    }
    }

  2. The implementation of SQLiteEncryptionPersistence is simply empty:

    // Implements the EncryptionPersistence interface using SQLite.
    type SQLiteEncryptionPersistence struct {
    *encryption.SQLitePersistence
    }
    var _ types.EncryptionPersistence = (*SQLiteEncryptionPersistence)(nil)
    func (e *SQLiteEncryptionPersistence) X3DHBundlesStorage() types.X3DHBundlesPersistence {
    return nil
    }
    func (e *SQLiteEncryptionPersistence) DRKeysStorage() types.DRKeysPersistence {
    return nil
    }
    func (e *SQLiteEncryptionPersistence) DRSessionStorage() types.DRSessionPersistence {
    return nil
    }
    func (e *SQLiteEncryptionPersistence) SharedSecretStorage() types.SharedSecretPersistence {
    return nil
    }
    func (e *SQLiteEncryptionPersistence) MultideviceStorage() types.MultidevicePersistence {
    return nil
    }
    func (e *SQLiteEncryptionPersistence) HashRatchetStorage() types.HashRatchetPersistence {
    return nil
    }

    ... yet it composes encryption.SQLitePersistence.
    And this is what the adapter relies on.

Part 3. Question

messagingtypes.EncryptionPersistence seem to be completely unused 🤔

Do we really need it? And the other persistence adapters?

Maybe we could just re-export the internal interfaces from messaging module?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants