the first release of client replica #1692

XiaochenCui · 2025-10-06T21:45:44Z

This is the first release of client-replica, check the RFC for details: https://github.com/XiaochenCui/puter/blob/fs/client-replica/doc/RFCS/20250821_client_replica_file_system.md

It's safe to merge and deploy it without update the config, no client will use client-replica in this case.

How to review

The implementation of fs-tree-manager process: src/fs_tree_manager
- entry: src/fs_tree_manager/server.go
- protobuf generated code: src/fs_tree_manager/go and src/fs_tree_manager/js
- merkle tree: src/fs_tree_manager/merkle
- core data structure and rpc: src/fs_tree_manager/proto/fs_tree_manager.proto
Client (puter-js): src/puter-js/src/modules/FileSystem/replica
puter backend (act as a proxy): src/backend/src/routers/filesystem_api/fs_tree_manager

Other code are hooks for integration with the existing codebase.

Config

init config file: cp src/fs_tree_manager/example-config.yaml src/fs_tree_manager/config.yaml
update config src/fs_tree_manager/config.yaml, set mysql args to the same as puter backend
update puter backend config, set the endpoint of fs_tree_manager:

{
  ...
  "services": {
    ...
    "client-replica": {
      "enabled": true,
      "fs_tree_manager_url": "<ip>:<port>"
    }
  }
}

Deploy

# build
docker build -t fs_tree_manager -f src/fs_tree_manager/Dockerfile src/fs_tree_manager

# run
docker compose -f src/fs_tree_manager/docker-compose.yml up

Whitelist

Whitelist is done by permission service:

        const svc_permission = Context.get('services').get('permission');
        const can_access = await svc_permission.check('endpoint:replica/fetch');
        if ( ! can_access ) {
            return socket.emit('replica/fetch/error', {
                success: false,
                error: { message: 'permission denied' },
            });
        }

XiaochenCui · 2025-10-10T19:13:57Z

@KernelDeimos @jelveh this PR is ready for review/merge

KernelDeimos

Biggest concerns are:

filesystem events should be used instead of adding hooks everywhere
database UPDATE added to move operation; I don't understand what it's fixing

Go code looks good to me, but I reviewed it very quickly since it's more outside the concern of the core backend.

KernelDeimos · 2025-10-15T20:07:57Z

src/backend/src/filesystem/hl_operations/hl_mkdir.js

(1) Why is this file changed? There do not appear to be any functional changes here - I think these changes were included by accident.

KernelDeimos · 2025-10-15T20:08:45Z

src/backend/src/filesystem/hl_operations/hl_mklink.js

See previous comment "(1)"

KernelDeimos · 2025-10-15T20:15:01Z

src/backend/src/modules/puterfs/lib/PuterFSProvider.js

+
+            const db = services.get('database').get(DB_WRITE, 'filesystem');
+            await db.write(
+                'UPDATE fsentries SET parent_uid = ? WHERE uuid = ?',


It's a bit concerning to see this change. If the parent_uid was not already being updated, that would mean move is currently broken. However, I just tried moving a file in prod - move is not broken.

A second area of concern is I was informed that the relica code won't do anything if it's not enabled, however this looks like a database operation that's happening every time a file is moved without any feature flag being checked.

it's not broken but the parent_uid is wrong without this patch, see here for details:

https://github.com/XiaochenCui/puter/blob/fs/client-replica/doc/RFCS/20250821_client_replica_file_system.md#fsentry-parent

the code below this patch doesn't set the parent_uid correctly:

puter/src/backend/src/modules/puterfs/lib/PuterFSProvider.js

Lines 223 to 232 in 847b3a0

const op_update = await svc_fsEntry.update(node.uid, {

...(

await node.get('parent_uid') !== await new_parent.get('uid')

? { parent_uid: await new_parent.get('uid') }

: {}

),

path: new_path,

name: new_name,

...(metadata ? { metadata } : {}),

});

I found parent_uid is correct after moving, I'll look into this later

KernelDeimos · 2025-10-15T20:29:02Z

src/backend/src/filesystem/hl_operations/hl_write.js

+                    console.error('user_id is missing');
+                    return;
+                }
+                await sendFSNew(user_id, response);


It isn't necessary to add these hooks into the filesystem implementation provider. This puts the responsibility on file system providers to inform the client-replica system of what's happening, requiring duplication of effort any time a new provider is added. This sort of thing is why we have filesystem events like fs.create.*, for example here. If the events don't have enough information or some events are missing (it looks like we don't have an event for delete yet) then the events should be updated to support that.

KernelDeimos · 2025-10-15T20:29:30Z

src/backend/src/modules/web/lib/eggspress.js

    }
-    if ( config.env === 'dev' && process.env.DEBUG ) {
-      console.log(`request url: ${req.url}, body: ${JSON.stringify(req.body)}`);
-    }


Why is this conditional debug log being removed? This only prints of the DEBUG environment variable is set - it allows easier debugging of endpoints when Puter is initiated with a terminal input like: DEBUG=1 npm start

oh, this log was added by me. I don't know others are using it - will revert this

Honestly I thought I added it, but let's keep it - it seems useful

KernelDeimos · 2025-10-15T20:33:26Z

src/backend/src/routers/filesystem_api/rename.js

+    })();
+
+    // ================== client-replica hook start ==================
+    // "rename" hook


If we can use events instead of this hook here that would be better.

src/fs_tree_manager/go/fs_tree_manager_grpc.pb.go

stash changes

stash changes stash changes stash changes stash changes stash changes stash changes stash changes

stash changes stash changes stash changes stash changes stash changes stash changes stash changes stash changes stash changes

stash changes stash changes stash changes stash changes stash changes stash changes stash changes stash changes stash changes stash changes

XiaochenCui force-pushed the fs/client-replica branch 4 times, most recently from dd647d5 to 97c78fc Compare October 10, 2025 19:08

XiaochenCui marked this pull request as ready for review October 10, 2025 19:13

XiaochenCui force-pushed the fs/client-replica branch from f8a8be2 to ede3cc2 Compare October 14, 2025 19:16

KernelDeimos requested changes Oct 15, 2025

View reviewed changes

XiaochenCui force-pushed the fs/client-replica branch 7 times, most recently from ac6de4a to 9f4d6d0 Compare October 21, 2025 00:08

XiaochenCui added 15 commits October 20, 2025 17:10

client-replica: the first release

38218b2

client-replica: hide the debug widget

899d111

client-replica: fix nil map

4ef3d8b

update logs

d9a7974

client-replica: docker config

900f86c

client-replica: add permission check (whitelist)

45559ca

client-replica: init playwright test (because of websocket)

edbafc4

client-replica: init multi-sesessions test

12a6e5f

client-replica: remove local config, update gitignore

9f4df5a

client-replica: update package.json

81c8bc3

client-replica: add e2e test to ci

3288eb5

ci: update work dir

54a3289

ci: add build action

8f430b7

ci: pass playwright test

6d738a9

test: fix lint

89a0396

stash changes

XiaochenCui added 10 commits October 20, 2025 17:10

test: seperate sessions

6b434e8

ci: update

40764f1

ci: init server config

094a327

ci: install deps

d242657

ci: add tmp action

ad49f36

stash changes stash changes stash changes stash changes stash changes stash changes stash changes

ci: update config

99ff426

ci: loosen the time limit

b0fc37a

ci: update config

a19c1ef

stash changes stash changes stash changes stash changes stash changes stash changes stash changes stash changes stash changes

ci: fix config error

c051480

test: update client-replica test

0647fed

stash changes stash changes stash changes stash changes stash changes stash changes stash changes stash changes stash changes stash changes

XiaochenCui force-pushed the fs/client-replica branch from 9f4d6d0 to 0647fed Compare October 21, 2025 00:10

XiaochenCui added 4 commits October 20, 2025 17:29

client-replica: update test

06a6728

ci: give fs-tree-manager more time

168a6fb

misc: test, ci, comments

967527f

client-replica: update docs

54bcbd5

	const op_update = await svc_fsEntry.update(node.uid, {
	...(
	await node.get('parent_uid') !== await new_parent.get('uid')
	? { parent_uid: await new_parent.get('uid') }
	: {}
	),
	path: new_path,
	name: new_name,
	...(metadata ? { metadata } : {}),
	});

the first release of client replica #1692

Are you sure you want to change the base?

the first release of client replica #1692

Uh oh!

Conversation

XiaochenCui commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XiaochenCui commented Oct 10, 2025

Uh oh!

KernelDeimos left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

XiaochenCui commented Oct 6, 2025 •

edited

Loading