Skip to content

feat(fs): implement slice upload #1141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 32 commits into
base: main
Choose a base branch
from
Draft

feat(fs): implement slice upload #1141

wants to merge 32 commits into from

Conversation

jenken827
Copy link
Contributor

@jenken827 jenken827 commented Aug 24, 2025 •
edited by xrgzs
Loading

修改点如下：

  1. 前端上传页面增加分片上传勾选框
  2. 如果勾选了分片上传，会调用新增的分片上传接口
  3. 目前实现了123open和baidu_netdisk的分片上传功能
    分片上传流程如下：
  4. 客户端调用FsUpHash获取上传所需的信息(目前主要是hash信息)
  5. 根据获取到的hash信息，客户端调用FsPreup上传必要的参数，获取分片大小，及需上传的分片列表，后端会把相关信息记录入库
  6. 客户端根据分片列表进行分片上传，如果分片是第一个，且需要sliceHash，那么需要把所有分片的hash带上
  7. 如果网盘本身支持分片上传，那么会直接走网盘的分片上传流程，本地不会缓存文件内容，如果网盘不支持分片，那么会先将分片缓存到本地
  8. 如果中途出现问题，可以重新进行分片上传流程，后端根据记录的信息进行恢复，不用重复上传已经上传的分片
  9. 分片上传完毕后，会根据用户是否勾选了AsTask来处理是直接上传还是以任务的形式处理（不论哪种形式，都是处理的本地缓存的文件，客户端到openlist服务的上传流程已经结束）。

Frontend: OpenListTeam/OpenList-Frontend#187

@jenken827 jenken827 mentioned this pull request Aug 24, 2025
Draft
@jenken827 jenken827 changed the title feat:支持分片上传 feat(fs):支持分片上传 Aug 24, 2025
@xrgzs xrgzs requested a review from j2rong4cn August 24, 2025 11:47
@xrgzs xrgzs changed the title feat(fs):支持分片上传 feat(fs): implement slice upload Aug 24, 2025
ILoveScratch2
ILoveScratch2 reviewed Aug 25, 2025
View reviewed changes
@jenken827
Copy link
Contributor Author

已调整

  1. 支持了流式复制
  2. 创建临时文件时，会加锁

@j2rong4cn j2rong4cn marked this pull request as draft August 28, 2025 04:04
@Suyunmeng Suyunmeng marked this pull request as ready for review September 5, 2025 05:18
Suyunmeng
Suyunmeng previously approved these changes Sep 5, 2025
View reviewed changes
@j2rong4cn j2rong4cn marked this pull request as draft September 5, 2025 10:07
j2rong4cn
j2rong4cn requested changes Sep 5, 2025
View reviewed changes
Copy link
Member

@j2rong4cn j2rong4cn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

上传完成后没有从数据库中移除tables.SliceUpload，导致不支持分片上传接口的驱动无法重复上传同一个文件。上传任务的id不应使用自增id吧？

@Suyunmeng Suyunmeng requested a review from j2rong4cn September 5, 2025 13:56
Suyunmeng and others added 3 commits September 5, 2025 23:54
@j2rong4cn
refactor(upload): Optimize SliceUploadManager with singleflight for s…
dc501f0 
…ession management and improve session loading logic
@j2rong4cn
Copy link
Member

还需要优化一下日志

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ILoveScratch2 ILoveScratch2 ILoveScratch2 left review comments

@Suyunmeng Suyunmeng Suyunmeng left review comments

@j2rong4cn j2rong4cn Awaiting requested review from j2rong4cn

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
enhancement go
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jenken827 @j2rong4cn @Suyunmeng @ILoveScratch2 @xrgzs