Skip to content

Restore a backup in-place, for restoring full to update a db with another full db without downtime on data #1217

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 29 commits into
base: master
Choose a base branch
from
Open

Restore a backup in-place, for restoring full to update a db with another full db without downtime on data #1217

wants to merge 29 commits into from

Conversation

minguyen9988
Copy link
Contributor

@minguyen9988 minguyen9988 commented Sep 1, 2025 •
edited by Slach
Loading

should fix #780

@minguyen9988 minguyen9988 changed the title restore a backup inplace, for restoring full to update a db with another full db without downtime on data WIP restore a backup inplace, for restoring full to update a db with another full db without downtime on data Sep 1, 2025
@nguyenminhduc9988
add metadata in backup
c84c9af 
fix create

remove unwanted
@minguyen9988 minguyen9988 force-pushed the minguyen/restore_remote branch from 936245e to c84c9af Compare September 1, 2025 03:41
@coveralls
Copy link

Pull Request Test Coverage Report for Build 17374127302

Details

  • 17 of 633 (2.69%) changed or added relevant lines in 6 files are covered.
  • 272 unchanged lines in 10 files lost coverage.
  • Overall coverage decreased (-4.3%) to 64.381%
Changes Missing Coverage Covered Lines Changed/Added Lines %
pkg/backup/restore_remote.go 4 6 66.67%
pkg/backup/backuper.go 0 3 0.0%
cmd/clickhouse-backup/main.go 5 9 55.56%
pkg/backup/restore.go 2 609 0.33%
Files with Coverage Reduction New Missed Lines %
pkg/backup/restore.go 1 52.04%
cmd/clickhouse-backup/main.go 2 75.36%
pkg/storage/object_disk/object_disk.go 5 66.48%
pkg/config/config.go 8 71.98%
pkg/status/status.go 9 67.43%
pkg/backup/backuper.go 17 71.64%
pkg/server/server.go 18 59.1%
pkg/storage/general.go 24 60.66%
pkg/storage/s3.go 25 47.49%
pkg/storage/gcs.go 163 0.0%
Totals Coverage Status
Change from base Build 17318832364: -4.3%
Covered Lines: 10254
Relevant Lines: 15927
💛 - Coveralls

@Slach Slach added this to the 2.7.0 milestone Sep 1, 2025
@nguyenminhduc9988
use the option without data flag
7ce426f
@nguyenminhduc9988
Merge upstream changes with enhanced in-place restore logging
5a4292c 
- Resolved merge conflicts in cmd/clickhouse-backup/main.go, pkg/backup/restore.go, pkg/backup/restore_remote.go, and pkg/server/server.go
- Integrated upstream named collections support with existing enhanced logging functionality
- Preserved in-place restore routing logic with additional named collections condition
- Maintained function signature compatibility by adding restoreNamedCollections and namedCollectionsOnly parameters
- Enhanced logging for table drops, part removals, and part downloads remains intact
@nguyenminhduc9988
Final merge conflict resolution and dependency sync
4043401 
- Resolved remaining merge conflict in cmd/clickhouse-backup/main.go restore_remote command
- Merged upstream named collections support with enhanced in-place restore functionality
- Updated go.mod and go.sum with upstream dependency changes
- All functionality preserved: enhanced logging, routing logic, and CLI parameter handling
@nguyenminhduc9988
remove my name
6a1e6fb
@nguyenminhduc9988
remove my name
71b9ef1
@minguyen9988 minguyen9988 changed the title WIP restore a backup inplace, for restoring full to update a db with another full db without downtime on data restore a backup inplace, for restoring full to update a db with another full db without downtime on data Sep 2, 2025
@minguyen9988 minguyen9988 changed the title restore a backup inplace, for restoring full to update a db with another full db without downtime on data Restore a backup in-place, for restoring full to update a db with another full db without downtime on data Sep 2, 2025
nguyenminhduc9988 and others added 13 commits September 2, 2025 02:29
@nguyenminhduc9988
fix for 1.25
33769c2
@nguyenminhduc9988
change test
4c26024
@Slach @nguyenminhduc9988
skip TestFIPS for 19.17 after switch to go 1.25, final fixes for Alti…
147b4f3 
…nity#961

Signed-off-by: Slach <[email protected]>
@Slach @nguyenminhduc9988
skip TestNamedCollections for 23.3, final fixes for Altinity#961
b49675f 
Signed-off-by: Slach <[email protected]>
@Slach @nguyenminhduc9988
fix TestNamedCollections for 25.8, final fixes for Altinity#961
c4daec2 
Signed-off-by: Slach <[email protected]>
@Slach @aider-chat-bot @nguyenminhduc9988
refactor: Consolidate named collection decryption and support encrypt…
9ee9f67 
…ed JSONL restore

Co-authored-by: aider (gemini/gemini-2.5-pro) <[email protected]>
@Slach @aider-chat-bot @nguyenminhduc9988
refactor: Restore named collections from jsonl by executing decrypted…
103a27c 
… SQL

Co-authored-by: aider (gemini/gemini-2.5-pro) <[email protected]>
@Slach @aider-chat-bot @nguyenminhduc9988
feat: Add ON CLUSTER to named collection restore queries
a787c74 
Co-authored-by: aider (gemini/gemini-2.5-pro) <[email protected]>
@Slach @nguyenminhduc9988
fix: Improve named collection restore and decryption error handling
e9fd9a5
@Slach @aider-chat-bot @nguyenminhduc9988
refactor: Switch DumpNode.Value to []byte and update related logic
1f2bd3e 
Co-authored-by: aider (gemini/gemini-2.5-pro) <[email protected]>
@Slach @nguyenminhduc9988
try to debug fingerprint for AES decryption, final fixes for Altinity…
97509ea 
…#961

Signed-off-by: Slach <[email protected]>
@Slach @aider-chat-bot @nguyenminhduc9988
refactor: Implement ClickHouse-compatible SipHash and support multipl…
b49c034 
…e header versions in restore

Co-authored-by: aider (gemini/gemini-2.5-pro) <[email protected]>
@Slach @nguyenminhduc9988
final fixes AES decryption, fix Altinity#961
3270aa3 
Signed-off-by: Slach <[email protected]>
@minguyen9988 minguyen9988 force-pushed the minguyen/restore_remote branch from b8f71ee to 4c5132c Compare September 5, 2025 08:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
2.7.0
Development

Successfully merging this pull request may close these issues.

Add --streaming option to create_remote and restore_remote
4 participants
@minguyen9988 @coveralls @Slach @nguyenminhduc9988