feat(ccr): add cluster-level task creation support #634
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
LemonCL
force-pushed
the
ccr_feature
branch
3 times, most recently
from
d5058c2 to
98b41cc
Compare
This commit introduces a comprehensive global synchronization feature that enables synchronization of global objects between Doris clusters. Features: - Support for global snapshot creation with configurable options - Flexible sync options: all, backup_privilege, backup_catalog, backup_workload_group - Automatic backup completion monitoring with timeout handling - Global snapshot information retrieval and processing - SQL statement execution on destination cluster - Comprehensive error handling and logging throughout the process API Endpoint: - POST /sync_global - Handles global synchronization requests Request Structure: - name: Unique identifier for the sync operation - src: Source cluster specification - dest: Destination cluster specification - Sync options (at least one required if 'all' is not set): - all: Sync all global objects - backup_privilege: Sync user privileges - backup_catalog: Sync catalog information - backup_workload_group: Sync workload group settings Workflow: 1. Create global snapshot on source cluster 2. Wait for backup completion with polling mechanism 3. Retrieve global snapshot details via RPC 4. Convert snapshot data to SQL statements 5. Execute SQL statements on destination cluster Additional improvements: - Internationalize all Chinese logs and comments to English - Refactor validation logic for flexible option combinations - Decompose syncGlobal function into smaller, focused functions This feature enables seamless migration and synchronization of global configurations between Doris clusters, supporting various deployment scenarios including disaster recovery and cluster migration.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add cluster-level CCR job creation.
By setting cluster_sync: true in the JSON body, a single POST to /create_ccr will automatically synchronize every database and table from the source cluster to the destination cluster
Usage example
curl -X POST -H "Content-Type: application/json" -d '{
"name": "ccr_full",
"cluster_sync": true,
"src": {
"host": "10.0.2.132",
"port": "9030",
"thrift_port": "9020",
"user": "root",
"password": "XX",
"database": "",
"table": ""
},
"dest": {
"host": "10.0.4.252",
"port": "9030",
"thrift_port": "9020",
"user": "root",
"password": "XX",
"database": "",
"table": ""
}
}' http://127.0.0.1:9190/create_ccr