-
Notifications
You must be signed in to change notification settings - Fork 38
feat: Add cluster-level CCR sync and dynamic monitor interval configuration #637
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
LemonCL
wants to merge
1
commit into
selectdb:dev
Choose a base branch
from
LemonCL:feature/cluster-ccr-sync-dev
base: dev
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,140 @@ | ||
# Cluster CCR Feature Changes | ||
|
||
## 概述 | ||
本次更改为CCR Syncer添加了集群级别同步功能,允许用户一次性同步整个集群的所有数据库,并提供了动态配置监控间隔的功能。 | ||
|
||
## 主要更改 | ||
|
||
### 1. 新增ClusterSync参数 | ||
|
||
#### 文件:`pkg/service/http_service.go` | ||
|
||
**新增字段:** | ||
```go | ||
type CreateCcrRequest struct { | ||
// ... 其他字段 | ||
// Whether it's cluster-level sync, if true, will get all databases from source cluster and create sync task for each database | ||
ClusterSync bool `json:"cluster_sync"` | ||
} | ||
``` | ||
|
||
**新增函数:** | ||
- `createClusterCcr()` - 创建集群级别的CCR同步任务 | ||
- `startDatabaseMonitor()` - 启动数据库监控守护进程 | ||
- `monitorDatabaseChanges()` - 检测数据库变化并处理 | ||
- `handleNewDatabases()` - 处理新增数据库 | ||
- `handleDeletedDatabases()` - 处理删除的数据库 | ||
- `getDatabaseList()` - 获取数据库列表 | ||
|
||
**功能特性:** | ||
- 自动获取源集群所有数据库 | ||
- 为每个数据库创建独立的同步任务 | ||
- 支持动态监控新增/删除的数据库 | ||
- 自动创建/删除对应的同步任务 | ||
- 支持失败重试机制 | ||
|
||
### 2. 动态监控间隔配置 | ||
|
||
#### 新增全局变量: | ||
```go | ||
var ( | ||
databaseMonitorInterval time.Duration = 2 * time.Minute | ||
intervalMutex sync.RWMutex | ||
intervalUpdateChan = make(chan time.Duration, 1) | ||
) | ||
``` | ||
|
||
#### 新增HTTP Handler: | ||
|
||
**`/update_monitor_interval` - 更新监控间隔** | ||
- 请求方法:POST | ||
- 请求格式: | ||
```json | ||
{ | ||
"interval_seconds": 300 | ||
} | ||
``` | ||
- 功能:动态更新数据库监控的检查间隔 | ||
|
||
**`/get_monitor_interval` - 获取当前监控间隔** | ||
- 请求方法:GET | ||
- 响应格式: | ||
```json | ||
{ | ||
"success": true, | ||
"interval_seconds": 120 | ||
} | ||
``` | ||
- 功能:获取当前的监控间隔设置 | ||
|
||
### 3. 性能优化 | ||
|
||
**锁优化:** | ||
- 使用channel通信替代频繁的锁操作 | ||
- 减少了`startDatabaseMonitor`函数中的锁竞争 | ||
- 提高了监控性能 | ||
|
||
**实现细节:** | ||
- 使用`select`语句监听ticker和interval更新 | ||
- 非阻塞的channel通信 | ||
- 只在间隔真正改变时才重置ticker | ||
|
||
## API使用示例 | ||
|
||
### 创建集群级同步任务 | ||
```bash | ||
curl -X POST http://localhost:9190/create_ccr \ | ||
-H "Content-Type: application/json" \ | ||
-d '{ | ||
"name": "cluster_sync_job", | ||
"src": { | ||
"host": "source-cluster", | ||
"port": 9030, | ||
"user": "root", | ||
"password": "password" | ||
}, | ||
"dest": { | ||
"host": "dest-cluster", | ||
"port": 9030, | ||
"user": "root", | ||
"password": "password" | ||
}, | ||
"cluster_sync": true | ||
}' | ||
``` | ||
|
||
### 更新监控间隔 | ||
```bash | ||
curl -X POST http://localhost:9190/update_monitor_interval \ | ||
-H "Content-Type: application/json" \ | ||
-d '{"interval_seconds": 300}' | ||
``` | ||
|
||
### 获取监控间隔 | ||
```bash | ||
curl -X GET http://localhost:9190/get_monitor_interval | ||
``` | ||
|
||
## 技术特点 | ||
|
||
1. **线程安全**:使用mutex保护全局变量 | ||
2. **高性能**:优化锁使用,减少竞争 | ||
3. **动态配置**:支持运行时修改监控间隔 | ||
4. **容错性**:支持失败重试和错误处理 | ||
5. **可观测性**:详细的日志记录 | ||
6. **向后兼容**:不影响现有的单数据库同步功能 | ||
|
||
## 测试建议 | ||
|
||
1. 测试集群级同步功能 | ||
2. 测试动态监控间隔更新 | ||
3. 测试新增/删除数据库的自动处理 | ||
4. 测试并发场景下的性能 | ||
5. 测试错误恢复机制 | ||
|
||
## 注意事项 | ||
|
||
1. 集群级同步会为每个数据库创建独立的同步任务 | ||
2. 任务命名格式:`{原任务名}_{数据库名}` | ||
3. 监控间隔最小值应大于0 | ||
4. 建议在生产环境中谨慎设置监控间隔 |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove this file