You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[Doc] Describe new partition lifecycle behaviours (#2992)
doc of apache/doris#57060,
apache/doris#57013
and fix of #2970
## Versions
- [x] dev
- [x] 3.x
- [ ] 2.1
- [ ] 2.0
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [ ] Checked by AI
- [ ] Test Cases Built
Copy file name to clipboardExpand all lines: docs/table-design/data-partitioning/auto-partitioning.md
+29-17Lines changed: 29 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -63,7 +63,7 @@ When creating a table, use the following syntax to populate the `partitions_defi
63
63
1. AUTO RANGE PARTITION:
64
64
65
65
```sql
66
-
AUTO PARTITION BY RANGE(<partition_expr>)
66
+
[AUTO] PARTITION BY RANGE(<partition_expr>)
67
67
<origin_partitions_definition>
68
68
```
69
69
@@ -98,6 +98,8 @@ When creating a table, use the following syntax to populate the `partitions_defi
98
98
);
99
99
```
100
100
101
+
In AUTO RANGE PARTITION, the `AUTO` keyword can be omitted, and it still conveys the meaning of automatic partitioning.
102
+
101
103
2. AUTO LIST PARTITION
102
104
103
105
```sql
@@ -228,33 +230,43 @@ Doris supports both Auto and Dynamic Partition. In this case, both functions are
228
230
229
231
There is no conflict between the two syntaxes, just set the corresponding clauses/attributes at the same time. Please note that it is uncertain whether the partition in current period is created by Auto Partition or Dynamic Partition. Different creation methods will lead to different naming formats for the partitions.
230
232
231
-
### Best Practice
233
+
## Lifecycle Management
234
+
235
+
:::info
236
+
Doris supports the simultaneous use of automatic partitioning and dynamic partitioning for lifecycle management, but it is now not recommended.
237
+
:::
238
+
239
+
In the AUTO RANGE PARTITION table, the property `partition.retention_count` is supported, which accepts a positive integer value as a parameter (denoted as`N`), indicating that **only the top `N` historical partitions with the largest partition values** are retained among all historical partitions. All current and future partitions are retained. Specifically:
232
240
233
-
In scenarios where you need to set a limiton the partition lifecycle, you can **disable the creation of Dynamic Partition, leaving the creation of partitions to be completed by Auto Partition**, and complete the management of the partition lifecycle through the Dynamic Partition's function of dynamically reclaiming partitions:
241
+
- Since RANGE partitions are always non-overlapping, `partition A's value > partition B's value` is equivalent to `partition A's lower bound value > partition B's upper bound value` which is equivalent to `partition A's upper bound value > partition B's upper bound value`.
242
+
- Historical partitions refer to **partitions whose upper bound is <= current time**.
243
+
- Current and future partitions refer to **partitions whose lower bound is >= current time**.
244
+
245
+
For example:
234
246
235
247
```sql
236
-
create table auto_dynamic(
237
-
k0 datetime(6) NOT NULL
238
-
)
239
-
auto partition by range (date_trunc(k0, 'year'))
240
-
(
248
+
create table auto_recycle(
249
+
k0 datetime(6) not null
241
250
)
242
-
DISTRIBUTED BY HASH(`k0`) BUCKETS 2
251
+
AUTO PARTITION BY RANGE (date_trunc(k0, 'day')) ()
252
+
DISTRIBUTED BY HASH(`k0`) BUCKETS 1
243
253
properties(
244
-
"dynamic_partition.enable" = "true",
245
-
"dynamic_partition.prefix" = "p",
246
-
"dynamic_partition.start" = "-50",
247
-
"dynamic_partition.end" = "0", --- Dynamic Partition No Partition Creation
248
-
"dynamic_partition.time_unit" = "year",
249
-
"replication_num" = "1"
254
+
"partition.retention_count" = "3"
250
255
);
251
256
```
252
257
253
-
This way we have both the flexibility of Auto Partition and consistency in partition names.
258
+
This represents keeping only the top 3 partitions with the largest datevaluesin the history. Assuming the current date is `2025-10-21`, and inserting data for each day from`2025-10-16` to `2025-10-23`, after one recycling, the remaining partitions are as follows:
259
+
260
+
- p20251018000000
261
+
- p20251019000000
262
+
- p20251020000000 (The following partition and above: Only keep three historical partitions)
263
+
- p20251021000000 (The following partition and below: The current and future partitions are not affected)
264
+
- p20251022000000
265
+
- p20251023000000
254
266
255
267
## Conjunct with Auto Bucket
256
268
257
-
Only AUTO RANGE PARTITION can be used together with the [Auto Bucket](./data-bucketing.md#auto-setting-bucket-number) feature. When using this feature, Doris assumes that the data import is incremental in time order, and each import only involves one partition. In other words, this usage is only recommended for tables that are incrementally imported daily.
269
+
Only AUTO RANGE PARTITION can be used together with the [Auto Bucket](./data-bucketing.md#auto-setting-bucket-number) feature. When using this feature, Doris assumes that the data import is incremental in time order, and each import only involves one partition. In other words, this usage is only recommended for tables that are incrementally imported batch by batch.
258
270
259
271
:::warning Note!
260
272
If the data import method does not conform to the above pattern, and both auto partitioning and auto bucketing are used at the same time, there is a possibility that the number of buckets in the new partition is extremely unreasonable, which may greatly affect query performance.
Copy file name to clipboardExpand all lines: versioned_docs/version-3.x/table-design/data-partitioning/auto-partitioning.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -263,10 +263,10 @@ In some early versions prior to 2.1.7, this feature was not disabled but not rec
263
263
## Conjunct with Auto Bucket
264
264
265
265
:::note
266
-
This feature has been working normally since Doris 3.1.1
266
+
This feature has been working normally since Doris 3.1.2
267
267
:::
268
268
269
-
Only AUTO RANGE PARTITION can be used together with the [Auto Bucket](./data-bucketing.md#auto-setting-bucket-number) feature. When using this feature, Doris assumes that the data import is incremental in time order, and each import only involves one partition. In other words, this usage is only recommended for tables that are incrementally imported daily.
269
+
Only AUTO RANGE PARTITION can be used together with the [Auto Bucket](./data-bucketing.md#auto-setting-bucket-number) feature. When using this feature, Doris assumes that the data import is incremental in time order, and each import only involves one partition. In other words, this usage is only recommended for tables that are incrementally imported batch by batch.
270
270
271
271
:::warning Note!
272
272
If the data import method does not conform to the above pattern, and both auto partitioning and auto bucketing are used at the same time, there is a possibility that the number of buckets in the new partition is extremely unreasonable, which may greatly affect query performance.
0 commit comments