Skip to content

Document ART index limitation with concurrent transactions and deletes #5724

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 25, 2025
Merged

Document ART index limitation with concurrent transactions and deletes #5724

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
88a675a
Document ART index limitation with concurrent transactions and deletes
rynoV Aug 27, 2025
bd2b25f
Update docs/1.2/sql/indexes.md
szarnyasg Sep 15, 2025
4388095
Enhance documentation on constraint checking after deletes
rynoV Sep 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion docs/1.2/sql/indexes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,4 +108,25 @@ Violates foreign key constraint because key "id: 1" is still referenced by a for
```

The reason for this is because DuckDB does not yet support “looking ahead”.
During the `INSERT`, it is unaware it will reinsert the foreign key value as part of the `UPDATE` rewrite.
During the `INSERT`, it is unaware it will reinsert the foreign key value as part of the `UPDATE` rewrite.

### Constraint Checking After Delete With Concurrent Transactions

When a delete is committed on a table with an ART index, data can only be removed from the index when no further transactions exist that refer to the deleted entry. This means if you perform a delete transaction, a subsequent transaction which inserts a record with the same key as the deleted record can fail with a constraint error if there is a concurrent transaction referencing the deleted record. Pseudocode to demonstrate:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

committed on a table with an index (happens for all our indexes, also those in extensions). There, we just don't notice, as we don't use them for constraint checking.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

referencing the deleted record. Note that constraint violations are only relevant for PK, FK, and UNIQUE indexes. Maybe expand on this? Not sure.


```
// Assume "someTable" is a table with an ART index preventing duplicates
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with an index enforcing a constraint violation - I think this can also happen for FKs, no (not only duplicate checks)? Oh, can we actually enter an inconsistent state here? I.e., I delete from the PK table, but keep the value alive. Then, I insert into the FK table, and now the delete goes through. ??

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just checked this using the .NET SDK and it does seem to get into an inconsistent state. This is the pseudocode:

// Setup: Create a primary table with UUID primary key and a secondary table with foreign key reference
primaryId = generateNewGUID()
conn = duckdbConnectInMemory()

// Create tables and insert initial record in primary table
duckdb(conn, "CREATE TABLE primary_table (id UUID PRIMARY KEY)")
duckdb(conn, "CREATE TABLE secondary_table (primary_id UUID, FOREIGN KEY (primary_id) REFERENCES primary_table(id))")
duckdbInsert(conn, "primary_table", {id: primaryId})

// Start transaction tx1 which will read from primary_table
tx1 = duckdbTxStart(conn)
readRecord = duckdb(tx1, "SELECT id FROM primary_table LIMIT 1")
// Note: tx1 remains open, holding locks/resources

// Outside of tx1, delete the record from primary_table
duckdbDelete(conn, "primary_table", {id: primaryId})

// Try to insert into secondary_table with foreign key reference to the now-deleted primary record
// This succeeds because tx1 is still open and the constraint isn't fully enforced yet
duckdbInsert(conn, "secondary_table", {primary_id: primaryId})

// Commit tx1, releasing any locks/resources
duckdbTxCommit(tx1)

// Verify the primary record is indeed deleted
count = duckdb(conn, "SELECT COUNT(*) FROM primary_table WHERE id = $primaryId",{primaryId: primaryId})
assert(count == 0, "Record should be deleted")

// Verify the secondary record with the foreign key reference exists
// This demonstrates the inconsistent state
count = duckdb(conn, "SELECT COUNT(*) FROM secondary_table WHERE primary_id = $primaryId",{primaryId: primaryId})
assert(count == 1, "Foreign key reference should exist")

If tx1 is committed before the secondary_table insert, the insert fails as expected, but if not the assertions succeed

tx1 = duckdbTxStart()
someRecord = duckdb(tx1, "select * from someTable using sample 1 rows")

tx2 = duckdbTxStart()
duckdbDelete(tx2, someRecord)
duckdbTxCommit(tx2)

// At this point someRecord is deleted, but the ART index is not updated, so the following would fail with a constraint error:
// tx3 = duckdbTxStart()
// duckdbInsert(tx3, someRecord)
// duckdbTxCommit(tx3)

duckdbTxCommit(tx1) // Following this, the above insert would succeed because the ART index was allowed to update
```