Add qdrant native engine #271
Conversation
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a new qdrant_native engine that communicates with Qdrant using the REST API directly via httpx, as an alternative to the existing qdrant engine that uses the official Qdrant Python client. The implementation provides native HTTP-based access for benchmarking purposes.
- Adds a complete qdrant_native engine implementation with configuration, upload, search, and parsing capabilities
- Integrates the new engine into the client factory
- Provides configuration presets for single-node benchmarking scenarios
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| experiments/configurations/qdrant-native-single-node.json | Adds 8 benchmark configuration presets for the qdrant_native engine with varying quantization and search parameters |
| engine/clients/qdrant_native/init.py | Module initialization exposing configurator, uploader, and searcher classes |
| engine/clients/qdrant_native/config.py | Configuration constants for collection name and API key from environment variables |
| engine/clients/qdrant_native/configure.py | Collection configuration and creation logic using REST API |
| engine/clients/qdrant_native/parser.py | Filter condition parser converting internal format to Qdrant REST API JSON |
| engine/clients/qdrant_native/search.py | Search implementation using httpx for direct REST API queries |
| engine/clients/qdrant_native/upload.py | Upload implementation with batch processing and post-upload optimization handling |
| engine/clients/client_factory.py | Registers qdrant_native engine in factory mappings |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| patch_url = f"{cls.host}/collections/{QDRANT_COLLECTION_NAME}" | ||
| patch_payload = { | ||
| "optimizers_config": { | ||
| "max_optimization_threads": 100_000, |
There was a problem hiding this comment.
The magic number 100_000 for max_optimization_threads is arbitrary and unexplained. Consider defining this as a named constant (e.g., UNLIMITED_OPTIMIZATION_THREADS = 100_000) or using a more semantically meaningful value that indicates the intent (e.g., using a very large number or checking if there's a documented way to signal 'unlimited' to the API).
| ): | ||
| """Create a payload index for a specific field""" | ||
| url = f"{self.host}/collections/{QDRANT_COLLECTION_NAME}/index" | ||
|
|
There was a problem hiding this comment.
The field type 'uuid' is checked but not present in the INDEX_TYPE_MAPPING dictionary (lines 16-22). This will cause INDEX_TYPE_MAPPING.get(field_type, 'keyword') on line 106 to fall back to the default 'keyword', which may not be the intended behavior. Either add 'uuid' to INDEX_TYPE_MAPPING or remove it from this condition.
| if collection_info["status"] == "green": | ||
| break | ||
|
|
||
| return total |
There was a problem hiding this comment.
The wait_collection_green method returns total wait time but this return value is not used by the caller in post_upload (line 94). Either utilize this return value for logging/metrics or remove it to clarify the method's intent.
| return total | |
| # No return value needed; method waits until collection is green |
This facilitates benchmarking of those changes in the core, that require python-client update. Qdrant "native" engine calls Rest APIs directly.