-
Notifications
You must be signed in to change notification settings - Fork 4.7k
Open
Labels
Description
🐛 Describe the bug
I noticed that support for new OpenAI embedding models such as text-embedding-3-small and text-embedding-3-large are added. They have an ability to reduce the output dimensions from default ones i.e 1536. I want to reduce the embedding dimensions to 1024 but seems like vector_dimension parameter is getting setup instead of dimensions parameter in OpenAIEmbedder class. I have below Yaml file:
vectordb:
provider: elasticsearch
config:
collection_name: 'collection_name'
es_url: ['es_host']
http_auth:
- id
- password
verify_certs: false
embedder:
provider: openai
config:
model: 'text-embedding-3-small'
vector_dimension: 1024
Above is giving me error when adding data in elastic: 'error': {'type': 'mapper_parsing_exception', 'reason': 'failed to parse', 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'Field [embeddings] of type [dense_vector] of doc has exceeded the number of dimensions [1024] defined in mapping'}}