Skip to content

Reduce Dimensions of new OpenAI embedding models is not working #1229

@ad3sai

Description

@ad3sai

🐛 Describe the bug

I noticed that support for new OpenAI embedding models such as text-embedding-3-small and text-embedding-3-large are added. They have an ability to reduce the output dimensions from default ones i.e 1536. I want to reduce the embedding dimensions to 1024 but seems like vector_dimension parameter is getting setup instead of dimensions parameter in OpenAIEmbedder class. I have below Yaml file:

vectordb:
  provider: elasticsearch
  config:
    collection_name: 'collection_name'
    es_url: ['es_host']
    http_auth:
      - id
      - password
    verify_certs: false

embedder:
  provider: openai
  config:
    model: 'text-embedding-3-small'
    vector_dimension: 1024

Above is giving me error when adding data in elastic: 'error': {'type': 'mapper_parsing_exception', 'reason': 'failed to parse', 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'Field [embeddings] of type [dense_vector] of doc has exceeded the number of dimensions [1024] defined in mapping'}}

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingeasyEasy difficulty

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions