Newbie asks for help: Error with environment variables and then model loading

[markod0925]

leetdebug_290125.log
Hi,

I have tried to run the tool in Windows and maybe it is not supported. Anyway, after the following cmds in Powershell:

ollama pull llama3.2
ollama pull nomic-embed-text
ollama serve

pip install leettools

I have created .env.ollama file in my Home dir with the following inside:

EDS_DEFAULT_LLM_BASE_URL=http://localhost:11434/api
EDS_DEFAULT_INFERENCE_MODEL=llama3.2:latest
EDS_DEFAULT_EMBEDDING_MODEL=nomic-embed-text:latest
EDS_EMBEDDING_MODEL_DIMENSION=768
EDS_LLM_API_KEY=60db5******fda69

Then after launching the command

leet flow -e .env.ollama -t answer -q "How does GraphRAG work?" -k graphrag -l debug
I have the following error (see attached file for all debug):

Creating embedding client with base_url: http://localhost:11434/v1 and api_key: 60db5******fda69 (C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\api_caller\api_utils.py:267 get_openai_embedder_client_for_user)
[28/01/25 23:44:26.047] ERROR [9764-11712] Embedding operation failed: Error code: 404 - {'error': {'message': 'model "text-embedding-3-small" not found, try pulling it first', 'type': 'api_error', 'param': None, 'code': None}} (C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\str_embedder\_impl\dense_embedder_openai.py:98 embed)

It seems to ignore my EDS_DEFAULT_EMBEDDING_MODEL using the default value. But it loads correctly the EDS_LLM_API_KEY, so maybe it is an input check bug?

[pengfeng]

Thanks for reporting the issue! One possible reason is that the graphrag KB was created using the OpenAI embedding model "text-embedding-3-small", and once a KB is created, the embedder has to be fixed so that the previously embedded content can be read back correctly using the same embedder.

We just updated a new version that will print out a warning if the default embedder and the KB embedder is not compatible. You can update your program and see if that was the reason.

Also, you can use a new KB name, say "graphrag.ollama" so that a new KB is created using the correct local embedder.

[markod0925]

Ok, now it seems to connect to the model since the error has changed.
I have deleted the leettools directory to have a fresh start.
Then I have launched the following cmd

leet flow -e .env.ollama -t answer -q "How does GraphRAG work?" -k graphrag.ollama -l debug > debuglog.log
.end.ollama is

EDS_DEFAULT_LLM_BASE_URL=http://localhost:11434/api
EDS_DEFAULT_INFERENCE_MODEL=llama3.2:latest
EDS_DEFAULT_EMBEDDING_MODEL=nomic-embed-text:latest
EDS_EMBEDDING_MODEL_DIMENSION=768
EDS_LLM_API_KEY=60*********da69
LANG=en_US.UTF-8

This is the debug log with the errors:
debuglog.log

Now it seems more related to internal issues... Any suggestion?

[markod0925]

I have tried another example with almost the same error:

PS C:\Users\Marco> leet kb add-url -e .env.ollama -k llmbook -r "https://arxiv.org/pdf/2501.09223"
C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\huggingface_hub\file_download.py:140: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\Marco\.cache\huggingface\hub\models--ds4sd--docling-models. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
  warnings.warn(message)
[29/01/25 10:29:00.516] ERROR [16300-2408] Embedding operation failed: 'NoneType' object is not iterable (C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\str_embedder\_impl\dense_embedder_openai.py:100 embed)
[29/01/25 10:29:00.519] ERROR [16300-10184] Embedding operation failed: 'NoneType' object is not iterable (C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\str_embedder\_impl\dense_embedder_openai.py:100 embed)
[29/01/25 10:29:00.520] ERROR [16300-6968] Embedding operation failed: 'NoneType' object is not iterable (C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\str_embedder\_impl\dense_embedder_openai.py:100 embed)
[29/01/25 10:29:00.525] ERROR [16300-10456] Embedding operation failed: 'NoneType' object is not iterable (C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\str_embedder\_impl\dense_embedder_openai.py:100 embed)
[29/01/25 10:29:00.527] ERROR [16300-16236] Embedding operation failed: 'NoneType' object is not iterable (C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\str_embedder\_impl\dense_embedder_openai.py:100 embed)
[29/01/25 10:29:00.531] ERROR [16300-13040] Embedding operation failed: 'NoneType' object is not iterable (C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\str_embedder\_impl\dense_embedder_openai.py:100 embed)
[29/01/25 10:29:00.533] ERROR [16300-17388] Embedding operation failed: 'NoneType' object is not iterable (C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\str_embedder\_impl\dense_embedder_openai.py:100 embed)
[29/01/25 10:29:00.534] ERROR [16300-14488] Embedding operation failed: 'NoneType' object is not iterable (C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\str_embedder\_impl\dense_embedder_openai.py:100 embed)
[29/01/25 10:29:00.535] ERROR [16300-1508] Embedding operation failed: 'NoneType' object is not iterable (C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\str_embedder\_impl\dense_embedder_openai.py:100 embed)
[29/01/25 10:29:00.748] ERROR [16300-7572] Error embedding document: Traceback (most recent call last):
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\pipeline\embed\segment_embedder_simple.py", line 63, in embed_segment_list
    rtn_code = self._embed(segments, display_logger)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\pipeline\embed\segment_embedder_simple.py", line 40, in _embed
    op_success = self.dense_vectorstore.save_segments(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\core\repo\_impl\duckdb\vector_store_dense_duckdb.py", line 205, in save_segments
    return self._upsert_embeddings_into_duckdb(org, kb, segments, dense_embedder)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\core\repo\_impl\duckdb\vector_store_dense_duckdb.py", line 179, in _upsert_embeddings_into_duckdb
    rtn_list = list(executor.map(partial_get_embedding, embed_batches))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures\_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures\_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures\_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures\thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\core\repo\_impl\duckdb\vector_store_dense_duckdb.py", line 60, in _batch_get_embedding
    embeddings = self._get_embedding(dense_embedder, texts)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\core\repo\_impl\duckdb\vector_store_dense_duckdb.py", line 121, in _get_embedding
    embed_result: DenseEmbeddings = dense_embedder.embed(embed_request)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\str_embedder\_impl\dense_embedder_openai.py", line 101, in embed
    raise e
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\str_embedder\_impl\dense_embedder_openai.py", line 93, in embed
    response = self.openai.embeddings.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\openai\resources\embeddings.py", line 124, in create
    return self._post(
           ^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\openai\_base_client.py", line 1283, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\openai\_base_client.py", line 960, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\openai\_base_client.py", line 1066, in _request
    return self._process_response(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\openai\_base_client.py", line 1165, in _process_response
    return api_response.parse()
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\openai\_response.py", line 319, in parse
    parsed = self._options.post_parser(parsed)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\openai\resources\embeddings.py", line 112, in parser
    for embedding in obj.data:
                     ^^^^^^^^
TypeError: 'NoneType' object is not iterable
 (C:\Users\Marco\AppData\Local\Programs\Python\Python312\Lib\site-packages\leettools\eds\pipeline\embed\segment_embedder_simple.py:67 embed_segment_list)
org:            org-default
kb:             llmbook
docource_id:    896d2089-f45e-4cd1-bc92-3e8b524daf27
docsink_id:     ee5bd644-a255-4263-a684-b0b6114a3654
document_uuid:  93aff710-c495-48b2-9463-886566930f40
URI:            https://arxiv.org/pdf/2501.09223

Maybe the issue is that I am trying to run everything without Administration privilege?

[pengfeng]

Interesting. I think the other user also reported this problem from a Windows machine, but we have also tested this on a Windows machine before and it works fine. I will tested a bit more on the Windows env and see if we can reproduce the error. Thanks for reporting!

[markod0925]

Maybe it is related to the fact that I have installed all tools (including Ollama and Python) without the admin user but with a local one? Il mer 29 gen 2025, 17:11 Feng ***@***.***> ha scritto:

…

Interesting. I think the other user also reported this problem from a Windows machine, but we have also tested this on a Windows machine before and it works fine. I will tested a bit more on the Windows env and see if we can reproduce the error. Thanks for reporting! — Reply to this email directly, view it on GitHub <#84 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BEQDIXSHXBVRYCP3IL5KTC32ND4UXAVCNFSM6AAAAABWBSPWGOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCOJZG42DQNI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[pengfeng]

I just did a fresh install on a Windows machine with Ollama using the default setting and it worked fine as shown above.

Maybe you can check the Ollama log to see if there is anything wrong with the Ollama service side?

[markod0925]

Good suggestion, but I cannot find anything relevant to my knowledge in that... I mean, there are few "warnings" like

llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
...
llama_new_context_with_model: n_ctx_pre_seq (8192) > n_ctx_train (2048) -- possible training context overflow

But after that the server seems to respond

time=2025-01-29T22:45:40.620+01:00 level=INFO source=server.go:594 msg="llama runner started in 1.27 seconds"
[GIN] 2025/01/29 - 22:45:40 | 200 |    2.1418861s |       127.0.0.1 | POST     "/api/embeddings"
[GIN] 2025/01/29 - 22:45:40 | 200 |    2.1514901s |       127.0.0.1 | POST     "/api/embeddings"
[GIN] 2025/01/29 - 22:45:40 | 200 |    2.1628667s |       127.0.0.1 | POST     "/api/embeddings"
[GIN] 2025/01/29 - 22:45:40 | 200 |    2.1411932s |       127.0.0.1 | POST     "/api/embeddings"
[GIN] 2025/01/29 - 22:45:40 | 200 |    2.1520612s |       127.0.0.1 | POST     "/api/embeddings"
[GIN] 2025/01/29 - 22:45:40 | 200 |    2.1639904s |       127.0.0.1 | POST     "/api/embeddings"
[GIN] 2025/01/29 - 22:45:40 | 200 |    2.1639904s |       127.0.0.1 | POST     "/api/embeddings"
[GIN] 2025/01/29 - 22:45:40 | 200 |    2.1347409s |       127.0.0.1 | POST     "/api/embeddings"
[GIN] 2025/01/29 - 22:45:40 | 200 |    2.1645187s |       127.0.0.1 | POST     "/api/embeddings"

There is something that I can do to perform a deep debug? I am no new to coding, but Python is not my ground... Can I run the code using VSCode in debugger style?

app.log
server.log

[markod0925]

I have tried to rerun everything in VSCode with debugger and I get this warning:
Warning: KB llmbook2 uses a dense embedder that is not compatible with the default specfied in .env.ollama:
KB embedder: DenseEmbedderOpenAI[model: nomic-embed-text:latest][dimension: 768]
Default : DenseEmbedderOpenAI[model: nomic-embed-text:latest][dimension: 1536].

It is stange, since I run two times the same command without changing the KB name. It seems that the first time it ignores the env parameter set in .env.ollama (EDS_EMBEDDING_MODEL_DIMENSION=768)

[pengfeng]

KB embedder: DenseEmbedderOpenAI[model: nomic-embed-text:latest][dimension: 768]
Default : DenseEmbedderOpenAI[model: nomic-embed-text:latest][dimension: 1536].

This warning message is saying that the KB was created with the correct dimension (768), but the second run has a default of 1536, which is kind of strange. This should be fine though since as long as the KB's embedder is correct and available through the settings, the embedding should work.

Also, from the Ollama log, it seems that the embedding request did get through, but the request return value is not handled correctly by the OpenAI client library. I think I will add a compatibility check tool to make sure the Ollama models play nicely with the OpenAI lib. (We are still evaluating other approaches such as OpenRouter, but will limit to use OpenAI client SDK for now.)

[markod0925]

And finally it works!

The issue was defining
EDS_DEFAULT_LLM_BASE_URL=http://localhost:11434/api instead of
EDS_DEFAULT_LLM_BASE_URL=http://localhost:11434/v1
Summarizing, first issue was due to an unintentional reuse of the same KB where the embedder used was different from the new one. Warning is necessary (but already implemented) and maybe ask the user to automatically create a new file _2.
The second was a mistake in the definition of the url, here maybe a sanity checker of the connection can be useful.

Thank you again for you patience!

[markod0925]

And finally it works!

The issue was defining
EDS_DEFAULT_LLM_BASE_URL=http://localhost:11434/api instead of
EDS_DEFAULT_LLM_BASE_URL=http://localhost:11434/v1
Summarizing, first issue was due to an unintentional reuse of the same KB where the embedder used was different from the new one. Warning is necessary (but already implemented) and maybe ask the user to automatically create a new file _2.
The second was a mistake in the definition of the url, here maybe a sanity checker of the connection can be useful.

Thank you again for you patience!

[pengfeng]

Ah, I see. The /api vs /v1 problem was subtle, it looked so natural lol. Yeah, the error message could be improved, similar to the KB mismatch issue. Thanks for reporting the issue!