Recover from messed up function call

[caramboleyo]

All models I try mess up function calls frequently. Right now the app crashes for example when the args parsing is wrong or when the llm tries to send null or an empty response. for the model to know it did something wrong a meaningful error response might be nice. Right now the whole app crashes on this.
if i catch the whole prompt i could fake the malformed request into the chat history and rerun, but for that i need the exact response from the llm that caused the crash and there is no way to get that as all on-handlers do not trigger before the crash.
also if i fake it into the chat session how to make the llm rerun it, maybe put the failed attempt in as a system message?

I was wondering how others handle this problem. node-llama-cpp does not seem to offer a way to properly handle (or recover from) errors, it just crashes on every little thing so i am guessing i am missing something crucial here in the handling?

I would have a try/catch block around the whole function calling code in 1357 of LlamaChat.js and call an onFunctionError callback containing all the tokens that were tried to be processed as a function call and the error on catch. But i guess thats to deep in the code base because we would have to return the whole big return object from line 1683?

[giladgd]

@caramboleyo It's impossible for models to call functions with parameters that don't 100% conform to the params schema that you specify, the model cannot technically generate tokens that don't conform to it as it's an integral part of the inference engine (via grammar).
There are also multiple validation layers that would throw an explicit error about schema violation if that somehow happens (please let me know if that's the case), and it's also covered via a testsuit.
I've worked very hard to ensure the params schema you specify is also reflected in the types of the handler arguments (via TypeScript type inference), so there couldn't be any confusion as to what parameters are actually available and how they could be used.

If you want to inform the model about an error that occurred in the function handler, make sure you return a string with an indicative error of what happened so the model can understand it and act upon it.
Throwing an error from inside a function handler just stops the inference and propagates the error upwards.

I recommend you to read the Using Function Calling and Using a JSON Schema Grammar guides, as they cover everything you need to know to use function calling and JSON schemas in node-llama-cpp.

If you still encounter issues with function calling, then I'd love to help you!
For that I'd need a simple and contained reproduction code that I can use to replicate the issue you're facing, as well as a link to the specific model file you used, the logs, and the exact errors that you get.
Please also use enable the debug mode on getLlama via (getLlama({debug: true})), and share the output of running the npx --yes node-llama-cpp inspect gpu command.

[giladgd]

@caramboleyo The LlamaChat class only handles the model generations while the LlamaChatSession class is a wrapper over LlamaChat for convenience, and is the one that calls the functions that the model attempts to call and feeds the results back into the model.
If you want to understand how LlamaChat is used in more detail, I recommend you to read the External Chat State guide.

[caramboleyo]

Well good luck going on then.