Incorrect tool parameter parsing when LLM call returns it as escaped json dict #52
Description
When working with llama, we observed that the tool call parameters can be occasionally an escaped json dict as opposed to json dict.
eg:
First request to LLM
{
"api_format": "GENERIC",
"frequency_penalty": null,
"is_echo": null,
"is_parallel_tool_calls": null,
"is_stream": false,
"log_probs": null,
"logit_bias": null,
"max_completion_tokens": null,
"max_tokens": null,
"messages": [
{
"content": [
{
"text": "You are a finance assistant. \nYou have access to these tools:\n- MonthlyVarianceSQL: fetch per-customer variance for a specific month and year\n- MultiMonthVarianceSQL: fetch variance summary for a year\nQuestion: {input}\nAlways use the appropriate tool first to get data, then explain in natural language.\n",
"type": "TEXT"
}
],
"name": null,
"role": "SYSTEM"
},
{
"content": [
{
"text": "Explain why cash inflow in August 2025 was below forecast.",
"type": "TEXT"
}
],
"name": null,
"role": "USER"
}
],
"metadata": null,
"num_generations": null,
"prediction": null,
"presence_penalty": null,
"reasoning_effort": null,
"response_format": null,
"seed": null,
"stop": null,
"stream_options": null,
"temperature": null,
"tool_choice": null,
"tools": [
{
"description": "Fetch variance by customer for a specific month/year. Input: month (str), year (int).",
"name": "monthly_variance_sql",
"parameters": {
"properties": {
"month": {
"description": "Analyze monthly",
"type": "string"
},
"year": {
"description": "Analyze year",
"type": "string"
}
},
"required": [
"month",
"year"
],
"type": "object"
},
"type": "FUNCTION"
},
{
"description": "Fetch variance summary across all months in a given year. Input: year (int).",
"name": "multi_month_variance_sql",
"parameters": {
"properties": {
"year": {
"description": "Analyze year",
"type": "string"
}
},
"required": [
"year"
],
"type": "object"
},
"type": "FUNCTION"
},
{
"description": "Customer list",
"name": "customers",
"parameters": {
"properties": {},
"required": [],
"type": "object"
},
"type": "FUNCTION"
}
],
"top_k": null,
"top_p": null,
"verbosity": null,
"web_search_options": null
},
"compartment_id": "ocid1.tenancy.oc1..xxx",
"serving_mode": {
"model_id": "meta.llama-3.3-70b-instruct",
"serving_type": "ON_DEMAND"
}
}tool call from lllm
{'modelId': 'meta.llama-3.3-70b-instruct', 'modelVersion': '1.0.0', 'chatResponse': {'apiFormat': 'GENERIC', 'timeCreated': '2025-10-22T18:22:21.606Z', 'choices': [{'index': 0, 'message': {'role': 'ASSISTANT', 'toolCalls': [{'type': 'FUNCTION', 'id': 'call_bd69677daf3f46a5bc382b4e', 'name': 'monthly_variance_sql', 'arguments': '{"month": "August", "year": "2025"}'}]}, 'finishReason': 'tool_calls', 'logprobs': {}}], 'usage': {'completionTokens': 25, 'promptTokens': 441, 'totalTokens': 466}}} and Headers: {'content-type': 'application/json', 'opc-request-id': 'xxx', 'content-encoding': 'gzip', 'content-length': '357'}
subsequent request to llm
{
"chat_request": {
"api_format": "GENERIC",
"frequency_penalty": null,
"is_echo": null,
"is_parallel_tool_calls": null,
"is_stream": false,
"log_probs": null,
"logit_bias": null,
"max_completion_tokens": null,
"max_tokens": null,
"messages": [
{
"content": [
{
"text": "You are a finance assistant. \nYou have access to these tools:\n- MonthlyVarianceSQL: fetch per-customer variance for a specific month and year\n- MultiMonthVarianceSQL: fetch variance summary for a year\nQuestion: {input}\nAlways use the appropriate tool first to get data, then explain in natural language.\n",
"type": "TEXT"
}
],
"name": null,
"role": "SYSTEM"
},
{
"content": [
{
"text": "Who are my customers?\n",
"type": "TEXT"
}
],
"name": null,
"role": "USER"
},
{
"annotations": null,
"content": [
{
"text": "Your request is incomplete. Please provide more details or specify the task you need help with.",
"type": "TEXT"
}
],
"name": null,
"refusal": null,
"role": "ASSISTANT",
"tool_calls": null
},
{
"content": [
{
"text": "Explain why cash inflow in August 2025 was below forecast.\n",
"type": "TEXT"
}
],
"name": null,
"role": "USER"
},
{
"annotations": null,
"content": [
{
"text": "",
"type": "TEXT"
}
],
"name": null,
"refusal": null,
"role": "ASSISTANT",
"tool_calls": [
{
"arguments": "{\"month\": \"August\", \"year\": \"2025\"}",
"id": "call_ed2635c686f449eea25915b2",
"name": "monthly_variance_sql",
"type": "FUNCTION"
}
]
},
{
"content": [
{
"text": "<tool_response!>",
"type": "TEXT"
}
],
"role": "TOOL",
"tool_call_id": "call_ed2635c686f449eea25915b2"
},
{
"annotations": null,
"content": [
{
"text": "<Agent_reponse>",
"type": "TEXT"
}
],
"name": null,
"refusal": null,
"role": "ASSISTANT",
"tool_calls": null
},
{
"content": [
{
"text": "Tell me more information about Acme Corp in this period",
"type": "TEXT"
}
],
"name": null,
"role": "USER"
}
],
"metadata": null,
"num_generations": null,
"prediction": null,
"presence_penalty": null,
"reasoning_effort": null,
"response_format": null,
"seed": null,
"stop": null,
"stream_options": null,
"temperature": null,
"tool_choice": null,
"tools": [
{
"description": "Fetch variance by customer for a specific month/year. Input: month (str), year (int).",
"name": "monthly_variance_sql",
"parameters": {
"properties": {
"month": {
"description": "Analyze monthly",
"type": "string"
},
"year": {
"description": "Analyze year",
"type": "string"
}
},
"required": [
"month",
"year"
],
"type": "object"
},
"type": "FUNCTION"
},
{
"description": "Fetch variance summary across all months in a given year. Input: year (int).",
"name": "multi_month_variance_sql",
"parameters": {
"properties": {
"year": {
"description": "Analyze year",
"type": "string"
}
},
"required": [
"year"
],
"type": "object"
},
"type": "FUNCTION"
},
{
"description": "Customer list",
"name": "customers",
"parameters": {
"properties": {},
"required": [],
"type": "object"
},
"type": "FUNCTION"
}
],
"top_k": null,
"top_p": null,
"verbosity": null,
"web_search_options": null
},
"compartment_id": "ocid1.tenancy.oc1..xxx",
"serving_mode": {
"model_id": "meta.llama-3.3-70b-instruct",
"serving_type": "ON_DEMAND"
}
} LLM response
{'modelId': 'meta.llama-3.3-70b-instruct', 'modelVersion': '1.0.0', 'chatResponse': {'apiFormat': 'GENERIC', 'timeCreated': '2025-10-22T19:48:12.726Z', 'choices': [{'index': 0, 'message': {'role': 'ASSISTANT', 'toolCalls': [{'type': 'FUNCTION', 'id': 'call_9049127a8871421db2b20b51', 'name': 'monthly_variance_sql', 'arguments': '"{\\"month\\": \\"August\\", \\"year\\": \\"2025\\"}"'}]}, 'finishReason': 'tool_calls', 'logprobs': {}}], 'usage': {'completionTokens': 27, 'promptTokens': 1022, 'totalTokens': 1049}}} and Headers: {'content-type': 'application/json', 'opc-request-id': '/xxx', 'content-encoding': 'gzip', 'content-length': '366'}
First tool call parameter extraction works fine,
'{"month": "August", "year": "2025"}'
but subsequent ones from the API call is sending the same params in an escaped json fashion
'"{\\"month\\": \\"August\\", \\"year\\": \\"2025\\"}"'
I found this in the docs for FunctionCall:
The arguments to call the function with, as generated by the model in JSON format. Note that the model does not always generate valid JSON, and may hallucinate parameters not defined by your function schema. Validate the arguments in your code before calling your function.
langchain-oci should try-escape the json to dict if it is not in right dict format.
Metadata
Metadata
Assignees
Labels
No labels