Is there a way to implement context window compacting/compressing?

[inno-odyssean]

I am trying to find a way so that when the context window limit is about to be reached I would call an llm and the llm would create a compacted summary of the conversation and insert this to the system message, while truncating messages to the latest message.

I am just confused as to where to insert this operation.

are there any best practices for this?

[lgrammel]

If you use streamText or generateText, then the prepareStep callback might be the best place: https://ai-sdk.dev/docs/reference/ai-sdk-core/generate-text#prepare-step

[inno-odyssean]

Thanks! I was able to implement it in the prepareStep callback.

[github-actions[bot]]

This discussion was automatically locked because it has not been updated in over 30 days. If you still have questions about this topic, please ask us at community.vercel.com/ai-sdk