You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Let's say you have a hypothetical LanguageModelMiddleware that performs RAG and as part of the transformParams function, appends/injects the context at the end of the user message.
It reads in configuration options via a ragMiddlewareproviderOption.
How can I either:
Access the param (e.g. params.providerMetadata that have been updated after running this transform?
Alternatively, how can I use wrapStream and wrapGenerate to add a source, message annotation, or data chunk to the response?
Here's a very basic example of what that middleware might do:
exportconstdocumentRagMiddleware: LanguageModelV1Middleware={transformParams: async({ params }: {type: 'generate'|'stream',params: LanguageModelV1CallOptions}): Promise<LanguageModelV1CallOptions>=>{constdocRagParams=params?.providerMetadata?.docRagif(docRag.enabled){docRagParams.startTime=Date.now()letragCandidates=[]// ... do whatever RAG logic here, let's pretend it's in ragCandidatesdocRagParams.ragCandidates=ragCandidates// We can return information about the rag config and rag candiates in the now updated providerMetadata, but there's no real way to access it from the calling `streamText` that wrapped this middleware?return{
...params,providerMetadata: {
...params.providerMetadata,docRag: {
...docRagParams}}}}else{return{ ...params}},// Alternatively, we try to use wrapStream and wrapGenerate to add either a source part but I have no idea how to do this rightwrapStream: async({ doStream, params })=>{constragMetadata=params.providerMetadata?.docRagconsttransformStream=newTransformStream<LanguageModelV1StreamPart,LanguageModelV1StreamPart>({start(controller){if(ragMetadata?.ragCandidates&&ragMetadata?.ragCandidates?.length>0){for(constmetadataofragMetadata?.ragCandidates){controller.enqueue({type: 'source',source: {id: metadata?.chunkId??`chunk_${generateId()}`,sourceType: 'url',url: metadata?.docId??'',title: metadata?.docName??'',providerMetadata: {documentMiddleware: {
...ragMetadata}}}})}}},transform(chunk,controller){controller.enqueue(chunk)},})constresult=awaitdoStream()return{
...result,stream: result.stream.pipeThrough(transformStream)}},wrapGenerate: async({ doGenerate, params })=>{const{ providerMetadata, ...rest}=awaitdoGenerate()// This just updates the provider metadata for the request, but same problem, can't access it// Alternatively, if we could inject sources or annotations here, that would work as well.return{
...rest,
providerMetadata
}}}
I've also tried using formatDataStreamPart in the transform stream, e.g.:
We could attempt to access the updated providerMetadata in the caller's side, but this is using the provider metadata passed for the configuration, not the results:
// ... initial providerMetadata was set here, languageModel is now wrappedLanguageModel with the docRag middlewarereturncreateDataStreamResponse({asyncexecute(dataStream){conststream=streamText({model: wrappedLanguageModel,// This has the docRag middleware appliedasynconFinish(event){// event.providerMetadata contains old metadata, likely since it was stripped in the provider, so this will not work.console.log(event.providerMetadata)if(event?.providerMetadata?.documentMiddleware){dataStream.writeMessageAnnotation({docRag: {
...event?.providerMetadata?.docRag}})}}})stream.mergeIntoDataStream(dataStream)}})
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Let's say you have a hypothetical
LanguageModelMiddlewarethat performs RAG and as part of thetransformParamsfunction, appends/injects the context at the end of the user message.It reads in configuration options via a
ragMiddlewareproviderOption.How can I either:
param(e.g.params.providerMetadatathat have been updated after running this transform?wrapStreamandwrapGenerateto add a source, message annotation, or data chunk to the response?Here's a very basic example of what that middleware might do:
I've also tried using
formatDataStreamPartin the transform stream, e.g.:We could attempt to access the updated providerMetadata in the caller's side, but this is using the provider metadata passed for the configuration, not the results:
But that doesn't seem to work either.
Beta Was this translation helpful? Give feedback.
All reactions