V5 Beta and Passing Files to Tools via sendMessage?

[andreaselia]

I have this working, but I'm just curious whether there are "better" ways to go about passing files for tool use.

Frontend:

sendMessage({ text }, { body: { imageUrl } });

Backend:

const { messages, imageUrl } = await c.req.json();

Then I can use imageUrl within my tool, but it doesn't feel as nice as if the image URL was specified within the user's message, and then automatically extracted using inputSchema.

The automated method works with the below setup, but it exposes the image URL to the user, which doesn't look nice.

Frontend:

sendMessage({ text: `${text} - Image URL: ${imageUrl}` });

Backend:

inputSchema: z.object({
  prompt: z
    .string()
    .describe("The text prompt describing what to edit on the image"),
  imageUrl: z.string().url().describe("The URL of the image to edit"),
}),

Any suggestions on improvements welcome, but at the moment I'm rolling with the former, and manually passing the imageUrl through to the tool.

[andreaselia]

@nicoalbanese based on our discussion on Twitter: https://x.com/digitalandreas/status/1948362388585033777

[andreaselia]

Discussed with @dancer and probably the best way to do it at the moment is by passing it through with the extra body param via sendMessage it seems.