Force structured response from model when requesting video metadata. #80
Conversation
Change-Id: Iae31272ff4e0d630652d9069e527f2a5a537549a
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @JolandaVerhoef, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request refactors the video metadata creation process to leverage structured responses from the Gemini model. By configuring the model to return JSON data according to predefined schemas, the need for complex string parsing is eliminated, leading to more reliable and maintainable code. This change simplifies the prompts sent to the model and streamlines the extraction of metadata like timestamps for thumbnails, hashtags, and account tags.
Highlights
- Structured Model Responses: The Gemini model is now configured to return metadata (e.g., thumbnails, hashtags, account tags, chapters, links) as structured JSON objects or arrays, enforced by
responseSchemaandresponseMimeTypein thegenerationConfig. - Simplified Prompts: Prompts sent to the Gemini model have been simplified by removing explicit instructions for output formatting, as the model is now expected to adhere to the defined JSON schemas.
- Improved Data Parsing: The project now utilizes Kotlinx Serialization to parse the structured JSON responses from the model, replacing manual string parsing logic (e.g., for timestamps).
- Dependency Updates: Updated Firebase BOM and introduced Kotlinx Serialization dependencies to support the new structured response handling.
- Code Cleanup: Removed the
TimestampUtility.ktfile and related parsing logic, as its functionality is now handled by structured JSON parsing.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request effectively refactors the video metadata generation to use structured responses from the Gemini model, which is a great improvement for robustness and simplifies the prompts. The code changes are well-aligned with this goal, removing the need for manual string parsing of timestamps.
My review includes a few suggestions to further improve the code quality and adherence to the provided architecture guidelines. Specifically, I've pointed out a deviation from the recommended ViewModel pattern, an inconsistency in the data schema for chapter timestamps, and a small refactoring opportunity for better readability.
...m/android/ai/samples/geminivideometadatacreation/viewmodel/VideoMetadataCreationViewModel.kt
Outdated
Show resolved
Hide resolved
...m/android/ai/samples/geminivideometadatacreation/viewmodel/VideoMetadataCreationViewModel.kt
Outdated
Show resolved
Hide resolved
...m/android/ai/samples/geminivideometadatacreation/viewmodel/VideoMetadataCreationViewModel.kt
Outdated
Show resolved
Hide resolved
Change-Id: Ieb48cfd9af18343e365b9b0469afd69a6a38c23d
...m/android/ai/samples/geminivideometadatacreation/viewmodel/VideoMetadataCreationViewModel.kt
Outdated
Show resolved
Hide resolved
...m/android/ai/samples/geminivideometadatacreation/viewmodel/VideoMetadataCreationViewModel.kt
Outdated
Show resolved
Hide resolved
…n use case now has all of its relevant code inside one file (configuring the model, calling the model with a custom prompt, handling the response, and transforming it into UI. The goal is to make the sample more readable. In addition - I've added some nice UI for some of the generated responses, that can e.g. seek the video to the chapter timestamp, or externally open urls. Change-Id: Iae8b575f799891f74bdcc7ae68a6f0c72a5caaab
JolandaVerhoef
force-pushed
the
jv/structured-response
branch
from
c530e0a to
4f279fd
Compare
...tion/src/main/java/com/android/ai/samples/geminivideometadatacreation/GenerateAccountTags.kt
Show resolved
Hide resolved
...tion/src/main/java/com/android/ai/samples/geminivideometadatacreation/GenerateAccountTags.kt
Show resolved
Hide resolved
Change-Id: Ieebc965a45ca8f2d93c98fbcbdb4e4dfd266316b
Change-Id: I4c3de2084d340d69bddec7c10497d5a891041f8a
Change-Id: I8d08d4f503b2f85dd4968d9842b6ad4fb26ee85c
...m/android/ai/samples/geminivideometadatacreation/viewmodel/VideoMetadataCreationViewModel.kt
Outdated
Show resolved
Hide resolved
| fileData(videoUri.toString(), "video/mp4") | ||
| text( | ||
| """ | ||
| Suggest relevant accounts to tag in the video's description or comments to |
There was a problem hiding this comment.
Wondering what accounts types it will find, would it suggest only YouTube profile urls?
There was a problem hiding this comment.
The reason we restricted this to Youtube was because it has moderated accounts. If we don't constrain to youtube we cannot guarantee the platforms where Gemini will search for relevant accounts.
|
@lethargicpanda can you please re-review? |
Simplify prompts and use structured responses instead.