-
Notifications
You must be signed in to change notification settings - Fork 100
feat: Add cookbook for Hallucination Risk Calculator #252
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
@julian-risch this is ready for review! It includes an example now where we create a RAG Query pipeline |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Except for my comment above, I don't have any comments on how to improve. The RAG example is really great to have! The output cell at the very end with an example of no answer due to missing relevant documents too.
Disclaimer: I did not run the code.
@@ -0,0 +1,435 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is there a risk here although the answer is basically the evidence provided in the prompt?
Reply via ReviewNB
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a risk bound and the risk is assessed based on this number and a threshold. I can't say I fully understand how it works this was mostly copied from the original repo.
@@ -0,0 +1,435 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left my comments! Can you also add the notebook to the index.toml for it to be rendered on the website. Here's an example for notebooks with experimental components.
Experimental release is done. I'll merge this notebook so that we can link to it in release notes. |
Cookbook for this PR deepset-ai/haystack-experimental#359
Some feedback on this from @julian-risch deepset-ai/haystack-experimental#359 (comment)