Simon Willson's Lethal Trifecta

[MentalGear]

It be interesting how superagent firewall compares to Simon Willson's Lethal Trifecta and Dual LLM separation.

[homanp]

@MentalGear As Simon correctly points out in his blog post, the real danger comes when you combine private data access, untrusted inputs, and external communications.

The Superagent Firewall is designed to break that chain in three ways:

• Strict tool and data scoping: each agent only has access to the minimum set of tools and data it needs, reducing the chance of accidental leakage.

• Runtime inspection and mediation: every inbound/outbound call or sensitive action passes through the firewall, which can block, sanitize, or require review before execution.

• Defense-in-depth with separation: the firewall complements dual-LLM or multi-agent separation by acting as a gatekeeper, ensuring that even if one component is exposed to untrusted inputs, it can’t directly trigger exfiltration of sensitive data.

[MentalGear]

Thanks for the details - I really appreciate your work on this fine tuned SLM and providing a pioneering AI firewall solution ! It's a really slick design and the docs are well made.

Dual LLMs: A guide / doc page on how to best integrate dual LLMs with the AI firewall would be much appreciated.

Regarding security claims verification: do you have any benchmarks / test suites that you run which can be used to benchmark your solution against others ? E.g. is there a list of public injection attacks (with synthetic augmentations/alternations) that you run your solution against ?

[homanp]

With regards to benchmarks, most benchmarks out there are focused on jailbreaks, but again as Simon points out that's the tip of the ice berg. Thus we have our own benchmarks and dataset. You can get access to it here: https://huggingface.co/datasets/superagent-ai/superagent-lm

Regarding Dual LLMs, depends a bit on your setup, do you have a specific setup in mind?

[homanp]

We are releasing some utilities to make it easier to consume Superagent, first up is LiteLLM integration:

BerriAI/litellm#14848 (comment)