The bug
For complex grammar, the generation becomes CPU bound and doesn't terminate. My guess is that the problem lies with empty or zero-width strings not being properly handled by the parser. By line profiling I see that all of the time is spent in these lines:

To Reproduce
I use this model and this code:
https://huggingface.co/cognitivecomputations/dolphin-2.9-llama3-8b
https://gist.github.com/bytemouse/6b8eaa647840c3793d5a4f23516b2a5f

System info
OS: Fedora 40
Guidance Version: 0.1.15