Add SmolLM3

[joelpaulkoch]

Hey, this is the SmolLM3 model from huggingface. It's smol, fully open and supports reasoning, so I figured it would be a nice addition to bumblebee.

I didn't implement YaRN extrapolation.

[jonatanklosko]

Hey @joelpaulkoch, this looks great! I dropped a few small comments and it's good to go :)

[joelpaulkoch]

The implementation is basically llama + NoPE support (in the transformer block) + architectures that are supported but missing in llama (i.e. :for_question_answering and :for_token_classification). So, would you prefer to add the optional NoPE support and architectures in the llama implementation and map smollm3 to llama?

[jonatanklosko]

So, would you prefer to add the optional NoPE support and architectures in the llama implementation and map smollm3 to llama?

It's separate in hf/transformers, so I would keep it separate here to for consistency. Also, I wouldn't necessarily add features to llama that are not in the hf/transformers implementation, otherwise it's harder to analyse for parity :)

[jonatanklosko]

Nit: we typically only include 4 decimal places, since the further ones are not relevant for the precision we compare with anyway.

Suggested change

	Nx.tensor([
	[
	[0.256240, -0.424804, -0.137127],
	[-0.806056, -0.141523, 0.364655],
	[-0.407146, -1.018769, -1.137962]
	]
	]),
	Nx.tensor([
	[
	[0.2562, -0.4248, -0.1371],
	[-0.8060, -0.1415, 0.3646],
	[-0.4071, -1.0187, -1.1379]
	]
	]),

[jonatanklosko]

This is unusual, pretty much all other LLMs work with the default atol 10e-4. If the numbers are slightly off like that, it often indicates a small difference, like a missing layer norm, layer norm in a different order, or something like that.

Do you know if that deviation is only for the test models, or is it similar for any real checkpoint?

[joelpaulkoch]

I checked and it's the same for HuggingFaceTB/SmolLM3-3B, not sure what is missing.
A missing layer norm would show up in the debug logs, right?
I'll investigate further when I find the time.