Is your feature request related to a problem? Please describe.
Matplotlib is not needed in headless environments and can cause compilation issues in some environments.

Describe the solution you'd like
Make the dependency on matplotlib options. Probably on by default, but allowed to be removed.

Describe alternatives you've considered
None so far.

Additional context
Happy to contribute this if maintainers agree.