`c.Authenticator.enable_auth_state = True` prevents JupyterHub from starting

[manics]

Bug description

Originally reported in https://discourse.jupyter.org/t/error-while-enabling-auth-state-persistance-for-genericoauthenticator/27746

How to reproduce

1. Install TLJH
2. Create a custom config file in /opt/tljh/config/jupyterhub_config.d:
   echo 'c.Authenticator.enable_auth_state = True' > /opt/tljh/config/jupyterhub_config.d/authstate.py
3. Run tljh-config reload, hub will fail to start
4. Check the logs sudo journalctl -u jupyterhub

Aug 22 12:39:49 ubuntu systemd[1]: Started jupyterhub.service.
Aug 22 12:39:50 ubuntu python3[6450]: [I 2024-08-22 12:39:50.825 JupyterHub app:2885] Running JupyterHub version 4.1.6
Aug 22 12:39:50 ubuntu python3[6450]: [I 2024-08-22 12:39:50.826 JupyterHub app:2915] Using Authenticator: firstuseauthenticator.firstuseauthenticator.FirstUseAuthenticator
Aug 22 12:39:50 ubuntu python3[6450]: [I 2024-08-22 12:39:50.826 JupyterHub app:2915] Using Spawner: tljh.user_creating_spawner.UserCreatingSpawner
Aug 22 12:39:50 ubuntu python3[6450]: [I 2024-08-22 12:39:50.826 JupyterHub app:2915] Using Proxy: jupyterhub_traefik_proxy.fileprovider.TraefikFileProviderProxy-1.1.0
Aug 22 12:39:50 ubuntu python3[6450]: [I 2024-08-22 12:39:50.842 JupyterHub app:1683] Loading cookie_secret from /opt/tljh/state/jupyterhub_cookie_secret
Aug 22 12:39:50 ubuntu python3[6450]: [E 2024-08-22 12:39:50.996 JupyterHub app:3408]
Aug 22 12:39:50 ubuntu python3[6450]:     Traceback (most recent call last):
Aug 22 12:39:50 ubuntu python3[6450]:       File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/app.py", line 3405, in launch_instance_async
Aug 22 12:39:50 ubuntu python3[6450]:         await self.initialize(argv)
Aug 22 12:39:50 ubuntu python3[6450]:       File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/app.py", line 2932, in initialize
Aug 22 12:39:50 ubuntu python3[6450]:         await self.init_users()
Aug 22 12:39:50 ubuntu python3[6450]:       File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/app.py", line 1957, in init_users
Aug 22 12:39:50 ubuntu python3[6450]:         ck = crypto.CryptKeeper.instance()
Aug 22 12:39:50 ubuntu python3[6450]:       File "/opt/tljh/hub/lib/python3.8/site-packages/traitlets/config/configurable.py", line 583, in instance
Aug 22 12:39:50 ubuntu python3[6450]:         inst = cls(*args, **kwargs)
Aug 22 12:39:50 ubuntu python3[6450]:       File "/opt/tljh/hub/lib/python3.8/site-packages/traitlets/config/configurable.py", line 119, in __init__
Aug 22 12:39:50 ubuntu python3[6450]:         self._load_config(self.config)
Aug 22 12:39:50 ubuntu python3[6450]:       File "/opt/tljh/hub/lib/python3.8/site-packages/traitlets/traitlets.py", line 687, in __get__
Aug 22 12:39:50 ubuntu python3[6450]:         return t.cast(G, self.get(obj, cls))  # the G should encode the Optional
Aug 22 12:39:50 ubuntu python3[6450]:       File "/opt/tljh/hub/lib/python3.8/site-packages/traitlets/traitlets.py", line 635, in get
Aug 22 12:39:50 ubuntu python3[6450]:         default = obj.trait_defaults(self.name)
Aug 22 12:39:50 ubuntu python3[6450]:       File "/opt/tljh/hub/lib/python3.8/site-packages/traitlets/traitlets.py", line 1897, in trait_defaults
Aug 22 12:39:50 ubuntu python3[6450]:         return t.cast(Sentinel, self._get_trait_default_generator(names[0])(self))
Aug 22 12:39:50 ubuntu python3[6450]:       File "/opt/tljh/hub/lib/python3.8/site-packages/traitlets/traitlets.py", line 1241, in __call__
Aug 22 12:39:50 ubuntu python3[6450]:         return self.func(*args, **kwargs)
Aug 22 12:39:50 ubuntu python3[6450]:       File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/crypto.py", line 93, in _config_default
Aug 22 12:39:50 ubuntu python3[6450]:         return JupyterHub.instance().config
Aug 22 12:39:50 ubuntu python3[6450]:       File "/opt/tljh/hub/lib/python3.8/site-packages/traitlets/config/configurable.py", line 592, in instance
Aug 22 12:39:50 ubuntu python3[6450]:         raise MultipleInstanceError(
Aug 22 12:39:50 ubuntu python3[6450]:     traitlets.config.configurable.MultipleInstanceError: An incompatible sibling of 'JupyterHub' is already instantiated as singleton: JupyterHub
Aug 22 12:39:50 ubuntu python3[6450]:     
Aug 22 12:39:51 ubuntu systemd[1]: jupyterhub.service: Main process exited, code=exited, status=1/FAILURE
Aug 22 12:39:51 ubuntu systemd[1]: jupyterhub.service: Failed with result 'exit-code'.
Aug 22 12:39:51 ubuntu systemd[1]: jupyterhub.service: Scheduled restart job, restart counter is at 5.
Aug 22 12:39:51 ubuntu systemd[1]: Stopped jupyterhub.service.
Aug 22 12:39:51 ubuntu systemd[1]: jupyterhub.service: Start request repeated too quickly.
Aug 22 12:39:51 ubuntu systemd[1]: jupyterhub.service: Failed with result 'exit-code'.
Aug 22 12:39:51 ubuntu systemd[1]: Failed to start jupyterhub.service.

This can also be reproduced by running the TLJH systemd start command

the-littlest-jupyterhub/tljh/systemd-units/jupyterhub.service

Line 21 in c49fada

ExecStart={python_interpreter_path} -m jupyterhub.app -f {jupyterhub_config_path} --upgrade-db

directly in a terminal:
/opt/tljh/hub/bin/python3 -m jupyterhub.app -f /opt/tljh/hub/lib/python3.8/site-packages/tljh/jupyterhub_config.py

Replacing /opt/tljh/hub/bin/python3 -m jupyterhub.app with /opt/tljh/hub/bin/python3 -m jupyterhub or /opt/tljh/hub/bin/jupyterhub both work
Note you'll also need to set JUPYTERHUB_CRYPT_KEY, but this is not the cause of this bug

Expected behaviour

Actual behaviour

Your personal set up

Killercoda Ubuntu 20.04.5 LTS VM https://killercoda.com/manics/scenario/the-littlest-jupyterhub

[minrk]

very strange that there are two JupyterHub classes trying to be instantiated, but they are not the same. This makes me think something fishy is happening with sys.path. Especially given that the reported env shows jupyterhub 5.1.0, but the startup logs show 4.1.6.

I'll need to think about how to debug this, but ipython/traitlets#909 might have helped. Further debugging cls.__file__ might help.

[minrk]

Ah, yeah. Learn something about Python every day:

In jupyterhub/app.py:

def main():
    from jupyterhub import app

    print(JupyterHub.__module__) # __main__
    print(JupyterHub) __main__.JupyterHub
    print(app.JupyterHub) jupyterhub.app.JupyterHub
    print(JupyterHub is app.JupyterHub) # False!

So jupyterhub shouldn't really allow -m jupyterhub.app at all, but it can technically be fixed.

The fix here should be to replace -m jupyterhub.app with -m jupyterhub.