I am filing this issue as something to possibly consider for the future.
Currently struct is used for converting bytes to ints (and a few other values like floats, but this issue is only about ints):
https://github.com/kaitai-io/kaitai_struct_python_runtime/blob/master/kaitaistruct.py#L170
For ints Python 3 has a different mechanism for converting from byte strings to ints namely from_bytes():
https://docs.python.org/3/library/stdtypes.html#int.from_bytes
What is very convenient is that it allows arbitrary length byte strings, so it becomes absolutely trivial to implement something like u3 or u5 or u11.