Defer debugpy and psutil imports until they are actually needed

[Carreau]

Both imports were paid for on every kernel startup even though most sessions never debug or ask for usage information.

• IPythonKernel.debugger is now a lazily-created property; the debugger (and the debugpy import) is only built on the first debug request. poll_stopped_queue is scheduled at that point rather than in start().
• debugger_class was a Type trait, which traitlets resolves — and therefore imports — as soon as the kernel is instantiated. It is replaced by a plain debugger_class_name string; debugger_class remains as a deprecated property, and subclasses still overriding it keep working with a DeprecationWarning.
• psutil is imported through _get_psutil(), which caches the (possibly None) result on first use.

On a local test (where I have optimisation in IPython and traitlets as well), this brings the startup time from 220ms to 170ms

[Carreau]

CI failure will be taken care of by #1544

[krassowski]

Both imports were paid for on every kernel startup even though most sessions never debug or ask for usage information.

They still get paid at every kernel startup after this PR. The debugger information is needed at the latest when generating kernel_info_request reply (which every frontend sends). Similar for psutil: init_connection_file → jupyter_client.connect.load_connection_file → _ip_default → localinterfaces._load_ips_psutil.

The import time of import ipykernel does not shrink either as neither debugpy nor psutil was imported on the pure import path (I worked on it in #1223)

A benchmark showing the benefits would help here. Otherwise, there is a number of regressions here:

1. kernel.debugger = X silently kills breakpoints (stock usage unaffected, but subclasses are)
2. IPythonKernel(debugger_class=X) is now silently ignored
3. _get_psutil() publishes the flag before the import (easy to fix though)
4. Debugger init failure is sticky and silent (debug_requests will return None after the first one)

[krassowski]

As per comment above. Thank you for working on it!

[Carreau]

Good point for subclasses, I'll see what I can do.

It's true that I tested only with #1542; that is minimal up to startup point and shut off immediately; and for import only for kernelapp.

The debugger information is needed at the latest when generating kernel_info_request reply (which every frontend sends)

Fair; I did not think about this – but I would not consider kernel_info_reply to be part of startup; I need to check but at least the startup before the first kernel_info_request is faster, I'm assuming the first kernel_info_request is sent only once the kernel is started (I'll verify), so I still think it's a (partial) win, and this should make the time to ready. But I least that's fixable separately and only affect the kernel_info_reply which for me is a separate issue – do we cache the result on disk ? Add a config to force the kernel to reply with 'debugger' always in or never and leave to to large deployment to select 'auto|true|false'

I'm also profiling locally with modified jupyter_client, traitlets and IPython (to defer more stuff from there as well); and yes some thing like psutils are otherwise still in startup path via other packages; but I can try to publish profiles even with modified versions to give an idea of what can be achieved.

[Carreau]

I suppose another big things, is are we ok dropping debugger_class traitlet and deprecating it to at least have some way of making debugger import lazy ?

[Carreau]

See #1545 for how I think we can handle making kerne_info_reply faster by providing a config option.

[Carreau]

Here are the result of

IPYKERNEL_BENCHMARK_STARTUP_SHUTDOWN=1 python tools/importtime_average.py -n 20 -- python -X importtime -m ipykernel  > totlog.log && tuna totlog.log

With #1542 merged in; to stop the kernel immediately and make it measurable. Also requires IPython'm main; traitlet and jupyter_client on latest releases

[Carreau]

Also technically you should test with -f fake-connection.json file; we can land a patch in jupyter_client that does not do the _ip_default → localinterfaces._load_ips_psutil when the connection file exists (which is the case most of the time as kernel are launched by frontend that likely can provide IPs in the connection files; but again; different repository. Different optimisation even if that goal is still to avoid importing psutil

[Carreau]

for debugger_class I can suggest something like this ipython/traitlets#972, make Type and conditional import to LazyType depending on traitlets version; or pull a minimalistic version that does not allow to check .klass and allow_none=True that i vendor here (which is what I did first):

class _LazyType(Type):  # type:ignore[type-arg]

    def instance_init(self, obj):
        """Deliberately does not resolve; see the class docstring."""
        pass

    def default(self, obj=None):
        self._resolve_classes()
        return super().default(obj)

    def validate(self, obj, value):
        self._resolve_classes()
        return super().validate(obj, value)

Though it becomes brittle as there is no more guard in MetaHasTraits.setup_class that prevent it from importing eagerly.