The "Getting Started" page has a "JupyterLab Support" support section which recommends using pip or conda to install the jupyterlab and ipywidgets packages only. Further down it mentions that for Jupyter Lab 2.x only you need to install some extensions manually using jupyter labextension install.

However, information in the Troubleshooting in Python page is somewhat contradictory and shows that you do in fact need the jupyterlab-plotly extension to be installed in version 3 (and presumably 4):

JupyterLab Problems

In order to use plotly in JupyterLab, you must have the jupyterlab-plotly extension installed as detailed in the Getting Started guide. When you install plotly, this extension is automatically made available to any JupyterLab 3.x installation in the same Python environment.

It seems to me that many people are installing the jupyterlab-plotly extension more or less accidentally because the kernel they use the plotly package with happens to be running in the same environment that the JupyterLab server/application is. However there are several issues filed here where people who are using separate environments are running into problems and it's not completely clear what the resolution is. In many cases they are trying to resolve them with jupyter labextension install ..., which is deprecated and often doesn't work properly in v3.

I think the documentation on the main page could be clearer that you do in fact need the extension with JupyterLab v3, and that you should make sure to install the plotly package in the same environment you're running JupyterLab in even if you're using a different environment for the kernel.