Jupyter(Hub) conceptual intro

[rkdarst]

At the JupyterHub/BinderHub workshop, one of our ideas was to make a conceptual intro to JupyterHub so that people could know what it does, and in particular what it doesn't do (what is handled by other components). We get many issues that end up misdirected or that have a root cause of not understanding what the components are.

This PR is my initial draft - comments welcome. I've written it informally and with a certain opinion - it's supposed to be like a teaching giving a first lesson, and not technical reference. It goes far beyond just JupyterHub, but the reason for having it here is that you really need to start knowing this stuff once you start administrating a JupyterHub, before that you can sort of get by without a perfect mental model.

Issues I know of:

• What is a proper title?
• Where should it go under the table of contents?
• I used the very verbose "single-user notebook server", I'd like a shorter term that still avoids ambiguity.
• All parts need fact-checking, because I haven't done that yet
• After the content is somewhat stable, we need to add links and references.
• Intro needs improvement and we should make sure it explains the purpose well
• Conclusion needs improvement and guides to what is next

[consideRatio]

@rkdarst thank you for putting in this excellent work to give an cohesive overview how things relate! I appreciate how it is written in a readable non-reference like manner, I'm thinking about the clarifications in the early part of the text for example where you clarify the purpose of the text is to give overview etc.

[rkdarst]

I did a big pass on this, and I think we should working on polishing. It's late and I'm sure everyone will be able to find little improvements - instead of making things perfect, I'll just let everyone read and make their suggestions.

The CI failures seem unrelated, but at least the docs one should be fixed.

[betatim]

Thanks for coming back to this!

The docs build error is:

Warning, treated as error:
/home/circleci/project/jupyterhub/auth.py:docstring of jupyterhub.auth.Authenticator.admin_users:1:duplicate object description of jupyterhub.auth.Authenticator.admin_users, other instance in api/auth, use :noindex: for one of them
Makefile:64: recipe for target 'html' failed
make: *** [html] Error 2

which I don't fully understand. There is one definition https://github.com/jupyterhub/jupyterhub/blob/master/docs/source/api/auth.rst (the "other instance"). But where is the first definition?

[betatim]

The weird thing is that on master the build seems to work. Maybe rebase your branch on the latest master ?

[rkdarst]

On Wed, Apr 15, 2020 at 03:09:51PM -0700, Tim Head wrote: There is one definition https://github.com/jupyterhub/jupyterhub/blob/master/docs/source/api/auth.rst (the "other instance"). But where is the first definition?

My current hypothesis is that it is about autodoc somehow: it's parsing the docstring to turn it into docs, but then one of the `.. autoconfigurable:: Authenticator` includes the same thing. And actually. scratch that. Look at the current docs: https://jupyterhub.readthedocs.io/en/stable/api/auth.html jupyterhub.auth.Authenticator has *every* attribute, including admin_users, duplicated! So, there *is* some actual, existing duplication. I guess something is stricter on warnings?

[rkdarst]

https://github.com/jupyterhub/autodoc-traits/blob/master/autodoc_traits/autodoc_traits.py#L28

I added some "print" statements right before this line, and sure enough, admin_users is in both trait_members and members. Same for Authenticator and LocalAuthenticator, but not for PAMAuthenticator or DummyAuthenticator. The first two have :members: in the rst file. Anyone know details of which way is correct?

@willingc ?

[rkdarst]

jupyterhub/autodoc-traits#1

After this, I am able to get the build to run further - but then there are other new errors. I guess sphinx really has gotten a lot stricter in more ways than one. Easy fix is removing -W from docs/Makefile...

[betatim]

Thanks for diving into autodocs! We introduced the -W flag a few years back to help keep the docs in "reasonable shape". At the time there the number of warnings was so big it was hard to tell if a change added more warnings or not. So the weary maintainer in me wants to keep -W to help us keep improving the quality of the docs or at least not adding new debt to them.

However we don't want to block progress with this. Maybe as a compromise we can limit ourselves to sphinx<3 for now? A next step would be to start a (hopefully) small coordinated effort to get the docs building with v3.

Curiously the last build on master still uses Sphinx-2.4.4 and it is green.

[rkdarst]

Discussion of docs failure moved to #3021.

[betatim]

Hurrah! All CI robots are happy again. Great work digging into sphinx and autotraits!

[rkdarst]

Now that CI works, would someone like to take a look at this? It is ready now, I've done multiple passes.

Some hints on what to check is at the top, but roughly a) is the placement within the docs good? Should it entirely be somewhere else?

... and b) fact-checking. I've done lots with Jupyter so I think there's not much wrong, but of course there's just so much that any other eyes will of course help.

[betatim]

"... provides multi-user login $thing" The sentence somehow ends abruptly. Could it be "...multi-user login capabilities." or "...functionality."?

[betatim]

Can we move the "The authenticator can also return..." sentence to the end of the paragraph? Lets tell people about the main thing the authenticator does "return a username" and then afterwards tell them about "but wait there is more".

[betatim]

Missing words in the second half of the sentence?

[rkdarst]

I think the problem was an extra "if". Removed.

[rkdarst]

Suggested change

	obvious how the parts relate. The knowledge contained here hasn't
	been assembled in one place before, and is essential to understand
	when setting up a sufficiently complex Jupyter(Hub) setup.
	This document was originally written to assist in debugging: very
	often, the actual problem is not where one thinks it is and thus
	people can't easily debug. In order to tell this story, we start at
	JupyterHub and go all the way down to the fundamental components of
	Jupyter.
	obvious how the parts relate.

Removing as per suggestion

[rkdarst]

yes, we should (not just here but many places scattered around here), but to get it out I'll save that for later...

[rkdarst]

I added a link to /reference/spawners, but didn't remove any from here since they somehow were chosen to represent the diversity of what's available... I'll let someone else pick what to remove.

[rkdarst]

I think I got this right now (in the push that is upcoming), might be worth someone checking that it matches modern standards...

[rkdarst]

I removed the sections but not the text, someone else can do that later. Anyway, there is jupyter_server now, which will some sort of updates here, right?

[rkdarst]

Does that mean this one? https://www.youtube.com/watch?v=JxyKBNJnfVM
Since it's mine and perhaps old, I'll let someone else decide to add it.

[rkdarst]

In the sprit of "getting it out and update", I made rebased to current upstream and did the quick revisions from the reviews. There are still more extensive things to do, but I'll leave that for someone else for later.

[rkdarst]

linkcheck failures seem unrelated

[yuvipanda]

I am a big fan of merging this one quickly. It has a lot of useful information and I'd prefer merging something imperfect and then iterating, rather than having all of this knowledge locked up in a PR draft.

I agree with @choldgraf. To that end, I'm going to do the following:

1. Rebase this on main
2. Move this out of tutorial to explanations/ where it fits best (I think)
3. Merge this
4. Make some smaller PRs to address some of the issues.

This is very helpful work @rkdarst, and I am grateful to you for doing it!

[yuvipanda]

I did also remove the reference to hubshare (It's dead I think), and linked to newer part of idle culler. Also removed the explicit TODO, and added a note about how this document is not as well cross linked as it should be.

[yuvipanda]

Thank you very much for your patience, @rkdarst :)