improvement: allow configuring node headers in child nodes

[zachdaniel]

improvement: allow extras to have child nodes
improvement: allow for hiding sidebar-items in the sidebar (but not in autocomplete)

I'm really hoping that we can get this merged despite the fact that almost certainly no one but users of Spark would want this. The changes that this makes allow for us to do more by virtue of post-processing sidebar-items-***.js. For DSLs (especially the big ones that Ash core has), it is not realistic to rely on generated module/function docs, primarily due to the hierarchical nature of the data in question, and the duplication of root node option names. i.e there is a public? and allow_nil? options on 10+ different DSL entities.

What we've done for a while is generated a markdown file that is a "guide", and then modified search-data to add entries for DSL options. But honestly this just isn't that useful, is super hidden for users, and barely if ever gets used.

We can't make a header-per option because that 1.) looks bad and 2.) it would be hundreds and hundreds of options. But only markdown headers are automatically lifted to sidebar-items. Additionally, sidebar-items are what drives search and what's in the sidebar, which is problematic for the same reasons. We can't just add sidebar-items because then the sidebar would have an item per option which would make it huge.

I decided to take a stab at modifying sidebar items in such a way that:

1. extras support having nodeGroups like modules do. I didn't need to do anything from this except remove headers and add sections and nodeGroups.
2. node groups' children can specify a header which will change how they appear in the sidebar but not in autocomplete
3. sidebar-items can be hidden, such that they only appear in search

None of these things should affect anyone who isn't doing post-processing of these files. I'm hoping that, while very specific to one use case, these changes are unobtrusive enough to be allowed to happen.

If so, I'm happy to test the changes and do any cleanup required to get this over the line.

EDIT: added one other improvement which is to allow for nested child nodes to specify their own label, used to show DSL Entity and DSL Option next to autocomplete items.

[github-actions]

📦 Docs artifacts are ready: https://github.com/elixir-lang/ex_doc/actions/runs/12661447721/artifacts/2398787267

[zachdaniel]

Video showing the UX here: https://bsky.app/profile/did:plc:asqrltlnmzecxz42wklq6yy2/post/3lf6szctzjs2g

and the generated docs can be seen here: https://ash-project.github.io/ash/readme.html

[zachdaniel]

Looks like mobile sidebar issues reported by a user, so I'll address those as well if this approach is considered reasonable.

[josevalim]

Thanks @zachdaniel. My biggest concern with this change is that we don't want to promote users to postprocess the sidebar-items, as those may change any time, or even removed altogether. I'd suggest for us to first think of a mechanism we could make this official, where we can configure how the h2 headers in a given page are processed.

[zachdaniel]

Hm…I had no idea that we didn’t want people to post process that file. In fact I thought an outcome of a previous conversation on how to customize ex_doc was to post process. Thinking on it now perhaps the idea was to post process html in some way but I guess even that would be unreliable for the same reasons 😅

I don’t think it’s enough to configure the h2 header handling because the things we want to allow autocompleting are in tables and don’t appear in headers, but I can try to find a different way to do this that doesn’t have these same problems.

I don’t think there is any solution that doesn’t involve stabilizing the shape of the data in sidebar-items.js at least to some degree.

some ideas:

custom search items

Supporting something like “custom-sidebar-items.json” that contains overrides of sidebar items by file, and stabilizing the subset of sidebar item structure needed for that.

injecting search items

supporting injecting some js into search that produces additional search results, which would only need to return a list of objects with the fields we pass into render autocomplete items. This stabilizes only those fields, and in a way where we can add more later.

We could couple this with some way to customize the sidebar heading to use nodeGroups for a certain level of nesting which could get me close to the UX we have now with this change.

[josevalim]

Hm…I had no idea that we didn’t want people to post process that file. In fact I thought an outcome of a previous conversation on how to customize ex_doc was to post process

I think the emphasis is in "promote". It is completely fine for you to post-process, as you are well aware of the risks, but at the moment we add features that rely on post-processing it gets a more official stamp than I am comfortable with. :)

Regarding your comments, are you sure this is about search_items/search_data? Anything in the document is searchable and I believe we index up to h3 as separate entries in search, but autocompletion+sidebar use the sidebar items. My understanding is that you want the latter, so search is not relevant.

[zachdaniel]

Sorry, I misspoke. I'm referring to sidebar-items.js. EDIT: I've updated my comment accordingly.

[zachdaniel]

Do those other approaches sound reasonable to you?

[josevalim]

Yes, I think something in this direction is acceptable, and it is similar to what I meant in configuring how pages are rendered. But I think we need to surface it up all the way to configuration, like extras_config: ["foo.md": [override_entries: [...]], as it is more general and it comes with the benefit of also working for EPUB.

[zachdaniel]

The big challenge with that approach is that the mix project is built in a way that I can't call dependencies while it's building, and Spark the dependency would be in charge of building this list based on the DSL definitions. So this approach only allows for it to be hard-coded by hand as far as I know, which won't work for this.

[zachdaniel]

Actually perhaps just configuring it to a function or an mfa is all that is necessary there. Will explore.

[josevalim]

Actually perhaps just configuring it to a function or an mfa is all that is necessary there. Will explore.

You can already pass an anonymous function to docs: fn -> ... end, which should be invoked a bit later. You actually want to use it as much as possible, otherwise you pay the price of docs processing on every mix invocation.

[zachdaniel]

I will close this in favor of exploring the suggested approach. I think unfortunately doing it this way is going to take a fair bit longer and so I'm not sure I will be able to prioritize it the same way, but I will try as it is a pretty serious QoL improvement for Ash users.

[josevalim]

Do reach out if you have questions. It probably makes sense to do a tiny proof of concept first so we can validate and evolve it together, instead of a huge change upfront!