Clean up modules

[akern40]

ndarray has a lot of modules! They tend to be a little hard to read through, and it can be hard to know where code should go when adding or modifying the library. I'd like for us to refactor our codebase to follow a few principles for module definition:

1. Always have module-level documentation, even for private modules. By telling readers what a module aims to do, we can help contributors know where to add/remove/change code.
2. Reduce or eliminate most module-level code, opting for just mod ... and ... use ... declarations. This should make it really fast for contributors to figure out how a module is structured.

This is especially important for public modules, but preferably should be applied to private modules, as well.

[jmg049]

It might make things simpler and more reproducible if there was an expected layout / format to the docs at both the module level and non-module level.

With what would essentially be a checklist of expected content, it would make it easier for people to review/contribute.

In some personal projects I have opted for setting the likes of the following as guidelines for docs (just a subset of things):

Module Level

• high level details, no technical implementation detail. What is the module and why does it exist?

• high level examples of what the module offers

Non-Module Level

• Every function should have

  • an explanation of what it does and why/when/where it should be used.
  • inputs and what they do
  • outputs
  • if it returns a result describe where and why the function can return an error
  • Same for options, when can it be none
  • unsafe code explanation, guarantees/assumptions for it to be true and safe.
  • if panics then when and why panic. It must be justified

• Similar for structs, enums, traits and other documentable items.

I think this is a great idea, just that it'll would be more enforceable if there's a set style guide / docs checklist.

[akern40]

Jack, I think that's a great call, and I like your suggested guidelines. I would change a few things:

1. For non-module documentation, we should hew a little closer to the Rust doc guidelines, which are less prescriptive than Python / NumPy. In particular, Rust does not ask you to explicitly document inputs / outputs as separate from the "explanation of what it does and why/when/where it should be used", and I don't think that we should, either.
2. I'd remove the line about "it must be justified" for panics, for two reasons. Mainly because that's an instruction about the implementation, not the documentation, but also because "justified" there is its own can of worms. We do need to address panics in the library (I don't love how many there are), but I feel that's a separate project.

On a related note, you seem to have a good eye for clear communication. If you're interested in contributing more, could I interest you in a larger project that I'd like to embark on? #1499 lays out my vision for a documentation website, and an argument for why we need one. I had been putting it on the backburner to focus on #1506 and implementation details, but I wonder if it would be better to stand it up "under construction" and have people make PRs to slowly build it (rather than waiting until a "v1" and then releasing). If I got the initial infrastructure stood up and working (with some initial documentation that I have), would you be interested in helping contribute there?

[jmg049]

Oh of course, my outline was just an example of what I personally use in my projects.

The whole point of them is just to try and make explicit anything that can/will go wrong.

Thanks for the comment on communication, I've just spent the past 4 years learning/ trying to prevent any kind of misunderstandings/ miscommunication in academic writing, so it's nice to see something is on the right track.

In terms of the site:

• definitely interested in helping out. I think having a dedicated website which encompasses all aspects of ndarray is a great idea. It introduces a single, primary entry point to the ecosystem. It supplements the likes of doc.rs and/or makes it possible to be more explanatory in nature compared to existing sources of information. Not to mention how easy it would be to add more design focused information on the project; design decisions, best practices, contributing and so on.
• then in terms of when to make it available/an initial getting it out there. For these types of things I lean more on the conservative side of things and would be in favour of making sure that whatever is put out first, is reflective in terms of how it means to go on. By this I mean having a really solid foundations before properly opening it up, i.e. aiming for a v1 that corresponds to some future release of ndarray. Make it part of the roadmap and commit to it coming with some future release. That way it's content is also tied to that release and a v1 starts from a fixed point in time.

Obviously take everything with a grain of salt, you have been involved in ndarray for much longer than me. Just my two cents.