Experiment with Cursor-generated docs for PowerShell scripts

[mokagio]

See discussion at #155 (comment)

This is on top #153 which has all the up to date PS1 scripts.

---

For the record, here's the prompts I gave.

• update the documentation for the ps1 commands based on the rules
• The PS1 scripts should not have a Bash shebang! Also, do it only for the scripts in bin/ the ones in tests/ are fine the way they are
• mmm... why did you skip @path_aware_refreshenv.ps1 ?
• thanks. revisit the requirements for @prepare_windows_host_for_app_distribution.ps1 are you sure about them?
  • at this point it added more requirements...
• but... doesn't the script itself install Chocolately and jq?

I also asked it to generate the list of prompts I gave it, which it did but left out quite a few

---

At this point, I don't like this setup. The documentation it generates is very verbose. I suppose we could update the rules to make it more concise, removing redundant entries (like having requirements in notes and in the "Requirements" section). But I felt it took me as much time to get it to generate the docs and iterate on them, plus my final read, as it would have taken me to write a few lines explaining what the script does.

I'm not going to push against having this kind of documentation if @Automattic/apps-infrastructure feels it's useful. I acknowledge I have a low threshold for how much commented text I can read before looking at code. And after all, we did discuss doing a better job at documentation to help everyone in the team be productive across our tooling, particularly in the areas where one has less experience. But... I wonder if verbose headers docs are the way here vs smaller scripts or ad hoc comments to explain what each chunk does, for example.

I'm also okay to keep using LLMs to generate the docs. If we do this, I'd love to have a way to make it clear that's the primary author of the documentation. I feel as a reader it would help me understand why the docs read the way they do. And as a dev, it would make it clear that I should ask an LLM to generate the docs instead of — big air quotes — "waisting" time writing them myself from scartch.

[dangermattic]

	1 Warning
⚠️	Please add an entry in the CHANGELOG.md file to describe the changes made by this PR

	1 Message
📖	This PR is still a Draft: some checks will be skipped.

Generated by 🚫 Danger

[mokagio]

Suggested change

	# Installs the Windows 10 SDK and Visual Studio Build Tools based on a version
	# specified in a .windows-10-sdk-version file.
	# Installs the Windows 10 SDK and Visual Studio Build Tools based on the
	# version specified in a .windows-10-sdk-version file.

[mokagio]

This seems redundant

Suggested change

	# - Saves PATH before refreshenv and restores unique entries after
	# - Use after installing packages that modify PATH at runtime

[mokagio]

Redundant because already in the requirements section

Suggested change

# - Requires administrator privileges

[mokagio]

On a second pass of the diff, I noticed that none of the docs used the full spec from .cursor/rules/powershell_scripts.mdc

.NOTES
    Author: [Author name]
    Last Edit: [Date]
    Version 1.0 - Initial release
    Requirements: Windows PowerShell 5.1
    Windows Requirements:
        - Windows Server 2019 or later
        - Required Windows Features: [list features]
        - Required Windows Roles: [list roles]
        - Administrator privileges: [Yes/No]

Maybe I didn't prompt it well enough?

[AliSoftware]

@mokagio I did the same experiment on my end asking my Cursor Composer to generate those docs too, you can see the result in #160. I left the generated content as-is in the draft PR to highlight what it generated without any manual tweaking.

PS: Did you close the folder/window and quit Cursor before switching to the commit that contained the .mdc files then reopening Cursor only then, before trying to ask it for the prompt?

[mokagio]

Did you close the folder/window and quit Cursor before switching to the commit that contained the .mdc files then reopening Cursor only then, before trying to ask it for the prompt?

I didn't record myself going through the process so I might be mistaken, but I'm pretty sure I did because I remember reading about that gotcha either in the other docs PR or on Slack.

[mokagio]

I left the generated content as-is in the draft PR to highlight what it generated without any manual tweaking.

This comment actually makes me think of something interesting / noteworthy / that should be kept in mind when thinking of LLM-generated docs: The result is non-deterministic!

I don't think that's a show stopper, and I'd like to think that with a set of rules well configured, the result should be similar across different prompt styles. But that's something I don't think I ever thought about and that makes me a bit uncomfortable, in a sense. In a world where we go through a lot of trouble to get deterministic result (VMs, lock-files, inject app state to decouple unit tests from it, etc.) getting different results every time is unsettling.

Again, I don't think it's a blocker, especially when it comes to documentation written in natural language and with the expectation that the better we get at configuring the .cursor/rules and at prompting, the better and more reliable the results will be.

[AliSoftware]

That is indeed good to keep in mind. Though, to be clear, the use of LLM here was never intended to generate (deterministic) documentation.

After all, the goal is not to have the code not documented and have a process that would generate documentation dynamically on the fly each time we need it, but rather to have a robot intern help us write some documentation for code so we can review and commit it once it's written. In that sense, the LLM is just like a robot intern. We might each have a slightly different intern each with its slightly different voice and tone and style of writing, but that doesn't really matter if the writing style is different between each attempt at asking the LLM the same question from different machines or at different times, what matters is that we end up with some sort of documentation that makes sense and is helpful.

Then the more documentation we'll write as headers of our shell scripts then commit to git, the more likely it will be, when the next person asking their LLM to write new documentation for another script, to pick us a similar tone to the documentation that was already written, as it'd likely use it as inspiration. But even if it didn't what matters is that the LLM is only here as an intern to help speed up the process and gather the information to write an initial take. It doesn't have to be deterministic.