Improve LLM agent readability of the docs site

[aivascu]

Summary

• Add /llms.txt and /robots.txt, plus build-time /sitemap.xml and /docs-markdown/** mirrors so agents can discover and fetch guide content as markdown.
• Wire prepare-agent-assets into pregenerate, just, and CI; keep generated agent assets gitignored.
• Preserve C# generics in API HTML and unescape DocFX \> / \(/\) so signatures read as Create<T>() in both HTML and /api-markdown.

Test plan

• CI build passes (prepare-api + prepare-agent-assets + generate)
• After deploy: /llms.txt, /robots.txt, /sitemap.xml return 200
• /docs-markdown/get-started/introduction.md matches /docs/get-started/introduction
• /api-markdown/autofixture/5-0-0-rc-1/autofixture.fixture.md has Build<T>() without backslashes
• Fixture API HTML heading shows Build<T>() (not stripped / not T\>)