Fix PathNotFound error in msexports ETL ingestion on first run

[Michael Flanakin (flanakin)]

🛠️ Description

The "Get Existing Parquet Files" GetMetadata activity fails with PathNotFound when the destination folder doesn't exist yet (e.g. first ingestion for a scope/dataset/month combination). Reservation recommendation exports hit this most often because their destination path adds an extra exportName segment, but any export type can hit it on a first run.

Fixed in both places this pattern exists: msexports_ETL_ingestion (Microsoft.CostManagement/Exports/app.bicep) and the analytics ingestion ETL pipeline (Microsoft.FinOpsHubs/Analytics/app.bicep). The two call sites aren't failing for the same reason, though: Exports writes to a destination folder that legitimately may not exist yet on first run, so this is a genuine fix there. Analytics reads a source folder that a manifest file just landed in, so that folder exists by construction whenever the pipeline runs — the change there is diagnostic, replacing an opaque PathNotFound ADLS error with the pipeline's own IngestionFilesNotFound error code if the folder is ever missing for some other reason (e.g. a malformed folderPath). It does not make a missing ingestion folder a no-op.

Fix: add 'exists' to the GetMetadata activity's fieldList. Per ADF's GetMetadata docs, specifying exists makes the activity return exists: false instead of throwing when the path is missing — "If exists isn't specified in the field list, the Get Metadata activity fails if the object isn't found." The downstream Filter activity's items expression now checks .output.exists before reading .output.childItems, so a missing folder resolves to an empty list instead of propagating the failure.

This is the same idiom already used elsewhere in this codebase for "path may not exist yet" scenarios: Check Schema in Exports/app.bicep, and both GetMetadata activities in IngestionQueries/app.bicep (one of which — "Get Existing Parquet Files" — already combines exists + childItems exactly like this fix does).

Root cause verification (re: issue #2088 comment)

A prior comment on the issue did solid root-cause analysis but its line numbers had drifted and its proposed fix options were speculative (custom error handling / parent-folder existence checks / wrapping in an If Condition). Re-verified against the current code:

• Microsoft.CostManagement/Exports/app.bicep: GetMetadata activity is at lines 1090-1122, Filter Out Current Exports at 1123-1147 (comment's estimate of ~1090-1122 / ~1123-1147 was accurate).
• Microsoft.FinOpsHubs/Analytics/app.bicep: identical vulnerable pattern confirmed at lines 1668-1707 (comment said ~1667-1706, essentially correct).
• The actual cleanest fix is ADF's documented, built-in behavior for this exact case (exists in fieldList), not a custom If Condition or Web-activity workaround — and this idiom was already present elsewhere in the repo, so this change makes the two vulnerable call sites consistent with the rest of the codebase rather than introducing a new pattern.

Fixes #2088

📷 Screenshots

Not applicable — backend ETL pipeline fix, no UI changes.

📋 Checklist

🔬 How did you test this change?

• 🤏 Lint tests
• 🤞 PS -WhatIf / az validate
• 👍 Manually deployed + verified
• 💪 Unit tests
• 🙌 Integration tests

• bicep build src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/app.bicep --stdout — builds cleanly
• bicep build src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/app.bicep --stdout — builds cleanly
• Verified compiled ARM JSON includes "exists" in both fieldList arrays
• pwsh -Command "./src/scripts/Test-PowerShell.ps1 -Lint" — 3418/3418 passed
• Manual end-to-end validation against a real deployment with no prior export data not yet performed — requesting automated CI deployment below

📦 Deploy to test?

• Hubs + ADX (managed)
• Hubs + Fabric (manual) — URI:
• Hubs (manual)
• Hubs (no data)
• Workbooks
• Alerts

🙋‍♀️ Do any of the following that apply?

• 🚨 This is a breaking change.
• 🤏 The change is less than 20 lines of code.

📑 Did you update docs/changelog.md?

• ✅ Updated changelog (required for dev PRs)
• ➡️ Will add log in a future PR (feature branch PRs only)
• ❎ Log not needed (small/internal change)

📖 Did you update documentation?

• ✅ Public docs in docs (required for dev)
• ✅ Public docs in docs-mslearn (required for dev)
• ✅ Internal dev docs in docs-wiki (required for dev)
• ✅ Internal dev docs in src (required for dev)
• ➡️ Will add docs in a future PR (feature branch PRs only)
• ❎ Docs not needed (small/internal change)

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Roland Krummenacher (RolandKrummenacher)]

Re-reviewed at 73a0076. Both of my threads are settled — the dependency-condition change is a better fix than what I suggested, and the Analytics hunk holds up once you account for how the pipeline is triggered. Details in the threads.

Net change against dev is +5/-3 across two files, and the three Get Existing Parquet Files call sites (Exports, Analytics, IngestionQueries) now use an identical Succeeded + exists + contains shape, which is worth more than the individual hunks.

One non-blocking ask left in the second thread: the description implies both call sites were failing the same way, and they weren't — Exports genuinely hits a missing destination folder on first run, Analytics doesn't. A sentence there would save the next reader the trace. Approving regardless.

[github-actions]

Hubs (no data) deployment failed. View logs.

[Michael Flanakin (flanakin)]

Holding for validation

[github-actions]

Hubs + ADX (managed) deployment failed. View logs.

[github-actions]

Hubs (manual) deployment failed. View logs.

[github-actions]

Hubs + ADX (managed) deployment failed. View logs.

[github-actions]

Hubs (manual) deployment failed. View logs.