[GLUTEN-12858][TEST] Unpersist cached DataFrames in flatten test

[sliortega295-ops]

What changes are proposed in this pull request?

Wrap each cached branch of the flatten function test in try/finally and unpersist its DataFrame after the cached assertions complete. The same lifecycle fix is applied to the Spark 3.3, 3.4, 3.5, 4.0, and 4.1 suites.

Previously, each version left three DataFrames cached for the rest of the suite. The updated test keeps at most one of these relations cached and releases it on both success and assertion failure, reducing unnecessary CPU-memory pressure and order sensitivity.

Fixes #12858.

How was this patch tested?

• Spotless passed for all five touched Spark modules
• Spark 3.5 targeted test-compile reactor: 12/12 modules passed, BUILD SUCCESS
• javap confirmed all three normal and exceptional paths call Dataset.unpersist()
• static lifecycle check per version: 3 cache, 3 try, 3 finally, and 3 unpersist calls
• git diff --check: passed

The JNI runtime suite was not run because this checkout has no prebuilt libgluten.so; building the native backend exceeds the available 30 GiB host-memory budget. The repository-wide formatter was also attempted but is currently blocked by a pre-existing formatting error in untouched gluten-iceberg/.../IcebergLocalFilesNode.java; all five touched modules pass their focused Spotless checks.

No GPU was used or required.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

The implementation and this description were generated at the account owner's request. Codex re-checked the final diff against repository code and ran the validations listed above. No separate human line-by-line code review was performed before submission.

[github-actions]

Run Gluten Clickhouse CI on x86