GH-3708: Inline parquet.thrift

[divjotarora]

Rationale for this change

It is currently not possible to build working POCs for unreleased parquet-format changes in parquet-java because the parquet-format dependency must be updated using a released version. This blocks reference implementations from merging and will become a bigger issue if the current versioning proposal goes through and we start having more "preview" features where writes are expected to support unreleased spec changes behind feature flags.

What changes are included in this PR?

This PR removes the dependency on github.com/apache/parquet-format and instead adds an inlined copy of parquet.thrift (parquet-format-structures/src/main/thrift/parquet.thrift) as well as a "sidecar" metadata file (parquet-format-structures/src/main/thrift/parquet-format.version) to indicate the parquet-format commit that's being inlined. There is a new dev/update-parquet-thrift.sh [$REF] script that takes in a parquet-format ref (commit SHA or tag) and updates the inlined parquet.thrift file to match that version.

Are these changes tested?

Edit inlined parquet.thrift with a meaningless change and then validate it gets overwritten:

-- Validate diff
> git diff
--- a/parquet-format-structures/src/main/thrift/parquet.thrift
+++ b/parquet-format-structures/src/main/thrift/parquet.thrift
...

-- Force an update
> ./dev/update-parquet-thrift.sh apache-parquet-format-2.13.0
Resolving apache-parquet-format-2.13.0 ...
  -> commit: c47e2a66e88943fc46fde1b028a9432f14fdf5c0
Fetching parquet.thrift at c47e2a66e88943fc46fde1b028a9432f14fdf5c0 ...
  -> written to /home/div.arora/parquet-java-inline-thrift/parquet-java/parquet-format-structures/src/main/thrift/parquet.thrift
  -> sidecar updated: /home/div.arora/parquet-java-inline-thrift/parquet-java/parquet-format-structures/src/main/thrift/parquet-format.version

Update complete:
  old commit: c47e2a66e88943fc46fde1b028a9432f14fdf5c0
  new commit: c47e2a66e88943fc46fde1b028a9432f14fdf5c0

-- Validate diff
> git diff
-- empty

-- Update to parquet-format commit containing FILE type
> ./dev/update-parquet-thrift.sh e94a5d090b324a0c0ee1adbb8ea6b099852dc3cc
Fetching parquet.thrift at e94a5d090b324a0c0ee1adbb8ea6b099852dc3cc ...
  -> written to /home/div.arora/parquet-java-inline-thrift/parquet-java/parquet-format-structures/src/main/thrift/parquet.thrift
  -> sidecar updated: /home/div.arora/parquet-java-inline-thrift/parquet-java/parquet-format-structures/src/main/thrift/parquet-format.version

Update complete:
  old commit: c47e2a66e88943fc46fde1b028a9432f14fdf5c0
  new commit: e94a5d090b324a0c0ee1adbb8ea6b099852dc3cc

-- Validate diff
> git diff
--- a/parquet-format-structures/src/main/thrift/parquet-format.version
+++ b/parquet-format-structures/src/main/thrift/parquet-format.version
@@ -1,2 +1,2 @@
 # Provenance of the inlined parquet.thrift. Maintained by dev/update-parquet-thrift.sh.
-parquet-format.commit=c47e2a66e88943fc46fde1b028a9432f14fdf5c0
+parquet-format.commit=e94a5d090b324a0c0ee1adbb8ea6b099852dc3cc
...
diff --git a/parquet-format-structures/src/main/thrift/parquet.thrift b/parquet-format-structures/src/main/thrift/parquet.thrift
...
+struct FileType {
+}
+

Are there any user-facing changes?

No

Closes #3708

[divjotarora]

@rdblue Thanks for the review! You raised a few big points, I'll recap them all here rather than replying to individual comments:

1. Release validation: the release check to ensure parquet-java tracks a released version of parquet-format was a suggestion from the mailing list thread. I audited the C++ and Rust implementations and didn't find a matching check there, so I've removed it.
2. Update script: I've removed the --version argument and now the update script just takes a single ref (either a full commit SHA or a tag) and it just downloads the corresponding parquet.thrift. I've also removed the retry logic.
3. Sidecar file: the parquet-format.version sidecar file now only stores a commit SHA, not a semantic version. In the case that the update script is invoked with a tag ref, it's first resolved to the corresponding SHA and that's stored in the sidecar.

[Fokko]

Thanks @divjotarora for working on this. I'm wondering if it wouldn't be easier to pull the structures from the snapshots repository: https://repository.apache.org/content/groups/snapshots/org/apache/parquet/parquet-format-structures/. It looks like it is missing some versions, but I think it would make sense to just take the snapshot from there and avoid a shell script.