Skip to content

[iceberg] Leave the nanosecond refusal to the validator that knows the table - #9503

Merged
JingsongLi merged 1 commit into
apache:masterfrom
plusplusjiajia:fix/iceberg-ns-type-mapping-scope
Sep 1, 2026
Merged

[iceberg] Leave the nanosecond refusal to the validator that knows the table#9503
JingsongLi merged 1 commit into
apache:masterfrom
plusplusjiajia:fix/iceberg-ns-type-mapping-scope

Conversation

@plusplusjiajia

Copy link
Copy Markdown
Member

Purpose

Follow-up to #9455, which was right that the Iceberg mirror must not publish nanosecond timestamps — Paimon writes them as Parquet INT96, which Iceberg reads as microseconds. It added two defences, and I think
only one belongs.

SchemaValidation#validateIcebergTimestampPrecisions is the right one: it refuses only when the table has Iceberg metadata enabled, and is registered by the same condition as the Iceberg publisher itself, so
no table that can publish is missing it.

The other narrowed IcebergDataField#toTypeString from 3..9 to 3..6. That method only names the Iceberg type for a Paimon type — it has no table to ask, so it also refuses callers that never let Paimon write
the data, for whom the INT96 hazard does not arise. It also deleted the mapping of 7..9 to the v3 types timestamp_ns and timestamptz_ns, which toDatatype still reads back as precision 9, so the class has
been claiming to read two names it stopped writing.

Whether nanoseconds may be published is a property of the table, not of the type. This restores the mapping and leaves the refusal to the validator.

TIME has the same shape but is left alone: Iceberg has no precision-carrying time type and the reverse mapping reads time back as precision 3, so widening it there would trade a loud error for a silent
precision loss.

Tests

The three classes #9455 touched all pass — IcebergCompatibilityTest 48, SchemaManagerTest 57, IcebergDataFieldTest 31.

IcebergDataFieldTest again requires 7..9 to map to the v3 names. IcebergCompatibilityTest's commit-path refusal test now runs from a MethodSource and additionally covers
TIMESTAMP_WITH_LOCAL_TIME_ZONE(9) — the family this restores a mapping for, and previously uncovered there. It still passes, i.e. the mirror still refuses it.

@plusplusjiajia plusplusjiajia reopened this Sep 1, 2026
@plusplusjiajia
plusplusjiajia marked this pull request as ready for review September 1, 2026 00:17
@plusplusjiajia plusplusjiajia reopened this Sep 1, 2026
@plusplusjiajia plusplusjiajia reopened this Sep 1, 2026
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit da71a66 into apache:master Sep 1, 2026
38 of 52 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants