Skip to content

feat(go): add BlobDescriptor batch and stream readers - #761

Merged
JingsongLi merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/go-blob-descriptor-reader
Aug 31, 2026
Merged

feat(go): add BlobDescriptor batch and stream readers#761
JingsongLi merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/go-blob-descriptor-reader

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • expose the merged Rust BlobReader and BlobStream through C and Go
  • add ordered batch reads and lazy io.ReadSeekCloser streams
  • let table.NewBlobReader() reuse the table FileIO, including DLF token refresh

Descriptor parsing, validation, range merging, concurrency limits, and storage I/O remain in Rust core (#762 and #763). The Go layer only manages FFI ownership and Go-friendly APIs.

Go API

reader, err := paimon.NewBlobReader(storageOptions)
reader, err := table.NewBlobReader()

values, err := reader.ReadBlobs(descriptors)
stream, err := reader.OpenBlob(descriptor) // io.ReadSeekCloser

ReadBlob delegates to ReadBlobs. Batch reads merge compatible ranges and preserve input order. Independent streams are lazy and are not merged.

Static options passed to NewBlobReader are not refreshed. table.NewBlobReader() retains the table's refreshing FileIO for DLF temporary data tokens.

Validation

  • cargo test --locked -p paimon-c --lib: 71 passed
  • Go BlobReader/BlobStream runtime tests: 4 passed; local table-fixture case skipped
  • go vet, rustfmt, clippy, dependency report, and diff checks passed

This PR adds only the C/Go binding layer; the Rust core implementation is already merged.

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title feat(go): add standalone BlobDescriptor reader feat(go): add BlobDescriptor batch and stream readers Aug 29, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/go-blob-descriptor-reader branch from 7e7eb31 to 407f4ce Compare August 31, 2026 09:44
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review August 31, 2026 09:45
Comment thread docs/src/go-binding.md Outdated
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/go-blob-descriptor-reader branch from 407f4ce to 132a2fe Compare August 31, 2026 11:22

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 0f6a9a4 into apache:main Aug 31, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants