HIVE-29853: Iceberg CompactionEvaluator double-counts input files - #6737
HIVE-29853: Iceberg CompactionEvaluator double-counts input files#6737Aggarwal-Raghav wants to merge 1 commit into
Conversation
|
CC @difin |
| return false; | ||
| } | ||
|
|
||
| addFiles(); |
There was a problem hiding this comment.
addFiles is here as well. So two times counting same data files. To keep the intention of q files same I have just pumped up more inserts to incease the total file size to be eligibe for compaction. The failinig q files can be checked in run 1
c06e88f to
48e5532
Compare
48e5532 to
e184a60
Compare
e184a60 to
8d00825
Compare
8d00825 to
b0d9bdd
Compare
|
Resolving merge conflicts. |
b0d9bdd to
e4e5395
Compare
|
CC @deniskuzZ , requesting review |
e4e5395 to
9ea1098
Compare
|
rebased to resolve another merge conflict |
|
|
cc @difin |
|
@Aggarwal-Raghav why change existing qfile? it's hard to follow if there is any regression in q.out |
@deniskuzZ , The run1 as I mentioned in initial comment is gone from jenkins. I have attached local q.out diff with just
|






What changes were proposed in this pull request?
Removed the redundant call to
addFiles()from theCompactionEvaluatorconstructor.Why are the changes needed?
The
addFiles()iterates through a table's Iceberg snapshot and adds files to theCommonPartitionEvaluator. Because this was being called in both the constructor and again inisEligibleForCompaction(), every file and byte was double-counted in memory.Does this PR introduce any user-facing change?
Yes. The evaluator will now accurately respect the configured byte sizes and file count thresholds (
compactor.threshold.target.sizeandcompactor.threshold.min.input.files).How was this patch tested?
CI outcome