fix: find_in_set internal error with two LargeUtf8 scalar arguments - #24802
Open
advitrocks9 wants to merge 2 commits into
Open
fix: find_in_set internal error with two LargeUtf8 scalar arguments#24802advitrocks9 wants to merge 2 commits into
advitrocks9 wants to merge 2 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24802 +/- ##
==========================================
- Coverage 81.53% 81.53% -0.01%
==========================================
Files 1123 1123
Lines 406042 406106 +64
Branches 406042 406106 +64
==========================================
+ Hits 331059 331108 +49
- Misses 55621 55633 +12
- Partials 19362 19365 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
find_in_set.sltis one of the porting stubs tracked by [EPIC] Completedatafusion-sparkSpark Compatible Functions #15914 and the bug fell out of filling it. I can open one if the changelog entry needs a link.Rationale for this change
find_in_setwith twoLargeUtf8scalar arguments does not return a value, it trips the planner's type assertion:return_typegoes throughutf8_to_int_type, which widens toInt64forLargeUtf8. The three array branches are generic overInt32Type/Int64Typeand honour that. The(Scalar, Scalar)branch computedSome(position as i32)and passed it toScalarValue::from, which is alwaysScalarValue::Int32, so it was the one branch that could contradict its ownreturn_type. Nothing had executed it:string_literal.sltcovers scalar/scalar forUtf8andUtf8Viewonly, and theLargeUtf8block instring_query.slt.partcasts columns, so they land in the array branches.What changes are included in this PR?
return_fieldpromised, with the samematchon the return type the array branches already use four times.spark/string/find_in_set.slt, previously a stub. Expected values come from Spark 4.2.0, read offUTF8String.findInSetand confirmed onpyspark==4.2.0, not from running DataFusion.The
.sltis the larger half of the diff at 37 queries. Coverage that exists nowhere else in the tree: a needle containing a comma, empty and consecutive and trailing-comma elements, no trimming around delimiters, first occurrence wins on duplicates, and a 20-element list that crossesFIND_IN_SET_LOOKUP_THRESHOLDso the lookup path from #23460 gets its first end-to-end test. DataFusion matches Spark on all 37.Are these changes tested?
Yes, the
LargeUtf8query above is the regression test. Reverting the source change and rebuilding gives exactly one failure, on that query, with the assertion above.I left
string/string_literal.sltalone. The new case runs in the same CI job, so a second copy next to theUtf8block there catches nothing extra.Are there any user-facing changes?
Yes.
find_in_setover twoLargeUtf8scalars returns a value instead of raising an internal error. No API change.