infer_rel_schema's read branch takes rel.read.base_schema.struct and never reads rel.read.projection (type_inference.py:698). The word projection does not appear anywhere in that file, on main either, so a masked read infers the columns the mask left out.
Read Operator, Direct Output Order: "Defaults to the schema of the data read after the optional projection (masked complex expression) is applied."
Reproduced with 0.31.0:
import substrait.algebra_pb2 as stalg
import substrait.type_pb2 as stt
from substrait.type_inference import infer_rel_schema
i64 = stt.Type(i64=stt.Type.I64(nullability=stt.Type.NULLABILITY_REQUIRED))
rel = stalg.Rel(read=stalg.ReadRel(
base_schema=stt.NamedStruct(names=["a", "b", "c"],
struct=stt.Type.Struct(types=[i64, i64, i64],
nullability=stt.Type.NULLABILITY_REQUIRED)),
projection=stalg.Expression.MaskExpression(
select=stalg.Expression.MaskExpression.StructSelect(struct_items=[
stalg.Expression.MaskExpression.StructItem(field=2)]))))
print(len(infer_rel_schema(rel).types))
# actual: 3
# expected: 1, the one column the mask selectsForeign plans only, as far as I can see: nothing in builders/ or dataframe/ sets ReadRel.projection, so this is reachable the same way #217 was rather than through the library's own API.
The narrowing would apply to base_schema.names as well as to struct.types, and it belongs before common/emit, whose indices are defined over the post-projection order. One scope question worth settling first: the repro uses a flat StructSelect, and masks also nest through struct_items.child, so it is worth deciding whether the fix covers the nested form now or later. Happy to send the PR.
infer_rel_schema's read branch takesrel.read.base_schema.structand never readsrel.read.projection(type_inference.py:698). The wordprojectiondoes not appear anywhere in that file, onmaineither, so a masked read infers the columns the mask left out.Read Operator, Direct Output Order: "Defaults to the schema of the data read after the optional projection (masked complex expression) is applied."
Reproduced with 0.31.0:
Foreign plans only, as far as I can see: nothing in
builders/ordataframe/setsReadRel.projection, so this is reachable the same way #217 was rather than through the library's own API.The narrowing would apply to
base_schema.namesas well as tostruct.types, and it belongs beforecommon/emit, whose indices are defined over the post-projection order. One scope question worth settling first: the repro uses a flatStructSelect, and masks also nest throughstruct_items.child, so it is worth deciding whether the fix covers the nested form now or later. Happy to send the PR.