[SPARK-53355][PYTHON][SQL][INFRA] test python udf type behavior

[benrobby]

What changes were proposed in this pull request?

• this captures Python UDF return type coercion behavior in a new test suite. It covers:
  • udf (script from [SPARK-52952][PYTHON] Add PySpark UDF Type Coercion Dev Script #51663)
  • pandas_udf (script from [SPARK-52943][PYTHON] Enable arrow_cast for all pandas UDF eval types #51635)
• Instead of manually regenerating results, we now capture the behavior in automated tests, this will allow us to catch type issues earlier. The tests also adds expected results in golden files.
• in addition to above return type coercion tests, this also adds a new test for input type behavior of udfs. It documents the Python datatypes that UDFs receive.
• also adds a helper script that prints a sensible diff between the tables. Example:

Why are the changes needed?

• it's too easy to miss inadvertent type coercion changes, and some of the existing tables in the codebase have become inconsistent.

Does this PR introduce any user-facing change?

No

How was this patch tested?

• this is a unit test

Was this patch authored or co-authored using generative AI tooling?

Yes, used claude to generate the table printing utils. Generated-by: claude-sonnet-4

[benrobby]

@HyukjinKwon and @asl could you take a look?

[asl3]

cc @zhengruifeng @HyukjinKwon

[asl3]

should we add a comment with context about the null handling importance for input type vs. return type?

[benrobby]

Would you have an example, what comment are you looking for exactly?

[asl3]

optional nit, but i meant we could clarify how the test cases for the return_types and input_types were decided

[HyukjinKwon]

Do we want to run this in CI? If so, we should add __init__.py ,and add this modul into dev/sparktestsupport/modules.py

[benrobby]

yes, let me try to add it. Do we need any other reviewer for the infra changes?

[HyukjinKwon]

My only concern is that I don't think we should say this is the standard behaviours ... as some of behaviours are weird. I am fine with this change as long as we're all on the same page that some behaviours here might change in the future.

[benrobby]

My only concern is that I don't think we should say this is the standard behaviours ... as some of behaviours are weird. I am fine with this change as long as we're all on the same page that some behaviours here might change in the future.

Yes, agreed. I've also updated the readme to reflect that these tests are purely internal and not an API documentation