Concatenating arrays with EmptyArray in the layout hits the python recursion limit #3663
Description
Version of Awkward Array
HEAD
Description and code to reproduce
Concatenating arrays with EmptyArray in the layout hits the python recursion limit.
All the arrays in the reproducer have EmptyArray in their layouts. If the array has Nones through an IndexedOptionArray for example, it is fine.
To reproduce:
import awkward as ak
array = ak.Array([None])
arrays = [array] * 5000
print(ak.concatenate(arrays, axis=0))
# or
import awkward as ak
array = ak.Array([{"x": None}])
arrays = [array] * 5000
print(ak.concatenate(arrays, axis=0))
# or
import awkward as ak
array = ak.Array([{"x": 1, "y": None}])
arrays = [array] * 5000
print(ak.concatenate(arrays, axis=0))is going to hit the following error unless you have a recursion limit higher than 5000 manually set.
Traceback (most recent call last):
File "/Users/iason/Dropbox/work/coffea_dev/awkward/tmp.py", line 5, in <module>
print(ak.concatenate(arrays, axis=0).typestr)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/_dispatch.py", line 41, in dispatch
with OperationErrorContext(name, args, kwargs):
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/_errors.py", line 80, in __exit__
raise self.decorate_exception(exception_type, exception_value)
File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/_dispatch.py", line 67, in dispatch
next(gen_or_result)
~~~~^^^^^^^^^^^^^^^
File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/operations/ak_concatenate.py", line 72, in concatenate
return _impl(arrays, axis, mergebool, highlevel, behavior, attrs)
File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/operations/ak_concatenate.py", line 176, in _impl
contents = [ak._do.mergemany(b) for b in batches]
~~~~~~~~~~~~~~~~^^^
File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/_do.py", line 216, in mergemany
return contents[0]._mergemany(contents[1:])
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/contents/recordarray.py", line 770, in _mergemany
merged = forfield[0]._mergemany(forfield[1:])
File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/contents/indexedoptionarray.py", line 837, in _mergemany
nextcontent = contents[0]._mergemany(tail_contents)
File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/contents/emptyarray.py", line 282, in _mergemany
return others[0]._mergemany(others[1:])
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/contents/emptyarray.py", line 282, in _mergemany
return others[0]._mergemany(others[1:])
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/contents/emptyarray.py", line 282, in _mergemany
return others[0]._mergemany(others[1:])
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
[Previous line repeated 990 more times]
RecursionError: maximum recursion depth exceeded