Best method to pass dataframe to web worker?

[joshuakoh1]

Hello,

I am trying to pass dataframe operations to a web worker and would like to know the method with the least overhead. I am currently passing {columns: df.columns, values: df.values} but it gets a little messy when passing an array of them.

Appreciate inputs.

Thanks

[flynnplatt]

Probably not of much help at this point (2 years too late) but maybe you have advice for me as I'm looking into doing something similar. My current (unimplemented) approach is to keep all of the dataframe operations on the worker. I load a bunch of data from an external source, load it into a SharedArrayBuffer of some sort and pass that to the worker. The worker then loads it into a DataFrame and then when I run queries on that worker I'll pass a reference to another SharedArrayBuffer and the worker can do all the fun Danfo stuff and put the results into the SharedArrayBuffer for the main thread to pick up.

I wish there was a good way to just pass the DataFrame itself around but unless Danfo switches to using ArrayBuffers under the hood I don't know that there's a way around the data duplication cost in order to enable that