DRAFT: Extension types #451
Conversation
pitrou
force-pushed
the
extension_types
branch
from
8b0da72 to
042a207
Compare
| When a reader encounters an extension type in a Parquet schema, it should try | ||
| to match it by name to its known extension types. If it does not recognize | ||
| the extension type, then it should read it as the underlying physical type | ||
| and should not try to interpret the column's statistics. It may however |
There was a problem hiding this comment.
min/max statistics, others should be valid?
|
Generally seems reasonable to me. |
| When a reader encounters an extension type in a Parquet schema, it should try | ||
| to match it by name to its known extension types. If it does not recognize | ||
| the extension type, then it should read it as the underlying physical type | ||
| and should not try to interpret the column's statistics. It may however |
| * | ||
| * If the extension type is not parametric, then `serialization` is empty. | ||
| */ | ||
| struct ExtensionTypeDescription { |
There was a problem hiding this comment.
Why choosing a dedicated ExtensionTypeDescription struct over list<KeyValue>? I'm afraid that a binary typed field may incur misuse from the users.
There was a problem hiding this comment.
What would the list<KeyValue> contain and where would it reside? I'm not following you.
There was a problem hiding this comment.
struct ExtensionTypeDescription {
1: optional list<KeyValue> metadata
}
And specify the required keys for each extension type, pretty much like what Arrow does.
There was a problem hiding this comment.
This does not make sense, does it? The keys will always be the same, so why not reify them in the Thrift spec as the PR currently does?
There was a problem hiding this comment.
Or are you thinking about extension-specific parameter keys as in https://arrow.apache.org/docs/dev/format/CanonicalExtensions.html ?
There was a problem hiding this comment.
Note we would still need the extension name, so this would be:
struct ExtensionTypeDescription {
1: required string name
2: optional list<KeyValue> parameters
}There was a problem hiding this comment.
Or are you thinking about extension-specific parameter keys as in https://arrow.apache.org/docs/dev/format/CanonicalExtensions.html ?
Yes, I mean something like this.
|
What makes something appropriate as an extension type rather than a basic supported type? GEOMETRY is currently supported natively, which seems to be simply a binary blob with a metadata CRS string. Is the problem that adding such types is cumbersome? |
Yes, mainly the fact that adding new types is cumbersome, so there is a trade-off between how useful we expect the type to be to the broader community. The other questions which comes into play are things like if stats are important, how well an extension type would work in this context (I forget if this design addresses that issue). IMO, GEOMETRY might have been considered for an extension type if we had this facility. As a datapiont, in the Arrow project most new types have been added as extension types I believe. |
|
Are you just providing a name for an introduced type? The examples don't show using any special handling -- IP address as FIXED_LEN_BYTE_ARRAY(16) and f64tensor as JSON -- and there is no example to help explain leaf vs. non-leaf handling. Is there some more complex vision? If so an example would be helpful. |
I think this needs to be fleshed out some more for leaf/non-leaf and possible custom statistics for aggregates types (e.g. point cloud data). I don't think f64tensor is supposed to be JSON, just its metadata. |
4 participants
Rationale for this change
What changes are included in this PR?
Do these changes have PoC implementations?