ParquetReadSupport, ParquetPartitionReaderFactory and Vectorized Parquet Decoding

Author: jaceklaskowski

docs/datasources/parquet/ParquetPartitionReaderFactory.md

@@ -89,7 +89,30 @@ createRowBaseReader(
   file: PartitionedFile): RecordReader[Void, InternalRow]
 ```
 
-`createRowBaseReader` [buildReaderBase](#buildReaderBase) (for the given [PartitionedFile](../PartitionedFile.md) and [createRowBaseParquetReader](#createRowBaseParquetReader)).
+`createRowBaseReader` [buildReaderBase](#buildReaderBase) for the given [PartitionedFile](../PartitionedFile.md) and with [createRowBaseParquetReader](#createRowBaseParquetReader) factory.
+
+### <span id="createRowBaseParquetReader"> createRowBaseParquetReader
+
+```scala
+createRowBaseParquetReader(
+  partitionValues: InternalRow,
+  pushed: Option[FilterPredicate],
+  convertTz: Option[ZoneId],
+  datetimeRebaseSpec: RebaseSpec,
+  int96RebaseSpec: RebaseSpec): RecordReader[Void, InternalRow]
+```
+
+`createRowBaseParquetReader` prints out the following DEBUG message to the logs:
+
+```text
+Falling back to parquet-mr
+```
+
+`createRowBaseParquetReader` creates a [ParquetReadSupport](ParquetReadSupport.md) (with [enableVectorizedReader](ParquetReadSupport.md#enableVectorizedReader) flag disabled).
+
+`createRowBaseParquetReader` creates a [RecordReaderIterator](../RecordReaderIterator.md) with a new `ParquetRecordReader`.
+
+In the end, `createRowBaseParquetReader` returns the `ParquetRecordReader`.
 
 ## <span id="createVectorizedReader"> Creating Vectorized Parquet RecordReader
 
@@ -157,3 +180,16 @@ buildReaderBase[T](
 `buildReaderBase` is used when:
 
 * `ParquetPartitionReaderFactory` is requested to [createRowBaseReader](#createRowBaseReader) and [createVectorizedReader](#createVectorizedReader)
+
+## Logging
+
+Enable `ALL` logging level for `org.apache.spark.sql.execution.datasources.v2.parquet.ParquetPartitionReaderFactory` logger to see what happens inside.
+
+Add the following line to `conf/log4j2.properties`:
+
+```text
+logger.ParquetPartitionReaderFactory.name = org.apache.spark.sql.execution.datasources.v2.parquet.ParquetPartitionReaderFactory
+logger.ParquetPartitionReaderFactory.level = all
+```
+
+Refer to [Logging](../../spark-logging.md).

docs/datasources/parquet/ParquetReadSupport.md

@@ -1,52 +1,48 @@
 # ParquetReadSupport
 
-`ParquetReadSupport` is a concrete `ReadSupport` (from Apache Parquet) of [UnsafeRows](../../UnsafeRow.md).
+`ParquetReadSupport` is a `ReadSupport` (Apache Parquet) of [UnsafeRows](../../UnsafeRow.md) for non-[Vectorized Parquet Decoding](../../vectorized-decoding/index.md).
 
-`ParquetReadSupport` is <<creating-instance, created>> exclusively when `ParquetFileFormat` is requested for a [data reader](ParquetFileFormat.md#buildReaderWithPartitionValues) (with no support for [Vectorized Parquet Decoding](../../vectorized-decoding/index.md) and so falling back to parquet-mr).
+`ParquetReadSupport` is the value of `parquet.read.support.class` Hadoop configuration property for the following:
 
-[[parquet.read.support.class]]
-`ParquetReadSupport` is registered as the fully-qualified class name for [parquet.read.support.class](ParquetFileFormat.md#parquet.read.support.class) Hadoop configuration when `ParquetFileFormat` is requested for a [data reader](ParquetFileFormat.md#buildReaderWithPartitionValues).
+* [ParquetFileFormat](ParquetFileFormat.md#buildReaderWithPartitionValues)
+* [ParquetScan](ParquetScan.md#createReaderFactory)
 
-[[creating-instance]]
-[[convertTz]]
-`ParquetReadSupport` takes an optional Java `TimeZone` to be created.
+## Creating Instance
 
-[[logging]]
-[TIP]
-====
-Enable `ALL` logging level for `org.apache.spark.sql.execution.datasources.parquet.ParquetReadSupport` logger to see what happens inside.
+`ParquetReadSupport` takes the following to be created:
 
-Add the following line to `conf/log4j2.properties`:
+* <span id="convertTz"> `ZoneId` (optional)
+* <span id="enableVectorizedReader"> `enableVectorizedReader`
+* <span id="datetimeRebaseSpec"> DateTime RebaseSpec
+* <span id="int96RebaseSpec"> int96 RebaseSpec
 
-```
-log4j.logger.org.apache.spark.sql.execution.datasources.parquet.ParquetReadSupport=ALL
-```
+`ParquetReadSupport` is created when:
 
-Refer to <<spark-logging.md#, Logging>>.
-====
+* `ParquetFileFormat` is requested to [buildReaderWithPartitionValues](ParquetFileFormat.md#buildReaderWithPartitionValues) (with [enableVectorizedReader](ParquetFileFormat.md#enableVectorizedReader) disabled)
+* `ParquetPartitionReaderFactory` is requested to [createRowBaseParquetReader](ParquetPartitionReaderFactory.md#createRowBaseParquetReader)
 
-=== [[init]] Initializing ReadSupport -- `init` Method
+## Logging
 
-[source, scala]
-----
-init(context: InitContext): ReadContext
-----
+Enable `ALL` logging level for `org.apache.spark.sql.execution.datasources.parquet.ParquetReadSupport` logger to see what happens inside.
 
-NOTE: `init` is part of the `ReadSupport` Contract to...FIXME.
+Add the following line to `conf/log4j2.properties`:
 
-`init`...FIXME
+```text
+logger.ParquetReadSupport.name = org.apache.spark.sql.execution.datasources.parquet.ParquetReadSupport
+logger.ParquetReadSupport.level = all
+```
 
-=== [[prepareForRead]] `prepareForRead` Method
+Refer to [Logging](../../spark-logging.md).
 
-[source, scala]
-----
-prepareForRead(
-  conf: Configuration,
-  keyValueMetaData: JMap[String, String],
-  fileSchema: MessageType,
-  readContext: ReadContext): RecordMaterializer[UnsafeRow]
-----
+<!---
+## Review Me
+
+`ParquetReadSupport` is <<creating-instance, created>> exclusively when `ParquetFileFormat` is requested for a [data reader](ParquetFileFormat.md#buildReaderWithPartitionValues) (with no support for [Vectorized Parquet Decoding](../../vectorized-decoding/index.md) and so falling back to parquet-mr).
 
-NOTE: `prepareForRead` is part of the `ReadSupport` Contract to...FIXME.
+[[parquet.read.support.class]]
+`ParquetReadSupport` is registered as the fully-qualified class name for [parquet.read.support.class](ParquetFileFormat.md#parquet.read.support.class) Hadoop configuration when `ParquetFileFormat` is requested for a [data reader](ParquetFileFormat.md#buildReaderWithPartitionValues).
 
-`prepareForRead`...FIXME
+[[creating-instance]]
+[[convertTz]]
+`ParquetReadSupport` takes an optional Java `TimeZone` to be created.
+-->

docs/datasources/parquet/VectorizedParquetRecordReader.md

@@ -1,6 +1,6 @@
 # VectorizedParquetRecordReader
 
-`VectorizedParquetRecordReader` is a [SpecificParquetRecordReaderBase](SpecificParquetRecordReaderBase.md) for [parquet](index.md) data source for [Vectorized Parquet Decoding](../../vectorized-decoding/index.md).
+`VectorizedParquetRecordReader` is a [SpecificParquetRecordReaderBase](SpecificParquetRecordReaderBase.md) for [Parquet Data Source](index.md) for [Vectorized Parquet Decoding](../../vectorized-decoding/index.md).
 
 ## Creating Instance
 

docs/datasources/parquet/index.md

@@ -11,6 +11,10 @@ Parquet is the default data source format based on the [spark.sql.sources.defaul
 
 Parquet data source uses `spark.sql.parquet` prefix for [parquet-specific configuration properties](../../configuration-properties.md).
 
+## Vectorized Parquet Decoding
+
+Parquet Data Source uses [VectorizedParquetRecordReader](VectorizedParquetRecordReader.md) for [Vectorized Parquet Decoding](../../vectorized-decoding/index.md) (and [ParquetReadSupport](ParquetReadSupport.md) otherwise).
+
 ## Demo
 
 ```scala

docs/vectorized-decoding/index.md

@@ -8,4 +8,4 @@ Quoting [SPARK-12854 Vectorize Parquet reader](https://issues.apache.org/jira/br
 
 Vectorized Parquet Decoding is used exclusively when `ParquetFileFormat` is requested for a [data reader](../datasources/parquet/ParquetFileFormat.md#buildReaderWithPartitionValues) when [spark.sql.parquet.enableVectorizedReader](../configuration-properties.md#spark.sql.parquet.enableVectorizedReader) property is enabled (`true`) and the read schema uses [AtomicTypes](../types/AtomicType.md) data types only.
 
-Vectorized Parquet Decoding uses [VectorizedParquetRecordReader](../datasources/parquet/VectorizedParquetRecordReader.md) for vectorized decoding.
+Vectorized Parquet Decoding uses [VectorizedParquetRecordReader](../datasources/parquet/VectorizedParquetRecordReader.md) for vectorized decoding (and [ParquetReadSupport](../datasources/parquet/ParquetReadSupport.md) otherwise).