Vector datatype support (#2634)

* Vector feature extension

* Added initial skeleton code for VECTOR support in JDBC

* Added Vector support and strategy in DataTypes and dtv

* Added changes which got removed from previous commit

* Added support to read vector object using getObject

* Updated read API for Vector support and created custom Vector class to decode/encode data, along with additional features like dimensionType, and dimensionCount.

* Updated getObject() API to skip 8-byte header in Vector class and improved constructor parameter positioning for clarity, removed VectorInputStream

* Added support for setVector() API, updated setObject() to handle dimensionCount for vector, fixed datatype and tdsType mappings along with setting correct typeDefinition in Parameter class.

* Updated the setObject API to support vector insertion and fixed RPC Vector handling to ensure insertion of vector data.

* Validated insert and read for sample and max-length (1998) vector data, and measured performance for max-length vectors at varying scales — 100, 1K, 10K, 100K, and 1M records.

* Updated insert logic to make batch size configurable for single-batch performance testing.

* Updated writeRPCVector to correctly write data to server and handle null insertion correctly, added test case to validate null insertion and test vectorsupport feature enabled. Removed getVector, setVector and updated existing setObject() along with adding helper functions in Vector class.

* Stored Procedure support for VECTOR Type using registerOutParameter, getObject, and setObject APIs.

* Added support for SQLServerBulkCopy, prepared statement using useBulkCopyForBatchInsert and added support for vector datatype in SQLServerBulkCSVFileRecord along with updating strategy for Vector type in dtv.java

* Updated to use scale as dimensionType and precision as dimensionCount for read, insert, bulkCopy, bulkCopyCSV operation and corrected typeInfo and Parameter.java along with adding test  cases for BulkCopy

* Added Table-Valued Parameter (TVP) support for the VECTOR type, validated insertion and retrieval of VECTOR data, and updated stored procedure and TVP definitions to use scale as dimensionType and precision as dimensionCount.

* Added performance testing tests for bulkCopy and insert using useBulkCopyForBatchInsert along with updating getObject() when data is null

* Added test case to validate vector as input and output in user-defined functions (UDFs); fixed previously failing test case.

* Added vector as non-searchable and Validated select*into, UDF and SVF by returning normalised vector, global and local temp tables

* Added test to validate transaction rollback, database view and Metadata information for vector data type

* Validate bulk copy scenarios with source and destination as -vector & varbinary combinations -vector and varchar combinations

* POC-BulkCopy usign CSV file and added tests to check different scenarios along with adding test for setObject(getObject())

* Added test scenarios to validate failures when insert and bulk copy operation is performed for mismatched source and destination tables

* Addressed comments, Updated dimensionType naming similar to server side, added functions to getScale, bytesperDimension and updated the logic to use functions instead of handling hardcoded values

* Added support to construct vector object using scaleByte

* Updated logic to support initialising vector object using precision and scale value also

* Updated convertStreamToObject to handle backward compatibility and updated toString() for Vector to return json formatted string

* handled backward compatibility by throwing error if getString() is used

* Refactored Vector class to use Object[] for vector data storage  and serialization instead of float[] and updated the test cases.

* Updated toString() method for vector and addressed test failures

* Added stored procedure support for null VECTOR type and validated registerOutParameter, getObject, and setObject APIs.

* Addressed comments

* Added vectorTypeSupport connection string property {off,v1}

* Addressed comments

* Removed getter, setter and updater conversion

* Updated getter, setter and updater conversion in DataTypes class

* Added xVectorType exclusion tag for tests using vector types.

* Updated tag name to xSQLv25 to reflect support for vector type introduced in SQL Server 2025 onwards

* Updated tag name to xVectorType

* Updated tag name to vectorTest.

* Updated failures in RequestBoundaryMethodsTest class

* Addressed comments

* Addressed comments and failures in ADO pipeline

* Updated setObject() for CallableStatement

* Addressed comments

* Updated build.gradle file to exclude vectorTest tag

* Updated Vector class

* Refactored Vector class and added VectorUtils.java to include helper functions

* Updated Vector class and VectorUtils class

* Added test scenarios

* git merge main and updated public pipeline to exclude vectorTest

* Refactored vector class to use FLOAT32 and addressed comments

* Addressed comments

Author: muskan124947

build.gradle

@@ -29,7 +29,7 @@ allprojects {
 
 test {
     useJUnitPlatform {
-        excludeTags (hasProperty('excludedGroups') ? excludedGroups : 'xSQLv15','xGradle','reqExternalSetup','NTLM','MSI','clientCertAuth','fedAuth','kerberos')
+        excludeTags (hasProperty('excludedGroups') ? excludedGroups : 'xSQLv15','xGradle','reqExternalSetup','NTLM','MSI','clientCertAuth','fedAuth','kerberos','vectorTest')
     }
 }
 
@@ -44,6 +44,11 @@ if (!hasProperty('buildProfile') || (hasProperty('buildProfile') && buildProfile
 	}
 	sourceCompatibility = 23
 	targetCompatibility = 23
+	test {
+    useJUnitPlatform {
+        excludeTags(hasProperty('excludedGroups') ? excludedGroups : 'vectorTest')
+    }
+}
 }
 
 if (hasProperty('buildProfile') && buildProfile == "jre21") {
@@ -57,6 +62,11 @@ if (hasProperty('buildProfile') && buildProfile == "jre21") {
 	}
 	sourceCompatibility = 21
 	targetCompatibility = 21
+	test {
+    useJUnitPlatform {
+        excludeTags(hasProperty('excludedGroups') ? excludedGroups : 'vectorTest')
+    }
+}
 }
 
 if (hasProperty('buildProfile') && buildProfile == "jre17") {
@@ -70,6 +80,11 @@ if (hasProperty('buildProfile') && buildProfile == "jre17") {
 	}
 	sourceCompatibility = 17
 	targetCompatibility = 17
+	test {
+    useJUnitPlatform {
+        excludeTags(hasProperty('excludedGroups') ? excludedGroups : 'vectorTest')
+    }
+}
 }
 
 if (hasProperty('buildProfile') && buildProfile == "jre11") {
@@ -83,6 +98,11 @@ if (hasProperty('buildProfile') && buildProfile == "jre11") {
 	}
 	sourceCompatibility = 11
 	targetCompatibility = 11
+	test {
+    useJUnitPlatform {
+        excludeTags(hasProperty('excludedGroups') ? excludedGroups : 'vectorTest')
+    }
+}
 }
 
 if(hasProperty('buildProfile') && buildProfile == "jre8") {
@@ -94,7 +114,7 @@ if(hasProperty('buildProfile') && buildProfile == "jre8") {
 	targetCompatibility = 1.8
 	test {
 		useJUnitPlatform {
-			excludeTags (hasProperty('excludedGroups') ? excludedGroups : 'xSQLv15','xGradle','NTLM','reqExternalSetup','MSI','clientCertAuth','fedAuth','xJDBC42')
+			excludeTags (hasProperty('excludedGroups') ? excludedGroups : 'xSQLv15','xGradle','NTLM','reqExternalSetup','MSI','clientCertAuth','fedAuth','xJDBC42','vectorTest')
 		}
 	}
 }

pom.xml

@@ -46,10 +46,11 @@
 			kerberos  - - - - - For tests using Kerberos authentication (excluded by default)
 			reqExternalSetup - For tests requiring external setup (excluded by default)
 			clientCertAuth - - For tests requiring client certificate authentication 
+			vectorTest - - For tests using vector data types (excluded by default)
 			setup (excluded by default) - - - - - - - - - - - - - - - - - - - - - - - 
 			- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
 			Default testing enabled with SQL Server 2019 (SQLv15) -->
-		<excludedGroups>xSQLv12,xSQLv15,NTLM,MSI,reqExternalSetup,clientCertAuth,fedAuth,kerberos</excludedGroups>
+		<excludedGroups>xSQLv12,xSQLv15,NTLM,MSI,reqExternalSetup,clientCertAuth,fedAuth,kerberos,vectorTest</excludedGroups>
 		<!-- Use -preview for preview release, leave empty for official release. -->
 		<releaseExt></releaseExt>
 		<!-- Driver Dependencies -->

src/main/java/com/microsoft/sqlserver/jdbc/Column.java

@@ -8,7 +8,6 @@
 import java.text.MessageFormat;
 import java.util.Calendar;
 
-
 /**
  * Column represents a database column definition (meta data) within a result set.
  */
@@ -345,7 +344,7 @@ else if (jdbcType.isBinary()) {
 
         // Update of binary SSType from textual JDBCType: Convert hex to binary.
         else if ((SSType.BINARY == ssType || SSType.VARBINARY == ssType || SSType.VARBINARYMAX == ssType
-                || SSType.IMAGE == ssType || SSType.UDT == ssType) &&
+                || SSType.IMAGE == ssType || SSType.UDT == ssType || SSType.VECTOR == ssType) &&
 
                 (JDBCType.CHAR == jdbcType || JDBCType.VARCHAR == jdbcType || JDBCType.LONGVARCHAR == jdbcType)) {
             jdbcType = JDBCType.VARBINARY;

src/main/java/com/microsoft/sqlserver/jdbc/DDC.java

@@ -726,6 +726,7 @@ static final Object convertStreamToObject(BaseInputStream stream, TypeInfo typeI
                 case VARBINARY:
                 case LONGVARBINARY:
                 case BLOB:
+                case VECTOR:
                     // Where allowed, streams convert directly to binary representation
                     if (StreamType.BINARY == getterArgs.streamType)
                         return stream;
@@ -798,6 +799,10 @@ static final Object convertStreamToObject(BaseInputStream stream, TypeInfo typeI
                             return new StringReader(new String(stream.getBytes(), typeInfo.getCharset()));
                     }
 
+                    if (typeInfo.getSSType() == SSType.VECTOR) {
+                        throw new SQLServerException(SQLServerException.getErrString("R_getStringNotSupportedForVector"), null, 0, null);
+                    }
+
                     // None of the special/fast textual conversion cases applied. Just go the normal route of converting
                     // via String.
                     return convertStringToObject(new String(stream.getBytes(), typeInfo.getCharset()),

src/main/java/com/microsoft/sqlserver/jdbc/DataTypes.java

@@ -66,6 +66,7 @@ enum TDSType {
     NTEXT(0x63), // 99
     UDT(0xF0), // -16
     XML(0xF1), // -15
+    VECTOR(0xF5), // 245
 
     // LONGLEN types
     SQL_VARIANT(0x62); // 98
@@ -149,7 +150,8 @@ enum SSType {
     XML(Category.XML, "xml", JDBCType.LONGNVARCHAR),
     TIMESTAMP(Category.TIMESTAMP, "timestamp", JDBCType.BINARY),
     GEOMETRY(Category.UDT, "geometry", JDBCType.GEOMETRY),
-    GEOGRAPHY(Category.UDT, "geography", JDBCType.GEOGRAPHY);
+    GEOGRAPHY(Category.UDT, "geography", JDBCType.GEOGRAPHY),
+    VECTOR(Category.VECTOR, "vector", JDBCType.VECTOR);
 
     final Category category;
     private final String name;
@@ -205,7 +207,8 @@ enum Category {
         TIMESTAMP,
         UDT,
         SQL_VARIANT,
-        XML;
+        XML,
+        VECTOR;
 
         private static final Category[] VALUES = values();
     }
@@ -267,7 +270,9 @@ enum GetterConversion {
 
         SQL_VARIANT(SSType.Category.SQL_VARIANT, EnumSet.of(JDBCType.Category.CHARACTER, JDBCType.Category.SQL_VARIANT,
                 JDBCType.Category.NUMERIC, JDBCType.Category.DATE, JDBCType.Category.TIME, JDBCType.Category.BINARY,
-                JDBCType.Category.TIMESTAMP, JDBCType.Category.NCHARACTER, JDBCType.Category.GUID));
+                JDBCType.Category.TIMESTAMP, JDBCType.Category.NCHARACTER, JDBCType.Category.GUID)),
+
+        VECTOR(SSType.Category.VECTOR, EnumSet.of(JDBCType.Category.VECTOR));    
 
         private final SSType.Category from;
         private final EnumSet<JDBCType.Category> to;
@@ -454,7 +459,9 @@ JDBCType getJDBCType(SSType ssType, JDBCType jdbcTypeFromApp) {
                     case NTEXT:
                         jdbcType = JDBCType.LONGVARCHAR;
                         break;
-
+                    case VECTOR:
+                        jdbcType = JDBCType.VECTOR;
+                        break;
                     case XML:
                     default:
                         jdbcType = JDBCType.LONGVARBINARY;
@@ -478,6 +485,7 @@ JDBCType getJDBCType(SSType ssType, JDBCType jdbcTypeFromApp) {
     READER(Reader.class, JDBCType.LONGVARCHAR),
     // Note: Only SQLServerSQLXML SQLXML instances are accepted by this driver
     SQLXML(SQLServerSQLXML.class, JDBCType.SQLXML),
+    VECTOR(microsoft.sql.Vector.class, JDBCType.VECTOR),
     OBJECT(Object.class, JDBCType.UNKNOWN);
 
     private final Class<?> javaClass;
@@ -677,7 +685,8 @@ enum JDBCType {
     SQL_VARIANT(Category.SQL_VARIANT, microsoft.sql.Types.SQL_VARIANT, Object.class.getName()),
     GEOMETRY(Category.GEOMETRY, microsoft.sql.Types.GEOMETRY, Object.class.getName()),
     GEOGRAPHY(Category.GEOGRAPHY, microsoft.sql.Types.GEOGRAPHY, Object.class.getName()),
-    LOCALDATETIME(Category.TIMESTAMP, java.sql.Types.TIMESTAMP, LocalDateTime.class.getName());
+    LOCALDATETIME(Category.TIMESTAMP, java.sql.Types.TIMESTAMP, LocalDateTime.class.getName()),
+    VECTOR(Category.VECTOR, microsoft.sql.Types.VECTOR, microsoft.sql.Vector.class.getName());
 
     final Category category;
     private final int intValue;
@@ -726,7 +735,8 @@ enum Category {
         GUID,
         SQL_VARIANT,
         GEOMETRY,
-        GEOGRAPHY;
+        GEOGRAPHY,
+        VECTOR;
 
         private static final Category[] VALUES = values();
     }
@@ -799,7 +809,9 @@ enum SetterConversion {
 
         GEOMETRY(JDBCType.Category.GEOMETRY, EnumSet.of(JDBCType.Category.GEOMETRY)),
 
-        GEOGRAPHY(JDBCType.Category.GEOGRAPHY, EnumSet.of(JDBCType.Category.GEOGRAPHY));
+        GEOGRAPHY(JDBCType.Category.GEOGRAPHY, EnumSet.of(JDBCType.Category.GEOGRAPHY)),
+
+        VECTOR(JDBCType.Category.VECTOR, EnumSet.of(JDBCType.Category.VECTOR));
 
         private final JDBCType.Category from;
         private final EnumSet<JDBCType.Category> to;
@@ -836,11 +848,11 @@ enum UpdaterConversion {
                 SSType.Category.DATETIMEOFFSET, SSType.Category.CHARACTER, SSType.Category.LONG_CHARACTER,
                 SSType.Category.NCHARACTER, SSType.Category.LONG_NCHARACTER, SSType.Category.XML,
                 SSType.Category.BINARY, SSType.Category.LONG_BINARY, SSType.Category.UDT, SSType.Category.GUID,
-                SSType.Category.TIMESTAMP, SSType.Category.SQL_VARIANT)),
+                SSType.Category.TIMESTAMP, SSType.Category.SQL_VARIANT, SSType.Category.VECTOR)),
 
         LONG_CHARACTER(JDBCType.Category.LONG_CHARACTER, EnumSet.of(SSType.Category.CHARACTER,
                 SSType.Category.LONG_CHARACTER, SSType.Category.NCHARACTER, SSType.Category.LONG_NCHARACTER,
-                SSType.Category.XML, SSType.Category.BINARY, SSType.Category.LONG_BINARY)),
+                SSType.Category.XML, SSType.Category.BINARY, SSType.Category.LONG_BINARY, SSType.Category.VECTOR)),
 
         CLOB(JDBCType.Category.CLOB, EnumSet.of(SSType.Category.LONG_CHARACTER, SSType.Category.LONG_NCHARACTER,
                 SSType.Category.XML)),
@@ -899,7 +911,10 @@ enum UpdaterConversion {
                 SSType.Category.DATETIMEOFFSET, SSType.Category.CHARACTER, SSType.Category.LONG_CHARACTER,
                 SSType.Category.NCHARACTER, SSType.Category.LONG_NCHARACTER)),
 
-        SQL_VARIANT(JDBCType.Category.SQL_VARIANT, EnumSet.of(SSType.Category.SQL_VARIANT));
+        SQL_VARIANT(JDBCType.Category.SQL_VARIANT, EnumSet.of(SSType.Category.SQL_VARIANT)),
+
+        VECTOR(JDBCType.Category.VECTOR, EnumSet.of(SSType.Category.CHARACTER, SSType.Category.LONG_CHARACTER, 
+                SSType.Category.VECTOR));
 
         private final JDBCType.Category from;
         private final EnumSet<SSType.Category> to;
@@ -1002,6 +1017,8 @@ int asJavaSqlType() {
                 case NVARCHAR:
                 case SQLXML:
                     return java.sql.Types.VARCHAR;
+                case VECTOR:
+                    return microsoft.sql.Types.VECTOR;
                 case LONGNVARCHAR:
                     return java.sql.Types.LONGVARCHAR;
                 case NCLOB:

src/main/java/com/microsoft/sqlserver/jdbc/IOBuffer.java

@@ -82,6 +82,7 @@
 import com.microsoft.sqlserver.jdbc.SQLServerConnection.FedAuthTokenCommand;
 import com.microsoft.sqlserver.jdbc.dataclassification.SensitivityClassification;
 
+import microsoft.sql.Vector;
 
 final class TDS {
     // application protocol
@@ -169,6 +170,11 @@ final class TDS {
     static final byte TDS_FEATURE_EXT_AZURESQLDNSCACHING = 0x0B;
     static final byte TDS_FEATURE_EXT_SESSIONRECOVERY = 0x01;
 
+    // Vector support
+    static final byte TDS_FEATURE_EXT_VECTORSUPPORT = 0x0E;
+    static final byte VECTORSUPPORT_NOT_SUPPORTED = 0x00;
+    static final byte MAX_VECTORSUPPORT_VERSION = 0x01;
+
     static final int TDS_TVP = 0xF3;
     static final int TVP_ROW = 0x01;
     static final int TVP_NULL_TOKEN = 0xFFFF;
@@ -237,6 +243,8 @@ static final String getTokenName(int tdsTokenType) {
                 return "TDS_FEATURE_EXT_AZURESQLDNSCACHING (0x0B)";
             case TDS_FEATURE_EXT_SESSIONRECOVERY:
                 return "TDS_FEATURE_EXT_SESSIONRECOVERY (0x01)";
+            case TDS_FEATURE_EXT_VECTORSUPPORT:
+                return "TDS_FEATURE_EXT_VECTORSUPPORT (0x0E)";
             default:
                 return "unknown token (0x" + Integer.toHexString(tdsTokenType).toUpperCase() + ")";
         }
@@ -5374,6 +5382,15 @@ else if (DataTypes.UNKNOWN_STREAM_LENGTH == dataLength)
                 internalJDBCType = javaType.getJDBCType(SSType.UNKNOWN, jdbcType);
                 writeInternalTVPRowValues(internalJDBCType, currentColumnStringValue, currentObject, columnPair, true);
                 break;
+
+            case VECTOR:
+                byte[] bValue = (currentObject == null) ? null : (byte[]) currentObject;
+                writeShort((short) (bValue == null ? -1 : bValue.length));
+                if (bValue != null) {
+                    writeBytes(bValue);
+                }
+                break;
+
             default:
                 assert false : "Unexpected JDBC type " + jdbcType.toString();
         }
@@ -5507,6 +5524,13 @@ void writeTVPColumnMetaData(TVP value) throws SQLServerException {
                     writeInt(TDS.SQL_VARIANT_LENGTH);// write length of sql variant 8009
 
                     break;
+                
+                case VECTOR:
+                    writeByte(TDSType.VECTOR.byteValue());
+                    writeShort((short) (VectorUtils.getVectorLength(pair.getValue().scale,  pair.getValue().precision))); // max length
+                    byte scaleByte = (byte) (VectorUtils.getScaleByte(pair.getValue().scale));
+                    writeByte((byte) scaleByte); // scale
+                    break;
 
                 default:
                     assert false : "Unexpected JDBC type " + jdbcType.toString();
@@ -5632,6 +5656,30 @@ void writeCryptoMetaData() throws SQLServerException {
         writeByte(cryptoMeta.normalizationRuleVersion);
     }
 
+    void writeRPCVector(String sName, Vector vectorValue, boolean bOut, int scale, int precision)
+            throws SQLServerException {
+        boolean vectorValueNull = (vectorValue == null);
+        byte[] bValue = vectorValueNull ? null : VectorUtils.toBytes(vectorValue);
+
+        writeRPCNameValType(sName, bOut, TDSType.VECTOR);
+
+        if (vectorValueNull) {
+            writeShort((short) (VectorUtils.getVectorLength(scale, precision))); // max length
+            byte scaleByte = (byte) (VectorUtils.getScaleByte(scale));
+            writeByte((byte) scaleByte); // scale (dimension type)
+            writeShort((short) -1); // actual len
+        } else {
+            writeShort((short) VectorUtils.getVectorLength(vectorValue)); // max length
+            writeByte((byte) VectorUtils.getScaleByte(vectorValue.getVectorDimensionType())); // scale (dimension type)
+            if (vectorValue.getData() == null) {
+                writeShort((short) -1); // actual len
+            } else {
+                writeShort((short) VectorUtils.getVectorLength(vectorValue)); // actual len
+                writeBytes(bValue); // data
+            }
+        }
+    }
+    
     void writeRPCByteArray(String sName, byte[] bValue, boolean bOut, JDBCType jdbcType,
             SQLCollation collation) throws SQLServerException {
         boolean bValueNull = (bValue == null);

src/main/java/com/microsoft/sqlserver/jdbc/ISQLServerConnection.java

@@ -652,4 +652,23 @@ CallableStatement prepareCall(String sql, int nType, int nConcur, int nHold,
      * @return boolean value for bulkCopyForBatchInsertAllowEncryptedValueModifications.
      */
     boolean getBulkCopyForBatchInsertAllowEncryptedValueModifications();
+
+    /**
+     * Sets the value of the vectorTypeSupport connection property.
+     *
+     * @param vectorTypeSupport
+     * A string that indicates the vector type support during connection initialization.
+     * Valid values are "off" (vector types are returned as strings) and "v1" (vectors of type FLOAT32 are returned as vectors). 
+     * Default is "v1".
+     */
+    void setVectorTypeSupport(String vectorTypeSupport);
+
+    /**
+     * Returns the value of the vectorTypeSupport connection property.
+     *
+     * @return vectorTypeSupport
+     *         The current vector type support setting ("off" or "v1").
+     */
+    String getVectorTypeSupport();
+
 }

src/main/java/com/microsoft/sqlserver/jdbc/ISQLServerDataSource.java

@@ -1084,6 +1084,24 @@ public interface ISQLServerDataSource extends javax.sql.CommonDataSource {
      * @return whether to allow encrypted value modifications during bulk copy operations created from batch insert operations.
      */
     boolean getBulkCopyForBatchInsertAllowEncryptedValueModifications();
+
+    /**
+     * Sets the value of the vectorTypeSupport connection property.
+     *
+     * @param vectorTypeSupport
+     * A string that indicates the vector type support during connection initialization.
+     * Valid values are "off" (vector types are returned as strings) and "v1" (vectors of type FLOAT32 are returned as vectors). 
+     * Default is "v1".
+     */
+    void setVectorTypeSupport(String vectorTypeSupport);
+
+    /**
+     * Returns the value of the vectorTypeSupport connection property.
+     *
+     * @return vectorTypeSupport
+     *         The current vector type support setting ("off" or "v1").
+     */
+    String getVectorTypeSupport();
 
     /**
      * Sets the client id to be used to retrieve the access token for a user-assigned Managed Identity.