Lack of Thread Safety causing ArrayIndexOutOfBoundsException in ArrayList in StoreReader

[jamesdbloom]

0.002% of our calls to load operations result in the following error, which means we get around 5 errors per hour. This looks like it is caused by a lack of thread safety around the way an ArrayList is used such that the underlying array inside the ArrayList is size zero but an item is being added at index 1. If you assume the ArrayList does not have a bug (which is likely due to how widely it is used) then the way the ArrayList is being used inside the StoreReader is not thead safe.

com.microsoft.azure.documentdb.DocumentClientException: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 0, 
RequestStartTime: "26 Oct 2019 00:46:55.566", RequestEndTime: "26 Oct 2019 00:46:55.587", Number of regions attempted: 1
StoreResponseStatistics{requestResponseTime="26 Oct 2019 00:46:55.587", storeReadResult=StoreReadResult{statusCode=200, storePhysicalAddress=https://cdb-ms-prod-westeurope1-fd14.documents.azure.com:14786/apps/413f848b-ce17-40fc-ad7f-14c0e21e9633/services/b6c727a9-51a9-434e-be69-84d1b99e26e2/partitions/bc2ffcc0-d4de-43d0-b854-453cabf7c920/replicas/132151774908527505s//, LSN=2019735, sessionToken=-1#2019735, partitionKeyRangeId='0', quorumAckedLSN=-1, globalCommittedLSN=2019735, itemLSN=-1, numberOfReadRegions=0, requestCharge=1.0, currentReplicaSetSize=-1, currentWriteQuorum=-1, isValid=true, isGoneException=false, isNotFoundException=false, exception='null'}, requestResourceType=Document, requestOperationType=Read}

	at com.microsoft.azure.documentdb.internal.directconnectivity.StoreReader.readMultipleReplica(StoreReader.java:161)
	at com.microsoft.azure.documentdb.internal.directconnectivity.QuorumReader.readQuorum(QuorumReader.java:164)
	at com.microsoft.azure.documentdb.internal.directconnectivity.QuorumReader.readStrong(QuorumReader.java:84)
	at com.microsoft.azure.documentdb.internal.directconnectivity.ConsistencyReader.read(ConsistencyReader.java:73)
	at com.microsoft.azure.documentdb.internal.directconnectivity.ReplicatedResourceClient.invoke(ReplicatedResourceClient.java:71)
	at com.microsoft.azure.documentdb.internal.directconnectivity.ServerStoreModel$1.apply(ServerStoreModel.java:101)
	at com.microsoft.azure.documentdb.internal.RetryUtility.executeStoreClientRequest(RetryUtility.java:147)
	at com.microsoft.azure.documentdb.internal.directconnectivity.ServerStoreModel.processMessage(ServerStoreModel.java:118)
	at com.microsoft.azure.documentdb.DocumentClient$8.apply(DocumentClient.java:3102)
	at com.microsoft.azure.documentdb.internal.RetryUtility.executeDocumentClientRequest(RetryUtility.java:73)
	at com.microsoft.azure.documentdb.DocumentClient.doRead(DocumentClient.java:3108)
	at com.microsoft.azure.documentdb.DocumentClient.readDocument(DocumentClient.java:1071)
	at com.tesco.crypt.common.persistence.cosmosdb.CosmosDbClient.getDocument(CosmosDbClient.java:218)
	at com.tesco.crypt.common.persistence.cosmosdb.CosmosDbClient.lambda$loadSynchronously$3(CosmosDbClient.java:155)
	... 15 more
Caused by: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 0
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at com.microsoft.azure.documentdb.internal.directconnectivity.StoreReader.readMultipleReplica(StoreReader.java:157)
	... 28 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 0
	at java.base/java.util.ArrayList.add(ArrayList.java:486)
	at java.base/java.util.ArrayList.add(ArrayList.java:498)
	at com.microsoft.azure.documentdb.internal.directconnectivity.StoreReader.recordReadResponse(StoreReader.java:406)
	at com.microsoft.azure.documentdb.internal.directconnectivity.StoreReader.access$100(StoreReader.java:33)
	at com.microsoft.azure.documentdb.internal.directconnectivity.StoreReader$1.call(StoreReader.java:221)
	at com.microsoft.azure.documentdb.internal.directconnectivity.StoreReader$1.call(StoreReader.java:217)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	... 4 more