Skip to content

TASK-6217 - Solr improvements: Stats and Update #2601

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 83 commits into
base: develop
Choose a base branch
from
Open

TASK-6217 - Solr improvements: Stats and Update #2601

wants to merge 83 commits into from

Conversation

@j-coll
Copy link
Member

@j-coll j-coll commented Jun 18, 2025

No description provided.

j-coll added 30 commits May 13, 2025 11:38
@j-coll
storage: Remove variant-secondary-annotation-index with samples. #TAS…
eade1a5 
…K-7669 #TASK-6217
@j-coll
storage: Extend ProjectMetadata to include explicit solr configuratio…
62dc57b 
…n #TASK-7676 #TASK-6217
@j-coll
storage: Use separate versioning for configset. #TASK-6217
26e8cd1
@j-coll
storage: Disable waitSearcher and openSearcher when writing into solr…
560a864 
…. #TASK-6217
@j-coll
storage: Replace the pending secondary annot index table with a file …
52a746d 
…based implementation. #TASK-6217
@j-coll
benchmark: Allow benchmark over https. Upgrade jmeter version. Add ge…
6950971 
…nefile. #TASK-6217
@j-coll
storage: Deprecate previos active searchmetadata if needed. #TASK-6217
6721225
@j-coll
storage: Remove "stored" fields from manage-schema. Update solr-confi…
b7c8095 
…g.xml . #TAKS-6217
@j-coll
storage: Add files integrity check. In fails, force MR execution. #TA…
cb335f4 
…SK-6217
@j-coll
storage: Fix VariantSearchToVariantConverter. #TASK-6217
7917f63
@j-coll
storage: Test with solr cloud instead of solr core. #TASK-6217
e4194fa
@j-coll
storage: Fix final search index synchronization. #TASK-6217
b4193cb
@j-coll
storage: Reduce dfs.replication for temporary files. #TASK-6217
bc90599
@j-coll
storage: Fix MiniSolrCloudCluster configuration. Add configsets. #TAS…
e758974 
…K-6217
@j-coll
storage: Log dfs.replication when writing MR writes to hdfs. #TASK-6217
6fb012a
@j-coll
storage: Create collections with Tlog replicas in multi-node solrclou…
486454f 
…d. #TASK-6217
@j-coll
storage: On solr write error, abort any hbase mutation or file clean.…
50b060a 
… #TASK-6217
@j-coll
storage: Do not clean pending variant files on reader error. #TASK-6217
5af7f6e
@j-coll
storage: Mark for cleaning only one file per variant. Ignore end posi…
105febc 
…tion. #TASK-6217
@j-coll
storage: Add automatic sharding on collection creation. #TASK-6217
aae1744
@j-coll
storage: Fix NPE reading status. Add numDocs and size in bytes. #TASK…
3d53309 
…-6217
@j-coll
storage: Move stats to a different solr collection. #TASK-6217
10e42c4
@j-coll
pom: Remove mortbay servlet-api dependencies. #TASK-6217
c686a72
@j-coll
storage: Fix active search index metadata selection. #TASK-6217
bead342
@j-coll
storage: Use join for filtering by stats in other collection. #TASK-6217
b3409c7
@j-coll
storage: Force sort by "id" on each solr query. #TASK-6217
54d8567
@j-coll
storage: Remove SEARCH_INDEX_LAST_TIMESTAMP. Use "lastUpdateDate" fro…
77e4e2a 
…m SearchIndexMetadata. #TASK-6217
@j-coll
storage: Increase insertBatchSize to 15k. #TASK-6217
b31dd45
@j-coll
storage: Add command opencga-admin.sh benchmark variant. #TASK-6217
5c843d5
@j-coll
storage: Store variantStats values as counters instead of freq values…
f0c7dad 
…. #TASK-6217
pfurio
pfurio previously approved these changes Jul 11, 2025
View reviewed changes
j-coll added 13 commits July 15, 2025 11:31
@j-coll
storage: fix converter setIncludeIndexStatus #TASK-6217
5e47f86
@j-coll
analysis: Start HadoopExternalResource before VariantSolrExternalReso…
2027394 
…urce. #TASK-6217
@j-coll
storage: Fix tests. #TASK-6217
c45fbae
@j-coll
storage: Fix MAF filter in solr #TASK-6217
7473c18
@j-coll
analyis: Fix VariantOperationsTest. #TASK-6217
fb23ae1
@j-coll
storage: Fix checkstyle. #TASK-6217
505937e
@j-coll
storage: Fix SolrQueryParser MAF. #TASK-6217
3486c5e
@j-coll
storage: Add searchIndex information to opencga-admin storage status.…
89a1f26 
… #TASK-6217
@j-coll
storage: Do not flip MAF filters over 0.5. #TASK-6217
0bb98d9
@j-coll
storage: Close output asynchronously at DiscoverVariantsFileBasedMapp…
6405b66 
…er. #TASK-6217
@j-coll
storage: Add default list of queries to benchmark. #TASK-6217
28a6f61
@j-coll
storage: set final name for configset. #TASK-6217
81e471a
@j-coll
storage: Increase "search.load.numThreads" from 1 to 2. #TASK-6217
01fb512
pfurio
pfurio previously approved these changes Jul 24, 2025
View reviewed changes
j-coll added 10 commits August 18, 2025 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pfurio pfurio pfurio left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@j-coll @halender @pfurio