Specify batch size for MME #134
Description
What did you find confusing? Please describe.
How do you specify batch size for MME models?
Describe how documentation can be improved
This blog describes using env vars to set batch size and other parameters for a single-model endpoint, however, I haven't found any documentation on setting batch size for individual models within a MME.
Additional context
Each model in my MME has a MAR-INF/MANIFEST.json within its model.tar.gz, so I tried to specify batchSize in these files, but I don't think it's being applied.