Document how to locally run the container

[ManuelVs]

What did you find confusing? Please describe.

I tried to extend the image adding my code and run it locally. However, the server does not start and it doesn't publish any logs from our scripts.

Dockerfile:

FROM 763104351884.dkr.ecr.eu-west-1.amazonaws.com/pytorch-inference:1.10.0-cpu-py38

ENV SAGEMAKER_PROGRAM "my_amazing_entrypoint.py"
ENV SAGEMAKER_REGION "eu-west-1"
ENV SAGEMAKER_SUBMIT_DIRECTORY "/opt/ml/model/code"

WORKDIR "/opt/ml/model/"
COPY model_new.tar.gz "/opt/ml/model/model.tar.gz"
RUN tar -xf model.tar.gz

model.tar.gz:

.
| - code/
  | - my_amazing_entrypoint.py
  | - more_packages/
| - pytorch_model.pth

Commands executed:

docker build -t pytorch-test .
docker run -ti pytorch-test

Output:

Warning: TorchServe is using non-default JVM parameters: -XX:+UseContainerSupport -XX:InitialRAMPercentage=8.0 -XX:MaxRAMPercentage=10.0 -XX:-UseLargePages -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2022-05-25T10:26:35,124 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager...
2022-05-25T10:26:35,219 [INFO ] main org.pytorch.serve.ModelServer - 
Torchserve version: 0.5.2
TS Home: /opt/conda/lib/python3.8/site-packages
Current directory: /opt/ml/model
Temp directory: /home/model-server/tmp
Number of GPUs: 0
Number of CPUs: 8
Max heap size: 3166 M
Python executable: /opt/conda/bin/python3.8
Config file: /home/model-server/config.properties
Inference address: http://0.0.0.0:8080
Management address: http://0.0.0.0:8081
Metrics address: http://127.0.0.1:8082
Model Store: /home/model-server
Initial Models: ALL
Log dir: /opt/ml/model/logs
Metrics dir: /opt/ml/model/logs
Netty threads: 0
Netty client threads: 0
Default workers per model: 8
Blacklist Regex: N/A
Maximum Response Size: 6553500
Maximum Request Size: 6553500
Limit Maximum Image Pixels: true
Prefer direct buffer: false
Allowed Urls: [file://.*|http(s)?://.*]
Custom python dependency for model allowed: false
Metrics report format: prometheus
Enable metrics API: true
Workflow Store: /home/model-server
Model config: N/A
2022-05-25T10:26:35,225 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager -  Loading snapshot serializer plugin...
2022-05-25T10:26:35,247 [DEBUG] main org.pytorch.serve.ModelServer - Loading models from model store: tmp
2022-05-25T10:26:35,249 [WARN ] main org.pytorch.serve.ModelServer - Failed to load model: /home/model-server/tmp
org.pytorch.serve.archive.model.ModelNotFoundException: Model not found at: tmp
        at org.pytorch.serve.archive.model.ModelArchive.downloadModel(ModelArchive.java:75) ~[model-server.jar:?]
        at org.pytorch.serve.wlm.ModelManager.createModelArchive(ModelManager.java:167) ~[model-server.jar:?]
        at org.pytorch.serve.wlm.ModelManager.registerModel(ModelManager.java:133) ~[model-server.jar:?]
        at org.pytorch.serve.wlm.ModelManager.registerModel(ModelManager.java:69) ~[model-server.jar:?]
        at org.pytorch.serve.ModelServer.initModelStore(ModelServer.java:194) [model-server.jar:?]
        at org.pytorch.serve.ModelServer.startRESTserver(ModelServer.java:356) [model-server.jar:?]
        at org.pytorch.serve.ModelServer.startAndWait(ModelServer.java:117) [model-server.jar:?]
        at org.pytorch.serve.ModelServer.main(ModelServer.java:98) [model-server.jar:?]
2022-05-25T10:26:35,264 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2022-05-25T10:26:35,325 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://0.0.0.0:8080
2022-05-25T10:26:35,325 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.
2022-05-25T10:26:35,327 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://0.0.0.0:8081
2022-05-25T10:26:35,327 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel.
2022-05-25T10:26:35,328 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8082
Model server started.
2022-05-25T10:26:35,588 [INFO ] pool-3-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:d90f998d682a,timestamp:1653474395
2022-05-25T10:26:35,589 [INFO ] pool-3-thread-1 TS_METRICS - DiskAvailable.Gigabytes:334.71700286865234|#Level:Host|#hostname:d90f998d682a,timestamp:1653474395
2022-05-25T10:26:35,590 [INFO ] pool-3-thread-1 TS_METRICS - DiskUsage.Gigabytes:157.27763748168945|#Level:Host|#hostname:d90f998d682a,timestamp:1653474395
2022-05-25T10:26:35,590 [INFO ] pool-3-thread-1 TS_METRICS - DiskUtilization.Percent:32.0|#Level:Host|#hostname:d90f998d682a,timestamp:1653474395
2022-05-25T10:26:35,590 [INFO ] pool-3-thread-1 TS_METRICS - MemoryAvailable.Megabytes:27624.3125|#Level:Host|#hostname:d90f998d682a,timestamp:1653474395
2022-05-25T10:26:35,591 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUsed.Megabytes:3569.87109375|#Level:Host|#hostname:d90f998d682a,timestamp:1653474395
2022-05-25T10:26:35,591 [INFO ] pool-3-thread-1 TS_METRICS - MemoryUtilization.Percent:12.7|#Level:Host|#hostname:d90f998d682a,timestamp:1653474395

Describe how documentation can be improved

It would be nice to add a section in the README file (or similar) with an example on how to run the image / container in a local docker installation.

Additional context

• This would improve developer experience by reducing the amount of time between trial and error. (It takes a while to deploy to Sagemaker)
• We can easily debug problems step by step.

[pharmapsychotic]

Did you figure out how to run it locally?

[Lunatik00]

I was able to run endpoints locally, it is..... not simple.

Well, let's assume you know a little about how sagemaker works, mainly what goes into the /opt/ml/model is not the .tar.gz file but what is inside that file, I also will be using boto3.

First, the easy way, just run the container directly, docker run --rm -p 8080:8080 --gpus all -v $PATH_TO_FOLDER:/opt/ml/model $CONTAINER_NAME serve and you use request to send whatever you want to http://localhost:8080/invocations

This works, with the bonus that you don't need to use python, but it will be outside the sagemaker environment, so, to get a little closer to sagemaker you have to use the local mode, install sagemaker[local] and boto3, now the long part.

first import and initialize the local sagemaker

import boto3
import sagemaker
from sagemaker.local import LocalSession

boto_session = boto3.Session(region_name='us-west-2')
session = LocalSession(boto_session=boto_session)
session.config = {'local': {'local_code': True}}

Explaination, botosession must be set with one region or you will get an error, the last line ensures that all will be done in local mode, or at least that is what it says in the documentation, now you have a sagemaker session, you can do everything and it will be done in the local version, but there are small differences, I use custom containers, so, I will be using the single container custom inference in sagemaker., first, we create the model

model = session.create_model(
    name='local',
    role='arn:aws:iam::123456789012:role/service-role/AmazonSageMaker', #dummy execution role
    primary_container={
        "Image": "$YOUR_IMAGE", 
        "ModelDataUrl": f"file://$FOLDER_WITH_TAR_GZ_CONTENTS",
        "Environment": {}, # Variables de entorno
    },
)

The file is from your current working directory, and you must set environments even when they are empty

you create the config

config = session.sagemaker_client.create_endpoint_config(
    EndpointConfigName='local-endpoint-config',
    ProductionVariants=[
        {
            'VariantName': 'local-variant',
            'ModelName': 'local',
            'InstanceType': 'local',
            'InitialInstanceCount': 1,
        }
    ]
)

As you can see, here InstanceType is different from the instances in aws, because it is local mode.

to finish you create the local endpoint

ep = session.sagemaker_client.create_endpoint(
    EndpointName='local-endpoint',
    EndpointConfigName='local-endpoint-config',
)

All this is done in memory, meaning that it should be done in a .ipynb file to test and when that file is closed it will not be available.

Now you can invoke the endpoint as you would do in your final script, but using the local endpoint name

predictor = sagemaker.predictor.Predictor(endpoint_name='local-endpoint', sagemaker_session=session)
response = predictor.predict(payload)

or

session.sagemaker_runtime_client.invoke_endpoint(
    EndpointName='local-endpoint',
    ContentType='application/json',
    Body=payload,
)['Body']

I couldn't find this info in any current guide and some of the details I have here were not explained in the little info I could gather, but this works as of today, and I used it 2 years ago, but with provided containers, I know it will probably not help you much, but I hope it would, to you or others that, like me, end up here when trying to make this work. AWS is seriously lacking on material to learn how to use their services.