Async server (#73)

Author: ra1han

.github/workflows/pypi-publish.yml renamed to .github/workflows/cd.yml

@@ -1,6 +1,9 @@
-name: Publish Python Package
+name: Publish PyPi Package
 
-on: [push, pull_request]
+on:
+  push:
+    tags:
+      - 'v*.*.*'  # triggers on tags like v1.2.3
 
 jobs:
   test_and_build:
@@ -30,7 +33,7 @@ jobs:
   deploy:
     needs: test_and_build
     runs-on: ubuntu-latest
-    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+    if: startsWith(github.ref, 'refs/tags/')
     steps:
       - name: Download artifact
         uses: actions/download-artifact@v4

.github/workflows/tests.yml renamed to .github/workflows/ci.yml

@@ -1,8 +1,8 @@
-name: PromptLab Tests
+name: Tests
 
 on:
   push:
-    branches: [main, feature/async-support]
+    branches: [main, feature/*]
   pull_request:
     branches: [main]
 

.gitignore

@@ -180,4 +180,6 @@ ide_docs/*
 .cursor/
 .trunk/
 AGENTS.md
-.env
+.env
+
+*.whl

docker/Dockerfile

@@ -13,11 +13,13 @@ RUN apt-get update && apt-get install -y \
     gcc \
     && rm -rf /var/lib/apt/lists/*
 
-# Copy requirements file and wheel file
-COPY requirements.txt .
+# # Uncomment for PROD image build
+# COPY requirements.txt .
+# RUN pip install --no-cache-dir -r requirements.txt
 
-# Install Python dependencies from requirements.txt
-RUN pip install --no-cache-dir -r requirements.txt
+# Uncomment for TEST image build
+COPY promptlab-0.1.6-py3-none-any.whl .
+RUN pip install --no-cache-dir promptlab-0.1.6-py3-none-any.whl
 
 # Create directory for database
 RUN mkdir -p /app/data

docker/requirements.txt

@@ -1 +1 @@
-promptlab==0.1.5
+promptlab==0.1.6

docs/self_hosting.md

@@ -29,26 +29,37 @@ This guide covers how to deploy PromptLab using Docker containers, both locally
 Run PromptLab with default settings:
 
 ```bash
-docker run -p 8000:8000 -p 8001:8001 --name promptlab-container promptlab:latest
+docker run -p 8000:8000 --name promptlab-container promptlab:latest
 ```
 
 #### Production Setup with Persistent Data
 
-For production use, mount a volume to persist your database:
+For production use, mount a volume to persist your database and set security environment variables:
 
 ```bash
 docker run -p 8000:8000 -p 8001:8001 \
   -v /your/local/data/path:/app/data \
+  -e PROMPTLAB_SECRET_KEY="your-secure-32-character-secret-key-here" \
   --name promptlab-container \
   --restart unless-stopped \
   promptlab:latest
 ```
 
 **Windows Example:**
 ```powershell
-docker run -p 8000:8000 -p 8001:8001 -v C:\work\promptlab_data:/app/data --name promptlab-container --restart unless-stopped promptlab:latest
+docker run -p 8000:8000 -p 8001:8001 -v C:\work\promptlab_data:/app/data -e PROMPTLAB_SECRET_KEY="your-secure-32-character-secret-key-here" --name promptlab-container --restart unless-stopped promptlab:latest
 ```
 
+#### Environment Variables
+
+| Variable | Required | Description | Default |
+|----------|----------|-------------|---------|
+| `PROMPTLAB_SECRET_KEY` | **Recommended** | JWT signing key for authentication. Should be 32+ characters for security. | Auto-generated (not persistent) |
+| `PROMPTLAB_LOG_LEVEL` | Optional | Logging level (`DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`) | `INFO` |
+| `PROMPTLAB_LOG_DIR` | Optional | Custom directory for log files | Platform-specific default |
+
+**⚠️ Security Note:** Always set `PROMPTLAB_SECRET_KEY` in production to ensure consistent JWT token validation across container restarts.
+
 ### Accessing the Application
 
 - **Main Application:** http://localhost:8000
@@ -109,7 +120,8 @@ az container create \
   --cpu 2 \
   --memory 4 \
   --os-type Linux \
-  --restart-policy Always
+  --restart-policy Always \
+  --environment-variables PROMPTLAB_SECRET_KEY="your-secure-32-character-secret-key-here"
 ```
 
 #### Step 4: Access Your Application
@@ -150,5 +162,8 @@ az containerapp create \
   --cpu 1.0 \
   --memory 2.0Gi \
   --min-replicas 1 \
-  --max-replicas 3
-```
+  --max-replicas 3 \
+  --env-vars PROMPTLAB_SECRET_KEY="your-secure-32-character-secret-key-here"
+```
+
+**Note:** For Container Apps, you can also set environment variables through the Azure portal or use Azure Key Vault for enhanced security.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "promptlab"
-version = "0.1.5"
+version = "0.1.6"
 description = "PromptLab is a free, lightweight, open-source experimentation tool for Gen AI applications."
 readme = "README.md"
 authors = [{name = "ra1han", email = "[email protected]"}]
@@ -30,12 +30,14 @@ dependencies = [
     "art>=6.5",
     "setuptools",
     "fastapi",
-    "uvicorn",
+    "uvicorn[standard]>=0.18.0",
     "sqlalchemy",
     "passlib",
     "python-jose",
     "python-multipart",
-    "bcrypt"
+    "bcrypt",
+    "uvloop>=0.17.0; sys_platform != 'win32'",
+    "httptools>=0.5.0; sys_platform != 'win32'",
 ]
 requires-python = ">=3.8"
 

samples/agent_evaluation/agent_evaluation.py

@@ -27,7 +27,7 @@ async def agent_proxy(inputs: dict) -> ModelResponse:
 dataset = Dataset(
     name=dataset_name, description=dataset_description, file_path=dataset_file_path
 )
-# ds = pl.asset.create(dataset)
+ds = pl.asset.create(dataset)
 
 # Retrieve assets
 ds = pl.asset.get(asset_name=dataset_name, version=0)

samples/hosted/README.md

@@ -1,10 +1,12 @@
-# Quickstart
+# Self-hosting
 
-This sample ([quickstart.py](quickstart.py)) demonstrates how to use PromptLab to evaluate a simple prompt. 
+This sample ([hosted.py](hosted.py)) demonstrates how to use self-hosted PromptLab. 
 
 ## Installation and Setup
 
-It's highly recommended to use a virtual environment (try venv or conda or uv).
+To host PromptLab in your environment, please follow this guide - [Self Hosting PromptLab](../../docs/self_hosting.md).
+
+To interact with the hosted PromptLab service, locally install the PromptLab package in a virtual environment (try venv or conda or uv).
 
 ```bash
 pip install promptlab
@@ -15,79 +17,15 @@ pip install promptlab
 The first step to use PromptLab is to initialize the PromptLab object. Please check [Tracer](../../docs/README.md#tracer) to learn more about the tracer configuration.
 
 ```python
-tracer_config = {"type": "sqlite", "db_file": "./promptlab.db"}
-pl = PromptLab(tracer_config)
-```
-
-Once the PromptLab object is ready, you can start the PromptLab Studio to check the assets and experiments.
-
-```python
-pl.studio.start(8000)
-```
-
-## Create a Prompt Template
-
-A prompt template is a prompt with or without placeholders. Please check [Prompt Template](../../docs/README.md#prompt-template) to learn more about it.
-
-A prompt template has two main attributes - `system_prompt` and `user_prompt`. The sample prompt used in this example is:
-
-```python
-prompt_name = "essay_feedback"
-prompt_description = "A prompt for generating feedback on essays"
-system_prompt = "You are a helpful assistant who can provide feedback on essays."
-user_prompt = """The essay topic is - <essay_topic>.
-               The submitted essay is - <essay>
-               Now write feedback on this essay."""
-prompt_template = PromptTemplate(name=prompt_name, description=prompt_description, system_prompt=system_prompt, user_prompt=user_prompt)
-pt = pl.asset.create(prompt_template)
-```
-
-Here, `<essay_topic>` and `<essay>` are placeholders that will be replaced with real data before sending to the LLM. PromptLab will search the dataset for columns with these exact names and use their values to replace the corresponding placeholders. Ensure that the dataset contains columns named `essay_topic` and `essay`.
-
-![PromptLab Studio](../../img/studio-pt.png)
-
-## Create Dataset
-
-A dataset is a JSONL file used to design the experiment. Please check [Dataset](../../docs/README.md#dataset) to learn more about it.
-
-```python
-dataset_name = "essay_samples"
-dataset_description = "dataset for evaluating the essay_feedback prompt"
-dataset_file_path = "./samples/data/essay_feedback.jsonl"
-dataset = Dataset(name=dataset_name, description=dataset_description, file_path=dataset_file_path)
-ds = pl.asset.create(dataset)
-```
-
-![PromptLab Studio](../../img/studio-ds.png)
-
-## Create Experiment
-
-An experiment evaluates the outcome of a prompt against a set of metrics for a given dataset. Developers can modify hyperparameters (such as prompt template and models) and compare experiment results to determine the best prompt to deploy in production. Please check [Experiment](../../docs/README.md#experiment) to learn more about it.
-
-In the [quickstart.py](quickstart.py), we are using the prompt template and dataset created in the previous steps to design an experiment. It also uses two evaluation metrics: `semantic_similarity` and `fluency`. Please check [Metric](../../docs/README.md#metric) to learn more about evaluation metrics.
-
-```python
-experiment_config = {
-    "completion_model": completion_model,
-    "embedding_model": embedding_model,
-    "prompt_template": pt,
-    "dataset": ds,
-    "evaluation": [
-        {
-            "metric": "semantic_similarity",
-            "column_mapping": {"response": "$completion", "reference": "feedback"},
-        },
-        {
-            "metric": "fluency",
-            "column_mapping": {"response": "$completion"},
-        },
-    ],
+tracer_config = {
+    "type": "api",
+    "endpoint": "http://HOST-URL:8001",
+    "jwt_token": "JWT_TOKEN",
 }
-pl.experiment.run(experiment_config)
+pl = PromptLab(tracer_config)
 ```
 
-![PromptLab Studio](../../img/studio-exp.png)
-
-Details of the experiment:
+- HOST-URL is the URL of the hosted service.
+- JWT_TOKEN can be found in the PromptLab page. Copy it from the top right corner profile link.
 
-![PromptLab Studio](../../img/studio-exp-details.png)
+Everything else for creating assets and running experiments are same as other examples.

samples/hosted/hosted.py

@@ -2,11 +2,11 @@
 from promptlab import PromptLab
 from promptlab.types import PromptTemplate, Dataset
 
-# Initialize PromptLab with api tracer
+# Initialize PromptLab with remote tracer
 tracer_config = {
-    "type": "api",
+    "type": "remote",
     "endpoint": "http://HOST-URL:8001",
-    "jwt_token": "your_jwt_token_here",
+    "jwt_token": "JWT_TOKEN",
 }
 pl = PromptLab(tracer_config)
 
@@ -68,6 +68,3 @@
 
 # # Run the experiment asynchronously
 asyncio.run(pl.experiment.run_async(experiment_config))
-
-# # Start the PromptLab Studio to view results
-# asyncio.run(pl.studio.start_async(8000))