Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -350,6 +350,7 @@
"models/automations/create-automations/webhook"
]
},
"models/automations/view-automation-history",
"models/automations/automation-events"
]
},
Expand Down
Binary file removed images/automations/project_automations_tab.png
Binary file not shown.
Binary file removed images/automations/registry_automations_tab.png
Binary file not shown.
45 changes: 29 additions & 16 deletions models/automations.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,31 +8,43 @@ import EnterpriseCloudOnly from "/snippets/en/_includes/enterprise-cloud-only.md
<EnterpriseCloudOnly/>
</Info>

This page describes _automations_ in W&B. [Create an automation](/models/automations/create-automations/) to trigger workflow steps, such as automated model testing and deployment, based on an event in W&B, such as when an [artifact](/models/artifacts/) artifact version is created or when a [run metric](/models/runs/) meets or changes by a threshold.
This page describes _automations_ in W&B. [Create an automation](/models/automations/create-automations/) to trigger workflow steps, such as automated model testing and deployment, based on an event in W&B.

For example, an automation can notify a Slack channel when a new version is created, trigger an automated testing webhook when the `production` alias is added to an artifact, or start a validation job only when a run's `loss` is within acceptable bounds.

## Overview
An automation can start when a specific [event](/models/automations/automation-events/) occurs in a registry or project.
<Info>
Looking for companion tutorials for automations?
- [Learn to automatically trigger a Github Action for model evaluation and deployment](https://wandb.ai/wandb/wandb-model-cicd/reports/Model-CI-CD-with-W-B--Vmlldzo0OTcwNDQw).
- [Watch a video demonstrating automatically deploying a model to a Sagemaker endpoint](https://www.youtube.com/watch?v=s5CMj_w3DaQ).
- [Watch a video series introducing automations](https://youtube.com/playlist?list=PLD80i8An1OEGECFPgY-HPCNjXgGu-qGO6&feature=shared).
</Info>

In a [Registry](/models/registry/), an automation can start:
- When a new artifact version is linked to a collection. For example, trigger testing and validation workflows for new candidate models.
- When an alias is added to an artifact version. For example, trigger a deployment workflow when an alias is added to a model version.
## Automation events

In a [project](/models/track/project-page/), an automation can start:
- When a new version is added to an artifact. For example, start a training job when a new version of a dataset artifact is added to a given collection.
- When an alias is added to an artifact version. For example, trigger a PII redaction workflow when the alias "redaction" is added to a dataset artifact.
- When a tag is added to an artifact version. For example, trigger a geo-specific workflow when the tag "europe" is added to an artifact version.
- When a metric for a run meets or exceeds a configured threshold.
- When a metric for a run changes by a configured threshold.
- When a run's status changes to **Running**, **Failed**, or **Finished**.
Automations can be triggered by events related to runs or artifacts in a project or registry. An automation can start:

Optionally, you can:
- Filter artifacts, aliases, or tags by name
- Filter runs by user or run name.
- When a run metric meets a defined absolute or relative threshold. For example, trigger when a run's `loss` is less than 0.01 or when a run's `accuracy` improves by 5%.
- When an artifact event occurs in a project or registry. For example, trigger when a new model version is created or when the `production` alias is added to a model.

For more details, see [Automation events and scopes](/models/automations/automation-events/).

## Automation actions

When an event triggers an automation, it can perform one of these actions:

### Slack notification
Send a message to a Slack channel with details about the triggering event. The message includes relevant information such as the entity name, user who triggered the event, and a link to view more details in W&B.

### Webhook
Call a webhook URL with a JSON payload containing information about the triggering event. This enables integration with external systems like CI/CD pipelines, model deployment services, or custom workflows. The webhook payload includes:
- Event details (type, entity information)

For implementation details, see:
- [Create a Slack automation](/models/automations/create-automations/slack/)
- [Create a webhook automation](/models/automations/create-automations/webhook/)

## How automations work

To [create an automation](/models/automations/create-automations/), you:

1. If required, configure [secrets](/platform/secrets/) for sensitive strings the automation requires, such as access tokens, passwords, or sensitive configuration details. Secrets are defined in your **Team Settings**. Secrets are most commonly used in webhook automations to securely pass credentials or tokens to the webhook's external service without exposing it in plain text or hard-coding it in the webhook's payload.
Expand All @@ -46,5 +58,6 @@ To [create an automation](/models/automations/create-automations/), you:

## Next steps
- [Create an automation](/models/automations/create-automations/).
- [View an automation's history](/models/automations/view-automation-history) to track executions and debug issues.
- Learn about [Automation events and scopes](/models/automations/automation-events/).
- [Create a secret](/platform/secrets/).
80 changes: 26 additions & 54 deletions models/automations/automation-events.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,81 +7,53 @@ import EnterpriseCloudOnly from "/snippets/en/_includes/enterprise-cloud-only.md
<EnterpriseCloudOnly/>
</Info>

An automation can start when a specific event occurs within a project or registry. This page describes the events that can trigger an automation within each scope. Learn more about automations in the [Automations overview](/models/automations/) or [Create an automation](/models/automations/create-automations/).
An automation can start when a specific event occurs within a project or registry. Learn more about automations in the [Automations overview](/models/automations/) or [Create an automation](/models/automations/create-automations/).

## Registry
This section describes the scopes and events for an automation in a [Registry](/models/registry/).
| Action | Registry scope | Project scope |
|--------------------------------------------------|----------------|---------------|
| [Artifact version](#artifact-events) linked | ✓ | |
| [Artifact version](#artifact-events) aliased | ✓ | ✓ |
| [Artifact version](#artifact-events) created | | ✓ |
| [Artifact version](#artifact-events) tagged | | ✓ |
| [Run status](#run-status-change) changed | | ✓ |
| [Run metric](#run-metrics-change) threshold met | | ✓ |

1. Navigate to the W&B Registry at `https://wandb.ai/registry/`.
1. Click the name of a registry, then view and create automations in the **Automations** tab.
## Artifact events
In a particular registry, an automation can start when an artifact version is linked or aliased. Automations in a given registry apply to all of the registry's collections.

<Frame>
![Screenshot of the Registry Automations tab with an automation](/images/automations/registry_automations_tab.png)
</Frame>

Learn more about [creating automations](/models/automations/create-automations/).

### Scopes
A [Registry](/models/registry/) automation watches for the event taking place on any collection within a specific registry, including collections added in the future.

### Events <a id="registry-events"></a>
A Registry automation can watch for these events:
- **A new version is linked to a collection**: Test and validate new models or datasets when they are added to a registry.
- **An artifact alias is added**: Trigger a specific step of your workflow when a new artifact version has a specific alias applied. For example, deploy a model when it has the `production` alias applied.

## Project
This section describes the scopes and events for an automation in a [project](/models/track/project-page/).

1. Navigate to your W&B project on the W&B App at `https://wandb.ai/<team>/<project-name>`.
1. View and create automations in the **Automations** tab.

<Frame>
![Screenshot of the Project Automations tab with an automation](/images/automations/project_automations_tab.png)
</Frame>

Learn more about [creating automations](/models/automations/create-automations/).

### Scopes
A project-level automation watches for the event taking place on any collection in the project. Depending on the event you specify, you can further limit the scope of the automation.

### Artifact events
This section describes the events related to an artifact that can trigger an automation.

- **A new version is added to an artifact**: Apply recurring actions to each version of an artifact. For example, start a training job when a new dataset artifact version is created. To limit the automation's scope, select a specific artifact in the **Artifact filter** field.
- **An artifact alias is added**: Trigger a specific step of your workflow when a new artifact version in a project has an alias applied that matches the **Alias regex** you specify. For example, run a series of downstream processing steps when an artifact has the `test-set-quality-check` alias applied, or run a workflow each time a new artifact version has the `latest` alias. Only one artifact version can have a given alias at a point in time.
- **An artifact tag is added**: Trigger a specific step of your workflow when an artifact version in a project has a tag applied that matches the **Tag regex** you specify. For example, specify `^europe.*` to trigger a geo-specific workflow when a tag beginning with the string `europe` is added to an artifact version. Artifact tags are used for grouping and filtering, and a given tag can be assigned to multiple artifact versions simultaneously.
In a project, an automation can start when an artifact version is created, tagged, or aliased. Project automations are scoped to the team and apply to all of the project's collections.

### Run events
An automation can be triggered by a change in a [run's status](/models/#run-states) or a change in a [metric value](/models/track/log/#what-data-is-logged-with-specific-wb-api-calls).
Regular expressions are supported for aliases and tags.

#### Run status change
## Run status change
<Note>
- Currently available only in [W&B Multi-tenant Cloud](/platform/hosting/#wb-multi-tenant-cloud).
- A run with **Killed** status cannot trigger an automation. This status indicates that the run was stopped forcibly by an admin user.
Currently available only in [W&B Multi-tenant Cloud](/platform/hosting/#wb-multi-tenant-cloud).
</Note>

Trigger a workflow when a run changes its [status](/models/runs/#run-states) to **Running**, **Finished**, or **Failed**. Optionally, you can further limit the runs that can trigger an automation by specifying a user or run name filter.
In a project, an automation can start when a run changes its [status](/models/runs/#run-states) to **Running**, **Finished**, or **Failed**. An automation cannot start when a run's status is **Killed**. This status indicates that the run was stopped forcibly by an admin user.

Optionally, you can further limit the runs that can trigger an automation by specifying a [user or run name filter](#run-filters).

Because run status is a property of the entire run, you can create a run status automation only from the **Automations** page.

<Frame>
![Screenshot showing a run status change automation](/images/automations/run_status_change.png)
</Frame>

Because run status is a property of the entire run, you can create a run status automation only from the the **Automations** page, not from a workspace.

#### Run metrics change
## Run metric events
<Note>
Currently available only in [W&B Multi-tenant Cloud](/platform/hosting/#wb-multi-tenant-cloud).
</Note>

Trigger a workflow based on a logged value for a metric, either a metric in a run's history or a [system metric](/models/ref/python/experiments/system-metrics) such as `cpu`, which tracks the percentage of CPU utilization. W&B logs system metrics automatically every 15 seconds.
In a project, an automation can start when a logged value for a metric meets a given threshold or changes by a given threshold. A [system metric](/models/ref/python/experiments/system-metrics), such as `cpu`, can trigger an automation. W&B logs system metrics automatically every 15 seconds.

You can create a run metrics automation from the project's **Automations** tab or directly from a line plot panel in a workspace.

To set up a run metric automation, you configure how to compare the metric's value with the threshold you specify. Your choices depend on the event type and on any filters you specify.

Optionally, you can further limit the runs that can trigger an automation by specifying a user or run name filter.
Optionally, you can further limit the runs that can trigger an automation by specifying a [user or run name filter](#run-filters).

##### Threshold
### Threshold
For **Run metrics threshold met** events, you configure:
1. The window of most recently logged values to consider (defaults to 5).
1. Whether to evaluate the **Average**, **Min**, or **Max** value within the window.
Expand All @@ -99,7 +71,7 @@ For example, trigger an automation when average `accuracy` is above `.6`.
![Screenshot showing a run metrics threshold automation](/images/automations/run_metrics_threshold_automation.png)
</Frame>

##### Change threshold
### Change threshold
For **Run metrics change threshold met** events, the automation uses two "windows" of values to check whether to start:

- The _current window_ of recently logged values to consider (defaults to 10).
Expand All @@ -122,7 +94,7 @@ For example, trigger an automation when average `loss` decreases by at least `.2
![Screenshot showing a run metrics change threshold automation](/images/automations/run_metrics_change_threshold_automation.png)
</Frame>

#### Run filters
## Run filters
This section describes how the automation selects runs to evaluate.

- By default, any run in the project triggers the automation when the event occurs. You can limit which runs trigger an automation by configuring one of the following filters:
Expand Down
1 change: 1 addition & 0 deletions models/automations/create-automations.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ For details, refer to:
View and manage automations from a project or registry's **Automations** tab.

- To view an automation's details, click its name.
- To view an automation's execution history, click its name and select the **History** tab. See [View an automation's history](/models/automations/view-automation-history) for details.
- To edit an automation, click its action `...` menu, then click **Edit automation**.
- To delete an automation, click its action `...` menu, then click **Delete automation**.

Expand Down
107 changes: 107 additions & 0 deletions models/automations/view-automation-history.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
---
title: View an automation's history
---

<Note>
This feature requires an Enterprise license. It is available only for:
- W&B Multi-tenant Cloud
- W&B Dedicated Cloud
- W&B Self-Managed v0.75.0+
</Note>

This page describes how to view and understand the execution history of your W&B [automations](/models/automations), including what triggered an automation, what actions were taken, and whether they succeeded or failed.

## Overview

Each executed automation generates a record that includes:
- **Execution timestamp**: When the automation was triggered.
- **Triggering event**: The specific event triggered the automation.
- **Status**: The execution's status. See [Execution status](#execution-status).
- **Action details**: Information about what action was performed, such as notifying a Slack channel or running a webhook.
- **Result details**: Additional information, if any, about the final outcome of executing the automation, including the error for a failed execution.

## View automation history

You can view automation history from multiple locations in the W&B App:

### From the Automations tab

<Tabs>
<Tab title="Registry">
1. Navigate to your registry by clicking on **Registry** in the left sidebar.
1. Select your registry from the list.
1. Click **Automations** tab to view the registry's automations. In each row, the **Last execution** column shows when the automation last executed.
1. In the **Automations history** tab, view all executions of the registry's automations in reverse chronological order, starting with the most recent execution. Each execution's metadata displays, including the event, action, and status.
</Tab>
<Tab title="Project">
1. Click the **Automations** tab in the left navigation. The project's automations display.

- Find the automation you want to investigate. You can use the search bar to filter by automation name, and you can sort by the last triggered date to find recently executed automations.

- Click an automation's name to view or edit its configuration.
1. In the **History** tab, view all executions of the project's automations in reverse chronological order, starting with the most recent execution. Each execution's metadata displays, including the event, action, and status. Click an execution timestamp to view details about a particular execution.
</Tab>
</Tabs>

### From the automation details page

1. Navigate to your project or registry.
1. Click the **Automations** tab.
1. Click an automation name to view or edit its configuration.
1. The **Execution history** section shows all executions for this specific automation. lick an execution timestamp to view details about a particular execution.

### From a specific entity

<Tabs>
<Tab title="Registry">
1. Navigate to your registry.
1. Select a specific registered model or artifact collection.
1. Click the **...** menu in the top right.
1. Select **View history** to see all automation executions related to this entity.
</Tab>
<Tab title="Project">
1. Navigate to your project.
1. Select a specific run or artifact.
1. In the Overview tab, look for the **Triggered automations** section.
1. Click on any listed automation to view its execution details.
</Tab>
</Tabs>

## Understanding execution details

### Execution status

Each automation execution has one of the following statuses:

- **Finished**: The automation completed all actions successfully.
- **Failed**: The automation encountered an error and could not complete.
- **Pending**: The automation is queued for execution.


### Execution details

Click on any execution in the history to view:

- **Event details**: The specific event that triggered the automation, including:
- Event type (e.g., "New artifact version", "Run completed")
- Entity information (run ID, artifact name, etc.)
- User who triggered the event (if applicable)

- **Action details**: Information about what the automation attempted to do:
- Action type (Slack notification or webhook)
- Target (Slack channel or webhook URL)
- Payload sent (for webhooks)

- **Result information**:
- Response status (for webhooks)
- Any error messages or stack traces (for failed executions)

## Find an automation execution

To find an automation execution, you can search by automation name, or you can sort the list of executions by any column.

## Next steps

- [Create an automation](/models/automations/create-automations)
- Learn about [Automation events and scopes](/models/automations/automation-events)
- [Create a secret](/platform/secrets)
4 changes: 0 additions & 4 deletions models/track/project-page.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -189,10 +189,6 @@ Click the **Group** button above the table to group all rows by the value in a p
## Automations tab
Automate downstream actions for versioning artifacts. To create an automation, define trigger events and resulting actions. Actions include executing a webhook or launching a W&B job. For more information, see [Automations](/models/automations/).

<Frame>
<img src="/images/automations/project_automations_tab.png" alt="Automation tab" />
</Frame>

## Sweeps tab

Start a new [sweep](/models/sweeps/) from your project.
Expand Down
Loading