FEAT Breaking: Adding Resiliency to Scenarios (#1164)

Author: rlundeen2

doc/_toc.yml

@@ -120,6 +120,7 @@ chapters:
       - file: code/memory/chat_message
     - file: code/setup/0_configuration
       sections:
+      - file: code/setup/1_resiliency
       - file: code/setup/default_values
       - file: code/setup/pyrit_initializer
     - file: code/auxiliary_attacks/0_auxiliary_attacks

doc/api.rst

@@ -215,6 +215,7 @@ API Reference
     MultiPromptSendingAttack
     MultiPromptSendingAttackContext
     MultiTurnAttackContext
+    AttackExecutorResult
     PromptSendingAttack
     RTASystemPromptPaths
     RedTeamingAttack
@@ -541,7 +542,6 @@ API Reference
     :toctree: _autosummary/
 
     AtomicAttack
-    AtomicAttackResult
     EncodingScenario
     FoundryStrategy
     FoundryScenario

doc/code/scenarios/scenarios.ipynb

@@ -32,10 +32,10 @@
     "\n",
     "## How It Works\n",
     "\n",
-    "Each `Scenario` contains a collection of `AttackRun` objects. When executed:\n",
+    "Each `Scenario` contains a collection of `AtomicAttack` objects. When executed:\n",
     "\n",
-    "1. Each `AttackRun` is executed sequentially\n",
-    "2. Every `AttackRun` tests its configured attack against all specified objectives and datasets\n",
+    "1. Each `AtomicAttack` is executed sequentially\n",
+    "2. Every `AtomicAttack` tests its configured attack against all specified objectives and datasets\n",
     "3. Results are aggregated into a single `ScenarioResult` with all attack outcomes\n",
     "4. Optional memory labels help track and categorize the scenario execution\n",
     "\n",
@@ -64,7 +64,7 @@
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "108387f8a81646c289f28c1e3255bb5a",
+       "model_id": "ecb802c8f9964212b3f6c3ff7a416e79",
        "version_major": 2,
        "version_minor": 0
       },
@@ -168,6 +168,7 @@
     "    ScenarioCompositeStrategy(strategies=[FoundryStrategy.Caesar, FoundryStrategy.CharSwap]),  # Composed strategy\n",
     "]\n",
     "\n",
+    "\n",
     "# Create a scenario from the pre-configured Foundry scenario\n",
     "foundry_scenario = FoundryScenario(\n",
     "    objective_target=objective_target,\n",
@@ -183,6 +184,30 @@
     "foundry_results = await foundry_scenario.run_async() # type: ignore\n",
     "await printer.print_summary_async(foundry_results) # type: ignore"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2",
+   "metadata": {},
+   "source": [
+    "## Resiliency\n",
+    "\n",
+    "Scenarios can run for a long time, and because of that, things can go wrong. Network issues, rate limits, or other transient failures can interrupt execution. PyRIT provides built-in resiliency features to handle these situations gracefully.\n",
+    "\n",
+    "### Automatic Resume\n",
+    "\n",
+    "If you re-run a `scenario`, it will automatically start where it left off. The framework tracks completed attacks and objectives in memory, so you won't lose progress if something interrupts your scenario execution. This means you can safely stop and restart scenarios without duplicating work.\n",
+    "\n",
+    "### Retry Mechanism\n",
+    "\n",
+    "You can utilize the `max_retries` parameter to handle transient failures. If any unknown exception occurs during execution, PyRIT will automatically retry the failed operation (starting where it left off) up to the specified number of times. This helps ensure your scenario completes successfully even in the face of temporary issues.\n",
+    "\n",
+    "### Dynamic Configuration\n",
+    "\n",
+    "During a long-running scenario, you may want to adjust parameters like `max_concurrency` to manage resource usage, or switch your scorer to use a different target. PyRIT's resiliency features make it safe to stop, reconfigure, and continue scenarios as needed.\n",
+    "\n",
+    "For more information, see [resiliency](../setup/1_resiliency.ipynb)"
+   ]
   }
  ],
  "metadata": {

doc/code/scenarios/scenarios.py

@@ -6,10 +6,6 @@
 #       format_name: percent
 #       format_version: '1.3'
 #       jupytext_version: 1.17.3
-#   kernelspec:
-#     display_name: pyrit-dev
-#     language: python
-#     name: python3
 # ---
 
 # %% [markdown]
@@ -40,10 +36,10 @@
 #
 # ## How It Works
 #
-# Each `Scenario` contains a collection of `AttackRun` objects. When executed:
+# Each `Scenario` contains a collection of `AtomicAttack` objects. When executed:
 #
-# 1. Each `AttackRun` is executed sequentially
-# 2. Every `AttackRun` tests its configured attack against all specified objectives and datasets
+# 1. Each `AtomicAttack` is executed sequentially
+# 2. Every `AtomicAttack` tests its configured attack against all specified objectives and datasets
 # 3. Results are aggregated into a single `ScenarioResult` with all attack outcomes
 # 4. Optional memory labels help track and categorize the scenario execution
 #
@@ -83,6 +79,7 @@
     ScenarioCompositeStrategy(strategies=[FoundryStrategy.Caesar, FoundryStrategy.CharSwap]),  # Composed strategy
 ]
 
+
 # Create a scenario from the pre-configured Foundry scenario
 foundry_scenario = FoundryScenario(
     objective_target=objective_target,
@@ -97,3 +94,22 @@
 # Execute the entire scenario
 foundry_results = await foundry_scenario.run_async()  # type: ignore
 await printer.print_summary_async(foundry_results)  # type: ignore
+
+# %% [markdown]
+# ## Resiliency
+#
+# Scenarios can run for a long time, and because of that, things can go wrong. Network issues, rate limits, or other transient failures can interrupt execution. PyRIT provides built-in resiliency features to handle these situations gracefully.
+#
+# ### Automatic Resume
+#
+# If you re-run a `scenario`, it will automatically start where it left off. The framework tracks completed attacks and objectives in memory, so you won't lose progress if something interrupts your scenario execution. This means you can safely stop and restart scenarios without duplicating work.
+#
+# ### Retry Mechanism
+#
+# You can utilize the `max_retries` parameter to handle transient failures. If any unknown exception occurs during execution, PyRIT will automatically retry the failed operation (starting where it left off) up to the specified number of times. This helps ensure your scenario completes successfully even in the face of temporary issues.
+#
+# ### Dynamic Configuration
+#
+# During a long-running scenario, you may want to adjust parameters like `max_concurrency` to manage resource usage, or switch your scorer to use a different target. PyRIT's resiliency features make it safe to stop, reconfigure, and continue scenarios as needed.
+#
+# For more information, see [resiliency](../setup/1_resiliency.ipynb)

doc/code/setup/0_configuration.ipynb

@@ -5,7 +5,7 @@
    "id": "0",
    "metadata": {},
    "source": [
-    "# Configuring PyRIT\n",
+    "# 0. Configuring PyRIT\n",
     "\n",
     "Before running PyRIT, you need to call the `initialize_pyrit` function which will set up your configuration.\n",
     "\n",

doc/code/setup/0_configuration.py

@@ -9,7 +9,7 @@
 # ---
 
 # %% [markdown]
-# # Configuring PyRIT
+# # 0. Configuring PyRIT
 #
 # Before running PyRIT, you need to call the `initialize_pyrit` function which will set up your configuration.
 #