Skip to content

Support for Azure Default Credentials in the Scaler #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Support for Azure Default Credentials in the Scaler #68

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a962eca
Support for Default Azure Identity
karpikpl Jun 11, 2024
5b13abd
Support for Default Azure Identity
karpikpl Jun 11, 2024
d435639
typo
karpikpl Jun 11, 2024
61df8d9
Fixing tests
karpikpl Jun 11, 2024
3dc96e3
logging
karpikpl Jun 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -452,3 +452,4 @@ $RECYCLE.BIN/
!.vscode/tasks.json
!.vscode/launch.json
!.vscode/extensions.json
src/Scaler.Demo/OrderProcessor/appsettings.Development.json
25 changes: 25 additions & 0 deletions .vscode/launch.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,31 @@
"name": ".NET Core Attach",
"type": "coreclr",
"request": "attach"
},
{
// Use IntelliSense to find out which attributes exist for C# debugging
// Use hover for the description of the existing attributes
// For further information visit https://github.com/OmniSharp/omnisharp-vscode/blob/master/debugger-launchjson.md
"name": "Order Processor",
"type": "coreclr",
"request": "launch",
"preLaunchTask": "build",
// If you have changed target frameworks, make sure to update the program path.
"program": "${workspaceFolder}/src/Scaler.Demo/OrderProcessor/bin/Debug/net6.0/Keda.CosmosDb.Scaler.Demo.OrderProcessor.dll",
"args": [],
"cwd": "${workspaceFolder}/src/Scaler.Demo/OrderProcessor",
"stopAtEntry": false,
// Enable launching a web browser when ASP.NET Core starts. For more information: https://aka.ms/VSCode-CS-LaunchJson-WebBrowser
"serverReadyAction": {
"action": "openExternally",
"pattern": "\\bNow listening on:\\s+(https?://\\S+)"
},
"env": {
"ASPNETCORE_ENVIRONMENT": "Development",
},
"sourceFileMap": {
"/Views": "${workspaceFolder}/Views"
}
}
]
}
14 changes: 9 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,12 @@ The specification below describes the `trigger` metadata in `ScaledObject` resou
- type: external
metadata:
scalerAddress: external-scaler-azure-cosmos-db.keda:4050 # Mandatory. Address of the external scaler service.
connection: <connection> # Mandatory. Connection string of Cosmos DB account with monitored container.
connection: <connection> # Optional. Connection string of Cosmos DB account with monitored container. Either `connection` or `endpoint` has to be provided.
endpoint: <endpoint> # Optional. Cosmos DB endpoint with monitored container. Either `connection` or `endpoint` has to be provided.
databaseId: <database-id> # Mandatory. ID of Cosmos DB database containing monitored container.
containerId: <container-id> # Mandatory. ID of monitored container.
leaseConnection: <lease-connection> # Mandatory. Connection string of Cosmos DB account with lease container.
leaseConnection: <lease-connection> # Optional. Connection string of Cosmos DB account with lease container. Either `leaseConnection` or `leaseEndpoint` has to be provided.
leaseEndpoint: <lease-endpoint> # Optional. Cosmos DB endpoint with lease container. Either `leaseConnection` or `leaseEndpoint` has to be provided.
leaseDatabaseId: <lease-database-id> # Mandatory. ID of Cosmos DB database containing lease container.
leaseContainerId: <lease-container-id> # Mandatory. ID of lease container.
processorName: <processor-name> # Mandatory. Name of change-feed processor used by listener application.
Expand All @@ -81,18 +83,20 @@ The specification below describes the `trigger` metadata in `ScaledObject` resou

- **`scalerAddress`** - Address of the external scaler service. This would be in format `<scaler-name>.<scaler-namespace>:<port>`. If you installed Azure Cosmos DB external scaler Helm chart in `keda` namespace and did not specify custom values, the metadata value would be `external-scaler-azure-cosmos-db.keda:4050`.

- **`connection`** - Connection string of the Cosmos DB account that contains the monitored container.
- **`connection`** or **`endpoint`** - Connection string of the Cosmos DB account or Cosmos DB endpoint that contains the monitored container.

- **`databaseId`** - ID of Cosmos DB database that contains the monitored container.

- **`containerId`** - ID of the monitored container.

- **`leaseConnection`** - Connection string of the Cosmos DB account that contains the lease container. This can be same or different from the value of `connection` metadata.
- **`leaseConnection`** or **`leaseEndpoint`**- Connection string of the Cosmos DB account or Cosmos DB endpoint that contains the lease container. This can be same or different from the value of `connection` metadata.

- **`leaseDatabaseId`** - ID of Cosmos DB database that contains the lease container. This can be same or different from the value of `databaseId` metadata.

- **`leaseContainerId`** - ID of the lease container containing the change feeds.

- **`processorName`** - Name of change-feed processor used by listener application. For more information on this, you can refer to [Implementing the change feed processor](https://docs.microsoft.com/azure/cosmos-db/sql/change-feed-processor#implementing-the-change-feed-processor) section.

> **Note** Ideally, we would have created `TriggerAuthentication` resource that would have prevented us from adding the connection strings in plain text in the `ScaledObject` trigger metadata. However, this is not possible since at the moment, the triggers of `external` type do not support referencing a `TriggerAuthentication` resource ([link](https://keda.sh/docs/scalers/external/#authentication-parameters)).
### Workload Identity support

To utilize Azure Workload Identity via Default Azure Credential use **`endpoint`** and **`leaseEndpoint`** parameters.
1 change: 1 addition & 0 deletions src/Scaler.Demo/OrderGenerator/Keda.CosmosDb.Scaler.Demo.OrderGenerator.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Azure.Identity" Version="1.11.4" />
<PackageReference Include="Bogus" Version="34.0.2" />
<PackageReference Include="Microsoft.Azure.Cosmos" Version="3.40.0" />
<PackageReference Include="Microsoft.Extensions.Hosting" Version="6.0.1" />
Expand Down
18 changes: 14 additions & 4 deletions src/Scaler.Demo/OrderGenerator/Program.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
using System;
using System.Linq;
using System.Threading.Tasks;
using Azure.Identity;
using Bogus;
using Bogus.DataSets;
using Keda.CosmosDb.Scaler.Demo.Shared;
Expand Down Expand Up @@ -85,7 +86,11 @@ private static bool ReadIsSingleArticle()

private static async Task CreateOrdersAsync(int count, bool isSingleArticle)
{
Container container = new CosmosClient(_cosmosDbConfig.Connection)
using var cosmosClient = _cosmosDbConfig.Connection.Contains("AccountKey")
? new CosmosClient(_cosmosDbConfig.Connection, new CosmosClientOptions { ConnectionMode = ConnectionMode.Gateway })
: new CosmosClient(_cosmosDbConfig.Connection, new DefaultAzureCredential(), new CosmosClientOptions { ConnectionMode = ConnectionMode.Direct });

Container container = cosmosClient
.GetContainer(_cosmosDbConfig.DatabaseId, _cosmosDbConfig.ContainerId);

int remainingCount = count;
Expand Down Expand Up @@ -127,8 +132,11 @@ private static async Task CreateOrderAsync(Container container, string article)
private static async Task SetupAsync()
{
Console.WriteLine($"Creating database: {_cosmosDbConfig.DatabaseId}");
using var cosmosClient = _cosmosDbConfig.Connection.Contains("AccountKey")
? new CosmosClient(_cosmosDbConfig.Connection, new CosmosClientOptions { ConnectionMode = ConnectionMode.Gateway })
: new CosmosClient(_cosmosDbConfig.Connection, new DefaultAzureCredential(), new CosmosClientOptions { ConnectionMode = ConnectionMode.Direct });

Database database = await new CosmosClient(_cosmosDbConfig.Connection)
Database database = await cosmosClient
.CreateDatabaseIfNotExistsAsync(_cosmosDbConfig.DatabaseId);

Console.WriteLine($"Creating container: {_cosmosDbConfig.ContainerId} with throughput: {_cosmosDbConfig.ContainerThroughput} RU/s");
Expand All @@ -142,12 +150,14 @@ await database.CreateContainerIfNotExistsAsync(

private static async Task TeardownAsync()
{
var client = new CosmosClient(_cosmosDbConfig.Connection);
using var cosmosClient = _cosmosDbConfig.Connection.Contains("AccountKey")
? new CosmosClient(_cosmosDbConfig.Connection, new CosmosClientOptions { ConnectionMode = ConnectionMode.Gateway })
: new CosmosClient(_cosmosDbConfig.Connection, new DefaultAzureCredential(), new CosmosClientOptions { ConnectionMode = ConnectionMode.Direct });

try
{
Console.WriteLine($"Deleting database: {_cosmosDbConfig.DatabaseId}");
await client.GetDatabase(_cosmosDbConfig.DatabaseId).DeleteAsync();
await cosmosClient.GetDatabase(_cosmosDbConfig.DatabaseId).DeleteAsync();
}
catch (CosmosException)
{
Expand Down
1 change: 1 addition & 0 deletions src/Scaler.Demo/OrderProcessor/Keda.CosmosDb.Scaler.Demo.OrderProcessor.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Azure.Identity" Version="1.11.4" />
<PackageReference Include="Microsoft.Azure.Cosmos" Version="3.40.0" />
<PackageReference Include="Microsoft.Extensions.Hosting" Version="6.0.1" />
<PackageReference Include="Microsoft.Extensions.Logging" Version="6.0.0" />
Expand Down
9 changes: 7 additions & 2 deletions src/Scaler.Demo/OrderProcessor/Worker.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
using System.Net;
using System.Threading;
using System.Threading.Tasks;
using Azure.Identity;
using Keda.CosmosDb.Scaler.Demo.Shared;
using Microsoft.Azure.Cosmos;
using Microsoft.Extensions.Hosting;
Expand All @@ -25,7 +26,11 @@ public Worker(CosmosDbConfig cosmosDbConfig, ILogger<Worker> logger)

public override async Task StartAsync(CancellationToken cancellationToken)
{
Database leaseDatabase = await new CosmosClient(_cosmosDbConfig.LeaseConnection)
var cosmosClient = _cosmosDbConfig.Connection.Contains("AccountKey")
Copy link
Collaborator

@JatinSanghvi JatinSanghvi Jul 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @karpikpl, the PR looks good overall. This particular cosmosClient should be based on LeaseConnection configuration as the sample code aims to work even when the Cosmos DB accounts are different for the monitored and lease containers. I have a couple of minor changes planned and should be able to fix this one too in the new PR if that's okay with you.

I have a few questions on AAD identity-based access. Would be helpful if you have answer to some or all of them.

  1. Were you able to test the demo code with default credentials? I had tough time trying to set the permissions as none of the existing role (and with custom role) seem to give me access to create database, etc.
  2. Do we have steps to allow use of default credentials inside locally running Docker container?
  3. Do we know the minimum (or a superset of) permissions required by the Cosmos DB scaler to be able to check change feeds and generate necessary KEDA metrics? If so, I can document and publish it in this repo.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@karpikpl - Reminder on this one. Let me know if you know answers to any of the questions.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JatinSanghvi I am highly interested in this one as well.

Were you able to test the demo code with default credentials? I had tough time trying to set the permissions as none of the existing role (and with custom role) seem to give me access to create database, etc.

Do you want me to take a stab at this?

You would most likely need a control plane role here:

Do we have steps to allow use of default credentials inside locally running Docker container?

I believe you would use Account Key based authentication there - there's no robust need or support for that imo.

Do we know the minimum (or a superset of) permissions required by the Cosmos DB scaler to be able to check change feeds and generate necessary KEDA metrics? If so, I can document and publish it in this repo.

here's a link to the built-in data plane roles

I believe you are looking for: Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/readChangeFeed

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cmeyertons , one of the teams in Microsoft is looking for adding support for accessing Cosmos DB account using Azure Kubernetes Service's managed identity. However, we noticed that supporting managed identities requires referencing TriggerAuthentication from the ScaledObject and that is currently not supported for external scalers AFAIK.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: The feature described in here is not generally supported and unless publicly documented, they should not be used on production deployments. There are based on internal documentation, and I haven't tested them to verify accuracy.

The roles to create/replace databases and containers are respectively as follows:

  • Microsoft.DocumentDB/databaseAccounts/sqlDatabases/write
  • Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/write

The following glob-expansion actions are also allowed:

  • Microsoft.DocumentDB/databaseAccounts/sqlDatabases/*
  • Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/*
  • Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/storedProcedures/*
  • Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/triggers/*
  • Microsoft.DocumentDB/databaseAccounts/throughputSettings/*

With glob expansion, the minimal set of actions needed to get access to all resources under a CosmosDB account are as follows:

  • Microsoft.DocumentDB/databaseAccounts/readMetadata
  • Microsoft.DocumentDB/databaseAccounts/sqlDatabases/*
  • Microsoft.DocumentDB/databaseAccounts/throughputSettings/*

These are expanded RBAC actions, so they will need to be stored in JSON file as follows to be able to apply them:

{
    "RoleName": "ExpandedRBACActions",
    "Type": "CustomRole",
    "AssignableScopes": ["/"],
    "Permissions": [{
        "DataActions": [
            "Microsoft.DocumentDB/databaseAccounts/readMetadata",
            "Microsoft.DocumentDB/databaseAccounts/sqlDatabases/*",
            "Microsoft.DocumentDB/databaseAccounts/throughputSettings/*"
        ]
    }]
}

PowerShell commands to create role using Azure CLI:

# Create RoleDefinition.
az cosmosdb sql role definition create --account-name $accountName --resource-group $resourceGroupName --body expandedActions.json

# Create RoleAssignment.
az cosmosdb sql role assignment create --account-name $accountName --resource-group $resourceGroupName  --role-definition-name "ExpandedRBACActions" --scope "/" --principal-id $principalId

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: The feature described in here is not generally supported and unless publicly documented, they should not be used on production deployments.

The external scaler is effectively a pod running outside of KEDA that KEDA talks to right? so the only steps necessary are just making sure that pod is appropriately decorated with the correct pod labels to run under the right service account as described in the workload identity docs - why is TriggerAuthentication necessary?

Is it non-trivial to provide an example of setting up the external autoscaler with the correct labels + service account to use workload identity? What's the gap I'm missing here after the code correctly supports it?

It sounds like you did the hard part already of defining the correct permissions needed for the role, which is great!

I agree that we cannot use the generic TriggerAuthentication here, but an Azure-first / AKS example will cover majority of use cases (I think they will be high overlap with AKS + Cosmos usage) - going cross-cloud is most likely a more niche use case where non-Entra authentication works fine.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apology, this PR fell of my radar.
The customer I was working with, decided not to go with Cosmos Scaler.

  1. Were you able to test the demo code with default credentials? I had tough time trying to set the permissions as none of the existing role (and with custom role) seem to give me access to create database, etc.

I've tested with workload identity in my AKS cluster

  1. Do we have steps to allow use of default credentials inside locally running Docker container?

That's fairly painful but there's a way. I usually include Azure CLI in one of the layers. Easiest way is to provide client id / client secret / tenant as environment variables from your service principal.

  1. Do we know the minimum (or a superset of) permissions required by the Cosmos DB scaler to be able to check change feeds and generate necessary KEDA metrics? If so, I can document and publish it in this repo.

I can find those

? new CosmosClient(_cosmosDbConfig.Connection)
: new CosmosClient(_cosmosDbConfig.Connection, new DefaultAzureCredential());

Database leaseDatabase = await cosmosClient
.CreateDatabaseIfNotExistsAsync(_cosmosDbConfig.LeaseDatabaseId, cancellationToken: cancellationToken);

Container leaseContainer = await leaseDatabase
Expand All @@ -37,7 +42,7 @@ public override async Task StartAsync(CancellationToken cancellationToken)
// Change feed processor instance name should be unique for each container application.
string instanceName = $"Instance-{Dns.GetHostName()}";

_processor = new CosmosClient(_cosmosDbConfig.Connection)
_processor = cosmosClient
.GetContainer(_cosmosDbConfig.DatabaseId, _cosmosDbConfig.ContainerId)
.GetChangeFeedProcessorBuilder<Order>(_cosmosDbConfig.ProcessorName, ProcessOrdersAsync)
.WithInstanceName(instanceName)
Expand Down
Loading