feat(*): Autonomous mode #75
Conversation
Conflicts: cmd/root.go
|
Tests from 2025-11-07 were conclusive for the fix on pulling large number of IPs with batch technique. |
rr404
left a comment
rr404
left a comment
There was a problem hiding this comment.
sorry forgot to push review
| * @param {KVNamespace} kvNamespace - Cloudflare KV namespace | ||
| */ | ||
| export async function markAsWarmed(kvNamespace) { | ||
| await kvNamespace.put(WARMUP_KEY, 'true'); |
There was a problem hiding this comment.
Maybe renamed it WARMED_UP. When i first read it I thought it meant the opposite
There was a problem hiding this comment.
That would be more correct written : WARMED_UP
| const origins = env.ONLY_INCLUDE_ORIGINS ? env.ONLY_INCLUDE_ORIGINS.split(',').map((s) => s.trim()) : []; | ||
|
|
||
| // Fetch decisions from LAPI | ||
| const decisions = await fetchDecisionsStream(lapiUrl, env.LAPI_KEY, { |
There was a problem hiding this comment.
pourrait t'on faire un test de charge, voir s'il timeout pas si on lui balance 100k+ ips
Meme s'il les met pas toutes en KV dû aux limitations FREE, juste voir s'il timeout pas au startup sur de GROSSES subscriptions
| // Step 1: Get existing IP_RANGES from KV | ||
| const existingRanges = await getIpRanges(env.CROWDSECCFBOUNCERNS); | ||
| // Step 2: Get all string-based keys (IP, AS, Country) that we need to check | ||
| const allStringKeys = [ |
There was a problem hiding this comment.
pareil ça à timer, voir si ce filter est pas un peu heavy et qu'il faille trouver une alternative.
On peut tester ça en local d'ailleurs pour le timer.
There was a problem hiding this comment.
je me dis que les workers Cloudflare un peu comme des lambda AWS ça aura pas un temps d'execution illimité autorisé ils veulent surmeent que ça soit plutot light weight comme scripts
|
|
||
| import logger from '../utils/logger.js'; | ||
|
|
||
| const BATCH_SIZE = 10000; // Cloudflare KV limit for batch operations |
There was a problem hiding this comment.
Ca pose peut etre problème si ta limite est de 1000 en free, peut etre pour du free passer le batch size à 100 ? pouvoir le changer ?
| }); | ||
|
|
||
| // Write each entry in the batch | ||
| const promises = batch.map((entry) => kvNamespace.put(entry.key, entry.value)); |
There was a problem hiding this comment.
tu fais des put individuels, du coup je comprend plus la notion de batch et batch size
There was a problem hiding this comment.
c'est par rapport aux promises ?
Description
decisions-sync-workerthat synchronizes decisions using a scheduled task and updates KV accordingly-Soption to configure the Cloudflare infrastructure and deploy this new worker. This is what we call "autonomous mode": the background go process is not needed anymore, and the deployed bouncers are sufficient to perform the bouncer role: one worker to synchronize decisions (decisions-sync-worker) and one worker to apply decisions (crowdsec-cloudflare-worker-bouncer)How to test
-Soption./crowdsec-cloudflare-worker-bouncer -S -c <PATH_TO_CONFIG.yaml>decisions-sync-workerworker is not deployed):./crowdsec-cloudflare-worker-bouncer -c <PATH_TO_CONFIG.yaml>Others
For the related documentation, please refer to the related PR