feat: Add Azure OpenAI configuration support to crwl config #1580
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add AZURE_API_BASE and AZURE_API_VERSION configuration options to allow users to configure Azure OpenAI settings through `crwl config` instead of requiring environment variables to be set manually. Changes: - Add AZURE_API_BASE setting to USER_SETTINGS in config.py - Add AZURE_API_VERSION setting to USER_SETTINGS in config.py - Update setup_llm_config() in cli.py to export Azure env vars from config - Users can now set Azure settings with: crwl config set AZURE_API_BASE "https://your-resource.openai.azure.com" crwl config set AZURE_API_VERSION "2024-08-01-preview" This improves the user experience for Azure OpenAI users by eliminating the need to export environment variables before each crwl command. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds support for configuring Azure OpenAI settings through
crwl config, eliminating the need to manually set environment variables before each command.Changes
AZURE_API_BASEconfiguration setting toUSER_SETTINGSinconfig.pyAZURE_API_VERSIONconfiguration setting toUSER_SETTINGSinconfig.pysetup_llm_config()incli.pyto export Azure environment variables from the global configurationMotivation
Currently, when using Azure OpenAI with crawl4ai, users must export
AZURE_API_BASEandAZURE_API_VERSIONenvironment variables before running each command:This is inconvenient and error-prone. With this PR, users can configure these settings once:
And then use crawl4ai with Azure OpenAI without any environment variables:
crwl https://example.com -q "What is this about?"Testing
Tested locally with Azure OpenAI GPT-5 deployment:
crwl config set~/.crawl4ai/global.ymlcrwlcommands without setting environment variablesImplementation Details
The
setup_llm_config()function now readsAZURE_API_BASEandAZURE_API_VERSIONfrom the global configuration and exports them as environment variables before LiteLLM is called. This ensures backward compatibility while providing a better user experience.🤖 Generated with Claude Code
Co-Authored-By: Claude [email protected]