Vmodak/pdm aiq agent #2
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add vanna, chromadb, bokeh, langchain_nvidia, tqdm, numpy, pandas, joblib - These packages were being imported but not declared as dependencies - Now 'uv pip install -e .' will install all required packages out of the box
- Add config-plotting-reasoning.yml based on proven reasoning config - Fix VannaManager unit counting SQL with automatic training examples - Implement golden plot generator with histogram and multi-line support - Update evaluation dataset with detailed expected answers - Resolve 429 rate limiting, parsing errors, and schema issues - Achieve 100% chart generation success rate in evaluation - Generate all 7 golden plots with data validation
- Add Evaluation (Optional) section after Observability - Include commands for simple queries (config-reasoning.yml) - Include commands for complex reasoning + plotting (config-plotting-reasoning.yml) - Document evaluation result locations and file descriptions - Maintain consistent formatting with existing optional sections
- Implemented unified master configuration (config-master.yml) for all query types - Consolidated evaluation datasets (eval_set.json + eval_set_master.json) - Organized backup directory with cleaned redundant files - Updated README with latest evaluation strategy and performance metrics (87% accuracy) - Added intelligent query classification for RUL lookups vs predictions - Cleaned up 30+ redundant files from root directory - Preserved essential directories with .gitkeep files Key improvements: - Single config handles text, prediction, and visualization queries - Fixed RUL query classification (100% accuracy on simple lookups) - Comprehensive 23-query evaluation dataset with performance insights - Clean repository structure with organized backup files
…cal consumption - Remove backup/ directory from git tracking while preserving local files - Add comprehensive .gitignore rules for: - backup/ directories (for local consumption only) - output_data/* and eval_output/* (except .gitkeep) - database files and dynamic folders - temporary and debug files - Maintain directory structure with .gitkeep files The backup folder contains organized old configs, evaluation data, and generated outputs that are useful for local reference but should not be committed to remote repository.
- Implement plot_analyzer_tool.py that uses reasoning LLM to analyze plot data - Generate natural language descriptions for better evaluation accuracy - Update config-master.yml with plot_analyzer integration and dynamic file coordination - Fix agent workflow to use actual file paths instead of placeholders - Add tool registration in __init__.py and register.py - Remove problematic langchain_nvidia dependency This significantly improves evaluation results by providing detailed plot analysis instead of generic metadata.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.