Skip to content

Commit f35550f

Browse files
robtaylorclaude
robtaylor
and
claude
committed
Add advanced tutorial testing workflows using Claude
Created two new GitHub workflows: 1. Tutorial Comprehension Test: - Uses Claude to analyze the tutorial for consistency and comprehensibility - Checks code examples for correctness - Assesses the tutorial's quality for beginners - Identifies potential improvements 2. Tutorial Execution Test: - Uses Claude to extract executable steps from the tutorial - Automatically runs each code example - Records and analyzes execution results - Provides detailed feedback on example executability - Archives all generated outputs as workflow artifacts 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
1 parent e6fa86a commit f35550f

File tree

2 files changed

+344
-0
lines changed

2 files changed

+344
-0
lines changed

‎.github/workflows/tutorial-comprehension-test.yml‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
name: Tutorial Comprehension Test
2+
3+
on:
4+
push:
5+
branches: [ main ]
6+
paths:
7+
- 'tutorial.md'
8+
- '.github/workflows/tutorial-comprehension-test.yml'
9+
pull_request:
10+
branches: [ main ]
11+
paths:
12+
- 'tutorial.md'
13+
- '.github/workflows/tutorial-comprehension-test.yml'
14+
workflow_dispatch: # Allow manual trigger
15+
16+
jobs:
17+
analyze-tutorial:
18+
name: Analyze Tutorial with Claude
19+
runs-on: ubuntu-latest
20+
21+
steps:
22+
- name: Checkout code
23+
uses: actions/checkout@v3
24+
25+
- name: Set up Node.js
26+
uses: actions/setup-node@v3
27+
with:
28+
node-version: '18'
29+
30+
- name: Install Anthropic SDK
31+
run: npm install @anthropic-ai/sdk
32+
33+
- name: Create tutorial analysis script
34+
run: |
35+
cat > analyze_tutorial.js << 'EOF'
36+
const fs = require('fs');
37+
const Anthropic = require('@anthropic-ai/sdk');
38+
39+
// Initialize Anthropic client
40+
const anthropic = new Anthropic({
41+
apiKey: process.env.ANTHROPIC_API_KEY,
42+
});
43+
44+
async function analyzeTutorial() {
45+
// Read the tutorial content
46+
const tutorialContent = fs.readFileSync('tutorial.md', 'utf8');
47+
48+
// Create the prompt for Claude
49+
const prompt = `<tutorial>
50+
${tutorialContent}
51+
</tutorial>
52+
53+
You are an expert in hardware design, HDLs, and educational content. Please analyze the above Amaranth HDL tutorial and perform the following tasks:
54+
55+
1. Consistency check:
56+
- Are all code examples syntactically correct?
57+
- Do the examples align with the explanations?
58+
- Are there any missing dependencies or imports?
59+
- Would a beginner be able to run these examples without errors?
60+
61+
2. Comprehensibility assessment:
62+
- How well does the tutorial explain hardware concepts to beginners?
63+
- Are there any concepts that need better explanation?
64+
- Is the progression of examples logical?
65+
- Are there any gaps in the learning journey?
66+
67+
3. Identify any potential improvements:
68+
- What could make this tutorial more effective?
69+
- Are there missing explanations for important concepts?
70+
- What additional examples might be helpful?
71+
72+
Provide your assessment in a structured format with clear headings and bullet points.`;
73+
74+
try {
75+
console.log("Sending request to Claude...");
76+
77+
// Call Claude with the prompt
78+
const response = await anthropic.messages.create({
79+
model: "claude-3-opus-20240229",
80+
max_tokens: 4000,
81+
messages: [
82+
{ role: "user", content: prompt }
83+
],
84+
temperature: 0.2,
85+
});
86+
87+
// Write Claude's analysis to a file
88+
fs.writeFileSync('tutorial_analysis.md', response.content[0].text);
89+
console.log("Analysis complete. Results written to tutorial_analysis.md");
90+
91+
// Also print a summary to the console
92+
console.log("\n=== SUMMARY OF ANALYSIS ===\n");
93+
console.log(response.content[0].text.substring(0, 1000) + "...");
94+
95+
} catch (error) {
96+
console.error("Error calling Claude API:", error);
97+
process.exit(1);
98+
}
99+
}
100+
101+
analyzeTutorial();
102+
EOF
103+
104+
chmod +x analyze_tutorial.js
105+
106+
- name: Analyze tutorial with Claude
107+
env:
108+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
109+
run: node analyze_tutorial.js
110+
111+
- name: Archive analysis results
112+
uses: actions/upload-artifact@v3
113+
with:
114+
name: tutorial-analysis
115+
path: tutorial_analysis.md

‎.github/workflows/tutorial-execution-test.yml‎

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
name: Tutorial Execution Test with Claude
2+
3+
on:
4+
push:
5+
branches: [ main ]
6+
paths:
7+
- 'tutorial.md'
8+
- '.github/workflows/tutorial-execution-test.yml'
9+
pull_request:
10+
branches: [ main ]
11+
paths:
12+
- 'tutorial.md'
13+
- '.github/workflows/tutorial-execution-test.yml'
14+
workflow_dispatch: # Allow manual trigger
15+
16+
jobs:
17+
execute-tutorial:
18+
name: Execute Tutorial with Claude
19+
runs-on: ubuntu-latest
20+
21+
steps:
22+
- name: Checkout code
23+
uses: actions/checkout@v3
24+
25+
- name: Set up Python and Node.js
26+
uses: actions/setup-node@v3
27+
with:
28+
node-version: '18'
29+
30+
- name: Install Anthropic SDK
31+
run: npm install @anthropic-ai/sdk
32+
33+
- name: Set up Python
34+
uses: actions/setup-python@v4
35+
with:
36+
python-version: '3.9'
37+
38+
- name: Install PDM
39+
run: |
40+
pip install pdm
41+
42+
- name: Install dependencies
43+
run: |
44+
sudo apt-get update
45+
sudo apt-get install -y gtkwave
46+
pdm install
47+
48+
- name: Create tutorial execution script
49+
run: |
50+
cat > execute_tutorial.js << 'EOF'
51+
const fs = require('fs');
52+
const { exec, execSync } = require('child_process');
53+
const Anthropic = require('@anthropic-ai/sdk');
54+
const util = require('util');
55+
const execAsync = util.promisify(exec);
56+
57+
// Initialize Anthropic client
58+
const anthropic = new Anthropic({
59+
apiKey: process.env.ANTHROPIC_API_KEY,
60+
});
61+
62+
async function executeTutorial() {
63+
// Read the tutorial content
64+
const tutorialContent = fs.readFileSync('tutorial.md', 'utf8');
65+
66+
// First, have Claude analyze the tutorial and extract executable steps
67+
const analysisPrompt = `<tutorial>
68+
${tutorialContent}
69+
</tutorial>
70+
71+
You are an expert in hardware design, HDLs, and Python. Please analyze the above Amaranth HDL tutorial and extract a step-by-step execution plan.
72+
73+
For each code example in the tutorial:
74+
1. Identify the filename it should be saved as
75+
2. Extract the exact code as shown in the tutorial
76+
3. Identify any dependencies or prerequisites needed to run this code
77+
4. Describe what the expected output or result should be
78+
79+
Format your response in JSON like this:
80+
{
81+
"steps": [
82+
{
83+
"name": "Step description",
84+
"file": "filename.py",
85+
"code": "Python code goes here",
86+
"dependencies": ["list", "of", "dependencies"],
87+
"expected_result": "Description of expected output",
88+
"validation": "How to verify it worked correctly"
89+
}
90+
]
91+
}
92+
93+
Only include steps that involve executing code. Focus on extracting the examples exactly as shown.`;
94+
95+
try {
96+
console.log("Analyzing tutorial to extract executable steps...");
97+
98+
// Call Claude to analyze the tutorial
99+
const analysisResponse = await anthropic.messages.create({
100+
model: "claude-3-sonnet-20240229",
101+
max_tokens: 4000,
102+
messages: [
103+
{ role: "user", content: analysisPrompt }
104+
],
105+
temperature: 0.2,
106+
});
107+
108+
// Parse Claude's response to get the execution plan
109+
const analysisText = analysisResponse.content[0].text;
110+
111+
// Extract JSON from Claude's response
112+
const jsonMatch = analysisText.match(/\{[\s\S]*\}/);
113+
if (!jsonMatch) {
114+
throw new Error("Could not extract JSON execution plan from Claude's response");
115+
}
116+
117+
const executionPlan = JSON.parse(jsonMatch[0]);
118+
fs.writeFileSync('execution_plan.json', JSON.stringify(executionPlan, null, 2));
119+
console.log(`Extracted ${executionPlan.steps.length} executable steps from tutorial`);
120+
121+
// Execute each step in the plan
122+
const results = [];
123+
124+
for (let i = 0; i < executionPlan.steps.length; i++) {
125+
const step = executionPlan.steps[i];
126+
console.log(`\n==== Executing Step ${i+1}: ${step.name} ====`);
127+
128+
// Save the code to a file
129+
fs.writeFileSync(step.file, step.code);
130+
console.log(`Created file: ${step.file}`);
131+
132+
// Execute the code
133+
try {
134+
console.log(`Running: pdm run python ${step.file}`);
135+
const { stdout, stderr } = await execAsync(`pdm run python ${step.file}`, { timeout: 60000 });
136+
137+
// Record the result
138+
results.push({
139+
step: i+1,
140+
name: step.name,
141+
file: step.file,
142+
success: true,
143+
stdout,
144+
stderr,
145+
error: null
146+
});
147+
148+
console.log("Output:", stdout);
149+
if (stderr) console.error("Errors:", stderr);
150+
151+
} catch (error) {
152+
console.error(`Error executing ${step.file}:`, error.message);
153+
154+
// Record the failure
155+
results.push({
156+
step: i+1,
157+
name: step.name,
158+
file: step.file,
159+
success: false,
160+
stdout: error.stdout || "",
161+
stderr: error.stderr || "",
162+
error: error.message
163+
});
164+
}
165+
}
166+
167+
// Save the execution results
168+
fs.writeFileSync('execution_results.json', JSON.stringify(results, null, 2));
169+
170+
// Have Claude analyze the results
171+
const resultsPrompt = `
172+
I've executed the code examples from an Amaranth HDL tutorial. Here are the results:
173+
174+
${JSON.stringify(results, null, 2)}
175+
176+
Please analyze these results and provide:
177+
178+
1. A summary of which examples worked and which failed
179+
2. For failed examples, analyze what might have gone wrong based on error messages
180+
3. Suggest possible improvements to the tutorial based on execution results
181+
4. Overall assessment of the tutorial's executability for beginners
182+
183+
Format your response with clear headings and bullet points.`;
184+
185+
console.log("\nAnalyzing execution results with Claude...");
186+
187+
const resultsAnalysisResponse = await anthropic.messages.create({
188+
model: "claude-3-sonnet-20240229",
189+
max_tokens: 4000,
190+
messages: [
191+
{ role: "user", content: resultsPrompt }
192+
],
193+
temperature: 0.2,
194+
});
195+
196+
// Save Claude's analysis of the results
197+
fs.writeFileSync('tutorial_execution_analysis.md', resultsAnalysisResponse.content[0].text);
198+
console.log("Analysis complete. Results written to tutorial_execution_analysis.md");
199+
200+
console.log("\n=== SUMMARY OF EXECUTION ANALYSIS ===\n");
201+
console.log(resultsAnalysisResponse.content[0].text.substring(0, 1000) + "...");
202+
203+
} catch (error) {
204+
console.error("Error during execution:", error);
205+
process.exit(1);
206+
}
207+
}
208+
209+
executeTutorial();
210+
EOF
211+
212+
chmod +x execute_tutorial.js
213+
214+
- name: Execute tutorial with Claude
215+
env:
216+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
217+
run: node execute_tutorial.js
218+
219+
- name: Archive execution results
220+
uses: actions/upload-artifact@v3
221+
with:
222+
name: tutorial-execution-results
223+
path: |
224+
*.py
225+
*.v
226+
*.vcd
227+
execution_plan.json
228+
execution_results.json
229+
tutorial_execution_analysis.md

0 commit comments

Comments
 (0)