feat: consume cogito for agent reasoning #320

mudler · 2025-10-10T20:36:37Z

Blocked by: mudler/cogito#9

some TODO:

Dropping methods used to extract tools/gaps
Write a Action -> cogito.Tool converter
Wire MCP servers with cogito.WithMCPs() and pass by MCP client sessions
For re-evaluating with goals, we can put that under a flag and use the ContentReview on the resulting fragment of the tool execution loop
QA

Signed-off-by: Ettore Di Giacinto <[email protected]>

richiejp · 2025-10-19T12:05:26Z

core/agent/agent.go

-				obs.Progress = append(obs.Progress, types.Progress{
-					ActionResult: res.Result,
-				})
-				a.observer.Update(*obs)


Have lines like this been recreated in the new version?

Signed-off-by: Ettore Di Giacinto <[email protected]>

Copilot

Pull Request Overview

This PR integrates the Cogito library to drive agent reasoning and tool execution, and removes the legacy “old” web UI in favor of the new React UI.

Replace in-house planning/reasoning and MCP tool wrapping with Cogito-based execution and evaluation
Remove legacy server-side templates and HTMX-based UI; simplify Fiber app initialization for React UI
Add Cogito integration points (status callbacks, tool callbacks, MCP sessions); update knowledgebase summarization to use Cogito

Reviewed Changes

Copilot reviewed 34 out of 37 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
webui/routes.go	Removes old embedded views/public routes; keeps React UI assets served.
webui/old/... (multiple)	Deletes legacy HTML/CSS/JS views and assets.
webui/app.go	Drops HTMX/templates engine usage; simplifies Fiber initialization.
go.mod	Adds cogito dependency; updates Go version/toolchain; adjusts deps.
core/types/job.go	Removes nextAction/pastActions; adds cogito.Fragment field.
core/types/actions.go	Adds cogito wrapper to expose Actions as Cogito tools; removes Plannable from interface.
core/agent/templates.go	Simplifies templates to align with Cogito-driven flow.
core/agent/mcp.go	Stops generating Actions for MCP tools; keeps MCP client sessions for Cogito.
core/agent/knowledgebase.go	Uses Cogito LLM to summarize conversation.
core/agent/evaluation.go	Removes custom evaluation; Cogito now handles evaluation.
core/agent/agent.go	Major refactor to Cogito execution, planning, and callbacks; cleans old codepaths.
core/agent/actions.go	Removes custom decision/planning utilities; simplifies available actions.
core/action/* (several)	Removes obsolete actions (reply, reasoning, plan, intention, goal).

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-19T19:54:26Z

core/agent/agent.go

+				chosenAction := availableActions.Find(tc.Name)
+
+				var obs *types.Observable
+				if job.Obs != nil {
+					obs = a.observer.NewObservable()
+					obs.Name = "action"
+					obs.Icon = "bolt"
+					obs.ParentID = job.Obs.ID
+					obs.Creation = &types.Creation{
+						FunctionDefinition: chosenAction.Definition().ToFunctionDefinition(),
+						FunctionParams:     types.ActionParams(tc.Arguments),
+					}


Possible nil pointer dereference if chosenAction is nil (e.g., for MCP tools or tools not in availableActions). Guard chosenAction before accessing Definition() and before later uses (job.Callback, SetResult). If nil, skip FunctionDefinition, and either construct a minimal placeholder for observer/callback or branch handling for non-local tools.

Copilot · 2025-10-19T19:54:26Z

core/agent/agent.go

+				cont := job.Callback(types.ActionCurrentState{
+					Job:       job,
+					Action:    chosenAction,
+					Params:    types.ActionParams(tc.Arguments),
+					Reasoning: reasoning})
+
+				if !cont {
+					job.Result.SetResult(
+						types.ActionState{
+							ActionCurrentState: types.ActionCurrentState{
+								Job:       job,
+								Action:    chosenAction,
+								Params:    types.ActionParams(tc.Arguments),
+								Reasoning: reasoning,
+							},
+							ActionResult: types.ActionResult{Result: "stopped by callback"},
+						})


Same nil dereference risk: chosenAction may be nil for MCP/external tools. Add a nil check and handle callback/result construction accordingly to avoid panics.

Copilot · 2025-10-19T19:54:26Z

core/agent/agent.go

+		toolToCall := fragment.Messages[len(fragment.Messages)-2].ToolCalls[0].Function.Name
+		switch toolToCall {
+		case action.StopActionName:
+			job.Result.Finish(nil)
 			return


Potential out-of-bounds access if fragment.Messages has fewer than 2 entries or if the assistant message has no ToolCalls. Guard lengths before indexing and verify ToolCalls is non-empty to avoid panics.

Suggested change

toolToCall := fragment.Messages[len(fragment.Messages)-2].ToolCalls[0].Function.Name

switch toolToCall {

case action.StopActionName:

job.Result.Finish(nil)

return

// Guard against out-of-bounds access

if len(fragment.Messages) >= 2 {

toolCalls := fragment.Messages[len(fragment.Messages)-2].ToolCalls

if len(toolCalls) > 0 {

toolToCall := toolCalls[0].Function.Name

switch toolToCall {

case action.StopActionName:

job.Result.Finish(nil)

return

}

}

Copilot · 2025-10-19T19:54:27Z

core/agent/knowledgebase.go

+		fragment := cogito.NewEmptyFragment().AddStartMessage("user", "Summarize the conversation below, keep the highlights as a bullet list:\n"+Messages(conv).String())
+		fragment, err := a.llm.Ask(a.context.Context, fragment)
 		if err != nil {
 			xlog.Error("Error summarizing conversation", "error", err)


If Ask returns an error, fragment might be nil or incomplete; calling fragment.LastMessage() can panic. Return early on error (or ensure fragment is valid) before accessing LastMessage.

Suggested change

xlog.Error("Error summarizing conversation", "error", err)

xlog.Error("Error summarizing conversation", "error", err)

return

Signed-off-by: Ettore Di Giacinto <[email protected]>

richiejp · 2025-10-20T08:41:29Z

core/agent/agent.go

+								},
+							},
+						},
+					})


This looks quite suspicious. It seems like this is creating and completing an observable for a decision step upon receiving the result of that decision. Really it should be created when the first request is submitted to the LLM as part of the decision process.

mmm you are right, I think I've did some mistake when porting the logic, can you check again?

I think you need to create the observable when making the request to the LLM and close it when you get the result. I don't know if you have the callbacks for this in Cogito? It's the kind of thing only needed for observability.

Presently the user will just see the decision appear after completion with the whole of conv as the query to the LLM which may or may not be what is actually passed to the LLM.

there is WithStatusCallback which is called with the reasoning, and WithToolCallResultCallback with the tool call result. WithToolCallBack instead accepts a func with a boolean, which if false will stop the tool execution and the agent.

I think I've misread a bit things initially, but now I've moved the update and the closing of the observable to use the callbacks above

Possibly there should be WithReasoningCallback and WithReasoningResultCallback?

Signed-off-by: Ettore Di Giacinto <[email protected]>

core/agent/agent.go

Signed-off-by: Ettore Di Giacinto <[email protected]>

This reverts commit 8b12a9f.

Signed-off-by: Ettore Di Giacinto <[email protected]>

mudler mentioned this pull request Oct 10, 2025

feat(tools): allow to automatically plan mudler/cogito#9

Merged

mudler linked an issue Oct 10, 2025 that may be closed by this pull request

Consume cogito #313

Open

WIP

8c7cf3f

Signed-off-by: Ettore Di Giacinto <[email protected]>

mudler force-pushed the cogito branch from b119fc5 to 8c7cf3f Compare October 15, 2025 16:40

mudler added 6 commits October 15, 2025 19:35

Drop old webui

8b7453a

Signed-off-by: Ettore Di Giacinto <[email protected]>

Almost there

a381208

Signed-off-by: Ettore Di Giacinto <[email protected]>

It builds

58f44a4

Signed-off-by: Ettore Di Giacinto <[email protected]>

Make it build

024b2b3

Signed-off-by: Ettore Di Giacinto <[email protected]>

fixups, still doesn't work

71d747a

Signed-off-by: Ettore Di Giacinto <[email protected]>

unused now

fae0641

Signed-off-by: Ettore Di Giacinto <[email protected]>

mudler force-pushed the cogito branch from 988ccf1 to fae0641 Compare October 18, 2025 21:08

richiejp reviewed Oct 19, 2025

View reviewed changes

mudler added 3 commits October 19, 2025 21:32

Send result before closing

910e642

Signed-off-by: Ettore Di Giacinto <[email protected]>

Fix observability

c088eef

Signed-off-by: Ettore Di Giacinto <[email protected]>

Drop MCP code and wire-up in cogito

77f68bf

Signed-off-by: Ettore Di Giacinto <[email protected]>

mudler marked this pull request as ready for review October 19, 2025 19:51

Copilot AI review requested due to automatic review settings October 19, 2025 19:51

Drop some templates

6c62819

Signed-off-by: Ettore Di Giacinto <[email protected]>

Copilot AI reviewed Oct 19, 2025

View reviewed changes

Keep reporting into conv

14950e0

Signed-off-by: Ettore Di Giacinto <[email protected]>

richiejp reviewed Oct 20, 2025

View reviewed changes

mudler added 4 commits October 20, 2025 11:11

tests fixups

467a66d

Signed-off-by: Ettore Di Giacinto <[email protected]>

tests fixups

e51407c

Signed-off-by: Ettore Di Giacinto <[email protected]>

fixups

3ffc8cd

Signed-off-by: Ettore Di Giacinto <[email protected]>

Do not complete observable during thought process

e6929d8

Signed-off-by: Ettore Di Giacinto <[email protected]>

richiejp reviewed Oct 21, 2025

View reviewed changes

core/agent/agent.go Show resolved Hide resolved

richiejp reviewed Oct 21, 2025

View reviewed changes

core/agent/agent.go Show resolved Hide resolved

richiejp reviewed Oct 21, 2025

View reviewed changes

core/agent/agent.go Show resolved Hide resolved

Update cogito

dc9fb71

Signed-off-by: Ettore Di Giacinto <[email protected]>

mudler force-pushed the cogito branch from fc12376 to a1f37e6 Compare October 21, 2025 14:18

Fixups

44ebedc

Signed-off-by: Ettore Di Giacinto <[email protected]>

mudler force-pushed the cogito branch from a1f37e6 to 44ebedc Compare October 21, 2025 14:21

mudler added 2 commits October 21, 2025 18:52

Fixups

a88eaab

Signed-off-by: Ettore Di Giacinto <[email protected]>

Fixups

e16a711

Signed-off-by: Ettore Di Giacinto <[email protected]>

mudler force-pushed the cogito branch from 2c49505 to e16a711 Compare October 21, 2025 20:19

Drop unneded option now

360d311

Signed-off-by: Ettore Di Giacinto <[email protected]>

mudler force-pushed the cogito branch 5 times, most recently from a8a0be6 to e68f122 Compare October 22, 2025 15:44

Fixups

3a978f6

Signed-off-by: Ettore Di Giacinto <[email protected]>

mudler force-pushed the cogito branch 2 times, most recently from 9fab92e to 91a91ed Compare October 23, 2025 13:27

Better handling of user tools

cf1cc69

Signed-off-by: Ettore Di Giacinto <[email protected]>

mudler force-pushed the cogito branch from 91a91ed to cf1cc69 Compare October 23, 2025 13:52

mudler added 3 commits October 23, 2025 16:10

TEST

8b12a9f

Signed-off-by: Ettore Di Giacinto <[email protected]>

Add flake attempts

c27715e

Signed-off-by: Ettore Di Giacinto <[email protected]>

Revert "TEST"

20d881c

This reverts commit 8b12a9f.

mudler force-pushed the cogito branch 5 times, most recently from b57f563 to 4c7d773 Compare October 24, 2025 08:59

Keep indexing MCP actions

43eec05

Signed-off-by: Ettore Di Giacinto <[email protected]>

mudler force-pushed the cogito branch from 4c7d773 to 43eec05 Compare October 24, 2025 13:13

	xlog.Error("Error summarizing conversation", "error", err)
	xlog.Error("Error summarizing conversation", "error", err)
	return

feat: consume cogito for agent reasoning #320

Are you sure you want to change the base?

feat: consume cogito for agent reasoning #320

Conversation

mudler commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

richiejp Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

richiejp Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

mudler Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

richiejp Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

mudler Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

richiejp Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mudler commented Oct 10, 2025 •

edited

Loading

mudler Oct 21, 2025 •

edited

Loading