[WIP] - SQL Model Data Layer

[hayescode]

Caution

This is currently a mockup and incomplete. This has not been tested and will not run.

This is an initial draft for consolidating all SQL data layers using FastAPI's sqlmodel (Pydantic + SQL Alchemy).

For now I have created the new models in a new models/ directory and are duplicated from step.py, element.py, etc. The idea would be to delete those once this is ready.

The main reasons for this change are:

1. Enable everyone who wants to use SQL data persistence to use a single implementation.
2. Reduce technical debt and bugs, and speed up future development.

replaces #1365

Notes:

• sqlmodel reserves the 'metadata' field for internal purposes and cannot be changed, so in Threads/Steps/User we have to rename this to something else so for now it's metadata_ instead, then aliased for proper database and API usage.

[niklasdiehm]

I think the initial file should be the creation of the tables, right? So that new users do not have to problem to create the tables by themself.

[hayescode]

Today our SQL tables are stored in cameCase which is abnormal so my idea for the 1st migration would rename the problematic columns like threadId to thread_id. That would be the most "hands-off" migration. I have not actually done that here yet though.

Alternatively we could create brand new tables in a new database schema and use COPY INTO scripts.

I really don't think keeping these as camelCase would be good long term as we'd have to handle this into the future for all changes and it's not the right way to make SQL tables.

[github-actions]

This PR is stale because it has been open for 14 days with no activity.

[asvishnyakov]

@github-actions Leave this PR open