fastn db

[amitu]

Stauts: In Discussion

Owner: @AbrarNitk

fpm serve will expose: /-/db/ API endpoint where SQL queries can be sent. fpm will run them against the package index and return the response over API.

Auth / Access Control

It would be super awesome if we could do some row-level access control. Every row in the DB came from one of the documents in the package. Now any logged-in user has either read access to any document or not. So if at query time we can ensure only the rows that the user has access to are visible then we have fantastic access control.

Not sure how to implement it tho. SQLite doesn't seem to support row-level security policies. PostgreSQL does but we can not bundle such a massive dependency on fpm. We can make it an optional feature, public packages won't need RLS and private packages can provide their PostgreSQL.

Indexer

We will have an indexer, which can be run by the fpm index. The indexer also runs concurrently in the background when fpm serve is running (and any changes to documents done via fpm will be picked and re-indexed). fpm index --watch will also file system changes and keep re-indexing.

Index Query Processor

Any document can also query the index.

-- string list names:
$processor$: fpm.db

SELECT name FROM [amitu.com/person#person];

What will be Indexed?

All variables and calls to components will be indexed in tables named after those types and components.

Calls to components from inside other components will not be indexed. Eg if ftd.text is used from inside another component, it will not be indexed.

If a document exists, amitu.com/person.ftd, which defines a record person:

-- record person:
caption name:

We will then define a table named amitu.com/person#person, with:

CREATE TABLE amitu.com/person#person 
    TEXT name
    TEXT $document-id$
    TEXT $name$

All The Lists Will Be Merged

For every record or type we create a single table. Both variables and lists of that type are inserted in that table.

If there are two lists containing 2 and 3 elements of type t, we will add 5 rows in the table corresponding to type t.

Processor Output Will Be Indexed

If a processor is used, the processor would be executed and it's output will be indexed.

Exception: $processor$: fpm.db. This processor will return zero result/default value when document is being rendered for indexing to avoid infinite recursion.

Opt Out From Indexing

Any variable can opt out of indexing by using $index$: false. This is largely to opt out of heavy processors. Some Processors can use $index$: false and their output will not be indexed (this acts as default). $index$: true can be used to force index output of a processor that opted out of indexing by default.

Simple Lookup

We can also do simple lookups on /-/db/ by passing document-id and variable/list-name to get only that variable/list.

fpm.sections CRUD

In tasks and dates we discussed fpm.p1-sections table:

CREATE TABLE [fpm.p1-sections]:
    INTEGER guid
    TEXT document-id,
    TEXT section-name, -- will be full name eg fifthtry.github.io/doc-site#h1
    INTEGER section-index;  -- index in the document is this first section or second and so on
    INTEGER level, -- if a ft.warning contains a ft.info, the level of ft.warning would be 0, and ft.info 1.
    TEXT kind,  -- record definition | record construction | component definition | component construction | variable creation | variable - updated

    TEXT caption,
    JSON headers,  -- will not include other sections?
    TEXT body,

    -- etc

What if we had a CRUD API /-/db/documents/ which lets you update the fpm.p1-sections table, and any update to this table is reflected in the actual document in the package.

We will support proper sync semantics, version, conflict etc to ensure there is no concurrent update issue.

CRUD on Sitemap

Sitemap should also be available as table.

TODO: Sitemap Table

[Arpita-Jaiswal]

Meta Data Table

We will have another table, [fpm.meta-data], which will list

CREATE TABLE `[fpm.meta-data]`
    TEXT document-id,
    TEXT name,
    TEXT type,

Say if you declare a variable:

-- import: amitu.com/foo
-- foo.person a:

type would be amitu.com/foo#person, name would be a and document-id would be foo.com/hello.

Terms Feature Should Use Package DB

We can have a table [fpm.headings]:

CREATE TABLE [fpm.headings]:
    TEXT document-id,
    TEXT term NULL,
    TEXT heading,
    TEXT component,
    INTEGER level,
    INTEGER section-index;

So all headings are stored in this table across entire package.

[amitu]

Also check out tasks and dates.