How we are parsing? 24th oct version (v3)

[amitu]

In the last update I discussed lalrpop. Since then I have used it some, and I have unfortunately not found it particularly great option.

It Makes Me Feel Dumb

This is kind of the first issue, every single simple thing I want to do, when I try to do it with lalrpop I feel like I have to really go get a degree in computer science. There is too much jargon, and too much emphasis on this is this and that is that, and I kind of do not really feel it's helping me. Things are hard. Things are very theoretical. Benefits quite dubious.

Grammar Is Not Real Grammar

One advantage I hoped we will have was that our .grammar file will become like official grammar of, at least p1 level of our language. It's true, but also not true. Like the grammar file is actually code, and is bogged down by details. This is not a readable grammar.

Too Much Fighting With The Tool

I am getting constant shift-reduce conflict, and other such, with page long error message, bravo, but it doesn't really tell you how to fix things really. Asks you to read the manual, but manual does not really tell you how to fix them either.

Moving Back To "Handwritten" Parser

So I have started writing the parser. The handwritten parser is kind of tedious, but it gives me "I am completely in command here, this is just basic Rust/programming". I am writing my own Scanner, derived from the ideas in A Beginner's Guide to Parsing in Rust.

Testing Using insta

I had earlier written custom snapshot testing code, I have since moved to use insta.rs. Feels pleasant.

Testing Difficulties

So our parser output is pretty "cheap" / light:

pub type Span = std::ops::Range<usize>;

pub struct Section {
    pub dashdash: Span, // for syntax highlighting and formatting
    pub name: KindedName,
    pub colon: Span, // for syntax highlighting and formatting
    pub caption: Option<HeaderValue>,
    pub headers: Vec<Header>,
    pub body: Option<HeaderValue>,
    pub sub_sections: Vec<Spanned<Section>>,
    pub function_marker: Option<Span>,
    pub is_commented: bool,
}

We only keep track of Range for every interesting thing. This is light on memory, but is pretty inconvenient when looking at the output. For example:

;; this is a comment

;-; this is some module doc
;-; multiline module docs!

;; some more comment

Parses into:

---
source: fastn-p1/src/lib.rs
expression: "fastn_p1::ParseOutput::new(\"foo\", &s)"
input_file: fastn-p1/t/003-module-doc.ftd
---
doc_name: foo
module_doc:
  start: 22
  end: 77
items:
  - span:
      start: 0
      end: 21
    value: Comment
  - span:
      start: 78
      end: 99
    value: Comment
line_starts: []

Reading all the start/end is kind of annoying.

One option we have, for testing purpose to create another Section (and all downstream types), and convert all the spans into the corresponding string slice in the input string, and show the string instead of (or along with) the start/end ranges.

Other option is to create some sort of HTML output, which takes the input string, and adds spans for each range, which on hover shows the data about that thing.

[amitu]

So I implemented a debug output for our ParseOutput, and now the module doc insta yml looks like this:

---
source: fastn-p1/src/lib.rs
expression: "fastn_p1::ParseOutput::new(\"foo\", &s).debug(&s)"
input_file: fastn-p1/t/003-module-doc.ftd
---
doc_name: foo
items:
  - comment: ";; this is a comment\n"
  - comment: ";; some more comment\n"
module_doc: ";-; this is some module doc\n;-; multiline module docs!\n"