Fix/numeric default none

[raph-topo]

Context

In columns of type Numeric (float) & Integer (int), a new row gets populated with value 0.

But 0 is a valid value, and actually neither more nor less valid than any other float or int. Reason for the existence of None in Python. 0 really means "cell has been defined with value 0" and not "no numeric value defined yet". A new numeric cell should thus be empty (None) by default.

As such I find myself creating trigger formulas for every numeric column =None.

Proposed solution

Set Python None & JS null & SQlite "NULL" as default for Numeric & Integer columns.

Related issues

#1530

Has this been tested?

• 👍 yes, I fixed the test suite:

yarn run test:client
yarn run test:common
yarn run test:python

Some functions now require the following to coalesce None to 0 for calculations:

($_col or 0) + …

Maybe this is not what you want? It feels more robust though (i.e. end user has to make sure his data is type-safe everywhere and not rely on silent old-default behaviour).

WIP: some tests still to fix.

[dsagal]

I think making that the new default could be disruptive to those with existing documents. New rows will suddenly behave differently from how they used to behave.

Also, there are certainly some cases where 0 is the better default, and it's hard to know which is more common -- for those users for whom 0 is better, there will be new a annoyance in having to create =0 trigger formula, just the same as it was annoying before to those who need None.

What do you think of instead making it easier to toggle the preferred default, so that it can be done with a click rather than with typing in a formula?

One other idea is to introduce a new type, so that there is both Numeric and Numeric:Empty (but it could be very confusing to new users, and too many types makes it harder to use them).

[raph-topo]

(examples are in Grist 1.4.2)

I think making that the new default could be disruptive to those with existing documents. New rows will suddenly behave differently from how they used to behave.

That is true, input behavior would change.

Looking at calculations though, it seems to me the disruption is contained by the way upper-case functions ignore None already today.

e.g. data:

0 in col B were filled automatically (default-0), empty in col C are deliberate entry. That gives summary table (all set to AVERAGE):

IMHO, since B4 & B5 did not receive my user input, the average is wronged by default-0, and the right behavior is C.

Obviously in such a small table, the automatic-0 values are easily seen & fixed, but what about 15-col/100-row docs? No way a 0 in the middle easily be spotted (happened to me twice already).

For standard ("lowercase") Python, default-0 already today does not avoid type issues :

Requiring $_col or 0-style formulas forces the user to deliberately act on missing data and not rely on unconscious defaults, which is IMHO much more transparent & reliable.

Matter of fact, the docs are already recommending $_col or 0 for cumulative/recursive functions: https://support.getgrist.com/formulas/#recursion

Also, there are certainly some cases where 0 is the better default, and it's hard to know which is more common -- for those users for whom 0 is better, there will be new a annoyance in having to create =0 trigger formula, just the same as it was annoying before to those who need None.

That is true.

From a user perspective, I would say:

• 0 is the default for additions & subtractions etc. (when working with sums of figures)
• 1 is the default for multiplication & division (when working e.g. with coefficients or fractions/percentages)
• but what about averages, or min/max? 0 weighs in as a value
• plus how could a column be certain of which usage is going to be made of it later? (i.e. same column could be used in an addition & multiplication/coefficient/percentage, the default should thus depend on downstream usage)

The only value that is neutral to any downstream to me seems to None.

What do you think of instead making it easier to toggle the preferred default, so that it can be done with a click rather than with typing in a formula?

Why not, but then we need multiple buttons, I see at least 0, 1 & None for now.

One other idea is to introduce a new type, so that there is both Numeric and Numeric:Empty (but it could be very confusing to new users, and too many types makes it harder to use them).

Agree. Plus they would basically be the same type in backends, because both would support deliberate None anyway.

[raph-topo]

I just imported my "20-table / 20+ col & 200+ rows each" doc with all sorts of lookups into my Grist test instance with this default-None patch.

The only issues that appeared were:

TableA.colAA is Reference to table B
TableA.colAB = $colA.colBA
TableB.colBA is Numeric but has some empty cells (i.e. None)

In Grist 1.4.2, $colA.colBA replaces None with 0.

Which in my case even creates errors because TableA is about coefficients specifically, as such I had to add visual error-markers to warn me about problematic 0:

In my Grist:dev it creates a new TypeError : unsupported operand type(s) for /: 'NoneType' and 'int', but that forces me to act and decide on how to treat unwanted values:

Easy to fix with: $_col or 0, and now I deliberately know what is going on if I did not fill correct data in another table.

Actually, that just helped me fix an unwanted coalescence in a formula column which I had not spotted yet.

[dsagal]

So, the behavior is of course different between 0 and None. Some formulas will produce errors, some different results (e.g. AVERAGE in a summary table will ignore None values but will average in the 0 values). I think there are several issues and decisions to make:

• It isn't easy to choose None as the default for a column (you have to know to write a formula, you have to know what None is, there is no in-app guidane).
  • Should it be easier to choose between these defaults when creating a column?
  • If yes, should we also allow other defaults, like 1, and other common defaults for other columns, like TODAY() for Dates?
• Should we change the initial default value for Numeric and Int columns from 0 to None:
  • For new documents?
  • For new columns in existing documents?
  • For existing columns in existing documents?

I feel that at least for the last question, we shouldn't change the default value. I.e. existing columns in existing documents should still default to 0 (new rows will get 0 values), to reduce disruption. For other questions, the risk of disruption from switching the default is lower, but there is still a tradeoff.

[tayflo]

I highly agree on the matter with @raph-topo, that None should be the default value for numeric value – I already thought about this one year ago (while discussing with some other users, who were also agreeing).

Indeed, I want to differentiate between a value that has been manually registered/validated, and a value that has not been reviewed. I prefer an error raised because a formula required a number and got None, than a silent erroneous calculation. So I'm always setting back value to None manually when I'm setting up numeric columns, and I don't recall an occurrence where I wanted zero to be the default value (once, I wanted 1 to be the default value, for quantities of items).

[tayflo]

Should we also allow other defaults, like 1, and other common defaults for other columns, like TODAY() for Dates?

I would say why not. Even if I'm quite fine with trigger formulas, having some preset defaults (in sync with init formula) may be a plus (if that does not make the interface looks too cluttered), especially for less tech-savvy users. It would both slightly reduce the time needed to configure columns, and give some ideas about (as does the "Shortcuts" option in the column creation menu).

If we think about what kind of default values would be interesting to have, I can think of:

• Numeric, Integer: None ; 0 ; 1 (I see Infinity does not seem supported)
• Date: None ; TODAY()
• DateTime: None ; NOW()
• Toggle: None ; False ; True (see discussion in #1530 about the None value)
• Choice, ChoiceList, Reference, ReferenceList: Some options among existing values
• Text, Attachment: Dunno

I don't know if it's worth it to add the functionalities for such options, but it may constitute a friendlier way to introduce user to trigger formulas. And it may also be the opportunity to think about better communicating about some special object interesting to use in trigger formulas, like the user object.

[tayflo]

Existing columns in existing documents should still default to 0 (new rows will get 0 values), to reduce disruption

It makes sense (even if, as far as I'm concerned, I would be happy for all my columns to be changed to defaulting to None, if I didn't make them do so myself already).

A way I see for doing this would be to set trigger formulas to =0 on preexisting columns (and the new default would be =None; should that =None be made transparent btw? that is, should a trigger formula always be set to make the default value explicit?).