Limits on expression contents.

[BengtGustafsson]

As has been shown in extract_fx it is possible to parse expression-fields to figure out which } or : is actually ending them. A few edge cases remain, but could be solved by small changes in other parts of the library:

• Using the comma operator in expressions. We can't forbid it at preprocessing time or MyTpl<A, B> would not work...
• format-specs starting with colon, which today has to be written as :: for range formats if you want to specify the element format. This has not been standardized yet.

Even with this proof of concept for not having to limit expression contents there remains the question if this is too much to ask of preprocessors and tools such as static analyzers. The implementation in extract_fx adds roughly 200 lines of C++ to a preprocessor implementation and handles all known quirks of non-erroneous expressions.

Regardless of which limitations are placed on expression-fields they have to be thoroughly specified to avoid compiler divergence, which could lead to portability problems. This specification work becomes easier with less limitations, and so does teaching the feature.

In Python, from which f-literals are modeled (although they are available in many languages) PEP-701 was just implemented in version 3.12 allowing most all of Python expressions in expression-fields. Previously there were limitations on nested string literals, line breaks and comments.

[hadrielk]

PEP 701 is indeed interesting.

Unfortunately we can't do exactly what they do for double-quote handling, for severals reasons that are probably obvious.

And besides C++ programmers expect and are accustomed to escaping double-quotes within string-literals. It would arguably be more confusing to require them to not be escaped.

But as the PEP notes, escaping quotes within it makes it challenging when multiple nested levels/depths of strings are being used.

BTW, does the PEP handle the case when an expression needs to use a single " double-quotes character? For example a f"name={dict['"']}"?

---

I should also note that they continue to require expressions to be wrapped by () parentheses if the expression contains a :, due to clashing with the colon separating the format-spec. That's what the current draft proposal requires as well, fwiw.

[BengtGustafsson]

I'm pretty sure Python 3.12 would handle f"name={dict['"']}" correctly as that double quote occurs inside another string literal and the point of the PEP is that such nested string literals are parsed "as usual". extract_fx handles this since yesterday at least.

While you could argue that it is confusing to experienced C++ programmers to suddenly not have to escape quotes in a string, but then imagine their relief when they understand that they can write their nested expressions just ignoring that the contents of {} is actually inside some kind of literal. It is not a regular literal as it has this f thing before it, so I think they would expect that there is some learning to do.

Also imagine the relief of the programmer tasked with making a clang-tidy function to convert std::format to f-literals (I mentioned this elsewhere as it was an argument in the PEP 701 motivation).

[hadrielk]

But I would think any C++ programmer should expect it to be fR"(...)" raw-string if they don't want to escape quotes, no?

Regardless, allowing any sequence of arbitrary characters within a set of quotes - without escaping or without delimiters like raw-string has - would be a hard sell to the WG, I would think?

Because to do that would require the main source file lexer to truly parse the string's internals just to figure out what constitutes a preprocessing-token.

That would be before the macro-invocation phase even, btw.

Whereas if we kept the current behavior of escaping quotes \", or using raw-string delimiters, leaves the main lexer unchanged. We'd only need to add a preprocessing-token for F"..." or X"..." or whatever, and the same lexer code that's in-use today would still work.

And then only the portion of the code that needs to parse the contents of that would be all contained solely within the _Extract() builtin operator.

[BengtGustafsson]

It may be a hard sell to WGs and it is true that with raw literals the discussion would be less heated. However, Python just showed that they thought the advantages outweighs the implementation cost, especially as they went back and redid it after having had f-literals at least since 2.x.

I don't know what you mean by main lexer, I think that all lexing needs to be done by the preprocessor. That said, I showed in my code and in a nearby issue that the workload to implement this "one more time" in the pre-preprocessor is close to negligable. All of this happens in phase 3 where tokenization occurs anyway.

When it comes to user expectations there may be an element of surprise the first time an experienced programmer compiles a format string with an escaped quote in an expression-field. But isn't this a pleasant surprise don't you think? Getting an error message saying that you don't have to write your code in a special way inside a f-literal would be a positive to most programmers, and programmers are quick learners in general. Note that it is always an error to write " outside a string literal.

Also it seems that if there is one other language many C++ programmers use it is most likely to be Python (build scripts, test frameworks etc.) so I think it is +1 to take inspiration from there rather than C# or something where escaping quotes is required (I suppose).

[hadrielk]

I don't know what you mean by main lexer, I think that all lexing needs to be done by the preprocessor.

Yeah, my description wasn't quite accurate.

What I really mean is: as far as I can tell, the lexer does not generally need to do very complicated work, to figure out what the potential dozen preprocessing-token types are, for the preprocessor to then work on for phase-4.

For string-literals, I think the hardest thing it has to do is figure out the ending of raw-strings, for which there's the bit in lex.pptoken 3.1 to make it work and undo the previous line-continuation handling of phase-2 translation.

With the exception of the stuff it has to do for raw-strings above, figuring out the beginning and end of string-literals is relatively trivial for it; and all other preprocessing-token types are simple and it does not need to parse internal details for more complicated grammar rules.

But if we define a new preprocessing-token for f-string-literal that requires parsing its contents for the full grammar rules of an interpolation-string, including the extraction-fields and such, just so that it can figure out what constitutes a single preprocessing-token of that type, then we'd have to define those rules within the lexer.

Alternatively, if we keep things the way everything is done now, then the lexer rules don't change. All the parsing logic of the f-string-literal would be contained to be only within the _Extract() operator's internal function.

And programmers can still avoid having to escape the quotes, by doing exactly what they do today to avoid that: they can use raw-strings. For example:

std::print(XR"({map.contains("foo")})");

[BengtGustafsson]

Sure, the parsing of string-literals gets more complicated to specify, possibly, but I don't think too much so, I will try to make a grammar for it, I think its not so many productions. The hard part is not actually nested comments or literals but setting up productions to eat matching parentheses and ternary operators.

[BengtGustafsson]

Here is a grammar starting out from an expanded string-literal production. As I see it the remaining problems is that doubling of left braces is not handled correctly: There is nothing that indicates that the case extraction-literal-contents { extraction-field } is not applicable when there are two consecutive left braces.

Unfortunately I had to add several new char categories to exclude a few characters which are parsed by the productions. This is only due to the fact that cases must be mutually exclusive. If a verbatim character is considered more specialized than s-char the new character groupings could be removed and s-char and r-char reused. e-char would just be "any character". However, I could not find any other productions within the preprocessor description that were not mutually exclusive so I wrote it out longhand like this, with above-mentioned exception for {{. I tried to solve that final problem by adding more productions but it just got ridiculous.

Note that if nested string literals were to use escaped quotes in expression fields this would work in another way: First phase 3 just notes the prefix and annotates a regularly parsed string-literal (possibly raw). Then in phase 3.5 the marked literals would be re-parsed by another parser, which would extract the fields. This parser would have to be described by some grammar but I guess it could be fairly simple if you don't allow {} or ?: on the top level, or expression-fields in format-specs.

I couldn't come up with a better way to format this in markdown than using a code block so here goes:

string-literal:
    encoding-prefix/opt " s-char-sequence/opt "
    encoding-prefix/opt R raw-string
    encoding-prefix/opt extraction-prefix " extraction-literal-contents "
    encoding-prefix/opt extraction-prefix R raw-extraction-string

# Do we want both upper and lower case? If not maybe choose upper as both L and R are upper and u and u8 are not usable with f anyway.
extraction-prefix:
    f
    x
    F
    X

extraction-literal-contents:
    x-char-sequence/opt
    extraction-literal-contents x-char-sequence
    extraction-literal-contents { extraction-field }
    extraction-literal-contents {{
    extraction-literal-contents }}

x-char-sequence:
    x-char
    x-char-sequence x-char

x-char:
    basic-x-char
    escape-sequence
    universal-character-name
    
basic-x-char
    any member of the translation character set except \, ", {, }, \n.

raw-extraction-string:
    " d-char-sequence/opt ( raw-extration-literal-contents } d-char-sequence/opt "

raw-extraction-literal-contents:
    y-char-sequence/opt
    raw-extraction-literal-contents y-char-sequence
    raw-extraction-literal-contents { extraction-field }
    raw-extraction-literal-contents {{
    raw-extraction-literal-contents }}

y-char-sequence:
    y-char
    y-char-sequence y-char

y-char:
    any member of the translation character set except {, } and ) followed by the initial d-char-sequence followed by "
    
extraction-field:
    expression-field
    expression-field : format-spec

expression-field:
    simple-expression-field
    simple-expression-field ? expression-field : expression-field

simple-expression-field:
    e-char-sequence/opt
    simple-expression-field e-char-sequence
    simple-expression-field ( simple-expression-field )
    simple-expression-field [ simple-expression-field ]
    simple-expression-field { simple-expression-field }
    simple-expression-field string-literal
    simple-expression-field char-literal
    simple-expression-field ::

e-char-sequence:
    e-char
    e-char-sequence x-char

e-char:
    any member of the translation character set except (. [, {, ", ', ), ], }, ?, :. Comments are replaced with single spaces as outside literals.
    
format-spec:
    f-char-sequence/opt
    format-spec f-char-sequence
    format-spec { expression-field }

f-char-sequence:
    f-char
    f-char-sequence f-char

f-char:
    any member of the translation character set except {, }.