Skip to content

Commit ec59a49

Browse files
floscrfloscr
committed
Document implicit strings
1 parent b695d8b commit ec59a49

File tree

1 file changed

+125
-5
lines changed

1 file changed

+125
-5
lines changed

‎docs-language/syntax.md‎

Lines changed: 125 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ import TokenScriptCodeBlock from '@site/src/components/TokenScriptCodeBlock';
1010

1111
## Character Set & Identifiers
1212

13-
- Identifiers may include standard ASCII letters, digits, hyphen (`-`), underscore (`_`), and extended Unicode characters (e.g., emoji).
13+
- Identifiers may include standard ASCII letters, digits, hyphens (`-`), underscores (`_`), and extended Unicode characters (e.g., emoji).
1414
- Identifiers must begin with an alphabetic or non-ASCII character (numbers are not allowed in the first position).
1515
- Identifiers are case-insensitive (`primaryColor` and `PrimaryColor` are treated equally).
1616

@@ -35,14 +35,134 @@ variable spacing: NumberWithUnit = 4px; // Trailing comments are allowed`}
3535
| Numbers | `42`, `3.1415`, `.5` | Integers and decimals; leading zero added for `.5` style numbers by the lexer. |
3636
| Hex Colors | `#FFAA00`, `#abc` | Interpreted as `Color.Hex`. |
3737
| Strings | `identifierStyle` | Bare identifiers not bound in scope fall back to string literals. |
38-
| Explicit Strings | `"quoted value"` or `'alternate quotes'` | Preserve whitespace and special characters. |
38+
| Explicit Strings | `"quoted value"` or `'alternate quotes'` | Preserves whitespace and special characters. |
3939
| Booleans | `true`, `false` | Reserved keywords mapped to `Boolean`. |
4040
| Null | `null` | Reserved keyword mapped to `Null`. |
4141
| Lists | `value1, value2, value3` | Comma-separated sequence; implicit lists use spaces when built by certain operations. |
4242

43+
## Implicit Strings and Lists
44+
45+
### Overview
46+
47+
TokenScript supports **implicit strings** - strings that don't require quotes. This is a fallback mechanism for when a string is not made explicit with quotes.
48+
49+
:::warning Recommendation: Use Explicit Strings
50+
Implicit strings exist for backward compatibility and convenience, but they have confusing edge cases.
51+
52+
**Prefer to use quotes around strings (`"hello"`) instead of relying on implicit strings (`hello`).**
53+
:::
54+
55+
### Basic Behavior
56+
57+
Implicit strings are recognized when the lexer/parser encounters text that:
58+
59+
1. Starts with a valid identifier character (letters, emoji, etc.)
60+
2. Or starts with a number followed by text that is **not a recognized unit**
61+
62+
<TokenScriptCodeBlock mode="script">
63+
{`Implicit strings supported`}
64+
</TokenScriptCodeBlock>
65+
66+
<TokenScriptCodeBlock mode="script">
67+
{`// Strings starting with numbers (not recognized units)
68+
// ⚠️ GOTCHA: Space is added between number and text!
69+
1unknown // Output: ["1", "unknown"] (List) - not "1unknown"!
70+
5test // Output: ["5", "test"] (List) - not "5test"!
71+
3D-Font // Output: ["3", "D-Font"] (List) - not "3D-Font"!`}
72+
</TokenScriptCodeBlock>
73+
74+
<TokenScriptCodeBlock mode="script">
75+
{`1px
76+
5rem
77+
10%`}
78+
</TokenScriptCodeBlock>
79+
80+
### Implicit Lists
81+
82+
When multiple values are combined without operators, they form **implicit lists**.
83+
<br />This allows natural composition of values:
84+
85+
<TokenScriptCodeBlock mode="script">
86+
{`1px solid black`}
87+
</TokenScriptCodeBlock>
88+
89+
### Edge Cases and Pitfalls
90+
91+
#### Arithmetic Operations with Implicit Strings
92+
93+
When you mix arithmetic operators with implicit strings, the behavior may surprise you:
94+
95+
<TokenScriptCodeBlock mode="script">
96+
{`// ⚠️ This evaluates to "2 unknown" not an error!
97+
// The arithmetic happens first: (1 + 1) = 2
98+
// Then creates implicit list with result and string: "2 unknown"
99+
1 + 1unknown`}
100+
</TokenScriptCodeBlock>
101+
102+
##### Recommended usage
103+
104+
Use recognized units for arithmetic:
105+
106+
<TokenScriptCodeBlock mode="script">
107+
{`1px + 1px`}
108+
</TokenScriptCodeBlock>
109+
110+
Use explicit strings to avoid the edge cases:
111+
112+
<TokenScriptCodeBlock mode="script">
113+
{`"1unknown"`}
114+
</TokenScriptCodeBlock>
115+
116+
So the aforementioned example will now throw an error:
117+
118+
<TokenScriptCodeBlock mode="script">
119+
{`1 + "1unknown"`}
120+
</TokenScriptCodeBlock>
121+
122+
#### 2. Strings Starting with Numbers
123+
124+
Unlike CSS (which disallows unquoted identifiers starting with numbers), TokenScript allows them as implicit strings:
125+
126+
<TokenScriptCodeBlock mode="script">
127+
{`3D Font`}
128+
</TokenScriptCodeBlock>
129+
130+
<TokenScriptCodeBlock mode="script">
131+
{`"3D Font"`}
132+
</TokenScriptCodeBlock>
133+
134+
### Best Practices
135+
136+
:::warning Always Use Explicit Strings When Possible
137+
While implicit strings are convenient, they can lead to confusion and errors.<br />**We recommend using explicit strings in most cases.**
138+
:::
139+
140+
<TokenScriptCodeBlock mode="script">
141+
{`// Use explicit strings (clear intent)
142+
"3D Font", "some other font", "Font with emoji 😼";`}
143+
</TokenScriptCodeBlock>
144+
145+
#### When Implicit Strings Are Acceptable
146+
147+
Implicit strings are mainly useful for:
148+
149+
1. **Simple identifiers without spaces or special characters**
150+
151+
<TokenScriptCodeBlock mode="script">
152+
{`primary`}
153+
</TokenScriptCodeBlock>
154+
155+
2. **CSS-like shorthand values**
156+
157+
<TokenScriptCodeBlock mode="script">
158+
{`1px solid black`}
159+
</TokenScriptCodeBlock>
160+
161+
3. **Backward compatibility** - Existing TokenScript code may rely on implicit strings
162+
43163
## Units
44164

45-
Numbers may be suffixed with a units like `px`, `em`, `rem`, `vw`, `vh`, `pt`, `in`, `cm`, `mm`, `deg`, `%`.
165+
Numbers may be suffixed with units like `px`, `em`, `rem`, `vw`, `vh`, `pt`, `in`, `cm`, `mm`, `deg`, `%`.
46166
<TokenScriptCodeBlock mode="script" showResult={false}>
47167
{`variable padding: NumberWithUnit = 1.5rem;
48168
variable angle: NumberWithUnit = 45deg;`}
@@ -81,8 +201,8 @@ variable ramp: List = accent, accent.to.oklch();`}
81201

82202
- Type annotations are required (`Type` or `Type.SubType`).
83203
- Initializers are optional; without an initializer, variables start with the type’s `empty()` value, which in most symbols is `null`
84-
- For [Dictionary](/language/types#dictionary) its an empty dictionary.
85-
- For [List](/language/types#list) its an empty list.
204+
- For [Dictionary](/language/types#dictionary), it's an empty dictionary.
205+
- For [List](/language/types#list), it's an empty list.
86206

87207
### Assignments & Reassignments
88208

0 commit comments

Comments
 (0)