6

I would like to understand TeX's behaviour when the catcode of the space character is set to 12.

The following two example input files differ in just one position: where the first has the character 4 the second has the character (space). Both 4 and have catcode 12 in these examples. I therefore expected the typeset output of the second example to be identical to that of the first, with the only difference that the typeset '4' would be replaced by a typeset 'space'. In other words, I expected the second example to have three typeset lines, the second line consisting of five of those "open boxes" that cmtt10 uses to typeset spaces. Instead, TeX ignored the second line, even though it is not empty in the input file (it consists of five catcode 12 spaces). What is going on here?

\catcode`\ =12 \obeylines \tt een twee drie 4 hoedje van papier \bye

enter image description here

\catcode`\ =12 \obeylines \tt een twee drie hoedje van papier \bye

enter image description here

Improve this question
7
  • 1
    Trailing spaces (ASCII 32, regardless of catcode) are trimmed in web2c TeX implementations. You can try that by setting the space to a show-stopping catcode, like 2: a line with a will be valid Commented Jan 26, 2021 at 16:34
  • @PhelypeOleinik The spaces are actually stripped off before looking at category codes and, as far as I know, this is in all TeX implementations. Commented Jan 26, 2021 at 16:37
  • @egreg Yeah, that's right (with "regardless of catcode" I should have said "before TeX assigns any catcode to them"). I don't know if this is in all implementations, though: this code is in some web2c C routines, so if some implementation doesn't use that, spaces may be preserved. This code is not part of TeX itself Commented Jan 26, 2021 at 16:40
  • 1
    @PhelypeOleinik it is documented in the texbook (and web2c recently changed to only strip space and not strip tab to match the texbook more exactly) Commented Jan 26, 2021 at 16:49
  • @PhelypeOleinik As David says: third double dangerous bend on page 46 of the TeXbook: “TeX deletes any ⟨space⟩ characters (number 32) that occur at the right end of an input line.” Commented Jan 26, 2021 at 17:13

1 Answer 1

Reset to default
3

On page 46 of the TeXbook, third double dangerous bend, we find

TeX deletes any ⟨space⟩ characters (number 32) that occur at the right end of an input line. Then it inserts a ⟨return⟩ character (number 13) at the right end of the line, except that it places nothing additional at the end of a line that you inserted with ‘I’ during error recovery. Note that ⟨return⟩ is considered to be an actual character that is part of the line; you can obtain special effects by changing its catcode.

The first sentence explains what you get. Note that tokenization hasn't even started yet: when needed, TeX puts a record in a buffer, doing the steps outlined in the text above; only after this it starts tokenizing and, indeed, the text above specifies characters number 32.

The ⟨return⟩ character actually is the one with character code \endlinechar which under \obeylines is still code 13, but active (category code 13). This is explained in the fifth double dangerous bend on page 48. Anyway it's inserted after the deletion of the space characters, so this doesn't help in your situation.

Improve this answer
2
  • Thanks for the answer. So I gather that it is impossible by design to typeset trailing spaces, because they are deleted and lost before TeX starts with the tokenization of the line. Commented Jan 26, 2021 at 20:10
  • 2
    @WillemVandenHeuvel Yes, that how it is. The problem is that some older operating systems used fixed length records and might pad them with spaces (or NUL characters, which is the reason for category code 9). Note that TAB characters are not removed, so you might use them. Commented Jan 26, 2021 at 20:14

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.