Why is \everyeof{\noexpand} needed to avoid "File ended while scanning definition of" with expandable read from file?

Digging in the TeX program code, my main finding is that there is neither an explicit nor an implicit end-of-file marker inserted in the token stream.

When TeX sees an \input command, it executes some file reading maintenance code (begin_file_reading) and switches to the new file for further reading of input lines. If it either sees an \endinput command or has reached the actual end of the input file, it sets a flag force_eof to true. The latter is used to determine if the current file should be popped from the reading stack (end_file_reading) when reading the next line. Processing then goes back to the place \input was called from.

If force_eof is set to true while TeX tries to read an input line, it calls check_outer_validity, which in turn checks if the end of file is allowed at the current source location. It checks that based on a variable called scanner_status. When the scanner status doesn't indicate that a definition is being formed, a parameter is being read etc., the check passes and TeX goes on processing the next token. Otherwise a "runaway" error message is issued, based on the current status. Additionally, ε-TeX adds the material defined in the \everyeof hook into the token stream.

Conceptually, one can imagine the input token stream of

\everyeof{\noexpand} \edef\@tempcontents{\@@input foo.txt \space} 

at the time \@tempcontents is being defined as the following sequence:

<contents of input file> <endline char> \noexpand <EOF> \space 

<endline char> is the character TeX inserts based on the value of \endlinechar after every line is has read. This is `\^^M by default, which expands to a space character and leads to the extra space after abc in your console output. Next comes the contents of the \everyeof token list, which is \noexpand in this example.

Now the mysterious <EOF> marker occurs. The important thing here is, <EOF> is not an actual or internal token, it's just the brief moment in which TeX checks if an \outer macro is allowed at this position of processing.

As TeX is in an \edef at that moment, the commands we may use in \everyeof to trick TeX into ignoring the outer test have to be a) expandable and b) allow an \outer macro to follow them. By hints in the code, at least the following three do work in this place:

• \noexpand
• \string
• \meaning

The reason those work is that they temporarily switch to scanner status normal before reading the next token, thus negating the error condition in check_outer_validity. TeX is dazed for a little moment and believes it is in normal processing mode, while actually still in the process of scanning for the end of the macro definition.

If you also want to remove the \space token, you can use the following:

\newcommand{\showexpanded}[1]{% \everyeof{\expandafter\@gobble\noexpand}% \long\edef\@tempcontents{#1}% \show\@tempcontents } 

which outputs

> \@tempcontents=\long macro: ->abc abc . \showexpanded ...ontents {#1}\show \@tempcontents