5 
\def\test{\catcode`!=\active \def!{test}}

Of course the above code would not work because nothing is executed when TeX is reading the replacement text of a macro definition. ! gets catcode 12 anyway. I've tried a search on this site and someone suggested using \scantokens. As far as I know, \scantokens is an expandable command which reads the following token list, stores it in a pseudo-file and then inputs it, so the tokens get re-tokenized according to the new catcodes. However, the following code still fails and I can't figure out why.

\def\test{\catcode`!=\active \scantokens{\def!}{test}} \test

I get the error of "runaway definition".

Improve this question

4 Answers 4

Reset to default
7

The e-TeX reference defines \scantokens as follows:

\scantokens, when followed by a <general text>, decomposes the <balanced text> of the <general text> into the corresponding sequence of characters as if the <balanced text> were written unexpanded to a file; it then uses TeX's \input mechanism to re-process these characters under the current \catcode regime.

So the \def! part of your macro is processed as if it were defined in an extra file that is then \input into your main file. This includes the use of an internal end-fo-file marker when the whole <balanced text> has been processed.

In The TeXbook, p. 206, Knuth makes a note on file reading and \outer macros (thanks to GuM for pointing this out):

An \outer macro cannot appear in an argument (not even when \par is allowed), nor can it appear in the parameter text or the replacement text of a definition [...] The end of an input file or alignment template is also considered to be \outer in this sense; for example, a file shouldn't end in the middle of a definition.

This is exactly the problem you run into here. TeX is still scanning the definition of ! when the end of file after \scantokens occurs. It then complains about File ended while scanning definition of !. You can see the same behavior and error message if you replace \scantokens{\def!} by an actual \input were the input file includes only \def!.

Bruno Le Floch found a nice way to get around this problem by using a \let instead of \def to define !:

\def\bangdef{test} \def\test{\catcode`!=\active \scantokens{\let!}\bangdef} \test

For completeness, here are two standard tricks that do not make use of \scantokens. One way to get the desired result is by defining \test in a regime which has the correct catcode already applied:

\begingroup \catcode`!=\active \gdef\test{\catcode`!=\active \def!{test}} \endgroup \test \show!

Instead of the \gdef you could also save the current catcode of ! and restore it after the definition of \test.

Another common trick is to change the lowercase code of an active character, here ~, to the character code of the character to be made active, and then use \lowercase to apply this change:

\def\test{% \begingroup \lccode`\~=`\! \lowercase{\endgroup \catcode`\!=\active \def~}{test}% } \test \show!
Improve this answer
5
  • On \scantokens, it is possible to avoid the issues with end-of-file but only by using an assignment. It's well-established that \scantokens is a bit 'fragile': it needs careful handling. Commented Oct 6, 2018 at 6:15
  • @JosephWright Thanks, I updated the second example. Could you explain what kind of assignment you mean for the \scantokens approach? Commented Oct 6, 2018 at 6:47
  • Typically just \everyeof{\noexpand} but one can be more sophisticated (see the expl3 code for \tl_rescan:nn). Commented Oct 6, 2018 at 6:51
  • As for a reference for the rule that forbids definitions to continue past the end of a file, see The TeXbook, p. 206, lines 7-13. If you want, I can insert the reference directly into your answer. Commented Oct 6, 2018 at 8:48
  • 1
    Amusingly \scantokens{\let\foo}\bar works, so one way could be to \def\temporary{test} and \expandafter\let\scantokens{!}\temporary. Commented Oct 6, 2018 at 8:58
3

OpTeX defines its \adef macro using \directlua:

 \_doc ------------------------------ \`\adef` `<char>{<body>}` defines active `<char>` as <body> and then puts the <char> as active character. I.e. the `<body>` can include the <char> as non-active charter (if it is non-active before `\adef`). For example `\adef ?{\,?}`. If the character is special, you can escape it, for example `\adef\%{...}`. The space can be declared by `\adef{ }{<body>}`. You can declare a macro with parameters too, for example `\adef @#1{...#1...}`. You can use prefixes `\protected`, `\global`, `\long` before `\adef`, they behave like prefixes before `\def`. \_cod ------------------------------ \_def\_adef#1#2#{\_adefA{#1}{#2}} \_def\_adefA#1#2#3{\_ea\_def\_directlua{tex.cprint(13,"\_luaescapestring{\_csstring#1}")}#2{#3}% \_catcode`#1=13 } \_public \adef ;
Improve this answer
1

Use etex or pdftex.

\def\test{\catcode`!=\active \begingroup\endlinechar-1 \everyeof{{test}\endgroup}\scantokens{\gdef!}} \test \tt \meaning! \bye

enter image description here

Less original, and dropping the idea to end the scantokens argument before replacement text:

\def\test{\catcode`!=\active \scantokens{\def!{test}\relax}}% \relax to absorb an EOL space token \tt +++\test+++ \show! \bye

Produces

> !=macro: ->test. l.10 \show!
Improve this answer
1
  • The first method has the advantage that the definition of ! can easily be made to use itself with catcode 12, for example try with \everyeof{{!!!}\endgroup}. This will define the active ! to non-active !!!. Commented Oct 6, 2018 at 21:45
0

"Academic" info:

You can (ab)use \futurelet for getting rid of end-of-file-markers that come into being due to \scantokens or due to the \input-primitive—in LaTeX the \input-primitive is renamed to \@@input and the control sequence \input is redefined:

% Characters of category 0, 5, 9, 14 and 15 don't make it into explicit % character tokens, thus allow syntax both with explicit character tokens % and control symbol tokens. % % E.g., under normal circumstances you can't do % \DefineActiveChar{%}{<prefixes like \global>}<parameter text>{<replacement text}% % , but you can do % \DefineActiveChar{\%}{<prefixes like \global>}<parameter text>{<replacement text}% % . % Reading a single space from a line of .tex-input, be it a real text file % or be it with \scantokens' faking of unexpanded-writing and then inputting % of a text file, requires special attention because at the time of % pre-processing a line of .tex-input all space-characters at the end of the % line get stripped off. % With some TeX-implementations this is also the case with the horizontal-tab- % character. % Thus writing and reading a space-character to/from external text file % requires appending an additional comment-character. % At least with writing and reading linefed and carriage return % special attention might be required, too. \newtoks\scratchtoks \def\thescratchtoks{\the\scratchtoks}% \long\def\firstoftwo#1#2{#1}% \long\def\secondoftwo#1#2{#2}% \def\GatherActiveCharacterAsRelax#1{% % #1 = active character token to (re)define \let#1=\relax\NeutraLizeEOFMarker{#1}% }% \def\NeutraLizeEOFMarker#1#2{% % #1 = active character token to (re)define % #2 = letter X or - in case X is the active character to redefine - active X % ; ensures that in case #1 is space that space is not removed % during TeX's pre-processing of lines of input \endgroup \begingroup \let#1=\relax \scratchtoks={\GatherDefinitionPrefixAndDefine{#1}}% \everyeof={}% \futurelet\scratchmacro\thescratchtoks }% \long\def\GatherDefinitionPrefixAndDefine#1#2{% % #1 = active character token to (re)define % #2 = prefixes for `\def` like `\global` or `\long` \endgroup#2\def#1% }% \def\DefineActiveChar#1{% % #1 = character or one-letter-control-sequence for alphabetic constant \begingroup \escapechar=-1\relax \newlinechar=-1\relax \endlinechar=-1\relax \catcode`\^=7\relax \catcode`\X=11\relax \catcode`#1=\active \expandafter\expandafter\expandafter\GatherActiveCharacterAsRelax\expandafter\noexpand \scantokens \csname \ifnum`#1=10 first\else second\fi oftwo\endcsname {% \expandafter{\expandafter^\expandafter^JX}% }{% \csname \ifnum`#1=13 first\else second\fi oftwo\endcsname {% \expandafter{\expandafter^\expandafter^MX}% }{% \expandafter{\string#1X}% }% }% }% {\catcode`!=\active \outer\gdef!{Huh?}}% %------------------------------------------------------------------------- \DefineActiveChar{!}{\long}#1#2{ACTIVE EXCLAMATION MARK Arg1: #1 test Arg2: #2} \begingroup \catcode`!=\active \show!\endgroup %------------------------------------------------------------------------- \DefineActiveChar{A}{\long}#1#2{ACTIVE A Arg1: #1 test Arg2: #2} \begingroup \catcode`A=\active \showA\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\!}{\long}#1#2{ACTIVE EXCLAMATION MARK AGAIN Arg1: #1 TEST Arg2: #2} \begingroup \catcode`!=\active \show!\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\%}{\long}#1#2{ACTIVE PERCENT Arg1: #1 Arg2: #2} \begingroup \catcode`\%=\active \show%\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\{}{\long}#1#2{ACTIVE CURLY LEFT BRACE Arg1: #1 Arg2: #2} \begingroup \catcode`\{=\active \show{\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\}}{\long}#1#2{ACTIVE CURLY RIGHT BRACE Arg1: #1 Arg2: #2} \begingroup \catcode`\}=\active \show}\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\#}{\long}#1#2{ACTIVE HASH Arg1: #1 Arg2: #2} \begingroup \catcode`\#=\active \show#\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\ }{\long}#1#2{ACTIVE SPACE Arg1: #1 Arg2: #2} \begingroup \catcode`\ =\active\show \endgroup %------------------------------------------------------------------------- \DefineActiveChar{ }{\long}#1#2{ACTIVE SPACE AGAIN Arg1: #1 Arg2: #2} \begingroup \catcode`\ =\active\show \endgroup %------------------------------------------------------------------------- \DefineActiveChar{\^^I}{\long}#1#2{ACTIVE HORIZONTAL TAB Arg1: #1 Arg2: #2} \begingroup \catcode`\^^I=\active\show^^I\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\\}{\long}#1#2{ACTIVE BACKSLASH Arg1: #1 Arg2: #2} \begingroup \catcode`\/=0 \catcode`\\=\active/show\/endgroup %------------------------------------------------------------------------- \DefineActiveChar{\$}{\long}#1#2{ACTIVE DOLLAR Arg1: #1 Arg2: #2} \begingroup \catcode`\$=\active\show$\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\&}{\long}#1#2{ACTIVE AMPERSAND Arg1: #1 Arg2: #2} \begingroup \catcode`\&=\active\show&\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\^}{\long}#1#2{ACTIVE CARET Arg1: #1 Arg2: #2} \begingroup \catcode`\^=\active\show^\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\_}{\long}#1#2{ACTIVE UNDERSCORE Arg1: #1 Arg2: #2} \begingroup \catcode`\_=\active\show_\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\^^A}{\long}#1#2{ACTIVE START OF HEADING Arg1: #1 Arg2: #2} \begingroup \catcode`\^^A=\active\show^^A\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\^^J}{\long}#1#2{ACTIVE LINE FEED Arg1: #1 Arg2: #2} \begingroup \catcode`\^^J=\active\show^^J\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\^^M}{\long}#1#2{ACTIVE CARRIAGE RETURN Arg1: #1 Arg2: #2} \begingroup \catcode`\^^M=\active\show^^M\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\~}{\long}#1#2{ACTIVE TILDE Arg1: #1 Arg2: #2} \begingroup \catcode`\~=\active\show~\endgroup %------------------------------------------------------------------------- \DefineActiveChar{!}{\long}{!} \begingroup \catcode`!=\active\show! \expandafter\show!% \endgroup %------------------------------------------------------------------------- % end in case of running LaTeX: \csname stop\endcsname % end in case of running eTeX with plain TeX format: \bye

In case of using Knuthian TeX, where ε-TeX-extensions like \scantokens and \everyeof are not available, you can, e.g., do:

% Characters of category 0, 5, 9, 14 and 15 don't make it into explicit % character tokens, thus allow syntax both with explicit character tokens % and control symbol tokens. % % E.g., under normal circumstances you can't do % \DefineActiveChar{%}{<prefixes like \global>}<parameter text>{<replacement text}% % , but you can do % \DefineActiveChar{\%}{<prefixes like \global><parameter text>{<replacement text}% % . % Reading a single space from a line of .tex-input, be it a real text file % or be it with \scantokens' faking of unexpanded-writing and then inputting % of a text file, requires special attention because at the time of % pre-processing a line of .tex-input all space-characters at the end of the % line get stripped off. % With some TeX-implementations this is also the case with the horizontal-tab- % character. % Thus writing and reading a space-character to/from external text file % requires appending an additional comment-character. % At least with writing and reading linefed and carriage return % special attention might be required, too. % In LaTeX the \input-primitive is renamed to \@@input and the control sequence % \input is redefined not to be a macro: \begingroup \expandafter\ifx\csname @@input\endcsname\relax \endgroup \let\myinput=\input \else \endgroup \expandafter\let\expandafter\myinput\expandafter=\csname @@input\endcsname \fi % In some TeX engines the `\everyeof`-primitive is available, in others it is not: \expandafter\ifx\csname everyeof\endcsname\relax \def\myeveryeof=#1{}% \else \let\myeveryeof=\everyeof \fi \newwrite\scratchwrite \newtoks\scratchtoks \def\thescratchtoks{\the\scratchtoks}% \def\GatherActiveCharacterAsRelax#1{% % #1 = active character token to (re)define \let#1=\relax\NeutraLizeEOFMarker{#1}% }% \def\NeutraLizeEOFMarker#1#2{% % #1 = active character token to (re)define % #2 = letter X or - in case X is the active character to redefine - active X % ; ensures that in case #1 is space that space is not removed % during TeX's pre-processing of lines of input \endgroup \begingroup \let#1=\relax \scratchtoks={\GatherDefinitionPrefixAndDefine{#1}}% \myeveryeof={}% \futurelet\scratchmacro\thescratchtoks }% \long\def\GatherDefinitionPrefixAndDefine#1#2{% % #1 = active character token to (re)define % #2 = prefixes for `\def` like `\global` or `\long` \endgroup#2\def#1% }% \def\DefineActiveChar#1{% % #1 = character or one-letter-control-sequence for alphabetic constant \begingroup \escapechar=-1\relax \newlinechar=-1\relax \immediate\openout\scratchwrite scratchfile.tex \immediate\write\scratchwrite{% \ifnum`#1=10 \string^\string^J\else \ifnum`#1=13 \string^\string^M\else \string#1\fi\fi X% }% \immediate\closeout\scratchwrite \endgroup \begingroup \endlinechar=-1\relax \catcode`\^=7\relax \catcode`\X=11\relax \catcode`#1=\active \expandafter\expandafter\expandafter\GatherActiveCharacterAsRelax\expandafter\noexpand \myinput "scratchfile.tex" % }% {\catcode`!=\active \outer\gdef!{Huh?}}% %------------------------------------------------------------------------- \DefineActiveChar{!}{\long}#1#2{ACTIVE EXCLAMATION MARK Arg1: #1 test Arg2: #2} \begingroup \catcode`!=\active \show!\endgroup %------------------------------------------------------------------------- \DefineActiveChar{A}{\long}#1#2{ACTIVE A Arg1: #1 test Arg2: #2} \begingroup \catcode`A=\active \showA\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\!}{\long}#1#2{ACTIVE EXCLAMATION MARK AGAIN Arg1: #1 TEST Arg2: #2} \begingroup \catcode`!=\active \show!\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\%}{\long}#1#2{ACTIVE PERCENT Arg1: #1 Arg2: #2} \begingroup \catcode`\%=\active \show%\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\{}{\long}#1#2{ACTIVE CURLY LEFT BRACE Arg1: #1 Arg2: #2} \begingroup \catcode`\{=\active \show{\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\}}{\long}#1#2{ACTIVE CURLY RIGHT BRACE Arg1: #1 Arg2: #2} \begingroup \catcode`\}=\active \show}\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\#}{\long}#1#2{ACTIVE HASH Arg1: #1 Arg2: #2} \begingroup \catcode`\#=\active \show#\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\ }{\long}#1#2{ACTIVE SPACE Arg1: #1 Arg2: #2} \begingroup \catcode`\ =\active\show \endgroup %------------------------------------------------------------------------- \DefineActiveChar{ }{\long}#1#2{ACTIVE SPACE AGAIN Arg1: #1 Arg2: #2} \begingroup \catcode`\ =\active\show \endgroup %------------------------------------------------------------------------- \DefineActiveChar{\^^I}{\long}#1#2{ACTIVE HORIZONTAL TAB Arg1: #1 Arg2: #2} \begingroup \catcode`\^^I=\active\show^^I\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\\}{\long}#1#2{ACTIVE BACKSLASH Arg1: #1 Arg2: #2} \begingroup \catcode`\/=0 \catcode`\\=\active/show\/endgroup %------------------------------------------------------------------------- \DefineActiveChar{\$}{\long}#1#2{ACTIVE DOLLAR Arg1: #1 Arg2: #2} \begingroup \catcode`\$=\active\show$\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\&}{\long}#1#2{ACTIVE AMPERSAND Arg1: #1 Arg2: #2} \begingroup \catcode`\&=\active\show&\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\^}{\long}#1#2{ACTIVE CARET Arg1: #1 Arg2: #2} \begingroup \catcode`\^=\active\show^\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\_}{\long}#1#2{ACTIVE UNDERSCORE Arg1: #1 Arg2: #2} \begingroup \catcode`\_=\active\show_\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\^^A}{\long}#1#2{ACTIVE START OF HEADING Arg1: #1 Arg2: #2} \begingroup \catcode`\^^A=\active\show^^A\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\^^J}{\long}#1#2{ACTIVE LINE FEED Arg1: #1 Arg2: #2} \begingroup \catcode`\^^J=\active\show^^J\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\^^M}{\long}#1#2{ACTIVE CARRIAGE RETURN Arg1: #1 Arg2: #2} \begingroup \catcode`\^^M=\active\show^^M\endgroup %------------------------------------------------------------------------- \DefineActiveChar{\~}{\long}#1#2{ACTIVE TILDE Arg1: #1 Arg2: #2} \begingroup \catcode`\~=\active\show~\endgroup %------------------------------------------------------------------------- \DefineActiveChar{!}{\long}{!} \begingroup \catcode`!=\active\show! \expandafter\show!% \endgroup %------------------------------------------------------------------------- % end in case of running LaTeX: \csname stop\endcsname % end in case of running eTeX with plain TeX format: \bye

Let's emphasize again:

This piece of code is an "academic thing". In practice applying \uccode+\uppercase/\lccode+\lowercase- or \scantokens+\everyeof-trickery is much more convenient.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.