9

This question led to a new package:
xpeek

I’ve been developing my xpeek package with lots of help from you folks on TeX.SX. It’s intended to help write commands which, like the familiar \xspace, look ahead into the input stream and choose what to do based on what follows. Although the interface needs to change, it has enough functionality to implement something like xspace, or the intelligent italic-correction from LaTeX's \textit.

Which means it has the same familiar limitation as \textit, best illustrated by example:

\textit{foof}\xspace. \textit{foof}\xspace! % bad italic correction before "." \def\nocorrlist{.,\xspace} \textit{foof}\xspace. \textit{foof}\xspace! % no italic correction before "!"

Thinking about the problem, it seems I need the ability to scan ahead in the input stream, ignoring tokens from one list while looking for tokens from another. In Expl3 terms, I’m hoping to define something like \peek_inlist_ignore_auxlist:nnTF.

As an initial step toward this end, I’m trying to simply scan ahead for the next input token not on the ignore list. For example:

\def\ignorelist{ ,.;:!} \newcommand{\nextnonpunc}{\scanignoring{\ignorelist}`\scanfound'} The next non-punctuation mark is \nextnonpunc.,;:! xyz

yielding

The next non-punctuation mark is ‘x’.,;:! xyz

(Actually, since I’m using Expl3, the syntax would be

\ExplSyntaxOn \tl_const:Nn \ignorelist {~,.;:!} \NewDocumentCommand { \nextnonpunc } { } { \peek_ignore_list:N \ignorelist `\l_peek_token' } \ExplSyntaxOff

or something like that.)

So how can I implement \peek_ignore_list:N?

The direction I’m considering is to read ahead, consuming tokens. Each token read is added to a save-list and compared to the ignore-list. If it’s on the ignore-list, continue; otherwise put the save-list back on the input stream and stop.

Does this sound reasonable so far?

To consume tokens one-by-one, I built this function:

\cs_new_protected:Npn \peek_meaning_really_remove:NTF #1 #2 #3 { \peek_meaning_remove:NTF #1 { #2 } { \peek_meaning_remove:NT \l_peek_token { #3 } } }

but the question above still stands: How can I implement \peek_ignore_list:N?

Improve this question
5
  • 1
    it's either feasible or not, depending on what the rules are. Eg, if you need to be able to cope with potentially seeing brace groups you probably have to use \futurelet and test for \bgroup like 2e's \@ifnextchar but futurelet _consumes the token, leaving it's definition in the token that it defines, but this means that if (say) you see an a and want to put it back you don;t know it was an a character just a \csname that has been \let to a. that may or may not matter, depending on what the tokens that you put back are supposed to do, but in general you don't know that. Commented Jul 19, 2012 at 19:46
  • See my edit; I’m OK with putting things back and am willing to accept some restrictions on what can be ignored so long as “\xspace” is ignorable. Commented Jul 19, 2012 at 19:50
  • 1
    I fear to misunderstand you once again today, @David: What do you mean by "\futurelet consumes the token"? Commented Jul 19, 2012 at 20:08
  • I meant if you go \futurelet\tmp\mymacro a then a is removed from the input stream. But as you are tying to politely point out, that's false. Maybe I'll go back to MathML:-) Commented Jul 19, 2012 at 20:15
  • 1
    Nooooo! (But thanks for confirming that it doesn't consume the token.) Commented Jul 19, 2012 at 20:17

2 Answers 2

Reset to default
5

Here is my humble attempt. I'm quite convinced that there must be better ways but it seems to do what you need.

The following lines

\AddToIgnoreList{,.;:!\xspace} \nextnonpunc,.;:!xyz \nextnonpunc,.;:! xyz \nextnonpunc,.;:!\xspace xyz \nextnonpunc,.;:!{something}

give this output:

enter image description here

Here is the complete code:

\documentclass{article} \usepackage{expl3,xparse,xspace} \ExplSyntaxOn % ---------------------------------------- % variables: \tl_new:N \l_solomon_current_ignore_token_tl \tl_new:N \l_solomon_ignorelist_tokens_tl \tl_new:N \l_solomon_ignored_tokens_tl % ---------------------------------------- % setup the list: \cs_new:Npn \solomon_add_to_ignorelist:n #1 { \tl_put_right:Nn \l_solomon_ignorelist_tokens_tl { #1 }} \cs_new:Npn \solomon_remove_from_ignorelist:n #1 { \tl_map_inline:nn { #1 } { \tl_remove_all:Nn \l_solomon_ignorelist_tokens_tl { ##1 } } } \NewDocumentCommand \AddToIgnoreList { m } { \solomon_add_to_ignorelist:n { #1 } } \NewDocumentCommand \RemoveFromIgnoreList { m } { \solomon_remove_from_ignorelist:n { #1 } } % ---------------------------------------- % check if next token is part of the ignore-list; store it in % \l_solomon_current_ignore_token_tl \cs_new:Npn \solomon_check_ignorelist:NTF #1#2 { \solomon_check_ignorelist_auxi:nTF { 0 } { #1 } { #2 } } \cs_new:Npn \solomon_check_ignorelist_auxi:nTF #1#2#3 { % get the first token from the list and compare: \solomon_get_head:NN \l_solomon_current_ignore_token_tl \l_solomon_ignorelist_tokens_tl \peek_meaning_remove:VTF \l_solomon_current_ignore_token_tl { #2 } { \solomon_check_ignorelist_auxii:nnn { #1 } { \solomon_check_ignorelist_auxi:VTF \l_tmpa_int { #2 } { #3 } } { \solomon_get_head:NN \l_solomon_current_ignore_token_tl \l_solomon_ignorelist_tokens_tl #3 } } } \cs_new:Npn \solomon_check_ignorelist_auxii:nnn #1#2#3 { % test was successful; rotate the list and go on if there are more \int_set:Nn \l_tmpa_int { #1 } \int_incr:N \l_tmpa_int \tl_remove_all:NV \l_solomon_ignorelist_tokens_tl \l_solomon_current_ignore_token_tl \tl_put_right:NV \l_solomon_ignorelist_tokens_tl \l_solomon_current_ignore_token_tl \int_compare:nTF { \l_tmpa_int < \tl_length:V \l_solomon_ignorelist_tokens_tl } { #2 } { #3 } } \cs_generate_variant:Nn \solomon_check_ignorelist_auxi:nTF { V } \cs_generate_variant:Nn \peek_meaning_remove:NTF { V } \cs_generate_variant:Nn \tl_remove_all:Nn { NV } \cs_generate_variant:Nn \tl_put_right:Nn { NV } \cs_new:Npn \solomon_get_head:NN #1#2 { \exp_after:wN \solomon_get_head_aux:Nw \exp_after:wN #1#2 \q_stop } \cs_new:Npn \solomon_get_head_aux:Nw #1#2#3 \q_stop { \tl_set:Nn #1 { #2 } } % ---------------------------------------- % now do some testing: \cs_new:Npn \solomon_nextnonpunc: { \solomon_check_ignorelist:NTF { \tl_put_right:NV \l_solomon_ignored_tokens_tl \l_solomon_current_ignore_token_tl \nextnonpunc } { (The~next~non-punctuation~mark~is:~`\l_peek_token')~ \tl_use:N \l_solomon_ignored_tokens_tl \tl_clear:N \l_solomon_ignored_tokens_tl } } \NewDocumentCommand \nextnonpunc {} { \solomon_nextnonpunc: } \ExplSyntaxOff \begin{document} \AddToIgnoreList{,.;:!\xspace} \nextnonpunc,.;:!xyz \nextnonpunc,.;:! xyz \nextnonpunc,.;:!\xspace xyz \nextnonpunc,.;:!{something} \end{document}
Improve this answer
8
  • Please note that \tl_length:(N|c|n|V|o) will be renamed into \tl_count:(N|c|n|V|o) Commented Jul 23, 2012 at 8:55
  • It's renamed in the new version Commented Jul 23, 2012 at 9:40
  • That was impressive! I'm going to have to study this before I can incorporate it into xpeek, especially with the all the :w commands. Commented Jul 24, 2012 at 3:36
  • @J.C.Salomon I'm glad if I could help and curious to see an updated version of xpeek! I just realised I mispelled your name, sorry! Commented Jul 24, 2012 at 8:10
  • \solomon_check_ignorelist:NTF should be \solomon_check_ignorelist:TF, since it takes two arguments. Commented Aug 3, 2012 at 13:04
1

Here, I use a non-expl3 token cycle to scan ahead in the input stream. I'm not sure if it is the desired behavior, but I made it so that any group (catcode 1) in the input stream is diagnosed as ending the search for additional punctuation.

\documentclass{article} \usepackage[T1]{fontenc} \usepackage{tokcycle,xspace} \makeatletter \def\abortcycle{\tcpush{\noexpand\endtokcycraw\sub@tcdepth}} \makeatother \newcommand\Attention[1]{(The next non-punctuation is `\detokenize{#1}')} \def\IgnoreList{} \long\def\AddToIgnoreList#1{\expandafter\def\expandafter\IgnoreList \expandafter{\IgnoreList#1}} \long\def\testIgnore#1{\long\def\testIgnoreaux##1#1##2\endIgnores{% \expandafter\ifx\expandafter\relax\detokenize{##2}\relax \Attention{#1}\abortcycle\else\fi\addcytoks{#1}}% \expandafter\testIgnoreaux\IgnoreList#1\endIgnores} \tokcycleenvironment\nextnonpunct {\testIgnore{##1}} {\groupedcytoks{\addcytoks{##1}}\Attention{{##1}}\abortcycle} {\testIgnore{##1}} {\testIgnore{##1}} \stripgroupingtrue \begin{document} \AddToIgnoreList{,.;:!\xspace} \nextnonpunct xyz \nextnonpunct,.;:! xyz \nextnonpunct,.;:!\xspace xyz \nextnonpunct,.;:!\xspace ,xyz \nextnonpunct,.;:!{something} else \end{document}

enter image description here

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.