11
$\begingroup$

According to the documentation of WordCharacter:

WordCharacter matches any character for which either LetterQ or DigitQ yields True.

However, the results of the following code seem not compatible with the doc:

StringMatchQ[#, WordCharacter] & /@ {"a", "1", ".", " ", "中", "あ"} LetterQ[#] || DigitQ[#] & /@ {"a", "1", ".", " ", "中", "あ"}

{True, True, False, False, True, True}

{True, True, False, False, False, False}

I suppose there is a special rule to handle CJK characters, so it's the doc which should be corrected, am I right?

(As the motivation of this "discovery", I was trying to match any characters not belong to CJK.)

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Your example does not violate what the documentation states, it's just that the reverse isn't true (it would seem): WordCharacter does not not match any character which neither LetterQ nor DigitQ matches. $\endgroup$ Commented Jul 25, 2015 at 3:14
  • $\begingroup$ @Pickett Your logic is right of course. But I can't agree with you, as it would make the doc sounding like playing with words... $\endgroup$ Commented Jul 25, 2015 at 3:34
  • 5
    $\begingroup$ The documentation is certainly wrong. LetterQ is only usable for the Latin and Greek alphabets, and even then only the letters that exist in English and modern Greek. StringMatchQ relies on the PCRE library, so it considers WordCharacter whatever PCRE does. This extends at least to some other alphabets/character types. $\endgroup$ Commented Jul 26, 2015 at 21:48
  • 1
    $\begingroup$ The result is difference with you here $\endgroup$ Commented Mar 21, 2017 at 20:14
  • $\begingroup$ Thanks! @yode I just tried and my 11.0 agrees with your result. I think Alexey Popkov's answer addressed the change. $\endgroup$ Commented Mar 22, 2017 at 18:43

2 Answers 2

Reset to default
4
$\begingroup$

Under the "Implementation Details" section of the official tutorial "Working with String Patterns" we read:

Because PCRE currently does not support preset character classes with characters beyond character code 255, the word and letter character classes (such as WordCharacter and LetterCharacter) only include character codes in the Unicode range 0–255. Thus LetterCharacter and _?LetterQ do not give equivalent results beyond character code 255.

The above-cited Documentation section wasn't updated since Mathematica version 5.2 (July-2005), and shouldn't be correct anymore because starting from PCRE version 8.32 (30-November-2012) the latter supports Unicode:

PCRE_UCP

This option changes the way PCRE processes \B, \b, \D, \d, \S, \s, \W, \w, and some of the POSIX character classes. By default, only ASCII characters are recognized, but if PCRE_UCP is set, Unicode properties are used instead to classify characters. More details are given in the section on generic character types in the pcrepattern page. If you set PCRE_UCP, matching one of the items it affects takes much longer. The option is available only if PCRE has been compiled with Unicode property support.

<...>

In UTF-8 (UTF-16, UTF-32) mode, characters with values greater than 255 (0xffff) can be included in a class as a literal string of data units, or by using the \x{ escaping mechanism.

Starting from Mathematica version 10 the two lines of code in the question produce identical results but in general these tests are different, and (as of Mathematica version 11.1.0) the results aren't the same for 705 characters from the Unicode table as one can see from the following:

unicode = FromCharacterCode /@ Range[0, 65276]; test1 = StringMatchQ[#, WordCharacter] & /@ unicode; test2 = LetterQ[#] || DigitQ[#] & /@ unicode; Pick[unicode, MapThread[Unequal, {test1, test2}]]

screenshot

Improve this answer
$\endgroup$
1
  • $\begingroup$ Nice finding! Thanks! So maybe I should stick with a self-maintained Unicode block table. $\endgroup$ Commented Mar 22, 2017 at 19:51
6
$\begingroup$

This is a pretty ugly hack, but maybe it will inspire you to something better.

leastCJK = ToCharacterCode["⺀"][[1]]; StringMatchQ[#, _?(ToCharacterCode[#][[1]] < leastCJK &)] & /@ {"a", "1", ".", " ", "中", "あ"} 
{True, True, True, True, False, False}

Note: "⺀" is unicode character U+2E80, CJK RADICAL REPEAT

This hack can be used with Repeat to ignore CJK strings

StringCases[ "the quick fox...本当に日本語を勉強していますか...jumped", Repeated[_?(ToCharacterCode[#][[1]] < leastCJK &)], Overlaps -> False] 
{"the quick fox...", "...jumped"}
Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks @m_goldberg ! This is how I'm dealing the problem right now. I hope there be a CJKCharacter :) $\endgroup$ Commented Jul 26, 2015 at 16:24
  • $\begingroup$ Hi @m_goldberg , I think Alexey's answer gives very insightful information. If it's fine with you, I would like to accept it. $\endgroup$ Commented Mar 22, 2017 at 18:48
  • $\begingroup$ @Silvia. Please feel free to do so. $\endgroup$ Commented Mar 22, 2017 at 18:51

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.