I want to perform a simple analysis of some Greek text: collect the different words used and count their frequency. It seems that some of the built-in commands do not work well with Greek letters. For instance, with
words = StringCases["α β1 rpr other", WordCharacter ..] the output is
{"1", "rpr", "other"} How do I get Mathematica to recognize other kinds of words?
I have partial success in splitting Greek text:
greek = ExampleData[{"Text", "HomerOdysseyGreek"}] Style[StringTake[greek, 100], FontFamily -> "Times"] Style[StringSplit[StringTake[greek, 100], " "], FontFamily -> "Times"] 
StringSplit is essentially all I need, and it works great. It correctly identifies words in the Greek. $\endgroup$ I'd say this is either a programming or a documentation bug.
The documentation for WordCharacter says:
WordCharacter matches any character for which either LetterQ or DigitQ yields True. »
Well, WordCharacter clearly doesn't consider alpha a letter:
StringMatchQ["α", WordCharacter] False
but LetterQ does:
LetterQ["α"] True
A workaround:
theGreeks=Alternatives@@Select[CharacterRange["\[CapitalAlpha]", "\[Omega]"], LetterQ]; StringCases["α β1 rpr other", (WordCharacter | theGreeks) ..] {"α", "β1", "rpr", "other"}
You may want to enlarge the set of characters included in "theGreeks". Now only, the basic Greek characters are in there.
StringMatchQ["α", RegularExpression["[[:word:]]"]] and StringMatchQ["α", RegularExpression["\\w"]] also return False. $\endgroup$
StringCases["α β1 rpr0 other", RegularExpression["(\\w|[\[CapitalAlpha]-ω])+"]]. A bit cumbersome, but there ya go... with the caveat that this can't handle characters with tonos; you'll have to modify the regex as needed. $\endgroup$