String.split(String) won't create regexp if your pattern is only one character long. When splitting by single character, it will use specialized code which is pretty efficient. StringTokenizer is not much faster in this particular case.
This was introduced in OpenJDK7/OracleJDK7. Here's a bug reportHere's a bug report and a commit. I've made a simple benchmark here.
$ java -version java version "1.8.0_20" Java(TM) SE Runtime Environment (build 1.8.0_20-b26) Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode) $ java Split split_banthar: 1231 split_tskuzzy: 1464 split_tskuzzy2: 1742 string.split: 1291 StringTokenizer: 1517