I'm having trouble understanding how tokens are counted in the StringTokenizer class with constructors that have multiple arguments.
String line = ("This is a sample sentence, how many tokens are inside this sentence?") ; new StringTokenizer(String str) new StringTokenizer(String str, String delims ) If i use the first StringToknenizer constructor with a single argument of line and write a sample program the result is "thisisasamplesentencehowmanytokensareinside?" and 12 tokens. It returns the whole sentence without any spaces. I understand how this works.
If use the second constructor with two arguments, (line, ",") my test program will get "this is a test sentence how many tokens are in this sentence? " with spaces for each word but no comma and only 2 tokens. I thought it should count BOTH the spaces and the comma as token separators but it counts everything before the comma as 1 token and everything after the comma as 1 token. This part is confusing me.
My problem is I don't understand how tokens are separated using delimiters using constructors that take multiple arguments such as (line, " ,"). Am i missing something or misunderstanding something?
