String.tokens : (char -> bool) -> string -> string list String.tokens takes a function, a string, and a list of
strings. The function takes a character and returns true if that character should be considered a delimiter, that is, something
that separates tokens. It then breaks the string into tokens accordingly, returning a list of those tokens represented as strings.
The delimiter(s) are not included.
- String.tokens Char.isSpace "this is a test" => ["this","is","a","test"]
- String.tokens (fn c => (c = #".") orelse (c = #"*")) "one.two*three" => ["one","two","three"];
This is not unlike the capabilities of the StringTokenizer and Scanner
classes in Java.
Of course, you can't count on tokens being nicely delimited in program
input; for example, given the string
x:=3+foo-(AnotherID/123.4) you need to be able to extract the tokens x, :=, +, foo, -, (, AnotherID, /, 123.4, and ).
There is another ML function that can help you with this: