Let gettoken() be a function that returns the next token in the input. if we call it repeatedly it will return one token after a
nother. when all the input is consumed, gettoken() returns eof (end of file). assume that longest prefix-matching rule is used by gettoken() and ties are broken in favor of the regular expression listed first.
