CPSC 220 Fall 2004
Program 1: A Token Scanner

1. Write a class Scanner with the following interface:
   • Scanner(String filename) -- creates a scanner object that reads from file filename.
   • String next() -- returns the next token in the file, null if there is no next token.
   • boolean hasNext() -- returns true if there is another token available (i.e., the file is open and it's not EOF), false otherwise.

   The Scanner constructor opens the given file for reading text, and nextToken() returns the next token in the file. For our purposes a token is one of two things:

   • A string of contiguous alphabetic and numeric characters terminated by any non-aphabetic/non-numeric character (including whitespace) or EOF.
   • A single non-alphabetic, non-numeric character.

   So the text below contains the 12 tokens shown:

      public class       Test5
      {
   
        int      
                x;
      }
      //Very silly
   
   
   Tokens:  public
            class
            Test5
            {
            int
            x
            ;
            }
            /
            /
            Very
            silly
   

   Things to keep in mind:
   • Whitespace may include spaces, tabs, and end-of-lines and may be encountered at any place, and in any combination, in the file. In particular, the file may start or end with whitespace.

   • Most Java IO methods throw exceptions, so you will have to deal with these using try and catch. However, your Scanner methods should not throw any IO exceptions. If an IO exception is thrown when you try to open the file (in the constructor), catch it, print an appropriate warning message, and initialize the scanner to indicate that there is no next token (as if it were EOF). If an IO exception is thrown while you are reading a token, catch it, print an appropriate warning message, and return null.

   • Since you are reading text from a file, you'll want to use a FileReader. (You may want to refer to the online Java documentation at http://java.sun.com/j2se/1.4.2/docs/api to review the FileReader API.) Remember that FileReader has a int read() method that returns the character read, or -1 if it's EOF. Of course, to use the returned value as a character you'll have to cast it to char.

   • Since you detect EOF based on the value of the last character read, you may find it useful to store that character in an instance variable. This variable will retain its value between calls to next(), and can be used to determine what hasNext will return.

   • You may find it useful to write a little method to skip whitespace, leaving the first non-whitespace character in an instance variable.

   • You may also find it useful to write a little method to read an alphanumeric token once the first character of one is encountered.

2. Write a class TestScanner that prompts for and reads in a filename, then creates a Scanner object and uses it to get the tokens in that file. You may use the methods of the Keyboard class to read if you like; remember to import cs1.Keyboard. As you get the tokens, list them to the standard output, one per line. At the end, print the total number of tokens found in the file.

   Be sure to test your program on a variety of input files, including (but not limited to) the following:

   • files that begin and/or end with whitespace
   • files that contain only a single token
   • an empty file
   • files containing only whitespace
   • various combinations of alphanumeric tokens (of various lengths), nonalphanumeric tokens (which will always be of length 1), and whitespace.

What to Turn In