(*

# Overview

In this lecture, will talk about parsing.  

In most programming languages, programs start off as text (files on
disk, which can be read in as strings or lists of characters). 
This is called the "concrete syntax".   

Next time, when we define the operational semantics of what it means to
run a program, we will see that it is much easier if the program is
represented as a tree, called an "abstract syntax tree".  This is
because the abstract syntax tree groups the operands of an operator into
subtrees of the operator.

Last time, we introduced a language that we will call Imp 
with the following kinds of expressions

* numeric constants for natural numbers, like 0 , 1 , 2 ... 
* addition, like (1 + 2) or (11 + (2 + 3))
* geting the contents of a memory address, like get(2)

The goal of parsing is to turn the text of a program, 
like the string 
  "(11 + (2 + 3))"
into an abstract syntax tree, like
  Plus(Num 11, Plus(Num 2, Num 3))
which can be visualized as

     Plus
      /\
     /  \
    /    \
 Num 11  Plus
          /\
         /  \
        /    \
     Num 2   Num 3

As another example, the string
  "get(1 + 1)"
should be parsed as
  Get(Plus(Num 1, Num 1))
i.e.

    Get
     |
     |
   Plus
     /\
    /  \
   /    \
Num 2   Num 3

*)


(*

# Tokenizing

The process of parsing is typically broken up into two stages.  Think of
the input string as a sequence of "letters".  The first, called lexing
or tokenizing, breaks the string up into a sequence of "words" instead
of letters.  What constitutes a "word" in a program?  Typically:

* numeric constants.  For example, in the input string, the digits 1 and
  1 of the number 11 are two separate characters, but lexing will turn
  them into a single token representing the number 11

* keywords. For example, the keyword get will be recognized as a single token.

* punctuation symbols. For example, parentheses, semicolons, and other
  characters that are part of the syntax of the language.

* variable names.    

For the language above, we will say that a token is either
NUM k, where k is an int, or
GET, or
PLUS, or
LPAREN, or 
RPAREN.

NUM k represents a numeric constant, GET represents the 'get' keyword,
while PLUS and LPAREN and RPAREN correspond to the + and ( and )
characters.

We can represent a token with the following datatype:

*)

datatype token =
    NUM of int
  | GET
  | PLUS
  | LPAREN
  | RPAREN

(* Here is a tokenizer for this set of tokens: *)

(* test if a char represents a digit by testing if its ASCII code is between
   the ASCII code for 0 (48) and the ASCII code for 9.

   Note that in SML a single character literal is written #"c".  
*)
fun is_digit(c : char) : bool =
    ord #"0" <= ord c andalso ord c <= ord #"9"

(* Given a list of characters l and an int sofar representing the
   integer that was to the left of l, combine sofar and whatever digits
   are at the front of l into a single number, and also return whatever
   is left of l starting with the first non-digit character.  *)
fun tokenize_number (l : char list, sofar : int) : int * char list =
    case l of
        [] => (sofar , [])
      | x :: xs =>
            if is_digit x
            then tokenize_number(xs , 10 * sofar + (ord x - ord #"0"))
            else (sofar, x::xs)
val (123, []) = tokenize_number(String.explode "23", 1)
val (123, [#" "]) = tokenize_number(String.explode "23 ", 1)
val (123, [#" "]) = tokenize_number(String.explode "3 ", 12)
    
fun tokenize (l : char list) : token list =
    case l of
        [] => []
      | #"(" :: xs => LPAREN :: tokenize xs
      | #")" :: xs => RPAREN :: tokenize xs
      | #"+" :: xs => PLUS :: tokenize xs
      | #"g" :: #"e" :: #"t" :: xs => GET :: tokenize xs
      | x :: xs => if is_digit x
                   then
                       let val (n , rest) = tokenize_number (xs, ord x - ord #"0")
                       in
                           NUM n :: tokenize rest
                       end 
                   else raise Fail "tokenizer: unexpected character"

val (NUM 123) :: [] = tokenize (String.explode "123")
val [GET, LPAREN, NUM 1, RPAREN ] = tokenize (String.explode "get(1)")
val [RPAREN, PLUS, NUM 1, LPAREN,GET ] = tokenize (String.explode ")+1(get")


(* 

For the first few cases, we turn a single charaacter into a single token.

Next, we recognize the sequence of characters g, then e, then t as the
single token get.

For numeric tokens, we need to convert the text of a numeral to a
number, which is what the tokenize_number function does.  Note that the
'ord' function gets the ASCII code of a character, and that ord x - ord #"0"
is a slick way of converting a single digit character into a number.

*)
    
(* 

# Parsing 

Now that we have a list of tokens, the parser's job is to make the list
into a tree.

We will represent a tree as follows: 

*)

datatype exp =
    Num of int
  | Get of exp
  | Plus of exp * exp

(* 

This says that a Num node contains an int,
a Get node has one subtree, and 
a Plus node has two subtrees.   

The parser will parse an expression from the front of a list of tokens,
and return whatever tokens are leftover after the ones that make up an
expression.  

Here are a few cases of the parser:

*)

fun parse (l : token list) : exp * token list =
    case l of
        [] => raise Fail "expected some token"
      | NUM k :: xs => (Num k , xs)
      | GET :: LPAREN :: xs =>
            (* If the first token is a GET, the next token needs to be an LPAREN
               Recursively, we parse an expression after that.
               Then, after that expression is parsed, the leftover tokens 
               must start with an RPAREN.  
               This makes the syntax tree Get e, where e is whatever expression 
               the recursive call found. 
               *)
            (case parse xs of
                 (e , RPAREN :: rest) => (Get e , rest)
               | _ => raise Fail "expected RPAREN")
      | GET :: _ => raise Fail "expected LPAREN"
      | _ => raise Fail "unexpected token"

fun run(s : string) : exp * token list =
    parse(tokenize (String.explode s))

val (Get(Num 11),[]) = run ("get(11)")
val (Get(Get(Num 11)),[]) = run ("get(get(11))")
    
(* 

# Grammars

The syntax of a programming language is typically specified with a
grammar, which is a description of which strings a parser should accept.

A grammar for the strings of the above language looks like this:

d ::= 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
k ::= d | dk 
e :: k | get(e) | (e1 + e2) 

This can be read as follows:
* a digit is either 0 or 1 or 2 ... (read ::= as "is either" and | as "or")
* a numeric constant is either a digit or a digit followed by a numeric constant
* an expression is either a numeric constant, or get( e ) where e is any string 
  that parses as an expression,
  or (e1 + e2) where e1 and e2 are any strings that parse as expressions.

We can also give a grammar for sequences of tokens:

e ::= NUM k | GET LPAREN e RPAREN | LPAREN e1 PLUS e2 RPAREN 

And finally for abstract syntax trees:

e ::= Num(k) | Get(e) | Plus(e1,e2)

End-to-end, the parser should turn any string matching the first grammar
into a syntax tree matching the third grammar.

After this week, we will mainly work at the level of abstract syntax
trees, with the understanding that you could write a parser to turn
concrete syntax into abstract syntax.

*)