• -- khellman 2020-05-04 15:22:50 Add rubric, correct minor typo ("it" for "its")

  • -- khellman 2020-05-03 16:48:11 Clarified (hopefully) the simplification of control structures.

  • -- khellman 2020-05-03 13:18:07 Clarified unary PLUS and added BOOLS in expression operation table.

  • You can download all the resources used in this writeup: zobos-resources.zip or zobos-resources.tar.bz2.
    The tarball contains source files (.src, .syn), token streams (.tok), AST PDFs, and verbose and OUTPUT filtered stdout listings (.vrb and .out) respectively.

  • Here is the change log for this assignment write-up. I will try to be descriptive in my log messages.

  • You can also subscribe to this page and receive Emails when changes are made.

In this assignment will you will implement a syntax checking LR(1) parser, a symbol table(s), and perform semantic analysis on a simple programming language grammar. The name of your application must be ZOBOS (case sensitive).

Input

ZOBOS will accept three command line parameters:

Argument

Value

Description

1

program.tok

The filename containing a token stream to be parsed, the format is identical to that produced by LUTHER

2

ast.dat

The filename for storing your AST

3

symtable.dat

The filename for storing symbol tables when instructed.

Coding

ZOBOS will do do the following:

  1. Your compiler will parse zlang.cfg using the LR knitting needles algorithm (pseudo code here, using this SLR table (the format is described here). Syntax error messages will be produced in a consistent manner to facilitate grading.

    • If program source has a syntax error, the remaining steps for ZOBOS are not to be performed and ZOBOS should exit() with a non-zero exit status.

    • It turns out that for zlang.cfg, an empty file is a valid program, but ZOBOS won't be tested with empty source files. So you don't need to contemplate "is ∅ a SYNTAX error?"

  2. Either during the parse (using syntax directed translation procedures during production rule reduction), after the parse using visitor patterns, or a combination of these methods; ZOBOS will construct an abstract syntax tree with the following properties

    1. all EXPR (sub)trees will be simplified so that

      1. leaves are either literals (intvals, floatvals) or variables

      2. the root and internal nodes are one of the operations associated with the non-terminals BOOLS, PLUS, MULT, UNARY, or CAST.

    2. control structures IF, IFELSE, WHILE, STMTS, and BRACESTMTS will be simplified in the spirit of textbook figure 7.15 (lga-sdt.pdf) --- essentially lose the terminals, keep the functional children (blocks of statements, predicate expressions).

    Simplification of all other grammar constructs are at the student's discretion and inclination. They will not be graded.
    There are example .src and AST PDFs in the zobos-resources archives for download.

    Outside of the rules listed above, you are not required to duplicate the ASTs provided, they are provided as examples!

  3. ZOBOS will write to disk a simple tree representation of its AST.

  4. ZOBOS will then use a scope respecting symbol table and visitor pattern(s) to perform several semantic checks on the variable and expression use of the parsed program source.

    1. When the special EMIT form of symtable is encountered during this process, your ZOBOS will write the symbol tables for all scopes to disk in a simple format to facilitate grading.

    2. Warnings or error messages will be emitted in a consistent and simple manner to facilitate grading.

  5. Lastly, ZOBOS will exit(0) if there were no errors detected in the program, and exit(1) otherwise.

    WARNING messages are not ERROR messages.

Syntax and Semantic Checks

Your ZOBOS compiler will inspect the program.tok for these types of errors and inconsistencies.

Keep in mind that we often clump all errors into the syntax error, pigeonhole. Technically, syntax errors occur only during the parse when an invalid pattern of terminals is detected. The other errors that compilers find for us, and most of them in this project, are semantic errors and inconsistencies.

Type

ID

Description

Terminal Location to report (line and column)

SYNTAX

SYNTAX

Any syntax error during the parse

Token causing error, or last token successfully processed

ERROR

NOVAR

Using an undeclared variable

Each occurrence of undeclared variable use

ERROR

CONV

Value conversion error (see the conversion errors table)

Location of the assign terminal

ERROR

EXPR

Expression tree operand error (see the expression errors table)

Location of the PLUS, TIMES or UNARY terminal operating on the erroneous value or variable

WARN

REVAR

Attempting to re-declare a variable

Each occurance of variable name in redeclaration

WARN

UNUSED

The variable is not used in an expression or assignment within it's scope

Location of variable name in its declaration

WARN

UNINIT

Using a variable in an expression before it has been initialized with a value

Location of variable name in expression

WARN

CONST

Attempting to store a value in a variable with the const attribute

Location of variable name on LHS of assign terminal

Conversion Errors (CONV)

Attempting to perform these types of data storage causes a CONV ERROR.

Value Type

Stored into Variable Type

string

int, float or bool

float

int, bool or string

bool

float or string

int

bool or string

Expression Value Propagation Rules and Errors (EXPR)

The type of value result in expression trees is always one of bool, float, int or string. An expression tree value type begins in the leaves and is implicit based on the type of literal value (intval, stringval, floatval) or variable is in the leaf.

Expression Operation Rules (generate EXPR ERROR).

unary PLUS and binary PLUS, TIMES, and BOOLS cannot be used on string or bool values

mod (modulo arithmetic) operation must be used on int values (both operands)

compl (bitwise complement) unary operation must be used on an int value

not (logical not) unary operation must be used on a BEXPR or bool value

Is it ever safe to stop looking for EXPR errors in poorly formed expression trees? Evaluation of an expression tree works from the bottom up (recursion is used to get down to the bottom).

  1. Each node should look for EXPR errors under each child branch.

  2. When an EXPR error has occurs in any child, a parent node can no long evaluate the correctness of its operation and so should not emit any EXPR errors.

  3. If none of a parent node's children report EXPR errors, the parent should evaluate the correctness of its operation.

  4. Except for the root, all nodes communicate either an error condition or a resultant value type for its operation to its own parent.

OUTPUT

ZOBOS will report three types of data to three different files.

Syntax, Warning and Error Messages

These will be written to system stdout and must be prefixed by the token OUTPUT as described by the course submission requirements. These messages will have four critical parts emitted in this order:

  1. the type of message :TYPE:, enclosed with colons1 where TYPE is one of (all capitals) SYNTAX, ERROR, or WARN.

  2. the line number of error in source (see Terminal Location column of syntax, error, and warning table)

  3. the character number of error in source (see Terminal Location column of syntax, error, and warning table)

  4. the ID for the message type

The line and character number come from the program.tok token stream. For a SYNTAX error it is the location of the offending token or the the last good token iff the syntax error occurs because there are no more tokens to be parsed. For WARN and ERROR messages, the location to report is detailed here.

Example Syntax Error Output

Since we abort all tasks on a syntax error, there should only be one SYNTAX error ever printed on stdout. You are permitted to print other information, just don't put it on the required OUTPUT line:

danglingbrace.vrb (the source danglingbrace.syn is available in the ZOBOS archive files)

OUTPUT :SYNTAX: 7 1 :SYNTAX:
state /1/ input ('rbrace', '}') expected bool,const,emit,float,id,if,int,lbrace,string,while,$

You aren't required to print out anything beyond the data on the OUTPUT line.

Abstract Syntax Tree

After grading the LG ptvis learning group check this weekend, I realized many groups are using the graphviz.org tools for generating their parse trees. Therefore, you have the option of writing a dot(1) instruction file instead of the format described below. I'll make sure the grader.sh script detects these cases correctly.

ZOBOS will write its abstract syntax tree to the filename specified by the second command line parameter. The format of this file is in two parts. The first part is a line by line declaration of a tree node id and name, separated by white space. If a name is not provided the id is used. An empty line separates the the first part from the second part.

The second part is a line by line description of tree edges. The first id on the line is the parent, subsequent ids are the children in left to right order. If a parent id is mentioned more than once, children are simply arranged in left to right, file order. The following two AST descriptions in this format yield the same tree graph:

Source (word.src)

AST Description (word.ast)

dot() Generated Graph

A A
B B
C C
D D

A B C D

A A
B B
C C
D D

A B
A C D

astfmt1.svg

Source (helloworld.src)

AST Description (helloworld.ast)

dot() Generated Graph

string word = "hello world";
emit word 0 11;

0 PROGRAM
0-0 STATEMENT
0-0-0 DECLLIST
0-0-0-0 string
0-0-0-1 DECLID
0-0-0-1-0 =
0-0-0-1-0-0 word
0-0-0-1-0-1 hello world
0-1 STATEMENT
0-1-0 EMIT
0-1-0-0 word
0-1-0-1 0
0-1-0-2 11
0-2 $

0-0-0-1-0 0-0-0-1-0-0 0-0-0-1-0-1
0-0-0-1 0-0-0-1-0
0-0-0 0-0-0-0 0-0-0-1
0-0 0-0-0
0-1-0 0-1-0-0 0-1-0-1 0-1-0-2
0-1 0-1-0
0 0-0 0-1 0-2

wordup.svg

The example above suggests an easy way to create unique identifiers for all AST nodes: the digit patterns are simply the path to each node, and each path element is the child index.

Symbol Tables

During symbol table generation and the search for semantic warnings and errors, if ZOBOS encounters the emit symtable statement all the symbols for all scopes should be written to the third command line argument. The format for this output is as follows:

  1. The global or deepest scope will be considered scope zero (0), each nested scope within it is assigned the next sequential scope value.

  2. Each symbol should be written on one line, and contain the following comma delimited fields: scope value, type, and id.

    1. The scope value is a non-negative integer, 0 is used for the deepest or "global" scope.

    2. type is the sequence of grammar terminals associated with the variable's DECLTYPE.

    3. The id is the identifier of the symbol.

    1. Yes, it is possible to have scopes without any symbols declared within them.
    2. Within the same scope, when a variable identifier is attempted to be re-used, an REDECL error should be generated and the attempted variable redeclaration is ignored.

NEED EXAMPLE

zlang.lr

zlang.cfg is not an SLR(1) language, nor is it LALR(1). This is due to the (55) UNARY->PLUS VALUE production rule, which causes two shift-reduce conflict during LR table generation. Both conflicts can be resolved by preferring the shift action over the reduce. Because of this (and the size of the grammar), we'll use a static fixed-up LR table that has these conflicts resolved.

The zlang.lr LR parsing table has 107 lines with 58 comma separated fields per line (that makes 59 fields each line). There is no white space except for new line separators (xA0).

  1. The first field of the first line is a period (.), which serves as a placeholder. The remaining fields on the first line are the column heading grammar symbols. There are 57 grammar symbols in the language (|$T+\Sigma_{\$}|$).

  2. The remaining lines are for the 106 item set states. The first field is the item set index. Subsequent fields are the actions to be performed on the associated grammar symbol. The actions are written in the same notation used in the text and lectures slides with one exception:

    1. sh-X means "shift the symbol to the parsing stack and goto to state X.

    2. r-N means reduce by rule number N (not index!) and push the resulting subtree back to the input queue.

    3. R-N means by rule number N, which has the grammar's goal symbol on the LHS --- accept the token stream as valid!

      In the text and slides R-N is written out across the entire row of the LR table, as shown here in item set 10.

zlang-pure.cfg does not contain #comments and should be immediately readable by your learning group code for grammars. zlang-rules.lis is a text file with numbered grammar rules, zlang-lr.pdf is a typeset LR table formatted in the same manner as lectures slides. And for the very curious, here is the CFSM item set graph of the language.

You may either hard code the LR parsing table and grammar rules into your ZOBOS application or read zlang.lr and the grammar rules from disk on each invocation. If the latter, include zlang.lr and any needed files in your submitted archive along side Build.sh or the Makefile. See to recall these submission details.

Hints and Testing

  1. You know you're going to dump symbol table debug info during development, so you might as format it according to the requirements in the first place.

grader.sh

To be provided...

Submit Your Work

Partners

You may complete this assignment in a group of two or three other students from the course (you may also work solo if you choose). Exams may have questions specific to course programming assignments — so be sure your group is truly working as a team and have a solid understanding of your submission's design and algorithms.

If you decide to complete this assignment as a group, you must tell your instructor one week before the assignment due date. Only one of you should submit the assignment for grading. You will all be able to see the grading result from your own logins.

Rubric

This work is worth 95 points.

Requirements

Points

Notes

Meets compilers course project requirements

10

Basic I/O and exit status requirements

10

exit status non-zero on unreadable input files or un-writable output files

Detect and report SYNTAX errors correctly

20

exit status should be non-zero when SYNTAX errors are detected

AST simplification of "raw" parse tree, output of AST graph information

20

emit symtable and symtable.dat requirements

15

ERROR class semantic issues properly detected and reported

10

exit status should be non-zero when ERROR issues are detected

WARN class semantic issues properly detected and reported

10

exit status should be zero when ERROR issues are detected