Token Struct Reference

Tokens are created by the Lexical Analyzer and provides an intermediate state for input consumed by the parser. More...

#include <token.h>

Collaboration diagram for Token:

Public Member Functions
	Token (Token *last, Symbol symbol, int pos)
	Token (Token *last, Symbol symbol, const char *text, int pos)
	~Token ()
int	GetPos () const
char *	GetText () const
Token *	GetLastToken () const
Token *	GetNextToken ()

Public Attributes
Symbol	symbol

Private Attributes
int	pos
char *	text
Token *	last
Token *	next

Friends
class	Lexer

Detailed Description

Tokens are created by the Lexical Analyzer and provides an intermediate state for input consumed by the parser.

Definition at line 46 of file token.h.

Constructor & Destructor Documentation

Token() [1/2]

Token::Token	(	Token *	last,
		Symbol	symbol,
		int	pos
	)

Definition at line 37 of file token.cpp.

References last, next, pos, symbol, and text.

Referenced by Lexer::GetLiteral(), GetNextToken(), Lexer::GetNextToken(), and Lexer::GetOperator().

{
    this->last = last;
    this->symbol = symbol;
    this->pos = pos;
    this->text = nullptr;
    this->next = nullptr;
}

Here is the caller graph for this function:

Token() [2/2]

Token::Token	(	Token *	last,
		Symbol	symbol,
		const char *	text,
		int	pos
	)

Definition at line 46 of file token.cpp.

References AllocAndCopy(), last, next, pos, symbol, and text.

Referenced by Lexer::GetDigitValue(), Lexer::GetLiteral(), and Lexer::GetQuotedIdent().

{
    this->last = last;
    this->symbol = symbol;
    this->pos = pos;
    this->next = nullptr;
    AllocAndCopy(&this->text, text);
}

Here is the call graph for this function:

Here is the caller graph for this function:

~Token()

Token::~Token

(

)

Definition at line 55 of file token.cpp.

References next, and text.

{
    if (text != nullptr)
    {
        delete [] text;
    }

    if (next != nullptr)
    {
        delete next;
    }
}

Member Function Documentation

GetLastToken()

Token * Token::GetLastToken

(

)

const

Definition at line 68 of file token.cpp.

References last.

Referenced by Parser::PutToken().

{
    return last;
}

Here is the caller graph for this function:

GetNextToken()

Token * Token::GetNextToken

(

)

Definition at line 73 of file token.cpp.

References next, pos, symend, and Token().

Referenced by Parser::GetToken(), and Parser::Peek().

{
    // Always return a token if requested
    if (next == nullptr)
    {
        next = new Token(this, symend, pos);
    }

    return next;
}

Here is the call graph for this function:

Here is the caller graph for this function:

GetPos()

int Token::GetPos

(

)

const

Definition at line 89 of file token.cpp.

References pos.

Referenced by Parser::Expect(), Parser::Parse(), Parser::ParseAtomic(), Parser::ParseDeleteStatement(), Parser::ParseDigistStatement(), Parser::ParseFileStatement(), Parser::ParseFunctionDef(), Parser::ParseIdent(), Parser::ParseListStatement(), and Parser::ParseNumeralStatement().

{
    return pos;
}

Here is the caller graph for this function:

GetText()

char * Token::GetText

(

)

const

Definition at line 84 of file token.cpp.

References text.

Referenced by Parser::ParseDeleteStatement(), Parser::ParseDigistStatement(), Parser::ParseDrawStatement(), Parser::ParseFileStatement(), Parser::ParseFunctionDef(), Parser::ParseHelpStatement(), Parser::ParseIdent(), Parser::ParseListStatement(), Parser::ParseNumber(), Parser::ParseNumeralStatement(), and Parser::ParsePromptStatement().

{
    return text;
}

Here is the caller graph for this function:

Friends And Related Function Documentation

Lexer

friend class Lexer

friend

Definition at line 64 of file token.h.

Member Data Documentation

last

Token* Token::last

private

Definition at line 62 of file token.h.

Referenced by GetLastToken(), and Token().

next

Token* Token::next

private

Definition at line 63 of file token.h.

Referenced by GetNextToken(), Token(), Lexer::Tokenize(), and ~Token().

pos

int Token::pos

private

Definition at line 60 of file token.h.

Referenced by GetNextToken(), GetPos(), and Token().

symbol

Symbol Token::symbol

Definition at line 53 of file token.h.

Referenced by Parser::Expect(), Parser::Parse(), Parser::ParseAddSubstract(), Parser::ParseAtomic(), Parser::ParseDeleteStatement(), Parser::ParseDigistStatement(), Parser::ParseDrawStatement(), Parser::ParseFactor(), Parser::ParseFileStatement(), Parser::ParseHelpStatement(), Parser::ParseIdent(), Parser::ParseListStatement(), Parser::ParseNumber(), Parser::ParseNumeralStatement(), Parser::ParsePower(), Parser::ParsePrefsStatement(), Parser::ParsePromptStatement(), Parser::ParseStatement(), Parser::ParseUnary(), Token(), Lexer::Tokenize(), and Parser::TryParseStatement().

text

char* Token::text

private

Definition at line 61 of file token.h.

Referenced by GetText(), Token(), and ~Token().

---

The documentation for this struct was generated from the following files:

• src/main/token.h
• src/main/token.cpp