Lexer Class Reference

Encapsulates an lexical analyzer. Provides token for the parser. More...

#include <lexer.h>

Collaboration diagram for Lexer:

Public Member Functions
	Lexer (const char *input)
	~Lexer ()
void	Tokenize ()
Token *	GetFirstToken () const
char *	GetInput () const

Static Public Member Functions
static char *	FindKeyword (Symbol symbol)

Private Member Functions
void	GetNextToken ()
bool	GetOperator ()
bool	GetQuotedIdent ()
bool	GetLiteral ()
bool	GetDigitValue ()

Static Private Member Functions
static bool	ShouldSkip (char character)
static Symbol	FindKeyword (const char *ident)

Private Attributes
char *	input
char *	str
unsigned int	pos
Token *	first
Token *	current

Detailed Description

Encapsulates an lexical analyzer. Provides token for the parser.

More info on lexical analysis is available at Wikipedia: https://wikipedia.org/wiki/Lexical_analysis

Definition at line 48 of file lexer.h.

Constructor & Destructor Documentation

Lexer()

Lexer::Lexer ( const char *  input )

explicit

Definition at line 38 of file lexer.cpp.

References AllocAndCopy(), current, first, input, pos, and str.

Referenced by Parser::Parser().

{
    AllocAndCopy(&this->input, input);
    pos = 0;
    str = this->input;
    first = nullptr;
    current = nullptr;
}

Here is the call graph for this function:

Here is the caller graph for this function:

~Lexer()

Lexer::~Lexer

(

)

Definition at line 47 of file lexer.cpp.

References first, and input.

{
    delete [] input;

    if (first != nullptr)
    {
        delete first;
    }
}

Member Function Documentation

FindKeyword() [1/2]

char * Lexer::FindKeyword ( Symbol  symbol )

static

Definition at line 248 of file lexer.cpp.

References operatordef::chr, keywords, keyworddef::name, operators, operatordef::symbol, and keyworddef::symbol.

Referenced by PreferencesBase::GetDescription().

{
    static const unsigned int kwcount = sizeof(keywords) / sizeof(keyworddef);
    for (unsigned int i = 0; i < kwcount; i++)
    {
        if (keywords[i].symbol == symbol)
        {
            return const_cast<char*>(keywords[i].name);
        }
    }

    static const unsigned int ocount = sizeof(operators) / sizeof(operatordef);
    for (unsigned int i = 0; i < ocount; i++)
    {
        if (operators[i].symbol == symbol)
        {
            return const_cast<char*>(&(operators[i].chr));
        }
    }

    return nullptr;
}

Here is the caller graph for this function:

FindKeyword() [2/2]

Symbol Lexer::FindKeyword ( const char *  ident )

staticprivate

Definition at line 243 of file lexer.cpp.

References Language::FindKeyword().

Referenced by GetLiteral().

{
    return Program->Language->FindKeyword(ident);
}

Here is the call graph for this function:

Here is the caller graph for this function:

GetDigitValue()

bool Lexer::GetDigitValue ( )

private

Definition at line 216 of file lexer.cpp.

References current, Program::Input, MemCopy(), NumeralSystem::Parse(), pos, str, symnumber, and Token::Token().

Referenced by GetNextToken().

{
    unsigned int len;
    char* end;

    // Only the numeral parser can determine if next token is a value
    Number* number = Program->Input->Parse(str, &len, &end);

    if (str == end)
    {
        delete number;
        return false;
    }

    char* text = new char[len + 1];
    MemCopy(text, str, len);
    *(text + len) = '\0';

    current = new Token(current, symnumber, text, pos);
    delete [] text;
    delete number;

    pos += len;
    str = end;
    return true;
}

Here is the call graph for this function:

Here is the caller graph for this function:

GetFirstToken()

Token * Lexer::GetFirstToken

(

)

const

Definition at line 62 of file lexer.cpp.

References first.

Referenced by Parser::GetToken(), and Parser::Peek().

{
    return first;
}

Here is the caller graph for this function:

GetInput()

char * Lexer::GetInput

(

)

const

Definition at line 57 of file lexer.cpp.

References input.

Referenced by Parser::Parse(), Parser::ParseDigistStatement(), Parser::ParseFileStatement(), Parser::ParseFunctionDef(), Parser::ParseIdent(), Parser::ParseNumeralStatement(), and Parser::TryParseStatement().

{
    return input;
}

Here is the caller graph for this function:

GetLiteral()

bool Lexer::GetLiteral ( )

private

Definition at line 166 of file lexer.cpp.

References Language::CharIsAlpha(), Language::CharIsDigit(), current, FindKeyword(), MemCopy(), pos, str, symident, and Token::Token().

Referenced by GetNextToken().

{
    const char* start = str;
    const unsigned int startPos = pos;
    int unsigned len = 0;
    bool found = true;
    Symbol ksymbol;

    while (found)
    {
        if (Program->Language->CharIsAlpha(*str))
        {
            str++;
            len++;
        }
        else if (len != 0 && Program->Language->CharIsDigit(*str))
        {
            str++;
            len++;
        }
        else
        {
            found = false;
        }
    }

    if (len == 0)
    {
        return false;
    }

    char* ident = new char[len + 1];
    MemCopy(ident, start, len);
    ident[len] = 0;

    if ((ksymbol = FindKeyword(ident)))
    {
        current = new Token(current, ksymbol, startPos);
    }
    else
    {
        current = new Token(current, symident, ident, startPos);
    }

    pos += len;

    delete [] ident;
    return true;
}

Here is the call graph for this function:

Here is the caller graph for this function:

GetNextToken()

void Lexer::GetNextToken ( )

private

Definition at line 86 of file lexer.cpp.

References Language::CharIsSpace(), current, GetDigitValue(), GetLiteral(), GetOperator(), GetQuotedIdent(), pos, ShouldSkip(), str, symend, symunknown, and Token::Token().

Referenced by Tokenize().

{
    // Skip spaces and non visible characters
    while (*str != 0 && ShouldSkip(*str))
    {
        str++;
        if (Program->Language->CharIsSpace(*str))
        {
            pos++;
        }
    }

    if (*str == 0)
    {
        current = new Token(current, symend, pos);
        return;
    }

    if (GetOperator() || GetQuotedIdent() || GetDigitValue() || GetLiteral())
        return;

    str++;
    pos++;
    current = new Token(current, symunknown, pos - 1);
}

Here is the call graph for this function:

Here is the caller graph for this function:

GetOperator()

bool Lexer::GetOperator ( )

private

Definition at line 112 of file lexer.cpp.

References operatordef::chr, current, operators, pos, str, operatordef::symbol, and Token::Token().

Referenced by GetNextToken().

{
    static const unsigned int count = sizeof(operators) / sizeof(operatordef);
    for (unsigned int i = 0; i < count; i++)
    {
        if (operators[i].chr == *str)
        {
            current = new Token(current, operators[i].symbol, pos);
            str++;
            pos++;
            return true;
        }
    }

    return false;
}

Here is the call graph for this function:

Here is the caller graph for this function:

GetQuotedIdent()

bool Lexer::GetQuotedIdent ( )

private

Definition at line 129 of file lexer.cpp.

References Language::CharIsCntrl(), current, MemCopy(), pos, str, symqident, and Token::Token().

Referenced by GetNextToken().

{
    if (*str != '"')
    {
        return false;
    }

    char* start = str;
    const unsigned int startPos = pos;
    int unsigned len = 0;
    str++;

    while (*str != 0 && *str != '"' && !Program->Language->CharIsCntrl(*str))
    {
        str++;
        len++;
    }

    if (len == 0 || *str != '"')
    {
        str = start;
        return false;
    }

    char* ident = new char[len + 1];
    MemCopy(ident, start + 1, len);
    ident[len] = 0;

    current = new Token(current, symqident, ident, startPos);

    str++;
    pos += len + 1;

    delete [] ident;
    return true;
}

Here is the call graph for this function:

Here is the caller graph for this function:

ShouldSkip()

bool Lexer::ShouldSkip ( char  character )

staticprivate

Definition at line 271 of file lexer.cpp.

References Language::CharIsCntrl(), and Language::CharIsSpace().

Referenced by GetNextToken().

{
    if (character == '\n')
    {
        return false;
    }

    if (Program->Language->CharIsCntrl(character))
    {
        return true;
    }

    if (Program->Language->CharIsSpace(character))
    {
        return true;
    }

    return false;
}

Here is the call graph for this function:

Here is the caller graph for this function:

Tokenize()

void Lexer::Tokenize

(

)

Definition at line 67 of file lexer.cpp.

References current, first, GetNextToken(), input, Token::next, pos, str, Token::symbol, and symend.

Referenced by Parser::Parse().

{
    pos = 0;
    str = input;
    first = nullptr;
    current = nullptr;

    GetNextToken();
    first = current;

    do
    {
        Token* last = current;
        GetNextToken();
        last->next = current;
    }
    while (current->symbol != symend);
}

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

current

Token* Lexer::current

private

Definition at line 64 of file lexer.h.

Referenced by GetDigitValue(), GetLiteral(), GetNextToken(), GetOperator(), GetQuotedIdent(), Lexer(), and Tokenize().

first

Token* Lexer::first

private

Definition at line 63 of file lexer.h.

Referenced by GetFirstToken(), Lexer(), Tokenize(), and ~Lexer().

input

char* Lexer::input

private

Definition at line 60 of file lexer.h.

Referenced by GetInput(), Lexer(), Tokenize(), and ~Lexer().

pos

unsigned int Lexer::pos

private

Definition at line 62 of file lexer.h.

Referenced by GetDigitValue(), GetLiteral(), GetNextToken(), GetOperator(), GetQuotedIdent(), Lexer(), and Tokenize().

str

char* Lexer::str

private

Definition at line 61 of file lexer.h.

Referenced by GetDigitValue(), GetLiteral(), GetNextToken(), GetOperator(), GetQuotedIdent(), Lexer(), and Tokenize().

---

The documentation for this class was generated from the following files:

• src/main/lexer.h
• src/main/lexer.cpp