Meta Tokens¶
There are two meta-tokens: StartOfDocumentToken
and EndOfDocumentToken. These tokens represent
a start and end of a token stream. These tokens have a length of 0 characters.
StartOfDocumentToken¶
This token starts a chain of double-linked tokens in a token stream. It’s used,
if the input source is a whole file, otherwise StartOfSnippetToken.
It is derived from base-class StartOfToken
and mixin StartOfDocument.
Interitance diagram:
EndOfDocumentToken¶
This token ends a chain of double-linked tokens in a token stream. It’s used,
if the input source is a whole file, otherwise EndOfSnippetToken.
It is derived from base-class EndOfToken
and mixin EndOfDocument.
Interitance diagram: