parser Package

parser Package

This package contains the actual wikicode parser, split up into two main modules: the tokenizer and the builder. This module joins them together under one interface.

class mwparserfromhell.parser.Parser

Represents a parser for wikicode.

Actual parsing is a two-step process: first, the text is split up into a series of tokens by the Tokenizer, and then the tokens are converted into trees of Wikicode objects and Nodes by the Builder.

parse(text, context=0, skip_style_tags=False)

Parse text, returning a Wikicode object tree.

If skip_style_tags is True, then '' and ''' will not be parsed, but instead be treated as plain text.

builder Module

class mwparserfromhell.parser.builder.Builder

Combines a sequence of tokens into a tree of Wikicode objects.

To use, pass a list of Tokens to the build() method. The list will be exhausted as it is parsed and a Wikicode object will be returned.

_handle_argument()

Handle a case where an argument is at the head of the tokens.

_handle_attribute(start)

Handle a case where a tag attribute is at the head of the tokens.

_handle_comment()

Handle a case where an HTML comment is at the head of the tokens.

_handle_entity()

Handle a case where an HTML entity is at the head of the tokens.

_handle_external_link(token)

Handle when an external link is at the head of the tokens.

_handle_heading(token)

Handle a case where a heading is at the head of the tokens.

_handle_parameter(default)

Handle a case where a parameter is at the head of the tokens.

default is the value to use if no parameter name is defined.

_handle_tag(token)

Handle a case where a tag is at the head of the tokens.

_handle_template()

Handle a case where a template is at the head of the tokens.

_handle_token(token)

Handle a single token.

_handle_wikilink()

Handle a case where a wikilink is at the head of the tokens.

_pop(wrap=True)

Pop the current node list off of the stack.

If wrap is True, we will call _wrap() on the list.

_push()

Push a new node list onto the stack.

_wrap(nodes)

Properly wrap a list of nodes in a Wikicode object.

_write(item)

Append a node to the current node list.

build(tokenlist)

Build a Wikicode object from a list tokens and return it.

contexts Module

This module contains various “context” definitions, which are essentially flags set during the tokenization process, either on the current parse stack (local contexts) or affecting all stacks (global contexts). They represent the context the tokenizer is in, such as inside a template’s name definition, or inside a level-two heading. This is used to determine what tokens are valid at the current point and also if the current parsing route is invalid.

The tokenizer stores context as an integer, with these definitions bitwise OR’d to set them, AND’d to check if they’re set, and XOR’d to unset them. The advantage of this is that contexts can have sub-contexts (as FOO == 0b11 will cover BAR == 0b10 and BAZ == 0b01).

Local (stack-specific) contexts:

• TEMPLATE

  • TEMPLATE_NAME
  • TEMPLATE_PARAM_KEY
  • TEMPLATE_PARAM_VALUE

• ARGUMENT

  • ARGUMENT_NAME
  • ARGUMENT_DEFAULT

• WIKILINK

  • WIKILINK_TITLE
  • WIKILINK_TEXT

• EXT_LINK

  • EXT_LINK_URI
  • EXT_LINK_TITLE

• HEADING

  • HEADING_LEVEL_1
  • HEADING_LEVEL_2
  • HEADING_LEVEL_3
  • HEADING_LEVEL_4
  • HEADING_LEVEL_5
  • HEADING_LEVEL_6

• TAG

  • TAG_OPEN
  • TAG_ATTR
  • TAG_BODY
  • TAG_CLOSE

• STYLE

  • STYLE_ITALICS
  • STYLE_BOLD
  • STYLE_PASS_AGAIN
  • STYLE_SECOND_PASS

• DL_TERM

• SAFETY_CHECK

  • HAS_TEXT
  • FAIL_ON_TEXT
  • FAIL_NEXT
  • FAIL_ON_LBRACE
  • FAIL_ON_RBRACE
  • FAIL_ON_EQUALS

Global contexts:

• GL_HEADING

Aggregate contexts:

• FAIL
• UNSAFE
• DOUBLE
• NO_WIKILINKS
• NO_EXT_LINKS

tokenizer Module

class mwparserfromhell.parser.tokenizer.Tokenizer

Creates a list of tokens from a string of wikicode.

END = <object object at 0x7f28c7a0e770>

MARKERS = [u'{', u'}', u'[', u']', u'<', u'>', u'|', u'=', u'&', u"'", u'#', u'*', u';', u':', u'/', u'-', u'\n', <object object at 0x7f28c7a0e760>, <object object at 0x7f28c7a0e770>]

MAX_CYCLES = 100000

MAX_DEPTH = 40

START = <object object at 0x7f28c7a0e760>

USES_C = False

_can_recurse()

Return whether or not our max recursion depth has been exceeded.

_context

The current token context.

_emit(token)

Write a token to the end of the current token stack.

_emit_all(tokenlist)

Write a series of tokens to the current stack at once.

_emit_first(token)

Write a token to the beginning of the current token stack.

_emit_style_tag(tag, markup, body)

Write the body of a tag and the tokens that should surround it.

_emit_text(text)

Write text to the current textbuffer.

_emit_text_then_stack(text)

Pop the current stack, write text, and then write the stack.

_fail_route()

Fail the current tokenization route.

Discards the current stack/context/textbuffer and raises BadRoute.

_handle_argument_end()

Handle the end of an argument at the head of the string.

_handle_argument_separator()

Handle the separator between an argument’s name and default.

_handle_blacklisted_tag()

Handle the body of an HTML tag that is parser-blacklisted.

_handle_dl_term()

Handle the term in a description list (foo in ;foo:bar).

_handle_end()

Handle the end of the stream of wikitext.

_handle_free_link_text(punct, tail, this)

Handle text in a free ext link, including trailing punctuation.

_handle_heading_end()

Handle the end of a section heading at the head of the string.

_handle_hr()

Handle a wiki-style horizontal rule (----) in the string.

_handle_invalid_tag_start()

Handle the (possible) start of an implicitly closing single tag.

_handle_list()

Handle a wiki-style list (#, *, ;, :).

_handle_list_marker()

Handle a list marker at the head (#, *, ;, :).

_handle_single_only_tag_end()

Handle the end of an implicitly closing single-only HTML tag.

_handle_single_tag_end()

Handle the stream end when inside a single-supporting HTML tag.

_handle_tag_close_close()

Handle the ending of a closing tag (</foo>).

_handle_tag_close_open(data, token)

Handle the closing of a open tag (<foo>).

_handle_tag_data(data, text)

Handle all sorts of text data inside of an HTML open tag.

_handle_tag_open_close()

Handle the opening of a closing tag (</foo>).

_handle_tag_space(data, text)

Handle whitespace (text) inside of an HTML open tag.

_handle_tag_text(text)

Handle regular text inside of an HTML open tag.

_handle_template_end()

Handle the end of a template at the head of the string.

_handle_template_param()

Handle a template parameter at the head of the string.

_handle_template_param_value()

Handle a template parameter’s value at the head of the string.

_handle_wikilink_end()

Handle the end of a wikilink at the head of the string.

_handle_wikilink_separator()

Handle the separator between a wikilink’s title and its text.

_is_free_link_end(this, next)

Return whether the current head is the end of a free link.

_parse(context=0, push=True)

Parse the wikicode string, using context for when to stop.

_parse_argument()

Parse an argument at the head of the wikicode string.

_parse_bold()

Parse wiki-style bold.

_parse_bracketed_uri_scheme()

Parse the URI scheme of a bracket-enclosed external link.

_parse_comment()

Parse an HTML comment at the head of the wikicode string.

_parse_entity()

Parse an HTML entity at the head of the wikicode string.

_parse_external_link(brackets)

Parse an external link at the head of the wikicode string.

_parse_free_uri_scheme()

Parse the URI scheme of a free (no brackets) external link.

_parse_heading()

Parse a section heading at the head of the wikicode string.

_parse_italics()

Parse wiki-style italics.

_parse_italics_and_bold()

Parse wiki-style italics and bold together (i.e., five ticks).

_parse_style()

Parse wiki-style formatting (''/''' for italics/bold).

_parse_tag()

Parse an HTML tag at the head of the wikicode string.

_parse_template()

Parse a template at the head of the wikicode string.

_parse_template_or_argument()

Parse a template or argument at the head of the wikicode string.

_parse_wikilink()

Parse an internal wikilink at the head of the wikicode string.

_pop(keep_context=False)

Pop the current stack/context/textbuffer, returing the stack.

If keep_context is True, then we will replace the underlying stack’s context with the current stack’s.

_push(context=0)

Add a new token stack, context, and textbuffer to the list.

_push_tag_buffer(data)

Write a pending tag attribute from data to the stack.

_push_textbuffer()

Push the textbuffer onto the stack as a Text node and clear it.

_read(delta=0, wrap=False, strict=False)

Read the value at a relative point in the wikicode.

The value is read from self._head plus the value of delta (which can be negative). If wrap is False, we will not allow attempts to read from the end of the string if self._head + delta is negative. If strict is True, the route will be failed (with _fail_route()) if we try to read from past the end of the string; otherwise, self.END is returned. If we try to read from before the start of the string, self.START is returned.

_really_parse_entity()

Actually parse an HTML entity and ensure that it is valid.

_really_parse_external_link(brackets)

Really parse an external link.

_really_parse_tag()

Actually parse an HTML tag, starting with the open (<foo>).

_remove_uri_scheme_from_textbuffer(scheme)

Remove the URI scheme of a new external link from the textbuffer.

_stack

The current token stack.

_textbuffer

The current textbuffer.

_verify_safe(this)

Make sure we are not trying to write an invalid character.

regex = <_sre.SRE_Pattern object at 0x148fdf0>

tag_splitter = <_sre.SRE_Pattern object at 0x163a0e8>

tokenize(text, context=0, skip_style_tags=False)

Build a list of tokens from a string of wikicode and return it.

exception mwparserfromhell.parser.tokenizer.BadRoute(context=0)

Raised internally when the current tokenization route is invalid.

tokens Module

This module contains the token definitions that are used as an intermediate parsing data type - they are stored in a flat list, with each token being identified by its type and optional attributes. The token list is generated in a syntactically valid form by the Tokenizer, and then converted into the :py:class`~.Wikicode` tree by the Builder.

class mwparserfromhell.parser.tokens.Token

A token stores the semantic meaning of a unit of wikicode.

class mwparserfromhell.parser.tokens.Text

class mwparserfromhell.parser.tokens.TemplateOpen

class mwparserfromhell.parser.tokens.TemplateParamSeparator

class mwparserfromhell.parser.tokens.TemplateParamEquals

class mwparserfromhell.parser.tokens.TemplateClose

class mwparserfromhell.parser.tokens.ArgumentOpen

class mwparserfromhell.parser.tokens.ArgumentSeparator

class mwparserfromhell.parser.tokens.ArgumentClose

class mwparserfromhell.parser.tokens.WikilinkOpen

class mwparserfromhell.parser.tokens.WikilinkSeparator

class mwparserfromhell.parser.tokens.WikilinkClose

class mwparserfromhell.parser.tokens.ExternalLinkOpen

class mwparserfromhell.parser.tokens.ExternalLinkSeparator

class mwparserfromhell.parser.tokens.ExternalLinkClose

class mwparserfromhell.parser.tokens.HTMLEntityStart

class mwparserfromhell.parser.tokens.HTMLEntityNumeric

class mwparserfromhell.parser.tokens.HTMLEntityHex

class mwparserfromhell.parser.tokens.HTMLEntityEnd

class mwparserfromhell.parser.tokens.HeadingStart

class mwparserfromhell.parser.tokens.HeadingEnd

class mwparserfromhell.parser.tokens.CommentStart

class mwparserfromhell.parser.tokens.CommentEnd

class mwparserfromhell.parser.tokens.TagOpenOpen

class mwparserfromhell.parser.tokens.TagAttrStart

class mwparserfromhell.parser.tokens.TagAttrEquals

class mwparserfromhell.parser.tokens.TagAttrQuote

class mwparserfromhell.parser.tokens.TagCloseOpen

class mwparserfromhell.parser.tokens.TagCloseSelfclose

class mwparserfromhell.parser.tokens.TagOpenClose

class mwparserfromhell.parser.tokens.TagCloseClose