mwparserfromhell (the MediaWiki Parser from Hell) is a Python package that provides an easy-to-use and outrageously powerful parser for MediaWiki wikicode. It supports Python 2 and Python 3.
Developed by Earwig with contributions from Σ, Legoktm, and others. Development occurs on GitHub.
The easiest way to install the parser is through the Python Package Index, so you can install the latest release with pip install mwparserfromhell (get pip). Alternatively, get the latest development version:
git clone https://github.com/earwig/mwparserfromhell.git
cd mwparserfromhell
python setup.py install
If you get error: Unable to find vcvarsall.bat while installing, this is because Windows can’t find the compiler for C extensions. Consult this StackOverflow question for help. You can also set ext_modules in setup.py to an empty list to prevent the extension from building.
You can run the comprehensive unit testing suite with python setup.py test.
Normal usage is rather straightforward (where text is page text):
>>> import mwparserfromhell
>>> wikicode = mwparserfromhell.parse(text)
wikicode is a mwparserfromhell.Wikicode object, which acts like an ordinary unicode object (or str in Python 3) with some extra methods. For example:
>>> text = "I has a template! {{foo|bar|baz|eggs=spam}} See it?"
>>> wikicode = mwparserfromhell.parse(text)
>>> print wikicode
I has a template! {{foo|bar|baz|eggs=spam}} See it?
>>> templates = wikicode.filter_templates()
>>> print templates
['{{foo|bar|baz|eggs=spam}}']
>>> template = templates[0]
>>> print template.name
foo
>>> print template.params
['bar', 'baz', 'eggs=spam']
>>> print template.get(1).value
bar
>>> print template.get("eggs").value
spam
Since nodes can contain other nodes, getting nested templates is trivial:
>>> text = "{{foo|{{bar}}={{baz|{{spam}}}}}}"
>>> mwparserfromhell.parse(text).filter_templates()
['{{foo|{{bar}}={{baz|{{spam}}}}}}', '{{bar}}', '{{baz|{{spam}}}}', '{{spam}}']
You can also pass recursive=False to filter_templates() and explore templates manually. This is possible because nodes can contain additional Wikicode objects:
>>> code = mwparserfromhell.parse("{{foo|this {{includes a|template}}}}")
>>> print code.filter_templates(recursive=False)
['{{foo|this {{includes a|template}}}}']
>>> foo = code.filter_templates(recursive=False)[0]
>>> print foo.get(1).value
this {{includes a|template}}
>>> print foo.get(1).value.filter_templates()[0]
{{includes a|template}}
>>> print foo.get(1).value.filter_templates()[0].get(1).value
template
Templates can be easily modified to add, remove, or alter params. Wikicode objects can be treated like lists, with append(), insert(), remove(), replace(), and more. They also have a matches() method for comparing page or template names, which takes care of capitalization and whitespace:
>>> text = "{{cleanup}} '''Foo''' is a [[bar]]. {{uncategorized}}"
>>> code = mwparserfromhell.parse(text)
>>> for template in code.filter_templates():
... if template.name.matches("Cleanup") and not template.has("date"):
... template.add("date", "July 2012")
...
>>> print code
{{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{uncategorized}}
>>> code.replace("{{uncategorized}}", "{{bar-stub}}")
>>> print code
{{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}}
>>> print code.filter_templates()
['{{cleanup|date=July 2012}}', '{{bar-stub}}']
You can then convert code back into a regular unicode object (for saving the page!) by calling unicode() on it:
>>> text = unicode(code)
>>> print text
{{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}}
>>> text == code
True
(Likewise, use str(code) in Python 3.)
For more tips, check out Wikicode's full method list and the list of Nodes.
mwparserfromhell is used by and originally developed for EarwigBot; Page objects have a parse() method that essentially calls mwparserfromhell.parse() on get().
If you’re using Pywikipedia, your code might look like this:
import mwparserfromhell
import wikipedia as pywikibot
def parse(title):
site = pywikibot.getSite()
page = pywikibot.Page(site, title)
text = page.get()
return mwparserfromhell.parse(text)
If you’re not using a library, you can parse templates in any page using the following code (via the API):
import json
import urllib
import mwparserfromhell
API_URL = "http://en.wikipedia.org/w/api.php"
def parse(title):
raw = urllib.urlopen(API_URL, data).read()
res = json.loads(raw)
text = res["query"]["pages"].values()[0]["revisions"][0]["*"]
return mwparserfromhell.parse(text)
Released April 22, 2014 (changes):
Released September 1, 2013 (changes):
Released August 29, 2013 (changes):
Released August 24, 2013 (changes):
Released June 20, 2013 (changes):
Released September 21, 2012 (changes):
mwparserfromhell (the MediaWiki Parser from Hell) is a Python package that provides an easy-to-use and outrageously powerful parser for MediaWiki wikicode.
Implements support for both Python 2 and Python 3 by defining common types in terms of their Python 2/3 variants. For example, str is set to unicode on Python 2 but str on Python 3; likewise, bytes is str on 2 but bytes on 3. These types are meant to be imported directly from within the parser’s modules.
This module contains the SmartList type, as well as its _ListProxy child, which together implement a list whose sublists reflect changes made to the main list, and vice-versa.
Bases: mwparserfromhell.smart_list._SliceNormalizerMixIn, list
Implements the list interface with special handling of sublists.
When a sublist is created (by list[i:j]), any changes made to this list (such as the addition, removal, or replacement of elements) will be reflected in the sublist, or vice-versa, to the greatest degree possible. This is implemented by having sublists - instances of the _ListProxy type - dynamically determine their elements by storing their slice info and retrieving that slice from the parent. Methods that change the size of the list also change the slice info. For example:
>>> parent = SmartList([0, 1, 2, 3])
>>> parent
[0, 1, 2, 3]
>>> child = parent[2:]
>>> child
[2, 3]
>>> child.append(4)
>>> child
[2, 3, 4]
>>> parent
[0, 1, 2, 3, 4]
The parent needs to keep a list of its children in order to update them, which prevents them from being garbage-collected. If you are keeping the parent around for a while but creating many children, it is advisable to call destroy() when you’re finished with them.
Raises IndexError if list is empty or index is out of range.
Bases: mwparserfromhell.smart_list._SliceNormalizerMixIn, list
Implement the list interface by getting elements from a parent.
This is created by a SmartList object when slicing. It does not actually store the list at any time; instead, whenever the list is needed, it builds it dynamically using the _render() method.
Raises ValueError if the value is not present.
Raises IndexError if list is empty or index is out of range.
This module contains the StringMixIn type, which implements the interface for the unicode type (str on py3k) in a dynamic manner.
Implement the interface for unicode/str in a dynamic manner.
To use this class, inherit from it and override the __unicode__() method (same on py3k) to return the string representation of the object. The various string methods will operate on the value of __unicode__() instead of the immutable self like the regular str type.
Contains data about certain markup, like HTML tags and external links.
Return the HTML tag associated with the given wiki-markup.
Return if the given tag‘s contents should be passed to the parser.
Return whether or not the given tag contains visible text.
Return whether or not the given tag can exist without a close tag.
This module contains accessory functions for other parts of the library. Parser users generally won’t need stuff from here.
Return a Wikicode for value, allowing multiple types.
This differs from Parser.parse() in that we accept more than just a string to be parsed. Unicode objects (strings in py3k), strings (bytes in py3k), integers (converted to strings), None, existing Node or Wikicode objects, as well as an iterable of these types, are supported. This is used to parse input on-the-fly by various methods of Wikicode and others like Template, such as wikicode.insert() or setting template.name.
If given, context will be passed as a starting context to the parser. This is helpful when this function is used inside node attribute setters. For example, ExternalLink‘s url setter sets context to contexts.EXT_LINK_URI to prevent the URL itself from becoming an ExternalLink.
Bases: mwparserfromhell.string_mixin.StringMixIn
A Wikicode is a container for nodes that operates like a string.
Additionally, it contains methods that can be used to extract data from or modify the nodes, implemented in an interface similar to a list. For example, index() can get the index of a node in the list, and insert() can add a new node at that index. The filter() series of functions is very useful for extracting and iterating over, for example, all of the templates in the object.
Insert value at the end of the list of nodes.
value can be anything parasable by parse_anything().
Return a list of nodes within our list matching certain conditions.
Iterate over arguments.
This is equivalent to filter() with forcetype set to Argument.
Iterate over comments.
This is equivalent to filter() with forcetype set to Comment.
Iterate over external_links.
This is equivalent to filter() with forcetype set to ExternalLink.
Iterate over headings.
This is equivalent to filter() with forcetype set to Heading.
Iterate over html_entities.
This is equivalent to filter() with forcetype set to HTMLEntity.
Iterate over templates.
This is equivalent to filter() with forcetype set to Template.
Iterate over wikilinks.
This is equivalent to filter() with forcetype set to Wikilink.
Return a list of sections within the page.
Sections are returned as Wikicode objects with a shared node list (implemented using SmartList) so that changes to sections are reflected in the parent Wikicode object.
Each section contains all of its subsections, unless flat is True. If levels is given, it should be a iterable of integers; only sections whose heading levels are within it will be returned. If matches is given, it should be either a function or a regex; only sections whose headings match it (without the surrounding equal signs) will be included. flags can be used to override the default regex flags (see ifilter()) if a regex matches is used.
If include_lead is True, the first, lead section (without a heading) will be included in the list; False will not include it; the default will include it only if no specific levels were given. If include_headings is True, the section’s beginning Heading object will be included; otherwise, this is skipped.
Return a hierarchical tree representation of the object.
The representation is a string makes the most sense printed. It is built by calling _get_tree() on the Wikicode object and its children recursively. The end result may look something like the following:
>>> text = "Lorem ipsum {{foo|bar|{{baz}}|spam=eggs}}"
>>> print mwparserfromhell.parse(text).get_tree()
Lorem ipsum
{{
foo
| 1
= bar
| 2
= {{
baz
}}
| spam
= eggs
}}
Iterate over nodes in our list matching certain conditions.
If recursive is True, we will iterate over our children and all of their descendants, otherwise just our immediate children. If forcetype is given, only nodes that are instances of this type are yielded. matches can be used to further restrict the nodes, either as a function (taking a single Node and returning a boolean) or a regular expression (matched against the node’s string representation with re.search()). If matches is a regex, the flags passed to re.search() are re.IGNORECASE, re.DOTALL, and re.UNICODE, but custom flags can be specified by passing flags.
Iterate over arguments.
This is equivalent to ifilter() with forcetype set to Argument.
Iterate over comments.
This is equivalent to ifilter() with forcetype set to Comment.
Iterate over external_links.
This is equivalent to ifilter() with forcetype set to ExternalLink.
Iterate over headings.
This is equivalent to ifilter() with forcetype set to Heading.
Iterate over html_entities.
This is equivalent to ifilter() with forcetype set to HTMLEntity.
Iterate over templates.
This is equivalent to ifilter() with forcetype set to Template.
Iterate over wikilinks.
This is equivalent to ifilter() with forcetype set to Wikilink.
Return the index of obj in the list of nodes.
Raises ValueError if obj is not found. If recursive is True, we will look in all nodes of ours and their descendants, and return the index of our direct descendant node within our list of nodes. Otherwise, the lookup is done only on direct descendants.
Insert value at index in the list of nodes.
value can be anything parasable by parse_anything(), which includes strings or other Wikicode or Node objects.
Insert value immediately after obj.
obj can be either a string, a Node, or another Wikicode object (as created by get_sections(), for example). If obj is a string, we will operate on all instances of that string within the code, otherwise only on the specific instance given. value can be anything parasable by parse_anything(). If recursive is True, we will try to find obj within our child nodes even if it is not a direct descendant of this Wikicode object. If obj is not found, ValueError is raised.
Insert value immediately before obj.
obj can be either a string, a Node, or another Wikicode object (as created by get_sections(), for example). If obj is a string, we will operate on all instances of that string within the code, otherwise only on the specific instance given. value can be anything parasable by parse_anything(). If recursive is True, we will try to find obj within our child nodes even if it is not a direct descendant of this Wikicode object. If obj is not found, ValueError is raised.
Do a loose equivalency test suitable for comparing page names.
other can be any string-like object, including Wikicode, or a tuple of these. This operation is symmetric; both sides are adjusted. Specifically, whitespace and markup is stripped and the first letter’s case is normalized. Typical usage is if template.name.matches("stub"): ....
A list of Node objects.
This is the internal data actually stored within a Wikicode object.
Remove obj from the list of nodes.
obj can be either a string, a Node, or another Wikicode object (as created by get_sections(), for example). If obj is a string, we will operate on all instances of that string within the code, otherwise only on the specific instance given. If recursive is True, we will try to find obj within our child nodes even if it is not a direct descendant of this Wikicode object. If obj is not found, ValueError is raised.
Replace obj with value.
obj can be either a string, a Node, or another Wikicode object (as created by get_sections(), for example). If obj is a string, we will operate on all instances of that string within the code, otherwise only on the specific instance given. value can be anything parasable by parse_anything(). If recursive is True, we will try to find obj within our child nodes even if it is not a direct descendant of this Wikicode object. If obj is not found, ValueError is raised.
Set the Node at index to value.
Raises IndexError if index is out of range, or ValueError if value cannot be coerced into one Node. To insert multiple nodes at an index, use get() with either remove() and insert() or replace().
Return a rendered string without unprintable code such as templates.
The way a node is stripped is handled by the __strip__() method of Node objects, which generally return a subset of their nodes or None. For example, templates and tags are removed completely, links are stripped to just their display part, headings are stripped to just their title. If normalize is True, various things may be done to strip code further, such as converting HTML entities like Σ, Σ, and Σ to Σ. If collapse is True, we will try to remove excess whitespace as well (three or more newlines are converted to two, for example).
This package contains Wikicode “nodes”, which represent a single unit of wikitext, such as a Template, an HTML tag, a Heading, or plain text. The node “tree” is far from flat, as most types can contain additional Wikicode types within them - and with that, more nodes. For example, the name of a Template is a Wikicode object that can contain text or more templates.
Represents the base Node type, demonstrating the methods to override.
__unicode__() must be overridden. It should return a unicode or (str in py3k) representation of the node. If the node contains Wikicode objects inside of it, __children__() should be a generator that iterates over them. If the node is printable (shown when the page is rendered), __strip__() should return its printable version, stripping out any formatting marks. It does not have to return a string, but something that can be converted to a string with str(). Finally, __showtree__() can be overridden to build a nice tree representation of the node, if desired, for get_tree().
Bases: mwparserfromhell.nodes.Node
Represents a template argument substitution, like {{{foo}}}.
The default value to substitute if none is passed.
This will be None if the argument wasn’t defined with one. The MediaWiki parser handles this by rendering the argument itself in the result, complete braces. To have the argument render as nothing, set default to "" ({{{arg}}} vs. {{{arg|}}}).
Bases: mwparserfromhell.nodes.Node
Represents a hidden HTML comment, like <!-- foobar -->.
Bases: mwparserfromhell.nodes.Node
Represents an external link, like [http://example.com/ Example].
Bases: mwparserfromhell.nodes.Node
Represents an HTML entity, like , either named or unnamed.
If the value is hexadecimal, this is the letter denoting that.
For example, the hex_char of "ሴ" is "x", whereas the hex_char of "ሴ" is "X". Lowercase and uppercase x are the only values supported.
Bases: mwparserfromhell.nodes.Node
Represents an HTML-style tag in wikicode, like <ref>.
Add an attribute with the given name and value.
name and value can be anything parasable by utils.parse_anything(); value can be omitted if the attribute is valueless. quoted is a bool telling whether to wrap the value in double quotes (this is recommended). pad_first, pad_before_eq, and pad_after_eq are whitespace used as padding before the name, before the equal sign (or after the name if no value), and after the equal sign (ignored if no value), respectively.
The list of attributes affecting the tag.
Each attribute is an instance of Attribute.
The closing tag, as a Wikicode object.
This will usually equal tag, unless there is additional spacing, comments, or the like.
Get the attribute with the given name.
The returned object is a Attribute instance. Raises ValueError if no attribute has this name. Since multiple attributes can have the same name, we’ll return the last match, since all but the last are ignored by the MediaWiki parser.
Return whether any attribute in the tag has the given name.
Note that a tag may have multiple attributes with the same name, but only the last one is read by the MediaWiki parser.
Whether the tag is implicitly self-closing, with no ending slash.
This is only possible for specific “single” tags like <br> and <li>. See definitions.is_single(). This field only has an effect if self_closing is also True.
Whether the tag starts with a backslash after the opening bracket.
This makes the tag look like a lone close tag. It is technically invalid and is only parsable Wikicode when the tag itself is single-only, like <br> and <img>. See definitions.is_single_only().
Bases: mwparserfromhell.nodes.Node
Represents a template in wikicode, like {{foo}}.
Add a parameter to the template with a given name and value.
name and value can be anything parasable by utils.parse_anything(); pipes and equal signs are automatically escaped from value when appropriate.
If showkey is given, this will determine whether or not to show the parameter’s name (e.g., {{foo|bar}}‘s parameter has a name of "1" but it is hidden); otherwise, we’ll make a safe and intelligent guess.
If name is already a parameter in the template, we’ll replace its value while keeping the same whitespace around it. We will also try to guess the dominant spacing convention when adding a new parameter using _get_spacing_conventions().
If before is given (either a Parameter object or a name), then we will place the parameter immediately before this one. Otherwise, it will be added at the end. If before is a name and exists multiple times in the template, we will place it before the last occurance. If before is not in the template, ValueError is raised. The argument is ignored if the new parameter already exists.
If preserve_spacing is False, we will avoid preserving spacing conventions when changing the value of an existing parameter or when adding a new one.
Get the parameter whose name is name.
The returned object is a Parameter instance. Raises ValueError if no parameter has this name. Since multiple parameters can have the same name, we’ll return the last match, since the last parameter is the only one read by the MediaWiki parser.
Return True if any parameter in the template is named name.
With ignore_empty, False will be returned even if the template contains a parameter with the name name, if the parameter’s value is empty. Note that a template may have multiple parameters with the same name, but only the last one is read by the MediaWiki parser.
Remove a parameter from the template, identified by param.
If param is a Parameter object, it will be matched exactly, otherwise it will be treated like the name argument to has() and get().
If keep_field is True, we will keep the parameter’s name, but blank its value. Otherwise, we will remove the parameter completely unless other parameters are dependent on it (e.g. removing bar from {{foo|bar|baz}} is unsafe because {{foo|baz}} is not what we expected, so {{foo||baz}} will be produced instead).
If the parameter shows up multiple times in the template and param is not a Parameter object, we will remove all instances of it (and keep only one if keep_field is True - the first instance if none have dependents, otherwise the one with dependents will be kept).
Bases: mwparserfromhell.nodes.Node
Represents ordinary, unformatted text with no special properties.
This package contains objects used by Nodes, but are not nodes themselves. This includes the parameters of Templates or the attributes of HTML tags.
Bases: mwparserfromhell.string_mixin.StringMixIn
Represents an attribute of an HTML tag.
This is used by Tag objects. For example, the tag <ref name="foo"> contains an Attribute whose name is "name" and whose value is "foo".
Bases: mwparserfromhell.string_mixin.StringMixIn
Represents a paramater of a template.
For example, the template {{foo|bar|spam=eggs}} contains two Parameters: one whose name is "1", value is "bar", and showkey is False, and one whose name is "spam", value is "eggs", and showkey is True.
This package contains the actual wikicode parser, split up into two main modules: the tokenizer and the builder. This module joins them together under one interface.
Combines a sequence of tokens into a tree of Wikicode objects.
To use, pass a list of Tokens to the build() method. The list will be exhausted as it is parsed and a Wikicode object will be returned.
Handle a case where a parameter is at the head of the tokens.
default is the value to use if no parameter name is defined.
This module contains various “context” definitions, which are essentially flags set during the tokenization process, either on the current parse stack (local contexts) or affecting all stacks (global contexts). They represent the context the tokenizer is in, such as inside a template’s name definition, or inside a level-two heading. This is used to determine what tokens are valid at the current point and also if the current parsing route is invalid.
The tokenizer stores context as an integer, with these definitions bitwise OR’d to set them, AND’d to check if they’re set, and XOR’d to unset them. The advantage of this is that contexts can have sub-contexts (as FOO == 0b11 will cover BAR == 0b10 and BAZ == 0b01).
Local (stack-specific) contexts:
TEMPLATE
- TEMPLATE_NAME
- TEMPLATE_PARAM_KEY
- TEMPLATE_PARAM_VALUE
ARGUMENT
- ARGUMENT_NAME
- ARGUMENT_DEFAULT
WIKILINK
- WIKILINK_TITLE
- WIKILINK_TEXT
EXT_LINK
- EXT_LINK_URI
- EXT_LINK_TITLE
HEADING
- HEADING_LEVEL_1
- HEADING_LEVEL_2
- HEADING_LEVEL_3
- HEADING_LEVEL_4
- HEADING_LEVEL_5
- HEADING_LEVEL_6
TAG
- TAG_OPEN
- TAG_ATTR
- TAG_BODY
- TAG_CLOSE
STYLE
- STYLE_ITALICS
- STYLE_BOLD
- STYLE_PASS_AGAIN
- STYLE_SECOND_PASS
DL_TERM
SAFETY_CHECK
- HAS_TEXT
- FAIL_ON_TEXT
- FAIL_NEXT
- FAIL_ON_LBRACE
- FAIL_ON_RBRACE
- FAIL_ON_EQUALS
Global contexts:
Aggregate contexts:
Creates a list of tokens from a string of wikicode.
Write the body of a tag and the tokens that should surround it.
Fail the current tokenization route.
Discards the current stack/context/textbuffer and raises BadRoute.
Handle text in a free ext link, including trailing punctuation.
Handle the (possible) start of an implicitly closing single tag.
Handle the end of an implicitly closing single-only HTML tag.
Handle a template parameter’s value at the head of the string.
Parse a template or argument at the head of the wikicode string.
Pop the current stack/context/textbuffer, returing the stack.
If keep_context is True, then we will replace the underlying stack’s context with the current stack’s.
Read the value at a relative point in the wikicode.
The value is read from self._head plus the value of delta (which can be negative). If wrap is False, we will not allow attempts to read from the end of the string if self._head + delta is negative. If strict is True, the route will be failed (with _fail_route()) if we try to read from past the end of the string; otherwise, self.END is returned. If we try to read from before the start of the string, self.START is returned.
Remove the URI scheme of a new external link from the textbuffer.
This module contains the token definitions that are used as an intermediate parsing data type - they are stored in a flat list, with each token being identified by its type and optional attributes. The token list is generated in a syntactically valid form by the Tokenizer, and then converted into the :py:class`~.Wikicode` tree by the Builder.
A token stores the semantic meaning of a unit of wikicode.