API Reference

Syntax Trees

Export the Python grammar and symbols.

class fissix.pygram.Symbols(grammar)

Bases: object

Python parse tree definitions.

This is a very concrete parse tree; we need to keep every token and even the comments and whitespace between tokens.

There’s also a pattern matching implementation here.

class fissix.pytree.Base(*args, **kwds)

Bases: object

Abstract base class for Node and Leaf.

This provides some default functionality and boilerplate using the template pattern.

A node may be a subnode of at most one parent.

changed()
clone()

Return a cloned (deep) copy of self.

This must be implemented by the concrete subclass.

depth()
get_lineno()

Return the line number which generated the invocant node.

get_suffix()

Return the string immediately following the invocant node. This is effectively equivalent to node.next_sibling.prefix

leaves()
post_order()

Return a post-order iterator for the tree.

This must be implemented by the concrete subclass.

pre_order()

Return a pre-order iterator for the tree.

This must be implemented by the concrete subclass.

remove()

Remove the node from the tree. Returns the position of the node in its parent’s children before it was removed.

replace(new)

Replace this node with a new one in the parent.

children = []
property next_sibling

The node immediately following the invocant in their parent’s children list. If the invocant does not have a next sibling, it is None

parent = None
property prev_sibling

The node immediately preceding the invocant in their parent’s children list. If the invocant does not have a previous sibling, it is None.

type = None
was_changed = False
was_checked = False
class fissix.pytree.BasePattern(*args, **kwds)

Bases: object

A pattern is a tree matching pattern.

It looks for a specific node type (token or symbol), and optionally for a specific content.

This is an abstract base class. There are three concrete subclasses:

  • LeafPattern matches a single leaf node;

  • NodePattern matches a single node (usually non-leaf);

  • WildcardPattern matches a sequence of nodes of variable length.

generate_matches(nodes)

Generator yielding all matches for this pattern.

Default implementation for non-wildcard patterns.

match(node, results=None)

Does this pattern exactly match a node?

Returns True if it matches, False if not.

If results is not None, it must be a dict which will be updated with the nodes matching named subpatterns.

Default implementation for non-wildcard patterns.

match_seq(nodes, results=None)

Does this pattern exactly match a sequence of nodes?

Default implementation for non-wildcard patterns.

optimize()

A subclass can define this as a hook for optimizations.

Returns either self or another node with the same effect.

content = None
name = None
type = None
class fissix.pytree.Leaf(*args, **kwds)

Bases: fissix.pytree.Base

Concrete implementation for leaf nodes.

clone()

Return a cloned (deep) copy of self.

leaves()
post_order()

Return a post-order iterator for the tree.

pre_order()

Return a pre-order iterator for the tree.

column = 0
lineno = 0
property prefix

The whitespace and comments preceding this token in the input.

class fissix.pytree.LeafPattern(*args, **kwds)

Bases: fissix.pytree.BasePattern

match(node, results=None)

Override match() to insist on a leaf node.

class fissix.pytree.NegatedPattern(*args, **kwds)

Bases: fissix.pytree.BasePattern

generate_matches(nodes)

Generator yielding all matches for this pattern.

Default implementation for non-wildcard patterns.

match(node)

Does this pattern exactly match a node?

Returns True if it matches, False if not.

If results is not None, it must be a dict which will be updated with the nodes matching named subpatterns.

Default implementation for non-wildcard patterns.

match_seq(nodes)

Does this pattern exactly match a sequence of nodes?

Default implementation for non-wildcard patterns.

class fissix.pytree.Node(*args, **kwds)

Bases: fissix.pytree.Base

Concrete implementation for interior nodes.

append_child(child)

Equivalent to ‘node.children.append(child)’. This method also sets the child’s parent attribute appropriately.

clone()

Return a cloned (deep) copy of self.

insert_child(i, child)

Equivalent to ‘node.children.insert(i, child)’. This method also sets the child’s parent attribute appropriately.

post_order()

Return a post-order iterator for the tree.

pre_order()

Return a pre-order iterator for the tree.

set_child(i, child)

Equivalent to ‘node.children[i] = child’. This method also sets the child’s parent attribute appropriately.

property prefix

The whitespace and comments preceding this node in the input.

class fissix.pytree.NodePattern(*args, **kwds)

Bases: fissix.pytree.BasePattern

wildcards = False
class fissix.pytree.WildcardPattern(*args, **kwds)

Bases: fissix.pytree.BasePattern

A wildcard pattern can match zero or more nodes.

This has all the flexibility needed to implement patterns like:

.* .+ .? .{m,n} (a b c | d e | f) (…)* (…)+ (…)? (…){m,n}

except it always uses non-greedy matching.

generate_matches(nodes)

Generator yielding matches for a sequence of nodes.

Args:

nodes: sequence of nodes

Yields:

(count, results) tuples where: count: the match comprises nodes[:count]; results: dict containing named submatches.

match(node, results=None)

Does this pattern exactly match a node?

match_seq(nodes, results=None)

Does this pattern exactly match a sequence of nodes?

optimize()

Optimize certain stacked wildcard patterns.

fissix.pytree.convert(gr, raw_node)

Convert raw node information to a Node or Leaf instance.

This is passed to the parser driver which calls it whenever a reduction of a grammar rule produces a new complete node, so that the tree is build strictly bottom-up.

fissix.pytree.generate_matches(patterns, nodes)

Generator yielding matches for a sequence of patterns and nodes.

Args:

patterns: a sequence of patterns nodes: a sequence of nodes

Yields:

(count, results) tuples where: count: the entire sequence of patterns matches nodes[:count]; results: dict containing named submatches.

fissix.pytree.type_repr(type_num)

Fixers

Base class for fixers (optional, but recommended).

class fissix.fixer_base.BaseFix(options, log)

Bases: object

Optional base class for fixers.

The subclass name must be FixFooBar where FooBar is the result of removing underscores and capitalizing the words of the fix name. For example, the class name for a fixer named ‘has_key’ should be FixHasKey.

cannot_convert(node, reason=None)

Warn the user that a given chunk of code is not valid Python 3, but that it cannot be converted automatically.

First argument is the top-level node for the code in question. Optional second argument is why it can’t be converted.

compile_pattern()

Compiles self.PATTERN into self.pattern.

Subclass may override if it doesn’t want to use self.{pattern,PATTERN} in .match().

finish_tree(tree, filename)

Some fixers need to maintain tree-wide state. This method is called once, at the conclusion of tree fix-up.

tree - the root node of the tree to be processed. filename - the name of the file the tree came from.

log_message(message)
match(node)

Returns match for a given parse tree node.

Should return a true or false object (not necessarily a bool). It may return a non-empty dict of matching sub-nodes as returned by a matching pattern.

Subclass may override.

new_name(template='xxx_todo_changeme')

Return a string suitable for use as an identifier

The new name is guaranteed not to conflict with other identifiers.

set_filename(filename)

Set the filename.

The main refactoring tool should call this.

start_tree(tree, filename)

Some fixers need to maintain tree-wide state. This method is called once, at the start of tree fix-up.

tree - the root node of the tree to be processed. filename - the name of the file the tree came from.

transform(node, results)

Returns the transformation for a given parse tree node.

Args:

node: the root of the parse tree that matched the fixer. results: a dict mapping symbolic names to part of the match.

Returns:

None, or a node that is a modified copy of the argument node. The node argument may also be modified in-place to effect the same change.

Subclass must override.

warning(node, reason)

Used for warning the user about possible uncertainty in the translation.

First argument is the top-level node for the code in question. Optional second argument is why it can’t be converted.

BM_compatible = False
PATTERN = None
explicit = False
filename = None
keep_line_order = False
numbers = count(1)
options = None
order = 'post'
pattern = None
pattern_tree = None
run_order = 5
syms = <fissix.pygram.Symbols object>
used_names = {}
class fissix.fixer_base.ConditionalFix(options, log)

Bases: fissix.fixer_base.BaseFix

Base class for fixers which not execute if an import is found.

should_skip(node)
start_tree(*args)

Some fixers need to maintain tree-wide state. This method is called once, at the start of tree fix-up.

tree - the root node of the tree to be processed. filename - the name of the file the tree came from.

skip_on = None

Utility functions, node construction macros, etc.

fissix.fixer_util.ArgList(args, lparen=Leaf(7, '('), rparen=Leaf(8, ')'))

A parenthesised argument list, used by Call()

fissix.fixer_util.Assign(target, source)

Build an assignment statement

fissix.fixer_util.Attr(obj, attr)

A node tuple for obj.attr

fissix.fixer_util.BlankLine()

A blank line

fissix.fixer_util.Call(func_name, args=None, prefix=None)

A function call

fissix.fixer_util.Colon()
fissix.fixer_util.Comma()

A comma leaf

fissix.fixer_util.Dot()

A period (.) leaf

fissix.fixer_util.FromImport(package_name, name_leafs)

Return an import statement in the form: from package import name_leafs

fissix.fixer_util.ImportAndCall(node, results, names)

Returns an import statement and calls a method of the module:

import module module.name()

fissix.fixer_util.KeywordArg(keyword, value)
fissix.fixer_util.LBrace()
fissix.fixer_util.LParen()
fissix.fixer_util.ListComp(xp, fp, it, test=None)

A list comprehension of the form [xp for fp in it if test].

If test is None, the “if test” part is omitted.

fissix.fixer_util.Name(name, prefix=None)

Return a NAME leaf

fissix.fixer_util.Newline()

A newline literal

fissix.fixer_util.Number(n, prefix=None)
fissix.fixer_util.RBrace()
fissix.fixer_util.RParen()
fissix.fixer_util.String(string, prefix=None)

A string leaf

fissix.fixer_util.Subscript(index_node)

A numeric or string subscript

fissix.fixer_util.attr_chain(obj, attr)

Follow an attribute chain.

If you have a chain of objects where a.foo -> b, b.foo-> c, etc, use this to iterate over all objects in the chain. Iteration is terminated by getattr(x, attr) is None.

Args:

obj: the starting object attr: the name of the chaining attribute

Yields:

Each successive object in the chain.

fissix.fixer_util.does_tree_import(package, name, node)

Returns true if name is imported from package at the top level of the tree which node belongs to. To cover the case of an import like ‘import foo’, use None for the package and ‘foo’ for the name.

fissix.fixer_util.find_binding(name, node, package=None)

Returns the node which binds variable name, otherwise None. If optional argument package is supplied, only imports will be returned. See test cases for examples.

fissix.fixer_util.find_indentation(node)

Find the indentation of node.

fissix.fixer_util.find_root(node)

Find the top level namespace.

fissix.fixer_util.in_special_context(node)

Returns true if node is in an environment where all that is required of it is being iterable (ie, it doesn’t matter if it returns a list or an iterator). See test_map_nochange in test_fixers.py for some examples and tests.

fissix.fixer_util.is_import(node)

Returns true if the node is an import statement.

fissix.fixer_util.is_list(node)

Does the node represent a list literal?

fissix.fixer_util.is_probably_builtin(node)

Check that something isn’t an attribute or function name etc.

fissix.fixer_util.is_tuple(node)

Does the node represent a tuple literal?

fissix.fixer_util.make_suite(node)
fissix.fixer_util.parenthesize(node)
fissix.fixer_util.touch_import(package, name, node)

Works like does_tree_import but adds an import statement if it was not imported.

Refactoring

Refactoring framework.

Used as a main program, this can refactor any number of files and/or recursively descend down directories. Imported as a module, this provides infrastructure to write your own refactoring tool.

exception fissix.refactor.FixerError

Bases: Exception

A fixer could not be loaded.

exception fissix.refactor.MultiprocessingUnsupported

Bases: Exception

class fissix.refactor.MultiprocessRefactoringTool(*args, **kwargs)

Bases: fissix.refactor.RefactoringTool

refactor(items, write=False, doctests_only=False, num_processes=1)

Refactor a list of files and directories.

refactor_file(*args, **kwargs)

Refactors a file.

class fissix.refactor.RefactoringTool(fixer_names, options=None, explicit=None)

Bases: object

gen_lines(block, indent)

Generates lines as expected by tokenize from a list of lines.

This strips the first len(indent + self.PS1) characters off each line.

get_fixers()

Inspects the options to load the requested patterns and handlers.

Returns:

(pre_order, post_order), where pre_order is the list of fixers that want a pre-order AST traversal, and post_order is the list that want post-order traversal.

log_debug(msg, *args)
log_error(msg, *args, **kwds)

Called when an error occurs.

log_message(msg, *args)

Hook to log a message.

parse_block(block, lineno, indent)

Parses a block into a tree.

This is necessary to get correct line number / offset information in the parser diagnostics and embedded into the parse tree.

print_output(old_text, new_text, filename, equal)

Called with the old version, new version, and filename of a refactored file.

processed_file(new_text, filename, old_text=None, write=False, encoding=None)

Called when a file has been refactored and there may be changes.

refactor(items, write=False, doctests_only=False)

Refactor a list of files and directories.

refactor_dir(dir_name, write=False, doctests_only=False)

Descends down a directory and refactor every Python file found.

Python files are assumed to have a .py extension.

Files and subdirectories starting with ‘.’ are skipped.

refactor_docstring(input, filename)

Refactors a docstring, looking for doctests.

This returns a modified version of the input string. It looks for doctests, which start with a “>>>” prompt, and may be continued with “…” prompts, as long as the “…” is indented the same as the “>>>”.

(Unfortunately we can’t use the doctest module’s parser, since, like most parsers, it is not geared towards preserving the original source.)

refactor_doctest(block, lineno, indent, filename)

Refactors one doctest.

A doctest is given as a block of lines, the first of which starts with “>>>” (possibly indented), while the remaining lines start with “…” (identically indented).

refactor_file(filename, write=False, doctests_only=False)

Refactors a file.

refactor_stdin(doctests_only=False)
refactor_string(data, name)

Refactor a given input string.

Args:

data: a string holding the code to be refactored. name: a human-readable name for use in error/log messages.

Returns:

An AST corresponding to the refactored input stream; None if there were errors during the parse.

refactor_tree(tree, name)

Refactors a parse tree (modifying the tree in place).

For compatible patterns the bottom matcher module is used. Otherwise the tree is traversed node-to-node for matches.

Args:
tree: a pytree.Node instance representing the root of the tree

to be refactored.

name: a human-readable name for this tree.

Returns:

True if the tree was modified, False otherwise.

summarize()
traverse_by(fixers, traversal)

Traverse an AST, applying a set of fixers to each node.

This is a helper method for refactor_tree().

Args:

fixers: a list of fixer instances. traversal: a generator that yields AST nodes.

Returns:

None

wrap_toks(block, lineno, indent)

Wraps a tokenize stream to systematically modify start/end.

write_file(new_text, filename, old_text, encoding=None)

Writes a string to a file.

It first shows a unified diff between the old text and the new text, and then rewrites the file; the latter is only done if the write option is set.

CLASS_PREFIX = 'Fix'
FILE_PREFIX = 'fix_'
PS1 = '>>> '
PS2 = '... '
fissix.refactor.get_all_fix_names(fixer_pkg, remove_prefix=True)

Return a sorted list of all available fix names in the given package.

fissix.refactor.get_fixers_from_package(pkg_name)

Return the fully qualified names for fixers in the package pkg_name.

Main program for 2to3.

class fissix.main.StdoutRefactoringTool(fixers, options, explicit, nobackups, show_diffs, input_base_dir='', output_dir='', append_suffix='')

Bases: fissix.refactor.MultiprocessRefactoringTool

A refactoring tool that can avoid overwriting its input files. Prints output to stdout.

Output files can optionally be written to a different directory and or have an extra file suffix appended to their name for use in situations where you do not want to replace the input files.

log_error(msg, *args, **kwargs)

Called when an error occurs.

print_output(old, new, filename, equal)

Called with the old version, new version, and filename of a refactored file.

write_file(new_text, filename, old_text, encoding)

Writes a string to a file.

It first shows a unified diff between the old text and the new text, and then rewrites the file; the latter is only done if the write option is set.

fissix.main.diff_texts(a, b, filename)

Return a unified diff of two strings.

fissix.main.main(fixer_pkg, args=None)

Main program.

Args:

fixer_pkg: the name of a package where the fixers are located. args: optional; a list of command line arguments. If omitted,

sys.argv[1:] is used.

Returns a suggested exit status (0, 1, 2).

fissix.main.warn(msg)