| Author: | Stefan Behnel |
|---|
This document describes how to read the source code of lxml and how to start working on it. You might also be interested in the companion document that describes how to build lxml from sources.
Contents
Cython is the language that lxml is written in. It is a very Python-like language that was specifically designed for writing Python extension modules.
The reason why Cython (or actually its predecessor Pyrex at the time) was chosen as an implementation language for lxml, is that it makes it very easy to interface with both the Python world and external C code. Cython generates all the necessary glue code for the Python API, including Python types, calling conventions and reference counting. On the other side of the table, calling into C code is not more than declaring the signature of the function and maybe some variables as being C types, pointers or structs, and then calling it. The rest of the code is just plain Python code.
The Cython language is so close to Python that the Cython compiler can actually compile many, many Python programs to C without major modifications. But the real speed gains of a C compilation come from type annotations that were added to the language and that allow Cython to generate very efficient C code.
Even if you are not familiar with Cython, you should keep in mind that a slow implementation of a feature is better than none. So, if you want to contribute and have an idea what code you want to write, feel free to start with a pure Python implementation. Chances are, if you get the change officially accepted and integrated, others will take the time to optimise it so that it runs fast in Cython.
First of all, read how to build lxml from sources to learn how to retrieve the source code from the GitHub repository and how to build it. The source code lives in the subdirectory src of the checkout.
The main extension modules in lxml are lxml.etree and lxml.objectify. All main modules have the file extension .pyx, which shows the descendence from Pyrex. As usual in Python, the main files start with a short description and a couple of imports. Cython distinguishes between the run-time import statement (as known from Python) and the compile-time cimport statement, which imports C declarations, either from external libraries or from other Cython modules.
lxml's tree API is based on proxy objects. That means, every Element object (or rather _Element object) is a proxy for a libxml2 node structure. The class declaration is (mainly):
cdef class _Element:
cdef _Document _doc
cdef xmlNode* _c_node
It is a naming convention that C variables and C level class members that are passed into libxml2 start with a prefixed c_ (commonly libxml2 struct pointers), and that C level class members are prefixed with an underscore. So you will often see names like c_doc for an xmlDoc* variable (or c_node for an xmlNode*), or the above _c_node for a class member that points to an xmlNode struct (or _c_doc for an xmlDoc*).
It is important to know that every proxy in lxml has a factory function that properly sets up C level members. Proxy objects must never be instantiated outside of that factory. For example, to instantiate an _Element object or its subclasses, you must always call its factory function:
cdef xmlNode* c_node cdef _Document doc cdef _Element element ... element = _elementFactory(doc, c_node)
A good place to see how this factory is used are the Element methods getparent(), getnext() and getprevious().
An important part of lxml is the documentation that lives in the doc directory. It describes a large part of the API and comprises a lot of example code in the form of doctests.
The documentation is written in the ReStructured Text format, a very powerful text markup language that looks almost like plain text. It is part of the docutils package.
The project web site of lxml is completely generated from these text documents. Even the side menu is just collected from the table of contents that the ReST processor writes into each HTML page. Obviously, we use lxml for this.
The easiest way to generate the HTML pages is by calling:
make html
This will call the script doc/mkhtml.py to run the ReST processor on the files. After generating an HTML page the script parses it back in to build the side menu, and injects the complete menu into each page at the very end.
Running the make command will also generate the API documentation if you have epydoc installed. The epydoc package will import and introspect the extension modules and also introspect and parse the Python modules of lxml. The aggregated information will then be written out into an HTML documentation site.
The main module, lxml.etree, is in the file lxml.etree.pyx. It implements the main functions and types of the ElementTree API, as well as all the factory functions for proxies. It is the best place to start if you want to find out how a specific feature is implemented.
At the very end of the file, it contains a series of include statements that merge the rest of the implementation into the generated C code. Yes, you read right: no importing, no source file namespacing, just plain good old include and a huge C code result of more than 100,000 lines that we throw right into the C compiler.
The main include files are:
Error log handling. All error messages that libxml2 generates internally walk through the code in this file to end up in lxml's Python level error logs.
At the end of the file, you will find a long list of named error codes. It is generated from the libxml2 HTML documentation (using lxml, of course). See the script update-error-constants.py for this.
The different schema languages (DTD, RelaxNG, XML Schema and Schematron) are implemented in the following include files:
The lxml package also contains a number of pure Python modules:
A Cython implemented extension module that uses the public C-API of lxml.etree. It provides a Python object-like interface to XML trees. The implementation resides in the file lxml.objectify.pyx.
A specialised toolkit for HTML handling, based on lxml.etree. This is implemented in pure Python.