path: root/mped.xml

<?xml version="1.0" encoding="utf-8"?>
<book xmlns="http://docbook.org/ns/docbook"
      xmlns:xi="http://www.w3.org/2001/XInclude"
      xmlns:mped="https://labo.planete-kraus.eu/mped.git" version="5.0">
  <info>
    <title>Meta-Programming in Extensible Documents</title>
    <author>
      <personname>Vivien Kraus</personname>
    </author>
    <abstract>
      <para>
        Literate Programming is a technique for writing programs, in a
        way that the code comes to support the ideas developed in a
        human language. Successful programs written with that
        technique are easy to understand, because you can get all the
        important ideas while reading the document, cover to cover.
      </para>
      <para>
        It is tempting to add code evaluation to the literate
        programming technique. With this common addition, the
        techniques becomes a meta-programming technique as it lets
        programmers use programs to write programs, possibly in other
        programming languages.
      </para>
      <para>
        While many tools provide such meta-programming capabilities to
        the literate programming task, it remains fairly uncommon to
        have it applied to extensible documents, in the XML
        ecosystem. This book provides a new extension to Docbook, to
        support meta-programming.
      </para>
    </abstract>
    <keywordset>
      <keyword>Meta-Programming</keyword>
      <keyword>Literate Programming</keyword>
      <keyword>XML</keyword>
    </keywordset>
    <copyright>
      <year>2022</year>
      <holder>Vivien Kraus</holder>
    </copyright>
    <legalnotice>
      <para>
        I have not made a decision about the license of the program.
      </para>
    </legalnotice>
  </info>
  <preface>
    <title>What this book is trying to do</title>
    <para>
      As developers, we like to broaden our ideas about how
      programming should be done. Repeating the same design process of
      computer programs is boring, to the point that it seems robots
      would do it better than us. After all, writing programs is very
      far from a purely scientific or engineering task, a large part
      of the writing process is deciding how to lay out the idea on
      the medium. Choices of styles, or technologies te write the
      program, are always more a question of personal preference than
      an objective cost of development.
    </para>
    <para>
      Lawyers worldwide seem to have noticed that, too, which is why
      it has been decided that computer programs would be ruled by
      copyright law: there are different ways to express an idea, none
      of which is inherently better than the others, so the law
      controls the expression of these ideas, not the ideas
      themselves.
    </para>
    <para>
      Among the different ways to write a program, literate
      programming must be one of the most appealing. Write a book,
      develop your ideas, and support them with code. I like the idea!
      Let’s do it. What do we need?
    </para>
    <para>
      First, we need to write a book. A very special book that is: it
      must feature text, programs, and documentation of this
      program. This is not very typical of a book, so we want the
      authoring process to be <emphasis>extensible</emphasis>, so that
      it lets authors add elements to their books without modifying
      the process they use to write their books. Books are typically
      written in a markup language: text is divided into elements that
      carry some intrinsic semantics, such as chapters. We want an
      extensible markup language, so that we can create new semantic
      elements without changing the language. I know two classes of
      extensible markup languages: ones where the extensions are code
      plug-ins to editors for that markup language, which is how you
      add features for org-mode through emacs plugins, for instance,
      and XML. For this present task, I want to use XML.
    </para>
    <para>
      We also need to write a program. Thus, our markup language
      should be able to take the pieces of code around and compile
      them to a program. While it is possible to write a program that
      would parse the document and extract the source code, I find it
      way more elegant to leverage XSLT, the stylesheet and
      transformation language for markup languages based on XML.
    </para>
    <para>
      Finally, we need to combine everything into a printable
      document. There, XSLT is a tool to be used too.
    </para>
    <para>
      The work presented here uses its own namespace:
      <uri>https://labo.planete-kraus.eu/mped.git</uri>, that we will
      now summarize as “mped”.
    </para>
  </preface>
  <chapter>
    <title>Tangling pieces of code from the document</title>
    <para>
      One of the most iconic features of literate programming is its
      ability to extract source code blocks and put them in files.
    </para>
    <section>
      <title>One source block to one file</title>
      <para>
        The document contains program listings that support the
        development of ideas. These are usually written in elements,
        siblings to paragraphs, and for Docbook, of type
        &lt;programlisting&gt;. The most important attribute,
        “language”, identifies the programming language.
      </para>
      <para>
        However, there is no attribute in Docbook that tells the
        tangling program where each piece of code should end up. This is
        why we introduce our first extension: the “mped:tangle-to”
        attribute.
      </para>
      <para>
        To tangle a document, an XSLT stylesheet is defined. It reads
        a Docbook document, and outputs a shell script that writes the
        correct pieces of code to the correct file names. The key
        template to do the task is <xref
        linkend="tangle-programlisting" />.
      </para>
      <figure xml:id="tangle-programlisting">
        <title>Copy a specific programlisting to disk</title>
        <programlisting language="xml">
          <![CDATA[
<xsl:template match="docbook:programlisting[@mped:tangle-to]">
  <xsl:text>mkdir -p $(dirname "</xsl:text>
  <xsl:value-of select="@mped:tangle-to" />
  <xsl:text>")&#xA;</xsl:text>

  <xsl:apply-templates select="." mode="source-code-formatter" />
  <xsl:text> > </xsl:text>
  <xsl:value-of select="@mped:tangle-to" />
  <xsl:text> &lt;&lt; "_MPED_EOF"&#xA;</xsl:text>

  <xsl:apply-templates mode="copy-source-code" />

  <xsl:text>&#xA;_MPED_EOF&#xA;</xsl:text>
</xsl:template>
          ]]>
        </programlisting>
      </figure>
      <para>
        This template starts by creating the directory where the file
        should go, then fills the file with the source code. XML has a
        precise behavior when it comes to whitespace preservation, but
        it’s not always the prettiest when we write it with whitespace
        output in mind. So, the code output is frequently not indented
        correctly, and has too many empty lines. To counter this
        effect, we use a code source <emphasis>formatter</emphasis>, a
        program that reads source code and indents it correctly. For
        XSLT, in <xref linkend="formatter-for-xml" />, we can use
        <command>xmllint</command> from
        <application>libxml2</application>. The important thing about
        the formatter is that it should take its input from standard
        input and write the formatted code to standard output.
      </para>
      <figure xml:id="formatter-for-xml">
        <title>Use xmllint as a formatter for XML languages</title>
        <programlisting language="xml">
          <![CDATA[
<xsl:template match="docbook:programlisting[@language = 'xml']" 
              mode="source-code-formatter">
  <xsl:text>xmllint --format -</xsl:text>
</xsl:template>
          ]]>
        </programlisting>
      </figure>
      <note>
        Unfortunately, xmllint does not accept an XML processing
        instruction after the first line, so you will still need to
        put no whitespace between the start tag of programlisting and
        the text for XML listings.
      </note>
      <para>
        If no formatter applies, then we can resort to
        <command>cat</command>, see <xref linkend="default-formatter"
        />.
      </para>
      <figure xml:id="default-formatter">
        <title>Use cat as the default formatter for any language</title>
        <programlisting language="xml">
          <![CDATA[
<xsl:template match="docbook:programlisting"
              mode="source-code-formatter">
  <xsl:text>cat</xsl:text>
</xsl:template>
          ]]>
        </programlisting>
      </figure>
      <para>
        We also need to specify how the source code is copied. It is
        very simple: <xref linkend="copy-source-code-text" /> copies
        the text verbatim.
      </para>
      <figure xml:id="copy-source-code-text">
        <title>Copy the source code as text</title>
        <programlisting language="xml">
          <![CDATA[
<xsl:template match="text()" mode="copy-source-code">
  <xsl:value-of select="." />
</xsl:template>
          ]]>
        </programlisting>
      </figure>
      <para>
        Tangling should never touch anything else. So, text should not
        be copied to output. This is why we disable text matching by
        default with <xref linkend="ignore-text-other-than-source" />.
      </para>
      <figure xml:id="ignore-text-other-than-source">
        <title>Ignore text by default when tangling</title>
        <programlisting language="xml">
          <![CDATA[
                 <xsl:template match="text()" />
          ]]>
        </programlisting>
      </figure>
    </section>
    <section>
      <title>Paste other listings in place</title>
      <para>
        Literate programming requires the author to be able to discuss
        bits of code in isolation, and then insert each bit into a
        larger bit. Mped provides this operation with a new tag,
        “mped:copy”. It has a “linkend” attribute that resolves to a
        program listing anywhere in the document. When copying source
        code, matching this element will insert the linked listing
        directly here. This is done in <xref
        linkend="tangle-mped-copy" />.
      </para>
      <para>
        More precisely, it looks if there is a single program listing
        that is directly under a figure with the given ID. This way,
        we can refer to listings as the figure they appear in, which
        makes cross-referencing easier.
      </para>
      <figure xml:id="tangle-mped-copy">
        <title>
          Insert literally a listing in another when tangling
        </title>
        <programlisting language="xml">
          <![CDATA[
<xsl:template match="mped:copy" mode="copy-source-code">
  <xsl:variable name="ref" select="@linkend" />
  <xsl:variable name="candidates"
                select="count(//docbook:figure[@xml:id = $ref]/docbook:programlisting)" />
  <xsl:if test="$candidates = 0">
    <xsl:message terminate="yes">
      <xsl:text>There are no listing directly within a figure with ID '</xsl:text>
      <xsl:value-of select="$ref" />
      <xsl:text>'.&#xA;</xsl:text>
    </xsl:message>
  </xsl:if>
  <xsl:if test="$candidates > 1">
    <xsl:message terminate="yes">
      <xsl:text>There are multiple listings directly within a figure with ID '</xsl:text>
      <xsl:value-of select="$ref" />
      <xsl:text>'.&#xA;</xsl:text>
    </xsl:message>
  </xsl:if>
  <xsl:apply-templates select="//docbook:figure[@xml:id = $ref]/docbook:programlisting"
                       mode="copy-source" />
</xsl:template>
          ]]>
        </programlisting>
      </figure>
    </section>
    <section>
      <title>Putting it all together</title>
      <para>
        The collection of all these templates gives a full stylesheet,
        in <xref linkend="whole-tangling-stylesheet" />.
      </para>
      <figure xml:id="whole-tangling-stylesheet">
        <title>The full stylesheet for tangling</title>
        <programlisting
            language="xml"
            mped:tangle-to="tangle.xsl"
            ><![CDATA[<?xml version="1.0"?>

<xsl:stylesheet version="1.0"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:mped="https://labo.planete-kraus.eu/mped.git"
                xmlns:docbook="http://docbook.org/ns/docbook">

  <xsl:output method="text" indent="no" />
  <xsl:strip-space elements="*" />
        ]]>
        <mped:copy linkend="tangle-programlisting" />
        <mped:copy linkend="formatter-for-xml" />
        <mped:copy linkend="default-formatter" />
        <mped:copy linkend="copy-source-code-text" />
        <mped:copy linkend="ignore-text-other-than-source" />
        <mped:copy linkend="tangle-mped-copy" />
        <![CDATA[
</xsl:stylesheet>
        ]]>
        </programlisting>
      </figure>
    </section>
  </chapter>
</book>