Kotej
How to test an XSLT migration by comparing old and new output
Rewriting a stylesheet for XSLT 3.0, moving from Xalan to Saxon, or just refactoring a 2,000-line transformation nobody dares to touch: the goal is always the same. For every input, the new version has to produce the same document as the old one. The safest way to prove it is to run both on real inputs and compare the outputs.
Step 1: collect representative inputs
Take a folder of real documents that covers the cases the stylesheet handles, including the strange ones:
empty optional elements, the longest message you have seen, special characters, every value of every
xsl:choose branch you can find. Production samples beat hand-written ones.
Step 2: run both versions
With Saxon from the command line, transforming a whole folder at once:
java -jar saxon-he.jar -s:inputs -xsl:old.xsl -o:out-old
java -jar saxon-he.jar -s:inputs -xsl:new.xsl -o:out-newWhen -s is a directory, Saxon transforms every file in it and writes each result under the
-o directory with the same name. If the old stylesheet is XSLT 1.0 and has to run on its original
processor, run it there and keep the outputs; what matters is that both folders hold the output of the same
inputs.
For a single stylesheet you are still editing, XSLT Playground runs XSLT 1.0, 2.0 and 3.0 in the browser on their own processors, keeps a version history of your stylesheet and its output, and can send the comparison between two versions straight to Kotej.
Step 3: compare the outputs as XML, not as text
A new processor or a rewritten template rarely produces the same bytes even when the result is right:
attributes come out in a different order, the indentation changes, a namespace gets a different prefix,
<x/> becomes <x></x>. A text diff flags all of it. Compare the two
folders in Kotej instead: it pairs the output files by name, compares each one
as XML, and marks the pairs that are equivalent (same document, different formatting) apart from the ones
that really differ.
Open the ones that differ and read the change. Typical findings in a migration:
- Whitespace in text nodes from a changed
xsl:strip-spaceor indentation setting, which may or may not matter to whoever reads the output. - Number and date formatting, where XSLT 2.0 and 3.0 are stricter about types than 1.0.
- Default namespace problems, where elements end up in no namespace or in the wrong one.
- Missing or duplicated elements from templates that match differently under the new priority rules.
Make it a regression test
Keep the input folder and the approved output folder in the repository. After any later change, run the stylesheet again and compare the new output with the approved one; an empty comparison is a passing test. The XSLT testing and regression guide on the XSLT Playground blog covers the other half: what to test and how to organise the cases.