Managing the eXtyles test suite

In a recent post about the eXtyles “build” process, we talked a bit about our test suite. What’s the test suite? It’s a representative set of thousands of files, assembled over many years and constantly evolving, on which we test-run every possible eXtyles process every weeknight.

Because its whole purpose is to allow automated testing of everything eXtyles does, the eXtyles test suite has been “in progress” through twenty years of eXtyles development—always expanding to cover new file types, new features, new use cases, and new customers. The test suite currently includes thousands of unique documents and test cases; we also add new test documents with each new eXtyles customer, so we’re always testing out our software in relevant real-world conditions. In addition, we add a test case whenever we fix a bug, to make sure we never inadvertently reverse a fix!

In building up our test suite over the years, we’ve also developed standard protocols to manage it. In addition to internally created files designed to test specific behaviors, the test suite consists of curated customer documents. Here’s how that process works.

Step 1: New files for new styles

First, whenever we set up a new build or a new export filter for a customer, we add to the test suite a sample file that includes all paragraph styles used by that customer. This ensures that no matter what changes we make to the generic code, or to any customer-specific file, all paragraph styles will still export correctly in the XML.

Step 2: Real files from the real world

Next, we add at least a few real documents from the new customer, to make sure that the average file their users run through eXtyles will always export in the same way.

Once a new customer starts using eXtyles in their workflow, they’ll send us any document that eXtyles doesn’t seem to be handling correctly—for example, an Advanced Process might be doing the wrong thing with a particular type of reference, or there might be errors in the exported XML that can’t be explained by problems in the Word file. If we determine that the error is caused by a bug, we’ll usually add the document to the test suite; this lets us test that the bug fix works and ensures that future code changes don’t disrupt it.

How many documents per customer are in the test suite? That depends! If a customer has been using eXtyles for some time, and has been helping us improve eXtyles by reporting unexpected behavior whenever it occurs, then a large number of their documents will be included in our test suite. On the other hand, a newer customer might have fewer than half a dozen documents, depending on how many sample files they provided to us during the initial implementation.

Step 3: The right tests for the right files

For every document in the test suite, a checklist identifies which eXtyles processes can run on it and how it’s exported once those processes have been run.

Those processes run each night during our automated testing, and every morning our QA team reviews the results of the night’s testing, looking for changes, no matter how small. We expect each day’s results to include a certain amount of “noise” (e.g., a redirected URL or a new Crossref link that didn’t appear the previous day); more substantive changes are reviewed in detail by the QA, Development, and/or Configuration teams.

When a file contains improvements (positive changes), QA will add it to the test suite, where it replaces the previous version and becomes the new standard. When a file contains a regression (a negative or unwanted change), Development and Configuration immediately start investigating the cause and working on a fix!

If you see something, send something

When you open a Support ticket or send us a query, we’ll almost always want to you to include all versions of the file you’re having trouble with.

Reviewing the problematic files help us reproduce, understand, and resolve your issue more quickly. Not only that, having these files gives us the option to add them to the test suite if they represent a new or unusual use case or will help us keep bug fixes on track.

If you’re an eXtyles customer, you have documents in the eXtyles test suite, doing their part every night to help us keep improving eXtyles!