racing · software · open-source

Building a mutation test framework in crystal - Part 1

Published on January 22, 2019 · 443 words · about 2 min reading time

In this series of posts I want to explain and document the basic building blocks of building a mutation test framework in crystal-lang. For a bit more context on the working framework named crytic, make sure to check out the introductory post.

So let's get into the first block:

Mutating code

To start off, let's have a look at how we can modify some code programmatically. This is after all one of the cornerstones of mutation testing: mutating the code under test. As an example, let's assume we have the subjects code present in a string, and we want to change each occurrence of true to false instead.

We could go ahead and pull our trusty Regex trickery out of the bag and do a string replacement on the code. This could work in a simple example, but in more complex scenarios, this would be problematic. Think comments, or future mutant's that not only do simple boolean replacements.

Are more robust way of doing this is to get an alternate representation of the code, versus only having a string. A better representation is the abstract syntax tree (AST). The AST is a data structure used in compilers, such as the crystal compiler. We can obtain an AST from a string by using crystals own AST parser, like so:

Now what? What good is the AST? Well, luckily, crystal also provides means of interacting with it. There are two types of classes in crystal to deal with the AST: Visitor and Transformer. Let's make a simplified distinction here. Visitor lives to inspect the AST, and Transformer is the best choice to modify the ast. So let's subclass Transformer and build ourselves a "mutant".

That's it! The AST will apply our transformer, which will be called for every BoolLiteral, and return a literal false.

Mutating only certain code

Let's go one step further and look at how we could decide whether to transform a certain piece of code, or not. Analysing the source code first can give us the benefit of knowing beforehand how many mutations we can perform, and possibly excluding certain nodes. Imagine multiple booleans inside the above def hello. Our current transformer would replace them all at the same time. Now for a mutation testing tool this is undesired, since we want to observe one change at a time. Implementing an aforementioned Visitor can give us a location to every boolean:

Now we can save that location, an instance of Crystal::Location, to the boolean or ignore it if we want.

Those two subclasses together, the Transformer and the Visitor allow crytic to actually mutate code.

Next time we will probably look at how to actually run the mutated code.

Introducing crytic - mutation testing in crystal-lang

Published on October 20, 2018 · 471 words · about 2 min reading time

While trying to be as pedantic as possible about test automation with this very blog's source code, I had two todos: performance and mutation testing. Both of those I did not have a plan yet as to how to tackle them. However, I have previously dabbled in mutation testing with ruby's mutant gem, so I at least had an idea of what to get out of a mutation test suite. Some searching revealed to me that there was no mutation test library yet for crystal. And what does the software engineer do in that case? Write his/her own of course! So I hereby present to you:

Crytic

crytic, distributed as a crystal shard, is my stab at mutation testing. It's new, it can't do much, it's probably buggy, but it already serves a simple purpose: Check parts of this blog's code for uncovered bits.

About code coverage

You might say: "why the heck do I need this? I have a code coverage report that shows 100%, so I am golden!". However, take the following code as an example:

This test will mark the valid? function as being 100% covered. But it does not at all make sure it works correctly. Returning false instead of true would still pass the test-suite at 100% coverage, but is obviously the opposite result.

So how to protect against this? Use crytic!

Usage

Running crytic --subject valid.cr valid_spec.cr yields the following output:

The line Original suite: ✅ is there to tell you that the original spec passed. This is helpful, because there is no point running mutated source code against a failing initial test-suite, because there is nothing to be learned from trying to make fail already failing tests.

❌ BoolLiteralFlip (x1) tells you that crytic has run one "mutation", more specificly the BoolLiteralFlip mutation on the valid? method. It changed the source code of the valid? method so that the boolean literal is false instead of true. It then ran the tests again. This mutation did not make the test-suite fail, so a wrong implementation slipped through the cracks of the suite. What follows is the diff that was made in order to detect this.

What to do with the result?

Improve the tests! So let's go ahead and run crytic on the improved test here:

This will now show you ✅ BoolLiteralFlip (x1) to tell you that indeed the test-suite detected the intentionally modified code. You can now be much more confident in your code coverage and refactoring your code.

Future

Obviously crytic is still very much new and just an MVP. I plan to improve all parts (output, speed, more mutations, ...) of it and use it in a few more places. Feel free to try it out to improve your testing efforts. The world needs better (and less 🤪) software and test automation is key to achieving it.

Quick test feedback with fswatch

Published on July 28, 2018 · 201 words · about 1 min reading time

Loads of tools exist for continuously monitoring files on your disk for changes and running the tests whenever a change happened. Some test runners do that by themselves, others don't and rely on additional tools. The crystal test runner doesn't provide such an option, so I looked for an alternative. The easiest I found was the following:

fswatch -or ./src ./spec | xargs -I{} crystal spec

fswatch is using the system's (e.g. macOS) filesystem events to get notified of changes in files. Combining this with a pipe to xargs is making for a super simple tool to execute crystal spec whenever I change either a test or a production code file. Now for the arguments used:

Opinion: Robotic racing

Published on July 15, 2018 · 206 words · about 1 min reading time

Roborace, the autonomous racing company and car, has recently competed in the Goodwood Festival Of Speed. I am a huge proponent of autonomy in the automotive sector as a whole, but sporadically following Roborace's public outings make me wonder if autonomous racing will ever be a thing. Watch the hillclimb first:

A few questions answered first: Do I think this is impressive? Absolutely. Massive props to the team of engineers that got this car speeding up that hill. Do I think this makes sense? Totally, those outings push forward the state of autonomous driving for sure. So this post is not at all a bash on the achievements or intentions of Roborace at all.

But can I imagine myself watching a race of autonomous vehicles, should it ever happen? Hard to tell, but it sure doesn't get me to the edge of my seat right now. Reflecting on why I dig watching F1 or AMA Pro Supercross, the "human" parts (the odd errors, emotions, sense of danger ...) are most certainly a factor. Let's see where this goes, and make a call once we actually see an autonomous race on real racetracks.

Bonus, the onboard:

My favorite continuous integration providers (May 2018)

Published on May 24, 2018 · 280 words · about 1 min reading time

Seeing the following tweet yesterday, which links to a great article comparing multiple CI/CD providers

made me realize that I did indeed go through most of them at some point in time. So very shortly, here are my two cents:

CircleCI

I agree with the article, it's great, I use it to do CI/CD for this very website. Nothing much to say. The build UI is not great on mobile though. Cannot do iOS/Android builds though?

Travis CI

Still use it, gets the job done. Nothing more to add. It's running the danger-todoist builds.

Bitrise

I found the configuration weird in the beginning, but you can get used to it. Major plus: Can do both my iOS and Android builds for PitBuddy App for private repositories.

Nevercode

I think Nevercode aquired Buddybuild at some point. I was using Buddybuild for iOS builds and when I started to get an Android build going it just was impossible to setup and massively unreliably. Can't say much about Nevercode, but I am happy I left Buddybuild for Bitrise.

Bitbucket Pipelines

Obviously I just recently started using it, so far so good. Can't do iOS/Android. Build speeds are great up until now.

Gitlab CI

Is this the solution to doing both web, iOS and Android in one tool? Hosting the code there as well? Seems tempting! I like Gitlab the SCM tool, and Gitlab CI has come a long way.

That's it, just a quick reminder on what I already tried and would like to try.

Next page »