racing · software · open-source

Measuring code coverage in crystal with kcov

Published on February 24, 2019 路 592 words 路 about 2 min reading time

Crystal, the programming language, does not yet provide a built in way of measuring the effectiveness of your test suite. So by running crystal spec you pretty much only have binary insight into the suite: it's passing or it's not. This lead me to build crytic in the first place. But while mutation coverage is a great tool to investigate the test suite, plain old code coverage is usually quicker to obtain and easier to glance at.

The players

Looking at the issue above mentions two possible libraries: anykeyh/crystal-coverage and SimonKagstrom/kcov. I had been running crystal-coverage for a while and used it as an inspiration quite a few times, after all it's a crystal library that basically injects coverage markers into the crystal source code before compiling and running the tests. Due to it's proof of concept nature however it has a few limitations and leads to varying results and coverage numbers. Not quite ready for production use yet, but I'm excited for it's future or possible integration of a similar concept into the crystal compiler itself!

kcov on the other hand has nothing to do with crystal. It can work on any binary that has debug information (in DWARF) available. This is great because it doesn't really impose any requirements onto our crystal program.

Generating coverage data with kcov

Usage of kcov is very straightforward. The macOS binary has to be built manually, but the instructions are easy enough and work as advertise. So once you have a kcov to run, you can pass it your command to execute the program, more specifically in this case, the tests. You also need to pass a folder to write the coverage report to.

kcov ./coverage run_tests for example invokes a program run_tests, uses its debug information to track code coverage, and creates the report inside the coverage folder.

The typical way to run a crystal program's tests is usually crystal spec, which means you don't have a binary for kcov to analyze. The way I solve this is by creating an entrypoint program to run the tests:

  1. Create a crystal file requiring all specs: echo "require \"./spec/**\"" > run_tests.cr
  2. Compile that file to a binary crystal build run_tests.cr -D skip-integration

The second part allows you to pass compile time flags as well, as I do to exclude integration tests from the code coverage analysis. I find that coverage generated by integration tests is low-value and more incidental that not. That's why I usually only consider unit-tests for code coverage.

Clean and include path

Two options that kcov provides were important to my workflow: --clean instructs kcov to always get a fresh analysis. If omitted, results from all runs are accumulated, which is not what I want.

Running the command without --include-path=./src will provide a coverage report that also prints the entirety of the crystal standard library (at least the parts that are required by the program). Usually the report is only interesting for the program that you are working on.

Putting it all together

So far we have everything together to create code coverage analysis for your crystal program's test suite. Awesome! Go ahead, look at it and see what parts of your code are uncovered by the tests 馃嵖. As in so many cases, the analysis only gets useful if continually integrated into the development workflow. Luckily for us, kcov produces output that can be understood by various tools down the chain. I use codecov.io to upload the report and have it easily browseable online. See this blogs report or crytics report. The script I use on CI is as follows:

Building a mutation test framework in crystal - Part 1

Published on January 22, 2019 路 443 words 路 about 2 min reading time

In this series of posts I want to explain and document the basic building blocks of building a mutation test framework in crystal-lang. For a bit more context on the working framework named crytic, make sure to check out the introductory post.

So let's get into the first block:

Mutating code

To start off, let's have a look at how we can modify some code programmatically. This is after all one of the cornerstones of mutation testing: mutating the code under test. As an example, let's assume we have the subjects code present in a string, and we want to change each occurrence of true to false instead.

We could go ahead and pull our trusty Regex trickery out of the bag and do a string replacement on the code. This could work in a simple example, but in more complex scenarios, this would be problematic. Think comments, or future mutant's that not only do simple boolean replacements.

Are more robust way of doing this is to get an alternate representation of the code, versus only having a string. A better representation is the abstract syntax tree (AST). The AST is a data structure used in compilers, such as the crystal compiler. We can obtain an AST from a string by using crystals own AST parser, like so:

Now what? What good is the AST? Well, luckily, crystal also provides means of interacting with it. There are two types of classes in crystal to deal with the AST: Visitor and Transformer. Let's make a simplified distinction here. Visitor lives to inspect the AST, and Transformer is the best choice to modify the ast. So let's subclass Transformer and build ourselves a "mutant".

That's it! The AST will apply our transformer, which will be called for every BoolLiteral, and return a literal false.

Mutating only certain code

Let's go one step further and look at how we could decide whether to transform a certain piece of code, or not. Analysing the source code first can give us the benefit of knowing beforehand how many mutations we can perform, and possibly excluding certain nodes. Imagine multiple booleans inside the above def hello. Our current transformer would replace them all at the same time. Now for a mutation testing tool this is undesired, since we want to observe one change at a time. Implementing an aforementioned Visitor can give us a location to every boolean:

Now we can save that location, an instance of Crystal::Location, to the boolean or ignore it if we want.

Those two subclasses together, the Transformer and the Visitor allow crytic to actually mutate code.

Next time we will probably look at how to actually run the mutated code.

Introducing crytic - mutation testing in crystal-lang

Published on October 20, 2018 路 471 words 路 about 2 min reading time

While trying to be as pedantic as possible about test automation with this very blog's source code, I had two todos: performance and mutation testing. Both of those I did not have a plan yet as to how to tackle them. However, I have previously dabbled in mutation testing with ruby's mutant gem, so I at least had an idea of what to get out of a mutation test suite. Some searching revealed to me that there was no mutation test library yet for crystal. And what does the software engineer do in that case? Write his/her own of course! So I hereby present to you:

Crytic

crytic, distributed as a crystal shard, is my stab at mutation testing. It's new, it can't do much, it's probably buggy, but it already serves a simple purpose: Check parts of this blog's code for uncovered bits.

About code coverage

You might say: "why the heck do I need this? I have a code coverage report that shows 100%, so I am golden!". However, take the following code as an example:

This test will mark the valid? function as being 100% covered. But it does not at all make sure it works correctly. Returning false instead of true would still pass the test-suite at 100% coverage, but is obviously the opposite result.

So how to protect against this? Use crytic!

Usage

Running crytic --subject valid.cr valid_spec.cr yields the following output:

The line Original suite: 鉁 is there to tell you that the original spec passed. This is helpful, because there is no point running mutated source code against a failing initial test-suite, because there is nothing to be learned from trying to make fail already failing tests.

鉂 BoolLiteralFlip (x1) tells you that crytic has run one "mutation", more specificly the BoolLiteralFlip mutation on the valid? method. It changed the source code of the valid? method so that the boolean literal is false instead of true. It then ran the tests again. This mutation did not make the test-suite fail, so a wrong implementation slipped through the cracks of the suite. What follows is the diff that was made in order to detect this.

What to do with the result?

Improve the tests! So let's go ahead and run crytic on the improved test here:

This will now show you 鉁 BoolLiteralFlip (x1) to tell you that indeed the test-suite detected the intentionally modified code. You can now be much more confident in your code coverage and refactoring your code.

Future

Obviously crytic is still very much new and just an MVP. I plan to improve all parts (output, speed, more mutations, ...) of it and use it in a few more places. Feel free to try it out to improve your testing efforts. The world needs better (and less 馃お) software and test automation is key to achieving it.

Quick test feedback with fswatch

Published on July 28, 2018 路 201 words 路 about 1 min reading time

Loads of tools exist for continuously monitoring files on your disk for changes and running the tests whenever a change happened. Some test runners do that by themselves, others don't and rely on additional tools. The crystal test runner doesn't provide such an option, so I looked for an alternative. The easiest I found was the following:

fswatch -or ./src ./spec | xargs -I{} crystal spec

fswatch is using the system's (e.g. macOS) filesystem events to get notified of changes in files. Combining this with a pipe to xargs is making for a super simple tool to execute crystal spec whenever I change either a test or a production code file. Now for the arguments used:

Opinion: Robotic racing

Published on July 15, 2018 路 206 words 路 about 1 min reading time

Roborace, the autonomous racing company and car, has recently competed in the Goodwood Festival Of Speed. I am a huge proponent of autonomy in the automotive sector as a whole, but sporadically following Roborace's public outings make me wonder if autonomous racing will ever be a thing. Watch the hillclimb first:

A few questions answered first: Do I think this is impressive? Absolutely. Massive props to the team of engineers that got this car speeding up that hill. Do I think this makes sense? Totally, those outings push forward the state of autonomous driving for sure. So this post is not at all a bash on the achievements or intentions of Roborace at all.

But can I imagine myself watching a race of autonomous vehicles, should it ever happen? Hard to tell, but it sure doesn't get me to the edge of my seat right now. Reflecting on why I dig watching F1 or AMA Pro Supercross, the "human" parts (the odd errors, emotions, sense of danger ...) are most certainly a factor. Let's see where this goes, and make a call once we actually see an autonomous race on real racetracks.

Bonus, the onboard:

Next page »