racing · software · open-source

Introducing crytic - mutation testing in crystal-lang

Published on October 20, 2018 · 471 words · about 2 min reading time

While trying to be as pedantic as possible about test automation with this very blog's source code, I had two todos: performance and mutation testing. Both of those I did not have a plan yet as to how to tackle them. However, I have previously dabbled in mutation testing with ruby's mutant gem, so I at least had an idea of what to get out of a mutation test suite. Some searching revealed to me that there was no mutation test library yet for crystal. And what does the software engineer do in that case? Write his/her own of course! So I hereby present to you:

Crytic

crytic, distributed as a crystal shard, is my stab at mutation testing. It's new, it can't do much, it's probably buggy, but it already serves a simple purpose: Check parts of this blog's code for uncovered bits.

About code coverage

You might say: "why the heck do I need this? I have a code coverage report that shows 100%, so I am golden!". However, take the following code as an example:

This test will mark the valid? function as being 100% covered. But it does not at all make sure it works correctly. Returning false instead of true would still pass the test-suite at 100% coverage, but is obviously the opposite result.

So how to protect against this? Use crytic!

Usage

Running crytic --subject valid.cr valid_spec.cr yields the following output:

The line Original suite: ✅ is there to tell you that the original spec passed. This is helpful, because there is no point running mutated source code against a failing initial test-suite, because there is nothing to be learned from trying to make fail already failing tests.

❌ BoolLiteralFlip (x1) tells you that crytic has run one "mutation", more specificly the BoolLiteralFlip mutation on the valid? method. It changed the source code of the valid? method so that the boolean literal is false instead of true. It then ran the tests again. This mutation did not make the test-suite fail, so a wrong implementation slipped through the cracks of the suite. What follows is the diff that was made in order to detect this.

What to do with the result?

Improve the tests! So let's go ahead and run crytic on the improved test here:

This will now show you ✅ BoolLiteralFlip (x1) to tell you that indeed the test-suite detected the intentionally modified code. You can now be much more confident in your code coverage and refactoring your code.

Future

Obviously crytic is still very much new and just an MVP. I plan to improve all parts (output, speed, more mutations, ...) of it and use it in a few more places. Feel free to try it out to improve your testing efforts. The world needs better (and less 🤪) software and test automation is key to achieving it.

Quick test feedback with fswatch

Published on July 28, 2018 · 201 words · about 1 min reading time

Loads of tools exist for continuously monitoring files on your disk for changes and running the tests whenever a change happened. Some test runners do that by themselves, others don't and rely on additional tools. The crystal test runner doesn't provide such an option, so I looked for an alternative. The easiest I found was the following:

fswatch -or ./src ./spec | xargs -I{} crystal spec

fswatch is using the system's (e.g. macOS) filesystem events to get notified of changes in files. Combining this with a pipe to xargs is making for a super simple tool to execute crystal spec whenever I change either a test or a production code file. Now for the arguments used:

Opinion: Robotic racing

Published on July 15, 2018 · 206 words · about 1 min reading time

Roborace, the autonomous racing company and car, has recently competed in the Goodwood Festival Of Speed. I am a huge proponent of autonomy in the automotive sector as a whole, but sporadically following Roborace's public outings make me wonder if autonomous racing will ever be a thing. Watch the hillclimb first:

A few questions answered first: Do I think this is impressive? Absolutely. Massive props to the team of engineers that got this car speeding up that hill. Do I think this makes sense? Totally, those outings push forward the state of autonomous driving for sure. So this post is not at all a bash on the achievements or intentions of Roborace at all.

But can I imagine myself watching a race of autonomous vehicles, should it ever happen? Hard to tell, but it sure doesn't get me to the edge of my seat right now. Reflecting on why I dig watching F1 or AMA Pro Supercross, the "human" parts (the odd errors, emotions, sense of danger ...) are most certainly a factor. Let's see where this goes, and make a call once we actually see an autonomous race on real racetracks.

Bonus, the onboard:

My favorite continuous integration providers (May 2018)

Published on May 24, 2018 · 280 words · about 1 min reading time

Seeing the following tweet yesterday, which links to a great article comparing multiple CI/CD providers

made me realize that I did indeed go through most of them at some point in time. So very shortly, here are my two cents:

CircleCI

I agree with the article, it's great, I use it to do CI/CD for this very website. Nothing much to say. The build UI is not great on mobile though. Cannot do iOS/Android builds though?

Travis CI

Still use it, gets the job done. Nothing more to add. It's running the danger-todoist builds.

Bitrise

I found the configuration weird in the beginning, but you can get used to it. Major plus: Can do both my iOS and Android builds for PitBuddy App for private repositories.

Nevercode

I think Nevercode aquired Buddybuild at some point. I was using Buddybuild for iOS builds and when I started to get an Android build going it just was impossible to setup and massively unreliably. Can't say much about Nevercode, but I am happy I left Buddybuild for Bitrise.

Bitbucket Pipelines

Obviously I just recently started using it, so far so good. Can't do iOS/Android. Build speeds are great up until now.

Gitlab CI

Is this the solution to doing both web, iOS and Android in one tool? Hosting the code there as well? Seems tempting! I like Gitlab the SCM tool, and Gitlab CI has come a long way.

That's it, just a quick reminder on what I already tried and would like to try.

FTP lives: Deploying a static site with Bitbucket pipelines

Published on May 23, 2018 · 529 words · about 2 min reading time

Yes. It still exists. Not the entire world has moved to a fancy pants containerized Kubernetes - Serverless - Cloud Hosting (I don't even know the actual latest trends) setup yet. Case in point is pitbuddyapp.com, website to my hobby iOS and Android app for rc car racing enthusiasts. The website is plain static html, no preprocessors, no javascript, build steps or the likes. It is hosted on my families 1und1 webspace. So how did I deploy this in the past? Yes, I took the trusty FTP program and copied the files (insert laughter here).

Bitbucket pipelines

The code for the site itself is versioned in a private repo on bitbucket.org, and bitbucket does come with their version of CI/CD tooling called "pipelines". I wanted to use this since setting up yet another external CI integration like travis or circleci felt a bit overkill for this. Granted, the site did not have any tests/verification at all so far.

BackstopJS once more

Having no tests or verification mechanism at all is not a great prerequisite for a continuous delivery toolchain, even for a static site. This would mean broken versions of the site would be deployed without warning. So once more backstopjs to the rescue. Setting up backstopjs was again without hassle, it can resolve file:// urls no problem. So a set of reference images were quickly created and I could look at reproducing those images in the pipeline.

The pipeline

Bitbucket pipelines are configured through a bitbucket-pipelines.yml file. No surprises here, thats in essence how most of the CI providers do it. The two cornerstones of the pipeline are two docker images. One to run the visual regression tests, the other to do the deployment. That's a "test->deploy" pipeline in a nutshell. This is the configuration in it's entirety:

The test step, run directly on the official backstopjs docker container, is simply executing the test task of backstopjs. This compares three reference images (three because I set up three viewport sizes once more) against what is currently rendered when opening the index.html. When they are identical, the deploy pipeline step is triggered. Deploy is, as mentioned, as simple as copying the relevant files to the correct directory on the webspace. Already a while ago I found a git plugin to accomplish this. It is called git-ftp and solves my issue very smoothly. As for backstop, a docker image already exists just to run git ftp. So all I needed to do (and I have struggled to get this working in the past) was to create a separate ftp user (for security reasons), add the FTP_USER, FTP_PASSWORD and FTP_URL environment variables to the bitbucket pipelines settings and voilá, continuous deployment for a static site with the safety net of visual regression tests it is 🎉. git ftp can use a separate ignore file to tune what content to deploy or omit.

Lessons learned

Next page »