Mutode: Generic JavaScript and Node.js Mutation Testing Tool

Mutode: Generic JavaScript and Node.js Mutation

Testing Tool

Diego Rodríguez B.

June, 2018Version: 1.0

Systems and Computing Engineering DepartmentSchool of Engineering

Bogotá, Colombia

Undergraduate Thesis

Mutode: Generic JavaScript and Node.jsMutation Testing Tool

Diego Rodríguez B. [email protected]

1. Adviser PhD. Mario Linares Vá[email protected] and Computing Engineering DepartmentUniversidad de los Andes

June, 2018

mailto:[email protected]

mailto:[email protected]

Diego Rodríguez B.

Mutode: Generic JavaScript and Node.js Mutation Testing Tool

Undergraduate Thesis, June, 2018

Adviser: PhD. Mario Linares Vásquez

Universidad de los Andes

Systems and Computing Engineering Department

Cra 1 # 18A - 12

111711 , Bogotá, Colombia

Abstract

Mutation testing is a technique in which faults (mutants) are injected into a programor application to assess its test suite effectiveness. It works by inserting mutants andrunning the applications test suite to identify if the mutants are detected (killed) ornot (survived) by the tests. Although computationally expensive, it has proven tobe an effective method to assess application test suites. Several mutation testingframeworks and tools have been built for the various programing languages, however,very few tools have been built for the JavaScript language, more specifically, there isa lack of mutation testing tools for the Node.js runtime and npm based applications.The npm Registry is a public collection of modules of open-source code for Node.js,front-end web applications, mobile applications, robots, routers, and countless otherneeds of the JavaScript community. The over 700,000 packages hosted in npmare downloaded more than 5 billion times per week. More and more software ispublished in npm every day, representing a huge opportunity to share code andsolutions, but also to share bugs and faulty software. In this thesis, I describe priorwork for mutation operators in JavaScript and Node.js, and propose Mutode, an opensource tool which leverages the npm package ecosystem to perform mutation testingfor JavaScript and Node.js applications. Finally, I empirically evaluated Mutodeeffectiveness by running it on the top 20 npm modules that have automated testsuites.

Resumen

Las pruebas de mutación son un técnica en la que fallas (mutantes), son inyectadasen un programa o aplicación para evaluar la efectividad de su suite de pruebas.Funciona insertando mutantes y corriendo la suite de pruebas de la aplicación,verificando si el mutante fue detectado (matado), o no (sobreviviente), por lapruebas. Aunque es computacionalmente costoso, ha demostrado ser un métodoefectivo para evaluar la suite de pruebas de una aplicación. Varias herramientas yframeworks se han construído para los diferentes lenguajes de programación. Sinembargo, muy pocas herramientas se han construído para el lenguaje JavaScript,más específicamente, hay una falta de herramientas de pruebas de mutación paraNode.js y aplicaciones basadas en npm. El repositorio de npm es una colección

v

pública de módulos de código abierto para Node.js, aplicaciones web, aplicacionesmóviles, robots, routers, y muchas otras necesidades de la comunidad de JavaScript.Los más de 700.000 módulos alojados en npm son descargados más de cinco milmillones de veces por semana. Más y más software es publicado en npm cada día,representando una inmensa oportunidad para compartir código y soluciones, perotambién para compartir errores y fallas de software. En esta tesis, describo trabajorelacionado de operadores de mutación en JavaScript y Node.js, y propongo Mutode,una herramienta de código abierto que aprovecha el ecosistema de paquetes denpm para realizar pruebas de mutación en aplicaciones de JavaScript y Node.js.Finalmente, evalúo empíricamente la efectividad de Mutode ejecutándolo sobre los20 módulos con más dependientes de npm, que tengan pruebas automáticas.

vi

Acknowledgements

We would like to thank Camilo Escobar and the Master Students from the AutomatedTesting course at Universidad de los Andes for their help testing Mutode.

vii

Contents

1 Introduction 1

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1.1 Objectives: Generic JavaScript and Node.js Mutation TestingTool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1.2 Expected results . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.3 Activities Schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Related work and existing tools 5

2.1 Prior work in other languages/ecosystems . . . . . . . . . . . . . . . 5

2.1.1 Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.2 C/C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1.3 .NET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1.4 Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.1.5 Ruby . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.1.6 PHP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.1.7 Android . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.2 Prior work in JavaScript and Node.js . . . . . . . . . . . . . . . . . . 10

2.2.1 Mutandis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.2.2 Stryker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.2.3 Mutant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.3 Differences between Mutode and previous tools . . . . . . . . . . . . 11

3 Mutode 15

3.1 Implemented Mutation Operations . . . . . . . . . . . . . . . . . . . 15

3.1.1 Mutators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.2 Mutode Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.2.1 Mutants Creation . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.2.2 Mutants Execution . . . . . . . . . . . . . . . . . . . . . . . . 25

3.3 Tool Usage and Extensibility . . . . . . . . . . . . . . . . . . . . . . . 26

3.4 Published Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.5 Development Environment . . . . . . . . . . . . . . . . . . . . . . . . 28

ix

4 Evaluation 294.1 Study Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294.2 Study Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.2.1 Execution Results . . . . . . . . . . . . . . . . . . . . . . . . . 30

5 Conclusion 415.1 Achieved Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . 415.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

5.3.1 New Features . . . . . . . . . . . . . . . . . . . . . . . . . . . 425.3.2 Existing Features . . . . . . . . . . . . . . . . . . . . . . . . . 42

Bibliography 43

x

1Introduction

JavaScript and Node.js applications, and their ecosystem, have been growing steadilyin the past few years thanks to the possibility of reusing code that runs both infront-end (web browser, mobile applications) and in back-end (server), and theeasiness to use published CommonJS-defined[40] and ECMAScript modules[12].This growth can be seen in comparison to other programming languages and moduleregistries [8], and in the statistics [48] published by the Chief Operating Officerof npm — the package manager for Node.js —, which was built as an open sourceproject in 2009 and later registered as a company in 2014. It is also an ever growingregistry of open source modules built for the JavaScript, Node.js and Web ecosystems[34].

The over 700,000 modules hosted in npm are downloaded more than 5 billiontimes per week [48]. More and more software is published in npm every day,representing an opportunity to share code and solutions, but also to share bugs andfaulty software. A recent survey of over sixteen thousand developers shows that 77%of respondents are concerned about the security in open source modules, and 52%of respondents are not satisfied with the tools they have got to evaluate open sourcecode security and quality[18]. As a reaction from the practitioners, current stateof the practice for testing, used in the most popular modules, includes unit testing,and an increasing number of developers are opting in using test code coveragetools such as Istanbul[5]. However, using unit testing frameworks and measuringcoverage is not enough by itself, because the quality of test suites needs also to bemeasured. Thus, here is an opportunity for going beyond test code coverage andtake advantage of mutation testing, which goes a step above and measures testsuites’ effectiveness.

Mutation testing is a technique in which faults (mutants) are inserted into a appli-cation to assess its test suite effectiveness [21]. It works by inserting mutants andrunning the applications test suite to identify if the mutants are detected (killed) ornot (survived). Although computationally expensive, it has proven to be an effectivemethod to assess applications’ test suites [21]. Recent papers have shown thatstatement and branch coverage are not enough for measuring test suite effectiveness[1, 24] and mutation testing is more effective than data flow testing for detectingfaults in applications [38].

1

1.1 Motivation

While for other types of applications there are well known and maintained mutationtesting frameworks, in the case of JavaScript and Node.js applications, practitionerslack an easy to use and well-maintained applications that allows mutation testing.Consequently, there is a big opportunity for creating a generic mutation testing toolthat targets the latest practices for software development in JavaScript and Node.js,for both back-end and front-end code; one that works with any type of testingframework; and one which requires zero or minor configuration.

With that in mind, we propose the following objectives

1.1.1 Objectives: Generic JavaScript and Node.js MutationTesting Tool

• Build a generic framework that leverages npm package.json definition usingthe npm test command to execute applications test suites.

• Implement a catalog of mutation operators based on PIT, prior work, personalexperience and new research.

• Empirically evaluate Mutode on the top 20 most depended upon npm modules.

1.1.2 Expected results

• A working and test mutation testing tool published in npm as a CLI tool withthe following features:

– Allow mutants tests to be executed in parallel, harnessing full CPU andreducing execution time.

– Unit tests to assess the tools core and mutation operators.

– Execution reports.

– Documentation of the tool and its operators.

• An implemented catalog of mutation operations grouped in mutators.

2 Chapter 1 Introduction

• Empirical benchmarking of the tool executed on the top 20 most dependedupon npm modules.

1.2 Methodology

The project was divided in three phases:

• Phase I

– Implement the basic generic framework core using basic mutation opera-tors applicable to any programming language.

– Search and review prior work in the topic.

– Implement parallel tests execution.

– Implement basic execution configuration.

• Phase II

– Implement all generic operators based on PIT.

– Search, review and implement previously defined JavaScript and Node.jsspecific mutation operators.

– Define and implement new Javascript and Node.js specific mutationoperators.

– Create tests to assess the framework and operators efficacy.

• Phase III

– Empirically evaluate the framework on the top 20 npm modules withautomated test suites.

– Generate execution reports.

Project wide, everything will be documented during the implementation process, thecode and the results obtained in tests and the empirical evaluation.

1.2 Methodology 3

Table 1.1: Activities Schedule

Activity/Week 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16Generic framework XPrior work X X X X XParallel execution X X XExecution configura-tion

X X

PIT Generic Operators X X XJS and Node.js opera-tors in prior work

X X X

New JS and Node.js Op-erators

X X X

Tests X X X X X XEmpirical evaluation X XDashboard and reports X XExecution state X XDocumentation X X X X X X X X X X X X X X X X

1.3 Activities Schedule

For the development of the project, the schedule described by the table 1.1 wasfollowed during the first semester of 2018.

1.4 Results

Mutode was successfully developed during the semester. The tool is published as annpm CLI module, available at [41], capable of receiving the file paths to mutate,concurrency parameter to set the number of mutant execution to run in parallel andthe list of mutators to run. Its code is hosted in GitHub and is freely available underthe MIT License. A README, user guide and video showing Mutode execution canbe found in the project repository [42].

A Tool Demonstration Paper was submitted and accepted in the ACM SIGSOFTInternational Symposium on Software Testing and Analysis [43].

Due to the available time and the lengthy process of running the benchmarking onthe top 20 most depended upon npm modules, the dashboard was not completed.Also, no execution state is currently saved, as any change any statement couldlead to a mutant surviving or being killed in another piece of code. Therefore, thisfunctionality originally proposed as an expected result was discarded.

4 Chapter 1 Introduction

2Related work and existing tools

In this chapter, I present prior work in mutation testing and existing mutation testingtools available for the different programming languages. A summary is available inTable 2.1.

2.1 Prior work in other languages/ecosystems

2.1.1 Java

PIT [6]

http://pitest.org/

PIT is a state of the art mutation testing system for Java and the jvm. It is activelydeveloped and supports. It is fast, easy to use and integrates with modern testand build tooling. It creates mutations based on the tests line coverage and runsthe specific tests which cover the code where the mutation was made, effectivelyreducing the time required to run. Its latest version was published in May 2018.

Javalanche [45]

http://javalanche.org/

Javalanche is framework for mutation testing of Java programs. It addresses boththe problems of efficiency and equivalent mutants. It is built for efficiency fromthe ground up, manipulating byte code directly and allowing mutation testing ofprograms that are several orders of magnitude larger than earlier research subjects.It addresses the problem of equivalent mutants by assessing the impact of mutationson dynamic invariants: The more invariants impacted by a mutation, the more likelyit is to be useful for improving test suites. Its latest version was published in 2012.

5

http://pitest.org/

http://javalanche.org/

Major [23]

http://mutation-testing.org/

Major is a mutation frame work for Java. It provides the following three components:(i) Compiler-integrated mutator for easy and fast fault seeding; (ii) Domain specificlanguage to configure the mutation process in detail; and (iii) Mutation analyzer forJUnit tests. Its latest version was published in 2017.

muJava [28]

https://cs.gmu.edu/~offutt/mujava/

µJava (muJava) is a mutation system for Java programs. It automatically gener-ates mutants for both traditional mutation testing and object oriented, class-level,mutation testing. Its latest version was published in 2016.

Bacterio [30]

Bacterio is a Java mutation testing tool that automates the tasks to perform mutationanalyses and that implements a set of mutation techniques that reduce the costs ofmutation and the execution mutant time drastically. It only works in the WindowsOperating System and is distributed as a rar compressed file . Its latest version waspublished in 2012.

Judy [29]

http://www.mutationtesting.org/

Judy is a mutation testing tool for Java applications that operates at the source codelevel and supports Object Oriented mutation operators. It has a high performancemutation testing process and integrates with various development environment tools.Its latest version was published in 2017.

6 Chapter 2 Related work and existing tools

http://mutation-testing.org/

https://cs.gmu.edu/~offutt/mujava/

http://www.mutationtesting.org/

2.1.2 C/C++

MILU [22]

https://github.com/yuejia/Milu

Milu is an efficient and flexible C mutation testing tool designed for both first orderand higher order mutation testing. Its distribution consists of the source files, whichmust be compiled for each platform. Its latest version was published in April, 2018.

Mull [9]

https://github.com/mull-project/mull

Mull is an open-source tool for mutation testing based on the LLVM framework. Mullworks with LLVM IR, a low-level intermediate representation, to perform mutations,and uses LLVM JIT for just-in-time compilation. Its latest version was published inMay 2018.

Mutate++ [27]

https://github.com/nlohmann/mutate_cpp

Mutate++ is a mutation test environment for C++ programs built in Python as aweb app that runs locally. Its latest version was published in 2017.

2.1.3 .NET

VisualMutator [10]

https://visualmutator.github.io/web/

Visual Mutator is a mutation testing tool integrated within the .NET programmingenvironment in the Visual Studio IDE. Its latest version was published in 2016.

2.1 Prior work in other languages/ecosystems 7

https://github.com/yuejia/Milu



https://visualmutator.github.io/web/

NinjaTurtles [33]

http://www.mutation-testing.net/

NinjaTurtles is an easy to use open source .NET mutation testing framework thatworks for C#, Visual Basic/VB.NET and other .NET code. It integrates with theexisting unit test framework. Its latest version was published in 2015.

NinjaTurtlesMutation [7]

https://github.com/criteo/NinjaTurtlesMutation

NinjaTurtlesMutation is a fork of NinjaTurtles that aims to provide a fast mutationtesting tool for codebases using NUnit v2.6.4. Its latest version was published in2016.

2.1.4 Python

Cosmic Ray [3]

https://github.com/sixty-north/cosmic-ray

Cosmic Ray is a tool for performing mutation testing on Python 3 code. Its a veryyoung tool with a very active development. Its latest version was published in May,2018.

MutPy [13]

https://github.com/mutpy/mutpy

MutPy is a mutation testing tool for Python 3.3+ source code. MutPy supportsstandard unittest module, generates YAML/HTML reports and has colorful output. Itapplies mutations on AST level. It supports high order mutations (HOM) and codecoverage analysis to improve the mutation testing process. Its latest version waspublished in 2017.






Mutmut [17]

https://github.com/boxed/mutmut

Mutmut is a young Python mutation testing library/tool that aims to be usable as aprogram directly, but also to have an easy to use API. Its actively being developed.Its latest version was published in May 2018.

2.1.5 Ruby

Mutant [44]

https://github.com/mbj/mutant

Mutant is a mutation testing tool for Ruby. Mutant supports ruby version >= 2.1and rspec tests. Support for JRuby is planned. It should also work under any Rubyengine that supports POSIX-fork(2) semantics. Mutations are applied in the ASTlevel. Its latest version was published in February, 2018.

2.1.6 PHP

Infection [39]

https://infection.github.io/

Infection is a PHP mutation testing framework based on AST (Abstract Syntax Tree)mutations. It works as a CLI tool and can be executed from your projects root, similarto the proposed workflow of Mutode. It currently supports PHPUnit and PhpSpectest frameworks, requires PHP 7+ and xDebug/phpdbg installed. Its latest versionwas published in May, 2018.

HumBug [4]

https://github.com/humbug/humbug

Humbug is a Mutation Testing framework for PHP. Its latest version was publishedin 2017 and has been deprecated in favor of Infection.

2.1 Prior work in other languages/ecosystems 9



https://infection.github.io/


2.1.7 Android

MDroid+ [26]

https://www.android-dev-tools.com/mdroid/

MDroid+ is an Android Mutation tool that aims to help mobile application developersand testers improve the quality of their test suites. Its project empirically derived aset of 38 Android-sepcific mutation operators which can be automatically seeded intoa target application and analyzed on a set of test cases. Currently it only generatesthe mutants, but execution is being developed. It was published in 2017.

µDroid [19]

https://github.com/Yuan-W/muDroid

MuDroid is a mutation testing tool for Android testing in integration level builtin Python 2.7+. It requires Pillow and pexpect installed. Its latest version waspublished in 2016.

2.2 Prior work in JavaScript and Node.js

In the case of JavaScript and Node.js apps, practitioners lack an easy to use andwell-maintained tool that allows mutation testing. However, there is significantresearch about JavaScript testing, bugs and their patterns [36, 14, 37, 11].

The following tools were found:

2.2.1 Mutandis

https://github.com/saltlab/mutandis/

Mutandis is a framework built in Java focused on client-side JavaScript applications.It utilizes Mozillas Rhino AST parser to mutate the application and intercepts networkrequests to analyze their content. If a test suite is available for a given application,its used to profile it. This tool, however, wasnt made for Node.js applications andnpm modules. Besides the initial work 4 years ago, no work or maintenance has


https://www.android-dev-tools.com/mdroid/

https://github.com/Yuan-W/muDroid

https://github.com/saltlab/mutandis/

been done to the library since. There is also a lack of documentation in its GitHubrepository and instructions on how to use it [32].

2.2.2 Stryker

https://stryker-mutator.io/

Stryker is an open source framework available at GitHub. It offers over 30 mutationoperations and allows running tests with Karma[25], Mocha[16], among otherframeworks through a custom built plug-in. It requires configuration and does notsupport other test frameworks besides Karma and Mocha without a plugin. Thisrepresents a room for improvement for a new tool that works out of the box withany test suite [20].

2.2.3 Mutant

https://github.com/benhartley/mutant

Mutant is a JavaScript mutation testing tool which has 7 mutation operations, islimited to test frameworks with TAP[46] output, and uses babel[47] to parse andregenerate AST; however it is unmaintained, only five versions were released [15].

2.3 Differences between Mutode and previoustools

Mutode focus is to be as simple and conventional as possible, allowing developers tostart using mutation testing with zero o minor configuration, while working withany testing framework.

Comparing Mutode with Mutandis, the key difference is that Mutandis allows fron-tend execution of mutants, intercepting network request and analyzing their content,while Mutode doesn’t. Mutandis was not made to be used within the npm ecosystemand its tools. Another difference is the AST parser, Mutandis uses Mozilla’s Rhinoand Mutode uses Babylon (Babel’s parser). On the development side, Mutandis wasbuilt in Java, while Mutode is built in Node.js. Both work on any environment thatruns Java.

2.3 Differences between Mutode and previous tools 11

https://stryker-mutator.io/


Table 2.1: Mutation Frameworks & Tools

Framework/Tool Language/Ecosystem Mutation Level Open SourcePIT Java Bytecode/Reflection YesJavalanche Java Bytecode YesMajor Java Bytecode YesMuJava Java Bytecode/Source YesBacterio Java Bytecode/Source NoJudy Java Source NoMilu C AST YesMull C/C++/Objective-C Source YesMutate++ C++ Source YesVisualMutator .NET Assembly YesNinjaTurtles .NET Assembly Yes, ArchivedNujaTurtlesMutation .NET Assembly YesCosmic Ray Python AST YesMutPy Python AST YesMutmut Python FST YesMutant Ruby AST YesInfection PHP AST YesHumBug PHP AST YesMdroid+ Android AST & Text YesmuDroid Android Smali YesMutandis JavaScript AST & Network YesStryker JavaScript AST YesMutant JavaScript AST Yes


Mutode approach on mutation operators is more broad, covering many Mutandisoperators within fewer Mutode operators. For example, while in Mutandis thereis an operator to remove break statements from switch cases, and remove returnstatements, in Mutode there is a line statements remover, which covers every single-line statement in any script or application. Another example of this is that, while inMutandis literal values of a condition are removed, and local or global values arechanged, in Mutode exists three mutators that mutate every boolean, numeric andboolean literal there is.

Comparing Mutode with Stryker, the key difference is the target. Mutode idea becameto be because of this: Stryker doesn’t work out of the box with any testing framework,tt only comes with support for Mocha [16] and Karma [25] out of the box, whileoffering supporting others through plugins. This, however, comes with a benefit,each test case knows exactly which lines of code it tests and therefore optimizationscan be made to only run certain test cases for certain mutations, reducing drasticallythe execution time. Another important difference is that Stryker supports and is builtwith TypeScript [31], a typed superset of JavaScript. Both run on any environmentthat supports Node.js, JavaScript.

Strykers operators are all covered by Mutode except for the Array object declaration.Mutode only supports removing elements from array literals and not the objectnotation. And while Stryker remove whole conditions or every element from arrays,Mutode generates mutants removing one by one, representing a closer escenario toreality, where you miss one value of an array, intead of all of the values.

2.3 Differences between Mutode and previous tools 13

3Mutode

With the purpose of creating an useful tool that targets a wide range of JavaScriptand Node.js applications, Mutode takes into account the following design decisions:(i) Mutode does not perform operations at the AST-level in order to keep complianceof the existing code with any linter the developer uses within the application ormodule; (ii) Mutode uses the npm test command in order to run the tests againsteach mutant, thus, no configuration or plug-ins are needed to execute existing tests;(iii) the mutants are executed in parallel by workers that consume tasks from aqueue to reduce the time required to test mutants and harness the CPU power.

3.1 Implemented Mutation Operations

In order to create an effective mutation testing tool for JavaScript and Node.js, wepropose a catalog of 43 mutation operations grouped in 16 mutators. Our operationscatalog is built on top of operations previously defined for (i) Java applications [6],(ii) JavaScript applications [32], (iii) new operations proposed by us based onour experience developing JavaScript/Node.js applications, (iv) and open sourcereported bugs in npm modules. The Mutode mutators and operations are explainedin detailed next and finally summarized in Table 3.6. It is worth noting that we call“mutator” to a group of mutation operations (e.g., Numeric literals), and “operations”to specific mutations applied to the source code (e.g., replace a true boolean literalwith false), in the same way that PIT[6] does.

3.1.1 Mutators

Boolean Literals Mutator

The boolean literals mutator mutates the true and false boolean literals the theircounterpart.

For example, the following code:

const myBoolean = true

15

Would be mutated to:

const myBoolean = false

Conditionals Boundary Mutator

The conditionals boundary mutator replaces the relational operators <, <=, >, >=with their boundary counterpart as listed in Table 3.1.

Table 3.1: Conditionals Boundary Mutator Operations

Original Mutant(s)< <=<= <> >=>= >


if (a < b) {...

}


if (a <= b) {...

}

Increments Mutator

The increments mutator mutates increments (i++) and decrements (i–) statementsto their counterparts.


const a = b++


const a = b--

16 Chapter 3 Mutode

Invert Negatives Mutator

The invert negatives mutator makes negative numbers positive.


const a = -10


const a = 10

Math Mutator

The math mutator mutates math and bitwise operators to their inverse. The modulusoperator % and the exponential operator ** are mutated to multiplication * as listedin Table 3.2.

Table 3.2: Math Mutator Operations

Original Mutant(s)+ -- +* // *% *& || &ˆ |« »» «** *


const a = b + c


const a = b - c

Negate Conditionals Mutator

The negate conditionals mutator mutates conditionals to their inverse as listed inTable 3.3.

3.1 Implemented Mutation Operations 17

Table 3.3: Negate Conditionals Mutator Operations

Original Mutant(s)== !=!= ===== !==!== ===<= >> <=>= << >=


if (a === b) {...

}


if (a !== b) {...

}

Numeric Literals Mutator

The numeric literals mutator creates different mutants for numeric literals as listedin Table 3.4.

Table 3.4: Numeric Literals Mutator Operations

Original Mutant(s)

a

a - 1a + 10 (If not previously 0)Random value


const PORT = 5432

Would generate the following mutants:

const PORT = 5431

const PORT = 5433

const PORT = 0

const PORT = 91284

18 Chapter 3 Mutode

Remove Array Elements Mutator

The remove array elements mutator removes single elements from arrays, generatingarray.length mutants.


const myArray = [a, b]


const myArray = [a]

const myArray = [b]

Remove Conditionals Mutator

The remove conditionals mutator replaces conditional statements with both literalboolean values, generating two mutants.


if (a. includes (b)) {...

}


if (true) {...

}

if (false ) {...

}

Remove Function Call Arguments Mutator

The remove function call arguments mutator removes single arguments from afunction call, generating arguments.length mutants.


myFunction (a, b)



myFunction (a)

myFunction (b)

Remove Function Parameters Mutator

The remove function parameter mutator removes single parameters from a functiondeclaration, generating parameters.length mutants. This is an easy to catchmutant by linters, as static analysis is enough to notice if a variable is being usedinside a function when it no longer exists.


function myFunction (a, b) {...

}


function myFunction (a) {...

}

function myFunction (b) {...

}

Remove Functions Mutator

The remove functions mutator comments (In order to keep code line numbers intact)functions. This is an easy to catch mutant by linters, as static analysis is enough tonotice if a function being called when it no longer exists.


function myFunction (a, b) {...

}


// function myFunction (a, b) {// ...//}

20 Chapter 3 Mutode

Remove Lines Mutator

The remove lines mutator comments (In order to keep code line numbers intact)single line statements. This might be an easy to catch mutant by linters, as staticanalysis is enough to notice if anything declared is later used in the code. It generatesas many mutants as single line valid statements there are.


...let myVariable = 'Hello world '...


...// let myVariable = 'Hello world '...

Remove Object Properties Mutator

The remove object properties mutator removes single properties from objects, gen-erating properties.length mutants. If object is spread in multiple lines, linesbelonging to the property are commented.


const myArray = {a: 10,b: 20

}


const myArray = {// a: 10,

b: 20}

const myArray = {a: 10,

// b: 20}


Remove Switch Cases Mutator

The remove switch cases mutator comments single cases from switch statements,generating cases.length mutants.


switch (value ) {case 'a':

...break

case 'b:...break

}


switch (value ) {// case 'a ':// ...// break

case 'b:...break

}

switch (value ) {case 'a':

...break

// case 'b:// ...// break}

String Literals Mutator

The string literals mutator creates different mutants for string literals as listed inTable 3.5.

Table 3.5: String Literals Mutator Operations

Original Mutant(s)

My string Empty string (if not previously empty)Random string


22 Chapter 3 Mutode

const NAME = 'Diego '


const NAME = 'a84mk1ixn3 '

const NAME = ''

Table 3.6: Mutators and Operations in Mutode

Mutator Operation Replacement or ActionOriginal Mutant(s)

Boolean Literals true falsefalse true

Conditionals Boundary

< <=<= <> >=>= >

Increments ++ –– ++

Invert Negatives -a a

Math

+ -- +* // *% *& || &ˆ |« »» «** *

Negate Conditionals

== !=!= ===== !==!== ===> <=>= << >=<= >

Numeric Literals a

a + 1a - 1Random value0

Remove Array Elements [a, b] [a][b]

Remove Conditionals a < 10 truefalse

Remove Function Call Arguments func(a, b) func(a)func(b)

Remove Function Parameters func(a, b) {} func (a) {}func (b) {}

Remove Functions Removes functions by commenting themtRemove Lines Removes single line statements by commenting

them

Remove Object Props {a:1,b:2} {a:1}{b:2}

Remove Switch Cases

switch(v) {case 1: ...case 2: ...}

switch(v) {case 1: ... }switch(v) {case 2: ... }

String Literals My string Empty stringRandom string


3.2 Mutode Execution

The workflow in Mutode is described as follows:

When an instance of Mutode is created, the index.js file and the lib/ folder of theapplication or module are analyzed (in the case the user does not provide a pathor array of paths) looking for files that could be mutated by Mutode; if no files arefound, then Mutode exits with an error.

Before starting the mutation process, Mutode deletes previous copies of the applica-tion (if any), creates a new copy of the application and times the test suite executionwith no mutants (i.e., the test suite is only executed on the original application) toset a timeout value that will discard mutants which execution do not exit after 2.5xthe expected timing; if the test suite does not pass (i.e., process exits with a codedifferent than 0), then Mutode exits with an error. After timing the test suite, Mutodecreates extra copies for each execution thread; the copies are used as the base forthe mutants generated by each thread.

For each file to be mutated an empty queue is created where mutants will be addedby mutators, the file content is read and its AST is parsed by babylon[2] (the ASTparser for babel[47]). Afterwards, mutators are executed to generate mutants fora given file and add them to the queue; a mutant in the queue is then processed(i.e., the test suite is executed on the mutant) by a free worker, each having itsown copy of the application, and logging the mutation results (i.e., killed, survivedor discarded) to the console. Finally once the queue is empty, a mutated file isreverted to the original version so no mutants of the file are left in subsequentmutant executions of other files’ mutants.

At the end of the process, copies of the application are deleted and execution timeis logged. A consolidated report is shown, summarizing mutants created, killed,survived and discarded, as well as the mutants coverage, calculated as:

Killed mutantsGenerated mutants - discarded mutants

× 100

3.2.1 Mutants Creation

Given the catalog of operations listed in Table 3.6, Mutode generates a mutantfor each line or set of lines where each mutator identifies a potential locationfor applying an operation. This process is performed by first using the AST andidentifying statement types that match those in which each mutator could create

24 Chapter 3 Mutode

a new mutant, and then using text comparisons with each operation that can beapplied. With the AST information of the statement, the operation is applied intothe source code of the file swapping exact text without affecting the code’s format,commenting lines without affecting the original code’s lines number or removingcode from the an exact line in the source code. A mutant ID is assigned and theinformation is logged to a mutants log to offer the developer a way of knowing whatwas changed. A full copy of the file with the mutant is then pushed to the tasksqueue.

3.2.2 Mutants Execution

Each mutant in the queue is inserted in one of the available workers’ copy of theapplication’s source. A child process running npm test is then launched on thecopy, which captures the exit code and marks it as either survived, if the exit codewas 0, or killed if else. In the case where the child process does not exit, a timeout,set to 2.5x the original test suite timing, is set to discard the mutant and kill thechild process. Result is logged to the console and used worker is freed for executingnext mutant available in the queue.

There are various reasons as to why a mutant could be discarded but the main onesare the following:

• The callback argument in a function call is removed, or the callback parameteris removed from a function declaration, therefore further execution does nothappen.

• A callback call or a returned function call is removed, therefore further execu-tion does not happen.

• A timeout timer’s value is modified by the numeric literals mutator to a largernumber than the mutant’s execution timeout.

• A timer or child process unreferencing is removed and process never dies, evenif execution is completed.

• An state variable is switched or removed, blocking the application’s continuity.

3.2 Mutode Execution 25

3.3 Tool Usage and Extensibility

Mutode is published as an npm module, available at [41] and intended to be used asa command-line utility in which the developer can decide the specific set of mutators,to be applied in the generation of mutants and execution, as well as a parameter toset the concurrency of mutants execution and a parameter to set the files to mutate.A README, user guide and video showing Mutode execution can be found in theproject repository[42].

3.4 Published Versions

The following list describes the published versions of Mutode in the npm registryduring the project with their changes:

• v1.4.1 (May 14, 2018): Ignore console/debug logs for functions call argumentsmutator. (Latest)

• v1.4.0 (May 13, 2018): Update dependencies. Update docs.

• v1.3.1 (April 26, 2018): Fix start execution timing.

• v1.3.0 (April 24, 2018): Add AppVeyor (Windows Testing). Improve Windowserror handling.

• v1.2.2 (April 20, 2018): Update docs

• v1.2.1 (April 20, 2018): Catch errors of bin. Improve discarding timeout.

• v1.2.0 (April 20, 2018): Show execution time.

• v1.1.1 (April 20, 2018): Add coveralls test coverage reporting.

• v1.1.0 (April 19, 2018): Force kill child processes and include dot files insubdirs.

• v1.0.3 (April 18, 2018): Improve AST parsing and handle errors.

• v1.0.2 (April 16, 2018): Change Travis badge URL.

• v1.0.1 (April 13, 2018): Update docs and add Travis CI badge.

26 Chapter 3 Mutode

• v1.0.0 (April 13, 2018): Generalize boolean, numeric and string mutators.Improve copying of source files.

• v0.2.6 (April 8, 2018): Run in shell for windows

• v0.2.5 (April 8, 2018): Add support for Windows

• v0.2.4 (April 8, 2018): Fixes.

• v0.2.3 (April 8, 2018): Improve mutators. Add remove functions mutator,remove array elements mutator and remove object properties mutator.

• v0.2.2 (April 8, 2018): Ignore block statements and improve tests.

• v0.2.1 (April 8, 2018): Don’t ignore bin folder.

• v0.2.0 (April 8, 2018): Add remove switch cases mutators. Remove debug ofAST nodes.

• v0.1.8 (April 5, 2018): Fix available mutator options in CLI help.

• v0.1.7 (April 5, 2018): Add string values mutator and switch cases mutator.

• v0.1.6 (April 4, 2018): Don’t try to copy extra copies if concurrency is one.

• v0.1.5 (April 4, 2018): Remove unneeded console log.

• v0.1.4 (April 4, 2018): Update docs.

• v0.1.3 (April 4, 2018): Add boolean and numeric values mutator. Improvedoc.

• v0.1.2 (March 31, 2018): Added invert negatives mutator. Fixes.

• v0.1.1 (March 27, 2018): Minor fixes of tabulations in logging.

• v0.1.0 (March 27, 2018): Improve documentation, README and configura-tion.

• v0.0.8 (March 26, 2018): Improve logging, clean previous executions. Addnegate conditionals mutator and remove conditionals mutator.

3.4 Published Versions 27

• v0.0.7 (March 16, 2018): Improve tests and concurrency.

• v0.0.6 (March 15, 2018): Use Babylon AST Parser.

• v0.0.5 (March 8, 2018): Moved increments to its own mutator. Added condi-tional boundaries mutator and debugging. Improved tests.

• v0.0.4 (February 24, 2018): Added parallelization, tests, JSDoc documentationand Travis CI.

• v0.0.3 (February 12, 2018): Use .npmignore to publish only needed files.

• v0.0.1 (February 12, 2018): Initial module with core functionality.

3.5 Development Environment

The development was done in a MacBook Pro 13-inch with Mac OS X 10.13.4.Node.js version 8.9.4 and npm verison 5.6.0 were used during the development forthe execution and testing of Mutode. WebStorm 2018 was used as the default IDE tocode.

28 Chapter 3 Mutode

4Evaluation

To evaluate Mutode, we conducted a study with the following goals:

• Test the applicability of Mutode in top npm modules

• Fix any execution bugs that arise during the top npm modules mutationanalysis

• Improve the core of Mutode and mutators’ operations

To accomplish this, we executed Mutode on the top 20 most depended npm moduleswith working automated test suites according to the npm registry [35].

4.1 Study Environment

The study was executed in a MacBook Pro 15" with a 2.8GHz quad-core 7th-generation Intel Core i7 processor, 16GB RAM and SSD storage. It’s importantto mention this as execution in the development environment would take at leasttwice the time.

4.2 Study Results

Mutation testing results are listed below and are summarized in Table 4.1. Tosuccessfully test Mutode on 20 modules, it was needed to try execute it in the top 33modules.

The following modules were tested but Mutode was not able to be used on them:

• [email protected] could not be tested because of excessive memory consumption.

• [email protected], [email protected] [email protected] and [email protected] not be tested because they did not include npm test command on theroot of the project nor an easy way to include it.

29

• bluebird @3.5.1 could not be tested because it has a restriction that its testsbe must run in a folder called ’bluebird’, and Mutode copies of the applicationare named as .mutode-{instance ID}-{copy number}.

• [email protected] and [email protected] could not be tested because their testswere failing.

• [email protected] was not included as all of the mutants survived and therewas no evidence its tests target the source.

• [email protected] was not included as it is built with TypeScript and has a buildstep.

• [email protected] was not included as it uses dotfiles for configuration, which arecurrently not copied by Mutode and needed in the Grunt task definition.

4.2.1 Execution Results

1. request

• Module Information

– Version: 2.87.1

– Test Cases: 1577

– Test Suite Timing: 28.61s

• Execution Command: mutode -c 7 lib *.js• Mutants Execution Timing: 4h 26m 46s• Mutants Execution Results

– Generated: 4247

– Killed: 4106

– Survived: 78

– Discarded: 63

– Mutant Coverage: 98.14%

2. chalk


– Version: 2.4.1

– Test Cases: 56


• Execution Command: mutode -c 2 index.js• Mutants Execution Timing: 17m 30.2s

30 Chapter 4 Evaluation

• Mutants Execution Results

– Generated: 322

– Killed: 284

– Survived: 35

– Discarded: 3


3. async


– Version: 2.6.1

– Test Cases: 516


• Execution Command: mutode -c 7 lib• Mutants Execution Timing: 2h 7m 46s• Mutants Execution Results

– Generated: 3636

– Killed: 3196

– Survived: 296

– Discarded: 144


4. express


– Version: 4.16.3

– Test Cases: 858


• Execution Command: mutode -c 7 lib index.js• Mutants Execution Timing: 43m 3s• Mutants Execution Results

– Generated: 3901

– Killed: 2335

– Survived: 0

– Discarded: 1566

– Mutant Coverage: 100%

4.2 Study Results 31

5. commander


– Version: 2.15.1

– Test Cases: 44


• Execution Command: mutode -c 7 index.js• Mutants Execution Timing: 44m 26.7s• Mutants Execution Results

– Generated: 1397

– Killed: 1016

– Survived: 373

– Discarded: 8


6. moment


– Version: 2.22.1



• Execution Command: mutode -c 6 src/lib src/moment.js• Mutants Execution Timing: 14h 22m 25s• Mutants Execution Results

– Generated: 10944

– Killed: 5175

– Survived: 1763

– Discarded: 4006


7. debug


– Version: 3.1.0

– Test Cases: 4


• Execution Command: mutode• Mutants Execution Timing: 4m 50.1s• Mutants Execution Results

– Generated: 1304

– Killed: 114

– Survived: 1186


– Discarded: 4


• Note: npm test command was created as the project uses a Makefile.

8. underscore


– Version: 1.9.0



• Execution Command: mutode -c 7 underscore.js• Mutants Execution Timing: 49m 41.2s• Mutants Execution Results

– Generated: 2852

– Killed: 2256

– Survived: 523

– Discarded: 73


9. colors


– Version: 1.2.5

– Test Cases: 32


• Execution Command: mutode -c 7 lib/• Mutants Execution Timing: 4m 34.1s• Mutants Execution Results

– Generated: 2258

– Killed: 449

– Survived: 1809

– Discarded: 0


10. body-parser


– Version: 1.18.3

– Test Cases: 231



• Execution Command: mutode -c 7 lib index.js• Mutants Execution Timing: 6m 17.3s• Mutants Execution Results

– Generated: 1095

– Killed: 949

– Survived: 106

– Discarded: 40


11. glob


– Version: 7.1.2



• Execution Command: mutode -c 7 common.js glob.js sync.js• Mutants Execution Timing: 6h 25m 24s• Mutants Execution Results

– Generated: 2117

– Killed: 1360

– Survived: 461

– Discarded: 296


12. uuid


– Version: 3.2.1

– Test Cases: 17


• Execution Command: mutode -c 7 lib *.js• Mutants Execution Timing: 7m 39.2s• Mutants Execution Results

– Generated: 3338

– Killed: 2920

– Survived: 396

– Discarded: 22



13. classnames


– Version: 2.2.5

– Test Cases: 55


• Execution Command: mutode -c 7 index.js bind.js dedupe.js• Mutants Execution Timing: 46.5s• Mutants Execution Results

– Generated: 262

– Killed: 184

– Survived: 73

– Discarded: 5


14. minimist


– Version: 1.2.0

– Test Cases: 140


• Execution Command: mutode -c 7 index.js• Mutants Execution Timing: 3m 27.8s• Mutants Execution Results

– Generated: 514

– Killed: 482

– Survived: 25

– Discarded: 7


15. yargs


– Version: 11.1.0

– Test Cases: 511

– Test Suite Timing: 15.94

• Execution Command: mutode -c 6 index.js lib yargs.js• Mutants Execution Timing: 3h 47m 27.9s• Mutants Execution Results

– Generated: 4586

– Killed: 3728

– Survived: 714

– Discarded: 144



16. through2


– Version: 2.0.3

– Test Cases: 44


• Execution Command: mutode -c 7 through2.js• Mutants Execution Timing: 17.3s• Mutants Execution Results

– Generated: 101

– Killed: 79

– Survived: 22

– Discarded: 0


17. yeoman-generator


– Version: 2.0.5

– Test Cases: 181

– Test Suite Timing: 15.79

• Execution Command: mutode -c 7 lib• Mutants Execution Timing: 43m 54.3s• Mutants Execution Results

– Generated: 1775

– Killed: 1769

– Survived: 0

– Discarded: 6

– Mutant Coverage: 100%

18. q


– Version: 1.5.1

– Test Cases: 243


• Execution Command: mutode -c 7 q.js queue.js• Mutants Execution Timing: 53m 26.3s• Mutants Execution Results

– Generated: 2181

– Killed: 1273

– Survived: 859

– Discarded: 49



19. axios


– Version: 0.18.0

– Test Cases: 30


• Execution Command: mutode -c 7 lib/**/*.js lib/*.js• Mutants Execution Timing: 23m 27.7s• Mutants Execution Results

– Generated: 2020

– Killed: 1569

– Survived: 402

– Discarded: 49


20. cheerio


– Version: 0.22.0

– Test Cases: 598


• Execution Command: mutode -c 7 lib/ index.js• Mutants Execution Timing: 21m 10.5s• Mutants Execution Results

– Generated: 2754

– Killed: 2424

– Survived: 292

– Discarded: 38


• Note: Makefile was modified to not lint, as it requires a dotfile configuration file, which iscurrently not copied by Mutode.

On average 2580 mutants were generated for each module, and the test suites in themodules achieved an average mutation coverage of 77.3%. The highest mutationcoverage was achieved by [email protected] and [email protected] at 100%and the lowest by [email protected] at 8.77%. Note that the latter only has four testscases in its test suite, and the test cases are oriented to the very core functionalityassuring it does not throw errors and does not repeat its logging when called.


Table 4.1: Study results on npm modules

Module & Version Tests Timing Execution Time Generated Killed Survived Discarded [email protected] 1577 28.61s 4h 26m 46s 4247 4106 78 63 98.14%[email protected] 56 6.75s 17m 30.2s 322 284 35 3 89.03%[email protected] 516 19.27s 2h 7m 46s 3636 3196 296 144 91.52%[email protected] 858 2.44s 43m 2.9s 3901 2335 0 1566 100.00%[email protected] 44 20.67s 44m 26.7s 1397 1016 373 8 73.15%moment @2.22.1 3359 14.59s 14h 22m 25s 10944 5175 1763 4006 74.59%[email protected] 4 0.96s 4m 50.1s 1304 114 1186 4 8.77%[email protected] 1572 5.96 49m 41.2s 2852 2256 523 73 81.18%[email protected] 32 0.46s 4m 34.1s 2258 449 1809 0 19.88%[email protected] 231 1.21s 6m 17.3s 1095 949 106 40 89.95%[email protected] 1706 35.31s 6h 25m 24s 2117 1360 461 296 74.68%[email protected] 17 0.47s 7m 39.2s 3338 2920 396 22 88.06%[email protected] 55 0.5s 46.5s 262 184 73 5 71.60%[email protected] 140 1.76s 3m 27.8s 514 482 25 7 95.07%[email protected] 511 15.94s 3h 47m 27.9s 4586 3728 714 144 83.93%[email protected] 44 0.43s 17.3s 101 79 22 0 78.22%[email protected] 181 15.79 43m 54.3s 1775 1769 0 6 100.00%[email protected] 243 12.27s 53m 26.3s 2181 1273 859 49 59.71%[email protected] 30 3.21s 23m 27.7s 2020 1569 402 49 79.60%[email protected] 598 1.33s 21m 10.5s 2754 2424 292 38 89.25%Averages 589 9.4s 1h 49m 42s 2580 1783 471 326 77.3%

Diving deeper into the mutants

To illustrate the reasons of surviving and discarded mutants, we analyzed Mutodeexecution on different modules.

[email protected]: Out of the 373 surviving mutants: 103 were string literal muta-tions, for example MUTANT 1265: SLM index.js Line 653: this.emit(’command:*’,args); mutated to this.emit(’D6RTbZ6PI’, args) survives, meaning that theemition of the event is not tested; 83 were numeric literal mutations, for ex-ample MUTANT 445: NLM index.js Line 1201: process.exit(0); mutated to‘process.exit(1); survives, meaning exit code of the process is not tested at all;53 were removed function call arguments mutations; 49 were removed line muta-tions; 28 were removed conditional mutations; and the rest were distributed in thefollowing mutators: math, remove array elements, remove object property, removefunctions, negate conditionals, conditionals boundary, and boolean literals. Out of theeight discarded mutants, four were because of removed arguments of a child processexecution started by the module, which then will not exit, for example MUTANT1007: RLM Commented line 556: args.unshift(bin); removed the line whichwill remove the binary from the args, so nothing is executed in the child process,never exiting; three were because of modified for statements’ step, which caused aninfinite loop; and one was because of a modified for statements’ boundary whichblocked execution by not finishing.

[email protected]: Out of the 1818 surviving mutants: 695 were string literal mutations,for example MUTANT 181: SLM Line 71: throw new TypeError(’Expected astring’); mutated to throw new TypeError(”); survives, meaning error messageis not tested; 401 were numeric literal mutations, for example MUTANT 553: NLMLine 44: blue: [34, 39], changed the array elements values to random numberswithout effect, implementing a test case for these scenario would imply a rendered


console to check that colors match; 304 were removed array elements mutations; 116were removed line mutations, 84 were removed functions call arguments mutations;59 removed objected properties mutations; and the rest were distributed in theremaining mutators. No mutants were discarded.

[email protected]: Out of the 550 surviving mutants: 236 were numeric literal mutations,for example MUTANT 253: NLM Line 46: var length32 = input.length * 32;mutated to var length32 = input.length * 31; survives, meaning length32 isnot tested; 86 were string literal mutations; 58 were removed line mutations, for ex-ample MUTANT 3321: RLM Commented line 26: return buf || bytesToUuid(rnds);survives, meaning that the return value is never tested; 50 were removed condi-tional mutations; 43 removed function call arguments mutations; and the restwere distributed in the following mutators: negate conditionals, math, conditionalsboundary, remove array elements, remove switch cases, remove function declarationparameters, increments and remove functions. Out of the 24 discard mutants, 21 werebecause of modified for statements’ step and three were because array access indexor length were modified by the numeric literal mutator, for example MUTANT 146:MM Line 66: x[(((len + 64) »> 9) « 4) + 14] = len; mutated to x[(((len- 64) »> 9) « 4) + 14] = len; survives, meaning that the changed index valueaffects the normal execution of the application, probably cause an infinite loop or anon-exit condition.

[email protected]: Out of the 167 surviving mutants: 56 were string literal mu-tations, for example MUTANT 468: SLM Line 54: ? bytes.parse(opts.limit ||’TpqCy’) sets a default invalid limit, meaning it is not tested, opts.limit is alwaysset or bytes.parse does not throw an error correctly; 26 were removed function call ar-guments mutations, for exmaple MUTANT 889: RFCAM Line 156: var arrayLimit= Math.max(100, paramCount) removed the 100 param from Math.max function,meaning that there’s no test case for the variable arrayLimit and paramCount as ar-ray limit is always good enough; 23 were removed line mutations; 19 were numericliteral mutations; 17 were removed object properties mutation; 11 were removedconditional mutations; 8 were boolean literal mutations; and 7 were negated condi-tions mutations. Out of the three discarded mutatants, two were debug calls and 1was a modified for statement’s step: MUTANT 518: IM Line 204: for (var i = 0;i < keys.length; i–), causing an infinite loop.


5Conclusion

In the development of the project, most objectives were achieved and Mutode waspublished in the npm registry, currently recording around a thousand downloadsper month. Following, I present the conclusions of the process and describe futurework.

• Mutation testing is an effective technique to measure test suites’ effectivenessand support developers improve their test suite. It is currently the best methodto go beyond test code coverage, which, even if is 100%, might not be enoughto tell that the code is good.

• The JavaScript and Node.js ecosystem is growing rapidly in usage, howevervalidation and analysis tools are not moving as fast, representing a securityissue. There’s much work to do in the ecosystem, which is mostly open source.

• Mutode is an effective tool that helps its users improve their test suites ,implementing 43 mutation operations grouped in 16 mutators and allowingeasy, zero or minimal configuration usage. Mutode is published as a free andopen source npm module, hosted on GitHub, under the MIT License.

• Mutode was evaluated against the top-20 most depended upon modules withautomated test suites of the npm registry and was shown to generate varioussurviving mutants which could provide insights on how to improve thosemodules’ test suites.

5.1 Achieved Objectives

• Mutode, a generic mutation testing tool was built, leveraging the npm package.jsondefinition using the npm test to execute applications’ test suites.

• A catalog of mutation operations based on different sources was implemented.

• Mutode was evaluated on the top 20 most depended upon modules

41

5.2 Limitations

• Mutode is currently limited to run with npm and the npm test command thatthe users have in their package.json file.

• Mutants execution times may be as long as days, depending on computingresources and the size of the project where it’s being executed.

• Large files in which Mutode generates many mutants may cause a runtimeerror due to lack of memory, as each mutant is a copy of the full file.

5.3 Future Work

There is much future work, both in terms of new functionalities and features, and ofexisting ones.

5.3.1 New Features

• Add new mutation operations.

• Implement a dash board.

• Generate HTML reports with charts and graphs.

• Allow users to specify testing command.

• Publish in other package managers such as brew.

5.3.2 Existing Features

• Improve current mutation operations catalog.

• Improve concurrency management to not exceed 100% CPU usage, whichcauses discarded mutants.

• Improve detection of redundant and equivalent mutants.

42 Chapter 5 Conclusion

Bibliography

[1] James H Andrews et al. „Using mutation analysis for assessing and comparingtesting coverage criteria“. In: TSE 32.8 (2006), pp. 608–624.

[2] Babel. Babylon. 2018. URL: https://github.com/babel/babel/tree/master/packages/babylon.

[3] Austin Bingham. Cosmic Ray: mutation testing for Python. 2017. URL: https://github.com/sixty-north/cosmic-ray.

[4] Pádraic Brady. HumBug. 2017. URL: https://github.com/humbug/humbug.

[5] Ben Coe. Istanbul. https: // istanbul. js. org/ . 2018.

[6] Henry Coles. PIT. http: // pitest. org/ . 2017.

[7] Criteo. NinjaTurtlesMutation. 2016. URL: https://github.com/criteo/NinjaTurtlesMutation.

[8] Erik DeBill. Modulecounts. http: // www. modulecounts. com/ . 2018.

[9] Alex Denisov and Stanislav Pankevich. „Mull it over: mutation testing basedon LLVM. https://github.com/mull-project/mull“. In: ().

[10] Anna Derezinska and Piotr Trzpil. „Mutation Testing Process Combined withTest-Driven Development in. NET Environment“. In: Theory and Engineeringof Complex Systems and Dependability. 2015, pp. 131–140.

[11] A. M. Fard and A. Mesbah. „JavaScript: The (Un)Covered Parts“. In: ICST’17.2017, pp. 230–240.

[12] Node Foundation. ECMAScript Modules. https: // nodejs. org/ api/ esm.html . 2018.

[13] Konrad Halas. MutPy: mutation testing tool for Python 3.x source code. 2017.URL: https://github.com/mutpy/mutpy.

[14] Quinn Hanam, Fernando S. de M. Brito, and Ali Mesbah. „Discovering BugPatterns in JavaScript“. In: ESEC/FSE’16. 2016, pp. 144–156.

[15] Ben Hartley. Mutant: A mutation testing framework for JavaScript. 2016. URL:https://github.com/benhartley/mutant.

[16] TJ Holowaychuk. Mocha. https: // mochajs. org/ . 2018.

[17] Anders Hovmöller. Mutmut. 2018. URL: https : / / github . com / boxed /mutmut.

43

https://github.com/babel/babel/tree/master/packages/babylon

https://github.com/babel/babel/tree/master/packages/babylon




https://istanbul.js.org/

http://pitest.org/



http://www.modulecounts.com/


https://nodejs.org/api/esm.html

https://nodejs.org/api/esm.html



https://mochajs.org/



[18] npm, Inc., Node.JS Foundation, and JS Foundation. Attitudes to security in theJavaScript community npm, Inc. Medium. 2018. URL: https://medium.com/npm-inc/security-in-the-js-community-4bac032e553b.

[19] Reyhaneh Jabbarvand and Sam Malek. „muDroid: An Energy-aware MutationTesting Framework for Android“. In: ESEC/FSE’17. 2017, pp. 208–219.

[20] Nico Jansen and Simon de Lang. Stryker: The JavaScript mutation testingframework. 2018. URL: https://github.com/stryker-mutator/stryker.

[21] Yue Jia and Mark Harman. „An analysis and survey of the development ofmutation testing“. In: TSE 37.5 (2011), pp. 649–678.

[22] Yue Jia and Mark Harman. „MILU: A customizable, runtime-optimized higherorder mutation testing tool for the full C language“. In: Practice and ResearchTechniques, 2008. TAIC PART’08. Testing: Academic & Industrial Conference.2008, pp. 94–98.

[23] René Just. „The Major mutation framework: Efficient and scalable mutationanalysis for Java“. In: ISSTA’14. 2014, pp. 433–436.

[24] René Just et al. „Are mutants a valid substitute for real faults in softwaretesting?“ In: ESEC/FSE 14. 2014, pp. 654–665.

[25] Vojta Jína. Karma. https: // karma- runner. github. io/ 2. 0/ index.html . 2018.

[26] Mario Linares-Vásquez et al. „Enabling mutation testing for Android apps“.In: ESEC/FSE’17. 2017, pp. 233–244.

[27] Niels Lohmann. Mutate++: C++ Mutation Test Environment. 2017. URL:https://github.com/nlohmann/mutate_cpp.

[28] Yu-Seung Ma, Jeff Offutt, and Yong Rae Kwon. „MuJava: An automated classmutation system“. In: Software Testing, Verification and Reliability 15.2 (2005),pp. 97–133.

[29] Lech Madeyski and Norbert Radyk. „Judy–a mutation testing tool for Java“.In: IET software 4.1 (2010), pp. 32–42.

[30] Pedro Reales Mateo and Macario Polo Usaola. „Bacterio: Java mutation testingtool: A framework to evaluate quality of tests cases“. In: Software Maintenance(ICSM), 2012 28th IEEE International Conference on. IEEE. 2012, pp. 646–649.

[31] Microsoft. TypeScript. https: // www. typescriptlang. org/ . 2018.

[32] Shabnam Mirshokraie, Ali Mesbah, and Karthik Pattabiraman. „EfficientJavaScript mutation testing“. In: ICST’13. 2013, pp. 74–83.

[33] David Musgrove. NinjaTurtles - .NET Mutation Testing http: // www. mutation-testing. net/ . 2015.

[34] Inc. npm. About npm. 2018. URL: https://www.npmjs.com/about.

[35] Inc. npm. Most depended upon packages. 2018. URL: https://www.npmjs.com/browse/depended.

[36] F. Ocariza et al. „An Empirical Study of Client-Side JavaScript Bugs“. In:ESEM’13. 2013, pp. 55–64.

44 Bibliography

https://medium.com/npm-inc/security-in-the-js-community-4bac032e553b

https://medium.com/npm-inc/security-in-the-js-community-4bac032e553b

https://github.com/stryker-mutator/stryker

https://karma-runner.github.io/2.0/index.html

https://karma-runner.github.io/2.0/index.html


https://www.typescriptlang.org/



https://www.npmjs.com/about

https://www.npmjs.com/browse/depended

https://www.npmjs.com/browse/depended

[37] Frolin S Ocariza Jr, Karthik Pattabiraman, and Benjamin Zorn. „JavaScripterrors in the wild: An empirical study“. In: ISSRE’11. 2011, pp. 100–109.

[38] A Jefferson Offutt et al. „An experimental evaluation of data flow and muta-tion testing“. In: Softw., Pract. Exper. 26.2 (1996), pp. 165–176.

[39] Maks Rafalko. Infection: PHP Mutation Testing Framework. 2018. URL: https://github.com/infection/infection.

[40] RequireJS. CommonJS. http: // requirejs. org/ docs/ commonjs. html .2018.

[41] Diego Rodríguez-Baquero. mutode. 2018. URL: https://www.npmjs.com/package/mutode.

[42] Diego Rodríguez-Baquero. Mutode: Mutation testing for JavaScript and Node.js.2018. URL: https://github.com/TheSoftwareDesignLab/mutode.

[43] Diego Rodríguez-Baquero and Mario Linares-Vásquez. „Mutode: GenericJavaScript and Node.js Mutation Testing Tool“. In: 27th International Sympo-sium on Software Testing and Analysis (ISSTA’18). 4 pages, to appear.

[44] Markus Schirp. Mutant: Mutation testing for Ruby. 2018. URL: https://github.com/mbj/mutant.

[45] David Schuler and Andreas Zeller. „Javalanche: Efficient mutation testingfor Java“. In: ESEC/FSE ’09: Proceedings of the 7th joint meeting of the Eu-ropean Software Engineering Conference and the ACM SIGSOFT InternationalSymposium on Foundations of Software Engineering. Amsterdam, Aug. 2009,pp. 297–298. ISBN: 9781605580012. DOI: 10.1145/1595696.1595750.

[46] Michael G Schwern and Andy Lester. TAP specification. https: // testanything.org/ tap- specification. html . 2018.

[47] Daniel Tschinder, Logan Smyth, and Henry Zhu. Babel. https: // babeljs.io/ . 2018.

[48] Laurie Voss. Rolling weekly downloads of npm packages. 2018. URL: https://twitter.com/seldo/status/988477780441481217.

Bibliography 45

https://github.com/infection/infection

https://github.com/infection/infection

http://requirejs.org/docs/commonjs.html

https://www.npmjs.com/package/mutode

https://www.npmjs.com/package/mutode

https://github.com/TheSoftwareDesignLab/mutode



http://dx.doi.org/10.1145/1595696.1595750

https://testanything.org/tap-specification.html

https://testanything.org/tap-specification.html

https://babeljs.io/

https://babeljs.io/

https://twitter.com/seldo/status/988477780441481217

https://twitter.com/seldo/status/988477780441481217

List of Tables

1.1 Activities Schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.1 Mutation Frameworks & Tools . . . . . . . . . . . . . . . . . . . . . . . 12

3.1 Conditionals Boundary Mutator Operations . . . . . . . . . . . . . . . 163.2 Math Mutator Operations . . . . . . . . . . . . . . . . . . . . . . . . . 173.3 Negate Conditionals Mutator Operations . . . . . . . . . . . . . . . . . 183.4 Numeric Literals Mutator Operations . . . . . . . . . . . . . . . . . . . 183.5 String Literals Mutator Operations . . . . . . . . . . . . . . . . . . . . 223.6 Mutators and Operations in Mutode . . . . . . . . . . . . . . . . . . . . 23

4.1 Study results on npm modules . . . . . . . . . . . . . . . . . . . . . . . 38

47

Colophon

This document was written inLATEX 2ε. It uses the style Clean Thesis, developed byRicardo Langner.

Node.js is a trademark of Joyent, Inc. and is used with its permission. I am notendorsed by or affiliated with Joyent.

Mutode: Generic JavaScript and Node.js Mutation Testing Tool

Documents

Transcript of Mutode: Generic JavaScript and Node.js Mutation Testing Tool