Necessist

Run tests with statements and method calls removed to help identify broken tests

Necessist currently supports Anchor, Foundry, Go, Hardhat, PHP, Rust, and Vitest.

A paper on Necessist (Test Harness Mutilation) appeared in Mutation 2024. (slides, preprint)

Contents

• Installation
• Running
• Overview
• Usage
• LLM-assisted auditing (experimental)
• Details
• Configuration files
• Source directives (experimental)
• Limitations
• Semantic versioning policy
• Goals
• Anti-goals
• References
• License

Installation

System requirements:

Install pkg-config and sqlite3 development files on your system, e.g., on Ubuntu:

sudo apt install pkg-config libsqlite3-dev

Install Necessist from crates.io:

cargo install necessist

Running

cd into your project’s directory and type necessist (with no arguments).

For example, if you cd into the fixtures/basic directory and type necessist, you should see the following:

4 candidates in 4 tests in 1 source file
src/lib.rs: dry running
src/lib.rs: mutilating
src/lib.rs:4:5-4:12: `n += 1;` passed

Note that there will be a delay while Necessist runs a test with a timeout.

See Usage for options that can be passed to necessist.

Overview

Necessist iteratively removes statements and method calls from tests and then runs them. If a test passes with a statement or method call removed, it could indicate a problem in the test. Or worse, it could indicate a problem in the code being tested.

Example

This example is from rust-openssl. The verify_untrusted_callback_override_ok test checks that a failed certificate validation can be overridden by a callback. But if the callback were never called (e.g., because of a failed connection), the test would still pass. Necessist reveals this fact by showing that the test passes without the call to set_verify_callback:

#[test]
fn verify_untrusted_callback_override_ok() {
    let server = Server::builder().build();

    let mut client = server.client();
    client
        .ctx()
        .set_verify_callback(SslVerifyMode::PEER, |_, x509| { //
            assert!(x509.current_cert().is_some());           // Test passes without this call
            true                                              // to `set_verify_callback`.
        });                                                   //

    client.connect();
}

Following this discovery, a flag was added to the test to record whether the callback is called. The flag must be set for the test to succeed:

#[test]
fn verify_untrusted_callback_override_ok() {
    static CALLED_BACK: AtomicBool = AtomicBool::new(false);  // Added

    let server = Server::builder().build();

    let mut client = server.client();
    client
        .ctx()
        .set_verify_callback(SslVerifyMode::PEER, |_, x509| {
            CALLED_BACK.store(true, Ordering::SeqCst);        // Added
            assert!(x509.current_cert().is_some());
            true
        });

    client.connect();
    assert!(CALLED_BACK.load(Ordering::SeqCst));              // Added
}

Comparison to conventional mutation testing

Possible theoretical foundation

Usage

Usage: necessist [OPTIONS] [TEST_FILES_OR_DIRS]... [-- ...]

Arguments:
  [TEST_FILES_OR_DIRS]...  Test files or directories to mutilate; if `--root ` is passed, relative paths are interpreted relative to 
  [ARGS]...                Additional arguments to pass to each test command

Options:
      --allow         Silence ; `--allow all` silences all warnings
      --check-skill      Check whether the `necessist-audit` skill at  is the current version or later, exiting with code 1 if it is not; add --write to update a nonexistent or outdated skill
      --default-config         Create a default necessist.toml file in the project's root directory
      --deny          Treat  as an error; `--deny all` treats all warnings as errors
      --dump                   Dump sqlite database contents to the console
      --dump-candidate-counts  Dump number of removal candidates in each file and exit
      --dump-candidates        Dump removal candidates and exit (for debugging)
      --find-skill             Check whether the `necessist-audit` skill is installed in a well-known directory, and whether it is the current version or later, exiting with code 1 if it is not; add --write to update an outdated skill
      --framework   Assume testing framework is  [possible values: anchor, auto, foundry, go, hardhat, php, rust, vitest]
      --no-lines-or-columns    Do not output line or column information (experimental)
      --no-sqlite              Do not output to an sqlite database
      --quiet                  Do not output to the console
      --reset                  Discard sqlite database contents
      --resume                 Resume from the sqlite database
      --root             Root directory of the project under test
      --timeout       Maximum number of seconds to run any test; 60 is the default, 0 means no timeout
      --verbose                Show test outcomes besides `passed`
  -h, --help                   Print help
  -V, --version                Print version

Output

By default, Necessist outputs to the console only when tests pass. Passing --verbose causes Necessist to instead output all of the removal outcomes below.

By default, Necessist outputs to both the console and to an sqlite database. For the latter, a tool like sqlitebrowser can be used to filter/sort the results.

LLM-assisted auditing (experimental)

Necessist provides a necessist-audit skill for using an LLM to investigate whether passing removals are evidence of bugs in tests or in the code being tested. The skill requires shell access and the necessist command to be available on PATH; it reads results with necessist --dump. If a necessist.db file exists in the current directory, the skill will use that; otherwise, the skill will run necessist to generate the file.

The necessist binary contains the skill. Passing --check-skill compares the skill at `` to the contained one and reports which is newer; adding --write installs a missing skill or updates an outdated one.

To check the standard Claude Code and Codex skill directories, use --find-skill; adding --write updates any outdated skills it finds. Unlike --check-skill, --find-skill reports but does not install missing skills. None of these operations requires network access.

Both options exit with code 1 if a skill is missing or outdated and --write was not passed. An exit code of 2 indicates an error, e.g., an unparsable skill or an invalid command line. These exit codes follow the convention used by grep and diff.

Instructions for installing and using the skill with Claude Code and Codex follow.

Claude Code

To install or update the skill:

necessist --check-skill ~/.claude/skills/necessist-audit/SKILL.md --write

Then, from a directory you would like to review, invoke the skill:

/necessist-audit

Codex

To install or update the skill:

necessist --check-skill ~/.codex/skills/necessist-audit/SKILL.md --write

Then, from a directory you would like to review, invoke the skill:

$necessist-audit

Details

Generally speaking, Necessist will not attempt to remove a statement if it is one the following:

• a statement containing other statements (e.g., a for loop)
• a declaration (e.g., a local or let binding)
• a break, continue, or return
• the last statement in a test

Similarly, Necessist will not attempt to remove a method call if:

• It is the primary effect of an enclosing statement (e.g., x.foo();).
• It appears in the argument list of an ignored function, method, or macro (see below).

Also, for some frameworks, certain statements and methods are ignored. Click on a framework to see its specifics.

Configuration files

A configuration file allows one to tailor Necessist’s behavior with respect to a project. The file must be named necessist.toml, appear in the project’s root directory, and be toml encoded. The file may contain one or more of the options listed below.

• ignored_functions, ignored_methods, ignored_macros: A list of strings interpreted as patterns. A function, method, or macro (respectively) whose path matches a pattern in the list is ignored. Note that ignored_macros is used only by the Rust backend currently.

• ignored_path_disambiguation: One of the strings None, Function, or Method. For a path that could refer to a function or method (see below), this option influences whether the function or method is ignored.

  • ignored_path_disambiguation = "None" (default): Ignore if the path matches either an ignored_functions or ignored_methods pattern.
  • ignored_path_disambiguation = "Function": Ignore only if the path matches an ignored_functions pattern.
  • ignored_path_disambiguation = "Method": Ignore only if the path matches an ignored_methods pattern.

• ignored_tests: A list of strings. A test whose name exactly matches a string in the list is ignored. For Mocha-based frameworks (e.g., Anchor and Hardhat), a test name is considered to be a message passed to it.

• visit_ignored_arguments: A boolean indicating whether Necessist should visit the arguments of ignored functions, methods, and macros. The default is false.

• walkable_functions: A list of strings interpreted as patterns. If a test calls a function that matches the pattern, and the function is declared in the same file as the test, then statements and method calls are removed from the function as though it were a test.

Patterns

A pattern is a string composed of letters, numbers, ., _, or *. Each character, other than *, is treated literally and matches itself only. A * matches any string, including the empty string.

The following are examples of patterns:

• assert: matches itself only
• assert_eq: matches itself only
• assertEqual: matches itself only
• assert.Equal: matches itself only
• assert.*: matches assert.Equal, but not assert, assert_eq, or assertEqual
• assert*: matches assert, assert_eq, assertEqual, and assert.Equal
• *.Equal: matches assert.Equal, but not Equal

Notes:

• Patterns match paths, not individual identifiers.
• . is treated literally like in a glob pattern, not like in regular expression.

Paths

A path is a sequence of identifiers separated by .. Consider this example (from Chainlink):

operator.connect(roles.oracleNode).signer.sendTransaction({
    to: operator.address,
    data,
}),

In the above, operator.connect and signer.sendTransaction are paths.

Note, however, that paths like operator.connect are ambiguous:

• If operator refers to package or module, then operator.connect refers to a function.
• If operator refers to an object, then operator.connect refers to a method.

By default, Necessist ignores such a path if it matches either an ignored_functions or ignored_methods pattern. Setting the ignored_path_disambiguation option above to Function or Method causes Necessist ignore the path only if it matches an ignored_functions or ignored_methods pattern (respectively).

Source directives (experimental)

Place the following comment in a source file to prevent Necessist from removing any candidate that begins on the immediately next line:

// necessist: skip

The comment must occupy its own line. Whitespace before or after // and after the colon is optional. Trailing text is allowed following a word boundary, as in // necessist: skip, reason for skipping.

To prevent all removals in a file, use:

// necessist: skip-file

This directive is honored only when every preceding line is whitespace-only, begins with optional whitespace followed by //, or is <?php surrounded by optional whitespace. The last case is needed to make the directive usable in a PHP test file.

A file containing an honored skip-file directive is not parsed, and thus can contain invalid syntax. This behavior may change in the future.

Necessist warns about mispositioned skip-file directives and unrecognized comment-only necessist: directives.

Limitations

• Slow. Modifying tests requires them to be rebuilt. Running Necessist on even moderately sized codebases can take several hours.

• Triage requires intimate knowledge of the source code. Generally speaking, Necessist does not produce “obvious” bugs. In our experience, deciding whether a statement/method call should be necessary requires intimate knowledge of the code under test. Necessist is best run on codebases for which one has (or intends to have) such knowledge.

Semantic versioning policy

We reserve the right to change the following, and to consider such changes non-breaking:

• the syntax that Necessist ignores by default

Changes to the following will be accompanied by a bump of at least Necessist’s minor version:

• the order in which removal candidates are output
• the order in which records are stored in necessist.db

Goals

• If a project uses a supported framework, then cding into the project’s directory and typing necessist (with no arguments) should produce meaningful output.

Anti-goals

• Become a general-purpose mutation testing tool. Good such tools already exist (e.g., universalmutator).

References

• Groce, A., Ahmed, I., Jensen, C., McKenney, P.E., Holmes, J.: How verified (or tested) is my code? Falsification-driven verification and testing. Autom. Softw. Eng. 25, 917–960 (2018). A preprint is available. See Section 2.3.

License

Necessist is licensed and distributed under the AGPLv3 license. Contact us if you’re looking for an exception to the terms.