Created
2 years ago
Modified
2 years ago
Author
avatar~jeff-hykin
Status
Todo
Labels

#123Extreme Coverage/Fuzzing

  • avatar~jeff-hykincommented · 2 years ago
    Author

    Note: This can't be a userland feature; it requires language/builtin integration.

    Whats the problem

    "100% coverage" is often bad/misleading

    Whats the deeper problem

    Let me use an example; the common counter-example to "100% coverage = good" is is array indexing. E.g. every logical branch is explored, but the but the test doesn't include a bad array-index, and misses an important case that causes the code to crash. One of the deeper issues is: nested exceptions/panics are not treated like branches. Builtins can throw, and coverage doesn't consider that.

    Exceptions as values fix this a bit (lets go 👌), but not necessarily all the way.

    Whats the solution

    // example func that "needs" coverage
fn doStuff(x, y) {
    return x[y] + jsonParse(y)
}
// create a per-operation list of "error branches":
//- array access (operation): [normal value], OutOfBounds, TypeError
//- jsonParse (fn): [normal value], SyntaxError
//- addition (operation): [normal value], TypeError
    1. Treat each of those as a coverage branch
    2. Provide some way of triggering errors directly. Like trigger an OutOfBounds error even if the given index was in-bounds

    This would be a form of fuzzing (it kind of combines fuzzing and coverage) and would make "100% coverage" way more robust/meaningful. Still not perfect, but I think its like a 50% improvement.

    Milestone: 0.3.0.

  • avatar~jeff-hykin changed the status to Todo · 2 years ago