Skip to content

Property testing distributed applications with fault injection and verification.

License

Notifications You must be signed in to change notification settings

nurturenature/fuzz_dist

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

fuzz_dist

A framework and customization of Jepsen for BEAM applications.

Generative property testing of...

  • actual application in a real cluster
  • real environmental faults

... And validate the results with a model/checker.


Using Jepsen

"We should do a Jepsen test!"

-- Every Developer ever...

Using Jepsen meaningfully, like all dist-sys endeavors, is a meaningful commitment.

fuzz_dist is an effort to bring Jepsen and BEAM applications together in a more approachable way.

  • generic'ish Elixir client
    • test operations as Elixir @behaviors vs Clojure (fn [])
    • common user connects via websocket to nearest local node which is geo clustered architecture
  • build/test/run environment pre-configured for BEAM
  • more efficient/effective fault injection

BEAM Aware

“Know the rules well, so you can break them effectively.”

-- Dalai Lama XIV

Distributed Erlang is resilient. Applications can use these primitives to implement powerful algorithms.

Is the application's implementation standing firmly on and aligned with the BEAM?

fuzz_dist directs Jepsen's nemesis generators to fuzz at distributed Erlang boundary values, triggers.

Fuzz near

  • partition duration ~ NetTick(Time/Intensity)/HeartBeat
  • network faults ~ GenServer.call(:timeout)
  • :nodedown/up, {:error, :timeout}
  • ...

The BEAM was built to be observed. Combine these observations with Jepsen fault and application telemetry to directly link anomalies to metrics/spans/logs.

  • erlang:monitor(:nodes, :time/clock_services, ...)
  • Jepsen operations,injected faults ->
    • Telemetry.execute(:trouble, :start)
  • ...

Current Development, Usage

"Always be suspicious of success."

-- Leslie Lamport

Please see the current AntidoteDB tests, and an environment to run them.


.next()

Tests

  • new types of faults
  • more generative workflows
  • improve analysis reports

Client

  • add :interactive transactions
    • ability to abort
  • more comprehensive Telemetry

Environment

  • more ergonomic
  • easier to package/deploy

Jepsen Tests Are Living Tests

Fuzzing distributed systems productively, i.e. actually improving documentation/code, increasing understanding/confidence, in a sustainable way is both an art and a science.

The Jepsen community has lead the way in showing us how to do it effectively, ethically, and in a way that's truly fun.

Many thanks to @Aphyr and https://jepsen.io for https://github.com/jepsen-io/jepsen.

About

Property testing distributed applications with fault injection and verification.

Topics

Resources

License

Stars

Watchers

Forks