Blog HCP Consul on Azure goes GA, plus more Consul news from HashiConf EU Read more
  • Overview
    • Consul on Kubernetes
    • Control access with Consul API Gateway
    • Discover Services with Consul
    • Enforce Zero Trust Networking with Consul
    • Load Balancing with Consul
    • Manage Traffic with Consul
    • Multi-Platform Service Mesh with Consul
    • Network Infrastructure Automation with Consul
    • Observability with Consul
  • Enterprise
  • Tutorials
  • Docs
  • API
  • CLI
  • Community
GitHub
Download
Try HCP Consul
    • v1.12.x (latest)
    • v1.11.x
    • v1.10.x
    • v1.9.x
    • v1.8.x
    • Overview
      • Overview
      • What is a Service Mesh?
      • Overview
      • Chef, Puppet, etc.
      • Nagios
      • SkyDNS
      • SmartStack
      • Serf
      • Eureka
      • Istio
      • Envoy and Other Proxies
      • Custom Solutions
    • Overview
    • Manual Bootstrap
    • Consul Agent
    • Glossary
    • Required Ports
    • Bootstrapping a Datacenter
    • Cloud Auto-join
    • Server Performance
    • Kubernetes
  • API
  • Commands (CLI)
    • Register Services - Service Definitions
    • Find Services - DNS Interface
    • Monitor Services - Check Definitions
    • Overview
    • How Service Mesh Works
    • Configuration
      • Overview
      • Ingress Gateway
      • Mesh
      • Exported Services
      • Proxy Defaults
      • Service Defaults
      • Service Intentions
      • Service Resolver
      • Service Router
      • Service Splitter
      • Terminating Gateway
      • Overview
      • Envoy
      • Built-in Proxy
      • Proxy Integration
      • Managed (Deprecated)
      • Overview
      • Proxy Service Registration
      • Sidecar Service Registration
    • Service-to-service permissions - Intentions
    • Service-to-service permissions - Intentions (Legacy Mode)
    • Transparent Proxy
      • Overview
      • UI Visualization
      • Overview
      • Discovery Chain
    • Connectivity Tasks
    • Distributed Tracing
      • Overview
        • WAN Federation
        • Enabling Service-to-service Traffic Across Datacenters
        • Enabling Service-to-service Traffic Across Admin Partitions
      • Ingress Gateways
      • Terminating Gateways
      • What is Cluster Peering
      • Create and Manage Peering Connections
      • Cluster Peering on Kubernetes
    • Nomad
    • Kubernetes
      • Overview
      • Go Integration
      • Overview
      • Built-In CA
      • Vault
      • ACM Private CA
    • Develop and Debug
    • Security
    • Overview
    • Installation
    • Technical Specifications
    • Common Errors
    • Upgrades
    • Overview
    • Architecture
      • Installing Consul on Kubernetes
      • Installing Consul K8s CLI
        • Minikube
        • Kind
        • AKS (Azure)
        • EKS (AWS)
        • GKE (Google Cloud)
        • Red Hat OpenShift
        • Self Hosted Kubernetes
        • Consul Clients Outside Kubernetes
        • Consul Servers Outside Kubernetes
        • Single Consul Datacenter in Multiple Kubernetes Clusters
        • Consul Enterprise
        • Overview
        • Federation Between Kubernetes Clusters
        • Federation Between VMs and Kubernetes
        • Overview
        • Systems Integration
          • Overview
          • Bootstrap Token
          • Enterprise License
          • Gossip Encryption Key
          • Partition Token
          • Replication Token
          • Server TLS
          • Service Mesh Certificates
          • Snapshot Agent Config
          • Webhook Certificates
        • WAN Federation
      • Overview
      • Transparent Proxy
      • Ingress Gateways
      • Terminating Gateways
      • Ingress Controllers
      • Configuring a Connect CA Provider
      • Health Checks
        • Metrics
    • Service Sync
      • Overview
      • Upgrade An Existing Cluster to CRDs
    • Annotations and Labels
    • Consul DNS
      • Upgrading Consul on Kubernetes
      • Upgrading Consul K8s CLI
      • Uninstall
      • Certificate Rotation
      • Gossip Encryption Key Rotation
      • Configure TLS on an Existing Cluster
      • Common Error Messages
      • FAQ
    • Compatibility Matrix
    • Helm Chart Configuration
    • Consul K8s CLI Reference
    • Overview
    • Requirements
    • Task Resource Usage
      • Installation
      • Secure Configuration
      • Migrate Existing Tasks
      • Installation
      • Secure Configuration
      • ACL Controller
    • Architecture
    • Consul Enterprise
    • Configuration Reference
    • Overview
    • Register Lambda Functions
    • Invoke Lambda Functions
    • Overview
      • Installation
      • Requirements
      • Configure
      • Run Consul-Terraform-Sync
    • Architecture
      • Overview
      • Status
      • Tasks
      • Health
      • Overview
      • task
      • start
    • Configuration
    • Tasks
    • Terraform Modules
      • Overview
      • License
      • Terraform Cloud Driver
      • Overview
      • Terraform
      • Terraform Cloud
    • Compatibility
    • Consul KV
    • Sessions
    • Watches
    • Overview
      • General
      • CLI Reference
      • Configuration Reference
    • Configuration Entries
    • Telemetry
    • Sentinel
    • RPC
    • Overview
      • ACL System Overview
      • Tokens
      • Policies
      • Roles
      • Rules Reference
      • Legacy Mode
      • Token Migration
      • ACLs in Federated Datacenters
        • Overview
        • Kubernetes
        • JWT
        • OIDC
        • AWS IAM
    • Encryption
      • Overview
      • Core
      • Network Infrastructure Automation
    • Overview
    • Admin Partitions
    • Audit Logging
    • Automated Backups
    • Automated Upgrades
    • Enhanced Read Scalability
    • Single sign-on - OIDC
    • Redundancy Zones
    • Advanced Federation
    • Network Segments
    • Namespaces
    • NIA with TFE
    • Sentinel
      • Overview
      • FAQ
    • Overview
    • Improving Consul Resilience
    • Anti-Entropy
    • Consensus Protocol
    • Gossip Protocol
    • Jepsen Testing
    • Network Coordinates
    • Consul Integration Program
    • NIA Integration Program
    • Vault Integration
    • Proxy Integration
  • Consul Tools
    • Overview
    • Compatibility Promise
    • Specific Version Details
      • Overview
      • General Process
      • Upgrading to 1.2.4
      • Upgrading to 1.6.9
      • Upgrading to 1.8.13
      • Upgrading to 1.10.0
    • Common Error Messages
    • FAQ
    • Overview
      • v1.11.x
      • v1.10.x
      • v1.9.x
      • v0.3.x
      • v0.2.x
      • v0.1.x
      • v0.4.x
      • v0.3.x
      • v0.2.x
      • v0.6.x
      • v0.5.x
    • Overview
    • ACL
  • Guides
Type '/' to Search

»Jepsen Testing

Jepsen is a tool, written by Kyle Kingsbury, designed to test the partition tolerance of distributed systems. It creates network partitions while fuzzing the system with random operations. The results are analyzed to see if the system violates any of the consistency properties it claims to have.

As part of our Consul testing, we ran a Jepsen test to determine if any consistency issues could be uncovered. In our testing, Consul gracefully recovered from partitions without introducing any consistency issues.

»Running the tests

At the moment, testing with Jepsen is rather complex as it requires setting up multiple virtual machines, SSH keys, DNS configuration, and a working Clojure environment. We hope to contribute our Consul testing code upstream and to provide a Vagrant environment for Jepsen testing soon.

»Output

Below is the output captured from Jepsen. We ran Jepsen multiple times, and it passed each time. This output is only representative of a single run and has been edited for length. Please reach out on Consul's Discuss if you would like to reproduce the Jepsen results.

$ lein test :only jepsen.system.consul-test

lein test jepsen.system.consul-test
INFO  jepsen.os.debian - :n5 setting up debian
INFO  jepsen.os.debian - :n3 setting up debian
INFO  jepsen.os.debian - :n4 setting up debian
INFO  jepsen.os.debian - :n1 setting up debian
INFO  jepsen.os.debian - :n2 setting up debian
INFO  jepsen.os.debian - :n4 debian set up
INFO  jepsen.os.debian - :n5 debian set up
INFO  jepsen.os.debian - :n3 debian set up
INFO  jepsen.os.debian - :n1 debian set up
INFO  jepsen.os.debian - :n2 debian set up
INFO  jepsen.system.consul - :n1 consul nuked
INFO  jepsen.system.consul - :n4 consul nuked
INFO  jepsen.system.consul - :n5 consul nuked
INFO  jepsen.system.consul - :n3 consul nuked
INFO  jepsen.system.consul - :n2 consul nuked
INFO  jepsen.system.consul - Running nodes: {:n1 false, :n2 false, :n3 false, :n4 false, :n5 false}
INFO  jepsen.system.consul - :n2 consul nuked
INFO  jepsen.system.consul - :n3 consul nuked
INFO  jepsen.system.consul - :n4 consul nuked
INFO  jepsen.system.consul - :n5 consul nuked
INFO  jepsen.system.consul - :n1 consul nuked
INFO  jepsen.system.consul - :n1 starting consul
INFO  jepsen.system.consul - :n2 starting consul
INFO  jepsen.system.consul - :n4 starting consul
INFO  jepsen.system.consul - :n5 starting consul
INFO  jepsen.system.consul - :n3 starting consul
INFO  jepsen.system.consul - :n3 consul ready
INFO  jepsen.system.consul - :n2 consul ready
INFO  jepsen.system.consul - Running nodes: {:n1 true, :n2 true, :n3 true, :n4 true, :n5 true}
INFO  jepsen.system.consul - :n5 consul ready
INFO  jepsen.system.consul - :n1 consul ready
INFO  jepsen.system.consul - :n4 consul ready
INFO  jepsen.core - Worker 0 starting
INFO  jepsen.core - Worker 2 starting
INFO  jepsen.core - Worker 1 starting
INFO  jepsen.core - Worker 3 starting
INFO  jepsen.core - Worker 4 starting
INFO  jepsen.util - 2   :invoke :read   nil
INFO  jepsen.util - 3   :invoke :cas    [4 4]
INFO  jepsen.util - 0   :invoke :write  4
INFO  jepsen.util - 1   :invoke :write  1
INFO  jepsen.util - 4   :invoke :cas    [4 0]
INFO  jepsen.util - 2   :ok :read   nil
INFO  jepsen.util - 4   :fail   :cas    [4 0]
(Log Truncated...)
INFO  jepsen.util - 4   :invoke :cas    [3 3]
INFO  jepsen.util - 4   :fail   :cas    [3 3]
INFO  jepsen.util - :nemesis    :info   :stop   nil
INFO  jepsen.util - :nemesis    :info   :stop   "fully connected"
INFO  jepsen.util - 0   :fail   :read   nil
INFO  jepsen.util - 1   :fail   :write  0
INFO  jepsen.util - :nemesis    :info   :stop   nil
INFO  jepsen.util - :nemesis    :info   :stop   "fully connected"
INFO  jepsen.core - nemesis done
INFO  jepsen.core - Worker 3 done
INFO  jepsen.util - 1   :invoke :read   nil
INFO  jepsen.core - Worker 2 done
INFO  jepsen.core - Worker 4 done
INFO  jepsen.core - Worker 0 done
INFO  jepsen.util - 1   :ok :read   3
INFO  jepsen.core - Worker 1 done
INFO  jepsen.core - Run complete, writing
INFO  jepsen.core - Analyzing
(Log Truncated...)
INFO  jepsen.core - Analysis complete
INFO  jepsen.system.consul - :n3 consul nuked
INFO  jepsen.system.consul - :n2 consul nuked
INFO  jepsen.system.consul - :n4 consul nuked
INFO  jepsen.system.consul - :n1 consul nuked
INFO  jepsen.system.consul - :n5 consul nuked
1964 element history linearizable. :D

Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
$ lein test :only jepsen.system.consul-test

lein test jepsen.system.consul-test
INFO  jepsen.os.debian - :n5 setting up debian
INFO  jepsen.os.debian - :n3 setting up debian
INFO  jepsen.os.debian - :n4 setting up debian
INFO  jepsen.os.debian - :n1 setting up debian
INFO  jepsen.os.debian - :n2 setting up debian
INFO  jepsen.os.debian - :n4 debian set up
INFO  jepsen.os.debian - :n5 debian set up
INFO  jepsen.os.debian - :n3 debian set up
INFO  jepsen.os.debian - :n1 debian set up
INFO  jepsen.os.debian - :n2 debian set up
INFO  jepsen.system.consul - :n1 consul nuked
INFO  jepsen.system.consul - :n4 consul nuked
INFO  jepsen.system.consul - :n5 consul nuked
INFO  jepsen.system.consul - :n3 consul nuked
INFO  jepsen.system.consul - :n2 consul nuked
INFO  jepsen.system.consul - Running nodes: {:n1 false, :n2 false, :n3 false, :n4 false, :n5 false}
INFO  jepsen.system.consul - :n2 consul nuked
INFO  jepsen.system.consul - :n3 consul nuked
INFO  jepsen.system.consul - :n4 consul nuked
INFO  jepsen.system.consul - :n5 consul nuked
INFO  jepsen.system.consul - :n1 consul nuked
INFO  jepsen.system.consul - :n1 starting consul
INFO  jepsen.system.consul - :n2 starting consul
INFO  jepsen.system.consul - :n4 starting consul
INFO  jepsen.system.consul - :n5 starting consul
INFO  jepsen.system.consul - :n3 starting consul
INFO  jepsen.system.consul - :n3 consul ready
INFO  jepsen.system.consul - :n2 consul ready
INFO  jepsen.system.consul - Running nodes: {:n1 true, :n2 true, :n3 true, :n4 true, :n5 true}
INFO  jepsen.system.consul - :n5 consul ready
INFO  jepsen.system.consul - :n1 consul ready
INFO  jepsen.system.consul - :n4 consul ready
INFO  jepsen.core - Worker 0 starting
INFO  jepsen.core - Worker 2 starting
INFO  jepsen.core - Worker 1 starting
INFO  jepsen.core - Worker 3 starting
INFO  jepsen.core - Worker 4 starting
INFO  jepsen.util - 2   :invoke :read   nil
INFO  jepsen.util - 3   :invoke :cas    [4 4]
INFO  jepsen.util - 0   :invoke :write  4
INFO  jepsen.util - 1   :invoke :write  1
INFO  jepsen.util - 4   :invoke :cas    [4 0]
INFO  jepsen.util - 2   :ok :read   nil
INFO  jepsen.util - 4   :fail   :cas    [4 0]
(Log Truncated...)
INFO  jepsen.util - 4   :invoke :cas    [3 3]
INFO  jepsen.util - 4   :fail   :cas    [3 3]
INFO  jepsen.util - :nemesis    :info   :stop   nil
INFO  jepsen.util - :nemesis    :info   :stop   "fully connected"
INFO  jepsen.util - 0   :fail   :read   nil
INFO  jepsen.util - 1   :fail   :write  0
INFO  jepsen.util - :nemesis    :info   :stop   nil
INFO  jepsen.util - :nemesis    :info   :stop   "fully connected"
INFO  jepsen.core - nemesis done
INFO  jepsen.core - Worker 3 done
INFO  jepsen.util - 1   :invoke :read   nil
INFO  jepsen.core - Worker 2 done
INFO  jepsen.core - Worker 4 done
INFO  jepsen.core - Worker 0 done
INFO  jepsen.util - 1   :ok :read   3
INFO  jepsen.core - Worker 1 done
INFO  jepsen.core - Run complete, writing
INFO  jepsen.core - Analyzing
(Log Truncated...)
INFO  jepsen.core - Analysis complete
INFO  jepsen.system.consul - :n3 consul nuked
INFO  jepsen.system.consul - :n2 consul nuked
INFO  jepsen.system.consul - :n4 consul nuked
INFO  jepsen.system.consul - :n1 consul nuked
INFO  jepsen.system.consul - :n5 consul nuked
1964 element history linearizable. :D

Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
github logoEdit this page
IntroGuidesDocsCommunityPrivacySecurityBrandConsent Manager