Blog HCP Consul on Azure goes GA, plus more Consul news from HashiConf EU Read more
  • Overview
    • Consul on Kubernetes
    • Control access with Consul API Gateway
    • Discover Services with Consul
    • Enforce Zero Trust Networking with Consul
    • Load Balancing with Consul
    • Manage Traffic with Consul
    • Multi-Platform Service Mesh with Consul
    • Network Infrastructure Automation with Consul
    • Observability with Consul
  • Enterprise
  • Tutorials
  • Docs
  • API
  • CLI
  • Community
GitHub
Download
Try HCP Consul
    • v1.12.x (latest)
    • v1.11.x
    • v1.10.x
    • v1.9.x
    • v1.8.x
    • Overview
      • Overview
      • What is a Service Mesh?
      • Overview
      • Chef, Puppet, etc.
      • Nagios
      • SkyDNS
      • SmartStack
      • Serf
      • Eureka
      • Istio
      • Envoy and Other Proxies
      • Custom Solutions
    • Overview
    • Manual Bootstrap
    • Consul Agent
    • Glossary
    • Required Ports
    • Bootstrapping a Datacenter
    • Cloud Auto-join
    • Server Performance
    • Kubernetes
  • API
  • Commands (CLI)
    • Register Services - Service Definitions
    • Find Services - DNS Interface
    • Monitor Services - Check Definitions
    • Overview
    • How Service Mesh Works
    • Configuration
      • Overview
      • Ingress Gateway
      • Mesh
      • Exported Services
      • Proxy Defaults
      • Service Defaults
      • Service Intentions
      • Service Resolver
      • Service Router
      • Service Splitter
      • Terminating Gateway
      • Overview
      • Envoy
      • Built-in Proxy
      • Proxy Integration
      • Managed (Deprecated)
      • Overview
      • Proxy Service Registration
      • Sidecar Service Registration
    • Service-to-service permissions - Intentions
    • Service-to-service permissions - Intentions (Legacy Mode)
    • Transparent Proxy
      • Overview
      • UI Visualization
      • Overview
      • Discovery Chain
    • Connectivity Tasks
    • Distributed Tracing
      • Overview
        • WAN Federation
        • Enabling Service-to-service Traffic Across Datacenters
        • Enabling Service-to-service Traffic Across Admin Partitions
      • Ingress Gateways
      • Terminating Gateways
      • What is Cluster Peering
      • Create and Manage Peering Connections
      • Cluster Peering on Kubernetes
    • Nomad
    • Kubernetes
      • Overview
      • Go Integration
      • Overview
      • Built-In CA
      • Vault
      • ACM Private CA
    • Develop and Debug
    • Security
    • Overview
    • Installation
    • Technical Specifications
    • Common Errors
    • Upgrades
    • Overview
    • Architecture
      • Installing Consul on Kubernetes
      • Installing Consul K8s CLI
        • Minikube
        • Kind
        • AKS (Azure)
        • EKS (AWS)
        • GKE (Google Cloud)
        • Red Hat OpenShift
        • Self Hosted Kubernetes
        • Consul Clients Outside Kubernetes
        • Consul Servers Outside Kubernetes
        • Single Consul Datacenter in Multiple Kubernetes Clusters
        • Consul Enterprise
        • Overview
        • Federation Between Kubernetes Clusters
        • Federation Between VMs and Kubernetes
        • Overview
        • Systems Integration
          • Overview
          • Bootstrap Token
          • Enterprise License
          • Gossip Encryption Key
          • Partition Token
          • Replication Token
          • Server TLS
          • Service Mesh Certificates
          • Snapshot Agent Config
          • Webhook Certificates
        • WAN Federation
      • Overview
      • Transparent Proxy
      • Ingress Gateways
      • Terminating Gateways
      • Ingress Controllers
      • Configuring a Connect CA Provider
      • Health Checks
        • Metrics
    • Service Sync
      • Overview
      • Upgrade An Existing Cluster to CRDs
    • Annotations and Labels
    • Consul DNS
      • Upgrading Consul on Kubernetes
      • Upgrading Consul K8s CLI
      • Uninstall
      • Certificate Rotation
      • Gossip Encryption Key Rotation
      • Configure TLS on an Existing Cluster
      • Common Error Messages
      • FAQ
    • Compatibility Matrix
    • Helm Chart Configuration
    • Consul K8s CLI Reference
    • Overview
    • Requirements
    • Task Resource Usage
      • Installation
      • Secure Configuration
      • Migrate Existing Tasks
      • Installation
      • Secure Configuration
      • ACL Controller
    • Architecture
    • Consul Enterprise
    • Configuration Reference
    • Overview
    • Register Lambda Functions
    • Invoke Lambda Functions
    • Overview
      • Installation
      • Requirements
      • Configure
      • Run Consul-Terraform-Sync
    • Architecture
      • Overview
      • Status
      • Tasks
      • Health
      • Overview
      • task
      • start
    • Configuration
    • Tasks
    • Terraform Modules
      • Overview
      • License
      • Terraform Cloud Driver
      • Overview
      • Terraform
      • Terraform Cloud
    • Compatibility
    • Consul KV
    • Sessions
    • Watches
    • Overview
      • General
      • CLI Reference
      • Configuration Reference
    • Configuration Entries
    • Telemetry
    • Sentinel
    • RPC
    • Overview
      • ACL System Overview
      • Tokens
      • Policies
      • Roles
      • Rules Reference
      • Legacy Mode
      • Token Migration
      • ACLs in Federated Datacenters
        • Overview
        • Kubernetes
        • JWT
        • OIDC
        • AWS IAM
    • Encryption
      • Overview
      • Core
      • Network Infrastructure Automation
    • Overview
    • Admin Partitions
    • Audit Logging
    • Automated Backups
    • Automated Upgrades
    • Enhanced Read Scalability
    • Single sign-on - OIDC
    • Redundancy Zones
    • Advanced Federation
    • Network Segments
    • Namespaces
    • NIA with TFE
    • Sentinel
      • Overview
      • FAQ
    • Overview
    • Improving Consul Resilience
    • Anti-Entropy
    • Consensus Protocol
    • Gossip Protocol
    • Jepsen Testing
    • Network Coordinates
    • Consul Integration Program
    • NIA Integration Program
    • Vault Integration
    • Proxy Integration
  • Consul Tools
    • Overview
    • Compatibility Promise
    • Specific Version Details
      • Overview
      • General Process
      • Upgrading to 1.2.4
      • Upgrading to 1.6.9
      • Upgrading to 1.8.13
      • Upgrading to 1.10.0
    • Common Error Messages
    • FAQ
    • Overview
      • v1.11.x
      • v1.10.x
      • v1.9.x
      • v0.3.x
      • v0.2.x
      • v0.1.x
      • v0.4.x
      • v0.3.x
      • v0.2.x
      • v0.6.x
      • v0.5.x
    • Overview
    • ACL
  • Guides
Type '/' to Search

»Manually Bootstrapping a Datacenter

When deploying Consul to a datacenter for the first time, there is an initial bootstrapping that must be done. As of Consul 0.4, an automatic bootstrapping is available and is the recommended approach. However, older versions only support a manual bootstrap that is documented here.

Generally, the first nodes that are started are the server nodes. Remember that an agent can run in both client and server mode. Server nodes are responsible for running the consensus protocol, and storing the cluster state. The client nodes are mostly stateless and rely on the server nodes, so they can be started easily.

Manual bootstrapping requires that the first server that is deployed in a new datacenter provide the -bootstrap configuration option. This option allows the server to assert leadership of the cluster without agreement from any other server. This is necessary because at this point, there are no other servers running in the datacenter! Lets call this first server Node A. When starting Node A something like the following will be logged:

2014/02/22 19:23:32 [INFO] consul: cluster leadership acquired
2014/02/22 19:23:32 [INFO] consul: cluster leadership acquired

Once Node A is running, we can start the next set of servers. There is a deployment table that covers various options, but it is recommended to have 3 or 5 total servers per datacenter. A single server deployment is highly discouraged as data loss is inevitable in a failure scenario. We start the next servers without specifying -bootstrap. This is critical, since only one server should ever be running in bootstrap mode. Once Node B and Node C are started, you should see a message to the effect of:

[WARN] raft: EnableSingleNode disabled, and no known peers. Aborting election.
[WARN] raft: EnableSingleNode disabled, and no known peers. Aborting election.

This indicates that the node is not in bootstrap mode, and it will not elect itself as leader. We can now join these machines together. Since a join operation is symmetric it does not matter which node initiates it. From Node B and Node C you can do the following:

$ consul join <Node A Address>
Successfully joined cluster by contacting 1 nodes.
$ consul join <Node A Address>
Successfully joined cluster by contacting 1 nodes.

Alternatively, from Node A you can do the following:

$ consul join <Node B Address> <Node C Address>
Successfully joined cluster by contacting 2 nodes.
$ consul join <Node B Address> <Node C Address>
Successfully joined cluster by contacting 2 nodes.

Once the join is successful, Node A should output something like:

[INFO] raft: Added peer 127.0.0.2:8300, starting replication
....
[INFO] raft: Added peer 127.0.0.3:8300, starting replication
[INFO] raft: Added peer 127.0.0.2:8300, starting replication
....
[INFO] raft: Added peer 127.0.0.3:8300, starting replication

As a sanity check, the consul info command is a useful tool. It can be used to verify raft.num_peers is now 2, and you can view the latest log index under raft.last_log_index. When running consul info on the followers, you should see raft.last_log_index converge to the same value as the leader begins replication. That value represents the last log entry that has been stored on disk.

This indicates that Node B and Node C have been added as peers. At this point, all three nodes see each other as peers, Node A is the leader, and replication should be working.

The final step is to remove the -bootstrap flag. This is important since we don't want the node to be able to make unilateral decisions in the case of a failure of the other two nodes. To do this, we send a SIGINT to Node A to allow it to perform a graceful leave. Then we remove the -bootstrap flag and restart the node. The node will need to rejoin the cluster, since the graceful exit leaves the cluster. Any transactions that took place while Node A was offline will be replicated and the node will catch up.

Now that the servers are all started and replicating to each other, all the remaining clients can be joined. Clients are much easier, as they can be started and perform a join against any existing node. All nodes participate in a gossip protocol to perform basic discovery, so clients will automatically find the servers and register themselves.

If you accidentally start another server with the flag set, do not fret. Shutdown the node, and remove the raft/ folder from the data directory. This will remove the bad state caused by being in -bootstrap mode. Then restart the node and join the cluster normally.

github logoEdit this page
IntroGuidesDocsCommunityPrivacySecurityBrandConsent Manager