June 20-22 Announcing HashiConf Europe full schedule: keynotes, sessions, labs & more Register Now
  • Overview
    • Consul on Kubernetes
    • Control access with Consul API Gateway
    • Discover Services with Consul
    • Enforce Zero Trust Networking with Consul
    • Load Balancing with Consul
    • Manage Traffic with Consul
    • Multi-Platform Service Mesh with Consul
    • Network Infrastructure Automation with Consul
    • Observability with Consul
  • Enterprise
  • Tutorials
  • Docs
  • API
  • CLI
  • Community
GitHub
Download
Try HCP Consul
    • v1.12.x (latest)
    • v1.11.x
    • v1.10.x
    • v1.9.x
    • v1.8.x
    • Overview
      • Overview
      • What is a Service Mesh?
      • Overview
      • Chef, Puppet, etc.
      • Nagios
      • SkyDNS
      • SmartStack
      • Serf
      • Eureka
      • Istio
      • Envoy and Other Proxies
      • Custom Solutions
    • Overview
    • Manual Bootstrap
    • Consul Agent
    • Glossary
    • Required Ports
    • Bootstrapping a Datacenter
    • Cloud Auto-join
    • Server Performance
    • Kubernetes
  • API
  • Commands (CLI)
    • Register Services - Service Definitions
    • Find Services - DNS Interface
    • Monitor Services - Check Definitions
    • Overview
    • How Service Mesh Works
    • Configuration
      • Overview
      • Ingress Gateway
      • Mesh
      • Exported Services
      • Proxy Defaults
      • Service Defaults
      • Service Intentions
      • Service Resolver
      • Service Router
      • Service Splitter
      • Terminating Gateway
      • Overview
      • Envoy
      • Built-in Proxy
      • Proxy Integration
      • Managed (Deprecated)
      • Overview
      • Proxy Service Registration
      • Sidecar Service Registration
    • Service-to-service permissions - Intentions
    • Service-to-service permissions - Intentions (Legacy Mode)
    • Transparent Proxy
      • Overview
      • UI Visualization
      • Overview
      • Discovery Chain
    • Connectivity Tasks
    • Distributed Tracing
      • Overview
        • WAN Federation
        • Enabling Service-to-service Traffic Across Datacenters
        • Enabling Service-to-service Traffic Across Admin Partitions
      • Ingress Gateways
      • Terminating Gateways
    • Nomad
    • Kubernetes
      • Overview
      • Go Integration
      • Overview
      • Built-In CA
      • Vault
      • ACM Private CA
    • Develop and Debug
    • Security
    • Overview
    • Installation
    • Technical Specifications
    • Common Errors
    • Upgrades
    • Overview
    • Architecture
      • Installing Consul on Kubernetes
      • Installing Consul K8s CLI
        • Minikube
        • Kind
        • AKS (Azure)
        • EKS (AWS)
        • GKE (Google Cloud)
        • Red Hat OpenShift
        • Self Hosted Kubernetes
        • Consul Clients Outside Kubernetes
        • Consul Servers Outside Kubernetes
        • Single Consul Datacenter in Multiple Kubernetes Clusters
        • Consul Enterprise
        • Overview
        • Federation Between Kubernetes Clusters
        • Federation Between VMs and Kubernetes
        • Overview
        • Systems Integration
          • Overview
          • Bootstrap Token
          • Enterprise License
          • Gossip Encryption Key
          • Partition Token
          • Replication Token
          • Server TLS
          • Service Mesh Certificates
          • Snapshot Agent Config
        • WAN Federation
      • Compatibility Matrix
      • Overview
      • Transparent Proxy
      • Ingress Gateways
      • Terminating Gateways
      • Ingress Controllers
      • Configuring a Connect CA Provider
      • Health Checks
        • Metrics
    • Service Sync
      • Overview
      • Upgrade An Existing Cluster to CRDs
    • Annotations and Labels
    • Consul DNS
      • Upgrading Consul on Kubernetes
      • Upgrading Consul K8s CLI
      • Uninstall
      • Certificate Rotation
      • Gossip Encryption Key Rotation
      • Configure TLS on an Existing Cluster
      • Common Error Messages
      • FAQ
    • Helm Chart Configuration
    • Consul K8s CLI Reference
    • Overview
    • Requirements
    • Task Resource Usage
      • Installation
      • Secure Configuration
      • Migrate Existing Tasks
      • Installation
      • Secure Configuration
      • ACL Controller
    • Architecture
    • Consul Enterprise
    • Configuration Reference
    • Overview
      • Installation
      • Requirements
      • Configure
      • Run Consul-Terraform-Sync
    • Architecture
      • Overview
      • Status
      • Tasks
      • Overview
      • task
    • Configuration
    • Tasks
    • Terraform Modules
      • Overview
      • License
      • Terraform Cloud Driver
      • Overview
      • Terraform
      • Terraform Cloud
    • Compatibility
    • Consul KV
    • Sessions
    • Watches
    • Overview
      • General
      • CLI Reference
      • Configuration Reference
    • Configuration Entries
    • Telemetry
    • Sentinel
    • RPC
    • Overview
      • ACL System Overview
      • Tokens
      • Policies
      • Roles
      • Rules Reference
      • Legacy Mode
      • Token Migration
      • ACLs in Federated Datacenters
        • Overview
        • Kubernetes
        • JWT
        • OIDC
        • AWS IAM
    • Encryption
      • Overview
      • Core
      • Network Infrastructure Automation
    • Overview
    • Admin Partitions
    • Audit Logging
    • Automated Backups
    • Automated Upgrades
    • Enhanced Read Scalability
    • Single sign-on - OIDC
    • Redundancy Zones
    • Advanced Federation
    • Network Segments
    • Namespaces
    • NIA with TFE
    • Sentinel
      • Overview
      • FAQ
    • Overview
    • Improving Consul Resilience
    • Anti-Entropy
    • Consensus Protocol
    • Gossip Protocol
    • Jepsen Testing
    • Network Coordinates
    • Consul Integration Program
    • NIA Integration Program
    • Vault Integration
    • Proxy Integration
  • Consul Tools
    • Overview
    • Compatibility Promise
    • Specific Version Details
      • Overview
      • General Process
      • Upgrading to 1.2.4
      • Upgrading to 1.6.9
      • Upgrading to 1.8.13
      • Upgrading to 1.10.0
    • Common Error Messages
    • FAQ
    • Overview
      • v1.11.x
      • v1.10.x
      • v1.9.x
      • v0.1.x
      • v0.2.x
      • v0.4.x
      • v0.3.x
      • v0.2.x
      • v0.5.x
      • v0.6.0-beta
    • Overview
    • ACL
  • Guides
Type '/' to Search

»Network Segments

Enterprise

This feature requires version 0.9.3+ of self-managed Consul Enterprise. Refer to the enterprise feature matrix for additional information.

Consul requires full connectivity between all agents (servers and clients) in a datacenter within a given LAN gossip pool. By default, all Consul agents will be a part of one shared Serf LAN gossip pool known as the <default> network segment, thus requiring full mesh connectivity within the datacenter.

Consul datacenter default agent connectivity: one network segment

In some environments, full connectivity between all agents is not possible—known as a "segmented network". This is usually the result of business policies enforced through network rules or firewalls. To use Consul in a segmented network, you must break up the LAN gossip pool along network communication boundaries into separate "network segments". Network segments are isolated LAN gossip pools that only require full connectivity between agent members on the same segment.

Consul datacenter agent connectivity with network segments

To get started with network segments you can review the tutorial on HashiCorp Learn for Network Segments.

Info: Network segments enable you to operate a Consul datacenter without full mesh (LAN) connectivity between agents. To federate multiple Consul datacenters without full mesh (WAN) connectivity between all server agents in all datacenters, use Network Areas (Enterprise).

»Consul Networking Models

To help set context for this feature, it is useful to understand the various Consul networking models and their capabilities.

Cluster: A set of Consul servers forming a Raft quorum along with a collection of Consul clients, all set to the same datacenter, and joined together to form what we will call a "local cluster". Consul clients discover the Consul servers in their local cluster through the gossip mechanism and make RPC requests to them. LAN Gossip (OSS) is an open intra-cluster networking model, and Network Segments (Enterprise) creates multiple segments within one cluster.

Federated Cluster: A set of connected clusters, each representing a unique Consul “datacenter”. These Consul servers are federated together over the WAN. Consul clients make use of resources in federated clusters by forwarding RPCs through the Consul servers in their local cluster, but they never interact with remote Consul servers directly. There are currently two inter-cluster network models which can be viewed on HashiCorp Learn: WAN gossip (OSS) and Network Areas (Enterprise).

LAN Gossip Pool: A set of Consul agents that have full mesh connectivity among themselves, and use Serf to maintain a shared view of the members of the pool for different purposes, like finding a Consul server in a local cluster, or finding servers in a remote cluster. A segmented LAN Gossip Pool limits a group of agents to only connect with the agents in its segment.

»Network Segments Configuration

»Server Configuration

Server agents are members of all segments. The datacenter includes a <default> segment, as well as additional segments defined in the segments server agent configuration option. Each additional segment is defined by:

  • a non-empty name
  • a unique port
  • optionally, a custom bind and advertise address for the additional segment's Serf LAN listener on the server

Note: Prior to Consul 1.7.3, a Consul server agent configured with too many network segments may not be able to start due to limitations in Serf.

»Example Server Configuration

The following server agent configuration will create two user-defined network segments: alpha and beta.

Example network segments configuration for server agents
Example network segments configuration for server agents
HCL
  • HCL
  • JSON
segments = [
  {
    name = "alpha"
    bind = "{{GetPrivateIP}}"
    advertise = "{{GetPrivateIP}}"
    port = 8303
  },
  {
    name = "beta"
    bind = "{{GetPrivateIP}}"
    advertise = "{{GetPrivateIP}}"
    port = 8304
  }
]
segments = [
  {
    name = "alpha"
    bind = "{{GetPrivateIP}}"
    advertise = "{{GetPrivateIP}}"
    port = 8303
  },
  {
    name = "beta"
    bind = "{{GetPrivateIP}}"
    advertise = "{{GetPrivateIP}}"
    port = 8304
  }
]
{
  "segments": [
    {
      "name": "alpha",
      "bind": "{{GetPrivateIP}}",
      "advertise": "{{GetPrivateIP}}",
      "port": 8303
    },
    {
      "name": "beta",
      "bind": "{{GetPrivateIP}}",
      "advertise": "{{GetPrivateIP}}",
      "port": 8304
    }
  ]
}
{
  "segments": [
    {
      "name": "alpha",
      "bind": "{{GetPrivateIP}}",
      "advertise": "{{GetPrivateIP}}",
      "port": 8303
    },
    {
      "name": "beta",
      "bind": "{{GetPrivateIP}}",
      "advertise": "{{GetPrivateIP}}",
      "port": 8304
    }
  ]
}

The server agent configuration options relevant to network segments are:

  • ports.serf_lan: The Serf LAN port on this server for the <default> network segment's gossip pool.
  • segments: A list of user-defined network segments on this server, including their names and Serf LAN ports.

»Client Configuration

Each client agent can only be a member of one segment at a time. This will be the <default> segment unless otherwise specified in the agent's segment agent configuration option.

»Join a Client to a Segment

For a client agent to join the Consul datacenter, it must connect to another agent (client or server) within its configured segment.

Clients A and B specify the same segment S. Client B is already joined to the segment S LAN gossip pool. Client A wants to join via Client B. In order to do so, Client A must connect to Client B's configured Serf LAN port.

There are several methods to specify the port used by the join operation, listed in order of precedence:

  1. Specify an explicit port in the join address. This can be done at the CLI when starting the agent (e.g., consul agent -retry-join "client-b-address:8303"), or in the agent's configuration using the retry-join option. This method is not compatible with cloud auto-join.

  2. Specify an alternate Serf LAN port for the agent. This can be done at the CLI when starting the agent (e.g., consul agent -retry-join "client-b-address" -serf-lan-port 8303), or in the agent's configuration using the serf_lan option. When a Serf LAN port is not explicitly specified in the join address, the agent will attempt to join the target host at the Serf LAN port specified in CLI or agent configuration.

  3. Use the default Serf LAN port (8301). The agent will attempt to join the target host on port 8301.

Agents within a segment can use different port numbers for their Serf LAN port. For example, on the <default> segment, Server S can use port 8301, Client A can use 8303, and Client B can use 8304. However, if an agent wishes to join a segment via an agent that uses a different port number, the target agent's Serf LAN port must be specified in the join address (method 1 above).

»Example Client Configuration

The following client agent configuration will cause the agent to:

  • Open a Serf LAN listener port on 8303.
  • Attempt to join the cluster via servers on port 8303 (since an alternate port is not specified in the retry_join addresses).
Example network segment configuration for client agents
Example network segment configuration for client agents
HCL
  • HCL
  • JSON
node_name = "consul-client1"
retry_join = ["consul-server1", "consul-server2", "consul-server3"]
segment = "alpha"
ports = {
  serf_lan = 8303
}
node_name = "consul-client1"
retry_join = ["consul-server1", "consul-server2", "consul-server3"]
segment = "alpha"
ports = {
  serf_lan = 8303
}
{
  "node_name": "consul-client1",
  "retry_join": ["consul-server1", "consul-server2", "consul-server3"],
  "segment": "alpha",
  "ports": {
    "serf_lan": 8303
  }
}
{
  "node_name": "consul-client1",
  "retry_join": ["consul-server1", "consul-server2", "consul-server3"],
  "segment": "alpha",
  "ports": {
    "serf_lan": 8303
  }
}

The client agent configuration options relevant to network segments are:

  • segment: The name of the network segment this client agent belongs to.
  • ports.serf_lan: Serf LAN port for the above segment on this client. This is not required to match the configured Serf LAN port for other agents on this segment.
  • retry_join or start_join: A list of agent addresses to join when starting. Ensure the correct Serf LAN port for this segment is used when joining the LAN gossip pool using one of the available configuration methods.
github logoEdit this page
IntroGuidesDocsCommunityPrivacySecurityBrandConsent Manager