»Network Segments

Consul requires full connectivity between all agents (servers and clients) in a datacenter within a given LAN gossip pool. By default, all Consul agents will be a part of one shared Serf LAN gossip pool known as the <default> network segment, thus requiring full mesh connectivity within the datacenter.

In some environments, full connectivity between all agents is not possible—known as a "segmented network". This is usually the result of business policies enforced through network rules or firewalls. To use Consul in a segmented network, you must break up the LAN gossip pool along network communication boundaries into separate "network segments". Network segments are isolated LAN gossip pools that only require full connectivity between agent members on the same segment.

To get started with network segments you can review the tutorial on HashiCorp Learn for Network Segments.

»Consul Networking Models

To help set context for this feature, it is useful to understand the various Consul networking models and their capabilities.

Cluster: A set of Consul servers forming a Raft quorum along with a collection of Consul clients, all set to the same datacenter, and joined together to form what we will call a "local cluster". Consul clients discover the Consul servers in their local cluster through the gossip mechanism and make RPC requests to them. LAN Gossip (OSS) is an open intra-cluster networking model, and Network Segments (Enterprise) creates multiple segments within one cluster.

Federated Cluster: A set of connected clusters, each representing a unique Consul “datacenter”. These Consul servers are federated together over the WAN. Consul clients make use of resources in federated clusters by forwarding RPCs through the Consul servers in their local cluster, but they never interact with remote Consul servers directly. There are currently two inter-cluster network models which can be viewed on HashiCorp Learn: WAN gossip (OSS) and Network Areas (Enterprise).

LAN Gossip Pool: A set of Consul agents that have full mesh connectivity among themselves, and use Serf to maintain a shared view of the members of the pool for different purposes, like finding a Consul server in a local cluster, or finding servers in a remote cluster. A segmented LAN Gossip Pool limits a group of agents to only connect with the agents in its segment.

»Network Segments Configuration

»Server Configuration

Server agents are members of all segments. The datacenter includes a <default> segment, as well as additional segments defined in the segments server agent configuration option. Each additional segment is defined by:

• a non-empty name
• a unique port
• optionally, a custom bind and advertise address for the additional segment's Serf LAN listener on the server

»Example Server Configuration

The following server agent configuration will create two user-defined network segments: alpha and beta.

segments = [
  {
    name = "alpha"
    bind = "{{GetPrivateIP}}"
    advertise = "{{GetPrivateIP}}"
    port = 8303
  },
  {
    name = "beta"
    bind = "{{GetPrivateIP}}"
    advertise = "{{GetPrivateIP}}"
    port = 8304
  }
]

segments = [
  {
    name = "alpha"
    bind = "{{GetPrivateIP}}"
    advertise = "{{GetPrivateIP}}"
    port = 8303
  },
  {
    name = "beta"
    bind = "{{GetPrivateIP}}"
    advertise = "{{GetPrivateIP}}"
    port = 8304
  }
]

{
  "segments": [
    {
      "name": "alpha",
      "bind": "{{GetPrivateIP}}",
      "advertise": "{{GetPrivateIP}}",
      "port": 8303
    },
    {
      "name": "beta",
      "bind": "{{GetPrivateIP}}",
      "advertise": "{{GetPrivateIP}}",
      "port": 8304
    }
  ]
}

{
  "segments": [
    {
      "name": "alpha",
      "bind": "{{GetPrivateIP}}",
      "advertise": "{{GetPrivateIP}}",
      "port": 8303
    },
    {
      "name": "beta",
      "bind": "{{GetPrivateIP}}",
      "advertise": "{{GetPrivateIP}}",
      "port": 8304
    }
  ]
}

The server agent configuration options relevant to network segments are:

• ports.serf_lan: The Serf LAN port on this server for the <default> network segment's gossip pool.
• segments: A list of user-defined network segments on this server, including their names and Serf LAN ports.

»Client Configuration

Each client agent can only be a member of one segment at a time. This will be the <default> segment unless otherwise specified in the agent's segment agent configuration option.

»Join a Client to a Segment

For a client agent to join the Consul datacenter, it must connect to another agent (client or server) within its configured segment.

There are several methods to specify the port used by the join operation, listed in order of precedence:

1. Specify an explicit port in the join address. This can be done at the CLI when starting the agent (e.g., consul agent -retry-join "client-b-address:8303"), or in the agent's configuration using the retry-join option. This method is not compatible with cloud auto-join.

2. Specify an alternate Serf LAN port for the agent. This can be done at the CLI when starting the agent (e.g., consul agent -retry-join "client-b-address" -serf-lan-port 8303), or in the agent's configuration using the serf_lan option. When a Serf LAN port is not explicitly specified in the join address, the agent will attempt to join the target host at the Serf LAN port specified in CLI or agent configuration.

3. Use the default Serf LAN port (8301). The agent will attempt to join the target host on port 8301.

»Example Client Configuration

The following client agent configuration will cause the agent to:

• Open a Serf LAN listener port on 8303.
• Attempt to join the cluster via servers on port 8303 (since an alternate port is not specified in the retry_join addresses).

node_name = "consul-client1"
retry_join = ["consul-server1", "consul-server2", "consul-server3"]
segment = "alpha"
ports = {
  serf_lan = 8303
}

node_name = "consul-client1"
retry_join = ["consul-server1", "consul-server2", "consul-server3"]
segment = "alpha"
ports = {
  serf_lan = 8303
}

{
  "node_name": "consul-client1",
  "retry_join": ["consul-server1", "consul-server2", "consul-server3"],
  "segment": "alpha",
  "ports": {
    "serf_lan": 8303
  }
}

{
  "node_name": "consul-client1",
  "retry_join": ["consul-server1", "consul-server2", "consul-server3"],
  "segment": "alpha",
  "ports": {
    "serf_lan": 8303
  }
}

The client agent configuration options relevant to network segments are:

• segment: The name of the network segment this client agent belongs to.
• ports.serf_lan: Serf LAN port for the above segment on this client. This is not required to match the configured Serf LAN port for other agents on this segment.
• retry_join or start_join: A list of agent addresses to join when starting. Ensure the correct Serf LAN port for this segment is used when joining the LAN gossip pool using one of the available configuration methods.