Blog HCP Consul on Azure goes GA, plus more Consul news from HashiConf EU Read more
  • Overview
    • Consul on Kubernetes
    • Control access with Consul API Gateway
    • Discover Services with Consul
    • Enforce Zero Trust Networking with Consul
    • Load Balancing with Consul
    • Manage Traffic with Consul
    • Multi-Platform Service Mesh with Consul
    • Network Infrastructure Automation with Consul
    • Observability with Consul
  • Enterprise
  • Tutorials
  • Docs
  • API
  • CLI
  • Community
GitHub
Download
Try HCP Consul
    • v1.12.x (latest)
    • v1.11.x
    • v1.10.x
    • v1.9.x
    • v1.8.x
    • Overview
      • Overview
      • What is a Service Mesh?
      • Overview
      • Chef, Puppet, etc.
      • Nagios
      • SkyDNS
      • SmartStack
      • Serf
      • Eureka
      • Istio
      • Envoy and Other Proxies
      • Custom Solutions
    • Overview
    • Manual Bootstrap
    • Consul Agent
    • Glossary
    • Required Ports
    • Bootstrapping a Datacenter
    • Cloud Auto-join
    • Server Performance
    • Kubernetes
  • API
  • Commands (CLI)
    • Register Services - Service Definitions
    • Find Services - DNS Interface
    • Monitor Services - Check Definitions
    • Overview
    • How Service Mesh Works
    • Configuration
      • Overview
      • Ingress Gateway
      • Mesh
      • Exported Services
      • Proxy Defaults
      • Service Defaults
      • Service Intentions
      • Service Resolver
      • Service Router
      • Service Splitter
      • Terminating Gateway
      • Overview
      • Envoy
      • Built-in Proxy
      • Proxy Integration
      • Managed (Deprecated)
      • Overview
      • Proxy Service Registration
      • Sidecar Service Registration
    • Service-to-service permissions - Intentions
    • Service-to-service permissions - Intentions (Legacy Mode)
    • Transparent Proxy
      • Overview
      • UI Visualization
      • Overview
      • Discovery Chain
    • Connectivity Tasks
    • Distributed Tracing
      • Overview
        • WAN Federation
        • Enabling Service-to-service Traffic Across Datacenters
        • Enabling Service-to-service Traffic Across Admin Partitions
      • Ingress Gateways
      • Terminating Gateways
      • What is Cluster Peering
      • Create and Manage Peering Connections
      • Cluster Peering on Kubernetes
    • Nomad
    • Kubernetes
      • Overview
      • Go Integration
      • Overview
      • Built-In CA
      • Vault
      • ACM Private CA
    • Develop and Debug
    • Security
    • Overview
    • Installation
    • Technical Specifications
    • Common Errors
    • Upgrades
    • Overview
    • Architecture
      • Installing Consul on Kubernetes
      • Installing Consul K8s CLI
        • Minikube
        • Kind
        • AKS (Azure)
        • EKS (AWS)
        • GKE (Google Cloud)
        • Red Hat OpenShift
        • Self Hosted Kubernetes
        • Consul Clients Outside Kubernetes
        • Consul Servers Outside Kubernetes
        • Single Consul Datacenter in Multiple Kubernetes Clusters
        • Consul Enterprise
        • Overview
        • Federation Between Kubernetes Clusters
        • Federation Between VMs and Kubernetes
        • Overview
        • Systems Integration
          • Overview
          • Bootstrap Token
          • Enterprise License
          • Gossip Encryption Key
          • Partition Token
          • Replication Token
          • Server TLS
          • Service Mesh Certificates
          • Snapshot Agent Config
          • Webhook Certificates
        • WAN Federation
      • Overview
      • Transparent Proxy
      • Ingress Gateways
      • Terminating Gateways
      • Ingress Controllers
      • Configuring a Connect CA Provider
      • Health Checks
        • Metrics
    • Service Sync
      • Overview
      • Upgrade An Existing Cluster to CRDs
    • Annotations and Labels
    • Consul DNS
      • Upgrading Consul on Kubernetes
      • Upgrading Consul K8s CLI
      • Uninstall
      • Certificate Rotation
      • Gossip Encryption Key Rotation
      • Configure TLS on an Existing Cluster
      • Common Error Messages
      • FAQ
    • Compatibility Matrix
    • Helm Chart Configuration
    • Consul K8s CLI Reference
    • Overview
    • Requirements
    • Task Resource Usage
      • Installation
      • Secure Configuration
      • Migrate Existing Tasks
      • Installation
      • Secure Configuration
      • ACL Controller
    • Architecture
    • Consul Enterprise
    • Configuration Reference
    • Overview
    • Register Lambda Functions
    • Invoke Lambda Functions
    • Overview
      • Installation
      • Requirements
      • Configure
      • Run Consul-Terraform-Sync
    • Architecture
      • Overview
      • Status
      • Tasks
      • Health
      • Overview
      • task
      • start
    • Configuration
    • Tasks
    • Terraform Modules
      • Overview
      • License
      • Terraform Cloud Driver
      • Overview
      • Terraform
      • Terraform Cloud
    • Compatibility
    • Consul KV
    • Sessions
    • Watches
    • Overview
      • General
      • CLI Reference
      • Configuration Reference
    • Configuration Entries
    • Telemetry
    • Sentinel
    • RPC
    • Overview
      • ACL System Overview
      • Tokens
      • Policies
      • Roles
      • Rules Reference
      • Legacy Mode
      • Token Migration
      • ACLs in Federated Datacenters
        • Overview
        • Kubernetes
        • JWT
        • OIDC
        • AWS IAM
    • Encryption
      • Overview
      • Core
      • Network Infrastructure Automation
    • Overview
    • Admin Partitions
    • Audit Logging
    • Automated Backups
    • Automated Upgrades
    • Enhanced Read Scalability
    • Single sign-on - OIDC
    • Redundancy Zones
    • Advanced Federation
    • Network Segments
    • Namespaces
    • NIA with TFE
    • Sentinel
      • Overview
      • FAQ
    • Overview
    • Improving Consul Resilience
    • Anti-Entropy
    • Consensus Protocol
    • Gossip Protocol
    • Jepsen Testing
    • Network Coordinates
    • Consul Integration Program
    • NIA Integration Program
    • Vault Integration
    • Proxy Integration
  • Consul Tools
    • Overview
    • Compatibility Promise
    • Specific Version Details
      • Overview
      • General Process
      • Upgrading to 1.2.4
      • Upgrading to 1.6.9
      • Upgrading to 1.8.13
      • Upgrading to 1.10.0
    • Common Error Messages
    • FAQ
    • Overview
      • v1.11.x
      • v1.10.x
      • v1.9.x
      • v0.3.x
      • v0.2.x
      • v0.1.x
      • v0.4.x
      • v0.3.x
      • v0.2.x
      • v0.6.x
      • v0.5.x
    • Overview
    • ACL
  • Guides
Type '/' to Search

»ACLs in Federated Datacenters

This topic describes how to set up Consul's access control list (ACL) system in cluster deployments that span multiple data centers. This documentation is applicable to new clusters rather than existing clusters.

»Requirements

Consul versions 1.4.0 and later

»Configure ACLs in the Primary Datacenter

In a federated Consul deployment, one of the datacenters is marked as the primary datacenter. The acl configuration block should be added to the primary datacenter server's configuration file as shown in the following example.

See the ACL Config Stanza for more detailed descriptions of each option.

Versions before 1.11.0: The initial_management token was called the master token in versions prior to 1.11.0

ACL Configuration in Primary
ACL Configuration in Primary
HCL
  • HCL
  • JSON
bootstrap_expect =  3
primary_datacenter =  "PRIMARY_DATACENTER_VALUE"
acl = {
 enabled =  true
 default_policy = "deny"
 down_policy = "deny"
 enable_token_persistence = true
 enable_token_replication = true
 tokens = {
   initial_management =  "ACL_MANAGEMENT_TOKEN"
   agent = "YOUR_ACL_AGENT_TOKEN"
  }
}

bootstrap_expect =  3
primary_datacenter =  "PRIMARY_DATACENTER_VALUE"
acl = {
 enabled =  true
 default_policy = "deny"
 down_policy = "deny"
 enable_token_persistence = true
 enable_token_replication = true
 tokens = {
   initial_management =  "ACL_MANAGEMENT_TOKEN"
   agent = "YOUR_ACL_AGENT_TOKEN"
  }
}

{
  "bootstrap_expect": N,
  "primary_datacenter": "PRIMARY_DATACENTER_VALUE",
  "acl": {
    "enabled": true,
    "default_policy": "deny",
    "down_policy": "deny",
    "enable_token_persistence": true,
    "enable_token_replication": true,
    "tokens": {
      "initial_management": "ACL_MANAGEMENT_TOKEN",
      "agent": "ACL_AGENT_TOKEN"
    }
  }
}
{
  "bootstrap_expect": N,
  "primary_datacenter": "PRIMARY_DATACENTER_VALUE",
  "acl": {
    "enabled": true,
    "default_policy": "deny",
    "down_policy": "deny",
    "enable_token_persistence": true,
    "enable_token_replication": true,
    "tokens": {
      "initial_management": "ACL_MANAGEMENT_TOKEN",
      "agent": "ACL_AGENT_TOKEN"
    }
  }
}

Warning: Note that most enterprise deployments have security requirements that prevent specifying tokens in configuration files. The enable_token_persistence flag is also set in the configuration example so that the token is stored to disk in the agent's data directory. Any future changes to the token that are made through the API will be persisted to the same location, and the value in the config file will be ignored.

The ACL agent token can also be set using the consul acl set-agent-token CLI as shown below.

$ consul acl set-agent-token agent "<agent token here>"
$ consul acl set-agent-token agent "<agent token here>"

»Configure Servers in Secondary Datacenters

Servers in secondary data centers must be configured to point to the primary data center as shown in the following example. Secondary data centers also need the ACL replication token provided to them.

»Create the replication token for ACL Management

Replication tokens are needed for ACL token replication and to create both configuration entries and auth methods in connected secondary datacenters.

Replication tokens require the following permissions:

  • acl = "write": The permission allows you to replicate tokens.
  • operator = "write": This permission enables the proxy-default configuration entries to be replicated and enables CA certificate signing in the secondary datacenter.
  • policy = "read" and intentions = "read" in the service_prefix field: These permissions enable service-default configuration entries, CA, and intention data to be replicated for all services.
Replication Token Policy
Replication Token Policy
HCL
  • HCL
acl = "write"
operator = "write"
service_prefix "" {
  policy = "read"
  intentions = "read"
}
replication-policy.hcl
acl = "write"
operator = "write"
service_prefix "" {
  policy = "read"
  intentions = "read"
}

Create a replication policy with the following command:

$ consul acl policy create -name replication -rules @replication-policy.hcl
$ consul acl policy create -name replication -rules @replication-policy.hcl

Use your newly created policy to create the replication token.

$ consul acl token create -description "replication token" -policy-name replication
$ consul acl token create -description "replication token" -policy-name replication

»Configure the replication token in Secondary Datacenters

Add the replication token generated above, to the ACL stanza in secondary datacenters.

ACL Configuration in Secondaries
ACL Configuration in Secondaries
HCL
  • HCL
  • JSON
primary_datacenter = "PRIMARY_DATACENTER_NAME"
acl = {
   enabled = true
   default_policy =  "deny"
   down_policy =  "deny"
   tokens = {
      agent =  "ACL_AGENT_TOKEN"
      replication = "ACL_REPLICATION_TOKEN"
   }
}
primary_datacenter = "PRIMARY_DATACENTER_NAME"
acl = {
   enabled = true
   default_policy =  "deny"
   down_policy =  "deny"
   tokens = {
      agent =  "ACL_AGENT_TOKEN"
      replication = "ACL_REPLICATION_TOKEN"
   }
}
{
  "primary_datacenter": "PRIMARY_DATACENTER_VALUE",
  "acl": {
    "enabled": true,
    "default_policy": "deny",
    "down_policy": "deny",
    "tokens": {
      "agent": "ACL_AGENT_TOKEN",
      "replication": "ACL_REPLICATION_TOKEN"
    }
  }
}
{
  "primary_datacenter": "PRIMARY_DATACENTER_VALUE",
  "acl": {
    "enabled": true,
    "default_policy": "deny",
    "down_policy": "deny",
    "tokens": {
      "agent": "ACL_AGENT_TOKEN",
      "replication": "ACL_REPLICATION_TOKEN"
    }
  }
}

Warning: When enabling ACL token replication in secondary datacenters, global tokens already present in the secondary datacenter will be lost. For production environments, consider configuring ACL replication in your initial datacenter bootstrapping process.

Warning: If you are using Consul Enterprise and the Admin Partitions feature, only ACL tokens in the default partition are replicated to other datacenters.

»WAN Join Servers

This step is needed for new federated cluster deployments in order for servers in each federated datacenter to discover each other.

Run the following command from one of the server nodes.

$ consul join -token="ACL_MANAGEMENT_TOKEN" -wan [server 1, server 2, ...]
$ consul join -token="ACL_MANAGEMENT_TOKEN" -wan [server 1, server 2, ...]

»Configure Clients in Secondary Datacenters

When ACLs are enabled, client agents need a special token known as the agent token to perform internal operations. Agent tokens need to have the right policies for node related actions, including registering itself in the catalog, updating node level health checks, and performing anti-entropy syncing.

»Generate Agent ACL Token

ACL Node Identities were introduced in Consul 1.8.1 and enable easily creating agent tokens with appropriately scoped policies.

To generate the ACL token using node identity, run the following command:

$ consul acl token create -node-identity=<NODE_NAME>:<DATACENTER>
$ consul acl token create -node-identity=<NODE_NAME>:<DATACENTER>

»Configure clients to use the ACL agent token

Update the client agents to include the token value from the previous step. Replace the ACL_AGENT_TOKEN value below with the secret ID value from the command output.

ACL Configuration in Client Agents
ACL Configuration in Client Agents
HCL
  • HCL
  • JSON
primary_datacenter = "PRIMARY_DATACENTER_NAME"
acl = {
   enabled = true
   default_policy =  "deny"
   down_policy =  "deny"
   tokens = {
      agent =  "ACL_AGENT_TOKEN"
   }
}
primary_datacenter = "PRIMARY_DATACENTER_NAME"
acl = {
   enabled = true
   default_policy =  "deny"
   down_policy =  "deny"
   tokens = {
      agent =  "ACL_AGENT_TOKEN"
   }
}
{
  "primary_datacenter": "PRIMARY_DATACENTER_VALUE",
  "acl": {
    "enabled": true,
    "default_policy": "deny",
    "down_policy": "deny",
    "tokens": {
      "agent": "ACL_AGENT_TOKEN"
    }
  }
}
{
  "primary_datacenter": "PRIMARY_DATACENTER_VALUE",
  "acl": {
    "enabled": true,
    "default_policy": "deny",
    "down_policy": "deny",
    "tokens": {
      "agent": "ACL_AGENT_TOKEN"
    }
  }
}

Note that client agents have to be restarted for ACL related configuration changes to take effect.

»Summary

After completing the above steps, a federated Consul cluster can be used with ACLs. Refer to ACL Replication Guide for more on this topic.

github logoEdit this page
IntroGuidesDocsCommunityPrivacySecurityBrandConsent Manager