June 20-22 Announcing HashiConf Europe full schedule: keynotes, sessions, labs & more Register Now
  • Overview
    • Consul on Kubernetes
    • Control access with Consul API Gateway
    • Discover Services with Consul
    • Enforce Zero Trust Networking with Consul
    • Load Balancing with Consul
    • Manage Traffic with Consul
    • Multi-Platform Service Mesh with Consul
    • Network Infrastructure Automation with Consul
    • Observability with Consul
  • Enterprise
  • Tutorials
  • Docs
  • API
  • CLI
  • Community
GitHub
Download
Try HCP Consul
    • v1.12.x (latest)
    • v1.11.x
    • v1.10.x
    • v1.9.x
    • v1.8.x
    • Overview
      • Overview
      • What is a Service Mesh?
      • Overview
      • Chef, Puppet, etc.
      • Nagios
      • SkyDNS
      • SmartStack
      • Serf
      • Eureka
      • Istio
      • Envoy and Other Proxies
      • Custom Solutions
    • Overview
    • Manual Bootstrap
    • Consul Agent
    • Glossary
    • Required Ports
    • Bootstrapping a Datacenter
    • Cloud Auto-join
    • Server Performance
    • Kubernetes
  • API
  • Commands (CLI)
    • Register Services - Service Definitions
    • Find Services - DNS Interface
    • Monitor Services - Check Definitions
    • Overview
    • How Service Mesh Works
    • Configuration
      • Overview
      • Ingress Gateway
      • Mesh
      • Exported Services
      • Proxy Defaults
      • Service Defaults
      • Service Intentions
      • Service Resolver
      • Service Router
      • Service Splitter
      • Terminating Gateway
      • Overview
      • Envoy
      • Built-in Proxy
      • Proxy Integration
      • Managed (Deprecated)
      • Overview
      • Proxy Service Registration
      • Sidecar Service Registration
    • Service-to-service permissions - Intentions
    • Service-to-service permissions - Intentions (Legacy Mode)
    • Transparent Proxy
      • Overview
      • UI Visualization
      • Overview
      • Discovery Chain
    • Connectivity Tasks
    • Distributed Tracing
      • Overview
        • WAN Federation
        • Enabling Service-to-service Traffic Across Datacenters
        • Enabling Service-to-service Traffic Across Admin Partitions
      • Ingress Gateways
      • Terminating Gateways
    • Nomad
    • Kubernetes
      • Overview
      • Go Integration
      • Overview
      • Built-In CA
      • Vault
      • ACM Private CA
    • Develop and Debug
    • Security
    • Overview
    • Installation
    • Technical Specifications
    • Common Errors
    • Upgrades
    • Overview
    • Architecture
      • Installing Consul on Kubernetes
      • Installing Consul K8s CLI
        • Minikube
        • Kind
        • AKS (Azure)
        • EKS (AWS)
        • GKE (Google Cloud)
        • Red Hat OpenShift
        • Self Hosted Kubernetes
        • Consul Clients Outside Kubernetes
        • Consul Servers Outside Kubernetes
        • Single Consul Datacenter in Multiple Kubernetes Clusters
        • Consul Enterprise
        • Overview
        • Federation Between Kubernetes Clusters
        • Federation Between VMs and Kubernetes
        • Overview
        • Systems Integration
          • Overview
          • Bootstrap Token
          • Enterprise License
          • Gossip Encryption Key
          • Partition Token
          • Replication Token
          • Server TLS
          • Service Mesh Certificates
          • Snapshot Agent Config
        • WAN Federation
      • Compatibility Matrix
      • Overview
      • Transparent Proxy
      • Ingress Gateways
      • Terminating Gateways
      • Ingress Controllers
      • Configuring a Connect CA Provider
      • Health Checks
        • Metrics
    • Service Sync
      • Overview
      • Upgrade An Existing Cluster to CRDs
    • Annotations and Labels
    • Consul DNS
      • Upgrading Consul on Kubernetes
      • Upgrading Consul K8s CLI
      • Uninstall
      • Certificate Rotation
      • Gossip Encryption Key Rotation
      • Configure TLS on an Existing Cluster
      • Common Error Messages
      • FAQ
    • Helm Chart Configuration
    • Consul K8s CLI Reference
    • Overview
    • Requirements
    • Task Resource Usage
      • Installation
      • Secure Configuration
      • Migrate Existing Tasks
      • Installation
      • Secure Configuration
      • ACL Controller
    • Architecture
    • Consul Enterprise
    • Configuration Reference
    • Overview
      • Installation
      • Requirements
      • Configure
      • Run Consul-Terraform-Sync
    • Architecture
      • Overview
      • Status
      • Tasks
      • Overview
      • task
    • Configuration
    • Tasks
    • Terraform Modules
      • Overview
      • License
      • Terraform Cloud Driver
      • Overview
      • Terraform
      • Terraform Cloud
    • Compatibility
    • Consul KV
    • Sessions
    • Watches
    • Overview
      • General
      • CLI Reference
      • Configuration Reference
    • Configuration Entries
    • Telemetry
    • Sentinel
    • RPC
    • Overview
      • ACL System Overview
      • Tokens
      • Policies
      • Roles
      • Rules Reference
      • Legacy Mode
      • Token Migration
      • ACLs in Federated Datacenters
        • Overview
        • Kubernetes
        • JWT
        • OIDC
        • AWS IAM
    • Encryption
      • Overview
      • Core
      • Network Infrastructure Automation
    • Overview
    • Admin Partitions
    • Audit Logging
    • Automated Backups
    • Automated Upgrades
    • Enhanced Read Scalability
    • Single sign-on - OIDC
    • Redundancy Zones
    • Advanced Federation
    • Network Segments
    • Namespaces
    • NIA with TFE
    • Sentinel
      • Overview
      • FAQ
    • Overview
    • Improving Consul Resilience
    • Anti-Entropy
    • Consensus Protocol
    • Gossip Protocol
    • Jepsen Testing
    • Network Coordinates
    • Consul Integration Program
    • NIA Integration Program
    • Vault Integration
    • Proxy Integration
  • Consul Tools
    • Overview
    • Compatibility Promise
    • Specific Version Details
      • Overview
      • General Process
      • Upgrading to 1.2.4
      • Upgrading to 1.6.9
      • Upgrading to 1.8.13
      • Upgrading to 1.10.0
    • Common Error Messages
    • FAQ
    • Overview
      • v1.11.x
      • v1.10.x
      • v1.9.x
      • v0.1.x
      • v0.2.x
      • v0.4.x
      • v0.3.x
      • v0.2.x
      • v0.5.x
      • v0.6.0-beta
    • Overview
    • ACL
  • Guides
Type '/' to Search

»Configuring Ingress Controllers with Consul on Kubernetes

This topic requires familiarity with Ingress Controllers

The following information requires Consul 1.10+, Consul-k8s 0.26+, Consul-helm 0.32+ configured with Transparent Proxy mode enabled.

This page describes a general approach for integrating Ingress Controllers with Consul on Kubernetes to secure traffic from the Controller to the backend services by deploying sidecars along with your Ingress Controller. This allows Consul to transparently secure traffic from the ingress point through the entire traffic flow of the service.

If you are looking for a fully supported solution for ingress traffic into Consul Service Mesh, please visit Consul API Gateway for instruction on how to install Consul API Gateway along with Consul on Kubernetes.

A few steps are generally required to enable an Ingress controller to join the mesh and pass traffic through to a service:

  • Enable connect-injection via an annotation on the Ingress Controller's deployment: consul.hashicorp.com/connect-inject is true.

  • Using the following annotations on the Ingress controller's deployment, set up exclusion rules for its ports.

    • consul.hashicorp.com/transparent-proxy-exclude-inbound-ports - Provides the ability to exclude a list of ports for inbound traffic that the service exposes from redirection. Typical configurations would require all inbound service ports for the controller to be included in this list.
    • consul.hashicorp.com/transparent-proxy-exclude-outbound-ports - Provides the ability to exclude a list of ports for outbound traffic that the service exposes from redirection. These would be outbound ports used by your ingress controller which expect to skip the mesh and talk to non-mesh services.
    • consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs - Provides the ability to exclude a list of CIDRs that the service communicates with for outbound requests from redirection. It is somewhat common that an Ingress controller will expect to make API calls to the Kubernetes service for service/endpoint management. As such including the ClusterIP of the Kubernetes service is common.

Note: Depending on which ingress controller you use, these stanzas may differ in name and layout, but it is important to apply these annotations to the pods of your ingress controller.

# An example list of pod annotations for an ingress controller, which need be applied to PODS for the controller, not the deployment itself.
  podAnnotations:
    consul.hashicorp.com/connect-inject: "true"
    # Add the container ports used by your ingress controller
    consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: "80,8000,9000,8443"
    # And the CIDR of your Kubernetes API: `kubectl get svc kubernetes --output jsonpath='{.spec.clusterIP}'
    consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs: "10.108.0.1/32"
# An example list of pod annotations for an ingress controller, which need be applied to PODS for the controller, not the deployment itself.
  podAnnotations:
    consul.hashicorp.com/connect-inject: "true"
    # Add the container ports used by your ingress controller
    consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: "80,8000,9000,8443"
    # And the CIDR of your Kubernetes API: `kubectl get svc kubernetes --output jsonpath='{.spec.clusterIP}'
    consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs: "10.108.0.1/32"
  • If the Ingress controller acts as a LoadBalancer and routes directly to Pod IPs instead of the ClusterIP of your Kubernetes Services a ServiceDefault CRD must be applied to each backend service allowing it to use the dialedDirectly features. By default this is disabled.

    # Example Service defaults config entry
    apiVersion: consul.hashicorp.com/v1alpha1
    kind: ServiceDefaults
    metadata:
      name: backend
    spec:
      transparentProxy:
        dialedDirectly: true
    
    # Example Service defaults config entry
    apiVersion: consul.hashicorp.com/v1alpha1
    kind: ServiceDefaults
    metadata:
      name: backend
    spec:
      transparentProxy:
        dialedDirectly: true
    
  • An intention from the Ingress Controller to the backend application must also be applied, this could be an L4 or L7 intention:

    # example L4 intention, but an L7 intention can also be used to control access to specific routes.
    apiVersion: consul.hashicorp.com/v1alpha1
    kind: ServiceIntentions
    metadata:
      name: ingress-backend
    spec:
      destination:
        name: backend
      sources:
        - name: ingress
          action: allow
    
    # example L4 intention, but an L7 intention can also be used to control access to specific routes.
    apiVersion: consul.hashicorp.com/v1alpha1
    kind: ServiceIntentions
    metadata:
      name: ingress-backend
    spec:
      destination:
        name: backend
      sources:
        - name: ingress
          action: allow
    

»Common Configuration Problems:

  • The Ingress Controller's ServiceAccount name and Service name differ by default in some platforms. Consul on Kubernetes requires the ServiceAccount and Service to have the same name. To resolve this be sure to explicitly set ServiceAccount name the same as the ingress controller service name using it's respective helm configurations.

  • If the Ingress Controller does not have the correct inbound ports excluded it will fail to start and the Ingress' service will not get created, causing the controller to hang in the init container. The required container ports are not always readily available in the helm charts, so in order to resolve this examine the ingress controller's underlying pod spec and look for the required container ports, adding these to the consul.hashicorp.com/transparent-proxy-exclude-inbound-ports annotation on the ingress controller deployment.

»Examples:

Here are a couple example configurations which can be used as reference points in setting up your own ingress controller configuration! These were used in dev environments and are not intended to be fully supported but should provide some idea how to extend the information above to your own uses cases.

  • Traefik Consul example - kschoche
  • Consul K8s Ingress Controllers examples - joatmon08
github logoEdit this page
IntroGuidesDocsCommunityPrivacySecurityBrandConsent Manager