»Ambassador Integration with Consul Connect
In addition to enabling Kubernetes services to discover and securely connect to each other, Connect also can help route traffic into a Kubernetes cluster from outside, when paired with an ingress controller like DataWire's Ambassador.
Ambassador is a popular Kubernetes-native service that acts as an ingress controller or API gateway. It supports an optional integration with Consul that allows it to route incoming traffic to the proxies for your Connect-enabled services.
This means you can have end-to-end encryption from the browser, to Ambassador, to your Kubernetes services.
Once you have tested and verified that everything is working, you can proceed with the Ambassador installation. Full instructions are available on the Ambassador site, but a summary of the steps is as follows:
If you are deploying to GKE, create a
RoleBinding to grant you cluster admin
kubectl create clusterrolebinding my-cluster-admin-binding \ --clusterrole=cluster-admin \ --user=$(gcloud info --format="value(config.account)")
Install Ambassador and a
LoadBalancer service for it:
kubectl apply -f https://www.getambassador.io/yaml/ambassador/ambassador-rbac.yaml kubectl apply -f https://www.getambassador.io/yaml/ambassador/ambassador-service.yaml
Install the Ambassador Consul Connector:
kubectl apply -f https://www.getambassador.io/yaml/consul/ambassador-consul-connector.yaml
Mapping annotations to your existing services, directing
HTTPS traffic to port 20000, which is opened by the Connect proxy. Here is an
example of doing this for the
static-server example used in the documentation
for the Connect sidecar:
apiVersion: v1 kind: Service metadata: name: static-service annotations: getambassador.io/config: | --- apiVersion: ambassador/v1 kind: TLSContext name: ambassador-consul hosts:  secret: ambassador-consul-connect --- apiVersion: ambassador/v1 kind: Mapping name: static-service_mapping prefix: /echo/ tls: ambassador-consul service: https://static-server:443 spec: type: NodePort selector: app: http-echo ports: - port: 443 name: https-echo targetPort: 20000
Once Ambassador finishes deploying, you should have a new
with a public-facing IP address. Connecting to the HTTP port on this address
should display the output from the static service.
kubectl describe service ambassador
»Enabling end-to-end TLS
The Ambassador service definition provided in their documentation currently does not serve pages over HTTPS. To enable HTTPS for full end-to-end encryption, follow these steps.
First, upload your public SSL certificate and private key as a Kubernetes secret.
kubectl create secret tls ambassador-certs --cert=fullchain.pem --key=privkey.pem
Download a copy of the ambassador-service.yaml file from Ambassador. Replace
metadata section with one that includes an Ambassador TLS configuration block,
using the secret name you created in the previous step. Then add an entry for port 443
LoadBalancer spec. Here is a complete example:
apiVersion: v1 kind: Service metadata: name: ambassador annotations: getambassador.io/config: | --- apiVersion: ambassador/v1 kind: Module name: tls config: server: enabled: True secret: ambassador-certs spec: type: LoadBalancer externalTrafficPolicy: Local ports: - port: 80 targetPort: http protocol: TCP name: http - port: 443 targetPort: https protocol: TCP name: https selector: service: ambassador
Update the service definition by applying it with
kubectl apply -f ambassador-service.yaml
You should now be able to test the SSL connection from your browser.
When Ambassador is unable to establish an authenticated connection to the Connect proxy servers, browser connections will display this message:
upstream connect error or disconnect/reset before headers
This error can have a number of different causes. Here are some things to check and troubleshooting steps you can take.
»Check intentions between Ambassador and your upstream service
If you followed the above installation guide, Consul should have registered a service called "ambassador". Make sure you create an intention to allow it to connect to your own services.
To check whether Ambassador is allowed to connect, use the
intention check subcommand.
$ consul intention check ambassador http-echo Allowed
»Confirm upstream proxy sidecar is running
First, find the name of the pod that contains your service.
$ kubectl get pods -l app=http-echo,role=server NAME READY STATUS RESTARTS AGE http-echo-7fb79566d6-jmccp 2/2 Running 0 1d
Then describe the pod to make sure that the sidecar is present and running.
$ kubectl describe pod http-echo-7fb79566d6-jmccp [...] Containers: consul-connect-envoy-sidecar: [...] State: Running Ready: True
»Start up a downstream proxy and try connecting to it
Log into one of your Consul server pods (or any pod that has a Consul binary in it).
$ kubectl exec -ti consul-server-0 -- /bin/sh
Once inside the pod, try starting a test proxy. Use the name of your service in place of
# consul connect proxy -service=ambassador -upstream http-echo:1234 ==> Consul Connect proxy starting... Configuration mode: Flags Service: http-echo-client Upstream: http-echo => :1234 Public listener: Disabled
If the proxy starts successfully, try connecting to it. Verify the output is as you expect.
# curl localhost:1234 "hello world"
Don't forget to kill the test proxy when you're done.
# kill %1 ==> Consul Connect proxy shutdown # exit
»Check Ambassador Connect sidecar logs
Find the name of the Connect Integration pod and make sure it is running.
$ kubectl get pods -l app=ambassador-pro,component=consul-connect NAME READY STATUS RESTARTS AGE ambassador-pro-consul-connect-integration-f88fcb99f-hxk75 1/1 Running 0 1d
Dump the logs from the integration pod. If the service is running correctly, there won't be much in there.
$ kubectl logs ambassador-pro-consul-connect-integration-f88fcb99f-hxk75 time="2019-03-13T19:42:12Z" level=info msg="Starting Consul Connect Integration" consul_host=10.142.0.21 consul_port=8500 version=0.2.3 2019/03/13 19:42:12 Watching CA leaf for ambassador time="2019-03-13T19:42:12Z" level=debug msg="Computed kubectl command and arguments" args="[kubectl apply -f -]" time="2019-03-13T19:42:14Z" level=info msg="Updating TLS certificate secret" namespace= secret=ambassador-consul-connect
»Check Ambassador logs
Make sure the Ambassador pod itself is running.
$ kubectl get pods -l service=ambassador NAME READY STATUS RESTARTS AGE ambassador-655875b5d9-vpc2v 2/2 Running 0 1d
Finally, check the logs for the main Ambassador pod.
$ kubectl logs ambassador-655875b5d9-vpc2v
»Check Ambassador admin interface
Forward the admin port from the Ambassador pod to your local machine.
$ kubectl port-forward pods/ambassador-655875b5d9-vpc2v 8877:8877
You should then be able to open http://localhost:8877/ambassador/v0/diag/ in your browser and view Ambassador's routing table. The table lists each URL mapping that has been set up. Service names will appear in green if Ambassador believes they are healthy, and red otherwise.
From this interface, you can also enable debug logging via the yellow "Set Debug On" button, which might give you a better idea of what's happening when requests fail.
If you have tried the above troubleshooting steps and are still stuck, DataWire provides support for Ambassador via the popular Slack chat app. You can request access and then join the
#ambassador room to get help.