OverviewEgress traffic is traffic going from OpenShift pods to external systems, outside of OpenShift. There are two main options for enabling egress traffic. Allow access to external systems from OpenShift physical node IPs or use egress router. In enterprise environments egress routers are often preferred. They allow granular access from a specific pod, group of pods or project to an external system or service. Access via node IP means all pods running on a given node can access external systems. An egress router is a pod that has two interfaces (eth0) and (macvlan0). Eth0 is sitting on the cluster network in OpenShift (internal) and macvlan0 has an IP and gateway from the external physical network. The network team can allow to external systems using the egress router IP. OpenShift administrators using labels can assign pods access to the egress router service thus enabling them to access external services. The egress router acts as a bridge between pods that can use the service and the external system.
ConfigurationIn this configuration we have deployed a simple OpenShift all-in-one environment running on a KVM hypervisor. We have also deployed a second VM running a web server. The KVM hypervisor has two virtual networks 192.168.122.0/24 and 192.168.123.0/24. OpenShift has two network interfaces, eth0 is on 192.168.122.0/24 and eth1 is on 192.168.123.0/24. The Web server has one interface, eth0 on 192.168.123.0/24. To test the egress router we will only allow access to the web server from the source IP of the egress router. Using another pod we will show how to access the web server using the egress router. [Web Server] Allow only the IP of the egress router (192.168.123.99) in OpenShift to access the web server on port 80.
# firewall-cmd --permanent --zone=public \ --add-rich-rule='rule family="ipv4" source address="192.168.123.99" \ port protocol="tcp" port="80" accept'
# firewall-cmd --reload[OpenShift Master] Create a new project
# oc new-project myprojDeploy Egress router in legacy mode
# vi egress-router.yaml
apiVersion: v1 kind: Pod metadata: name: egress-1 labels: name: egress-1 annotations: pod.network.openshift.io/assign-macvlan: "true" spec: containers: - name: egress-router image: openshift3/ose-egress-router securityContext: privileged: true env: - name: EGRESS_SOURCE value: 192.168.123.99 - name: EGRESS_GATEWAY value: 192.168.123.1 - name: EGRESS_DESTINATION value: 192.168.123.91
# oc create -f egress-router.yamlCheck Egress pod and ensure it can access web server
# oc get pods NAME READY STATUS RESTARTS AGE egress-1 1/1 Running 0 2h
# oc rsh egress-1Once connected to egress router you will notice it is running as root. This is the difference between legacy mode and init. For troubleshooting and testing it is easier to run in legacy mode and then switch to init mode when things are working.
sh-4.2# curl http://192.168.123.91 Hello World! My App deployed via Ansible V6.Here we see we can access our web server.
Deploy Egress service The egress service allows other pods to access external services using the egress router.
# vi egress-service.yaml
apiVersion: v1 kind: Service metadata: name: egress-1 spec: ports: - name: http port: 80 type: ClusterIP selector: name: egress-1
# oc create -f efress-service.yamlDeploy Ruby hello world pod The ruby example pod will be used to access our web server (192.168.123.91) via the egress router. Remember only source IP 192.168.123.99 (egress router) can access web server.
# oc new-app \ centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git
# oc get pods NAME READY STATUS RESTARTS AGE egress-1 1/1 Running 0 2h ruby-ex-1-wt3q9 1/1 Running 1 15hHere we now see the ruby example pod is running. Get service for the egress router
# oc get service NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE egress-1 172.30.137.86 80/TCP 2h ruby-ex 172.30.205.126 8080/TCP 15hCheck that web server pod cannot access web server directly
# oc rsh ruby-ex-1-wt3q9
sh-4.2$ curl http://192.168.123.91 curl: (7) Failed connect to 192.168.123.91:80; No route to hostCheck that web server pod can access web server using egress service
sh-4.2$ curl http://172.30.137.86 Hello World! My App deployed via Ansible V6.
sh-4.2$ curl http://egress-1 Hello World! My App deployed via Ansible V6.
Configure Egress Router Init ModeOnce egress router is working it is recommend re-configuring using init mode. This ensures that the egress router is not running as root.
apiVersion: v1 kind: Pod metadata: name: egress-1 labels: name: egress-1 annotations: pod.network.openshift.io/assign-macvlan: "true" spec: initContainers: - name: egress-router image: openshift3/ose-egress-router securityContext: privileged: true env: - name: EGRESS_SOURCE value: 192.168.123.99 - name: EGRESS_GATEWAY value: 192.168.123.1 - name: EGRESS_DESTINATION value: 192.168.123.91 - name: EGRESS_ROUTER_MODE value: init containers: - name: egress-router-wait image: openshift3/ose-pod
TroubleshootingIn order to troubleshoot the egress router it is recommended to run in legacy mode so you have access to the IP space. View Network Configuration Below we can see that eth0 has a IP from pod network and macvlan0 has IP on our external network.
# oc rsh egress-1
sh-4.2# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP link/ether 0a:58:0a:80:00:51 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.128.0.81/23 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::858:aff:fe80:51/64 scope link valid_lft forever preferred_lft forever 4: macvlan0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether b6:6d:62:ee:2e:bb brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.123.99/32 scope global macvlan0 valid_lft forever preferred_lft forever inet6 fe80::b46d:62ff:feee:2ebb/64 scope link valid_lft forever preferred_lft foreverSend ARP requests from egress router to gateway of external network
sh-4.2# arping -I macvlan0 -c 2 192.168.123.91 ARPING 192.168.123.91 from 192.168.123.99 macvlan0 Unicast reply from 192.168.123.91 [52:54:00:7F:0A:4A] 0.944ms Unicast reply from 192.168.123.91 [52:54:00:7F:0A:4A] 0.671ms Sent 2 probes (1 broadcast(s)) Received 2 response(s)View ARP table of OpenShift node Notice both the egress router (192.168.123.99) and web server (192.168.123.91) show up in arp cache. The egress router is incomplete because the node cannot see the MAC address. This is important and reason you need to enable promiscuous mode if you are running OpenShift on top of a virtualization platform. Otherwise hypervisor will not recognize MAC address and simply drop the packets.
# arp Address HWtype HWaddress Flags Mask Iface 10.128.0.71 ether 0a:58:0a:80:00:47 C tun0 10.128.0.66 ether 0a:58:0a:80:00:42 C tun0 10.128.0.74 ether 0a:58:0a:80:00:4a C tun0 10.128.0.69 ether 0a:58:0a:80:00:45 C tun0 192.168.123.99 (incomplete) eth1 10.128.0.81 ether 0a:58:0a:80:00:51 C tun0 10.128.0.72 ether 0a:58:0a:80:00:48 C tun0 10.128.0.67 ether 0a:58:0a:80:00:43 C tun0 192.168.122.1 ether 52:54:00:18:40:b7 C eth0 10.128.0.75 ether 0a:58:0a:80:00:4b C tun0 10.128.0.70 ether 0a:58:0a:80:00:46 C tun0 192.168.123.1 ether 52:54:00:03:3f:fd C eth1 192.168.123.91 ether 52:54:00:7f:0a:4a C eth1 10.128.0.73 ether 0a:58:0a:80:00:49 C tun0Use tcpdump on OpenShift node to analyze packets While running tcpdump connect to web server through egress router using curl.
# tcpdump -i eth1 -e 14:29:46.550889 b6:6d:62:ee:2e:bb (oui Unknown) > 52:54:00:03:3f:fd (oui Unknown), ethertype IPv4 (0x0800), length 74: 192.168.123.99.55892 > 192.168.123.91.http: Flags [S], seq 2654909605, win 28200, options [mss 1410,sackOK,TS val 11141299 ecr 0,nop,wscale 7], length 0 14:29:46.550991 52:54:00:03:3f:fd (oui Unknown) > 52:54:00:7f:0a:4a (oui Unknown), ethertype IPv4 (0x0800), length 74: 192.168.123.99.55892 > 192.168.123.91.http: Flags [S], seq 2654909605, win 28200, options [mss 1410,sackOK,TS val 11141299 ecr 0,nop,wscale 7], length 0 14:29:46.551107 52:54:00:7f:0a:4a (oui Unknown) > b6:6d:62:ee:2e:bb (oui Unknown), ethertype IPv4 (0x0800), length 74: 192.168.123.91.http > 192.168.123.99.55892: Flags [S.], seq 2459314841, ack 2654909606, win 28960, options [mss 1460,sackOK,TS val 10909669 ecr 11141299,nop,wscale 7], length 0Notice we now see the MAC address of the egress router (b6:6d:62:ee:2e:bb). We can also see the egress router talking to the web server (192.168.123.91) through the gateway (192.168.123.1).