How to set up wordpress on OpenShift in 10 minutes

Standard

What this is about?

A lot of customers would like to give the brave new container world (based on Docker technology) a try with real life workload. The WordPress content management system (yes, it has become more than a simple blog) seems to be an application that many customers know and use (and that I’ve been asked for numerous times). From a technical point of view the WordPress use case is rather simple, since we only need a PHP runtime and a database such as MySQL. Therefore it is a perfect candidate to pilot container aspects on OpenShift Container Platform.

Preparation

Install Container Development Kit

I highly recommend to install the freely available Red Hat Container Development Kit (shortly CDK). It will give you a ready to use installation of OpenShift Container Platform based on a Vagrant image. So you’re up to speed in absolutely no time:

Please follow the installation instructions here: https://developers.redhat.com/products/cdk/get-started/

Setup resources on OpenShift

Spin up your CDK environment and ssh into the system:

vagrant up
vagrant ssh

Create a new project and import the template for an ephemeral MySQL (since this is not included in the CDK V2.3 distribution by default). If you prefer to use another database or even one with persistent storage, then you can find additional templates here.

oc new-project wordpress
oc create -f https://raw.githubusercontent.com/openshift/openshift-ansible/master/roles/openshift_examples/files/examples/v1.3/db-templates/mysql-ephemeral-template.json

Now we create one pod for our MySQL database and create our WordPress application based on the source code. OpenShift will automatically determine that it is based on PHP and will therefore choose the PHP builder image to create a Docker image from our WordPress source code.

oc new-app mysql-ephemeral
oc new-app https://github.com/wordpress/wordpress
oc expose service wordpress

Now let’s login to the OpenShift management console and see what has happened:

We now have a pod that runs our WordPress application (web server, PHP, source code) and one pod running our ready to use ephemeral (= non-persistent) MySQL database.

Install wordpress

Before we need to note down the connection settings for our MySQL database. Firstly we look up the cluster IP of our mysql service; secondly we look up the database name, username & password. Have a look at the following screenshots:

Now it is time to setup and configure wordpress. Simply click on the route that has been created for your wordpress pod (in my case the hostname is “http://wordpress-wordpress.rhel-cdk.10.1.2.2.xip.io/wp-admin/setup-config.php”).

Congratulations for installing WordPress on OpenShift!

What’s next

For now we’ve created all the resources manually in a not yet reusable fashion. Therefore one of the next steps could be to create a template from our resources, import it into the OpenShift namespace and make it available for our users as a service catalog item. So our users could provision a fully installed WordPress with the click of a button.

Installing and Configuring Red Hat Satellite 6 via shell script

Standard

 

 

Overview

I would like to have a satellite 6.newest server to showcase common Satellite usecases. So the idea is to have a quick but flexible way to set up a Satellite 6.x server from scratch.

Goal

The idea is to get most parts somehow automated (but still not very flexible). In this blog i’m relying on shell scripting for the main Satellite configuration part, as the well known “hammer” command is a shell command. The script will stop when run into an error. If the error is fixed and you rerun the script it will skip over any task fulfilled already. There are some pre-configuration tasks done through Ansible though.

The server we set up is “owning” it’s own subnet, on which dhco, tftp and domain name resolution is fully under control of the satellite server. This means, there will be no conflicts e.g. with network wide dhcp servers.

Solution

Important Words on name resolution:

Satellite 6 needs to have a forward and reverse records for it’s own hostname (bound to the interface of the deployment network) in place before starting satellite-install in order to get the certificates set up correctly.

You will want to achieve the following:

 

  1. assure DNS forward and reverse lookup for the hostname of your server
  2. assure the resolved ip corresponds to the interface which points to the network of the clients
  3. assure no other ip address will resolve to the same name
  4. assure no other name resolves to the same ip address

If you do not take this serious enough, you might run into trouble when rolling out some servers. They might not register with subscription management. (internal server error).

The Demo Satellite server, will have it’s own (very private) network interface (for deployment) delivering DNS, DHCP, tfp and so forth for that network.

The Satellite-DNS server will resolve it’s own name and be master over the whole network configuration. satellite-install will be given all directives to configure DNS zones and dhcp ranges.

This brings us into a chicken-egg issue:

The dns-server will be in place after satellite-install has run and only thereafter you can add the records for the satellite itself, but satellite-install also needs the dns in place before hand otherwise the certificates will be wrong.

The approach which should work:

  1. on the satellite host
    1. put it’s hostname and the private ip in /etc/hosts
    2. set up resolv.conf to work with official nameserver (otherwise subscription-manager and yum won’t work)
  2. install satellite-packages and install satellite (with satellite-insall sript)
    1. this should bring your local DNS server for your (private) domain and your subnet
    2. the local DNS-server should forward request to official nameservers
  3. add satellite host to your local DNS server
  4. reconfigure your resolver to only ask localhost
  5. continue with satellite configuration (mostly via hammer)

System to deploy on

The system we deploy on can be virtual or physical. It needs connectivety to the Red Hat CDN and needs a “local” subnet”

I installed RHEL 7.2 on a virtual machine .

Memory: 16 GB are recommended for Satellite.

Storage

I set up a default sized VM which brings 100 GB Boot-Volume. As Satellite is very disk-space hungry i added a second 400 GB volume – meant for /var/lib.

Networking interfaces

I have a private VLAN attached to the VM with a subnet 172.24.101.0/24. Which can be used to deploy hosts and where i am able to run dhcp et all. The satellite nevertheless needs an interface in the rhevm – network to be reached from the outside world.

I configured the server to use dhcp (address only) on eth0 on rhevm network.

And i configured a static ip on the private VLAN. Important: eth1 must deactivate “default route”

installation/configuration of OS

I specified to boot from RHEL 7.2 DVD iso. In the grub menu i added to the default install item the following parameter:

inst.ks=https://raw.githubusercontent.com/mschreie/coe_sat62/master/anaconda-ks.cfg

which pulls in some definitions on how to install, so that things get kind of unattanded (except of providing the parameter).

Hint: You need to retype this as i found no way to cut&paste into the RHEV-provided VM-Console…

But it does save quit some manual work and prevents mistakes.

For reference:
Storage
I set up filesystems as follows:

/ 50 GB
/home no separate volume, no separate filesystem
/var/lib 250 GB on separate volume group on separate volume leaving 150 GB for growth)

Networking interfaces

I set up eth0 to use dhcp and eth1 to have a static ip on my private vlan.

Disabled ipv6 on all devices. Disabled change of resolv.conf via dhcp.

Config needs to looks like this in the end:

eth0: DHCP (address only), attach automatically
eth1: manual, attach automatically
     IP: 172.24.101.3/24 (note it is .3 now!)
     router: 172.24.101.253
no DNS configured
HOSTNAME: msisat62.example.com

Looking at the files (and correcting them if neccessary):

[root@msisat62 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE="Ethernet"
BOOTPROTO="dhcp"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="no"
NAME="eth0"
UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
DEVICE="eth0"
ONBOOT="yes"
PEERDNS="no"
PEERROUTES="yes"

 

[root@msisat62 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=eth1
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
DEVICE=eth1
ONBOOT=yes
IPADDR=172.24.101.3
PREFIX=24
GATEWAY=172.24.101.253

Preparation – done through Ansible

Disclaimer: With this project i used /etc/ansible as main project path, which i would not do so today and which i would not recommend. As paths are coded in on many places i did not want to change this without good testing. I therefor leave the unpleasant paths the way they work.

[mschreie@mschreie coe]$ ssh-copy-id root@msisat62

I added to Ansibles inventory file ( /etc/ansible/hosts):

[coe]
msisat62 ansible_user=root

As i do grep the output of some commands it is essential to be in the expected language. In /etc/ansible/ansible.cfg i assured this:

[defaults]
#module_lang = C
module_lang = en_US.UTF-8

I stored the secrets on how to reach cdn in a separate encrypted vault file.

[root@mschreie coe]# ansible-vault create secret.yml
cdnuser: myusername
cdnpass: mypassword

The Playbook itself can be downloaded like this:

[mschreie@mschreie ~ ]$ cd /etc/ansible/coe/
[mschreie@mschreie coe]$ wget https://github.com/mschreie/coe_sat62/raw/master/secret.yml
[mschreie@mschreie coe]$ wget https://github.com/mschreie/coe_sat62/raw/master/satellite_install.yml
[mschreie@mschreie coe]$ cd /etc/ansible/templates/
[mschreie@mschreie templates]$ wget https://github.com/mschreie/coe_sat62/raw/master/resolv.conf.j2
 
[mschreie@mschreie templates]$ cd /etc/ansible/coe/

I then ran:

[mschreie@mschreie coe]$ ansible-playbook -vv satellite_install.yml --ask-vault-pass
Using /etc/ansible/ansible.cfg as config file
Vault password:
Loaded callback default of type stdout, v2.0
....

Installation – shell script part

I used the Book of Adrian Bradshaw Introduction · Getting Started with Satellite 6 Command Line  to set up my satellite server. I put all commands in a shell script. Up to now this script is very unspectacular: no big intelligence or algorithm inside.

I added some mechanism for logging and to stop at error – so that the script won’t mess up any further. These mechanisms work pretty well but are not tested throughout. Feedback welcome😉

I’ve separated Script and configuration:

First you find the configuration:

[root@msisat62 ~]# wget https://raw.githubusercontent.com/mschreie/coe_sat62/master/satenv.sh

and the script itself:

[root@msisat62 ~]# wget https://raw.githubusercontent.com/mschreie/coe_sat62/master/satellite_setup.sh

Please find some explanation:

  1. Commands really changing setup are wrapped with a doit – function (as mentioned above)
  2. This function puts each correctly executed command in a donefile. The doit function also exits the script when a command returned with error. This gives you the chance to correct the issue before everything is messed up. When rerunning the script will skip all commands found in the donefile. You can safely rerun the script and it continues exactly where it stopped before.
  3. all output should be seen on the screen and in a logfile called $0.log

To run the script simply call:

[root@msisat62 ~]# vi satenv.sh
[root@msisat62 ~]# bash satellite_setup.sh

Explanation on the satellite-installer cmd:

satellite-installer --scenario satellite \
   --foreman-proxy-dhcp true \
   --foreman-proxy-dhcp-interface eth1 \
   --foreman-proxy-dhcp-range "$RANGEFROM $RANGETO" \
   --foreman-proxy-dhcp-nameservers "$DNSSERVER" \
   --foreman-proxy-dns true \
   --foreman-proxy-dns-forwarders "$DNSFORWARDERS" \
   --foreman-proxy-dns-interface $SATINTERFACE \
   --foreman-proxy-dns-zone "$DNSDOMAIN" \
   --foreman-proxy-dns-reverse "$DNSREVERSDOM" \
   --foreman-proxy-tftp true \
   --katello-proxy-url=http://proxy.coe.muc.redhat.com \
   --katello-proxy-port=3128 \
   --enable-foreman-plugin-openscap \
   --enable-foreman-proxy-plugin-openscap

I choose the satellite-installer cmd-line as seen in the satellite_setup.sh script.

It is wise to analyze the logs /var/log/katello-installer/…log and unfortunately i did not catch the admin credentials. Therefor i ran (also part of the script):

[mschreie@mschreie coe]$ foreman-rake permissions:reset
Reset to user: admin, password: FYdURRwgxAqbYD5N

and put the new credentials into /root/.hammer/cli_config.yml to use hammer without passing any credentials (also part of the script). To log in on Web-UI you need to look up the current password in /root/.hammer/cli_config.yml.

Note: I wanted to set the timezone but the timezone-module from ansible was not on my notebook…. It is now, but i did not update the script yet.

Checking what i did
I need to check that dhcp / dns are somehow what i expected:

[mschreie@mschreie coe]$ cat /etc/named.conf
[mschreie@mschreie coe]$ less /etc/named/options.conf
[mschreie@mschreie coe]$ cat /etc/zones.conf
[mschreie@mschreie coe]$ less /etc/dhcp/dhcpd.conf
 
[mschreie@mschreie coe]$ dig @localhost www.google.de
[mschreie@mschreie coe]$ dig @localhost mdschreier.coe.muc.redhat.com AXFR
[mschreie@mschreie coe]$ dig @localhost rhevm.coe.muc.redhat.com
[mschreie@mschreie coe]$ dig @localhost rhev.coe.muc.redhat.com

Manual tweaking

DNS records

I did the DNS changes inside the script already. Nothing to do here anymore.

DNS resolv.conf

same here – this is corrected through the script.

enable content in activation keys

As you know the activation key contains a content-view. All Repositories of the CV are available through the AK. But some Repositories default to “not enabled”. You can then enable them on the server with “subscription-manager repos” cmd. I prefere having them enabled per default.

I did not manage to get the right hammer-command in place yet.

The direction might be:

[root@msi-sat62 ~]# hammer activation-key list
[root@msi-sat62 ~]# hammer activation-key info --id 1  --organization "$ORG"  << you do not see the repositories here :-(
[root@msi-sat62 ~]# hammer activation-key product-content --id 1 --organization "$ORG"
-----|--------------------------------------------------------|------|-----|---------|----------------------------------------|---------
ID   | NAME                                                   | TYPE | URL | GPG KEY | LABEL                                  | ENABLED?
-----|--------------------------------------------------------|------|-----|---------|----------------------------------------|---------
4831 | Red Hat Satellite Tools 6.2 (for RHEL 7 Server) (RPMs) |      |     |         | rhel-7-server-satellite-tools-6.2-rpms | 1      
2455 | Red Hat Enterprise Linux 7 Server (Kickstart)          |      |     |         | rhel-7-server-kickstart                | default
2472 | Red Hat Enterprise Linux 7 Server - RH Common (RPMs)   |      |     |         | rhel-7-server-rh-common-rpms           | default
2456 | Red Hat Enterprise Linux 7 Server (RPMs)               |      |     |         | rhel-7-server-rpms                     | default
-----|--------------------------------------------------------|------|-----|---------|----------------------------------------|---------

[root@msi-sat62 ~]# hammer activation-key content-override --content-label rhel-7-server-rh-common-rpms --value 1 --id 1 --organization "$ORG"
Updated content override

FixMe: Needs to be automated and added in the script.

First Deployment

I created a VM via RHEVM-WebUI:

  • small
  • nic VLAN 101
  • 50 GB thin provisioned disk, bootable
  • Boot sequence: PXE, Hard Disk

I noted the mac adress: 00:1a:4a:7f:6a:38

And run

[root@msi-sat62 ~]# hammer host create --hostgroup "$HG1" \
   --name "msi-provisiontest1" --mac "00:1a:4a:7f:6a:38" \
   --root-password "redhat00" \
   --organization "${ORG}" --location "${LOC}"

Host created

This host deployed charmingly well.

Conclusion

We are now able to set up a Satellite Server 80% in an automated way. The result will be a Satellite 6.2 up and running and able to provision an existing server.

There are still quite some pitfalls and i believe all this needs quite some tweeking.

Update: I’m now working on an “Ansible only” solution to set up a Satellite Server but thought at least the script might help someone.


My personal look at the German eID system (“Neuer Personalausweis”)

Standard

Business Problem

Many business processes in Germany involve paper (or better TONS OF PAPER!) and surely many manual steps: think of opening a bank account or registering a car at your local “Zulassungsstelle”. In my opinion one of the main reasons for that is that the identity of a user cannot be properly verified online. You could now argue that things like video identification or Deutsche Post PostIdent came up to address this problem. However this only solves part of the problem, since the signature still needs to be done manually.

In Germany the so called nPA (neuer Personalausweis) is able to solve this problem by providing a qualified signature. So you will be able to digitally sign contracts online. Therein lays the potential to completely transform tons of paper-based processes. And huge amounts of time and money could be saved as well!

Source: http://www.personalausweisportal.de/

Use cases of the eID system

The nPA has two main functions “Identification with Online-Ausweisfunktion” and “Electronic Signature”, which allow to implement many exciting use cases. These range from simple verifications (like age check, address validation) to login mechanisms for websites (the nPA can be considered as a single-sign-on system in this context). Moreover the nPA also allows to apply a qualified digital signature to documents, which is equal to a genuine signature (according to German law).

Since its launch in 2010 a couple of federal institutes and enterprises have made their services ready for the nPA:

  • ElsterOnline (German tax)
  • Rentenkonto online (German pension fund)
  • Punkteauskunft aus dem Verkehrszentralregister (VZR)
  • UrkundenService
  • Allianz Kundenportal

A complete list of applications can be found here: http://www.personalausweisportal.de/DE/Buergerinnen-und-Buerger/Anwendungen/Anwendungen_node.html. However, from my perception the adoption still leaves a lot of room for improvement.

Architectural overview

There is extensive documentation available which describes the technical architecture behind the eID system (personally I recommend the information from BSI found here: https://www.bsi.bund.de/DE/Publikationen/TechnischeRichtlinien/tr03130/tr-03130.html). That it why I do not want to go into the nitty gritty details.

However, to give you a rough understanding have a look at the following illustration, which looks similar to what is available in token based authentication systems (think of SAML and/or OpenID Connect concepts). There is something like a service provider (“WebServer”) who wants to protect a service; then an authority that is able to validate the identity (“eID-Server”); and a login component (“AusweisApp”) that allows the end user to enter login information like a PIN. Last but not least the user must have a card reader connected to his local system, which talks to the login component (“AusweisApp”).

Source: http://www.bsi.bund.de

It is important to understand that the login component (“AusweisApp”) is implemented as a standalone application, which must run on the user’s computer (and of course be installed beforehand). For 2017 it is planned to release mobile versions of the app (see Google Play Store) in order to use a mobile device as a card reader. In my opinion this will help to reduce the overall complexity from an end user’s perspective.

When looking at the system from a service provider’s point of view (e.g. I am an online shop provider who wants to enable users to login with their nPA), you have to consider a lot of things. Since their is neither a public instance of the “eID-Server” nor source code available, you have two options: create your own implementation based on the BSI spec or buy the service from a provider. Additionally you will have to think of how to integrate the token into your application: since there is no “reference implementation” of the “eID-Server” spec there is little to no documentation available. Overall the process feels rather complex and intransparent to me.

A detailed description of the application process can be found here: “Become Service Provider”.

Conclusion

The opportunity behind the German eID system is really huge and could speed up lots of processes and make all of our lives easier. But in my opinion there are a lot of things hindering the adoption and success of the system:

  1. There is no public eID-Server instance that can be used by public and private institutions. This makes the adoption unnecessarily complicated because all service providers have to find a solution for themselves.
  2. Little documentation for service providers available. Instead only tons of specs available that need a lot of work lifted by the service provider.
  3. Many services require that you map your eID to the identity in their system (at least once). This makes the process very uncomfortable for the end user.
  4. Currently an external card reader is needed. Firstly it has to be bought by the end user and secondly this does not work on the go. Fortunately this caveat has already been addressed with the mobile app version.

My final thoughts: the adoption cannot be forced by laws. Instead, I think that the eID system should be developed in a more transparent and community based manner. Moreover the integration by service providers should be as easy as putting a social login on my personal website.

 

Scalable Network of Active MQ Brokers for handing massive connections

Standard

Abstract

screenshot-from-2016-12-07-18-07-08

Many organisations are now facing a challenge when it comes to choosing and setting up the right messaging infrastructure. It must be able to scale and handle massive parallel connections. This challenge often emerges with IoT & Big Data projects where a massive number of sensors are potentially connected to produce messages that need to be processed and stored.

An important thing to bare in mind when speaking of scaling the messaging infrastructure is that one must consider 2 distinct capabilities

  • scaling up when there is heavy load
  • scaling down to save costs when compute power is not need

This post explains how to address this challenge using the concept of network brokers in JBoss Active MQ.

At the time of writing the uses cases in this post have been tested with JBoss Active MQ 6.3.0

Choosing the right topology

Choosing the right broker network topology is a crucial step.

There are many examples of network topology that can be found in the official JBoss ActiveMQ documentation :

https://access.redhat.com/documentation/en/red-hat-jboss-a-mq/6.3/paged/using-networks-of-brokers/chapter-6-network-topologies

To have a better understanding, we need to have the following concepts in mind.

Each broker can have :

  • 1 or many transport connector to accept client connections (consumers or producers) and network connections from other brokers (receiving messages from other brokers). In our sample here we only have 1 transport connector per broker instance
  • 1 or many network connectors to allow messages to be forwarded to other brokers

In this article we will take the example of an IoT use case where the main focus is on :

  • handing a large number of connected sensors
  • securing a central system that collects and processes messages

In such scenario, we would like to set up a concentrator topology as shown below.

brokernetworkconcentrator

This topology allows us to setup stable and controlled connections between the brokers of layer 1 towards layer 2. The central brokers do not have to deal with sensors connecting and disconnecting.

The green arrows define a network connector. Note that these have a direction. It means that brokers from layer 1 can forward messages to all brokers from layer 2.

We also allow messages to travel between brokers of layer 2. We will see later that this allows the network to better balance the workload and to facilitates scale ups/scale downs.

How to scale up ?

Scaling up is pretty straight in this setup :

  • To handle more connections, brokers can be added in layer 1
  • If the overall volume grows, extend the number of brokers in layer 2 to absorb the workload

Usually when scaling up, we want the clients already connected to be redistributed among the newly available brokers. This is especially important for the clients connected to the layer 2 as we might not want to statically bind them to one broker.

To achieve this we can set the following options within the transport connectors of the central brokers.

  • rebalanceClusterClients : allows the broker to disconnect clients and force them to reconnect to a less busy broker
  • updateClusterClients : updates client connections when a broker joins the network. In that way the clients do not need to be aware of all the brokers available when connecting for the first time. The clients only need to be able to connect to at least one.
  • updateClusterClientsOnRemove : updates client connections when a broker leaves the network.
  • updateClusterFilter : a regular expression that allows to trigger client updates only on certain brokers that join the network. In our case for example we only want brokers from layer 2 to trigger reconnects.
 <transportConnectors>
 <transportConnector name="clients"
 rebalanceClusterClients="true" updateClusterClients="true"
 updateClusterClientsOnRemove="true" updateClusterFilter="amq-c.*"
 uri="tcp://localhost:61620" />
 </transportConnectors>

Further options on the transport connector can be found here :

https://access.redhat.com/documentation/en/red-hat-jboss-a-mq/6.3/paged/connection-reference/appendix-c-server-options

How to scale down?

Scaling down is a bit more challenging. This is because when shutting down an instance, there might be still messages remaining in the broker.

The strategy we can apply to shut down a broker (named c01 in this example) properly in a network is the following :

  • Stop the transport connector through JMX/Jolokia of broker c01. This has the effect of disconnecting all consumers and producers from broker c01. It also disconnects all incoming network connectors.
  • The clients (producers & consumers) will fail over to an active broker instance
  • Unconsumed messages in broker c01 will be forwarded to another broker instance since the network connect of broker c01 towards its neighbor is still active
  • Once all messages in broker c01 are consumed, we can shut down the entire instance

screenshot-from-2016-12-07-16-03-33

There might be situations where this procedure is not enough. Messages may remain when no active consumers for those exist anywhere in the network. In this case, I recommend you to look at Josh Reagan’s blog post:

http://joshdreagan.github.io/2016/08/22/decommissioning_jboss_a-mq_brokers/

It describes how to read the message store to be decommissioned and forward all remaining messages.

How is the load distributed ?

Deactivate conduit subscriptions on queues for better load balancing

When messages are sent to brokers in layer 1, by default these are distributed in a round robin manner to brokers in layer 2.

For better load balancing on queues, conduit subscriptions can be deactivated on the brokers of layer 1. If consumers are not evenly distributed on layer 2, it allows brokers from layer 1 to be aware of that and apply the round robin algorithm on the different consumers rather than the brokers they are connected to.

To have a deeper understanding you can refer to the following documentation :

https://access.redhat.com/documentation/en/red-hat-jboss-a-mq/6.3/paged/using-networks-of-brokers/chapter-9-load-balancing

Message hopping on layer 2 to avoid slow consumers

screenshot-from-2016-12-07-16-47-36

If you run the tests that are provided in the sample code (see below). You might notice that sometimes a message that reaches a broker in layer 2 is forwarded to its neighbor on layer 2 instead of being consumed by the local client.

This is a very powerful feature of Active MQ that allows messages to be consumed as fast as possible within the network.

This usually happens when the local consumer is slowing down. A good messaging system is one that tends to contain 0 messages (all messages are consumed as soon as they are produced). So what we can observe here is that the network of brokers is doing it’s best to move the messages around so that they are consumed as fast as possible.

On the other hand though, the consequences of this is that messages might get stuck at some point. There is a rule saying that a message cannot go back to a broker that has already been visited. So imagine if the fast consumer suddenly disconnects, our message here will get stuck in c01.

How to avoid stuck messages ?

To avoid messages getting stuck we need to add the replayWhenNoConsumers policy on the queues that are used within the network.

<policyEntry queue="sensor.events" queuePrefetch="1">
 <networkBridgeFilterFactory>
 <conditionalNetworkBridgeFilterFactory
 replayDelay="1000" replayWhenNoConsumers="true" />
 </networkBridgeFilterFactory>
</policyEntry>

This allows messages that are on a broker without any local consumer to revisit a previous broker.

Other important configurations

To ensure that the messages are prioritizing the shortest paths, these options need to be set on the network connectors :

  • decreaseNetworkConsumerPriority=”true”
  • suppressDuplicateQueueSubscriptions=”true”

 

Sample code & configurations

Nothing is better than a working example. A sample project can be found in the following GitHub repo : https://github.com/alainpham/amq-broker-network

This project populates 3 AMQ instances configured as a netwok of brokers similar to what is described in the sections above :

screenshot-from-2016-12-07-14-35-08

  • s01 is the instance where the producers will connect to (layer 1). We only need on of those to understand the different concepts. This instance has 2 network connectors, one towards c01 and another one towards c02
  • c01 and c02 are the instances where the central consumers connect to in order to process messages (alyer 2). They each have a network connector towards each other

There are these test cases that you can run with the command

mvn test
  • nominalSend1000Msg
    • create the network of 3 brokers with 1 producer and 2 consumers
    • send 1000 messages
    • each consumer receives roughly half of the messages
    • some messages hop between c01 and c02 when consumers are a bit slow
[ main] TestAMQNetwork INFO Received on collector01 500
[ main] TestAMQNetwork INFO Received on collector02 500
[ main] TestAMQNetwork INFO Total received 1000
[ main] TestAMQNetwork INFO amqs01 enqueue 1000
[ main] TestAMQNetwork INFO amqs01 dequeue 1000
[ main] TestAMQNetwork INFO amqs01 dispatch 1000
[ main] TestAMQNetwork INFO amqc01master enqueue 501
[ main] TestAMQNetwork INFO amqc01master dequeue 501
[ main] TestAMQNetwork INFO amqc01master dispatch 501
[ main] TestAMQNetwork INFO amqc02master enqueue 501
[ main] TestAMQNetwork INFO amqc02master dequeue 501
[ main] TestAMQNetwork INFO amqc02master dispatch 501
  •  scaleDown
    • create the network of 3 brokers with 1 producer and 2 consumers
    • send 500 messages
    • stop transport connector on c01
    • consumer on c01 reconnects to c02, remaining messages on c01 are drained
    • send another 500 messages
    • each consumer receives roughly half of the messages
############### before scaleDown #########################
[ main] TestAMQNetwork INFO ###################################
[ main] TestAMQNetwork INFO ###################################
[ main] TestAMQNetwork INFO Received on collector01 250
[ main] TestAMQNetwork INFO Received on collector02 250
[ main] TestAMQNetwork INFO Total received 500
[ main] TestAMQNetwork INFO amqs01 enqueue 500
[ main] TestAMQNetwork INFO amqs01 dequeue 500
[ main] TestAMQNetwork INFO amqs01 dispatch 500
[ main] TestAMQNetwork INFO amqc01master enqueue 250
[ main] TestAMQNetwork INFO amqc01master dequeue 250
[ main] TestAMQNetwork INFO amqc01master dispatch 250
[ main] TestAMQNetwork INFO amqc02master enqueue 250
[ main] TestAMQNetwork INFO amqc02master dequeue 250
[ main] TestAMQNetwork INFO amqc02master dispatch 250
 ############### scaleDown #########################
[ main] TestAMQNetwork INFO ##############################
[ main] TestAMQNetwork INFO ##############################
[ main] TestAMQNetwork INFO Received on collector01 501
[ main] TestAMQNetwork INFO Received on collector02 499
[ main] TestAMQNetwork INFO Total received 1000
[ main] TestAMQNetwork INFO amqs01 enqueue 1000
[ main] TestAMQNetwork INFO amqs01 dequeue 1000
[ main] TestAMQNetwork INFO amqs01 dispatch 1000
[ main] TestAMQNetwork INFO amqc01master enqueue 250
[ main] TestAMQNetwork INFO amqc01master dequeue 250
[ main] TestAMQNetwork INFO amqc01master dispatch 250
[ main] TestAMQNetwork INFO amqc02master enqueue 750
[ main] TestAMQNetwork INFO amqc02master dequeue 750
[ main] TestAMQNetwork INFO amqc02master dispatch 750

  • scaleUp
    • c01 is not started at the beginning, only s01 and c02 are up.
    • send 500 msgs
    • start c01
    • one of the consumer is forced to reconnected to c01
    • send another 500 msgs
    • each consumer receives roughly half of the messages
    • c01 starts to broke messages once it’s up
############### before scaleUp #########################
[ main] TestAMQNetwork INFO Received on collector01 250
[ main] TestAMQNetwork INFO Received on collector02 250
[ main] TestAMQNetwork INFO Total received 500
[ main] TestAMQNetwork INFO amqs01 enqueue 500
[ main] TestAMQNetwork INFO amqs01 dequeue 500
[ main] TestAMQNetwork INFO amqs01 dispatch 500
[ main] TestAMQNetwork INFO amqc01master enqueue 0
[ main] TestAMQNetwork INFO amqc01master dequeue 0
[ main] TestAMQNetwork INFO amqc01master dispatch 0
[ main] TestAMQNetwork INFO amqc02master enqueue 500
[ main] TestAMQNetwork INFO amqc02master dequeue 500
[ main] TestAMQNetwork INFO amqc02master dispatch 500

############### scaleUp #########################
[ main] TestAMQNetwork INFO Received on collector01 500
[ main] TestAMQNetwork INFO Received on collector02 500
[ main] TestAMQNetwork INFO Total received 1000
[ main] TestAMQNetwork INFO amqs01 enqueue 1000
[ main] TestAMQNetwork INFO amqs01 dequeue 1000
[ main] TestAMQNetwork INFO amqs01 dispatch 1000
[ main] TestAMQNetwork INFO amqc01master enqueue 252
[ main] TestAMQNetwork INFO amqc01master dequeue 252
[ main] TestAMQNetwork INFO amqc01master dispatch 252
[ main] TestAMQNetwork INFO amqc02master enqueue 752
[ main] TestAMQNetwork INFO amqc02master dequeue 752
[ main] TestAMQNetwork INFO amqc02master dispatch 752

The Jeavons’s Paradox for software development

Standard

A few months ago, whilst idling researching something using the Wikipedia rat-hole method (where you find an interesting link which you follow eventually forgetting what you were looking for in the first place), I stumbled across the Jevons’s Paradox, which is an economic effect first noticed in 1865 by an English economist William Jevons. The Paradox is that whilst technology improvements may make a single use of any resource more efficiency, the effect is that overall demand for that resource increases. Whilst Jevons initial focus was on around coal consumption, continued research has focused on economics, in particular the Rebound effect. This looks at why particular improvements might not provide the expected gains, that may have been seen at an experimental level and not seen during mainstream adation.

In the 1980’s the Khazzoom-Brookes postulate looked specifically at energy efficiency following the oil crisis in the mid late 70’s and how a drive to produce more fuel efficient cars would infact drive up demand for oil, rather than reducing it. The individual reaction to a more efficient car might be to drive further on the same budget and would encourage some people to use cars over other forms of travel (such as train, bus or walking). The overall collective impact would be for an increased compusion of the resource, in this case oil.

Can this postulate be applied to IT ? For sure, technical developments since the 1960’s and 70’s have seen greater resources (power, people) being used to fulfill the collective demand for Information Technology solutions, as they’ve automated and improved upon analogue and manual systems, but this is a change of system, so the resource increases should be expected, alongside a reduction in resources being used for systems being replaced. Over the last 5 years, one of the arguments being about DevOps, Platform-as-a-Service, microservices etc, is that new wave of IT development is that it is more efficient. Is it analogous to say that by having a more efficient system for developing and operating a specific application and workload that the overall demand for IT resources will increase ?

For application development and operation, resource could be:
* developers, operators and other IT staff, like project managers and architects. This can be measured in terms of hours worked, needs for training, as well as recruiting and onboarding.
* hardware and infrastructure software (used as the basis for operations and development) in terms of licensing, support etc. CPU, storage etc.
* networking.

From purely a development point of view, the implementation of a more efficient platform and associated processes might reduce the time a developer needs to do some tasks on that specific project, such as the need to set-up development or staging environments or write unique test scripts, as this will be automated or they will be able to reuse things from other projects, on the shared platform. Using DevOps principles and the removal of bottlenecks on the process, the efficiency of developing a specific application should improve significantly. Whilst metrics are used to record this for specific projects and for IT organisations as a whole, they are used to show an improvement in efficient and highlight what work is being done.

For example, the DevOps scorecard shown above is from a short article by Payal Chakravarty of IBM with some suggested metrics. It doesn’t include any indicators on resource apart from #7 which is about customer usage, and it would be interesting to see if the number of developers, the hours they work, size of disk and CPU usage increased by moving to a DevOps process from one which was simply just Agile or Waterfall. These suggested, additional metrics could be used to measure the resource usage and to see if it increased or decreased. This might be essential if one of the briefs of implementing DevOps or modernizing applications or development is to reduce the overall cost of development. Specific application development costs might be cheaper, but the overall IT spend might increase as per the Khazzoom-Brookes postulate might apply.

To measure this, you would need to look at:
* overall developer hours for the IT department (as well as measuring PM, Ops and other staff)
* number of IT staff
* number of concurrent application workloads, in development and in production
* number of requests for application development, whether this is new, modernization or update

The aim would be see if the resources applied to application development and operation is increasing and if it is, as what proportion is this against the number of projects and applications being managed. As well as looking for evidence for the existence of Jeavons’s Paradox, an IT organisaton will need to look for the diminishing returns (or possibly negative returns) through development of the system. However, when looking at any proposed modernization or development of an IT technology, process or organisation, it might be good to assume that overall resource usage is going up. You might expect it for a number of reasons anyway, as
* cost of providing training on new techniques and methods
* need to design and deliver a new or migrated application platform
* increased complexity required with new technology
* need to both maintain existing applications as well as developing new ones.

But if these additional costs are taken on board, simply the increased adoption and reliance of the improved systems, as well as the need to maintain the legacy (or technical debt) will mean that overall costs will increase anyway. Therefore in calculating return on investment, the time for payback on any decision for development should to take into account the increased consumption of resources and the need to provide more. Some further analysis will appear in another article, as well as looking how the tragedy of the commons should be taken into account in measuring the impact of IT development and change.

Deploying OpenShift Enterprise from Ansible Tower

Standard

ansible-tower-logotype-large-rgb-fullgrey-300x124 plus_sign openshiftlogo

Overview

In this article we will look at how to use Ansible Tower to deploy and manage OpenShift environments. OpenShift of course uses Ansible as its deployment and configuration tool already. While that is great, using Tower provides several major advantages:

  • UI for OpenShift deployment and configuration management
  • Secure store for credentials
  • RBAC and ability to delegate different responsibilities for OpenShift deployments
  • Easy to visualize and manage multiple OpenShift environments and even versions of OpenShift
  • History, audit trail and detailed logging in central location for all OpenShift environments and deployments

Prepare OpenShift Environment

In this example we will be doing an all-in-one deployment of OpenShift. The following steps should be done on OpenShift masters and nodes. Again here we just have one node since it is an all-in-one.

CONFIGURE A VMs WITH FOLLOWING:

  • RHEL 7.2
  • 2 CPUs
  • 4096 RAM
  • 30GB disk for OS
  • 25GB disk for docker images

REGISTER VALID SUBSCRIPTION

# subscription-manager register
# subscription-manager attach --pool=843298293829382
# subscription-manager repos --disable="*"
#subscription-manager repos \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-server-ose-3.3-rpms"

INSTALL REQUIRED TOOLS

# yum install -y wget git net-tools bind-utils iptables-services bridge-utils bash-completion

UPDATE

# yum update -y

RESTART OPENSHIFT MASTER

# systemctl reboot

CONFIGURE DOCKER

# yum install -y docker-1.10.3

ENABLE DOCKER DAEMON TO PULL FROM OPENSHIFT REGISTRY

# vi /etc/sysconfig/docker
OPTIONS='--selinux-enabled --insecure-registry 172.30.0.0/16'

SETUP DOCKER STORAGE FOR OPENSHIFT REGISTRY

Note: we will use the second disk for configuring docker storage.

# cat < /etc/sysconfig/docker-storage-setup
DEVS=/dev/vdb
VG=docker-vg
EOF
# docker-storage-setup

ENABLE AND START DOCKER DAEMON

# systemctl enable docker
# systemctl start docker

Import OpenShift inventory into Ansible Tower

These steps should be done directly on the host running Ansible Tower.

Create Inventory in Ansible Tower

Under inventories add a new inventory.

ans_group

Create directors for OpenShift inventory

# mkdir /root/ose3

Setup ansible-hosts file

# vi /root/ose3/ansible-hosts
##########################
### OSEv3 Server Types ###
##########################
[OSEv3:children]
masters
nodes
etcd

##############################
### host group for masters ###
##############################
[masters]
ose3-master2.lab.com

###################################
### host group for etcd servers ###
###################################
[etcd]
ose3-master2.lab.com

##################################################
### host group for nodes, includes region info ###
##################################################
[nodes]
ose3-master2.lab.com openshift_schedulable=True

Create directory for group_vars

Note: this is required because Tower import tool does not yet support [groupname:vars] directly in inventory file.

# mkdir /root/ose3/group_vars

Setup OpenShift parameters using group_vars file

# vi /root/ose3/group_Vars/OSEv3
ansible_ssh_user: root
os_sdn_network_plugin_name: 'redhat/openshift-ovs-subnet'
deployment_type: openshift-enterprise
openshift_master_default_subdomain: apps.lab.com
openshift_master_identity_providers: [{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
openshift_node_kubelet_args: {'maximum-dead-containers': ['100'], 'maximum-dead-containers-per-container': ['2'], 'minimum-container-ttl-duration': ['10s'], 'max-pods': ['110'], 'image-gc-high-threshold': ['90'], 'image-gc-low-threshold': ['80']}
logrotate_scripts: [{"name": "syslog", "path": "/var/log/cron\n/var/log/maillog\n/var/log/messages\n/var/log/secure\n/var/log/spooler\n", "options": ["daily", "rotate 7", "compress", "sharedscripts", "missingok"], "scripts": {"postrotate": "/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true"}}]
openshift_docker_options: "--log-opt max-size=1M --log-opt max-file=3"
openshift_node_iptables_sync_period: 5s
openshift_master_pod_eviction_timeout: 3m
osm_controller_args: {'resource-quota-sync-period': ['10s']}
osm_api_server_args: {'max-requests-inflight': ['400']}
openshift_use_dnsmasq: false

Import OpenShift inventory

# tower-manage inventory_import --source=/root/ose --inventory-name="OSE_3.3" --overwrite --overwrite-vars

After import is complete you should see inventory. Under OSE_3.3 inventory, a group called OSEv3 should be visible. If you edit the OSEv3 group you should see the variables used to drive OpenShift deployment. Here you can easily change things in order to update or change OpenShift deployment.

ansible_inv2

Under the OSEv3 group you should see all the OpenShift server groups and under those the actual systems.

ansible_inv1

Configure Ansible Tower

Create Project in Tower

Under projects add a new project. Add Github URL to ansible-openshift project. Ensure you add the correct branch. OpenShift v3.3 correlates to branch release-1.3. You should add a separate project for every release.

Note: make sure you check what version of ansible-openshift correlates to version of OpenShift you want to deploy!

ans1

Add credentials for OpenShift nodes

Under settings->credentials add new credentials called OSE.

Note: In this example I added the root user and password but you can use non-root user or ssh keys instead of password. In fact there is already a group_var parameter to use sudo.

ans_pwd

Add job template

Under job templates add a new template. Select the inventory, project and machine credentials. Select playbooks/byo/config.yaml for the playbook.

ans_template

Deploy OpenShift

To deploy or update your OpenShift deployment you simply need to run the playbook from Tower by clicking the rocket next to your job template.

ans_deploy

You can follow the deployment status by looking at the job in Tower.

ans_job

Summary

In this article we looked at how to deploy OpenShift using Ansible Tower. The default method for deploying and managing OpenShift is Ansible Core. Tower however gives you a lot of advantages providing central management, credentials store, RBAC, maintain multiple versions or multiple OpenShift environments and of course the more you do with ansible the more sense it makes to start using Tower. I hope you found this article informative and interesting. Looking forward to hearing your thoughts and feedback.

Happy OpenShifting!

(c) 2016 Keith Tenzer


Standard Operating Environment – Part II: Workflows in Detail

Standard

Introduction

In the previous article in this series we introduced the basic concepts of a Standard Operating Environment (SOE). Here we go into further details of the workflows and processes involved in an SOE. In the third part of this series we will discuss the implementation and operation of an SOE in practice.

Build Lifecycle

Previously we discussed how a build goes through the following lifecycle stages:

  • Inception
  • Development
  • Release
  • Maintenance
  • Retirement

We will model each of these lifecycle stages as an Activity. As discussed in the previous article is usual to have multiple builds within the organisation, each build potentially at a different lifecycle stage. NB Comments in these activity diagrams refer to how activities might be implemented in Red Hat Satellite.

Inception Activity

The inception activity can be modelled as shown below:
inception-activity

As discussed in the previous article, every build is identified by a unique name and settling the name is the first step of Inception. The name will be static throughout the entire lifecycle of the build and used to reference it in all documentation, communications etc.

The next step is to create the repositories that will be used to contain the build. Typically, these consists of a software repository to contain (in the Red Hat world) RPMs, an SCM repository (such as one or more git repositories (or branches on existing repositories) to contain text-based artefacts such as deployment templates (kickstart files, snippets, cloud-init scripts) and configuration information (Puppet modules or Ansible playbooks and roles). Finally, a requirements tracker, such as a JIRA project, is created to manage the build development process.

The two final stages of the activity are to create test machines and a development build plan. A common scenario is to create one test machine per profile and many organisation will also create hardware (as opposed to virtual) test machines in order to increase test coverage to things like firmware and hardware compatibility. The build plan to be created will be used to automate integration and testing of the build in a CI engine such as Jenkins or Bamboo.

Once the Inception of the build is complete and the build development framework is in place, the Development Activity is triggered.

Development Activity

The development activity can be modelled as shown below:
development-activity

The Development Activity starts with requirements gathering and task backlog creation. The need for a a structured requirements gathering process is clear when the expectations placed on a modern SOE build are noted. The build will be required to:

  • Run on multiple platforms: hardware, virtualised, and cloud.
  • Support a large variety of 3rd party software, and be aligned with the support matrix of each vendor
  • Integrate with the organisation’s authentication, security, monitoring, logging, backup, storage, and network management systems
  • Meet non-functional requirements, such as performance and reliability requirements, for critical workloads

The complex requirements that are required of a modern SOE merit a more formal approach to requirements gathering and tracking, and of course many of the more static requirements will simply be inherited from earlier builds so implementation in the new build is likely to consist purely of ensuring successful test execution.

The main loop of the Development Activity consists of writing tests, software and configurations, integrating them, and testing the results; in other words a typical development-test loop. Many organisations will choose to use an Agile methodology, often Scrumban, to manage this process. A CI build plan implemented in Jenkins, Bamboo, or equivalent handles the creations of any artefacts that need to be built (such as RPMs or Puppet Modules), the integration of upstream content (such as Red Hat Enterprise Linux errata), the presentation of the new build version to test servers (via Red Hat Satellite Content Views), the re-deployment of the test servers and the execution of automated tests on the test servers.

Once all requirements have been implemented, and all tests are passing, the Release Activity is triggered.

Release Activity

The Release Activity can be modelled as shown below:
release-activity

This is a simple linear activity that production-ises the build. Along with the normal documentation tasks that might be expected such as publishing release notes and a defined support lifecycle for the build – including the all-important EOL date – the build is exposed to server environments. In the case of the Red Hat Satellite, this would be acheived by promoting the build content view into a lifecycle environment in which the targeted servers reside. Finally, servers that have been scheduled to be migrated to this build are re-assigned to it (Content View reassignment in the Red Hat Satellite world) and updated. It is important to note that SOE does not mandate the updating of all servers to the latests build – as described in the previous article it is expected that multiple builds would be supported at any one time.

Maintenance Activity

The Maintenance Activity can be modelled as shown below:
maintenance-activity

This activity is similar to the Development Activity but concerns itself with the ongoing maintenance and updating of a build that is in production. There are in fact two activities under consideration:

  • The triage and integration of updated vendor content e.g. Red Hat Enterprise Linux errata
  • The resolution of issues raised by users

A general triage plan has been shown, however many organisations will use a risk plan to determine how frequently, or if it all, vendor patches are incorporated into the build, for example (using the security errata ratings used by Red Hat):

  • Critical security errata are incorporated into the build within 48 hours
  • Serious security errata are incorporated into the build within 7 days
  • Moderate and Low security errata are incorporated into the build within 30 days
  • Bugfixes are only incorporated into the build if there is an operational reason to do so i.e. they address a bug which is causing loss of service, otherwise they are deferred until the next build
  • Enhancements are always deferred until the next build

Likewise the resolution of issues fixed by users is staged for incorporation into the current build, or postponed until the next build – for example a fix for a minor problem that does not cause operational loss may not warrant the release of a new build version.

It is important to note that every time the Maintenance Activity loop completes this results in the release of a new Build Version, not of a new build. The unique build name is unchanged, it is only the version of the build that increments. Red Hat Satellite supports this with Content View versioning.

Once a new Build Version has successfully tested it is exposed (for example via Content View promotion) to client servers which can then be updated as required.

Retirement Activity

The Retirement Activity can be modelled as shown below:
retirement-activity

This is a simple linear activity to take the build out of production. As the Maintenance Activity for this particular build will be terminated, it is important that all servers currently registered to the build are migrated to a more current build.

NEXT: Standard Operating Environment – Part III: Implementation with Red Hat Satellite, Jenkins, and Ansible Tower

OpenShift Enterprise 3.3: all-in-one Lab Environment with Jenkins Build Pipeline

Standard

Screenshot from 2016-08-04 14:40:07

Overview

In this article we will setup a OpenShift Enterprise 3.3 all-in-one configuration. We will also configure OpenShift router, registry, aggregate logging, metrics, CloudForms integration and finally an integrated jenkins build pipeline.

OpenShift has several different roles: masters, nodes, etcd and load balancers. An all-in-one setup means running all service on a single system. Since we are only using a single system, a load balancer or ha-proxy won’t be configured. If you would like to read more about OpenShift I can recommend the following:

Prerequisites

Configure a VM with following:

  • RHEL 7.2
  • 2 CPUs
  • 4096 RAM
  • 30GB disk for OS
  • 25GB disk for docker images

Register valid subscription

# subscription-manager register
# subscription-manager attach --pool=843298293829382
# subscription-manager repos --disable="*"
#subscription-manager repos \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-server-ose-3.3-rpms"

Install required tools

# yum install -y wget git net-tools bind-utils iptables-services bridge-utils bash-completion

Update

# yum update -y

Install OpenShift tools

# yum install -y atomic-openshift-utils

Restart OpenShift master

# systemctl reboot

Configure Docker

# yum install -y docker-1.10.3

Enable Docker daemon to pull from OpenShift registry

# vi /etc/sysconfig/docker
OPTIONS='--selinux-enabled --insecure-registry 172.30.0.0/16'

Setup Docker storage for OpenShift registry

Note: we will use the second disk for configuring docker storage.

# cat < /etc/sysconfig/docker-storage-setup
DEVS=/dev/vdb
VG=docker-vg
EOF
# docker-storage-setup

Enable and start Docker daemon

# systemctl enable docker
# systemctl start docker

Setup ssh access without password for Ansible

# ssh-keygen
# ssh-copy-id -i /root/.ssh/id_rsa.pub ose3-master.lab.com

DNS Setup

DNS is a requirement for OpenShift Enterprise. In fact most issues you may run into are a result of not having a properly working DNS environment. For OpenShift you can either use dnsmasq or bind. I recommend using dnsmasq but in this article I will cover both options.

Note: Since OpenShift also has skydns for providing DNS services to containers running on port 53 you need to setup dnsmasq or bind on a separate system. The OpenShift environment should resolv to that DNS server.

Option 1: DNSMASQ

A colleague Ivan Mckinely was nice enough to create an ansible playbook for deploying dnsmasq. To deploy dnsmasq run following steps on OpenShift master.

# git clone https://github.com/ivanthelad/ansible-aos-scripts.git
#cd ansible-aos-scripts

Edit inventory file and set dns to IP of the system that should be providing DNS

Also ensure nodes and masters have correct IPs for your OpenShift servers. In our case 192.168.122.60 is master, node and DNS.

#vi inventory
# ip of DNS server
[dns] 192.168.122.59
# ip of OpenShift nodes
[nodes] 192.168.122.60
# ip of OpenShift masters
[masters] 192.168.122.60

Configure dnsmasq and add wildcard DNS so all hosts with

# vi playbooks/roles/dnsmasq/templates/dnsmasq.conf

strict-order
domain-needed
local=/lab.com/
bind-dynamic
resolv-file=/etc/resolv.conf.upstream
no-hosts
address=/.cloudapps.lab.com/192.168.122.60
address=/ose3-master.lab.com/192.168.122.60
address=/dns.lab.com/192.168.122.59
log-queries

Ensure all hosts you want in DNS are also in /etc/hosts

The dnsmasq service reads /etc/hosts upon startup so all entries in hosts file can be queried through DNS.

#vi /etc/hosts
192.168.122.60  ose3-master.lab.com     ose3-master

Install dnsmasq via ansible

# ansible-playbook -i inventory playbooks/install_dnsmas.yml

If you need to make changes you can edit the /etc/dnsmasq.conf file and restart dnsmasq service.

Below is a sample dnsmasq.conf

# vi /etc/dnsmasq.conf
strict-order
domain-needed
local=/example.com/
bind-dynamic
resolv-file=/etc/resolv.conf.upstream
no-hosts
address=/.apps.lab.com/192.168.122.60
address=/ose3-master.lab.com/192.168.122.60
address=/ose3-master.lab.com/192.168.122.60
address=/dns.lab.com/192.168.122.59
address=/kubernetes.default.svc/192.168.122.60
log-queries

Option 2: NAMED

Install DNS tools and utilities

# yum -y install bind bind-utils
# systemctl enable named
# systemctl start named

Set firewall rules using iptables

# iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 53 -j ACCEPT
# iptables -A INPUT -p udp -m state --state NEW -m udp --dport 53 -j ACCEPT

Save the iptables Using

# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]

Note: If you are using firewalld you can just enable service DNS using firwall-cmd utility.

Example of zone file for lab.com

vi /var/named/dynamic/lab.com.zone 

$ORIGIN lab.com.
$TTL 86400
@ IN SOA dns1.lab.com. hostmaster.lab.com. (
 2001062501 ; serial
 21600 ; refresh after 6 hours
 3600 ; retry after 1 hour
 604800 ; expire after 1 week
 86400 ) ; minimum TTL of 1 day
;
;
 IN NS dns1.lab.com.
dns1 IN A 192.168.122.1
 IN AAAA aaaa:bbbb::1
ose3-master IN A 192.168.122.60
*.cloudapps 300 IN A 192.168.122.60

Example of named configuration

# vi /etc/named.conf 
//
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//

options {
 listen-on port 53 { 127.0.0.1;192.168.122.1; };
 listen-on-v6 port 53 { ::1; };
 directory "/var/named";
 dump-file "/var/named/data/cache_dump.db";
 statistics-file "/var/named/data/named_stats.txt";
 memstatistics-file "/var/named/data/named_mem_stats.txt";
 allow-query { localhost;192.168.122.0/24;192.168.123.0/24; };

/* 
 - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
 - If you are building a RECURSIVE (caching) DNS server, you need to enable 
 recursion. 
 - If your recursive DNS server has a public IP address, you MUST enable access 
 control to limit queries to your legitimate users. Failing to do so will
 cause your server to become part of large scale DNS amplification 
 attacks. Implementing BCP38 within your network would greatly
 reduce such attack surface 
 */
 recursion yes;

dnssec-enable yes;
 dnssec-validation yes;
 dnssec-lookaside auto;

/* Path to ISC DLV key */
 bindkeys-file "/etc/named.iscdlv.key";

managed-keys-directory "/var/named/dynamic";

pid-file "/run/named/named.pid";
 session-keyfile "/run/named/session.key";

//forward first;
 forwarders {
 //10.38.5.26;
 8.8.8.8;
 };
};

logging {
 channel default_debug {
 file "data/named.run";
 severity dynamic;
 };
};

zone "." IN {
 type hint;
 file "named.ca";
};

zone "lab.com" IN {
 type master;
 file "/var/named/dynamic/lab.com.zone";
 allow-update { none; };
};

//zone "122.168.192.in-addr.arpa" IN {
// type master;
// file "/var/named/dynamic/122.168.192.db";
// allow-update { none; };
//};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

Install OpenShift

OpenShift is installed and managed through Ansible. You have two options to install. Either advanced installation where you configure Ansible playbook or basic installation that simply runs a vanilla playbook with default options. I recommend always doing an advanced install and as such will not cover basic installation.

Configure the inventory

Note: If you have different dns names then change items in bold below.

# vi /etc/ansible/hosts
##########################
### OSEv3 Server Types ###
##########################
[OSEv3:children]
masters
nodes
etcd

################################################
### Set variables common for all OSEv3 hosts ###
################################################
[OSEv3:vars]
ansible_ssh_user=root
os_sdn_network_plugin_name='redhat/openshift-ovs-subnet'
deployment_type=openshift-enterprise
openshift_master_default_subdomain=apps.lab.com
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
openshift_node_kubelet_args={'maximum-dead-containers': ['100'], 'maximum-dead-containers-per-container': ['2'], 'minimum-container-ttl-duration': ['10s'], 'max-pods': ['110'], 'image-gc-high-threshold': ['90'], 'image-gc-low-threshold': ['80']}
logrotate_scripts=[{"name": "syslog", "path": "/var/log/cron\n/var/log/maillog\n/var/log/messages\n/var/log/secure\n/var/log/spooler\n", "options": ["daily", "rotate 7", "compress", "sharedscripts", "missingok"], "scripts": {"postrotate": "/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true"}}]
openshift_docker_options="--log-opt max-size=1M --log-opt max-file=3"
openshift_node_iptables_sync_period=5s
openshift_master_pod_eviction_timeout=3m
osm_controller_args={'resource-quota-sync-period': ['10s']}
osm_api_server_args={'max-requests-inflight': ['400']}
openshift_use_dnsmasq=false

##############################
### host group for masters ###
##############################
[masters]
ose3-master.lab.com

###################################
### host group for etcd servers ###
###################################
[etcd]
ose3-master.lab.com

##################################################
### host group for nodes, includes region info ###
##################################################
[nodes]
ose3-master.lab.com openshift_schedulable=True

Run playbook to install OpenShift

# ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml

Configure OpenShift

Once OpenShift is installed we need to configure an admin user and also setup the router and registry.

Create local admin account and enable permissions

# oc login -u system:admin -n default
# htpasswd -c /etc/origin/master/htpasswd admin
# oadm policy add-cluster-role-to-user cluster-admin admin
# oc login -u admin -n default

Configure OpenShift registry

Image streams and Docker images are stored in registry. When you build application, your application code will be added as a image stream. This enables S2I (Source to Image) and allows for fast build times.

#oadm registry --service-account=registry \
--config=/etc/origin/master/admin.kubeconfig \
--images='registry.access.redhat.com/openshift3/ose-${component}:${version}'

Note: normally you would want to setup registry in HA configuration.

Configure OpenShift router

The OpenShift router is basically an HA-Proxy that sends incoming requests to node where pod/container are running.

Note: normally you would want to setup the router in an HA configuration.

#oadm router router --replicas=1 \
    --credentials='/etc/origin/master/openshift-router.kubeconfig' \
    --service-account=router

Optional: CloudForms Integration

CloudForms is a cloud management platform. It integrates not only with OpenShift but also other Cloud platforms (OpenStack, Amazon, GCE, Azure) and traditional virtualization platforms (VMware, RHEV, Hyper-V). Since OpenShift is usually running on cloud or traditional virtualization platforms, CloudForms enables true end-to-end visibility. CloudForms provides not only performance metrics, events, smart state analysis of containers (scanning container contents) but also can provide chargeback for OpenShift projects. CloudForms is included in OpenShift subscription for purpose of managing OpenShift. To add OpenShift as provider in CloudForms follow the below steps.

The management-infra project in OpenShift is designed for scanning container images. A container is started in this context and the image to be scanned is mounted. A service account management-admin exists and should be used to provide CloudForms access.

Note: scanning images is CPU intensive so it is important to ensure the management-admin project schedules pods/containers on infrastructure nodes.

List tokens that are configured in management-infra project (this is created at install time).

# oc project management-infra
# oc get sa management-admin -o yaml
apiVersion: v1
imagePullSecrets:
- name: management-admin-dockercfg-ln1an
kind: ServiceAccount
metadata:
 creationTimestamp: 2016-07-24T11:36:58Z
 name: management-admin
 namespace: management-infra
 resourceVersion: "400"
 selfLink: /api/v1/namespaces/management-infra/serviceaccounts/management-admin
 uid: ee6a1426-5192-11e6-baff-001a4ae42e01
secrets:
- name: management-admin-token-wx17s
- name: management-admin-dockercfg-ln1an

Use describe to get token to enable CloudForms to accesss the management-admin project.

# oc describe secret management-admin-token-wx17s
Name: management-admin-token-wx17s
Namespace: management-infra
Labels: 
Annotations: kubernetes.io/service-account.name=management-admin,kubernetes.io/service-account.uid=ee6a1426-5192-11e6-baff-001a4ae42e01

Type: kubernetes.io/service-account-token

Data
====
ca.crt: 1066 bytes
namespace: 16 bytes
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtYW5hZ2VtZW50LWluZnJhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1hbmFnZW1lbnQtYWRtaW4tdG9rZW4td3gxN3MiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibWFuYWdlbWVudC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImVlNmExNDI2LTUxOTItMTFlNi1iYWZmLTAwMWE0YWU0MmUwMSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptYW5hZ2VtZW50LWluZnJhOm1hbmFnZW1lbnQtYWRtaW4ifQ.Y0IlcwhHW_CpKyFvk_ap-JMAT69fbIqCjkAbmpgZEUJ587LP0pQz06OpBW05XNJ3cJg5HeckF0IjCJBDbMS3P1W7KAnLrL9uKlVsZ7qZ8-M2yvckdIxzmEy48lG0GkjtUVMeAOJozpDieFClc-ZJbMrYxocjasevVNQHAUpSwOIATzcuV3bIjcLNwD82-42F7ykMn-A-TaeCXbliFApt6q-R0hURXCZ0dkWC-za2qZ3tVXaykWmoIFBVs6wgY2budZZLhT4K9b4lbiWC5udQ6ga2ATZO1ioRg-bVZXcTin5kf__a5u6c775-8n6DeLPcfUqnLucaYr2Ov7RistJRvg

Add OpenShift provider to CloudForms using the management-admin service token.

CF_CONTAINER_2

Optional: CloudForms Container Provider

CloudForms is a cloud management platform. It integrates not only with OpenShift but also other Cloud platforms (OpenStack, Amazon, GCE, Azure) and traditional virtualization platforms (VMware, RHEV, Hyper-V). Since OpenShift is usually running on cloud or traditional virtualization platforms, CloudForms enables true end-to-end visibility. CloudForms provides not only performance metrics, events, smart state analysis of containers (scanning container contents) but also can provide chargeback for OpenShift projects. CloudForms is included in OpenShift subscription for purpose of managing OpenShift. To add OpenShift as provider in CloudForms follow the below steps.

Get token for CloudForms Access

Use management-admin token that is created in management-admin project during install to provide access to CloudForms.

# oc describe sa -n management-infra management-admin
Name: management-admin
Namespace: management-infra
Labels: 

Mountable secrets: management-admin-token-vr21i
 management-admin-dockercfg-5j3m3

Tokens: management-admin-token-mxy4m
 management-admin-token-vr21i

Image pull secrets: management-admin-dockercfg-5j3m3
# oc describe secret -n management-infra management-admin-token-mxy4m
Name: management-admin-token-mxy4m
Namespace: management-infra
Labels: 
Annotations: kubernetes.io/service-account.name=management-admin,kubernetes.io/service-account.uid=87f8f4e4-4c0f-11e6-8aca-52540057bf27

Type: kubernetes.io/service-account-token

Data
====
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtYW5hZ2VtZW50LWluZnJhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1hbmFnZW1lbnQtYWRtaW4tdG9rZW4tbXh5NG0iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibWFuYWdlbWVudC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6Ijg3ZjhmNGU0LTRjMGYtMTFlNi04YWNhLTUyNTQwMDU3YmYyNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptYW5hZ2VtZW50LWluZnJhOm1hbmFnZW1lbnQtYWRtaW4ifQ.dN-CmGdSR2TRh1h0qHvwkqnW6TLvhXJtuHX6qY2jsrZIZCg2LcyuQI9edjBhl5tDE6PfOrpmh9-1NKAA6xbbYVJlRz52gnEdtm1PVgvzh8_WnKiQLZu-xC1qRX_YL7ohbglFSf8b5zgf4lBdJbgM_2P4sm1Czhu8lr5A4ix95y40zEl3P2R_aXnns62hrRF9XpmweASGMjooKOHB_5HUcZ8QhvdgsveD4j9de-ZzYrUDHi0NqOEtenBThe5kbEpiWzSWMAkIeC2wDPEnaMTyOM2bEfY04bwz5IVS_IAnrEF7PogejgsrAQRtYss5yKSZfwNTyraAXSobgVa-e4NsWg
ca.crt: 1066 bytes
namespace: 16 bytes

Add OpenShift provider to CloudForms using token

OSE3_login_cf

Configure metrics by supplying the service name exposed by OpenShift

OSE3_hawkular_cf

Choose a container image to scan

OSEV3_SMartState

Check for scanning container

You should see scanning container start in the project management-infra.

[root@ose3-master ~]# oc project management-infra
[root@ose3-master ~]# oc get pods
NAME READY STATUS RESTARTS AGE
manageiq-img-scan-24297 0/1 ContainerCreating 0 12s
[root@ose3-master ~]# oc get pods
NAME READY STATUS RESTARTS AGE
manageiq-img-scan-24297 1/1 Running 0 1m

Check image in CloudForms

You should now see an OpenSCAP report and in addition visibility into packages that are actually installed in the container itself.

Compute->Containers-Container Images->MySQL

OSEv3_SMart_State_REsults

Packages

OSEV3_Packages

OpenScap HTML Report

OSEV3_OpenScap

Optional: Performance Metrics

OpenShift provides ability to collect performance metrics using Hawkular. This runs as container and uses cassandra to persist the data. CloudForms is able to display capacity and utilization metrics for OpenShift using Hawkular.

Switch to openshift-infra project

[root@ose3-master ~]# oc project openshift-infra

Create service account for metrics-deployer pod

[root@ose3-master ~]# oc create -f - <

Enable permissions and set secret

[root@ose3-master ~]# oadm policy add-role-to-user edit system:serviceaccount:openshift-infra:metrics-deployer
[root@ose3-master ~]#oadm policy add-cluster-role-to-user cluster-reader system:serviceaccount:openshift-infra:heapster
[root@ose3-master ~]# oc secrets new metrics-deployer nothing=/dev/null

Deploy metrics environment for OpenShift

[root@ose3-master ~]# oc new-app -f /usr/share/openshift/examples/infrastructure-templates/enterprise/metrics-deployer.yaml \
-p HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.apps.lab.com \
-p USE_PERSISTENT_STORAGE=false -p MASTER_URL=https://ose3-master.lab.com:8443

Add the metrics URL to the OpenShift master config file

# vi /etc/origin/master/master-config.yaml
assetConfig:
metricsPublicURL: "https://hawkular-metrics.apps.lab.com/hawkular/metrics"

Restart OpenShift Master

# systemctl restart atomic-openshift-master

Cleanup

# oc delete all,sa,templates,secrets,pvc --selector="metrics-infra"
# oc delete sa,secret metrics-deployer

Optional: Aggregate Logging

OpenShift Enterprise supports Kibana and the ELK Stack for log aggregation. Any pod and container that log to STDOUT will have all their log messages aggregated. This provides centralized logging for all application components. Logging is completely integrated within OpenShift and the ELK Stack runs of course containerized within OpenShift.

In openshift-infra project create service account for logging and necessary permissions.

Switch to logging project

# oc project logging

Create service accounts, roles and bindings

# oc new-app logging-deployer-account-template

Setup permissions

# oadm policy add-cluster-role-to-user oauth-editor \
       system:serviceaccount:logging:logging-deployer
# oadm policy add-scc-to-user privileged  \
    system:serviceaccount:logging:aggregated-logging-fluentd
# oadm policy add-cluster-role-to-user cluster-reader \
    system:serviceaccount:logging:aggregated-logging-fluentd

Create configmap

# oc create configmap logging-deployer \
 --from-literal kibana-hostname=kibana.lab.com \
 --from-literal public-master-url=https://ose3-master.lab.com:8443 \
 --from-literal es-cluster-size=1 \
 --from-literal es-instance-ram=2G

Create secret

# oc secrets new logging-deployer nothing=/dev/null

Deploy aggregate logging

# oc new-app logging-deployer-template

Option 1:Enable fluentd on specific nodes

# oc label node/node.example.com logging-infra-fluentd=true

Option 2: Enable fluentd all nodes

# oc label node --all logging-infra-fluentd=true

Cleanup

# oc new-app logging-deployer-template –param MODE=uninstall

Optional: Jenkins Build Pipeline

As of OpenShift 3.3 Jenkins build pipelines are integrated within OpenShift. This means you can not only execute but also observe and configure build pipelines without leaving OpenShift. In OpenShift 3.3 this feature is technology preview.

Before we build a pipleline we need to create some projects and deploy an application. For this example we have prepared a basic helloworld nodejs application. The application has two versions both are available via branches in Github. Using the build pipeline an end-to-end application upgrade and rollout will be demonstrated. The application will be deployed across three stages: development, integration and production. The development stage has both versions of the application while integration and test have one version, either v1 or v2. In development and integration the application will be scaled to only a single pod. Production is using scaling of 4 pods.

Create Projects

# oc new-project dev
# oc new-project int
# oc new-project prod

Switch to dev project

# oc project dev

Build v1 of nodejs application

# oc new-app --name v2 registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest~https://github.com/ktenzer/nodejs-ex.git

Build v2 of nodejs application

# oc new-app --name v2 registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest~https://github.com/ktenzer/nodejs-ex.git#v2

Enable Jenkins build pipelines in OpenShift

# vi /etc/origin/tech-preview/pipelines.js
window.OPENSHIFT_CONSTANTS.ENABLE_TECH_PREVIEW_FEATURE.pipelines = true;
# vi /etc/origin/master/master-config.yaml
jenkinsPipelineConfig:
  autoProvisionEnabled: true
  templateNamespace: openshift
  templateName: jenkins-ephemeral
  serviceName: jenkins
assetConfig:
  extensionScripts:
    - /etc/origin/tech-preview/pipelines.js

Restart OpenShift master

# systemctl restart atomic-openshift-master

Setup permissions for build pipeline

In this case we have three projects. Both the integration and production projects need to pull images from development. In addition jenkins service account in development where jenkins will run needs access to both integration and production projects.

# oc policy add-role-to-user edit system:serviceaccount:dev:jenkins -n prod
# oc policy add-role-to-user edit system:serviceaccount:dev:jenkins -n int
# oc policy add-role-to-user edit system:serviceaccount:dev:jenkins -n dev
# oc policy add-role-to-user system:image-puller system:serviceaccount:int:default -n dev
# oc policy add-role-to-user system:image-puller system:serviceaccount:prod:default -n dev

Reconsile roles

In case you did an upgrade from OpenShift roles might need to be reconsiled. If you have problems or issues I would recommend these steps.

# oadm policy reconcile-cluster-roles --confirm
# oadm policy reconcile-cluster-role-bindings --confirm
# oadm policy reconcile-sccs --confirm

Create Jenkins Build Pipeline

# vi pipeline.json
{
    "kind": "List",
    "apiVersion": "v1",
    "metadata": {},
    "items": [{
        "kind": "BuildConfig",
        "apiVersion": "v1",
        "metadata": {
            "name": "nodejs-pipeline-master",
            "labels": {
                "app": "nodejs-integration"
            },
            "annotations": {
                "pipeline.alpha.openshift.io/uses": "[{\"name\": \"master\", \"namespace\": \"\", \"kind\": \"DeploymentConfig\"}]"
            }
        },
        "spec": {
            "triggers": [{
                "type": "GitHub",
                "github": {
                    "secret": "EgXVqyOOobmMzjVzQHSh"
                }
            }, {
                "type": "Generic",
                "generic": {
                    "secret": "bz6uJc9u-0-58EoYKgL3"
                }
            }],
            "source": {
                "type": "Git",
                "git": {
                    "uri": "https://github.com/ktenzer/nodejs-ex.git",
                    "ref": "master"
                }
            },
            "strategy": {
                "type": "JenkinsPipeline",
                "jenkinsPipelineStrategy": {
                    "jenkinsfilePath": "jenkins-pipeline.dsl"
                }
            }
        }
    }, {
        "kind": "BuildConfig",
        "apiVersion": "v1",
        "metadata": {
            "name": "nodejs-pipeline-v2",
            "labels": {
                "app": "nodejs-integration"
            },
            "annotations": {
                "pipeline.alpha.openshift.io/uses": "[{\"name\": \"v2\", \"namespace\": \"\", \"kind\": \"DeploymentConfig\"}]"
            }
        },
        "spec": {
            "triggers": [{
                "type": "GitHub",
                "github": {
                    "secret": "EgXVqyOOobmMzjVzQHSh"
                }
            }, {
                "type": "Generic",
                "generic": {
                    "secret": "bz6uJc9u-0-58EoYKgL3"
                }
            }],
            "source": {
                "type": "Git",
                "git": {
                    "uri": "https://github.com/ktenzer/nodejs-ex.git",
                    "ref": "v2"
                }
            },
            "strategy": {
                "type": "JenkinsPipeline",
                "jenkinsPipelineStrategy": {
                    "jenkinsfilePath": "jenkins-pipeline.dsl"
                }
            }
        }
    }]
}
# oc create -f pipeline.json

Setup Pipeline for nodejs application master branch

Here we setup v1 of application in development project. Under Build->pipelines select the nodejs-pipeline-master. On right select actions->edit. You will need to change “Jenkins Type” to inline. Once you have selected inline copy/paste below Jenkins DSL:

node {
stage 'build'
openshiftBuild(buildConfig: 'v1', showBuildLogs: 'true')
stage 'deploy development'
openshiftVerifyDeployment(deploymentConfig: 'v1')
stage 'promote to int'
openshiftTag(alias: 'false', apiURL: '', authToken: '', destStream: 'helloworld', destTag: 'v1', destinationAuthToken: '', destinationNamespace: 'int', namespace: 'dev', srcStream: 'v1', srcTag: 'latest', verbose: 'false')
openshiftTag(alias: 'false', apiURL: '', authToken: '', destStream: 'acceptance', destTag: 'latest', destinationAuthToken: '', destinationNamespace: 'int', namespace: 'int', srcStream: 'helloworld', srcTag: 'v1', verbose: 'false')
stage 'deploy int'
openshiftVerifyDeployment(namespace: 'int', deploymentConfig: 'acceptance')
openshiftScale(namespace: 'int', deploymentConfig: 'acceptance',replicaCount: '1')
stage 'promote to production'
openshiftTag(alias: 'false', apiURL: '', authToken: '', destStream: 'helloworld', destTag: 'v1', destinationAuthToken: '', destinationNamespace: 'prod', namespace: 'int', srcStream: 'helloworld', srcTag: 'v1', verbose: 'false')
openshiftTag(alias: 'false', apiURL: '', authToken: '', destStream: 'production', destTag: 'latest', destinationAuthToken: '', destinationNamespace: 'prod', namespace: 'prod', srcStream: 'helloworld', srcTag: 'v1', verbose: 'false')
stage 'deploy production'
openshiftVerifyDeployment(namespace: 'prod', deploymentConfig: 'production')
openshiftScale(namespace: 'prod', deploymentConfig: 'production', replicaCount: '4')
}

Setup Pipeline for nodejs application v2 branch

Here we setup v2 of application in development project. Under Build->pipelines select the nodejs-pipeline-v2. On right select actions->edit. You will need to change “Jenkins Type” to inline. Once you have selected inline copy/paste below Jenkins DSL:

node {
stage 'build'
openshiftBuild(buildConfig: 'v2', showBuildLogs: 'true')
stage 'deploy development'
openshiftVerifyDeployment(deploymentConfig: 'v2')
stage 'promote to int'
openshiftTag(alias: 'false', apiURL: '', authToken: '', destStream: 'helloworld', destTag: 'v2', destinationAuthToken: '', destinationNamespace: 'int', namespace: 'dev', srcStream: 'v2', srcTag: 'latest', verbose: 'false')
openshiftTag(alias: 'false', apiURL: '', authToken: '', destStream: 'acceptance', destTag: 'latest', destinationAuthToken: '', destinationNamespace: 'int', namespace: 'int', srcStream: 'helloworld', srcTag: 'v2', verbose: 'false')
stage 'deploy int'
openshiftVerifyDeployment(namespace: 'int', deploymentConfig: 'acceptance')
openshiftScale(namespace: 'int', deploymentConfig: 'acceptance',replicaCount: '1')
stage 'promote to production'
openshiftTag(alias: 'false', apiURL: '', authToken: '', destStream: 'helloworld', destTag: 'v2', destinationAuthToken: '', destinationNamespace: 'prod', namespace: 'int', srcStream: 'helloworld', srcTag: 'v2', verbose: 'false')
openshiftTag(alias: 'false', apiURL: '', authToken: '', destStream: 'production', destTag: 'latest', destinationAuthToken: '', destinationNamespace: 'prod', namespace: 'prod', srcStream: 'helloworld', srcTag: 'v2', verbose: 'false')
stage 'deploy production'
openshiftVerifyDeployment(namespace: 'prod', deploymentConfig: 'production')
openshiftScale(namespace: 'prod', deploymentConfig: 'production', replicaCount: '4')
}

Optional: OpenShift Upgrade from 3.2

If you have an OpenShift 3.2 environment and want to try the integrated jenkins build pipleline follow the steps below.

OpenShift Upgrade

subscription-manager repos --disable="rhel-7-server-ose-3.2-rpms" \
    --enable="rhel-7-server-ose-3.3-rpms"\
    --enable="rhel-7-server-extras-rpms"
yum clean all
yum update atomic-openshift-utils
# ansible-playbook -i /etc/ansible/hosts /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_3/upgrade.yml
# systemctl reboot

Update images

# NS=openshift;for img in `oc get is -n ${NS}|awk '{print $1;}'|grep -v NAME`; do oc import-image -n ${NS} $img;done

Verify

# oc get nodes
 NAME STATUS AGE
 ose3-master.lab.com Ready 115d
# oc get -n default dc/docker-registry -o json | grep \"image\"
 "image": "openshift3/ose-docker-registry:v3.3.1.3",
# oc get -n default dc/router -o json | grep \"image\"
 "image": "openshift3/ose-haproxy-router:v3.3.1.3",

Upgrade Aggregate Logging

# oc project logging
# oc apply -n openshift -f \
    /usr/share/openshift/examples/infrastructure-templates/enterprise/logging-deployer.yaml
# oc process logging-deployer-account-template | oc apply -f -
# oadm policy add-cluster-role-to-user oauth-editor \
       system:serviceaccount:logging:logging-deployer

Upgrade Cluster Metrics

To upgrade metrics I recommend just deleting and re-creating. Before doing so you should prune the existing images so the new 3.3 images are downloaded.

Summary

In this article we have seen how to configure an OpenShift 3.3 all-in-one lab environment. We have also seen how install and configuration can be adapted through ansible playbook. We have seen how to configure various DNS options required by OpenShift. It should be repeated that most OpenShift problems are a direct result of improper DNS setup! We have seen how to integrate OpenShift in CloudForms, setup aggregate logging and how to configure metrics using hawkular. Finally we have seen how to create a Jenkins build pipeline using a nodejs application across three stages. As always if you have any feedback please share.

Happy OpenShifting!

(c) 2016 Keith Tenzer


bpmNEXT recording on case management

Standard
The recordings from the bpmNEXT 2016 conference (which I blogged about here) are available (for quite some time already it seems, I must have missed it somehow), but wanted to share the video of the presentation I did related to case management. 

It's a topic we are asked about regularly, and this presentation + demo might give you a good idea of where we are going:



Since then, we've made good progress, if you want to know more, take for example a look at the blog series by Maciej about case management as well.

[Short Tip] Show all variables of a host

Standard

Ansible Logo

There are multiple sources where variables for Ansible can be defined. Most of them can be shown via the setup module, but there are more.

For example, if you use a dynamic inventory script to access a Satellite server many variables like the organization are provided via the inventory script – and these are not shown in setup usually.

To get all variables of a host use the following notation:

---
- name: dump all
  hosts: all

  tasks:
  - name: get variables
    debug: var=hostvars[inventory_hostname]

Use this during debug to find out if the variables you’ve set somewhere are actually accessible in your playbooks.


Filed under: Ansible, Cloud, Debian & Ubuntu, Fedora & RHEL, Linux, Microsoft, Shell, Short Tip, SUSE, Technology