OpenStack: Integrating Ceph as Storage Backend

Standard

cephopenstack

Overview

In this article we will discuss why Ceph is Perfect fit for OpenStack. We will see how to integrate three prominent OpenStack use cases with Ceph: Cinder (block storage), Glance (images) and Nova (VM virtual disks).

Ceph provides unified scale-out storage, using commodity x86 hardware, that is self-healing and intelligently anticipates failures. It has become the defacto standard for software-defined storage. Ceph being an OpenSource project has enabled many vendors the ability to provide Ceph based software-defined storage systems. Ceph is not just limited to Companies like Red Hat, Suse, Mirantis, Ubuntu, etc. Integrated solutions from SanDisk, Fujitsu, HP, Dell, Samsung and many more exist today. There are even large-scale community built environments, Cern comes to mind, that provide storage services for 10,000s of VMs.

Ceph is by no means limited to OpenStack, however this is where Ceph started gaining traction. Looking at latest OpenStack user survey, Ceph is by a large margin the clear leader for OpenStack storage. Page 42 in OpenStack April 2016 User Survey reveals Ceph is 57% of OpenStack storage. The next is LVM (local storage) with 28% followed by NetApp with 9%. If we remove LVM, Ceph leads any other storage company by 48%, that is incredible. Why is that?

There are several reasons but I will give you my top three:

  • Ceph is a scale-out unified storage platform. OpenStack needs two things from storage: ability to scale with OpenStack itself and do so regardless of block (Cinder), File (Manila) or Object (Swift). Traditional storage vendors need to provide two or three different storage systems to achieve this. They don’t scale the same and in most cases only scale-up in never-ending migration cycles. Their management capabilities are never truly integrated across broad spectrum of storage use-cases.
  • Ceph is cost-effective. Ceph leverages Linux as an operating system instead of something proprietary. You can choose not only whom you purchase Ceph from, but also where you get your hardware. It can be same vendor or different. You can purchase commodity hardware, or even buy integrated solution of Ceph + Hardware from single vendor. There are even hyper-converged options for Ceph that are emerging (running Ceph services on compute nodes).
  • Ceph is OpenSource project just like OpenStack. This allows for a much tighter integration and cross-project development. Proprietary vendors are always playing catch-up since they have secrets to protect and their influence is usually limited in Opensource communities, especially in OpenStack context.

Below is an architecture Diagram that shows all the different OpenStack components that need storage. It shows how they integrate with Ceph and how Ceph provides a unified storage system that scales to fill all these use cases.

ceph-openstack

source: Red Hat Summit 2016

If you are interested in more topics relating to ceph and OpenStack I recommend following: http://ceph.com/category/ceph-and-openstack/

Ok enough talking about why Ceph and OpenStack are so great, lets get our hands dirty and see how to hook it up!

If you don’t have a Ceph environment you can follow this article on how to set one up quickly.

Glance Integration

Glance is the image service within OpenStack. By default images are stored locally on controllers and then copied to compute hosts when requested. The compute hosts cache the images but they need to be copied again, every time an image is updated.

Ceph provides backend for Glance allowing images to be stored in Ceph, instead of locally on controller and compute nodes. This greatly reduces network traffic for pulling images and increases performance since Ceph can clone images instead of copying them. In addition it makes migrating between OpenStack deployments or concepts like multi-site OpenStack much simpler.

Install ceph client used by Glance.

[root@osp9 ~]# yum install -y python-rbd

Create Ceph user and set home directory to /etc/ceph.

[root@osp9 ~]# mkdir /etc/ceph
[root@osp9 ~]# useradd ceph
[root@osp9 ~]# passwd ceph

Add ceph user to sudoers.

cat << EOF >/etc/sudoers.d/ceph
ceph ALL = (root) NOPASSWD:ALL
Defaults:ceph !requiretty
EOF

On Ceph admin node.

Create Ceph RBD Pool for Glance images.

[ceph@ceph1 ~]$ sudo ceph osd pool create images 128

Create keyring that will allow Glance access to pool.

[ceph@ceph1 ~]$ sudo ceph auth get-or-create client.images mon 'allow r' osd 'allow class-read object_prefix rdb_children, allow rwx pool=images' -o /etc/ceph/ceph.client.images.keyring

Copy the keyring to /etc/ceph on OpenStack controller.

[ceph@ceph1 ~]$ scp /etc/ceph/ceph.client.images.keyring root@osp9.lab:/etc/ceph

Copy /etc/ceph/ceph.conf configuration to OpenStack controller.

[ceph@ceph1 ~]$ scp /etc/ceph/ceph.conf root@osp9.lab:/etc/ceph

Set permissions so Glance can access Ceph keyring.

[root@osp9 ~(keystone_admin)]# chgrp glance /etc/ceph/ceph.client.images.keyring
[root@osp9 ~(keystone_admin)]#chmod 0640 /etc/ceph/ceph.client.images.keyring

Add keyring file to Ceph configuration.

[root@osp9 ~(keystone_admin)]# vi /etc/ceph/ceph.conf
[client.images]
keyring = /etc/ceph/ceph.client.images.keyring

Create backup of original Glance configuration.

[root@osp9 ~(keystone_admin)]# cp /etc/glance/glance-api.conf /etc/glance/glance-api.conf.orig

Update Glance configuration.

[root@osp9 ~]# vi /etc/glance/glance-api.conf
[glance_store]
stores = glance.store.rbd.Store
default_store = rbd
rbd_store_pool = images
rbd_store_user = images
rbd_store_ceph_conf = /etc/ceph/ceph.conf

Restart Glance.

[root@osp9 ~(keystone_admin)]# systemctl restart openstack-glance-api

Download Cirros images and add it into Glance.

[root@osp9 ~(keystone_admin)]# wget http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img

Convert QCOW2 to RAW. It is recommended for Ceph to always use RAW format.

[root@osp9 ~(keystone_admin)]#qemu-img convert cirros-0.3.4-x86_64-disk.img cirros-0.3.4-x86_64-disk.raw

Add image to Glance.

[root@osp9 ~(keystone_admin)]#glance image-create --name "Cirros 0.3.4" --disk-format raw --container-format bare --visibility public --file cirros-0.3.4-x86_64-disk.raw
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | ee1eca47dc88f4879d8a229cc70a07c6 |
| container_format | bare |
| created_at | 2016-09-07T12:29:23Z |
| disk_format | qcow2 |
| id | a55e9417-67af-43c5-a342-85d2c4c483f7 |
| min_disk | 0 |
| min_ram | 0 |
| name | Cirros 0.3.4 |
| owner | dd6a4ed994d04255a451da66b68a8057 |
| protected | False |
| size | 13287936 |
| status | active |
| tags | [] |
| updated_at | 2016-09-07T12:29:27Z |
| virtual_size | None |
| visibility | public |
+------------------+--------------------------------------+

Check that glance image exists in Ceph.

[ceph@ceph1 ceph-config]$ sudo rbd ls images
a55e9417-67af-43c5-a342-85d2c4c483f7
[ceph@ceph1 ceph-config]$ sudo rbd info images/a55e9417-67af-43c5-a342-85d2c4c483f7
rbd image 'a55e9417-67af-43c5-a342-85d2c4c483f7':
 size 12976 kB in 2 objects
 order 23 (8192 kB objects)
 block_name_prefix: rbd_data.183e54fd29b46
 format: 2
 features: layering, striping
 flags:
 stripe unit: 8192 kB
 stripe count: 1

Cinder Integration

Cinder is the block storage service in OpenStack. Cinder provides an abstraction around block storage and allows vendors to integrate by providing a driver. In Ceph, each storage pool can be mapped to a different Cinder backend. This allows for creating storage services such as gold, silver or bronze. You can decide for example that gold should be fast SSD disks that are replicated three times, while silver only should be replicated two times and bronze should use slower disks with erasure coding.

Create Ceph pool for cinder volumes.

[ceph@ceph1 ~]$ sudo ceph osd pool create  128

Create keyring to grant cinder access.

[ceph@ceph1 ~]$ sudo ceph auth get-or-create client.volumes mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images' -o /etc/ceph/ceph.client.volumes.keyring

Copy keyring to OpenStack controllers.

[ceph@ceph1 ~]$ scp /etc/ceph/ceph.client.volumes.keyring root@osp9.lab:/etc/ceph

Create file that contains just the authentication key on OpenStack controllers.

[ceph@ceph1 ~]$ sudo ceph auth get-key client.volumes |ssh osp9.lab tee client.volumes.key

Set permissions on keyring file so it can be accessed by Cinder.

[root@osp9 ~(keystone_admin)]# chgrp cinder /etc/ceph/ceph.client.volumes.keyring
[root@osp9 ~(keystone_admin)]# chmod 0640 /etc/ceph/ceph.client.volumes.keyring

Add keyring to Ceph configuration file on OpenStack controllers.

[root@osp9 ~(keystone_admin)]#vi /etc/ceph/ceph.conf

[client.volumes]
keyring = /etc/ceph/ceph.client.volumes.keyring

Give KVM Hypervisor access to Ceph.

[root@osp9 ~(keystone_admin)]# uuidgen |tee /etc/ceph/cinder.uuid.txt

Create a secret in virsh so KVM can access Ceph pool for cinder volumes.

[root@osp9 ~(keystone_admin)]#vi /etc/ceph/cinder.xml

<secret ephemeral="no" private="no">
 <uuid>ce6d1549-4d63-476b-afb6-88f0b196414f</uuid>
 <usage type="ceph">
 <name>client.volumes secret</name>
 </usage>
</secret>
[root@osp9 ceph]# virsh secret-define --file /etc/ceph/cinder.xml
[root@osp9 ~(keystone_admin)]# virsh secret-set-value --secret ce6d1549-4d63-476b-afb6-88f0b196414f --base64 $(cat /etc/ceph/client.volumes.key)

Add Ceph backend for Cinder.

[root@osp9 ~(keystone_admin)]#vi /etc/cinder/cinder.conf

[rbd]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2
rbd_user = volumes
rbd_secret_uuid = ce6d1549-4d63-476b-afb6-88f0b196414f

Restart Cinder service on all controllers.

[root@osp9 ~(keystone_admin)]# openstack-service restart cinder

Create a cinder volume.

[root@osp9 ~(keystone_admin)]# cinder create --display-name="test" 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-09-08T10:58:17.000000 |
| description | None |
| encrypted | False |
| id | d251bb74-5c5c-4c40-a15b-2a4a17bbed8b |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | test |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | dd6a4ed994d04255a451da66b68a8057 |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | 783d6e51e611400c80458de5d735897e |
| volume_type | None |
+--------------------------------+--------------------------------------+


List new cinder volume

[root@osp9 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| d251bb74-5c5c-4c40-a15b-2a4a17bbed8b | available | test | 1 | - | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+

List cinder volume in ceph.

[ceph@ceph1 ~]$ sudo rbd ls volumes
volume-d251bb74-5c5c-4c40-a15b-2a4a17bbed8b
[ceph@ceph1 ~]$ sudo rbd info volumes/volume-d251bb74-5c5c-4c40-a15b-2a4a17bbed8b
rbd image 'volume-d251bb74-5c5c-4c40-a15b-2a4a17bbed8b':
 size 1024 MB in 256 objects
 order 22 (4096 kB objects)
 block_name_prefix: rbd_data.2033b50c26d41
 format: 2
 features: layering, striping
 flags:
 stripe unit: 4096 kB
 stripe count: 1

Integrating Ceph with Nova Compute

Nova is the compute service within OpenStack. Nova stores virtual disks images associated with running VMs by default, locally on Hypervisor under /var/lib/nova/instances. There are a few drawbacks to using local storage on compute nodes for virtual disk images:

  • Images are stored under root filesystem. Large images can cause filesystem to fill up, thus crashing compute nodes.
  • A disk crash on compute node could cause loss of virtual disk and as such a VM recovery would be impossible.

Ceph is one of the storage backends that can integrate directly with Nova. In this section we will see how to configure that.

[ceph@ceph1 ~]$ sudo ceph osd pool create vms 128

Create authentication keyring for Nova.

[ceph@ceph1 ~]$ sudo ceph auth get-or-create client.nova mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=vms, allow rx pool=images' -o /etc/ceph/ceph.client.nova.keyring

Copy keyring to OpenStack controllers.

[ceph@ceph1 ~]$ scp /etc/ceph/ceph.client.nova.keyring root@osp9.lab:/etc/ceph

Create key file on OpenStack controllers.

[ceph@ceph1 ~]$ sudo ceph auth get-key client.nova |ssh osp9.lab tee client.nova.key

Set permissions on keyring file so it can be accessed by Nova service.

[root@osp9 ~(keystone_admin)]# chgrp nova /etc/ceph/ceph.client.nova.keyring
[root@osp9 ~(keystone_admin)]# chmod 0640 /etc/ceph/ceph.client.nova.keyring

Ensure the required rpm packages are installed.

[root@osp9 ~(keystone_admin)]# yum list installed python-rbd ceph-common
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Installed Packages
ceph-common.x86_64 1:0.94.5-15.el7cp @rhel-7-server-rhceph-1.3-mon-rpms
python-rbd.x86_64 1:0.94.5-15.el7cp @rhel-7-server-rhceph-1.3-mon-rpms

Update Ceph configuration.

[root@osp9 ~(keystone_admin)]#vi /etc/ceph/ceph.conf

[client.nova]
keyring = /etc/ceph/ceph.client.nova.keyring

Give KVM access to Ceph.

[root@osp9 ~(keystone_admin)]# uuidgen |tee /etc/ceph/nova.uuid.txt

Create a secret in virsh so KVM can access Ceph pool for cinder volumes.

[root@osp9 ~(keystone_admin)]#vi /etc/ceph/nova.xml

<secret ephemeral="no" private="no">
<uuid>c89c0a90-9648-49eb-b443-c97adb538f23</uuid>
<usage type="ceph">
<name>client.volumes secret</name>
</usage>
</secret>
[root@osp9 ~(keystone_admin)]# virsh secret-define --file /etc/ceph/nova.xml
[root@osp9 ~(keystone_admin)]# virsh secret-set-value --secret c89c0a90-9648-49eb-b443-c97adb538f23 --base64 $(cat /etc/ceph/client.nova.key)

Make backup of Nova configuration.

[root@osp9 ~(keystone_admin)]# cp /etc/nova/nova.conf /etc/nova/nova.conf.orig

Update Nova configuration to use Ceph backend.

[root@osp9 ~(keystone_admin)]#vi /etc/nova/nova.conf
force_raw_images = True
disk_cachemodes = writeback

[libvirt]
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = nova
rbd_secret_uuid = c89c0a90-9648-49eb-b443-c97adb538f23

Restart Nova services.

[root@osp9 ~(keystone_admin)]# systemctl restart openstack-nova-compute

List Neutron networks.

[root@osp9 ~(keystone_admin)]# neutron net-list
+--------------------------------------+---------+-----------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+-----------------------------------------------------+
| 4683d03d-30fc-4dd1-9b5f-eccd87340e70 | private | ef909061-34ee-4b67-80a9-829ea0a862d0 10.10.1.0/24 |
| 8d35a938-5e4f-46a2-8948-b8c4d752e81e | public | bb2b65e7-ab41-4792-8591-91507784b8d8 192.168.0.0/24 |
+--------------------------------------+---------+-----------------------------------------------------+

Start ephemeral VM instance using Cirros image that was added in the steps for Glance.

[root@osp9 ~(keystone_admin)]# nova boot --flavor m1.small --nic net-id=4683d03d-30fc-4dd1-9b5f-eccd87340e70 --image='Cirros 0.3.4' cephvm
+--------------------------------------+-----------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | wzKrvK3miVJ3 |
| config_drive | |
| created | 2016-09-08T11:41:29Z |
| flavor | m1.small (2) |
| hostId | |
| id | 85c66004-e8c6-497e-b5d3-b949a1666c90 |
| image | Cirros 0.3.4 (a55e9417-67af-43c5-a342-85d2c4c483f7) |
| key_name | - |
| metadata | {} |
| name | cephvm |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | dd6a4ed994d04255a451da66b68a8057 |
| updated | 2016-09-08T11:41:33Z |
| user_id | 783d6e51e611400c80458de5d735897e |
+--------------------------------------+-----------------------------------------------------+

Wait until the VM is active.

[root@osp9 ceph(keystone_admin)]# nova list
+--------------------------------------+--------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+--------+------------+-------------+---------------------+
| 8ca3e74e-cd52-42a6-acec-13a5b8bda53c | cephvm | ACTIVE | - | Running | private=10.10.1.106 |
+--------------------------------------+--------+--------+------------+-------------+---------------------+

List images in Ceph vms pool. We should now see the image is stored in Ceph.

[ceph@ceph1 ~]$ sudo rbd -p vms ls
8ca3e74e-cd52-42a6-acec-13a5b8bda53c_disk

Troubleshooting

Unable to delete Glance Images stored in Ceph RBD

[root@osp9 ceph(keystone_admin)]# nova image-list
+--------------------------------------+--------------+--------+--------+
| ID | Name | Status | Server |
+--------------------------------------+--------------+--------+--------+
| a55e9417-67af-43c5-a342-85d2c4c483f7 | Cirros 0.3.4 | ACTIVE | |
| 34510bb3-da95-4cb1-8a66-59f572ec0a5d | test123 | ACTIVE | |
| cf56345e-1454-4775-84f6-781912ce242b | test456 | ACTIVE | |
+--------------------------------------+--------------+--------+--------+
[root@osp9 ceph(keystone_admin)]# rbd -p images snap unprotect cf56345e-1454-4775-84f6-781912ce242b@snap
[root@osp9 ceph(keystone_admin)]# rbd -p images snap rm cf56345e-1454-4775-84f6-781912ce242b@snap
[root@osp9 ceph(keystone_admin)]# glance image-delete cf56345e-1454-4775-84f6-781912ce242b

Summary

In this article we discussed how OpenStack and Ceph fit perfectly together. We discussed some of the use cases around glance, cinder and nova. Finally we went through steps to integrate Ceph with those use cases. I hope you enjoyed the article and found the information useful. Please share your feedback.

Happy Cephing!

(c) 2016 Keith Tenzer

 


Ceph 1.3 Lab Installation and Configuration Guide

Standard

1015767

Overview

In this article we will setup a Ceph 1.3 cluster for purpose of learning or a lab environment.

 

Ceph Lab Environment

For this environment you will need three VMs (ceph1, ceph2 and ceph3). Each should have 20GB root disk and 100GB data disk. Ceph has three main components: Admin console, Monitors and OSDs.

Admin console – UI and CLI used for managing Ceph cluster. In this environment we will install on ceph1.

Monitors – Monitor health of Ceph cluster. One or more monitors forms a paxos part-time parliment, providing extreme reliability and durability of cluster membership. Monitors maintain the various maps: monitor, osd, placement group (pg) and crush. Monitors will be installed on ceph1, ceph2 and ceph3.

OSDs – Object storage daemon handles storing data, recovery, backfilling, rebalancing and replication. OSDs sit on top of a disk / filesystem. Bluestore enables OSDs to bypass filesystem but is not an option in Ceph 1.3. An OSD will be installed on ceph1, ceph2 and ceph3.

On All Cephs nodes.

#subscription-manager repos --disable=*
#subscription-manager repos --enable=rhel-7-server-rpms
#subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-rhceph-1.3-calamari-rpms --enable=rhel-7-server-rhceph-1.3-installer-rpms --enable=rhel-7-server-rhceph-1.3-tools-rpms

Configure firewalld.

sudo systemctl start firewalld
sudo systemctl enable firewalld
sudo firewall-cmd --zone=public --add-port=80/tcp --permanent
sudo firewall-cmd --zone=public --add-port=2003/tcp --permanent
sudo firewall-cmd --zone=public --add-port=4505-4506/tcp --permanent
sudo firewall-cmd --zone=public --add-port=6789/tcp --permanent
sudo firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent
sudo firewall-cmd --reload

Configure NTP.

yum -y install ntp
systemctl enable ntpd.service
systemctl start ntpd

Ensure NTP is scychronozing.

ntpq -p
remote refid st t when poll reach delay offset jitter
 ==============================================================================
 +privatewolke.co 131.188.3.222 2 u 12 64 1 26.380 -2.334 2.374
 +ridcully.episod 148.251.68.100 3 u 11 64 1 26.626 -2.425 0.534
 *s1.kelker.info 213.172.96.14 2 u 12 64 1 26.433 -6.116 1.030
 sircabirus.von- .STEP. 16 u - 64 0 0.000 0.000 0.000

Create ceph user for deployer.

#useradd ceph
#passwd ceph
#cat << EOF >/etc/sudoers.d/ceph
ceph ALL = (root) NOPASSWD:ALL
Defaults:ceph !requiretty
EOF
#chmod 0440 /etc/sudoers.d/ceph
#su - ceph
#ssh-key-gen
#ssh-copy-id ceph@ceph1
#ssh-copy-id ceph@ceph2
#ssh-copy-id ceph@ceph3

Set SELinux to permissive. Ceph 2.0 now supports SELinux but for 1.3 it was not possible out-of-box.

#vi /etc/selinux/config

SELINUXTYPE=permissive

Create ceph-config dir.

mkdir ~/ceph-config
cd ~/ceph-config

On Monitors.

#subscription-manager repos --enable=rhel-7-server-rhceph-1.3-mon-rpms
#yum update -y

On OSD Nodes.

#subscription-manager repos --enable=rhel-7-server-rhceph-1.3-osd-rpms
#yum update -y

On admin node (ceph1).

[ceph@ceph1 ~]$ vi .ssh/config

Host node1
  Hostname ceph1
  User ceph
Host node2
  Hostname ceph2
  User ceph
Host node3
  Hostname ceph3
  User ceph
chmod 600 ~/.ssh/config

On admin node (ceph1).

Setup Admin Console and Calamari.

#sudo yum -y install ceph-deploy calamari-server calamari-clients
#sudo calamari-ctl initialize
#su - ceph
[ceph@ceph1 ceph-config]$cd ~/ceph-config

Create Ceph Cluster.

#ceph-deploy new ceph1 ceph2 ceph3

Deploy Ceph monitors and OSDs.

[ceph@ceph1 ceph-config]$ceph-deploy install --mon ceph1 ceph2 ceph3
[ceph@ceph1 ceph-config]$ceph-deploy install --osd ceph1 ceph2 ceph3
[ceph@ceph1 ceph-config]$ceph-deploy --overwrite-conf mon create-initial

Connect Ceph monitors to Calamari.

[ceph@ceph1 ceph-config]$ceph-deploy calamari connect --master ceph1.lab ceph1 ceph2 ceph3
[ceph@ceph1 ceph-config]$ceph-deploy install --cli ceph1
[ceph@ceph1 ceph-config]$ceph-deploy admin ceph1

Check Ceph quorum status.

[ceph@ceph1 ceph-config]$ sudo ceph quorum_status --format json-pretty

{
 "election_epoch": 6,
 "quorum": [
 0,
 1,
 2
 ],
 "quorum_names": [
 "ceph1",
 "ceph2",
 "ceph3"
 ],
 "quorum_leader_name": "ceph1",
 "monmap": {
 "epoch": 1,
 "fsid": "188aff9b-7da5-46f3-8eb8-465e014a472e",
 "modified": "0.000000",
 "created": "0.000000",
 "mons": [
 {
 "rank": 0,
 "name": "ceph1",
 "addr": "192.168.0.31:6789\/0"
 },
 {
 "rank": 1,
 "name": "ceph2",
 "addr": "192.168.0.32:6789\/0"
 },
 {
 "rank": 2,
 "name": "ceph3",
 "addr": "192.168.0.33:6789\/0"
 }
 ]
 }
}

Set crush tables to optimal.

[ceph@ceph1 ceph-config]$sudo ceph osd crush tunables optimal

Configure OSDs.

[ceph@ceph1 ceph-config]$ceph-deploy disk zap ceph1:vdb ceph2:vdb ceph3:vdb
[ceph@ceph1 ceph-config]$ceph-deploy osd prepare ceph1:vdb ceph2:vdb ceph3:vdb
[ceph@ceph1 ceph-config]$ceph-deploy osd activate ceph1:vdb1 ceph2:vdb1 ceph3:vdb1

Connect Calamari to Ceph nodes.

[ceph@ceph1 ceph-config]$ceph-deploy calamari connect --master ceph1.lab ceph1 ceph2 ceph3

Tips and Tricks

Remove OSD from Ceph

[ceph@ceph1 ~]$sudo ceph osd out osd.0
[ceph@ceph1 ~]$sudo ceph osd crush remove osd.0
[ceph@ceph1 ~]$sudo ceph auth del osd.0
[ceph@ceph1 ~]$sudo ceph osd down 0
[ceph@ceph1 ~]$sudo ceph osd rm 0

Ceph Placement Group Calculation for Pool

  • OSDs * 100 / Replicas
  • PGs should always be power of two 62, 128, 256, etc

Re-deploy Ceph

In case at anytime you want to start over you can run below commands to uninstall Ceph. This of course deletes any data so be careful.

[ceph@ceph1 ~]$ sudo service ceph restart osd.3
ceph-deploy purge <ceph-node> [<ceph-node>]

Summary

In this article we installed a Ceph cluster on virtual machines. We deployed the cluster, setup monitors and configured OSDs.  This environment should provide the basis for a journey into software-defined storage and Ceph. The economics of scale have brought down barriers and paved the way for a software-defined world. Storage is only the next logical boundry. Ceph being an OpenSource project is already the defacto software-defined standard and is in position to become the key beneficiary of software-defined storage. I hope you found the information in this article of use, please share your experiences.

Happy Cephing!

(c) 2016 Keith Tenzer

 

 


Ceph: the future of Storage

Standard

storage-cloud1

Overview

Since joining Red Hat in 2015, I have intentionally stayed away from the topic of storage. My background is storage but I wanted to do something else as storage became completely mundane and frankly boring. Why?

Storage hasn’t changed much in 20 years. I started my career as a Linux and Storage engineer in 2000 and everything that existed then, exists today. Only things became bigger, faster, cheaper due to technologies such as flash.

I realized in late 2015 that storage industry is starting a challenging period for all vendors but didn’t really have feeling for when that could lead to real change. I did know that the monolithic storage array built on proprietary Linux/Unix with proprietary x86 hardware we all know and love was a thing of the past. If you think about it storage is a scam today, you get opensource software running on x86 hardware packaged as a proprietary solution that doesn’t interoperate with anything else. So you get none of the value of opensource and pay extra for it. I like to think that economics like gravity, eventually always wins.

Challenges

There are many challenges facing our storage industry but I will focus on three: cost, scale and agility.

Cost

Linux has become an equalizer and storage companies not building on Linux are forced to be operating system companies in addition to storage companies, a distinct disadvantage. Their R&D costs to maintain their proprietary storage operating systems reduce overall value and increase costs of their products. This is why we saw a large amount of storage startups over last 3-5 years, because of Linux and opensource. In addition most storage platforms don’t allow you to choose hardware and therefore you are often paying a premium for standard x86. Disks are a great example, typically they cost twice what you would pay through Amazon. Oh but our disks are tested and have a mean time between failure of xxxxxxx. That is total ********, what about if disks failures weren’t a big deal, didn’t cause impact and storage system automatically adjusted?

Some vendors allow you to choose different hardware platforms but they are usually limited and certainly you can’t get the storage software itself from multiple vendors. While these points may be interesting, at the end of the day cost comes down to one thing. Everyone is measured against Amazon S3. You are either cheaper than S3 (3 cents per GB per Month) or you have some explaining to do. If we consider small or medium environments this may be doable with traditional storage but, as soon as scale and complexity come into play (multiple use-cases with block, file and object) those costs explode.

Scale

Scalability is a very complex problem to solve. Sure everything scales until it doesn’t but jokes aside, we have reached a point where scalability needs to be applied more generally, especially in storage. Storage arrays today are use-case driven. Each customer has many use-cases and the larger the customer the more use-cases. This means many types of dissimilar storage systems. While a single storage system may scale to some degree, many storage systems together don’t. Customers are typically stuck in a 3-year cycle of forklift upgrades because storage systems don’t truly scale-out they only scale-up.

Agility

The result of many storage appliances and arrays is technology sprawl which in turn creates technology debt. Storage vendors don’t even have a vision for data management within their own ecosystem no less interoperating with other vendors. Everything is completely fragmented to point where a few use-cases equals a new storage system. Storage systems today require a fairly low entry cost for management overhead but as the environment grows and scales those costs increase further reducing agility. They don’t stay consistent, how could they when you don’t have a unified data management strategy. Storage systems are designed to prevent failure at all costs, they certainly don’t anticipate failure. At scale we have more failures, this in turn correlates to more time spent keeping lights on. The goal of every organization is to reduce that and maximize time spent on innovation. Finally automation suffers and it is increasingly harder to build blueprints around storage. There is just too much variation.

I could certainly go on and there are other challenges to discuss but I think you get the picture. We have reached the dead-end of storage.

Why Software-defined?

As mentioned the main problem I see today is for every use-case, there is a storage array or appliance. The startups are all one-trick ponies solving only a few use cases. Traditional storage vendors throw a different storage systems at each use-case then call it a solution. You end with not real data management strategy. I laugh when I hear vendors talking about data lakes. If you buy into storage array mindset, you end up at same place, a completely unmanageable environment at scale where operational costs are not linear but constantly going up, welcome to most storage environments today.

As complexity increases you reach a logical point where abstractions are needed. Today storage not only needs to provide file, block and object but also needs to interoperate with large ecosystem of vendors, cloud providers and applications. Decoupling the data management software from the hardware is the logical next step. This is the same thing we have already observed with server virtualization and are observing in networking with NFV. The economics of cost and advantages of decoupling hardware and software simply make sense. Organizations have been burned over and over making technology decisions that later are replaced or reverted, because newer better technologies become available in other platforms. Software-defined storage allows easy introduction of new technologies without having to purchase new storage system because your old storage was designed before that technology was invented. Finally storage migrations. Aren’t we tired of always migrating data when changing storage systems? A common data management platform using common x86 hardware and Linux could do away with storage migrations.

Why Ceph?

 

Ceph has become the defacto standard for software-defined storage. Currently the storage industry is at beginning of major disruption period where software-defined storage will drive out traditional proprietary storage systems. Ceph is of course opensource, which enables a rich eco-system of vendors to provide storage systems based on Ceph. The software-defined world is not possible without opensource and doing things the opensource way.

Ceph delivers exactly what is needed to disrupt the storage industry. Ceph provides a unified scale-out storage system based on common x86 hardware, is self healing and not only anticipates failures but expects them. Ceph does away with storage migrations and since hardware is decoupled give you choice of when to deploy new hardware technologies.

You can buy Ceph separately from hardware, so you have choice not only whom you buy Ceph from (Red Hat, Suse, Mirantis, Unbuntu, etc) but also who you purchase hardware from (HP, DELL, Fujitsu, IBM, etc). In addition you can even buy ceph together with hardware for an integrated appliance (SanDisk, Fujitsu, etc). You have choice and are free from vendor lock-in.

Ceph is extremely cost efficient. Even the most expensive all-flash, integrated solutions are less than S3 (3 cents per GB per Month). If you really want to go cheap you can by off-the-shelf commodity hardware from companies like SuperMicro and still get enterprise Ceph from Red Hat, Suse, Ubuntu, etc while being a lot cheaper than S3.

Ceph scales, one example I will give is Cern 30 PB test. Ceph can be configured to optimize different workloads such as block, file and object. You can create storage pools and decide to co-locate journals or put journals on SSDs for optimal performance. Ceph allows you to tune your storage to specific use-cases, while maintaining unified approach. In tech-preview is a new feature called bluestore. This allows Ceph to completely bypass file-system layer and store data directly on raw devices. This will greatly increase performance and there is a ton of optimizations planned after that, this is just the beginning!

Ceph enables agility by providing unified storage system that supports file, block and object. Ceph can run on VMs or physical hardware so you can easily bridge private and public clouds. Ceph provides a storage management layer for anything you can present as a disk device. Finally Ceph simplifies management, it is the same level of effort to manage a 10 node Ceph cluster or a 100 node Ceph cluster so running costs are linear.

Below is a diagram showing how Ceph addresses file, block and object storage using a unified architecture built around RADOS.

CEPH1

source: http://docs.ceph.com/docs/hammer/architecture

Summary

In this article we spent time discussing current storage challenges and the value of not just software-defined storage but also Ceph. We can’t keep doing what we have been doing for last 20+ years in storage industry. The economics of scale have brought down barriers and paved the way for a software-defined world. Storage is only the next logical boundry. Ceph being an OpenSource project is already the defacto software-defined standard and is in position to become the key beneficiary as software-defined storage becomes more mainstream. If you are interested in Ceph I will be producing some how-to guides soon, so stay tuned. Please feel free to debate whether you agree or disagree with my views. I welcome all feedback.

(c) 2016 Keith Tenzer


[Short Tip] Fix mount problems in RHV during GlusterFS mounts

Standard

Gluster Logo

When using Red Hat Virtualization or oVirt together with GLusterFS, there might be a strange error during the first creation of a storage domain:

Failed to add Storage Domain xyz.

One of the rather easy to fix reasons might be a permission problem: an initial Gluster exported file system belongs to the user root. However, the virtualization manager (ovirt-m bzw. RHV-M) does not have root rights and such needs another ownership.

In such cases, the fix is to mount the exported volume & set the user rights to the rhv-m user.

$ sudo mount -t glusterfs 192.168.122.241:my-vol /mnt
# cd /mnt/
# chown -R 36.36 .

Afterwarsd, the volume can be mounted properly. Some more general details can be found at RH KB 78503.


Filed under: Business, Cloud, Debian & Ubuntu, Fedora & RHEL, Linux, Shell, Short Tip, SUSE, Technology, Virtualization

OpenStack Mitaka Lab Installation and Configuration Guide

Standard

rdo

Overview

In this article we will focus on installing and configuring OpenStack Mitaka using RDO and the packstack installer. RDO is a community platform around Red Hat’s OpenStack Platform. It allows you to test the latest OpenStack capabilities on a stable platform such as Red Hat Enterprise Linux (RHEL) or CentOS. This guide will take you through installing the OpenStack Liberty release, configuring networking, security groups, flavors, images and are other OpenStack related services. The outcome is a working OpenStack environment based on the Mitaka release that you can use as a baseline for testing your applications with OpenStack capabilities.

Install and Configure OpenStack Liberty

  • Install RHEL or CentOS 7.1.
  • Ensure name resolution is working.
# vi /etc/hosts
192.168.122.80 osp9.lab osp9
  • Set hostname.
# hostnamectl set-hostname osp9.lab
  • Disable firewalld since this is for a lab environment.
# systemctl disable firewalld
# systemctl stop firewalld
  • Disable NetworkManager, it is still not recommended for Liberty (at least RDO).
# systemctl stop NetworkManager
# systemctl disable NetworkManager
  • For RHEL systems register with subscription manager.
# subscription-manager register
# subscription-manager subscribe --auto
# subscription-manager repos --disable=*
# subscription-manager repos --enable=rhel-7-server-rpms
# subscription-manager repos --enable=rhel-7-server-extras-rpms 
# subscription-manager repos --enable=rhel-7-server-rh-common-rpms
# subscription-manager repos --enable=rhel-7-server-openstack-9-rpms
  • Install yum-utils and update the system.
# yum install -y yum-utils
# yum update -y
  • Reboot.
# systemctl reboot
  • Install packstack packages.
# yum install -y openstack-packstack
You can install packstack by providing command-line options or using the answers file.

Option 1: Install using command-line options

 # packstack --allinone --os-neutron-ovs-bridge-mappings=extnet:br-ex \
 --os-neutron-ovs-bridge-interfaces=br-ex:eth0 \
 --os-neutron-ml2-type-drivers=vxlan,flat \
 --os-heat-install=y --os-heat-cfn-install=y \
 --os-sahara-install=y --os-trove-install=y \
 --os-neutron-lbaas-install=y \
 --keystone-admin-passwd=redhat01

Option 2: Install using answers file

  • Create packstack answers file for customizing the installer.
# packstack --gen-answer-file /root/answers.txt
  • Update the packstack answers file and enable other OpenStack services. Note: as of the writing of this guide SSL is not working in combination with Horizon so don’t enable SSL.
# vi /root/answers.txt
 CONFIG_KEYSTONE_ADMIN_PW=redhat
 CONFIG_PROVISION_DEMO=n
 CONFIG_HORIZON_SSL=y
 CONFIG_HEAT_INSTALL=y
 CONFIG_HEAT_CFN_INSTALL=y
 CONFIG_HEAT_CLOUDWATCH_INSTALL=y
 CONFIG_SAHARA_INSTALL=y
 CONFIG_CEILOMETER_INSTALL=y
 CONFIG_LBAAS_INSTALL=y
  • Install OpenStack Liberty using packstack.
# packstack --answer-file /root/answers.txt
  • Source the keystone admin profile.
# . /root/keystonerc_admin
  • Check status of openstack services.
# openstack-status
  • Backup the ifcfg-etho script.
# cp /etc/sysconfig/network-scripts/ifcfg-eth0 /root/
  • Configure external bridge for floating ip networks.
# vi /etc/sysconfig/network-scripts/ifcfg-eth0
 DEVICE=eth0
 ONBOOT=yes
 TYPE=OVSPort
 DEVICETYPE=ovs
 OVS_BRIDGE=br-ex
# vi /etc/sysconfig/network-scripts/ifcfg-br-ex
 DEVICE=br-ex
 BOOTPROTO=static
 ONBOOT=yes
 TYPE=OVSBridge
 DEVICETYPE=ovs
 USERCTL=yes
 PEERDNS=yes
 IPV6INIT=no
 IPADDR=<www.xxx.yyy.zzz>
 NETMASK=255.255.255.0
 GATEWAY=<GW IP>
 DNS1=<DNS IP>
  • Add the eht0 physical interface to the br-ex bridge in openVswitch for floating IP networks.
# ovs-vsctl add-port br-ex eth0 ; systemctl restart network.service

Configure OpenStack

  • Create private network.
# neutron net-create private
# neutron subnet-create private 10.10.1.0/24 --name private_subnet --allocation-pool start=10.10.1.100,end=10.10.1.200
  • Create public network. Note: these steps assume the physical network connected to eth0 is 192.168.122.0/24.
# neutron net-create public --router:external
# neutron subnet-create public 192.168.122.0/24 --name public_subnet --allocation-pool start=192.168.122.100,end=192.168.122.200 --disable-dhcp --gateway 192.168.122.1
  • Add a new router and configure router interfaces.
# neutron router-create router1 --ha False
# neutron router-gateway-set router1 public
# neutron router-interface-add router1 private_subnet
  • Upload a glance image. In this case we will use a Cirros image because it is small and thus good for testing OpenStack.
# yum install -y wget
# wget http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img
# glance image-create --name "Cirros 0.3.4" --disk-format qcow2 --container-format bare --visibility public --file /root/cirros-0.3.4-x86_64-disk.img
  • Create a new m1.nano flavor for running Cirros image.
# nova flavor-create m1.nano 42 64 0 1
  • Create security group and allow all TCP ports.
# nova secgroup-create all "Allow all tcp ports"
# nova secgroup-add-rule all TCP 1 65535 0.0.0.0/0
  • Create security group for base access
# nova secgroup-create base "Allow Base Access"
# nova secgroup-add-rule base TCP 22 22 0.0.0.0/0
# nova secgroup-add-rule base TCP 80 80 0.0.0.0/0
# nova secgroup-add-rule base ICMP -1 -1 0.0.0.0/0
  • Create a private ssh key for connecting to instances remotely.
# nova keypair-add admin
  • Create admin.pem file and add private key from output of keypair-add command.
# vi /root/admin.pem
# chmod 400 /root/admin.pem
  • List the network IDs.
# neutron net-list
 +--------------------------------------+---------+-------------------------------------------------------+
 | id | name | subnets |
 +--------------------------------------+---------+-------------------------------------------------------+
 | d4f3ed19-8be4-4d56-9f95-cfbac9fdf670 | private | 92d82f53-6e0b-4eef-b8b9-cae32cf40457 10.10.1.0/24     |
 | 37c024d6-8108-468c-bc25-1748db7f5e8f | public  | 22f2e901-186f-4041-ad93-f7b5ccc30a81 192.168.122.0/24 |
  • Start an instance and make sure to replace network id from above command.
# nova boot --flavor m1.nano --image "Cirros 0.3.4" --nic net-id=92d82f53-6e0b-4eef-b8b9-cae32cf40457 --key-name admin --security-groups all mycirros
  • Create a floating IP and assign it to the mycirros instance.
# nova floating-ip-create
# nova floating-ip-associate mycirros <FLOATING IP>
  • Connect to mycirros instance using the private ssh key stored in the admin.pem file. Note: The first floating IP in the range 192.168.122.201.
# ssh -i admin.pem cirros@192.168.122.201

Nova Nested Virtualization

Most OpenStack Lab or test environments will install OpenStack on a hypervisor platform inside virtual machines. I would strongly recommend KVM. If you are running OpenStack on KVM (Nova nested virtualization) make sure to follow these tips and tricks to get the best performance.

Summary

This article was intended as a hands on guide for standing up an OpenStack Mitaka lab environment using RDO. As mentioned RDO is a stable community platform built around Red Hat’s OpenStack Platform. It provides the ability to test the latest OpenStack features against either an enterprise platform (RHEL) or community platform (CentOS). Hopefully you found the information in this article useful. If you have anything to add or feedback, feel free to leave your comments.Happy OpenStacking!(c) 2016 Keith Tenzer

Cloud Systems Management: Satellite 6.2 Getting Started Guide

Standard

sat_6_components

Overview

In this article we will look at how to install Satellite 6.2 and configure a base environment. This article builds on a similar article I published for Satellite 6.1. In addition to installing and configuring Satellite we will also look at one of the long awaited new features remote-cmd execution.

If you are coming from Satellite 5 world then you will want to familiarize yourself with the concepts and how they apply in Satellite 6. The biggest change is around how you manage content through stages (life-cycle enviornments) but there is also a lot more.

Red_Hat_Satellite_6_Life_Cycle

source: https://www.windowspro.de/sites/windowspro.de/files/imagepicker/6/Red_Hat_Satellite_6_Life_Cycle.png

Features

Satellite 6.2 continues to build on that release and provides the following features:

  • Automated Workflows — This includes remote execution, scheduling for remote execution jobs and expanded bootstrap and provisioning options.

  • Air-gapped security and federation — Inter-Satellite sync is now available to export RPM content from one Satellite to import into another

  • Software Management Improvements — Simplified Smart Variable management is now available.

  • Capsule improvements — Users now have deeper insight into Capsule health and overall performance; Capsules are lighter-weight and can be configured to store only the content that has been requested by its clients; and a new Reference Architecture including deploying a Highly Available Satellite Capsule is now available.

  • Atomic OSTree and containers — Mirror, provision and manage RHEL Atomic hosts and content with Satellite; mirror container repositories such as Red Hat Registry, DockerHub™ and other 3rd-party sources; and Satellite provides a secure, curated point of entry for container content

  • Enhanced documentation — Both new and improved documentation is available. (https://access.redhat.com/documentation/red-hat-satellite/)

    • New Guides

      • Virtual Instance Guide (How to configure virt-who)

      • Hammer CLI Guide (How to use Satellite’s CLI)

      • Content Management Guide (How to easily manage Satellite’s content )

      • Quickstart Guide (How to get up and running quickly)

    • Improved/more user-friendly documentation

  • User Guide split to make more topical and easier to follow:

    • Server Administration Guide

    • Host Configuration Guide

    • “Cheat Sheets” available for specific topics (Hammer)

    • Updated Feature Overviews

Prerequisites

In order to install Satellite we need a subscription and of course RHEL 6 or 7.

subscription-manager register
subscription-manager list --available
subscription-manager attach --pool=934893843989289
subscription-manager repos --disable "*"

RHEL 6

subscription-manager repos --enable=rhel-6-server-rpms \
--enable=rhel-server-rhscl-6-rpms \
--enable=rhel-6-server-satellite-6.2-rpms

RHEL 7

subscription-manager repos --enable=rhel-7-server-rpms \
--enable=rhel-server-rhscl-7-rpms \
--enable=rhel-7-server-satellite-6.2-rpms

Update all packages.

# yum update -y

Add Firewall rules.

RHEL 6

# iptables -A INPUT -m state --state NEW -p udp --dport 53 -j ACCEPT \
&& iptables -A INPUT -m state --state NEW -p tcp --dport 53 -j ACCEPT \
&& iptables -A INPUT -m state --state NEW -p udp --dport 67 -j ACCEPT \
&& iptables -A INPUT -m state --state NEW -p udp --dport 69 -j ACCEPT \
&& iptables -A INPUT -m state --state NEW -p tcp --dport 80 -j ACCEPT \
&& iptables -A INPUT -m state --state NEW -p tcp --dport 443 -j ACCEPT \
&& iptables -A INPUT -m state --state NEW -p tcp --dport 5647 -j ACCEPT \
&& iptables -A INPUT -m state --state NEW -p tcp --dport 8140 -j ACCEPT \
&& iptables-save > /etc/sysconfig/iptables

RHEL 7

# firewall-cmd --add-service=RH-Satellite-6
# firewall-cmd --permanent --add-service=RH-Satellite-6

NTP

# yum install chrony

# systemctl start chronyd

# systemctl enable chronyd

SOS

Setting up SOS is a good idea to get faster responses from Red Hat support.

#yum install -y sos

DNS Configuration

This is only required if you want to setup an external DNS server. If you use the integrated DNS provided by Satellite you can skip this step.

[root@ipa ]# vi /etc/named.conf
 //
 // named.conf
 //
 // Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
 // server as a caching only nameserver (as a localhost DNS resolver only).
 //
 // See /usr/share/doc/bind*/sample/ for example named configuration files.
 //

include "/etc/rndc.key";

controls {
 inet 192.168.0.26 port 953 allow { 192.168.38.27; 192.168.38.26; } keys { "capsule"; };
 };

options {
 listen-on port 53 { 192.168.0.26; };
 listen-on-v6 port 53 { ::1; };
 directory "/var/named";
 dump-file "/var/named/data/cache_dump.db";
 statistics-file "/var/named/data/named_stats.txt";
 memstatistics-file "/var/named/data/named_mem_stats.txt";
 //allow-query { localhost; };
 //forwarders {
 //8.8.8.8;
 //};
 forwarders { 8.8.8.8; 8.8.4.4; };

/*
 - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
 - If you are building a RECURSIVE (caching) DNS server, you need to enable
 recursion.
 - If your recursive DNS server has a public IP address, you MUST enable access
 control to limit queries to your legitimate users. Failing to do so will
 cause your server to become part of large scale DNS amplification
 attacks. Implementing BCP38 within your network would greatly
 reduce such attack surface
 */
 recursion yes;

dnssec-enable yes;
 dnssec-validation yes;

/* Path to ISC DLV key */
 bindkeys-file "/etc/named.iscdlv.key";

managed-keys-directory "/var/named/dynamic";

pid-file "/run/named/named.pid";
 session-keyfile "/run/named/session.key";
 };

logging {
 channel default_debug {
 file "data/named.run";
 severity dynamic;
 };
 };

zone "0.168.192.in-addr.arpa" IN {
 type master;
 file "/var/named/dynamic/0.168.192-rev";
 //allow-update { 192.168.0.27; };
 update-policy {
 grant capsule zonesub ANY;
 };
 };

zone "lab" IN {
 type master;
 file "/var/named/dynamic/lab.zone";
 //allow-update { 192.168.0.27; };
 update-policy {
 grant capsule zonesub ANY;
 };

};

include "/etc/named.rfc1912.zones";
 include "/etc/named.root.key";

DNS ZONES

[root@ipa ]# cat /var/named/dynamic/
 0.168.192-rev 0.168.192-rev.old lab.zone lab.zone.jnl managed-keys.bind testdns.sh
 [root@ipa dynamic]# cat /var/named/dynamic/lab.zone
 $ORIGIN lab.
 $TTL 86400
 @ IN SOA dns1.lab. hostmaster.lab. (
 2001062501 ; serial
 21600 ; refresh after 6 hours
 3600 ; retry after 1 hour
 604800 ; expire after 1 week
 86400 ) ; minimum TTL of 1 day
 ;
 ;
 IN NS dns1.lab.
 rhevm IN A 192.168.0.20
 rhevh01 IN A 192.168.0.21
 osp8 IN A 192.168.0.22
 cf IN A 192.168.0.24
 ipa IN A 192.168.0.26
 sat6 IN A 192.168.0.27
 IN AAAA aaaa:bbbb::1
 ose-master IN A 192.168.0.25
 * 300 IN A 192.168.0.25
 ;
 ;
[root@ipa ]# cat /var/named/dynamic/0.168.192-rev
 $TTL 86400 ; 24 hours, could have been written as 24h or 1d
 $ORIGIN 0.168.192.IN-ADDR.ARPA.

@ IN SOA dns1.lab. hostmaster.lab. (
 2001062501 ; serial
 21600 ; refresh after 6 hours
 3600 ; retry after 1 hour
 604800 ; expire after 1 week
 86400 ) ; minimum TTL of 1 day

; Name servers for the zone - both out-of-zone - no A RRs required
 IN NS dns1.lab.
 ; server host definitions
 20 IN PTR rhevm.lab.
 21 IN PTR rhevh01.lab.
 22 IN PTR osp8.lab.
 24 IN PTR cf.lab.
 25 IN PTR ose3-master.lab.
 26 IN PTR ipa.lab.
 27 IN PTR sat6.lab.

Installation

Before we start with manual install, a Solution Architect and colleague of mine, Sebastian Hetze, created an automated setup script that also can integrate IDM. I strongly recommend using this script and contributing to further enhancements.

https://github.com/shetze/hammer-scripts/blob/master/sat62-setup.sh

If you are of course doing this to learn then it is definitely good to walk through manual steps so you understand concepts and what is involved.

#yum install -y satellite
With Integrated DNS
# satellite-installer --scenario satellite --foreman-admin-username admin --foreman-admin-password redhat01 --foreman-proxy-dns true --foreman-proxy-dns-interface eth0 --foreman-proxy-dns-zone example.com --foreman-proxy-dns-forwarders 8.8.8.8 --foreman-proxy-dns-reverse 0.168.192.in-addr.arpa --foreman-proxy-dhcp true --foreman-proxy-dhcp-interface eth0 --foreman-proxy-dhcp-range "192.168.0.100 192.168.0.199" --foreman-proxy-dhcp-gateway 192.168.0.1 --foreman-proxy-dhcp-nameservers 192.168.0.27 --foreman-proxy-tftp true --foreman-proxy-tftp-servername $(hostname) --capsule-puppet true --foreman-proxy-puppetca true
Without Integrated DNS
# satellite-installer --scenario satellite --foreman-admin-username admin --foreman-admin-password redhat01 --foreman-proxy-dns true --foreman-proxy-dns-interface eth0 --foreman-proxy-dns-zone sat.lab --foreman-proxy-dns-forwarders 192.168.0.26 --foreman-proxy-dns-reverse 0.168.192.in-addr.arpa --foreman-proxy-dhcp true --foreman-proxy-dhcp-interface eth0 --foreman-proxy-dhcp-range "192.168.0.225 192.168.0.250" --foreman-proxy-dhcp-gateway 192.168.0.1 --foreman-proxy-dhcp-nameservers 192.168.0.27 --foreman-proxy-tftp true --foreman-proxy-tftp-servername $(hostname) --capsule-puppet true --foreman-proxy-puppetca true

Configuration

At this point you should be able to reach the web UI using HTTPS. In this environment the url is https://sat6.lab.com. Next we need to setup the hammer CLI. Configure hammer so that we automatically pass authentication credentials.

mkdir ~/.hammer
cat > .hammer/cli_config.yml <<EOF
:foreman:
    :host: 'https://sat6.lab/'
    :username: 'admin'
    :password: 'redhat01'

EOF

External DNS Configuration

If you setup external DNS then you need to allow Satellite server to update DNS records on your external DNS server.

# vi /etc/foreman-proxy/settings.d/dns.yml
---
:enabled: true
:dns_provider: nsupdate
:dns_key: /etc/rndc.key
:dns_server: 192.168.38.2
:dns_ttl: 86400
# systemctl restart foreman-proxy

Register Satellite Server in Red Hat Network (RHN).

SAT6_REGISTER

Assign subscriptions to the Satellite server and download manifest from RHN.

SAT_ATTACHED_SUB

Note: In the next section we will be using the hammer CLI to configure Satellite. In this environment we are using the organization “Default Organization”, you would probably change this to a more specific organization name. If so you need to first create a new organization.

Upload manifest file to Satellite server.

#hammer subscription upload --organization "Default Organization" --file /root/manifest_d586388b-f556-4623-b6a0-9f76857bedbc.zip

Create a subnet in Satellite 6 under Infrastructure->Subnet. In this environment the subnet is 192.168.122.0/24 and we are using external D.

Screenshot from 2016-08-03 14:33:06

Enable basic repositories.

At minimum you need RHEL Server, Satellite Tools and RH Common.

#hammer repository-set enable --organization "Default Organization" --product 'Red Hat Enterprise Linux Server' --basearch='x86_64' --releasever='7Server' --name 'Red Hat Enterprise Linux 7 Server (RPMs)'
#hammer repository-set enable --organization "Default Organization" --product 'Red Hat Enterprise Linux Server' --basearch='x86_64' --releasever='7Server' --name 'Red Hat Enterprise Linux 7 Server (Kickstart)'
#hammer repository-set enable --organization "Default Organization" --product 'Red Hat Enterprise Linux Server' --basearch='x86_64' --name 'Red Hat Satellite Tools 6.2 (for RHEL 7 Server) (RPMs)'

#hammer repository-set enable --organization "Default Organization" --product 'Red Hat Enterprise Linux Server' --basearch='x86_64' --name 'Red Hat Enterprise Linux 7 Server - RH Common RPMs x86_64 7Server'

Enable EPEL repository for 3rd party packages.

#wget -q https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7  -O /root/RPM-GPG-KEY-EPEL-7
#hammer gpg create --key /root/RPM-GPG-KEY-EPEL-7  --name 'GPG-EPEL-7' --organization "Default Organization"

Create a new product for the EPEL repository. In Satellite 6 products are a groupings of external content outside of RHN. Products can contain RPM repositories, Puppet modules or container images.

#hammer product create --name='EPEL 3rd Party Packages' --organization "Default Organization" --description 'EPEL 3rd Party Packages'
#hammer repository create --name='EPEL 7 - x86_64' --organization "Default Organization" --product='EPEL 3rd Party Packages' --content-type='yum' --publish-via-http=true --url=http://dl.fedoraproject.org/pub/epel/7/x86_64/ --checksum-type=sha256 --gpg-key=GPG-EPEL-7

 Synchronize the repositories. This will take a while as all of the RPM packages will be downloaded. Note: you can also use the –async option to run tasks in parallel.

#hammer repository synchronize --async --organization "Default Organization" --product 'Red Hat Enterprise Linux Server'  --name 'Red Hat Enterprise Linux 7 Server Kickstart x86_64 7Server'
#hammer repository synchronize --async --organization "Default Organization" --product 'Red Hat Enterprise Linux Server'  --name 'Red Hat Satellite Tools 6.2 for RHEL 7 Server RPMs x86_64'
#hammer repository synchronize --async --organization "Default Organization" --product 'Red Hat Enterprise Linux Server'  --name 'Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server'

#hammer repository synchronize --async --organization "Default Organization" --product 'Red Hat Enterprise Linux Server'  --name 'Red Hat Enterprise Linux 7 Server - RH Common RPMs x86_64 7Server'

#hammer repository synchronize --async --organization "$ORG" --product 'EPEL 3rd Party Packages  --name  'EPEL 7 - x86_64'

Create life cycles for development and production.

#hammer lifecycle-environment create --organization "Default Organization" --description 'Development' --name 'DEV' --label development --prior Library
#hammer lifecycle-environment create --organization "Default Organization" --description 'Production' --name 'PROD' --label production --prior 'DEV'

Create content view for RHEL 7 base.

#hammer content-view create --organization "Default Organization" --name 'RHEL7_base' --label rhel7_base --description 'Core Build for RHEL 7'

#hammer content-view add-repository --organization "Default Organization" --name 'RHEL7_base' --product 'Red Hat Enterprise Linux Server' --repository 'Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server'
#hammer content-view add-repository --organization "Default Organization" --name 'RHEL7_base' --product 'Red Hat Enterprise Linux Server' --repository 'Red Hat Satellite Tools 6.2 for RHEL 7 Server RPMs x86_64'

#hammer content-view add-repository --organization "Default Organization" --name 'RHEL7_base' --product 'Red Hat Enterprise Linux Server' --repository 'Red Hat Enterprise Linux 7 Server - RH Common RPMs x86_64 7Server'

#hammer content-view add-repository --organization "Default Organization" --name 'RHEL7_base' --product 'EPEL 3rd Party Packages'  --repository  'EPEL 7 - x86_64'

Publish and promote content view to the environments.

#hammer content-view publish --organization "Default Organization" --name RHEL7_base --description 'Initial Publishing'
#hammer content-view version promote --organization "Default Organization" --content-view RHEL7_base --to-lifecycle-environment DEV
#hammer content-view version promote --organization "Default Organization" --content-view RHEL7_base --to-lifecycle-environment PROD

Add activation keys for both stage environments.

#hammer activation-key create --organization "Default Organization" --description 'RHEL7 Key for DEV' --content-view 'RHEL7_base' --unlimited-hosts --name ak-Reg_To_DEV --lifecycle-environment 'DEV'
#hammer activation-key create --organization "Default Organization" --description 'RHEL7 Key for PROD' --content-view 'RHEL7_base' --unlimited-hosts --name ak-Reg_To_PROD --lifecycle-environment 'PROD'

Add subscriptions to activation keys.

Screenshot from 2016-08-03 12:53:41

Get medium id needed for hostgroup creation.

#hammer medium list
---|----------------------------------------------------------------------------------|---------------------------------------------------------------------------------
ID | NAME | PATH 
---|----------------------------------------------------------------------------------|---------------------------------------------------------------------------------
1 | CentOS mirror | http://mirror.centos.org/centos/$version/os/$arch 
8 | CoreOS mirror | http://$release.release.core-os.net 
2 | Debian mirror | http://ftp.debian.org/debian 
9 | Default_Organization/Library/Red_Hat_Server/Red_Hat_Enterprise_Linux_7_Server... | http://sat6.lab/pulp/repos/Default_Organization/Library/content/dist/rhel/ser...
4 | Fedora Atomic mirror | http://dl.fedoraproject.org/pub/alt/atomic/stable/Cloud_Atomic/$arch/os/ 
3 | Fedora mirror | http://dl.fedoraproject.org/pub/fedora/linux/releases/$major/Server/$arch/os/ 
5 | FreeBSD mirror | http://ftp.freebsd.org/pub/FreeBSD/releases/$arch/$version-RELEASE/ 
6 | OpenSUSE mirror | http://download.opensuse.org/distribution/$version/repo/oss 
7 | Ubuntu mirror | http://archive.ubuntu.com/ubuntu 
---|----------------------------------------------------------------------------------|---------------------------------------------------------------------------------

Create a host group. A host group is a foreman construct and is used for automation of provisioning parameters. A host is provisioned based on its host group. The host group contains kickstart/provisioning templates, OS information, network information, activation keys, parameters, puppet environment and if virtual a compute profile. Note: you will need to change the hostname sat6.lab.com.

#hammer hostgroup create --architecture x86_64 --content-source-id 1 --content-view RHEL7_base --domain lab --lifecycle-environment DEV --locations 'Default Location' --name RHEL7_DEV_Servers --organizations "Default Organization" --puppet-ca-proxy sat6.lab --puppet-proxy sat6.lab --subnet VLAN_0 --partition-table 'Kickstart default' --operatingsystem 'RedHat 7.2' --medium-id 9

Add compute resource for RHEV.

# hammer compute-resource create --provider Ovirt --name RHEV --url https://rhevm.lab/api --organizations "Default Organization" --locations 'Default Location' --user admin@internal --password redhat01

Satellite 6 Bootstrapping

Satellite bootstrapping is the process for taking an already provisioned system and attaching it to the Satellite server. The minimum process is outlined below:

Install Katello package from Satellite server.

#rpm -Uvh http://sat6.lab.com/pub/katello-ca-consumer-latest.noarch.rpm

Subscribe using activation key.

#subscription-manager register --org="Default_Organization" --activationkey="DEV_CORE"

Enable Satellite tools repository and install katello agent.

#yum -y install --enablerepo rhel-7-server-satellite-tools-6.2-rpms katello-agent

In addition to get full functionality you would also need to install and configure Puppet. Many customers are also looking for a solution that would gracefully move a system attached to Satellite 5 into Satellite 6. As such there is a bootstrapping script I can recommend from Evgeni Golov (one of our top Satellite Consultants @Red Hat):

https://github.com/buildout/buildout/blob/master/bootstrap/bootstrap.py

Remote Command Execution

A new feature many Satellite 5 customers have been waiting for is remote-cmd execution. This feature allows you to run and schedule commands on clients connected to Satellite 6 server. You can think of this as poor-man’s Ansible.

Ensure remote-cmd execution is configured for Satellite capsule.

Screenshot from 2016-08-23 12:47:15

Note: If you aren’t using provisioning template “Satellite Kickstart Default” and you upgraded from Satellite 6.1, you will need to re-clone the “Satellite Kickstart Default” template and apply your changes. A snippet was added to “Satellite Kickstart Default” in order to automatically configure foreman-proxy ssh keys.

Screenshot from 2016-08-23 12:53:18.png

For systems that are already provisioned you need to copy foreman-proxy ssh key.

# ssh-copy-id -i /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy.pub root@192.168.122.210

Run “ls” command for a given client using remote-cmd execution.

# hammer job-invocation create --async --inputs "command=ls -l /root" --search-query name=client1.lab.com --job-template "Run Command - SSH Default"

Run command using input file or script.

# hammer job-invocation create --async --input-files command=/root/script.sh --search-query name=client1.lab.com --job-template "Run Command - SSH Default"

Run command on multiple hosts.

# hammer job-invocation create --async --inputs "command=ls -l /root" --search-query "name ~ client1.lab.com|client2.lab.com" --job-template "Run Command - SSH Default"

List jobs that are running, completed or failed.

# hammer job-invocation list
---|---------------------|-----------|---------|--------|---------|-------|------------------------
ID | DESCRIPTION | STATUS | SUCCESS | FAILED | PENDING | TOTAL | START 
---|---------------------|-----------|---------|--------|---------|-------|------------------------
4 | Run ls -l /root | succeeded | 1 | 0 | 0 | 1 | 2016-08-23 11:10:50 UTC
3 | Run puppet agent -t | succeeded | 1 | 0 | 0 | 1 | 2016-08-23 10:43:20 UTC
2 | Run puppet agent -t | failed | 0 | 1 | 0 | 1 | 2016-08-23 10:37:18 UTC
1 | Run puppet agent -t | failed | 0 | 1 | 0 | 1 | 2016-08-23 10:19:41 UTC
---|---------------------|-----------|---------|--------|---------|-------|------------------------

Show details of a completed job.

# hammer job-invocation info --id 3
ID: 4
Description: Run ls -l /root
Status: succeeded
Success: 2
Failed: 0
Pending: 0
Total: 2
Start: 2016-08-23 11:34:05 UTC
Job Category: Commands
Mode: 
Cron line: 
Recurring logic ID: 
Hosts: 
 - client1.lab.com
 - client2.lab.com

Show output from a completed command for a given host. Satellite will show stdout and return code from command.

# hammer job-invocation output --host client1.lab.com --id 4
total 36
-rw-------. 1 root root 4256 Aug 23 10:34 anaconda-ks.cfg
-rw-r--r--. 1 root root 21054 Aug 23 10:34 install.post.log
-rw-r--r--. 1 root root 54 Aug 23 10:33 install.postnochroot.log
Exit status: 0

Summary

In this article we learned how to deploy Satellite 6.2 environment. We looked at some different options regarding DNS. Provided a guideline for getting a basic Satellite 6.2 environment up and running. Finally we looked a bit more into the new feature remote-cmd execution. I hope you found this article useful. If you have anything to share or otherwise feedback please don’t be shy.

Happy Satelliting!

(c) 2016 Keith Tenzer

 

 

 


Using github to version some files

Standard

Overview

I want to store some config files, scripts or Ansible playbooks  on a central web service to have

  • some versioning
  • know where the newest truth resides

I’m not a developer and don’t do lot of collaborative coding in a big spread team. So  i manly need to add files, change files and consume them.

Up to now i know git is “very easy” but git has many different commands to achieve similar things. “pull, fetch, merge, clone” are all downloading content to a local repository or maybe they do  part of what you need. As i want to manage real brain work output (written scripts, playbooks or config files), i prefer to not accidentally loose some content.

I found answer 12 in [1] very useful.

Solution

I’m keeping together all files belonging to the same Project or Subject in one github repository.

Creating a new repository

I do have an account on https://github.com/ and logged in through the web frontend. Clicking “+” on the right upper corner gets me a new repository. I gave the repository a Name e.g. “myrepo”.

Note: I try to find names representing the proejctide or the environment the content is for. “myrepo” is only goodas placeholder inside this documentation.

I fill readme.md with some reasonable content and may also file a lisence.md file.

That’s all.

Editing and creating new Files – first time

To edit, change or add files to your repository you need to clone the repository to your local filesystem. Change files there , commit them and push them back to the original repository.

I define some global user information:

$ git config --global user.name "mschreie"
$ git config --global user.email mschreie@redhat.com
$ ### cerating a starting-Directory where my (cloned) repositories reside
$ mkdir git
$ cd $_
$ git clone https://github.com/mschreie/myrepo
$ cd myrepo/

If you want to add a new file

… copy file from some other place or create a new one in this directory and add all files not added yet, e.g.

$ copy /etc/ansible/hosts.cfg .
$ git add .

If you want to edit a file

just  do so, e.g.

$ vi <file>

 In both cases

you need to commit your changes and push them to the central/original repository

$ git commit -m "some relevant comment"
$ git status -s 
$ git push origin master

Editing and creating new Files – second time

If you edited some files on the local  server before. You already have a git-directory and a local clone of the git-Repo.

You would then need to update your repo, do the changes, commit and push.

To update your local clone repository with the newest content of your github repository:

$ git pull

Consuming files

You could clone the whole repository and make use of all files, but you can also download distinct files through

wget https://github.com/mschreie/myrepo/raw/master/file

Pleas note  “raw” in the URL. If you navigate through the web front end to the file you are looking for, you will get exactly the same URL but as “blob” with more fancy layout around.

Conclusion

We now are able to have one authoritative space, where the self created files reside. You are able to add new files, change files and consume what you created. You are also able to organize the files in different repositories.

git can do much more – it’s all i need in the first place from a non-developer perspective.

 

[1] http://stackoverflow.com/questions/9577968/how-to-upload-files-on-github

 


OpenShift v3: Basic Release Deployment Scenarios

Standard

3d small people - Males with four puzzle together

source: http://snsoftwarelabs.com/

Overview

One of the hardest things companies struggle with today is release management. Of course many methodologies and even more tools or technologies exist, but how do we bring everything together and work across functional boundaries of an organization? A product release involves everyone in the company not just a single team. Many companies struggle with this and the result is a much slower innovation cycle. In the past this used to be something that at least wasn’t a deal breaker. Unfortunately that is no longer the case. Today companies live and die by their ability to not only innovate but release innovation. I would say innovating is the easy part, the ability to provide those innovations in a controlled fashion through products and services is the real challenge.

Moving to micro-services architectures and container based technologies such as Docker have simplified or streamlined many technological aspects. Certainly at least providing light at the end of the tunnel, but until OpenShift there wasn’t a platform to bring it all together, allowing development and operations teams to work together while still maintaining their areas of focus or control. In this article we will look at three scenarios for handling application deployments within OpenShift that involve both operations and development. Each scenario builds on the other and should give you a good idea of the new possibilities with OpenShift. Also keep in mind, these are basic scenarios and we are just scratching the surface so this should be viewed as a starting point for doing application deployments or release management in the new containerized world.

Scenario 1: Development leveraging customized image from Operations

Typically operations teams will want to control application runtime environment. Ensure that the application runtime environment meets all security policies, provides needed capabilities and is updated on regular basis.

Development teams want to focus on innovation through application functionality, stability and capabilities.

OpenShift allows both teams to focus on their core responsibility while also providing a means to integrate inputs/outputs of various teams into a end-to-end release.

There are many ways to integrate DevOps teams in OpenShift. One simple way is by separating development and operations into different projects and allowing development to their application runtime environment from operations. In this scenario we will see how to do that using a basic ruby hello-world application as example.

Create Projects

Create operations and development projects for our ruby application.

# oc login -u admin
# oc new-project ruby-ops
# oc new-project ruby-dev

Setup Users

Create a user for development and operations.

# htpasswd /etc/origin/master/htpasswd dev
# htpasswd /etc/origin/master/htpasswd ops

Create ops and dev users

# htpasswd /etc/origin/master/htpasswd dev
# htpasswd /etc/origin/master/htpasswd ops

Enable permissions.

Create three groups that allow operations to edit the ruby-ops project, allow development to view the ruby-ops project and also edit the ruby-dev project. In addition the ruby-dev project needs permission to pull images from the ruby-ops project.

Create groups and add users to correct groups.

# oadm groups new ops-edit && oadm groups new dev-view && oadm groups new dev-edit
# oadm groups add-users ops-edit ops && oadm groups add-users dev-view dev && \
oadm groups add-users dev-edit dev

Associate groups to projects and setup pull permissions to allow ruby-dev to pull images from ruby-ops.

# oadm policy add-role-to-group edit ops-edit -n ruby-ops && \
# oadm policy add-role-to-group view dev-view -n ruby-ops && \
# oadm policy add-role-to-group edit dev-edit -n ruby-dev && \
# oadm policy add-role-to-group system:image-puller system:serviceaccounts:ruby-dev -n ruby-ops

Operations Ruby Environment

As ops user create a ruby runtime image using application test code.

# oc login -u ops
# oc project ruby-ops
# oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-hello-world.git

Application requires database and service name called database.

# oc new-app mysql-ephemeral -p DATABASE_SERVICE_NAME=database
# oc env dc database --list | oc env dc ruby-hello-world -e -

Development Ruby Environment

As dev user pull the operations ruby runtime image and build using latest code from different Github branch or project.

# oc login -u dev
# oc project ruby-dev
# oc new-app ruby-ops/ruby-22-centos7:latest~https://github.com/ktenzer/ruby-hello-world.git

Application requires database and service name called database.

# oc new-app mysql-ephemeral -p DATABASE_SERVICE_NAME=database
# oc env dc database --list | oc env dc ruby-hello-world -e -

Scenario 2: Development to Production Promotion

Once development has an application version with required functionality and having passed all tests it can be promoted to other environments. Typically that would be qa, test, integration and eventually production. In this simple example using the ticket-monster application we will promote directly from development to production.

Using similar concepts as describe in scenario 1 this technique relies on production pulling appropriate images from development. In this scenario however we will create deployment configs and within setup a trigger that when an image with particular name/tag is updated in development, promotion to production will occur automatically. Scenario 1 shows how to do this manually, here we do it automatically and in scenario 3 we will see how to do promotion using jenkins that enables building complex pipelines with approval processes.

Create projects and setup pull permissions

# oc new-project ticket-monster-dev
# oc new-project ticket-monster-prod
# oc policy add-role-to-group system:image-puller system:serviceaccounts:ticket-monster-prod -n ticket-monster-dev

Create ticket monster template for development

# vi monster.yaml
kind: Template
apiVersion: v1
metadata:
 name: monster
 annotations:
 tags: instant-app,javaee
 iconClass: icon-jboss
 description: |
 Ticket Monster is a moderately complex application that demonstrates how
 to build modern applications using JBoss web technologies

parameters:
- name: GIT_URI
 value: git://github.com/kenthua/ticket-monster-ose
- name: MYSQL_DATABASE
 value: monster
- name: MYSQL_USER
 value: monster
- name: MYSQL_PASSWORD
 from: '[a-zA-Z0-9]{8}'
 generate: expression


objects:
- kind: ImageStream
 apiVersion: v1
 metadata:
 name: monster

- kind: BuildConfig
 apiVersion: v1
 metadata:
 name: monster
 spec:
 triggers:
 - type: Generic
 generic:
 secret: secret
 - type: ImageChange
 - type: ConfigChange
 strategy:
 type: Source
 sourceStrategy:
 from:
 kind: ImageStreamTag
 name: jboss-eap64-openshift:latest
 namespace: openshift
 source:
 type: Git
 git:
 uri: ${GIT_URI}
 ref: master
 output:
 to:
 kind: ImageStreamTag
 name: monster:latest

- kind: DeploymentConfig
 apiVersion: v1
 metadata:
 name: monster
 spec:
 replicas: 1
 selector:
 deploymentConfig: monster
 template:
 metadata:
 labels:
 deploymentConfig: monster
 name: monster
 spec:
 containers:
 - name: monster
 image: monster
 ports:
 - name: http
 containerPort: 8080
 - name: jolokia
 containerPort: 8778
 - name: debug
 containerPort: 8787
 readinessProbe:
 exec:
 command:
 - /bin/bash
 - -c
 - /opt/eap/bin/readinessProbe.sh
 env:
 - name: DB_SERVICE_PREFIX_MAPPING
 value: monster-mysql=DB
 - name: TX_DATABASE_PREFIX_MAPPING
 value: monster-mysql=DB
 - name: DB_JNDI
 value: java:jboss/datasources/MySQLDS
 - name: DB_DATABASE
 value: ${MYSQL_DATABASE}
 - name: DB_USERNAME
 value: ${MYSQL_USER}
 - name: DB_PASSWORD
 value: ${MYSQL_PASSWORD}
 - name: JAVA_OPTS
 value: "-Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.logmanager -Djava.awt.headless=true -Djboss.modules.policy-permissions=true"
 - name: DEBUG
 value: "true"
 triggers:
 - type: ImageChange
 imageChangeParams:
 automatic: true
 containerNames:
 - monster
 from:
 kind: ImageStream
 name: monster

- kind: DeploymentConfig
 apiVersion: v1
 metadata:
 name: monster-mysql
 spec:
 triggers:
 - type: ImageChange
 imageChangeParams:
 automatic: true
 containerNames:
 - monster-mysql
 from:
 kind: ImageStreamTag
 name: mysql:latest
 namespace: openshift
 replicas: 1
 selector:
 deploymentConfig: monster-mysql
 template:
 metadata:
 labels:
 deploymentConfig: monster-mysql
 name: monster-mysql
 spec:
 containers:
 - name: monster-mysql
 image: mysql
 ports:
 - containerPort: 3306
 env:
 - name: MYSQL_USER
 value: ${MYSQL_USER}
 - name: MYSQL_PASSWORD
 value: ${MYSQL_PASSWORD}
 - name: MYSQL_DATABASE
 value: ${MYSQL_DATABASE}

- kind: Service
 apiVersion: v1
 metadata:
 name: monster
 spec:
 ports:
 - name: http
 port: 8080
 selector:
 deploymentConfig: monster

- kind: Service
 apiVersion: v1
 metadata:
 name: monster-mysql
 spec:
 ports:
 - port: 3306
 selector:
 deploymentConfig: monster-mysql

- kind: Route
 apiVersion: v1
 metadata:
 name: monster
 spec:
 to:
 name: monster
# oc create -n openshift -f monster.yaml

Create template for ticket monster production environment

Below trigger will only deploy production environment when the image stream in development is tagged with monster:prod.

# vi monster-prod.yaml
kind: Template
apiVersion: v1
metadata:
 name: monster-prod
 annotations:
 tags: instant-app,javaee
 iconClass: icon-jboss
 description: |
 Ticket Monster is a moderately complex application that demonstrates how
 to build modern applications using JBoss web technologies. This template
 is for "production deployments" of Ticket Monster.

parameters:
- name: MYSQL_DATABASE
 value: monster
- name: MYSQL_USER
 value: monster
- name: MYSQL_PASSWORD
 from: '[a-zA-Z0-9]{8}'
 generate: expression

objects:
- kind: DeploymentConfig
 apiVersion: v1
 metadata:
 name: monster
 spec:
 replicas: 3
 selector:
 deploymentConfig: monster
 template:
 metadata:
 labels:
 deploymentConfig: monster
 name: monster
 spec:
 containers:
 - name: monster
 image: monster
 ports:
 - name: http
 containerPort: 8080
 - name: jolokia
 containerPort: 8778
 readinessProbe:
 exec:
 command:
 - /bin/bash
 - -c
 - /opt/eap/bin/readinessProbe.sh
 env:
 - name: DB_SERVICE_PREFIX_MAPPING
 value: monster-mysql=DB
 - name: TX_DATABASE_PREFIX_MAPPING
 value: monster-mysql=DB
 - name: DB_JNDI
 value: java:jboss/datasources/MySQLDS
 - name: DB_DATABASE
 value: ${MYSQL_DATABASE}
 - name: DB_USERNAME
 value: ${MYSQL_USER}
 - name: DB_PASSWORD
 value: ${MYSQL_PASSWORD}
 - name: JAVA_OPTS
 value: "-Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.logmanager -Djava.awt.headless=true -Djboss.modules.policy-permissions=true"
 triggers:
 - type: ImageChange
 imageChangeParams:
 automatic: true
 containerNames:
 - monster
 from:
 kind: ImageStreamTag
 name: monster:prod
 namespace: ticket-monster-dev

- kind: DeploymentConfig
 apiVersion: v1
 metadata:
 name: monster-mysql
 spec:
 triggers:
 - type: ImageChange
 imageChangeParams:
 automatic: true
 containerNames:
 - monster-mysql
 from:
 kind: ImageStreamTag
 name: mysql:latest
 namespace: openshift
 replicas: 1
 selector:
 deploymentConfig: monster-mysql
 template:
 metadata:
 labels:
 deploymentConfig: monster-mysql
 name: monster-mysql
 spec:
 containers:
 - name: monster-mysql
 image: mysql
 ports:
 - containerPort: 3306
 env:
 - name: MYSQL_USER
 value: ${MYSQL_USER}
 - name: MYSQL_PASSWORD
 value: ${MYSQL_PASSWORD}
 - name: MYSQL_DATABASE
 value: ${MYSQL_DATABASE}

- kind: Service
 apiVersion: v1
 metadata:
 name: monster
 spec:
 ports:
 - name: http
 port: 8080
 selector:
 deploymentConfig: monster

- kind: Service
 apiVersion: v1
 metadata:
 name: monster-mysql
 spec:
 ports:
 - port: 3306
 selector:
 deploymentConfig: monster-mysql

- kind: Route
 apiVersion: v1
 metadata:
 name: monster
 spec:
 to:
 name: monster
# oc create -n openshift -f monster-prod.yaml

Deploy ticket-monster development environment

Using the UI you can now choose template monster (monster).

monster

Deploy ticket-monster production environment

Deploy ticket-monster production template (monster-prod).

monster-prod

You will notice in development the environment is built and you can access the application using the service URL: http://monster-ticket-monster-dev.apps.lab

app

If you look at production environment, the database is running and a service endpoint exists however ticket monster application is not running. The production template is as mentioned setup to pull images from development automatically that have a certain name/tag. The production environment also runs scale of 4 for application where development only has scale of 1.

Promote development application version to production

Get the image stream pull spec.

# oc get is monster -o yaml
apiVersion: v1
kind: ImageStream
metadata:
 annotations:
 openshift.io/image.dockerRepositoryCheck: 2016-08-09T13:37:47Z
 creationTimestamp: 2016-08-09T13:14:53Z
 generation: 7
 name: monster
 namespace: ticket-monster-dev
 resourceVersion: "107170"
 selfLink: /oapi/v1/namespaces/ticket-monster-dev/imagestreams/monster
 uid: 42740a3d-5e33-11e6-aa8d-001a4ae42e01
spec:
 tags:
 - annotations: null
 from:
 kind: ImageStreamImage
 name: monster@sha256:3a48a056a58f50764953ba856d90eba73dd0dfdee10b8cb6837b0fd9461da7f9
 generation: 7
 importPolicy: {}
 name: prod
status:
 dockerImageRepository: 172.30.139.50:5000/ticket-monster-dev/monster
 tags:
 - items:
 - created: 2016-08-09T13:26:04Z
 dockerImageReference: 172.30.139.50:5000/ticket-monster-dev/monster@sha256:3a48a056a58f50764953ba856d90eba73dd0dfdee10b8cb6837b0fd9461da7f9
 generation: 1
 image: sha256:3a48a056a58f50764953ba856d90eba73dd0dfdee10b8cb6837b0fd9461da7f9
 tag: latest

Once you have the image stream pull specification (above in bold), tag the image stream monster:prod.

# oc tag monster@sha256:3a48a056a58f50764953ba856d90eba73dd0dfdee10b8cb6837b0fd9461da7f9 monster:prod

You can verify that the image stream in fact has a tag prod.

# oc get is
NAME DOCKER REPO TAGS UPDATED
monster 172.30.139.50:5000/ticket-monster-dev/monster prod,latest 2 minutes ago

As soon as the image stream in ticket-monster-dev is tagged with monster:prod it will be deployed to production. As mentioned above the scale in production is 4 instead of 1.

ticket-monster-prod

Scenario 3: AB Deployment using Jenkins

In this scenario we will look at how to do a simple AB deployment using Jenkins. This scenario builds off what we learned in scenario 1 and 2. In this scenario we will create a slightly more complex environment with three environments: development, integration and production. We will also build two version of our application, v1 and v2 in dev environment. Using Jenkins we will show how to promote v2 of application from development to integration to production. Finally we will show how to rollback production from v2 to v1 of application. I want to thank a colleague, Torben Jaeger who created a lot of the below content.

Create projects

# oc new-project dev && \
oc new-project int && \
oc new-project prod

Setup pull permissions

Allow int environment to pull images from dev environment and prod to pull images from both dev and int environments.

# oc policy add-role-to-group system:image-puller system:serviceaccounts:int -n dev
# oc policy add-role-to-group system:image-puller system:serviceaccounts:prod -n int
# oc policy add-role-to-group system:image-puller system:serviceaccounts:prod -n dev

Setup jenkins in development environment

Using the UI go to the dev project, select add to project and choose jenkins-ephemeral.

jenkins

Clone Github repository

# git clone https://github.com/ktenzer/openshift-demo.git

Update auth tokens for Jenkins

Get Auth token for OpenShift API. This is needed to allow Jenkins to access OpenShift environment. You need to update the auth tokens.

# oc login -u admin
# oc whoami -t
DMzhKyEN87DZiDYV6i1d8L8NL2e6gFVFPpT5FnozKtU

Update the below jenkins jobs and replace authToken and destinationAuthToken using your token from above.

# ls jenkins-jobs/
promote-int.xml promote-prod.xml rel-v2.xml rollback-prod.xml

Setup all three environements

Using templates you updated above with your auth token create all environments.

# cd openshift-demo
oc create -f template.json -n dev
# oc create -f acceptance.template.json -n int
# oc create -f production.template.json -n prod

Deploy nodejs hello-world application in dev environment

Integration is setup to pull from development and production is setup to pull from integration.

# oc new-app -f template.json -n dev

This template creates two versions of the application: v1-ab and v2-ab.

dev

Test applications in development

Connecting to v1-ab via http or curl should print “Hello World!”. Connecting to v2-ab via http or curl should print “Hello World, welcome to Frankfurt!”.

Deploy v1 from development to integration

Deploy v1 to int using tags. A trigger is set on int (acceptance) template to deploy when the image acceptance:latest is updated. You should see that a pod is started and running version v1 of application.

oc tag v1:latest v1:1.0 -n dev
oc tag dev/v1:1.0 acceptance:v1 -n int
oc tag acceptance:v1 acceptance:latest -n int

Deploy v1 from integration to production

Deploy v1 to prod using tags. A trigger is set on prod template to deploy when the image production:latest is updated.

oc tag int/acceptance:v1 production:v1 -n prod
oc tag production:v1 production:latest -n prod

Notice the application scale is 4 and the version of code running in prod is v1. Again this is all defined in the template we provided.

prod-dep

Configure Jenkins

You have seen how to promote application versions manually using image tags and triggers in the templates. Next lets get a bit more sophisticated and orchestrate the same things through jenkins using the OpenShift plugin. This time we will promote the version v2 through int and prod environments.

There are jenkins jobs to promote a release from development to integration and also from integration to production. There is a job to build a new version of the v2 application and finally perform a rollback in production.

# curl -k -u admin:password -XPOST -d @jenkins-jobs/rel-v2.xml 'https://jenkins-dev.apps.lab/createItem?name=rel-v2' -H "Content-Type: application/xml"
# curl -k -u admin:password -XPOST -d @jenkins-jobs/promote-int.xml 'https://jenkins-dev.apps.lab/createItem?name=promote-int' -H "Content-Type: application/xml"
# curl -k -u admin:password -XPOST -d @jenkins-jobs/promote-prod.xml 'https://jenkins-dev.apps.lab/createItem?name=promote-prod' -H "Content-Type: application/xml"
# curl -k -u admin:password -XPOST -d @jenkins-jobs/rollback-prod.xml 'https://jenkins-dev.apps.lab/createItem?name=rollback-prod' -H "Content-Type: application/xml"

Once you create the jobs you can login to jenkins using the url with user: admin password: password.

jenkins_ippes

Optional Step: make change in v2 code

In order for this to work you need to fork below nodejs-ex Github repository and update the template.json with your github URL. You would then need to redeploy the dev environment using new template.

 "source": {
 "type": "Git",
 "git": {
 "uri": "https://github.com/ktenzer/nodejs-ex.git",
 "ref": "master"
 },
# git clone https://github.com/ktenzer/nodejs-ex
# cd nodejs-ex

Checkout v2 branch and make commit.

# git checkout v2
# vi index.html
Hello World, welcome to Munich!

Commit changes.

# git commit -a -m "updated to munich"
# git push origin v2

Run rhel-v2 build from Jenkins

rhel-v2-ex

If you cloned the nodejs-ex repository then you should see your changes in v2 by clicking the URL or using curl.

Promote v2 from development to integration

Run promote-int build from Jenkins. You will see a new pod is started with v2 code next to v1 pod.

promote-int-ex

Promote v2 from integration to production

Here we will take a deeper look into what is actually happening under the hood.

Using curl you can see how the application is switched from v1 to v2.

# for i in {1..10000};do curl prod-ab.apps.lab; sleep 1;done

Run promote-prod build from Jenkins.

promote-prod-ex

 

The deployment is started and a v2 pods are started next to v1 pods. At this point application service still is set to v1.

prod1

Two v2 pods are running and readiness checks are done to ensure v2 application is responding. The v1 application pods are set to scale down.

prod2

All four v2 pods are running and the v1 pods are scaling to 0. We can see that both v1 and v2 are responding to requests while we are in transition.

prod3

Only v2 pods are running and the AB deployment of v2 is complete.

prod4

Now lets assume we aren’t satisfied with v2 and want to do a rollback of v1.

Run rollback-prod build from Jenkins.

rollback-ex

Here we observe the same thing as in our previous step, only we are switching from v2 to v1.

prod7

Summary

In this article we have seen how OpenShift simplifies application deployments and integrates with tools such as Jenkins that enable release management. You have many options using OpenShift and we have only really begun to scratched the surface. With Jenkins you can create very complex build pipelines that allow you to not only control but also visualize your application deployment processes. We have looked at one type of common deployment, the AB deployment but there are also other deployment types such as blue-green or canary. In a future article I will take a look at additional deployment models in OpenShift. If you have feedback or experience then please share your thoughts. I hope you have found this article useful and informative.

Happy OpenShifting!

(c) 2016 Keith Tenzer


OpenShift Enterprise 3.2: all-in-one Lab Environment

Standard

Screenshot from 2016-08-04 14:40:07

Overview

In this article we will setup a OpenShift Enterprise 3.2 all-in-one configuration. We will also setup the integration with CloudForms that allows additional management of OpenShift environments.

OpenShift has several different roles: masters, nodes, etcd and load balancers. An all-in-one setup means running all service on a single system. Since we are only using a single system a load balancer or ha-proxy won’t be configured. If you would like to read more about OpenShift I can recommend the following:

Prerequisites

Configure a VM with following:

  • RHEL 7.2
  • 2 CPUs
  • 4096 RAM
  • 30GB disk for OS
  • 25GB disk for docker images
# subscription-manager repos --disable="*"
# subscription-manager repos \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-server-ose-3.2-rpms"
# yum install -y wget git net-tools bind-utils iptables-services bridge-utils bash-completion
# yum update -y
# yum install -y atomic-openshift-utils
# systemctl reboot
# yum install -y docker-1.10.3
# vi /etc/sysconfig/docker
OPTIONS='--selinux-enabled --insecure-registry 172.30.0.0/16'
# cat <<EOF > /etc/sysconfig/docker-storage-setup
DEVS=/dev/vdb
VG=docker-vg
EOF
# docker-storage-setup
# systemctl enable docker
# systemctl start docker
# ssh-keygen
# ssh-copy-id -i /root/.ssh/id_rsa-pub ose3-master.lab.com

DNS Setup

DNS is a requirement for OpenShift Enterprise. In fact most issues you may run into are a result of not having a properly working DNS environment. For OpenShift you can either use dnsmasq or bind. I recommend using bind but in this article I will cover both options.

DNSMASQ

A colleague Ivan Mckinely was nice enough to create an ansible playbook for deploying dnsmasq. To deploy dnsmasq run following steps on OpenShift master.

# git clone https://github.com/ivanthelad/ansible-aos-scripts.git
#cd ansible-aos-scripts

Edit inventory file and set dns to IP of the system that should be providing DNS. Also ensure nodes and masters have correct IPs for your OpenShift servers. In our case 192.168.122.60 is master, node and DNS.

#vi inventory

[dns]
192.168.122.60
[nodes]
192.168.122.60
[masters]
192.168.122.60

Configure dnsmasq and add wildcard DNS so all hosts with

# vi playbooks/roles/dnsmasq/templates/dnsmasq.conf

strict-order
domain-needed
local=/lab.com/
bind-dynamic
resolv-file=/etc/resolv.conf.upstream
no-hosts
address=/.cloudapps.lab.com/192.168.122.60
address=/ose3-master.lab.com/192.168.122.60
log-queries

Ensure all hosts you want in DNS are also in /etc/hosts. The dnsmasq service reads /etc/hosts upon startup so all entries in hosts file can be queried through DNS.

#vi /etc/hosts

192.168.122.60  ose3-master.lab.com     ose3-master

Configure ssh on DNS host

#ssh-keygen
#ssh-copy-id -i ~/.ssh/id_rsa.pub ose3-master.lab.com

Install dnsmasq via ansible

# ansible-playbook -i inventory playbooks/install_dnsmas.yml

If you need to make changes you can edit the /etc/dnsmasq.conf file and restart dnsmasq service. Below is a sample dnsmasq.conf.

# vi /etc/dnsmasq.conf
strict-order
domain-needed
local=/example.com/
bind-dynamic
resolv-file=/etc/resolv.conf.upstream
no-hosts
address=/.apps.lab.com/192.168.122.60
address=/ose3-master.lab.com/192.168.122.60
address=/kubernetes.default.svc/192.168.122.60
log-queries

NAMED

Install DNS tools and utilities.

# yum -y install bind bind-utils
# systemctl enable named
# systemctl start named

Set firewall rules using iptables.

# iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 53 -j ACCEPT
# iptables -A INPUT -p udp -m state --state NEW -m udp --dport 53 -j ACCEPT

Save the iptables Using.

# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]

Note: If you are using firewalld you can just enable service DNS using firwall-cmd utility.

Example of zone file for lab.com.

vi /var/named/dynamic/lab.com.zone 

$ORIGIN lab.com.
$TTL 86400
@ IN SOA dns1.lab.com. hostmaster.lab.com. (
 2001062501 ; serial
 21600 ; refresh after 6 hours
 3600 ; retry after 1 hour
 604800 ; expire after 1 week
 86400 ) ; minimum TTL of 1 day
;
;
 IN NS dns1.lab.com.
dns1 IN A 192.168.122.1
 IN AAAA aaaa:bbbb::1
ose3-master IN A 192.168.122.60
*.cloudapps 300 IN A 192.168.122.60

Example of named configuration.

# vi /etc/named.conf 
//
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//

options {
 listen-on port 53 { 127.0.0.1;192.168.122.1; };
 listen-on-v6 port 53 { ::1; };
 directory "/var/named";
 dump-file "/var/named/data/cache_dump.db";
 statistics-file "/var/named/data/named_stats.txt";
 memstatistics-file "/var/named/data/named_mem_stats.txt";
 allow-query { localhost;192.168.122.0/24;192.168.123.0/24; };

/* 
 - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
 - If you are building a RECURSIVE (caching) DNS server, you need to enable 
 recursion. 
 - If your recursive DNS server has a public IP address, you MUST enable access 
 control to limit queries to your legitimate users. Failing to do so will
 cause your server to become part of large scale DNS amplification 
 attacks. Implementing BCP38 within your network would greatly
 reduce such attack surface 
 */
 recursion yes;

dnssec-enable yes;
 dnssec-validation yes;
 dnssec-lookaside auto;

/* Path to ISC DLV key */
 bindkeys-file "/etc/named.iscdlv.key";

managed-keys-directory "/var/named/dynamic";

pid-file "/run/named/named.pid";
 session-keyfile "/run/named/session.key";

//forward first;
 forwarders {
 //10.38.5.26;
 8.8.8.8;
 };
};

logging {
 channel default_debug {
 file "data/named.run";
 severity dynamic;
 };
};

zone "." IN {
 type hint;
 file "named.ca";
};

zone "lab.com" IN {
 type master;
 file "/var/named/dynamic/lab.com.zone";
 allow-update { none; };
};

//zone "122.168.192.in-addr.arpa" IN {
// type master;
// file "/var/named/dynamic/122.168.192.db";
// allow-update { none; };
//};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

Note: I have left out reverse DNS, PTR records. If you need this you can of course add zone file and set that up but it isn’t required for a lab configuration.

Install OpenShift.

Here we are enabling ovs-subnet SDN and setting authentication to use htpasswd. This is the most basic configuration as we are doing an all-in-one setup. For actual deployments you would want multi-master, dedicated nodes and seperate nodes for handling etcd.

##########################
### OSEv3 Server Types ###
##########################
[OSEv3:children]
masters
nodes
etcd

################################################
### Set variables common for all OSEv3 hosts ###
################################################
[OSEv3:vars]
ansible_ssh_user=root
os_sdn_network_plugin_name='redhat/openshift-ovs-subnet'
deployment_type=openshift-enterprise
openshift_master_default_subdomain=apps.lab.com
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
openshift_node_kubelet_args={'maximum-dead-containers': ['100'], 'maximum-dead-containers-per-container': ['2'], 'minimum-container-ttl-duration': ['10s'], 'max-pods': ['110'], 'image-gc-high-threshold': ['90'], 'image-gc-low-threshold': ['80']}
logrotate_scripts=[{"name": "syslog", "path": "/var/log/cron\n/var/log/maillog\n/var/log/messages\n/var/log/secure\n/var/log/spooler\n", "options": ["daily", "rotate 7", "compress", "sharedscripts", "missingok"], "scripts": {"postrotate": "/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true"}}]
openshift_docker_options="-log-driver json-file --log-opt max-size=1M --log-opt max-file=3"
openshift_node_iptables_sync_period=5s
openshift_master_pod_eviction_timeout=3m
osm_controller_args={'resource-quota-sync-period': ['10s']}
osm_api_server_args={'max-requests-inflight': ['400']}
openshift_use_dnsmasq=true

##############################
### host group for masters ###
##############################
[masters]
ose3-master.lab.com

###################################
### host group for etcd servers ###
###################################
[etcd]
ose3-master.lab.com

##################################################
### host group for nodes, includes region info ###
##################################################
[nodes]
ose3-master.lab.com openshift_schedulable=True

Run Ansible playbook to install and configure OpenShift.

# ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml

Configure OpenShift

Create local admin account and enable permissions.

[root@ose3-master ~]#oc login -u system:admin -n default
[root@ose3-master ~]#htpasswd -c /etc/origin/master/htpasswd admin
[root@ose3-master ~]#oadm policy add-cluster-role-to-user cluster-admin admin
[root@ose3-master ~]#oc login -u admin -n default

Configure OpenShift image registry. Image streams are stored in registry. When you build application, your application code will be added as a image stream. This enables S2I (Source to Image) and allows for fast build times.

[root@ose3-master ~]#oadm registry --service-account=registry \
--config=/etc/origin/master/admin.kubeconfig \
--images='registry.access.redhat.com/openshift3/ose-${component}:${version}'

Configure OpenShift router. The OpenShift router is basically an HA-Proxy that sends incoming service requests to node where pod is running.

[root@ose3-master ~]#oadm router router --replicas=1 \
    --credentials='/etc/origin/master/openshift-router.kubeconfig' \
    --service-account=router

CloudForms Integration

CloudForms is a cloud management platform. It integrates not only with OpenShift but also other Cloud platforms (OpenStack, Amazon, GCE, Azure) and traditional virtualization platforms (VMware, RHEV, Hyper-V). Since OpenShift is usually running on cloud or traditional virtualization platforms, CloudForms enables true end-to-end visibility. CloudForms provides not only performance metrics, events, smart state analysis of containers (scanning container contents) but also can provide chargeback for OpenShift projects. CloudForms is included in OpenShift subscription for purpose of managing OpenShift. To add OpenShift as provider in CloudForms follow the below steps.

The management-admin project is designed for scanning container images. A container is started in this context and the image to be scanned mounted. List tokens that are configured in management-admin project (this is created at install time).

[root@ose3-master ~]# oc get sa management-admin -o yaml
apiVersion: v1
imagePullSecrets:
- name: management-admin-dockercfg-ln1an
kind: ServiceAccount
metadata:
 creationTimestamp: 2016-07-24T11:36:58Z
 name: management-admin
 namespace: management-infra
 resourceVersion: "400"
 selfLink: /api/v1/namespaces/management-infra/serviceaccounts/management-admin
 uid: ee6a1426-5192-11e6-baff-001a4ae42e01
secrets:
- name: management-admin-token-wx17s
- name: management-admin-dockercfg-ln1an

Use describe to get token to enable CloudForms to accesss the management-admin project.

[root@ose3-master ~]# oc describe secret management-admin-token-wx17s
Name: management-admin-token-wx17s
Namespace: management-infra
Labels: <none>
Annotations: kubernetes.io/service-account.name=management-admin,kubernetes.io/service-account.uid=ee6a1426-5192-11e6-baff-001a4ae42e01

Type: kubernetes.io/service-account-token

Data
====
ca.crt: 1066 bytes
namespace: 16 bytes
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtYW5hZ2VtZW50LWluZnJhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1hbmFnZW1lbnQtYWRtaW4tdG9rZW4td3gxN3MiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibWFuYWdlbWVudC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImVlNmExNDI2LTUxOTItMTFlNi1iYWZmLTAwMWE0YWU0MmUwMSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptYW5hZ2VtZW50LWluZnJhOm1hbmFnZW1lbnQtYWRtaW4ifQ.Y0IlcwhHW_CpKyFvk_ap-JMAT69fbIqCjkAbmpgZEUJ587LP0pQz06OpBW05XNJ3cJg5HeckF0IjCJBDbMS3P1W7KAnLrL9uKlVsZ7qZ8-M2yvckdIxzmEy48lG0GkjtUVMeAOJozpDieFClc-ZJbMrYxocjasevVNQHAUpSwOIATzcuV3bIjcLNwD82-42F7ykMn-A-TaeCXbliFApt6q-R0hURXCZ0dkWC-za2qZ3tVXaykWmoIFBVs6wgY2budZZLhT4K9b4lbiWC5udQ6ga2ATZO1ioRg-bVZXcTin5kf__a5u6c775-8n6DeLPcfUqnLucaYr2Ov7RistJRvg

Add OpenShift provider to CloudForms using the management-admin service token.

CF_CONTAINER_2

Performance Metrics

OpenShift provides ability to collect performance metrics using Hawkular. This runs as container and uses cassandra to persist the data. CloudForms is able to display capacity and utilization metrics for OpenShift using Hawkular.

Switch to openshift-infra project. This is intended for running infrastructure containers such as hawkular or ELK for logging.

[root@ose3-master ~]# oc project openshift-infra

Create service account for metrics-deployer pod.

[root@ose3-master ~]# oc create -f - <<API
apiVersion: v1
kind: ServiceAccount
metadata:
 name: metrics-deployer
secrets:
- name: metrics-deployer
API

Enable permissions and set secret.

[root@ose3-master ~]# oadm policy add-role-to-user edit system:serviceaccount:openshift-infra:metrics-deployer
[root@ose3-master ~]#oadm policy add-cluster-role-to-user cluster-reader system:serviceaccount:openshift-infra:heapster
[root@ose3-master ~]# oc secrets new metrics-deployer nothing=/dev/null

Deploy metrics environment for OpenShift.

[root@ose3-master ~]# oc new-app -f /usr/share/openshift/examples/infrastructure-templates/enterprise/metrics-deployer.yaml \
-p HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.apps.lab.com \
-p USE_PERSISTENT_STORAGE=false -p MASTER_URL=https://ose3-master.lab.com:8443

CloudForms Container Provider

CloudForms is a cloud management platform. It integrates not only with OpenShift but also other Cloud platforms (OpenStack, Amazon, GCE, Azure) and traditional virtualization platforms (VMware, RHEV, Hyper-V). Since OpenShift is usually running on cloud or traditional virtualization platforms, CloudForms enables true end-to-end visibility. CloudForms provides not only performance metrics, events, smart state analysis of containers (scanning container contents) but also can provide chargeback for OpenShift projects. CloudForms is included in OpenShift subscription for purpose of managing OpenShift. To add OpenShift as provider in CloudForms follow the below steps.

Use management-admin token that is created in management-admin project during install to provide access to CloudForms.

[root@ose3-master ~]# oc describe sa -n management-infra management-admin
Name: management-admin
Namespace: management-infra
Labels: <none>

Mountable secrets: management-admin-token-vr21i
 management-admin-dockercfg-5j3m3

Tokens: management-admin-token-mxy4m
 management-admin-token-vr21i

Image pull secrets: management-admin-dockercfg-5j3m3
[root@ose3-master ~]# oc describe secret -n management-infra management-admin-token-mxy4m
Name: management-admin-token-mxy4m
Namespace: management-infra
Labels: <none>
Annotations: kubernetes.io/service-account.name=management-admin,kubernetes.io/service-account.uid=87f8f4e4-4c0f-11e6-8aca-52540057bf27

Type: kubernetes.io/service-account-token

Data
====
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtYW5hZ2VtZW50LWluZnJhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1hbmFnZW1lbnQtYWRtaW4tdG9rZW4tbXh5NG0iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibWFuYWdlbWVudC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6Ijg3ZjhmNGU0LTRjMGYtMTFlNi04YWNhLTUyNTQwMDU3YmYyNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptYW5hZ2VtZW50LWluZnJhOm1hbmFnZW1lbnQtYWRtaW4ifQ.dN-CmGdSR2TRh1h0qHvwkqnW6TLvhXJtuHX6qY2jsrZIZCg2LcyuQI9edjBhl5tDE6PfOrpmh9-1NKAA6xbbYVJlRz52gnEdtm1PVgvzh8_WnKiQLZu-xC1qRX_YL7ohbglFSf8b5zgf4lBdJbgM_2P4sm1Czhu8lr5A4ix95y40zEl3P2R_aXnns62hrRF9XpmweASGMjooKOHB_5HUcZ8QhvdgsveD4j9de-ZzYrUDHi0NqOEtenBThe5kbEpiWzSWMAkIeC2wDPEnaMTyOM2bEfY04bwz5IVS_IAnrEF7PogejgsrAQRtYss5yKSZfwNTyraAXSobgVa-e4NsWg
ca.crt: 1066 bytes
namespace: 16 bytes

Add OpenShift provider to CloudForms using token.

OSE3_login_cf

Configure metrics by supplying the service name exposed by OpenShift.

OSE3_hawkular_cf

Choose a container image to scan.

OSEV3_SMartState

You should see scanning container start in the project management-infra.

[root@ose3-master ~]# oc project management-infra
[root@ose3-master ~]# oc get pods
NAME READY STATUS RESTARTS AGE
manageiq-img-scan-24297 0/1 ContainerCreating 0 12s
[root@ose3-master ~]# oc get pods
NAME READY STATUS RESTARTS AGE
manageiq-img-scan-24297 1/1 Running 0 1m

Check image in CloudForms and you should now see an OpenSCAP report and in addition visibility into packages that are actually installed in the container itself.

Compute->Containers-Container Images->MySQL

OSEv3_SMart_State_REsults

Packages

OSEV3_Packages

OpenScap HTML Report

OSEV3_OpenScap

Aggregate Logging

OpenShift Enterprise supports Kibana and the ELK Stack for log aggregation. Any pod and container that log to STDOUT will have all their log messages aggregated. This provides centralized logging for all application components. Logging is completely integrated within OpenShift and the ELK Stack runs of course containerized within OpenShift.

In openshift-infra project create service account for logging and necessary permissions.

[root@ose3-master ~]# oc project openshift-infra
[root@ose3-master ~]# oc secrets new logging-deployer nothing=/dev/null
[root@ose3-master ~]# oc create -f - <<API
apiVersion: v1
kind: ServiceAccount
metadata:
  name: logging-deployer
secrets:
- name: logging-deployer
API

[root@ose3-master ~]# oc policy add-role-to-user edit --serviceaccount logging-deployer
[root@ose3-master ~]# oadm policy add-scc-to-user privileged system:serviceaccount:logging:aggregated-logging-fluentd
[root@ose3-master ~]# oadm policy add-cluster-role-to-user cluster-reade system:serviceaccount:logging:aggregated-logging-fluentd

Deploy the logging stack. This creates a template or blueprint.

[root@ose3-master ~]# oc new-app logging-deployer-template \
             --param KIBANA_HOSTNAME=kibana.apps.lab.com \
             --param ES_CLUSTER_SIZE=1 \
             --param KIBANA_OPS_HOSTNAME=kibana-ops.apps.lab.com \
             --param PUBLIC_MASTER_URL=https://ose3-master.lab.com:8443
[root@ose3-master ~]# oc get pods
NAME READY STATUS RESTARTS AGE
logging-deployer-1de06 0/1 Completed 0 3m

Once logger deployer is complete, execute the template.

[root@ose3-master ~]# oc new-app logging-support-template

If you don’ see containers creating then you need to manually import image streams.

[root@ose3-master ~]#oc import-image logging-auth-proxy:3.2.0 --from registry.access.redhat.com/openshift3/logging-auth-proxy:3.2.0
[root@ose3-master ~]# oc import-image logging-kibana:3.2.0 --from registry.access.redhat.com/openshift3/logging-kibana:3.2.0
[root@ose3-master ~]# oc import-image logging-elasticsearch:3.2.0 --from registry.access.redhat.com/openshift3/logging-elasticsearch:3.2.0
[root@ose3-master ~]# oc import-image logging-fluentd:3.2.0 --from registry.access.redhat.com/openshift3/logging-fluentd:3.2.0
[root@ose3-master ~]#  oc get pods
NAME READY STATUS RESTARTS AGE
logging-deployer-9lqkt 0/1 Completed 0 15m
logging-es-pm7uamdy-2-rdflo 1/1 Running 0 8m
logging-kibana-1-e13r3 2/2 Running 0 13m

Once ELK Stack is running update deployment so that persistent storage is used (optional). Note: this requires configuring persistent storage and that is explained in a different blog post referenced above.

#vi pvc.json

{ 
    "apiVersion": "v1",
    "kind": "PersistentVolumeClaim",
    "metadata": {
         "name": "logging-es-1"
    },
    "spec": {
        "accessModes": [ "ReadWriteOnce" ],
            "resources": {
                "requests": {
                    "storage": "10Gi"
                }
            }
     }
}
[root@ose3-master ~]# oc create -f pvc.json
[root@ose3-master ~]# oc get dc
NAME TRIGGERS LATEST
logging-es-pm7uamdy ConfigChange, ImageChange 2
[root@ose3-master ~]# oc volume dc/logging-es-pm7uamdy --add --overwrite --name=elasticsearch-storage --type=persistentVolumeClaim --claim-name=logging-es-1

OpenShift on RHEV or VMware.

If you are running OpenShift on traditional virtualization platform ensure mac spoofing is enabled or allowed. If it isn’t the hypervisor will most likely drop packets going outbound from OpenShift. Enabling mac spoofing for RHEV is documented here under configure RHEV for OpenStack (same issue exists when running OpenStack nested in RHEV). For VMware or Hyper-V, I am not sure so just keep this in mind.

Summary

In this article we have seen how to configure an OpenShift 3.2 all-in-one lab environment. We have also seen how install and configuration can be adapted through ansible playbook. We have seen how to configure various DNS options. It should be repeated that most OpenShift problems are a direct result of improper DNS setup! We have seen how to integrate OpenShift in CloudForms and how to configure metrics using hawkular. Finally we even covered configuring log aggregation using containerized ELK stack. If you have any feedback please share.

Happy OpenShifting!

(c) 2016 Keith Tenzer

 

 

 


Adding hosts to DNS via Ansible

Standard

Overview

I’m running a dynamic Nameservice for my datacenter authorative for example.com. Now i want to add address records and reverse pointer records to dns to be able to resolv names.

Solution

To achieve this i added variables to my inventory:

cat /etc/ansible/hosts 

[glusterleft]
gluster11 fqdn=gluster11.example.com. ipaddress=172.16.20.103 reverse=103.20.16.172.in-addr.arpa.
gluster12 fqdn=gluster12.example.com. ipaddress=172.16.20.104 reverse=104.20.16.172.in-addr.arpa.

[glusterright]
gluster21 fqdn=gluster21.example.com. ipaddress=172.16.20.203 reverse=203.20.16.172.in-addr.arpa.
gluster22 fqdn=gluster22.example.com. ipaddress=172.16.20.204 reverse=204.20.16.172.in-addr.arpa.

[rhevleft]
rhev11 fqdn=rhev11.example.com. ipaddress=172.16.20.101 reverse=101.20.16.172.in-addr.arpa.
rhev12 fqdn=rhev12.example.com. ipaddress=172.16.20.102 reverse=102.20.16.172.in-addr.arpa.

[rhevright]
rhev21 fqdn=rhev21.example.com. ipaddress=172.16.20.201 reverse=201.20.16.172.in-addr.arpa.
rhev22 fqdn=rhev22.example.com. ipaddress=172.16.20.202 reverse=202.20.16.172.in-addr.arpa.

Note: It would definitely be possible to get the reverse name created out of the ip automatically. – I leave this as an shell exercise for you.

And wrote the following playbook:

[root@jump ansible]# cat named_addhosts.yml
---
- hosts: servers
 gather_facts: False
 serial: 1

tasks:
 - name: check dns
 local_action: shell host {{ fqdn }}
 register: dnsout
 ignore_errors: yes

- name: add dnsentry
 local_action: script /etc/ansible/named_update.sh {{ fqdn }} {{ ipaddress }} {{ reverse }} 
 when: dnsout.stdout.find('{{ ipaddress }}') == -1
 run_once: true
which is supported by a small helper shell script:

[root@jump ansible]# cat /etc/ansible/named_update.sh
#! /usr/bin/bash
# small script which updates dns van nsupdate
# needs 3 parameters
# hostname - full qualified (with a . at the end)
# ipaddress
# reverse - reverse ipaddress full qualified (with in-addr.arpa. ) 
if [ $# -ne 3 ]
then
 echo "usage: $0 hostname ipaddress reverse" >&2
 echo " with:" >&2
 echo " hostname - full qualified (with a . at the end)" >&2
 echo " ipaddress" >&2
 echo " reverse - reverse ipaddress full qualified (with in-addr.arpa. ) " >&2
 exit 1
fi
echo $1
echo $2
echo $3
nsupdate -k /etc/rndc.key << EOF
update add $1 3600 A $2
send
update add $3 3600 PTR $1
send
EOF
 

It is also necessary to grant access to modify the DNS server:. This access is granted through /etc/rndc.key which i put in place before.

Conclusion

I’m now easily able to add Ansible managed hosts into a dns service.