Ceph 1.3 Lab Installation and Configuration Guide



In this article we will setup a Ceph 1.3 cluster for purpose of learning or a lab environment.


Ceph Lab Environment

For this environment you will need three VMs (ceph1, ceph2 and ceph3). Each should have 20GB root disk and 100GB data disk. Ceph has three main components: Admin console, Monitors and OSDs.

Admin console – UI and CLI used for managing Ceph cluster. In this environment we will install on ceph1.

Monitors – Monitor health of Ceph cluster. One or more monitors forms a paxos part-time parliment, providing extreme reliability and durability of cluster membership. Monitors maintain the various maps: monitor, osd, placement group (pg) and crush. Monitors will be installed on ceph1, ceph2 and ceph3.

OSDs – Object storage daemon handles storing data, recovery, backfilling, rebalancing and replication. OSDs sit on top of a disk / filesystem. Bluestore enables OSDs to bypass filesystem but is not an option in Ceph 1.3. An OSD will be installed on ceph1, ceph2 and ceph3.

On All Cephs nodes.

#subscription-manager repos --disable=*
#subscription-manager repos --enable=rhel-7-server-rpms
#subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-rhceph-1.3-calamari-rpms --enable=rhel-7-server-rhceph-1.3-installer-rpms --enable=rhel-7-server-rhceph-1.3-tools-rpms

Configure firewalld.

sudo systemctl start firewalld
sudo systemctl enable firewalld
sudo firewall-cmd --zone=public --add-port=80/tcp --permanent
sudo firewall-cmd --zone=public --add-port=2003/tcp --permanent
sudo firewall-cmd --zone=public --add-port=4505-4506/tcp --permanent
sudo firewall-cmd --zone=public --add-port=6789/tcp --permanent
sudo firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent
sudo firewall-cmd --reload

Configure NTP.

yum -y install ntp
systemctl enable ntpd.service
systemctl start ntpd

Ensure NTP is scychronozing.

ntpq -p
remote refid st t when poll reach delay offset jitter
 +privatewolke.co 2 u 12 64 1 26.380 -2.334 2.374
 +ridcully.episod 3 u 11 64 1 26.626 -2.425 0.534
 *s1.kelker.info 2 u 12 64 1 26.433 -6.116 1.030
 sircabirus.von- .STEP. 16 u - 64 0 0.000 0.000 0.000

Create ceph user for deployer.

#useradd ceph
#passwd ceph
#cat << EOF >/etc/sudoers.d/ceph
ceph ALL = (root) NOPASSWD:ALL
Defaults:ceph !requiretty
#chmod 0440 /etc/sudoers.d/ceph
#su - ceph
#ssh-copy-id ceph@ceph1
#ssh-copy-id ceph@ceph2
#ssh-copy-id ceph@ceph3

Set SELinux to permissive. Ceph 2.0 now supports SELinux but for 1.3 it was not possible out-of-box.

#vi /etc/selinux/config


Create ceph-config dir.

mkdir ~/ceph-config
cd ~/ceph-config

On Monitors.

#subscription-manager repos --enable=rhel-7-server-rhceph-1.3-mon-rpms
#yum update -y

On OSD Nodes.

#subscription-manager repos --enable=rhel-7-server-rhceph-1.3-osd-rpms
#yum update -y

On admin node (ceph1).

[ceph@ceph1 ~]$ vi .ssh/config

Host node1
  Hostname ceph1
  User ceph
Host node2
  Hostname ceph2
  User ceph
Host node3
  Hostname ceph3
  User ceph
chmod 600 ~/.ssh/config

On admin node (ceph1).

Setup Admin Console and Calamari.

#sudo yum -y install ceph-deploy calamari-server calamari-clients
#sudo calamari-ctl initialize
#su - ceph
[ceph@ceph1 ceph-config]$cd ~/ceph-config

Create Ceph Cluster.

#ceph-deploy new ceph1 ceph2 ceph3

Deploy Ceph monitors and OSDs.

[ceph@ceph1 ceph-config]$ceph-deploy install --mon ceph1 ceph2 ceph3
[ceph@ceph1 ceph-config]$ceph-deploy install --osd ceph1 ceph2 ceph3
[ceph@ceph1 ceph-config]$ceph-deploy --overwrite-conf mon create-initial

Connect Ceph monitors to Calamari.

[ceph@ceph1 ceph-config]$ceph-deploy calamari connect --master ceph1.lab ceph1 ceph2 ceph3
[ceph@ceph1 ceph-config]$ceph-deploy install --cli ceph1
[ceph@ceph1 ceph-config]$ceph-deploy admin ceph1

Check Ceph quorum status.

[ceph@ceph1 ceph-config]$ sudo ceph quorum_status --format json-pretty

 "election_epoch": 6,
 "quorum": [
 "quorum_names": [
 "quorum_leader_name": "ceph1",
 "monmap": {
 "epoch": 1,
 "fsid": "188aff9b-7da5-46f3-8eb8-465e014a472e",
 "modified": "0.000000",
 "created": "0.000000",
 "mons": [
 "rank": 0,
 "name": "ceph1",
 "addr": "\/0"
 "rank": 1,
 "name": "ceph2",
 "addr": "\/0"
 "rank": 2,
 "name": "ceph3",
 "addr": "\/0"

Set crush tables to optimal.

[ceph@ceph1 ceph-config]$sudo ceph osd crush tunables optimal

Configure OSDs.

[ceph@ceph1 ceph-config]$ceph-deploy disk zap ceph1:vdb ceph2:vdb ceph3:vdb
[ceph@ceph1 ceph-config]$ceph-deploy osd prepare ceph1:vdb ceph2:vdb ceph3:vdb
[ceph@ceph1 ceph-config]$ceph-deploy osd activate ceph1:vdb1 ceph2:vdb1 ceph3:vdb1

Connect Calamari to Ceph nodes.

[ceph@ceph1 ceph-config]$ceph-deploy calamari connect --master ceph1.lab ceph1 ceph2 ceph3

Tips and Tricks

Remove OSD from Ceph

[ceph@ceph1 ~]$sudo ceph osd out osd.0
[ceph@ceph1 ~]$sudo ceph osd crush remove osd.0
[ceph@ceph1 ~]$sudo ceph auth del osd.0
[ceph@ceph1 ~]$sudo ceph osd down 0
[ceph@ceph1 ~]$sudo ceph osd rm 0

Ceph Placement Group Calculation for Pool

  • OSDs * 100 / Replicas
  • PGs should always be power of two 62, 128, 256, etc

Re-deploy Ceph

In case at anytime you want to start over you can run below commands to uninstall Ceph. This of course deletes any data so be careful.

[ceph@ceph1 ~]$ sudo service ceph restart osd.3
ceph-deploy purge <ceph-node> [<ceph-node>]


In this article we installed a Ceph cluster on virtual machines. We deployed the cluster, setup monitors and configured OSDs.  This environment should provide the basis for a journey into software-defined storage and Ceph. The economics of scale have brought down barriers and paved the way for a software-defined world. Storage is only the next logical boundry. Ceph being an OpenSource project is already the defacto software-defined standard and is in position to become the key beneficiary of software-defined storage. I hope you found the information in this article of use, please share your experiences.

Happy Cephing!

(c) 2016 Keith Tenzer




Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: