13.8.14

ubuntu samba4 cluster: CTDB ocfs2 DRBD

Mr. Shuttleworth and the packagers at Ubuntu do a great job, but clustering is not one of them. Their samba packages are not built with cluster support and their ctdb is broken. To make matters even worse, the latest samba4 comes without winbind AD support. If that hasn't put you off yet, you then find out that the ctdb source doesn't use standard locations for its directories. Open Source at its very best.

conclusion
On Ubuntu, clustering means build it yourself. It can be done and when it's up, it's bulletproof. So, here goes. Breathe deeply and. . .

reference
This guide is an ubuntuified version of our openSUSE cluster post. If you think it's missing here, it'll be there instead.

hardware
Each node needs a minimum of 2 physical network interface cards, one for the cluster traffic between the nodes and the other that goes out to the domain. Here, we use 2 real cards and a virtual bond for the third.

for domain: altea.site
DC: palmera, 192.168.1.132
DNS: palmera, bind9_dlz master for altea.site forwarding 192.168.1.1
node 1: uc1 192.168.1.82, 192.168.1.80, 192.168.1.81 and 192.168.0.10
node 2: uc2 192.168.1.83, 192.168.1.80, 192.168.1.81 and 192.168.0.11
gateway: 192.168.1.1

eth0: domain traffic. nic is disabled
eth1: cluster traffic, no route out
bond0: domain and local traffic bound to eth0

cluster disk
You will need a second disk drive for your cluster data. We have a spare disk installed on each of the nodes which is recognised by fdisk as /dev/sdb. We have partitioned it as /dev/sdb1. We have used sdb1 as the shared storage. Under DRBD the cluster device becomes /dev/drbd1. /dev/drbd1 is formatted with ocfs2. We have drbd primary:primary with both nodes up and mounted.

DNS
Get this right and the rest is easy.
/etc/hosts
127.0.0.1 localhost
192.168.0.10 uc1
192.168.0.11 uc2

If you know the ifup syntax for bonding and you want to mount the cluster partition on boot:
sudo apt-get install ifenslave-2.6
 /etc/network/interfaces
# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual
bond-master bond0

auto eth1
iface eth1 inet static
address 192.168.0.10
netmask 255.255.255.0

auto bond0
iface bond0 inet static
address 192.168.1.82
gateway 192.168.1.1
netmask 255.255.255.0
dns-nameserver 192.168.1.132
bond-slaves none

If not, use the pretty pictures to set up DNS as follows. Here are the screenshots for setting up networkmanager for uc1 on node1. Repeat for uc2 but using the second set of IPs from above. But be warned that you will have to use ifup if you want to mount the o2cb startup stuff.

install the cluster and kerberos stuff
sudo apt-get install drbd8-utils linux-server ocfs2-tools build-essential libattr1-dev krb5-user libblkid-dev gdb libgnutls-dev libreadline-gplv2-dev python-dev autoconf python-dnspython pkg-config pam-krb5utils libpopt-dev apparmor-utils  ldap-utils  libsasl2-modules-gssapi-mit

download and unpack ctdb-2.5.3.tar.gz
then it's just: 
./configure && make
and install it:
sudo make install

configure the cluster
 /usr/local/etc/ctdb/nodes
192.168.0.10
192.168.0.11
 /usr/local/etc/ctdb/public_addresses
192.168.1.80/24 bond0
192.168.1.81/24 bond0

 /etc/drbd.conf
global {
  usage-count yes;
}
common {
  net {
    protocol C;
  }
}
resource r0 {
 net { 
                allow-two-primaries; 
                after-sb-0pri discard-zero-changes;
                after-sb-1pri discard-secondary;
                after-sb-2pri disconnect;
        }
startup { become-primary-on both; }
  on uc1 {
    device    /dev/drbd1;
    disk      /dev/sdb1;
    address   192.168.0.10:7789;
    meta-disk internal;
  }
  on uc2 {
    device    /dev/drbd1;
    disk      /dev/sdb1;
    address   192.168.0.11:7789;
    meta-disk internal;
  }
}

/etc/ocfs2/cluster.conf
node:
        ip_port = 7777
        ip_address = 192.168.0.10
        number = 1
        name = uc1
        cluster = ocfs2
node:
        ip_port = 7777
        ip_address = 192.168.0.11
        number = 2
        name = uc2
        cluster = ocfs2
cluster:
        node_count = 2
        name = ocfs2

configure the fs
sudo dpkg-reconfigure ocfs2-tools
Setting cluster stack "o2cb": OK
Starting O2CB cluster ocfs2: OK

partition the spare disk
use fdisk to end up with /dev/sdb1 where:
fdisk -l
gives something like:
Disk /dev/sda: 12.9 GB, 12884901888 bytes, 25165824 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000f1bbc

   Device Boot      Start         End      Blocks   Id  System

/dev/sda1            2048     1525759      761856   82  Linux swap / Solaris
/dev/sda2   *     1525760    25165823    11820032   83  Linux

Disk /dev/sdb: 2147 MB, 2147483648 bytes, 4194304 sectors

Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000ab2e5

   Device Boot      Start         End      Blocks   Id  System

/dev/sdb1            2048     4194303     2096128   83  Linux

create the drbd metadata on both nodes
sudo drbdadm create-md r0
sudo drbdadm up r0

sync the nodes
Be careful. If you have data you wish to keep on either node then choose that node to begin the sync.
sudo drbdadm primary --force r0
wait for
cat /proc/drbd
to stabilise, then make it active
sudo drbdadm primary r0

format the partition
Unless you just want fail-over, you must use something that doesn't crash if mounted simultaneously.
mkfs -t ocfs2 -N 2 -L ubuntucluster /dev/drbd1

mount it
sudo mkdir /cluster 
sudo mount /dev/drbd1 /cluster
cd /cluster
sudo mkdir users && sudo mkdir profiles && sudo chmod 1777 profiles

samba
download and unpack the latest samba (ATM 4.1.11.tar.gz)
 ./configure --with-cluster-support  --without-ad-dc  --with-shared-modules=idmap_ad --with-ads && make
install it:
sudo make install

/usr/local/samba/etc/samba/smb.conf
[global]
workgroup = ALTEA
netbios name = SMBCLUSTER
realm = ALTEA.SITE
security = ADS
kerberos method = secrets only
winbind enum users = Yes
winbind enum groups = Yes
winbind use default domain = Yes
winbind nss info = rfc2307
idmap config * : backend = tdb
idmap config * : range = 19900-19999
idmap config ALTEA : backend  = ad
idmap config ALTEA : range = 20000-4000000
idmap config ALTEA : schema_mode = rfc2307
clustering = Yes
ctdbd socket = /usr/local/var/run/ctdb/ctdb.socket
[users]
path = /cluster/users
read only = No
[profiles]
path = /cluster/profiles
read only = No

winbind
create the stack at /etc/pam.d:
auth sufficient pam_winbind.so
auth [success=2 default=ignore] pam_unix.so nullok_secure
auth requisite pam_deny.so
auth required pam_permit.so
auth optional pam_cap.so 
account sufficient      pam_winbind.so
account [success=1 new_authtok_reqd=done default=ignore] pam_unix.so 
account requisite pam_deny.so
account required pam_permit.so
account sufficient pam_localuser.so

create the winbind links:
sudo ln -s /usr/local/samba/lib/libnss_winbind.so.2 /lib/libnss_winbind.so
sudo ln -s /lib/libnss_winbind.so /lib/libnss_winbind.so.2
sudo cp /usr/local/samba/lib/security/pam_winbind.so /lib/security

start the cluster
sudo ctdbd --dbdir=/usr/local/var/lib/ctdb --dbdir-persistent=/usr/local/var/lib/ctdb/persistent --event-script-dir=/usr/local/etc/ctdb/events.d --public-addresses=/usr/local/etc/ctdb/public_addresses --socket=/usr/local/var/run/ctdb/ctdbd.socket --logfile=/var/log/syslog 

(tail the logs and wait for them to stabilise)

 sudo ctdb status
Number of nodes:2
pnn:0 192.168.0.10     OK (THIS NODE)
pnn:1 192.168.0.11     DISCONNECTED|UNHEALTHY|INACTIVE
Generation:2138318810
Size:1
hash:0 lmaster:0
Recovery mode:NORMAL (0)
Recovery master:0

create a domain user
uidNumber: 3000016
loginShell: /bin/bash
homeDrive: Z:
homeDirectory: \\smbcluster\users\stevep
profilePath: \\smbcluster\profiles\stevep
unixHomeDirectory: /cluster/users/stevep

'posixify' the group Domain Users
gidNumber: 20513

start smbd
sudo smbd

join the domain
on one node only:
net ads join -UAdministrator

start winbind
sudo winbindd

check
id stevep
uid=3000016(stevep) gid=20513(domain users) groups=20513(domain users),19903(BUILTIN\users)

add the round robin A records for the cluster to the DC
 host smbcluster
smbcluster.altea.site has address 192.168.1.81
smbcluster.altea.site has address 192.168.1.80

 ping smbcluster
PING smbcluster.altea.site (192.168.1.80) 56(84) bytes of data.
64 bytes from smbcluster (192.168.1.80): icmp_seq=1 ttl=64 time=0.863 ms

finally
sudo chgrp Domain\ Users /cluster/users && sudo chgrp Domain\ Users /cluster/profiles

Linux client:
sudo mount.cifs //smbcluster/users /cluster/users -osec=krb5,username=cifsuser,multiuser
 su stevep
Password: 
stevep@uc1:/home/steve$ cd ~
stevep@uc1:~$ touch cluster.txt
stevep@uc1:~$ ls -l
total 0
-rw-r--r-- 1 stevep domain users 0 ago 13 10:51 cluster.txt
stevep@uc1:~$ exit

bring up the other node
sudo ctdbd --dbdir=/usr/local/var/lib/ctdb --dbdir-persistent=/usr/local/var/lib/ctdb/persistent --event-script-dir=/usr/local/etc/ctdb/events.d --public-addresses=/usr/local/etc/ctdb/public_addresses --socket=/usr/local/var/run/ctdb/ctdbd.socket --logfile=/var/log/syslog 

sudo drbdadm primary r0

mount /dev/drbd1 /cluster

sudo ctdb enable

sudo ctdb status
Number of nodes:2
pnn:0 192.168.0.10     OK (THIS NODE)
pnn:1 192.168.0.11     OK
Generation:569123968
Size:2
hash:0 lmaster:0
hash:1 lmaster:1
Recovery mode:NORMAL (0)
Recovery master:0

disable node 1
sudo ctdb disable && sudo ctdb status
Number of nodes:2
pnn:0 192.168.0.10     DISABLED (THIS NODE)
pnn:1 192.168.0.11     OK
Generation:569123968
Size:2
hash:0 lmaster:0
hash:1 lmaster:1
Recovery mode:NORMAL (0)
Recovery master:0

(tail the logs and wait for the takeover to finish)

kill smbd on node 1
 killall smbd
 ps aux|grep smbd
steve    18577  0.0  0.1   7124   840 pts/5    S+   11:07   0:00 grep --color=auto smbd
(wait for the log activity to stabilise)

Back to the Linux workstation, check the fail-over:
stevep@uc1:~$ touch cluster2.txt
stevep@uc1:~$ ls -l
total 0
-rw-r--r-- 1 stevep domain users 0 ago 13 11:09 cluster2.txt
-rw-r--r-- 1 stevep domain users 0 ago 13 10:51 cluster.txt
You now admire your work as you fail-over and fail-back time and time again convincing yourself that it does indeed work. At first, you just don't believe it.

More graphical examples below with both Linux and XP clients working on the same data.

Choose your poison: Ubuntu or XP:
all together now, clusters give you. . .
peace of mind. . .