1. Whatever happened to cman?
Christine “Chrissie” Caulfield, Red Hat
ccaulfie@redhat.com
Version history
0.1 30th January 2009 First draft
0.2 3rd February 2009 Add a chapter on migrating from libcman
0.2 27th February 2009 Add some commas and fix some typos

2. 1. Introduction
CMAN was the mainstay of Red Hat's cluster suite in Red Hat Enterprise Linux 4, 5 and 6. Actually
CMAN in RHEL4 was a totally different thing to that found in RHEL5 and 6 but the external interfaces
looked very similar.
So what was CMAN?
CMAN stood for “Cluster MANager” (boring eh?) and provided the low-level infrastructure for the
cluster; node communications, votes, quorum etc. Most of the useful bits of the cluster stack were built
on top of cman so few people got to see it unless things went horribly wrong.
Starting with RHEL5 we dispensed with the custom-written kernel CMAN and based the cluster
manager on OpenAIS. For more details about this see my document on the subject:
http://people.redhat.com/ccaulfie/docs/aiscman.pdf.
OK, a short summary then. CMAN in RHEL 5/6 is really just a quorum module that loads into openais
(these days known as corosync). It provides other API services such as messaging but those are really
just wrappers around other openais services.
At the cluster summit in 2008 it was decided that keeping companies' own clustering modules was a
waste of effort and almost always fails to build a community. Even though CMAN (all versions of it)
were released as free software, they lived in their own repositories and were seldom modified by the
original author (ie me!). So efforts were made to make the software even more open and included in the
components of their parent projects. Also part of the collaboration efforts were the pooling of resource
and fence agents and several others. But this is just about CMAN OK? Come on now, concentrate.
2. corosync/openais – what happened there?
In Red Hat Enterprise Linux 5 cman was a module for openais. In RHEL6 (we're not actually supposed
to call it RHEL, but “Red Hat Enterprise Linux” is a lot of typing. And even if I do a search/replace
afterwards it'll be unwieldy to read ... sorry marketing, but this is an engineering paper, not a sales one)
it's a module for corosync. So, what happened there?
Just in case you weren't already confused enough, openais has now been split into two parts. The parts
still called openais are the modules for corosync that implement the SAF AIS services and APIs – you
know: the ones with the VeryAnnoyinglyCapitalisedNames. Corosync is the communications layer
below that, but it also includes other services such as cpg (Closed Process Groups ... very popular
amongst service writers than one) and, now, quorum. Most people won't really notice this split, apart
from, possibly, the name of the process in 'ps' output.
Just to put a little bit of detail on this, the services that are included with corosync are:
cfg cpg evs quorum

3. Services included in openais are:
Amf Ckpt Clm Evt Lck Msg (see what I mean about the capitalisation ... and that's just the names!)
corosync is the name of the daemon, replacing aisexec.
I hope that's all clear because it gets more complicated from here on in.
3. Intermediate status – RHEL6
Before getting onto the really new code, I'll talk a little about the RHEL6 code which is sort of a half-
way house between RHEL5 and the code that will be in RHEL7. I suppose that makes some sort of
sense, numerically if nothing else.
The first thing you'll notice about RHEL6 is that ccsd is not there any more. Actually I very much
doubt it'll be really the very first thing you notice unless you're incredibly sad or a real clustering nerd,
but I do hope you might have spotted it at some stage. I'll talk (well, write – unless you want to invite
me, all expenses paid, to give a presentation) about that in the next chapter.
Apart from this, you will notice very little difference in the RHEL6 cman. We were under pressure to
allow rolling upgrades from RHEL5 (ie no wire protocol changes) and time constraints mean that the
code discussed below was not stable enough to be included for the deadline. The only other substantial
difference is that cman now is a quorum provider for corosync. I'll also write more about this this later
on (you're really agog now aren't you?), but basically it means that the rest of the corosync modules
will know about the quorum status and can act on it if they feel the urge, and that applications that are
only interested in the quorum status and node list can link straight into libquorum and bypass cman
altogether. This might help some people with the upgrade path ... or not ... we'll see.
4. Configuring corosync
This chapter is also relevant to RHEL6 by the way, just in case you were going to skip the rest of the
document and have an early night.
So, as I mentioned above, ccsd is now gone and I can hear the cheers already. The distribution of
cluster.conf files is now delegated to ricci and a new command ccs_sync. I will not focus too closely on
that bit any more but it brings me to the next main change in the way CMAN (as it still is at this stage)
is loaded. Actually you don't have to use ccs_sync to distribute cluster config files, conga, scp, nfs and
several other methods work perfectly well. Pick that one that suits you best; “choice is good” as the
free-marketeers love to say.
The way that corosync (not aisexec remember?) is launched is still via cman_tool as before. But it now
sets a few environment variables first. The most important of these tells corosync which configuration
modules to use, and there are up to three of those now.
The first, xmlconfig, simply loads the contents of /etc/cluster/cluster.conf into the corosync object
database, where it keeps all the configuration information. The second, cman-preconfig, interprets that
format into a format that is native to corosync by resolving nodenames, generating multicast addresses
etc.

4. 4.1. Those modules in full
Corosync config modules are different from the normal service modules that corosync uses. They are
specified in the environment variable COROSYNC_DEFAULT_CONFIG_IFACE before corosync is
started. This variable can contain a colon-separated list of modules that will be called before corosync
is started for real. They also manage the reloading of data when the underlying configuration file or
device changes. By default cman_tool starts corosync with
COROSYNC_DEFAULT_CONFIG_IFACE=xmlconfig:cmanpreconfig:openaisserviceenable
4.2. xmlconfig & friends
xmlconfig is simply an XML parser with a default filename of /etc/cluster/cluster.conf. It reads the tree
of objects it finds in that file and loads it, unmodified, into corosync's object database (objdb). There
are a few other config modules that can be used here. The is an ldap one (called, rather obviously I
hope, ldapconfig) and even a ccsd one too if you're hopelessly addicted to the past. These modules are
quite easy to write and if you have a different back-end database you want to use it's probably quite
simple to write a corosync config module for it.
Just in case you were wondering, objdb is simply an in-memory hierarchical database where corosync
holds configuration data and some state.
The point about xmlconfig and it's similarly-named friends (actually, maybe they're relatives ... hmm)
is that they do not try to interpret the data, they simply copy what is there into objdb. If it makes no
sense to corosync then it will tell you when it tries to start up properly shortly afterwards. You don't
need cman-preconfig (see below) to use these if the format of the data matches the hierarchy of
corosync.conf(5).
cman_tool join allows you to specify a different main config module than xmlconfig on the command-
line with cman_tool join -C <module>. It will always add cman-preconfig though. eg the command:
# cman_tool join -C ldapconfig
will load corosync with the environment variable:
COROSYNC_DEFAULT_CONFIG_IFACE=ldapconfig:cmanpreconfig:openaisserviceenable
4.3 cman-preconfig
Cman-preconfig is a rather strange beast in some ways. It's main purpose is to re-interpret the loaded
configuration file. Let me endeavour to explain.
CMAN since RHEL4 has a very specific XML-based format which has not changed significantly in the
meantime. It contains details of all the nodes in the cluster, along with their fence details and also
rgmanager services too. This format bears no resemblance to corosync.conf. If you load a normal
RHEL cluster.conf into corosync using xmlconfig alone, without passing it through cman-preconfig
then corosync will not start. It will not be able to find a single entry that it needs to run.
cman-preconfig re-interprets this configuration data into a form that corosync expects, and adds some
sensible, we hope, defaults for things not specified at all, eg the cluster multicast address. It also has a

5. mode where it can start up cman/corosync with no configuration data at all (see cman_tool -X) which is
very useful for time-poor or lazy people who just want to test the cluster a little, or perhaps for
commissioning virtualised clusters. There's a whole chapter on this later on.
4.4. openaisserviceenable
This module simply tells corosync to also load the openais services as well it its own default set.
Normally this is what you will want as many cluster daemons use openais services like checkpointing.
You can disable this by adding the -A switch to cman_tool join.
5. CMAN itself
Now we get to the nitty-gritty. The CMAN module exists in RHEL5 and RHEL6 and does all the
cman-y things that cman does such as quorum and messaging. It also services the libcman library using
an IPC mechanism all of its own. It does not use corosync's IPC (library to daemon communications)
system for reasons that are unlikely to become clear any more.
The new code, as I've said, dispenses with anything called cman at all. The main work of managing
quorum is done by a built-in corosync module called votequorum. This code is actually largely lifted
from the original cman module, with all the non-quorum bits removed, and converted to use the
corosync IPC layer.
This module also includes code for managing quorum devices (eg qdiskd) as well as the hated
“Disallowed” mode. Though both are inactive in a default corosync system.
At the time of writing libcman still exists. It provides the same API as the libcman in previous versions
though there might be small semantic differences. It's also not ABI compatible so the soname has been
incremented to 4. This libcman uses a combination of calls to quorum, confdb and a custom cman
module to provide these services. It is provided largely as a compatibility layer for older applications
though the source code to libcman can also provide ideas on how to do cman-y type things with the
existing corosync services. All the programs in Red Hat clustering will be changed to use the standard
APIs that are provided with corosync.
6. The CMAN module
Now hang on there, you just said “a custom cman module”. So cman does still exist?
Well, yes and no.
There is a module called “cmanbits” which provides services for some libcman APIs. These are:
cman_send_data,
cman_start_recv_data

6. cman_end_recv_data
cman_is_listening
Of these, the messaging API can be replaced with the normal openais messaging system. There is no
suitable replacement for is_listening.
Missing totally from this new cman is the barriers API which nobody used, to my knowledge. If you
did and are fuming with me for leaving it out ... sorry. But there are actually better ways of achieving
the same effect with existing corosync or openais calls.
Some other functions that were in libcman are now provided by the corosync cfg service. These are
corosync_cfg_kill_node
corosync_cfg_try_shutdown
corosync_cfg_replyto_shutdown
corosync_cfg_get_node_addrs
The are wrapped in the new libcman but are deprecated.
7. corosync quorum
7.1. quorum plugins
A brief word about corosync's quorum system might he helpful here. Corosync has a basic notion of
quorum built in to the executive. Services cannot be synchronised without it and several IPC calls will
be blocked if the cluster is inquorate.
By default corosync does not load the quorum module. This is for backwards compatibility so that
existing clusters will work. If the quorum module was loaded and no quorum provider was specified
then the cluster would never be quorate and never get any work done. Not a popular system I would
guess. If the quorum module is absent then everything just works all the time.
To load the quorum module, you need to add the following stanza to corosync.conf (or equivalent):
service {
name: corosync_quorum
ver:0
}
cman-preconfig does this for you by the way. Corosync will then load the quorum module and allow
users to use the basic quorum API as defined in /usr/include/corosync/quorum.h. This simply allows
programs to query the quorate state and to receive notifications when it changes. This notification also
includes a list of nodes that are part of the quorum. It's useful (I hope) to realise that this plugin is also
available in RHEL6 and that cman provides quorum for it. So if you only need to know the quorum
state (ie you don't need any of the other libcman gubbins) then calling into libquorum instead will
provide you with forward-compatibility.

7. However, as I mentioned above (or on the previous page if you were ungreen enough to print this
document out) this will not give you quorum, all it will do is to block a lot of useful activity in
corosync. To make it work you need a quorum provider.
A quorum provider is a corosync module that tells the corosync quorum plugin when the quorate state
changes, based in its own internal algorithm. There are many ways of calculating quorum so there are
several modules available, and I hope people will write more to suit their own needs. The quorum
provider is loaded by putting its name in corosync.conf as follows:
quorum {
provider: testquorum
}
This example, as I sincerely hoped you have guessed, loads the 'testquorum' module. Testquorum is an
extremely simple module which I recommend as a template to other quorum provider writers. It simply
changes the corosync quorate status based on the quorum.quorate variable in the object database. Using
corosync-objctl the quorate state can be switched on and off at will. If you have some form of external
quorum system this might actually be useful to you.
Another quorum provider shipped with corosync is YKD. This is the YKD system that has been
shipped with openais for some time, it has merely been re-designated a quorum plugin now. Sadly the
developers did not have time to give it a shiny new logo too so it looks pretty much like it always did.
To be quite frank, I know almost nothing about YKD and it scares me a little, so unless a knight in
shining armour (ideally looking like David Tennant) comes along to rescue me I'll leave it there.
Now, where was I, sorry I got distracted by the David Tennant reference. Oh yes, votequorum is the
quorum module based on cman. This is the one I know most about and is probably the most interest to
those of you coming from a cman background. So I'll devote an entire subsection it. Favouritism? Pah!
7.2. votequorum
Votequorum is the name of the corosync module that replaces most of that cman used to do. Only it
does it in a more 'corosync-y' way. It uses the corosync IPC system for communication with users, it
provides quorum to the central core of the corosync daemon and provides familiar tracking callbacks.
I'll outline the API calls for votequorum here. If you are familiar with libcman then you can go and get
a hot chocolate or something while this bit is going on ... see you in chapter 8.
cs_error_t votequorum_initialize(votequorum_handle_t *handle, votequorum_callbacks_t
*callbacks)
cs_error_t votequorum_finalize(votequorum_handle_t handle)
These two are the familiar corosync initialise & finalise calls that corosync/openais developers should
already be familiar with. libcman users will note that the callbacks are specified here rather than in later
API calls. The callbacks are for quorum change or expected votes change – remember votequorum
does not have a messaging API.

8. cs_error_t votequorum_fd_get(votequorum_handle_t handle, int *fd);
cs_error_t votequorum_dispatch(votequorum_handle_t handle,cs_dispatch_flags_t
dispatch_types)
Again, these should be familiar sort of call to both corosync and libcman users. You call
votequorum_dispatch when the FD returned from votequrum_fd_get becomes active from select or
poll.
cs_error_t votequorum_getinfo (votequorum_handle_t handle, unsigned int nodeid, struct
votequorum_info *info)
This call fills in the info structure with information about the quorum subsystem and the node for
which you requested information. Members starting node_ are specific to the node in question, the
others are for the cluster as a whole.
cs_error_t votequorum_setexpected(votequorum_handle_t handle, unsigned int expected_votes)
This call changes the expected_votes value for the cluster. This will persist until it is changed again.
NOTE: if you are using a configuration file then a forced reload of that file will overwrite a manually
set votes value, as will adding a new node to the cluster.
cs_error_t votequorum_setvotes (votequorum_handle_t handle, unsigned int nodeid, unsigned
int votes)
This call changes the number of votes assigned to a existing node in the cluster. This will persist until it
is changed again. NOTE: if you are using a configuration file then a forced reload of that file will
overwrite a manually set votes value.
cs_error_t votequorum_qdisk_register(votequorum_handle_t handle, char *name, unsigned int
votes)
cs_error_t votequorum_qdisk_unregister(votequorum_handle_t handle)
cs_error_t votequorum_qdisk_poll(votequorum_handle_t handle, unsigned int state)
cs_error_t votequorum_qdisk_getinfo(votequorum_handle_t handle, struct
votequorum_qdisk_info *info)
This group of calls manages an optional extra quorum device, often a quorum disk. There can only be
one quorum device per cluster and it is up to the quorum device software to ensure that its state is
consistent across the cluster or to remove it from quorum. The device can have any number of votes,
but those votes cannot be changed without re-registering the device – doing this might cause a
temporary loss of quorum. Quorum devices must poll corosync regularly to be regarded as alive, by
default if a quorum device has not called votequorum_qdisk_poll() after ten seconds then it will be
regarded as dead and removed from quorum. This poll period is customisable, see below.

9. cs_error_t votequorum_setstate(votequorum_handle_t handle)
This call sets a non-resettable flag “HasState” in votequorum. This state is available from the getinfo
call above and used by some subsystems to prevent cluster merges. It only makes sense to call this if
the dreaded “disallowed” flag is set. (see configuration variable below).
cs_error_t votequorum_trackstart(votequorum_handle_t handle, uint64_t context, unsigned int
flags)
cs_error_t votequorum_trackstop(votequorum_handle_t handle)
These are the usual tracking start/stop calls for a corosync plugin. Trackstart enables quorum or
expected_votes change messages to be sent to the notification callback. To avoid race conditions it is
recommended that the flags CS_TRACKSTART | CS_TRACKCHANGES are used so the program
will get one immediate callback containing the current quorum state, then further callbacks as nodes
join or leave the cluster.
cs_error_t votequorum_leaving(votequorum_handle_t handle)
This call sets a flag inside all nodes in the cluster telling them that the node will be leaving the cluster
soon. When the node actually does leave the cluster, then quorum will be reduced and activity will
carry on. This is like 'cman_tool leave remove' in cman. If this is not called before removing a node
then the number of active votes will be reduced and the cluster could lose quorum.
Not that this call does not remove the node from the cluster, it merely sets the flag saying it will. There
is a timeout on this flag of ten seconds. If the node is not shut down within this time it will be
cancelled. It is important that this timeout is longer than CFGs shutdown timeout otherwise leaving will
be cancelled if daemons don't respond.
cs_error_t votequorum_context_get(votequorum_handle_t handle, void **context)
cs_error_t votequorum_context_set(votequorum_handle_t handle, void *context)
These two calls simply allow the context variable returned to the callbacks to be changed.
7.2.1 configuring votequorum
The votequorum module has a small number of variables that you can set in objdb. They are all held
under the main quorum{} section.
quorumdev_poll This is the time between quorum device polls, in milliseconds.
The default is 10000.
leaving_timeout If the node is not shutdown <leaving_timeout> milliseconds after
votequorum_leave is called then the leave flag will be cancelled.
The default is 10000.
votes The number of votes this node has.
The default is 1.

10. expected_votes The number of votes that this cluster expects. votequorum will use the
largest value of all the nodes in the cluster.
The default is 1024. ie you will not have quorum unless you change it!
two_node Enables a special two-node mode where a cluster with only two nodes
and no quorum device can continue running with only one node. This
expects you to have hardware fencing configured. two_node mode can be
cleared without a cluster restart (ie you can add a third node) but it
cannot be set without restarting all nodes.
The default is 0.
disallowed This is an odd mode that prevents two clusters with state (See setstate()
above) from merging. If this happens then nodes will be marked
“disallowed” and not included in quorum calculations. Those who have
read up on RHEL5 cman will know this mode and fear it. It cannot be
changed without a cluster restart.
The default is 0.
8 Configuration free clusters
Configuring a cluster is hard work. Don't you wish you could just plug in a node and have it “just
work”? Well, within several limitations you can. This mode has existed in cman since RHEL4 but is
not well documented (oops, maybe that's my fault) and is hardly used to my knowledge.
The key command to remember is
# cman_tool join -X
On a freshly installed system this will fail with the error:
corosync died: Could not read cluster configuration
OK, so it's not that configuration-free. You don't get something for nothing in this life. However, don't
despair. All you need to do is generate a security key for the cluster and try again:
# mkdir /etc/cluster
# corosync-keygen
# mv /etc/ais/authkey /etc/cluster/cman_authkey
You now need to copy this file to all the nodes you want to join in this cluster. Once you've done that
try cman_tool join -X again and it should magically start up. If you start up the other nodes then they
will join the same cluster.
cman_tool join -X sets up several defaults to get you going. Maybe the first thing you will notice is that
the node numbers are HUGE! They are the IP addresses of the nodes. This isn't a problem, it just looks
a little unwieldy, especially on little-endian systems where adjacently numbered nodes might appear to

11. have random and massively different nodeids. eg:
# cman_tool nodes
Node Id Sts Name
16951488 M amy
33728704 M anna
50505920 M clara
67283136 M fanny
117614784 M nadia
134392000 M sofia
Anyway, if you can cope with those this system should get you going with a simple single cluster that
has no fencing requirements (ie GFS cannot be used).
You can override several things on the cman_tool command-line to add things like sensible nodeids
and a different cluster name or cluster ID. The last two will allow you to run multiple configuration-
less clusters on a single network. By default all nodes will join a cluster called “RHCluster”.
Actually, you can change the nodeids by adding -n<nodeid> to the cman_tool command-line, but that
probably doesn't fit in too well with the zero-configuration system I suppose. See the cman_tool man
page for a list of options.
Quorum is the big problem with this system. There is no way to know how many nodes are needed in
such a cluster. You can set expected_votes on the cman_tool command-line, and I urge you to do so,
but without this quorum will default to 1. Yes ONE. Your cluster will always be quorate. I know this
goes against the principles of a lot of clustering but there are two main reasons why it is done this way.
1. It's easy to get going. A lot of the time, this mode will be used by people just testing clusters and
they want to get going and see what all the fuss is about. Once they understand clusters they can get
involved in sorting out quorum, fencing and configuration files.
2. A lot of the time these clusters are run as virtual machines in systems like Xen or KVM. In this case
there is no danger of a “split-brain” occurring as the systems are either all on the same host computer or
on nodes that are already part of a properly configured cluster with real hardware fencing.
9. Migrating code from libcman
This chapter is just a list of libcman calls showing you where in corosync to find equivalents. Most of
them have very similar names so it shouldn't be too hard I hope. The compatibility libcman provides
most of these calls so you might like to look at the source code for that library if you are unsure about
how to achieve a particular result.
cman_admin_init
cman_init
cman_finish

12. cman_setprivdata
cman_getprivdata
cman_start_notification
cman_stop_notification
cman_start_confchg
cman_stop_confchg
cman_get_fd
cman_dispatch
Use the equivalent calls for the subsystem(s) you are using.
cman_set_votes
cman_set_expected_votes
cman_set_dirty
cman_register_quorum_device
cman_unregister_quorum_device
cman_poll_quorum_device
cman_get_quorum_device
cman_get_node_count
cman_get_nodes
cman_get_disallowed_nodes
cman_get_node
Use libvotequorum. Note though that this does not provide you with node names for the cman_get_
calls.
cman_is_quorate
Use libquorum, or libvotequorum. You should only use libvotequorum if you are also using other
features of that module. If all you need to know is the quorum status then you should use libquorum as
that will work regardless of quorum provider.
cman_get_version
cman_set_version
cman_get_cluster
cman_set_debuglog
Use libconfdb or libccs to read/write to the objdb.
cman_kill_node
cman_shutdown
cman_replyto_shutdown
cman_get_node_addrs
Use libcfg
cman_leave_cluster
If you just need to leave the cluster then libcfg will work. If you need to leave the cluster and make the
remaining nodes keep quorum then you also need to tell votequorum.
cman_send_data
cman_start_recv_data
cman_end_recv_data

13. Use libcpg
cman_get_node_extra
cman_get_subsys_count
cman_is_active
cman_is_listening
cman_barrier_register
cman_barrier_change
cman_barrier_wait
cman_barrier_delete
cman_get_extra_info
cman_get_fenceinfo
cman_node_fenced
These calls are not implemented at all, sorry. You're on your own.