In this post we will see how to get instant health alerts if any replica of any collection becomes unhealthy in a cluster.
For this puporse, we will use zookeeper watches. But before we go there, lets see how solr maintains state information for a collection.
State management in solr
All state related information of solr collections is stored in zookeeper and shared by all nodes in the cluster. For each collection, solr maintains a
You can see zookeeper data created by solr from command line:
1 2 3 4 5
collections is a zNode and for each newly created collection, a new child will be created under this node. Lets try creating a collection:
Here’s what zk data looks like after successful collection creation:
1 2 3 4 5
You will see that it created a a child zkNode named
collection1 and under that it created a
state.json file. Lets see content of that file:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
state key on line 14. It maintains state information for each replica. So by checking this value we can know the health of each replica. Try create another collection with multiple shards and replicas and observer the content of
Solr also exposes this information through CLUSTERSTATUS API mentioned below, but there are some rare instances where this API fails to gather and return data from zk. So we can’t rely on that:
Zookeeper chroot used by solr is also exposed in solr admin UI. Navigate to
cloud -> tree to access it and then expand the collections zNode. Here’s what it will look like:
Whenever a replica’s state changes, it will be reflected here. What we want is that whenever any replica’s state is non-active, we want to send an alert. Basically we need to watch each state.json for any change and then trigger an alert, if its not active.
To watch for changes, we will use zookeeper watches.
Zookeeper watches and curator
Zk watches allow clients to get notified when a znode changes in some way. Watches are set by operations, and are triggered by ZooKeeper when anything gets changed. For example, a watch can be placed on a znode which will be triggered when the znode data changes or the znode itself gets deleted.
Watch management is a low level task, so we will use a library called curator(https://curator.apache.org/) which abstract out lot of task like connection management, etc.
We will use
TreeCache recipies provided by curator for setting up watches:
It will be used to keep track of collection being created and deleted. Whenever a child is added, updated or removed, the Path Cache will change its state to contain the current set of children, the children’s data and the children’s state.
To watch for list of collection, we can define a listener on zk’s
/collections zNode like below:
1 2 3 4 5 6 7 8 9
Line 9 will initialize the
collectionCache with list of existing colletions and
PathChildrenCacheListener will make sure that its always in sync with list of collection in zookeeper
Next we want to set watch on
state.json of each collection. Instead of setting a watch individually on each collection’s
state.json zNode, we will use TreeCache recipie which will watch all the files inside the
/collections zNode. TreeCache attempts to keep all data from all children of a ZK path locally cached. This class will watch the ZK path, respond to update/create/delete events, pull down the data, etc. We can register a listener that will get notified when changes occur.
Since we are only intreseted in zNodes named
state.json, we can ignore changes to other children. Here’s how to set the watch:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
data in line 30 is same json which we saw at the starting.
processState will parse this json and filter out unhealthy replicas, which will be send over through an email as an alert.
Note that a
non-active state not necessarily means that a replica is unhealthy and needs manual intervention, so there might be few false alerts too. For instance, when a replica is under recovery due to load on a certain node, its state will show as
recovery but it will auto recover after few minutes. To reduce noise from such alerts, instead of sending email immediately, an alert for a replica can be put on hold and rechecked after few minutes whether it auto-recovered on not
You can find full code for this utility at https://bitbucket.org/saumitra/solr-health-alerts