One or two of our servers have been a little bit overloaded recently. <\/p>\n
They’re going to be replaced with beefier machines, but due to a number of issues I haven’t been able to replace them yet. <\/p>\n
Issue #1 – Pre expo, we weren’t allowed to replace anything.
\nIssue #2 – Post Expo, I’m no longer allowed in the data center! <\/p>\n
We’re working on sorting issue #2 out, but in the interim I need to keep the older machines running.<\/p>\n
I was previously using Monit to monitor system load.<\/p>\n
Monit would be a good solution – it has a web ui, it can stop services if system load goes too high, and generally works when everything else is failed. This is great when things go poopy, but it has one fatal issue.<\/p>\n
It doesn’t know how to restart stuff if load is back to normal.
\nThis typically means that something will put the server load into unusability for a sustained period of time (due to lots of visitors), monit will go ooh, apache has gone awol, and stop it.
\nUnfortunately if its back to normal, monit doesn’t have a way to start it up again, so I need to manually go to the monit page, and start the service. I do get emailed on things like this, but it leads to complaints from the 2 clients that appear to monitor their particular websites more than monit does. <\/p>\n
So, I’ve been looking at other solutions.<\/p>\n
One such solution is sysfence<\/p>\n
While sysfence is severly underdocumented, the examples provided don’t even work!, and appears to be abandoned, it does do the job.
\nSysfence is a no bells and no whistles precursor to monit, but it has that killer feature that monit is missing.<\/p>\n
So, how do we use sysfence?<\/p>\n
Will install it, but unfortunately no config is installed.<\/p>\n So, start off by creating a \/etc\/sysfence folder<\/p>\n We’ll need to create a config file for it, so <\/p>\n My sample sysfence script is below (explanation underneath script)<\/p>\n I'm having issues with apache causing load to rocket, so I've setup some rules as follows:<\/p>\n If load average for 1 minute > 10 (ie server is going bonkers), and load average for the last 5 minutes > 6 then stop apache. eg Issue? All rules need to have a \"rule name\" specified.<\/p>\n So a corrected working version would be:<\/p>\n Back to our setup..<\/p>\n Now we've setup a ruleset, we need to run it. Calling Will run it as a daemon.<\/p>\napt-get install sysfence<\/code><\/p>\nmkdir \/etc\/sysfence
\ncd \/etc\/sysfence<\/code><\/p>\npico sysfence.conf<\/code><\/p>\n
\nrule \"ApacheStop\" {
\n la1 >= 10.00 or la5 >= 6.0
\n}
\nrun '\/etc\/init.d\/apache2 stop;'
\n<\/code>
\n
\nrule \"ApacheStart\" {
\n la1 <=2 \n}\nrun once '\/etc\/init.d\/apache2 start;'\n<\/code>
\n
\nrule \"warning\" { la1 >= 8.00 } run once 'echo \"Load High: BACKUP\" | mail lawrence@computersolutions.cn'
\n<\/code><\/p>\n
\nif load average for 1 minute > 8 send me an email.
\nif load average for 1 minute < 2 then start apache. This will only run one time if load is below 2. \n\nThe documentation http:\/\/sysfence.sourceforge.net\/<\/a> goes over how to write a rule. Note that the examples are broken; <\/p>\n
\nif {
\n la1 >= 8.00
\n} run once 'echo \"SHOW FULL PROCESSLIST\" | mysql | mail my@email.com'
\n<\/code><\/p>\nif \"some rule\" {
\n la1 >= 8.00
\n} run once 'echo \"SHOW FULL PROCESSLIST\" | mysql | mail my@email.com'
\n<\/code><\/p>\nsysfence \/etc\/sysfence.conf<\/code><\/p>\n