tmt – Tomcat monitor tool

Our Tomcat server sometimes crushes either process killed or no response from httpd. Before falling into the ‘long-term’ debugging, we have implemented ‘tmt’, which is used to moniter the status of Tomcat process and httpd by restarting the Tomcat timely. In ‘tmt’, we will learn the use of cronjob, ps, ulimit, coredump and curl/wget respectively and the combination of these tools to construct a simple but practical script – ‘tmt’. May it help:)

1. cronjob: basic framework for system monitoring under Unix/Linux
[root@aimlab ~]# crontab -l
*/1 * * * * /home/daveti/aimlab.cron
[root@aimlab ~]# crontab -e

2. coredump: useful way for debugging application/process written in C/C++
If you are wondering why the coredump is not generated or where the coredump is generated, please read on. Generally, whether coredump is enabled or not depends on ‘ulimit’. The name pattern and directory of coredump depend on ‘/proc/sys/kernel/core_pattern’ and ‘/proc/sys/kernel/core_uses_pid’. Detailed info please refer to the kernel document for ‘sysctl’: http://www.kernel.org/doc/Documentation/sysctl/kernel.txt

[daveti@aimlab ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 71680
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 71680
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[daveti@aimlab ~]$ ulimit -c unlimited
[daveti@aimlab ~]$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 71680
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 71680
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[daveti@aimlab ~]$ cat /proc/sys/kernel/core_pattern
core
[daveti@aimlab ~]$ cat /proc/sys/kernel/core_uses_pid
1
[daveti@aimlab ~]$ cd Ctest
[daveti@aimlab Ctest]$ ll
total 120
drwxrwxr-x 2 daveti daveti   4096 Nov  6 13:18 .
drwx—— 9 daveti daveti   4096 Nov  9 14:40 ..
-rw——- 1 daveti daveti 159744 Nov  6 13:18 core.4530
-rwxrwxr-x 1 daveti daveti   6575 Nov  6 13:17 coreTry
-rw-rw-r– 1 daveti daveti    117 Nov  6 13:17 coreTry.c
[daveti@aimlab Ctest]$

3. curl/wget: easy command to detect the response from httpd
Both ‘curl’ and ‘wget’ are commands used to send HTTP HEAD request to ‘httpd’ and to get the corresponding response, through which the status of httpd could be determined then. There may be the case where the process of Tomcat is still alive but the ‘httpd’ is hung somewhere. In this case, the 2 commands could help.

[daveti@aimlab ~]$ curl -I http://daveti.blog.com
HTTP/1.1 200 OK
Server: nginx/0.7.67
Date: Fri, 09 Nov 2012 22:56:08 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
Vary: Accept-Encoding
X-Powered-By: PHP/5.3.6
X-UA-Compatible: IE=edge
X-Pingback: http://daveti.blog.com/xmlrpc.php
Content-Length: 249424
X-Varnish: 224924436
Age: 0
Via: 1.1 varnish

[daveti@aimlab ~]$ wget -S –spider daveti.blog.com
Spider mode enabled. Check if remote file exists.
–2012-11-09 14:56:49–  http://daveti.blog.com/
Resolving daveti.blog.com… 195.170.168.1
Connecting to daveti.blog.com|195.170.168.1|:80… connected.
HTTP request sent, awaiting response…
HTTP/1.1 200 OK
Server: nginx/0.7.67
Date: Fri, 09 Nov 2012 22:56:49 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
Vary: Accept-Encoding
X-Powered-By: PHP/5.3.6
X-UA-Compatible: IE=edge
X-Pingback: http://daveti.blog.com/xmlrpc.php
Content-Length: 249424
X-Varnish: 224928456 224924436
Age: 41
Via: 1.1 varnish
Length: 249424 (244K) [text/html]
Remote file exists and could contain further links,
but recursion is disabled — not retrieving.

[daveti@aimlab ~]$

4. tmt – Tomcat monitor tool

#!/bin/bash
# cron table used to detect if the apache server process
# is still alive. If alive, then leave it; otherwise
# will restart the server process again
# NOTE: this is a workaround for segV in JVM/Apache…
# Sep 18, 2012
# daveti

# Enable core dump for debugging
# ulimit -c ulimit
# Nov 5th, 2012
# daveti

# Add httpd checking if the process is still alive
# but no response from httpd
# curl or wget
# Nov 7th, 2012
# daveti

# Define the process name for apache server
export AIMAPACHE=”/usr/lib/jvm/java-7-sun/bin/java -Djava.util.logging.config.file=/usr/apache-tomcat-7.0.26/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/apache-tomcat-7.0.26/endorsed -classpath /usr/apache-tomcat-7.0.26/bin/bootstrap.jar:/usr/apache-tomcat-7.0.26/bin/tomcat-juli.jar -Dcatalina.base=/usr/apache-tomcat-7.0.26 -Dcatalina.home=/usr/apache-tomcat-7.0.26 -Djava.io.tmpdir=/usr/apache-tomcat-7.0.26/temp org.apache.catalina.startup.Bootstrap start”
export MYHOME=”/home/daveti”
# Define the boolean flag to see if the apache server is alive
export ALIVE=”false”
# Check for the response from http if the tomcat process is there
export SERVERURL=”http://aimlab.cs.uoregon.edu”
export AIMHTTP=”HTTP/1.1 200 OK”
# Try to get the corresponding process…
date > $MYHOME/aimlab.tmp
ps -eo args | grep “org.apache.catalina.startup.Bootstrap start” >> $MYHOME/aimlab.tmp

# Make sure this is the process of apache
while read line
do
if [ “$line” == “$AIMAPACHE” ]
then
ALIVE=”true”
fi
done < $MYHOME/aimlab.tmp

# Check the flag to see if we need to restart apache manually
if [ “$ALIVE” == “false” ]
then
# Save the time of restarting apache…
date >> $MYHOME/aimlab_apache_restart.log
echo “Tomcat process(JVM) got core dump” >> $MYHOME/aimlab_apache_restart.log
# Enable core dump before each restart for further debugging
ulimit -c unlimited
ulimit -a >> $MYHOME/aimlab_apache_restart.log
# Check for the current directory
echo $PWD >> $MYHOME/aimlab_apache_restart.log
# NOTE; this log is used just for debugging to see the frequency
# of crush of apache – the log may need to be deleted if it is too large…
# Restart the server
/usr/apache-tomcat-7.0.26/bin/restart.sh >> /usr/apache-tomcat-7.0.26/logs/cron_restart.log
echo “=================================” >> $MYHOME/aimlab_apache_restart.log
else
# NOTE: the httpd test should not be executed if we have already restarted the tomcat process
# On the other hand, the httpd test should only happen when the process is still alive
# Send http HEAD request
curl -I $SERVERURL >> $MYHOME/aimlab.tmp
# Get the first line of response
# NOTE: need to truncate the extra newline and control characters at the tail
HTTP200OK=`cat $MYHOME/aimlab.tmp | grep “$AIMHTTP” | tr -d nr[:cntrl:]`
# Debug
echo $HTTP200OK >> $MYHOME/aimlab.tmp
echo $AIMHTTP >> $MYHOME/aimlab.tmp
# Restart the tomcat again if bad response from httpd
if [ “$HTTP200OK” != “$AIMHTTP” ]
then
# Debug
echo “NOT 200 OK?” >> $MYHOME/aimlab.tmp
echo $HTTP200OK >> $MYHOME/aimlab.tmp
echo $AIMHTTP >> $MYHOME/aimlab.tmp
# Restart the tomcat
date >> $MYHOME/aimlab_apache_restart.log
echo “Bad http HEAD response from httpd” >> $MYHOME/aimlab_apache_restart.log
curl -I $SERVERURL >> $MYHOME/aimlab_apache_restart.log
/usr/apache-tomcat-7.0.26/bin/restart.sh >> /usr/apache-tomcat-7.0.26/logs/cron_restart.log
echo “=================================” >> $MYHOME/aimlab_apached_restart.log
fi
fi

About daveti

Interested in kernel hacking, compilers, machine learning and guitars.
This entry was posted in Dave's Tools, Programming and tagged , , , , , , , , , . Bookmark the permalink.

One Response to tmt – Tomcat monitor tool

  1. daveti says:

    NOTE: the response of ‘curl’ may be delayed when httpd gets stuck. In this case, running the cronjob so frequently may cause trouble – the last process is restarting while the new process detecting is beginning. Tuning the frequencey in [3, 5] mins should be OK.
    Another cronjob could be run in another server to ping this server. As long as bad pings within certain range of time, like 10min, emails should be triggered to leave a msg to admin:
    daveti@ix: ~/Tools 6$ ping aimlab.cs.uoregon.edu
    no answer from aimlab.cs.uoregon.edu
    daveti@ix: ~/Tools 7$ ping daveti.blog.com
    daveti.blog.com is alive
    daveti@ix: ~/Tools 97$ echo “test email from ix server” | mailx -s “ix to daveti” daveti@cs.uoregon.edu

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s