Tomcat Cluster Farm Deployer issue

Question

I have setup apache-tomcat cluster with mod_jk on ubuntu server. I have some issues with FarmDeployer which deploys war files to other web servers in the cluster and I got some hope that it works as I posted and got FarmDeployer working here.

The war deployer works perfectly with the below configuration from the post but not working trustfully as it sometimes gives 404 error when accessed the page:

 <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true" xmlValidation="false" xmlNamespaceAware="false"> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"> <Channel className="org.apache.catalina.tribes.group.GroupChannel"> <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" address="192.168.1.4" port="4000" selectorTimeout="100" maxThreads="6"/> <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/> <Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor"> <Member className="org.apache.catalina.tribes.membership.StaticMember" port="4000" securePort="-1" host="192.168.1.2" domain="staging-cluster" uniqueId="{0,1,2,3,4,5,6,7,8,9}"/> </Interceptor> </Channel> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer" tempDir="/usr/share/tomcat/temp/" deployDir="/usr/share/tomcat/webapps/" watchDir="/usr/share/tomcat/watch/" watchEnabled="true"/> </Cluster> </Host> 

---

The following is the configuration(from tomcat cluster docs) which works perfectly for HA and LB clustering. But it fails in Farm Deployer and there is no trace in logs that why the farm deployer is not working.

<Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true" xmlValidation="false" xmlNamespaceAware="false"> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8"> <Manager className="org.apache.catalina.ha.session.DeltaManager" expireSessionsOnShutdown="false" notifyListenersOnReplication="true"/> <Channel className="org.apache.catalina.tribes.group.GroupChannel"> <Membership className="org.apache.catalina.tribes.membership.McastService" address="228.0.0.4" port="45564" frequency="500" dropTime="3000"/> <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter"> <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/> </Sender> <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" address="auto" port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6"/> <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/> <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/> </Channel> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/> <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/> <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/> </Cluster> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer" tempDir="/usr/share/tomcat/temp/" deployDir="/usr/share/tomcat/webapps/" watchDir="/usr/share/tomcat/watch/" watchEnabled="true"/> </Host> 

Anybody has any idea or any working configuration on how to make farm deployer working perfectly with the above HA & LB configuration?.

Thank you!

---

Update:1

I configured with 3 nodes as per your configuration and I could not access the site with the domain name(point to LB) and it's giving tomcat ROOT page but with the ip/myapp it's rendering.

The war file myapp.war is deployed to all the servers but it's not extracting on other 2 nodes other than manager node(deployer web server).

logs on other 2 webservers:

dead.[org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 2, 56}:4000,{-64, -88, 2, 56},4000, alive=2908632,id={64 21 -13 -74 -38 -98 70 117 -117 77 40 -79 124 -110 56 59 }, payload={}, command={}, domain={}, ]] 18 Aug, 2011 3:17:33 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 2, 56}:4000,{-64, -88, 2, 56},4000, alive=1020,id={-65 -93 19 -47 -112 -115 78 -60 -118 99 120 107 -51 115 -88 115 }, payload={}, command={}, domain={}, ] 18 Aug, 2011 3:19:55 AM org.apache.catalina.core.StandardContext stop INFO: Container org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/myapp] has not been started 18 Aug, 2011 3:19:55 AM org.apache.catalina.startup.HostConfig checkResources INFO: Undeploying context [/myapp] 18 Aug, 2011 3:19:59 AM org.apache.catalina.startup.HostConfig checkResources INFO: Undeploying context [/myapp] 18 Aug, 2011 3:20:01 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive myapp.war 18 Aug, 2011 3:20:01 AM org.apache.catalina.startup.ContextConfig init SEVERE: Exception fixing docBase: {0} java.util.zip.ZipException: invalid CEN header (bad signature) at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:131) at java.util.jar.JarFile.<init>(JarFile.java:150) at java.util.jar.JarFile.<init>(JarFile.java:87) at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:90) at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:66) at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:86) at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122) at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89) at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:141) at org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:883) at org.apache.catalina.startup.ContextConfig.init(ContextConfig.java:1012) at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:279) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117) at org.apache.catalina.core.StandardContext.init(StandardContext.java:5338) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4086) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:830) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:515) at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1231) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:297) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795) at org.apache.catalina.ha.deploy.FarmWarDeployer.check(FarmWarDeployer.java:623) at org.apache.catalina.ha.deploy.FarmWarDeployer.messageReceived(FarmWarDeployer.java:246) at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:899) at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:880) at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:269) at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79) at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79) at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:110) at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79) at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:241) at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:225) at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:188) at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:91) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) 18 Aug, 2011 3:20:01 AM org.apache.catalina.core.StandardContext resourcesStart SEVERE: Error starting static Resources java.lang.IllegalArgumentException: Invalid or unreadable WAR file : invalid CEN header (bad signature) at org.apache.naming.resources.WARDirContext.setDocBase(WARDirContext.java:135) at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:3957) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4126) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:830) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:515) at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1231) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:297) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795) at org.apache.catalina.ha.deploy.FarmWarDeployer.check(FarmWarDeployer.java:623) at org.apache.catalina.ha.deploy.FarmWarDeployer.messageReceived(FarmWarDeployer.java:246) at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:899) at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:880) at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:269) at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79) at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79) at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:110) at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79) at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:241) at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:225) at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:188) at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:91) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) 18 Aug, 2011 3:20:01 AM org.apache.catalina.core.StandardContext start SEVERE: Error in resourceStart() 18 Aug, 2011 3:20:01 AM org.apache.catalina.ha.deploy.FileMessageFactory writeMessage WARNING: Receive Message again -- Sender ActTimeout to short [ path: /myapp war: /usr/share/tomcat/webapps/myapp.war data: [B@107108e data length: 6535 ] 18 Aug, 2011 3:20:01 AM org.apache.catalina.core.StandardContext start SEVERE: Error getConfigured 18 Aug, 2011 3:20:01 AM org.apache.catalina.core.StandardContext start SEVERE: Context [/myapp] startup failed due to previous errors 18 Aug, 2011 3:20:01 AM org.apache.catalina.core.StandardContext stop INFO: Container org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/myapp] has not been started 18 Aug, 2011 3:20:09 AM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive myapp.war 18 Aug, 2011 3:20:09 AM org.apache.catalina.startup.ContextConfig init SEVERE: Exception fixing docBase: {0} java.util.zip.ZipException: invalid CEN header (bad signature) at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:131) at java.util.jar.JarFile.<init>(JarFile.java:150) at java.util.jar.JarFile.<init>(JarFile.java:87) at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:90) at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:66) at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:86) at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122) at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89) at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:141) at org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:883) at org.apache.catalina.startup.ContextConfig.init(ContextConfig.java:1012) at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:279) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117) at org.apache.catalina.core.StandardContext.init(StandardContext.java:5338) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4086) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:830) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:719) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:490) at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1217) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:293) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590) at java.lang.Thread.run(Thread.java:636) 18 Aug, 2011 3:20:09 AM org.apache.catalina.core.StandardContext resourcesStart SEVERE: Error starting static Resources java.lang.IllegalArgumentException: Invalid or unreadable WAR file : invalid CEN header (bad signature) at org.apache.naming.resources.WARDirContext.setDocBase(WARDirContext.java:135) at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:3957) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4126) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:830) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:719) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:490) at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1217) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:293) at at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590) at java.lang.Thread.run(Thread.java:636) 18 Aug, 2011 3:20:09 AM org.apache.catalina.core.StandardContext start SEVERE: Error in resourceStart() 18 Aug, 2011 3:20:09 AM org.apache.catalina.core.StandardContext start SEVERE: Error getConfigured 18 Aug, 2011 3:20:09 AM org.apache.catalina.core.StandardContext start SEVERE: Context [/myapp] startup failed due to previous errors 18 Aug, 2011 3:20:09 AM org.apache.catalina.core.StandardContext stop INFO: Container org.apache.catalina.core.ContainerBase.[Catalina].[www.mydomain.com].[/myapp] has not been started 18 Aug, 2011 3:28:09 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:/myapp 18 Aug, 2011 3:28:09 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:/myapp 

---

logs on deployer web server:

18 Aug, 2011 4:50:57 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0.18 18 Aug, 2011 4:50:57 PM org.apache.catalina.core.StandardContext resourcesStart SEVERE: Error starting static Resources java.lang.IllegalArgumentException: Document base /usr/share/tomcat/webapps/myapp does not exist or is not a readable directory at org.apache.naming.resources.FileDirContext.setDocBase(FileDirContext.java:141) at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:3957) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4126) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardHost.start(StandardHost.java:719) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) at org.apache.catalina.core.StandardService.start(StandardService.java:516) at org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at org.apache.catalina.startup.Catalina.start(Catalina.java:578) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413) 18 Aug, 2011 4:50:57 PM org.apache.catalina.core.StandardContext start SEVERE: Error in resourceStart() 18 Aug, 2011 4:50:57 PM org.apache.catalina.core.StandardContext start SEVERE: Error getConfigured 18 Aug, 2011 4:50:57 PM org.apache.catalina.core.StandardContext start SEVERE: Context [] startup failed due to previous errors 18 Aug, 2011 4:50:57 PM org.apache.catalina.core.StandardContext stop INFO: Container org.apache.catalina.core.ContainerBase.[Catalina].[www.mydomain.com].[/] 18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.deploy.FarmWarDeployer start INFO: Cluster deployment is watching /usr/share/tomcat/watch/ for changes. 18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.deploy.FarmWarDeployer start INFO: Cluster FarmWarDeployer started. 18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.JvmRouteBinderValve start INFO: JvmRouteBinderValve started 18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager start INFO: Register manager /manager to cluster element Host with name localhost 18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager start INFO: Starting clustering manager at /manager 18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions INFO: Manager [/manager]: skipping state transfer. No members active in cluster group. 18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager start INFO: Register manager /docs to cluster element Host with name localhost INFO: Starting clustering manager at 18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions INFO: Manager []: skipping state transfer. No members active in cluster group. 18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager start INFO: Register manager /host-manager to cluster element Host with name localhost 18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager start INFO: Starting clustering manager at /host-manager 18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions INFO: Manager [/host-manager]: skipping state transfer. No members active in cluster group. 18 Aug, 2011 4:51:00 PM org.apache.catalina.ha.session.DeltaManager start INFO: Register manager /examples to cluster element Host with name localhost 18 Aug, 2011 4:51:00 PM org.apache.catalina.ha.session.DeltaManager start INFO: Starting clustering manager at /examples 18 Aug, 2011 4:51:00 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions INFO: Manager [/examples]: skipping state transfer. No members active in cluster group. 18 Aug, 2011 4:51:00 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-9080 18 Aug, 2011 4:51:00 PM org.apache.jk.common.ChannelSocket init INFO: JK: ajp13 listening on /0.0.0.0:8009 18 Aug, 2011 4:51:00 PM org.apache.jk.server.JkMain start INFO: Jk running ID=0 time=0/57 config=null 18 Aug, 2011 4:51:00 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 3206 ms 18 Aug, 2011 4:51:01 PM org.apache.catalina.tribes.io.BufferPool getBufferPool INFO: Created a buffer pool with max size:104857600 bytes of notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://192.168.1.101:4002,192.168.1.101,4002, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={115 116 97 103 105 110 103 45 99 ...(15)}, ] 18 Aug, 2011 4:51:04 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://192.168.1.101:4002,192.168.1.101,4002, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={115 116 97 103 105 110 103 45 99 ...(15)}, ] 18 Aug, 2011 4:51:06 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 2, 53}:4001,{-64, -88, 2, 53},4001, alive=1007,id={80 88 -34 -11 98 72 71 -70 -75 -107 -48 -91 -90 50 -126 116 }, payload={}, command={}, domain={}, ] 18 Aug, 2011 4:51:09 PM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector performBasicCheck WARNING: Member added, even though we werent notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://192.168.1.102:4001,192.168..102,4001, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={115 116 97 103 105 110 103 45 99 ...(15)}, ] 18 Aug, 2011 4:51:09 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://192.168.1.102:4001,192.168.1.102,4001, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={115 116 97 103 105 110 103 45 99 ...(15)}, ] 18 Aug, 2011 4:51:20 PM org.apache.catalina.ha.deploy.WarWatcher check INFO: check cluster wars at /usr/share/tomcat/watch 18 Aug, 2011 4:51:20 PM org.apache.catalina.ha.deploy.FarmWarDeployer fileModified INFO: Installing webapp[/myapp] from /usr/share/tomcat/webapps/myapp.war 18 Aug, 2011 4:51:20 PM org.apache.catalina.ha.deploy.FarmWarDeployer remove INFO: Cluster wide remove of web app /myapp 18 Aug, 2011 4:51:26 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive myapp.war 18 Aug, 2011 4:51:27 PM org.apache.catalina.ha.session.DeltaManager start INFO: Register manager /myapp to cluster element Host with name localhost 18 Aug, 2011 4:51:27 PM org.apache.catalina.ha.session.DeltaManager start INFO: Starting clustering manager at /myapp 18 Aug, 2011 4:51:27 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions WARNING: Manager [/myapp], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 2, 101}:4002,{-64, -88, 2, 101},4002, alive=26275,id={-120 -66 103 32 -56 -122 77 51 -95 20 25 88 83 -69 -59 -115 }, payload={}, command={}, domain={}, ]. This operation will timeout if no session state has been received within 60 seconds. 18 Aug, 2011 4:52:27 PM org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions SEVERE: Manager [/myapp]: No session state send at 18/8/11 4:51 PM received, timing out after 60,073 ms. 18 Aug, 2011 4:52:44 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive myapp.war 18 Aug, 2011 4:52:45 PM org.apache.catalina.core.ApplicationContext log INFO: Initializing Spring root WebApplicationContext 18 Aug, 2011 4:52:48 PM org.apache.catalina.core.ApplicationContext log INFO: org.tuckey.web.filters.urlrewrite.utils.Log ERROR: logLevelConf: null 18 Aug, 2011 4:52:48 PM org.apache.catalina.core.ApplicationContext log INFO: org.tuckey.web.filters.urlrewrite.UrlRewriteFilter INFO: loaded (conf ok) 18 Aug, 2011 4:52:48 PM org.apache.catalina.core.ApplicationContext log INFO: Initializing Spring FrameworkServlet 'SpringAction' 18 Aug, 2011 4:53:00 PM org.apache.catalina.ha.deploy.WarWatcher check INFO: check cluster wars at /usr/share/tomcat/watch 18 Aug, 2011 4:53:20 PM org.apache.catalina.ha.deploy.WarWatcher check 

---

Update:2 for EDIT

ifconfig listed the below entry for eth0 on all web servers

UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

Executed the below command on all the web servers

route add -host 228.0.0.4 dev eth0

ping to 228.0.0.4 didn't work on any of the system

lsof -i :4000 listed similar to what is listed to you(could not paste here as characters exceeded for this post) on deployer web server(also on other servers):

IMPROVEMENT: I happily Used the Multicast configuration changing auto to ip in Receiver and I saw myapp.war on another web server when I placed the war file in watch directory on deployer web server. The Problem is, war is only extracting on the deployer web server(creating a direcotry) but not on other servers.

Answer 1

How do I update for 3 web servers?

Do the same as when doing with 2 nodes.

On the deployment node, add one more member in StaticMembershipInterceptor:

<Member className="org.apache.catalina.tribes.membership.StaticMember" port="4002" securePort="-1" host="192.168.5.208" domain="staging-cluster" uniqueId="{0,1,2,3,4,5,6,7,8,9}"/> 

On the node 3, create a cluster like belows:

 <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"> <Channel className="org.apache.catalina.tribes.group.GroupChannel"> <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" address="192.168.5.208" port="4002" selectorTimeout="100" maxThreads="6"/> </Channel> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer" tempDir="/opt/tomcat6/tempdir/" deployDir="/opt/tomcat6/webapps/" watchDir="/opt/tomcat6/watchdir/" watchEnabled="false"/> </Cluster> 

And copy a *.war file to watchdir directory on the deployment node, the below lines will appears in catalina.out:

INFO: check cluster wars at /usr/share/tomcat6/watchdir Aug 17, 2011 2:07:53 PM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector performBasicCheck WARNING: Member added, even though we werent notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://192.168.5.208:40 02,192.168.5.208,4002, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={115 116 97 103 105 110 103 45 99 ...(15)}, ] Aug 17, 2011 2:07:53 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://192.168.5.208:4002,192.168.5.208,4002, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={115 116 97 103 105 110 103 45 99 ...(15)}, ] Aug 17, 2011 2:08:12 PM org.apache.catalina.ha.deploy.WarWatcher check INFO: check cluster wars at /usr/share/tomcat6/watchdir Aug 17, 2011 2:08:12 PM org.apache.catalina.ha.deploy.FarmWarDeployer fileModified INFO: Installing webapp[/cas] from /usr/share/tomcat6/webapps/cas.war Aug 17, 2011 2:08:12 PM org.apache.catalina.ha.deploy.FarmWarDeployer remove INFO: Cluster wide remove of web app /cas Aug 17, 2011 2:08:13 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive cas.war 

Check the webapps folder on the node 3:

drwxr-xr-x 8 root root 4096 Aug 17 14:08 cas -rw-r--r-- 1 root root 19962865 Aug 17 14:08 cas.war ... 

and the catalina.out:

Aug 17, 2011 2:08:13 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive cas.war 

---

EDIT

Invalid or unreadable WAR file : invalid CEN header (bad signature)

It looks like the WAR file is corrupted when deploying to 2 other nodes. How often does it happen?

Isn't it possible making FarmDeployer work with multicast from tomcat cluster docs which works perfectly for me apart from FarmDeployer?

I figured it out! The reason is you haven't added a multicast route for your network interface, so the cluster message cannot be send/receive between Tomcat nodes:

lsof -i :4000 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME java 9968 tomcat 37u IPv6 121232775 TCP localhost:terabase (LISTEN) 

Do it by following steps:

• Make sure that your network interface is enabled for multicast:

  UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

• Add a multicast route:

  route add -host 228.0.0.4 dev bond1

(you should change bond1 to the network interface that receives the cluster message)

Testing with ping -c 4 228.0.0.4 didn't work for me. So, I downloaded multicast test tool, run it and I see the belows response:

./bin/multicast-test.sh -local 192.168.5.149 -group 228.0.0.4:45564 PATH=/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/ossec/bin java version "1.6.0_13" Java(TM) SE Runtime Environment (build 1.6.0_13-b03) Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode) Oracle Coherence Version 3.3.1/389 Grid Edition: Development mode Copyright (c) 2000-2007 Oracle. All rights reserved. Starting test on ip=/192.168.5.149, group=/228.0.0.4:45564, ttl=4 Configuring multicast socket... Starting listener... Fri Aug 19 10:19:14 ICT 2011: Sent packet 1. Fri Aug 19 10:19:14 ICT 2011: Received test packet 1 from self (sent 9ms ago). Fri Aug 19 10:19:14 ICT 2011: Received 69 bytes from an unknown multicast application at 192.168.5.199: ??? Fri Aug 19 10:19:14 ICT 2011: Received 69 bytes from an unknown multicast application at 192.168.5.149: ??? 

Or you can check with lsof, netstat, ...

lsof -i :4000 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME java 11084 tomcat 37u IPv6 121236631 TCP localhost:terabase (LISTEN) java 11084 tomcat 59u IPv6 121236830 TCP localhost:terabase->192.168.5.199:43519 (ESTABLISHED) java 11084 tomcat 66u IPv6 121236835 TCP localhost:55538->192.168.5.199:terabase (ESTABLISHED) java 11084 tomcat 67u IPv6 121236836 TCP localhost:55539->192.168.5.199:terabase (ESTABLISHED) 

You are ready for configure Tomcat clustering with multicast.

Enable clustering configuration same as mentioned in the doc, but change <Receiver address from "auto" to specified interface:

<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" address="192.168.5.149" port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6"/> 

Do the same for all the nodes in the cluster to see it works.

---

EDIT 2

IMPROVEMENT: I happily Used the Multicast configuration changing auto to ip in Receiver and I saw myapp.war on another web server when I placed the war file in watch directory on deployer web server. The Problem is, war is only extracting on the deployer web server(creating a direcotry) but not on other servers.

It means that the connection between deployment node and the other nodes is established. Did you still get the same error "Invalid or unreadable WAR file : invalid CEN header (bad signature)"? Everytime?

---

java.util.zip.ZipException: invalid END header (bad central directory offset) 

According to the bug 45851, you should upgrade your Tomcat version (at least 6.0.19) and try again. I'm using version 6.0.29 and as you can see, it works fine.

---

It's needing tomcat restart on other web servers then it copies and extracts and the site is available with other-web-severs-ip/myapp but not with domain name(404 error). Where do you point the site domain name, I pointed the site domain name in the DNS to LB and created a virtual host in all the tomcat web servers with same domain name

No. It seems that you haven't cleared thoroughly. No need to create virtual host in all Tomcat nodes.

Assuming that you have Apache HTTP server and Tomcat on node 1, with IP 192.168.1.101. Two other Tomcat nodes at 1.102, 1.103. jvmRoute corresponding are: tomcat1, tomcat2, tomcat3.

The mod_jk.conf on Apache HTTP server:

LoadModule jk_module modules/mod_jk.so JkWorkersFile /etc/httpd/conf/workers.properties JkShmFile /var/log/httpd/jk.shm JkLogFile /var/log/httpd/mod_jk.log JkLogLevel info JkLogStampFormat "[%a %b %d %H:%M:%S %Y] " JkMount /myapp/* cluster 

The workers.properties is something like this:

worker.list = cluster worker.tomcat1.port=8009 worker.tomcat1.host=localhost worker.tomcat1.type=ajp13 worker.tomcat1.lbfactor=1 worker.tomcat2.port=8009 worker.tomcat2.host=192.168.1.102 worker.tomcat2.type=ajp13 worker.tomcat2.lbfactor=1 worker.tomcat3.port=8009 worker.tomcat3.host=192.168.1.103 worker.tomcat3.type=ajp13 worker.tomcat3.lbfactor=1 worker.cluster.type=lb worker.cluster.balance_workers=tomcat1,tomcat2,tomcat3 worker.cluster.sticky_session=1 

Try again.