欢迎来到云服务器租用和托管数据中心

网络技术

linux+nginx+tomcat负载均衡,实现session同步

花了一个上午的时间研究nginx+tomcat的负载均衡测试,集群环境搭建比较顺利,但是session同步的问题折腾了几个小时才搞定,现把我的过程贴上来,以备用。
软件及环境是:
虚拟机上装centos 5.5
IP为:192.168.0.51 装上nginx和tomcat  6.0.32 命名为 Tomcat1
一台win7上装tomcat  6.0.32  IP为:192.168.0.50  命名为 Tomcat2

首先装nginx,我是参照http://blog.s135.com/nginx_php_v6/ 配了一个Nginx + PHP(FastCGI)环境,然后再加上Tomcat的转发。
我的nginx.conf 修改如下
linux+nginx+tomcat负载均衡,实现session同步#######################################################################
linux+nginx+tomcat负载均衡,实现session同步#
linux+nginx+tomcat负载均衡,实现session同步# This is the main Nginx configuration file.  
linux+nginx+tomcat负载均衡,实现session同步#
linux+nginx+tomcat负载均衡,实现session同步# More information about the configuration options is available on 
linux+nginx+tomcat负载均衡,实现session同步#   
* the English wiki - http://wiki.nginx.org/Main
linux+nginx+tomcat负载均衡,实现session同步#   * the Russian documentation - http://sysoev.ru/nginx/
linux+nginx+tomcat负载均衡,实现session同步#
linux+nginx+tomcat负载均衡,实现session同步#######################################################################
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步#
----------------------------------------------------------------------
linux+nginx+tomcat负载均衡,实现session同步# Main Module - directives that cover basic functionality
linux+nginx+tomcat负载均衡,实现session同步#
linux+nginx+tomcat负载均衡,实现session同步#   http:
//wiki.nginx.org/NginxHttpMainModule
linux+nginx+tomcat负载均衡,实现session同步#
linux+nginx+tomcat负载均衡,实现session同步#
----------------------------------------------------------------------
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步user              nginx;
linux+nginx+tomcat负载均衡,实现session同步worker_processes  
8;
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步error_log  
/var/log/nginx/error.log;
linux+nginx+tomcat负载均衡,实现session同步#error_log  
/var/log/nginx/error.log  notice;
linux+nginx+tomcat负载均衡,实现session同步#error_log  
/var/log/nginx/error.log  info;
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步pid        
/var/run/nginx.pid;
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步#
----------------------------------------------------------------------
linux+nginx+tomcat负载均衡,实现session同步# Events Module 
linux+nginx+tomcat负载均衡,实现session同步#
linux+nginx+tomcat负载均衡,实现session同步#   http:
//wiki.nginx.org/NginxHttpEventsModule
linux+nginx+tomcat负载均衡,实现session同步#
linux+nginx+tomcat负载均衡,实现session同步#
----------------------------------------------------------------------
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步events 
{
linux+nginx+tomcat负载均衡,实现session同步    worker_connections  
10240;
linux+nginx+tomcat负载均衡,实现session同步}

linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步#
----------------------------------------------------------------------
linux+nginx+tomcat负载均衡,实现session同步# HTTP Core Module
linux+nginx+tomcat负载均衡,实现session同步#
linux+nginx+tomcat负载均衡,实现session同步#   http:
//wiki.nginx.org/NginxHttpCoreModule 
linux+nginx+tomcat负载均衡,实现session同步#
linux+nginx+tomcat负载均衡,实现session同步#
----------------------------------------------------------------------
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步http 
{
linux+nginx+tomcat负载均衡,实现session同步    include       
/etc/nginx/mime.types;
linux+nginx+tomcat负载均衡,实现session同步    default_type  application
/octet-stream;
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步    log_format  main  
'$remote_addr - $remote_user [$time_local] "$request" '
linux+nginx+tomcat负载均衡,实现session同步                      '$status $body_bytes_sent "$http_referer" '
linux+nginx+tomcat负载均衡,实现session同步                      '"$http_user_agent" "$http_x_forwarded_for"';
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步    access_log  
/var/log/nginx/access.log  main;
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步server_names_hash_bucket_size  
128;
linux+nginx+tomcat负载均衡,实现session同步client_header_buffer_size  32k;
linux+nginx+tomcat负载均衡,实现session同步large_client_header_buffers  
4  32K;
linux+nginx+tomcat负载均衡,实现session同步client_max_body_size 8m;
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步    sendfile        on;
linux+nginx+tomcat负载均衡,实现session同步    tcp_nopush     on;
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步    #keepalive_timeout  
0;
linux+nginx+tomcat负载均衡,实现session同步    keepalive_timeout  
65;
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步    #gzip  on;
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步gzip_min_length 1k;
linux+nginx+tomcat负载均衡,实现session同步gzip_buffers 
4  16k;
linux+nginx+tomcat负载均衡,实现session同步gzip_http_version 
1.1;
linux+nginx+tomcat负载均衡,实现session同步gzip_comp_level 
2;
linux+nginx+tomcat负载均衡,实现session同步gzip_types text
/plain application/x-javascript text/css application/xml;
linux+nginx+tomcat负载均衡,实现session同步gzip_vary  on;
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步upstream  tserver  
{
linux+nginx+tomcat负载均衡,实现session同步        server   
192.168.0.51:8080 weight=1;
linux+nginx+tomcat负载均衡,实现session同步        server   
192.168.0.50:8080 weight=1;
linux+nginx+tomcat负载均衡,实现session同步 }

linux+nginx+tomcat负载均衡,实现session同步    
linux+nginx+tomcat负载均衡,实现session同步    #
linux+nginx+tomcat负载均衡,实现session同步    # The 
default server
linux+nginx+tomcat负载均衡,实现session同步    #
linux+nginx+tomcat负载均衡,实现session同步    server 
{
linux+nginx+tomcat负载均衡,实现session同步        listen       
80;
linux+nginx+tomcat负载均衡,实现session同步        server_name  _;
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步        #charset koi8
-r;
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步        #access_log  logs
/host.access.log  main;
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步         location 
~ .*.(php|php5)?$
linux+nginx+tomcat负载均衡,实现session同步        
{
linux+nginx+tomcat负载均衡,实现session同步          root  
/www/ROOT;      
linux+nginx+tomcat负载均衡,实现session同步          #fastcgi_pass  unix:
/tmp/php-cgi.sock;
linux+nginx+tomcat负载均衡,实现session同步         fastcgi_pass  
127.0.0.1:9000;
linux+nginx+tomcat负载均衡,实现session同步         fastcgi_index index.php;
linux+nginx+tomcat负载均衡,实现session同步         include fcgi.conf;
linux+nginx+tomcat负载均衡,实现session同步        }

linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步        location 
~ .(jsp|jspx|do|htm)?{
linux+nginx+tomcat负载均衡,实现session同步         proxy_set_header  Host $host;  
linux+nginx+tomcat负载均衡,实现session同步         proxy_set_header  X
-Real-IP  $remote_addr;
linux+nginx+tomcat负载均衡,实现session同步         proxy_pass http:
//tserver;#转向tomcat处理       
linux+nginx+tomcat负载均衡,实现session同步        }

linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步        location 
~ (/dwr/)? {
linux+nginx+tomcat负载均衡,实现session同步         proxy_set_header  Host $host;
linux+nginx+tomcat负载均衡,实现session同步         proxy_set_header  X
-Real-IP  $remote_addr;
linux+nginx+tomcat负载均衡,实现session同步         proxy_pass http:
//tserver;#转向tomcat处理
linux+nginx+tomcat负载均衡,实现session同步        }

linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步        location 
/ {
linux+nginx+tomcat负载均衡,实现session同步            root   
/www/ROOT;
linux+nginx+tomcat负载均衡,实现session同步            index  index.html index.jsp 
default.jsp index.do default.do;
linux+nginx+tomcat负载均衡,实现session同步        }

linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步        error_page  
404              /404.html;
linux+nginx+tomcat负载均衡,实现session同步        location 
= /404.html {
linux+nginx+tomcat负载均衡,实现session同步            root   
/www/ROOT;
linux+nginx+tomcat负载均衡,实现session同步        }

linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步        # redirect server error pages to the 
static page /50x.html
linux+nginx+tomcat负载均衡,实现session同步        #
linux+nginx+tomcat负载均衡,实现session同步        error_page   
500 502 503 504  /50x.html;
linux+nginx+tomcat负载均衡,实现session同步        location 
= /50x.html {
linux+nginx+tomcat负载均衡,实现session同步            root   
/www/ROOT;
linux+nginx+tomcat负载均衡,实现session同步        }

linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步        # proxy the PHP scripts to Apache listening on 
127.0.0.1:80
linux+nginx+tomcat负载均衡,实现session同步        #
linux+nginx+tomcat负载均衡,实现session同步        #location 
~ .php$ {
linux+nginx+tomcat负载均衡,实现session同步        #    proxy_pass   http:
//127.0.0.1;
linux+nginx+tomcat负载均衡,实现session同步        #}

linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步        # pass the PHP scripts to FastCGI server listening on 
127.0.0.1:9000
linux+nginx+tomcat负载均衡,实现session同步        #
linux+nginx+tomcat负载均衡,实现session同步        #location 
~ .php$ {
linux+nginx+tomcat负载均衡,实现session同步        #    root           html;
linux+nginx+tomcat负载均衡,实现session同步        #    fastcgi_pass   
127.0.0.1:9000;
linux+nginx+tomcat负载均衡,实现session同步        #    fastcgi_index  index.php;
linux+nginx+tomcat负载均衡,实现session同步        #    fastcgi_param  SCRIPT_FILENAME  
/scripts$fastcgi_script_name;
linux+nginx+tomcat负载均衡,实现session同步        #    include        fastcgi_params;
linux+nginx+tomcat负载均衡,实现session同步        #}

linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步        # deny access to .htaccess files, 
if Apache's document root
linux+nginx+tomcat负载均衡,实现session同步        # concurs with nginx's one
linux+nginx+tomcat负载均衡,实现session同步        #
linux+nginx+tomcat负载均衡,实现session同步        #location 
~ /.ht {
linux+nginx+tomcat负载均衡,实现session同步        #    deny  all;
linux+nginx+tomcat负载均衡,实现session同步        #}

linux+nginx+tomcat负载均衡,实现session同步    }

linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步    # Load config files from the 
/etc/nginx/conf.d directory
linux+nginx+tomcat负载均衡,实现session同步    include 
/etc/nginx/conf.d/*.conf;
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步}
linux+nginx+tomcat负载均衡,实现session同步


下一步是配置Tomcat集群。分别打开tomcat1和tomcat2下conf中server.xml
< Server port="8005" shutdown="SHUTDOWN">
< !-- Define a non-SSL HTTP/1.1 Connector on port 8080 -->
< Connector port="8080" maxHttpHeaderSize="8192"
   maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
   enableLookups="false" redirectPort="8443" acceptCount="100"
   connectionTimeout="20000" disableUploadTimeout="true" />
< !-- Define an AJP 1.3 Connector on port 8009 -->
< Connector port="8009" enableLookups="false" redirectPort="8443" protocol="AJP/1.3" />

找到<Engine name="Catalina" defaultHost="localhost">
Tomcat1下的修改为<Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat1">
Tomcat2下的修改为<Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat2">

tomcat1找到
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
修改为
linux+nginx+tomcat负载均衡,实现session同步<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" 
linux+nginx+tomcat负载均衡,实现session同步            channelSendOptions
="6">
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步            
<Manager className="org.apache.catalina.ha.session.DeltaManager"
linux+nginx+tomcat负载均衡,实现session同步                expireSessionsOnShutdown="false"
linux+nginx+tomcat负载均衡,实现session同步                notifyListenersOnReplication="true"/>
linux+nginx+tomcat负载均衡,实现session同步            <!--
linux+nginx+tomcat负载均衡,实现session同步            <Manager className="org.apache.catalina.ha.session.BackupManager"
linux+nginx+tomcat负载均衡,实现session同步                expireSessionsOnShutdown="false"
linux+nginx+tomcat负载均衡,实现session同步                notifyListenersOnReplication="true"
linux+nginx+tomcat负载均衡,实现session同步                mapSendOptions="6"/>
linux+nginx+tomcat负载均衡,实现session同步            -->
linux+nginx+tomcat负载均衡,实现session同步            <Channel className="org.apache.catalina.tribes.group.GroupChannel">
linux+nginx+tomcat负载均衡,实现session同步                <Membership className="org.apache.catalina.tribes.membership.McastService" 
linux+nginx+tomcat负载均衡,实现session同步                    address
="228.0.0.5" 
linux+nginx+tomcat负载均衡,实现session同步                    bind
="192.168.0.51" 
linux+nginx+tomcat负载均衡,实现session同步                    port
="45564" 
linux+nginx+tomcat负载均衡,实现session同步                    frequency
="500" 
linux+nginx+tomcat负载均衡,实现session同步                    dropTime
="3000"/>
linux+nginx+tomcat负载均衡,实现session同步                <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" 
linux+nginx+tomcat负载均衡,实现session同步                    address
="192.168.0.51" 
linux+nginx+tomcat负载均衡,实现session同步                    autoBind
="100" 
linux+nginx+tomcat负载均衡,实现session同步                    port
="4001" 
linux+nginx+tomcat负载均衡,实现session同步                    selectorTimeout
="100" 
linux+nginx+tomcat负载均衡,实现session同步                    maxThreads
="6"/>
linux+nginx+tomcat负载均衡,实现session同步                <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
linux+nginx+tomcat负载均衡,实现session同步                    <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
linux+nginx+tomcat负载均衡,实现session同步                </Sender>
linux+nginx+tomcat负载均衡,实现session同步                <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
linux+nginx+tomcat负载均衡,实现session同步                <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
linux+nginx+tomcat负载均衡,实现session同步            </Channel>
linux+nginx+tomcat负载均衡,实现session同步            <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=".*.gif;.*.js;.*.jpg;.*.png;.*.htm;.*.html;.*.css;.*.txt;"/>
linux+nginx+tomcat负载均衡,实现session同步            
linux+nginx+tomcat负载均衡,实现session同步            
<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
linux+nginx+tomcat负载均衡,实现session同步            <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
linux+nginx+tomcat负载均衡,实现session同步        </Cluster>


tomcat2找到
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
修改为
linux+nginx+tomcat负载均衡,实现session同步<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" 
linux+nginx+tomcat负载均衡,实现session同步            channelSendOptions
="6">
linux+nginx+tomcat负载均衡,实现session同步
linux+nginx+tomcat负载均衡,实现session同步            
<Manager className="org.apache.catalina.ha.session.DeltaManager"
linux+nginx+tomcat负载均衡,实现session同步                expireSessionsOnShutdown="false"
linux+nginx+tomcat负载均衡,实现session同步                notifyListenersOnReplication="true"/>
linux+nginx+tomcat负载均衡,实现session同步            <!--
linux+nginx+tomcat负载均衡,实现session同步            <Manager className="org.apache.catalina.ha.session.BackupManager"
linux+nginx+tomcat负载均衡,实现session同步                expireSessionsOnShutdown="false"
linux+nginx+tomcat负载均衡,实现session同步                notifyListenersOnReplication="true"
linux+nginx+tomcat负载均衡,实现session同步                mapSendOptions="6"/>
linux+nginx+tomcat负载均衡,实现session同步            -->
linux+nginx+tomcat负载均衡,实现session同步            <Channel className="org.apache.catalina.tribes.group.GroupChannel">
linux+nginx+tomcat负载均衡,实现session同步                <Membership className="org.apache.catalina.tribes.membership.McastService" 
linux+nginx+tomcat负载均衡,实现session同步                    address
="228.0.0.5" 
linux+nginx+tomcat负载均衡,实现session同步                    bind
="192.168.0.50" 
linux+nginx+tomcat负载均衡,实现session同步                    port
="45564" 
linux+nginx+tomcat负载均衡,实现session同步                    frequency
="500" 
linux+nginx+tomcat负载均衡,实现session同步                    dropTime
="3000"/>
linux+nginx+tomcat负载均衡,实现session同步                <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" 
linux+nginx+tomcat负载均衡,实现session同步                    address
="192.168.0.50" 
linux+nginx+tomcat负载均衡,实现session同步                    autoBind
="100" 
linux+nginx+tomcat负载均衡,实现session同步                    port
="4000" 
linux+nginx+tomcat负载均衡,实现session同步                    selectorTimeout
="100" 
linux+nginx+tomcat负载均衡,实现session同步                    maxThreads
="6"/>
linux+nginx+tomcat负载均衡,实现session同步                <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
linux+nginx+tomcat负载均衡,实现session同步                    <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
linux+nginx+tomcat负载均衡,实现session同步                </Sender>
linux+nginx+tomcat负载均衡,实现session同步                <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
linux+nginx+tomcat负载均衡,实现session同步                <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
linux+nginx+tomcat负载均衡,实现session同步            </Channel>
linux+nginx+tomcat负载均衡,实现session同步            <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=".*.gif;.*.js;.*.jpg;.*.png;.*.htm;.*.html;.*.css;.*.txt;"/>
linux+nginx+tomcat负载均衡,实现session同步            
linux+nginx+tomcat负载均衡,实现session同步            
<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
linux+nginx+tomcat负载均衡,实现session同步            <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
linux+nginx+tomcat负载均衡,实现session同步        </Cluster>



还需要在程序的web.xml里面</web-app>前面加入以下这句话
<distributable/>
session同步到这里设置完毕

服务器的启动顺序如下:
tomcat1 --> |tomcat2 --> |nginx
先启动tocmat1,tomcat1启动完全的时候再启动tomcat2,等两个tocmat全启动之后,再启动nginx

负载的时候可以用APACHE或NGINX,如果什么都不用,两台一接防问了话,可能会出错极session不同步的问题,网上也很多人介意不要做seeeion同步,这样会降低机器的性能,有一个方法可以,就是NGINX的ip_hash,这样至少可以保证客户端去防问同一台TOCMAT,除非那台TOCMAT挂了

Copyright © 2003-2020 香港服务器和服务器租用 梦飞数据中心 版权所有