OS运维 自动化

一. 限制每秒请求数

ngx_http_limit_req_module模块通过漏桶原理来限制单位时间内的请求数，一旦单位时间内请求数超过限制，就会返回503错误。配置需要在两个地方设置：

• nginx.conf的http段内定义触发条件，可以有多个条件
• 在location内定义达到触发条件时nginx所要执行的动作

例如：

http {
    limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s; //触发条件，所有访问ip 限制每秒10个请求
    ...
    server {
        ...
        location  ~ \.php$ {
            limit_req zone=one burst=5 nodelay;   //执行的动作,通过zone名字对应
               }
           }
     }

参数说明：

$binary_remote_addr  二进制远程地址
zone=one:10m    定义zone名字叫one，并为这个zone分配10M内存，用来存储会话（二进制远程地址），1m内存可以保存16000会话
rate=10r/s;     限制频率为每秒10个请求
burst=5         允许超过频率限制的请求数不多于5个，假设1、2、3、4秒请求为每秒9个，那么第5秒内请求15个是允许的，反之，如果第一秒内请求15个，会将5个请求放到第二秒，第二秒内超过10的请求直接503，类似多秒内平均速率限制。
nodelay         超过的请求不被延迟处理，设置后15个请求在1秒内处理。

二.限制IP连接数

ngx_http_limit_conn_module的配置方法和参数与http_limit_req模块很像，参数少，要简单很多

http {
    limit_conn_zone $binary_remote_addr zone=addr:10m; //触发条件
    ...
    server {
        ...
        location /download/ {
            limit_conn addr 1;    // 限制同一时间内1个连接，超出的连接返回503
                }
           }
     }

三.白名单设置

http_limit_conn和http_limit_req模块限制了单ip单位时间内的并发和请求数，但是如果Nginx前面有lvs或者haproxy之类的负载均衡或者反向代理，nginx获取的都是来自负载均衡的连接或请求，这时不应该限制负载均衡的连接和请求，就需要geo和map模块设置白名单：

geo $whiteiplist  {
        default 1;
        10.11.15.161 0;
    }
map $whiteiplist  $limit {
        1 $binary_remote_addr;
        0 "";
    }
limit_req_zone $limit zone=one:10m rate=10r/s;
limit_conn_zone $limit zone=addr:10m;

geo模块定义了一个默认值是1的变量whiteiplist，当在ip在白名单中，变量whiteiplist的值为0，反之为1
如果在白名单中--> whiteiplist=0 --> $limit="" --> 不会存储到10m的会话状态（one或者addr）中 --> 不受限制
反之，不在白名单中 --> whiteiplist=1 --> $limit=二进制远程地址 -->存储进10m的会话状态中 --> 受到限制

四.测试

使用ab命令来模拟CC攻击，http_limit_conn和http_limit_req模块要分开测试，同时注意http_limit_conn模块只统计正在被处理的请求（这些请求的头信息已被完全读入）所在的连接。如果请求已经处理完，连接没有被关闭时，是不会被统计的。这时用netstat看到连接数可以超过限定的数量，不会被阻止。

ab -n 请求数 -c 并发 http://10.11.15.174/i.php

如果被阻止前台会返回503，同时在nginx的error_log中会看到如下错误日志：
被限制连接数：

2015/01/28 14:20:26 [error] 4107#0: *65525 limiting connections by zone "addr", client: 10.11.15.161, server: , request: "GET /i.php?=PHPE9568F35-D428-11d2-A769-00AA001ACF42 HTTP/1.1", host: "10.11.15.174", referrer: "http://10.11.15.174/i.php"

被限制请求数：

2015/01/28 14:18:59 [error] 4095#0: *65240 limiting requests, excess: 5.772 by zone "one", client: 10.11.15.161, server: , request: "GET /i.php?=PHPE9568F34-D428-11d2-A769-00AA001ACF42 HTTP/1.1", host: "10.11.15.174", referrer: "http://10.11.15.174/i.php"

五.其它一些防CC的方法

1.Nginx模块 ModSecurity、http_guard、ngx_lua_waf

• ModSecurity 应用层WAF，功能强大，能防御的攻击多，配置复杂
• ngx_lua_waf 基于ngx_lua的web应用防火墙，使用简单，高性能和轻量级
• http_guard 基于openresty

2.软件+Iptables

• fail2ban 通过分析日志来判断是否使用iptables拦截
• DDoS Deflate 通过netstat判断ip连接数，并使用iptables屏蔽

开头说过抗DDOS是一个系统工程，通过优化系统和软件配置，只能防御小规模的CC攻击，对于大规模攻击、四层流量攻击、混合攻击来说，基本上系统和应用软件没挂，带宽就打满了。下面是我在工作中使用过的防御DDOS的方式：

1. 高防服务器和带流量清洗的ISP 通常是美韩的服务器，部分ISP骨干供应商有流量清洗服务，例如香港的PCCW。通常可以防御10G左右的小型攻击
2. 流量清洗服务 例如：akamai(prolexic),nexusguard 我们最大受到过80G流量的攻击，成功被清洗，但是费用非常贵
3. CDN 例如：蓝讯 网宿 cloudflare 等，CDN针对DDOS的分布式特点，将流量引流分散，同时对网站又有加速作用，效果好，成本相对低。

1.修改最大连接数

最大连接数不够的话，nginx日志中会出现"Too many open files"错误。系统默认的1024太小了，在/etc/security/limits.conf中增加：

* soft nproc 65535
* hard nproc 65535
* soft nofile 65535
* hard nofile 65535

2.sysctl优化

这个比较考验内功，暂时还没太多研究，从网上搬运了一份，以后在慢慢学习：

###
### GENERAL SYSTEM SECURITY OPTIONS ###
###

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

#Allow for more PIDs
kernel.pid_max = 65535

# The contents of /proc/<pid>/maps and smaps files are only visible to
# readers that are allowed to ptrace() the process
kernel.maps_protect = 1

#Enable ExecShield protection
kernel.exec-shield = 1
kernel.randomize_va_space = 2

# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65535

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65535

# Restrict core dumps
fs.suid_dumpable = 0

# Hide exposed kernel pointers
kernel.kptr_restrict = 1

###
### IMPROVE SYSTEM MEMORY MANAGEMENT ###
###

# Increase size of file handles and inode cache
fs.file-max = 209708

# Do less swapping
vm.swappiness = 30
vm.dirty_ratio = 30
vm.dirty_background_ratio = 5

# specifies the minimum virtual address that a process is allowed to mmap
vm.mmap_min_addr = 4096

# 50% overcommitment of available memory
vm.overcommit_ratio = 50
vm.overcommit_memory = 0

# Set maximum amount of memory allocated to shm to 256MB
kernel.shmmax = 268435456
kernel.shmall = 268435456

# Keep at least 64MB of free RAM space available
vm.min_free_kbytes = 65535

###
### GENERAL NETWORK SECURITY OPTIONS ###
###

#Prevent SYN attack, enable SYNcookies (they will kick-in when the max_syn_backlog reached)
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_max_syn_backlog = 4096

# Disables packet forwarding
net.ipv4.ip_forward = 0
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.default.forwarding = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.default.forwarding = 0

# Disables IP source routing
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0

# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0

# Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 7

# Decrease the time default value for connections to keep alive
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15

# Don't relay bootp
net.ipv4.conf.all.bootp_relay = 0

# Don't proxy arp for anyone
net.ipv4.conf.all.proxy_arp = 0

# Turn on the tcp_timestamps, accurate timestamp make TCP congestion control algorithms work better
net.ipv4.tcp_timestamps = 1

# Don't ignore directed pings
net.ipv4.icmp_echo_ignore_all = 0

# Enable ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Enable bad error message Protection
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Allowed local port range
net.ipv4.ip_local_port_range = 16384 65535

# Enable a fix for RFC1337 - time-wait assassination hazards in TCP
net.ipv4.tcp_rfc1337 = 1

# Do not auto-configure IPv6
net.ipv6.conf.all.autoconf=0
net.ipv6.conf.all.accept_ra=0
net.ipv6.conf.default.autoconf=0
net.ipv6.conf.default.accept_ra=0
net.ipv6.conf.eth0.autoconf=0
net.ipv6.conf.eth0.accept_ra=0

###
### TUNING NETWORK PERFORMANCE ###
###

# For high-bandwidth low-latency networks, use 'htcp' congestion control
# Do a 'modprobe tcp_htcp' first
net.ipv4.tcp_congestion_control = htcp

# For servers with tcp-heavy workloads, enable 'fq' queue management scheduler (kernel > 3.12)
net.core.default_qdisc = fq

# Turn on the tcp_window_scaling
net.ipv4.tcp_window_scaling = 1

# Increase the read-buffer space allocatable
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.udp_rmem_min = 16384
net.core.rmem_default = 262144
net.core.rmem_max = 16777216

# Increase the write-buffer-space allocatable
net.ipv4.tcp_wmem = 8192 65536 16777216
net.ipv4.udp_wmem_min = 16384
net.core.wmem_default = 262144
net.core.wmem_max = 16777216

# Increase number of incoming connections
net.core.somaxconn = 32768

# Increase number of incoming connections backlog
net.core.netdev_max_backlog = 16384
net.core.dev_weight = 64

# Increase the maximum amount of option memory buffers
net.core.optmem_max = 65535

# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks
net.ipv4.tcp_max_tw_buckets = 1440000

# try to reuse time-wait connections, but don't recycle them (recycle can break clients behind NAT)
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1

# Limit number of orphans, each orphan can eat up to 16M (max wmem) of unswappable memory
net.ipv4.tcp_max_orphans = 16384
net.ipv4.tcp_orphan_retries = 0

# Increase the maximum memory used to reassemble IP fragments
net.ipv4.ipfrag_high_thresh = 512000
net.ipv4.ipfrag_low_thresh = 446464

# don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1

# Increase size of RPC datagram queue length
net.unix.max_dgram_qlen = 50

# Don't allow the arp table to become bigger than this
net.ipv4.neigh.default.gc_thresh3 = 2048

# Tell the gc when to become aggressive with arp table cleaning.
# Adjust this based on size of the LAN. 1024 is suitable for most /24 networks
net.ipv4.neigh.default.gc_thresh2 = 1024

# Adjust where the gc will leave arp table alone - set to 32.
net.ipv4.neigh.default.gc_thresh1 = 32

# Adjust to arp table gc to clean-up more often
net.ipv4.neigh.default.gc_interval = 30

# Increase TCP queue length
net.ipv4.neigh.default.proxy_qlen = 96
net.ipv4.neigh.default.unres_qlen = 6

# Enable Explicit Congestion Notification (RFC 3168), disable it if it doesn't work for you
net.ipv4.tcp_ecn = 1
net.ipv4.tcp_reordering = 3

# How many times to retry killing an alive TCP connection
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_retries1 = 3

# Avoid falling back to slow start after a connection goes idle
# keeps our cwnd large with the keep alive connections (kernel > 3.6)
net.ipv4.tcp_slow_start_after_idle = 0

# Allow the TCP fastopen flag to be used, beware some firewalls do not like TFO! (kernel > 3.7)
net.ipv4.tcp_fastopen = 3

# This will enusre that immediatly subsequent connections use the new values
net.ipv4.route.flush = 1
net.ipv6.route.flush = 1

    # 具体值根据服务器硬件计算，配置不当可能导致过早关闭TCP连接
    # net.netfilter.nf_conntrack_max = 1048576
    # net.netfilter.nf_conntrack_tcp_timeout_established = 1200

3.nginx和lua防御cc攻击

参考了opencdn团队的做法，通过nginx和lua来防御cc，原理见下面的参考文章，效果很棒
nginx需要编译lua模块，见：http://www.52os.net/articles/nginx-install-lua-and-lua-based-waf.html
在nginx.conf的http段中加入：

limit_req_zone $cookie_token zone=session_limit:20m rate=1r/s;
limit_req_zone $binary_remote_addr $uri zone=auth_limit:20m rate=1r/m;

在server段中加入：

location  / {
    limit_req zone=session_limit burst=5;
    rewrite_by_lua '
        local random = ngx.var.cookie_random
        if (random == nil) then
            return ngx.redirect("/auth?url=" .. ngx.var.request_uri)
        end
        local token = ngx.md5("opencdn" .. ngx.var.remote_addr .. random)
        if (ngx.var.cookie_token ~= token) then
            return ngx.redirect("/auth?url=".. ngx.var.request_uri)
        end
    ';
            proxy_redirect         off;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
                            proxy_pass http://backend;
    }

location /auth {
        limit_req zone=auth_limit burst=1; 
        if ($arg_url = "") {
            return 403;
        }     
        access_by_lua '
            local random = math.random(9999)
            local token = ngx.md5("opencdn" .. ngx.var.remote_addr .. random)
            if (ngx.var.cookie_token ~= token) then
                ngx.header["Set-Cookie"] = {"token=" .. token, "random=" .. random}
                return ngx.redirect(ngx.var.arg_url)
            end
        ';
    }

这个方法会造成搜索引擎蜘蛛一直处在302中，不利于seo，可以通过智能dns来为蜘蛛指定单独的线路。和被打到宕机比起来，seo几乎可以无视

4.iptables限制tcp连接和频率

通过上述的配置，cc攻击流量就处在302中了，但是保险起见对ip进行连接频率和并发限制，限制单ip连接和频率，在/etc/sysconfig/iptables中加入：

#单个IP在60秒内只允许新建20个连接
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -m state --state NEW -m recent --update --seconds 60 --hitcount 20 --name DEFAULT --rsource -j DROP
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -m state --state NEW -m recent --set --name DEFAULT --rsource

#控制单个IP的最大并发连接数为20
-I INPUT -p tcp --dport 80 -m connlimit  --connlimit-above 20 -j REJECT  

#每个IP最多20个初始连接
-A INPUT -p tcp --syn -m connlimit --connlimit-above 20 -j DROP

这样配置后，单个ip能建立的连接不是只有20个，具体能建立多少连接还要看tcp的超时设置，但单个ip不会建立大量的tcp连接消耗系统资源

5.使用fail2ban屏蔽攻击ip

通过上面设置nginx后，cc攻击请求变为302，直接由性能强劲的nginx处理。但是攻击ip还是在不停的访问服务器，消耗着服务器的资源，一旦达到一定数量级，也会严重影响到系统的性能，所以通过分析nginx的访问日志彻底屏蔽这些ip

安装fail2ban并升级iptables至最新：

yum install -y epel-release 
yum install -y fail2ban iptables python-inotify

先看下我nginx的访问日志格式 ：

log_format  main  '$remote_addr $status $request $body_bytes_sent [$time_local]  $http_user_agent $http_referer  $http_x_forwarded_for $upstream_addr $upstream_status $upstream_cache_status $upstream_response_time';

攻击日志的效果：

159.138.198.106 302 GET /auth?url=/ HTTP/1.1 235 [17/Oct/2015:21:06:22 +0800]  Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/600.4.10 (KHTML, like Gecko) Version/8.0.4 Safari/600.4.10 -  - - - - -

cc攻击的ip会经过nginx和lua处理后，访问状态变为302，根据nginx的访问日志格式，过滤这些ip和302状态，加入黑名单即可。
新建fail2ban的规则文件/etc/fail2ban/filter.d/nginx-302-cc.conf，内容为：

[Definition]
failregex = <HOST> 302.(GET|POST)*.*HTTP/1.*$
ignoreregex =

新建fail2ban的配置文件/etc/fail2ban/jail.d/nginx-anti-302.conf,内容为：

[nginx-anti-302]
enabled = true
port = http
filter = nginx-302-cc
logpath = /opt/nginx/logs/52os.net/access_web.log
findtime = 60     #检测60秒内的日志
bantime = 900     #屏蔽ip的时间为15分钟
maxretry = 90      #达到90次就屏蔽
    backend = pyinotify #使用pyinotify检测日志变化，被攻击时检测海量日志时性能最好
    banaction = iptables-ipset-proto6-allports  #使用ipset屏蔽IP，使用iptables屏蔽大量IP需要时非常慢，并且资源占用非常大

访客访问一次网站会产生2次302，这样配置后60秒内允许45次正常的访问，基本上不会屏蔽正常访客

如果使用iptables屏蔽，需注意fail2ban-0.9.3在执行iptables命令时，会加上了-w参数防止规则冲突，iptables-1.4.20以后才有这个参数，而CentOS 6 的iptables是1.4.7,导致iptables规则添加失败，解决方法是删除iptables-common.conf中的<lockingopt>：

sed -i 's/iptables = iptables <lockingopt>/iptables = iptables/' /etc/fail2ban/action.d/iptables-common.conf

启动fail2ban ：

service fail2ban start    

通过以上设置实现了：

1. 增大了系统的吞吐量
2. cc流量直接由高性能的nginx返回302，不会proxy_pass到后端的服务器或应用
3. 限制单个ip建立的tcp连接数量和频率
4. 恶意攻击ip实时黑名单

实际使用效果非常不错。面对专业的ddos玩家，在好的系统终有薄弱的环节，攻击达到一定规模，基本上是不可防的，但是可以尽量利用有限的资源和攻击者周旋，提高攻击的门槛。当然，要是烧的起钱，这篇文章可以无视

http://mp.weixin.qq.com/s/3nL_qcNqcn7JgamDB9rNaw

http://mp.weixin.qq.com/s/tx6l9d38vb_80z4-LJq0XQ