正文
基于queryperf 和 perftcpdns 的DNS压力测试
提示：扫一扫查出行【扫一扫了解最新限行尾号】
复制提示
最近在AWS上安装了PPTP VPN 做代理，手机, pad 也可以无缝FQ，甚是开心。最近工作不太忙，研究一下缓存加速的调优。系统已经安装的nscd文件级的缓存和dnsmasq，cpu级的dns缓存。为什么说nscd是文件级的缓存呢，事情得先从dnsmasq说起，大家都知道nscd -g是可以看到他对hosts的缓存命中率的：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
|
[root@shanker:~06:24]
#nscd -g
nscd configuration:
1 server debug level
3m 55s server runtime
4 current number of threads
32 maximum number of threads
0 number of
times
clients had to wait
no paranoia mode enabled
3600 restart internal
5 reload count
passwd
cache:
no cache is enabled
yes
cache is persistent
yes
cache is shared
0 suggested size
0 total data pool size
0 used data pool size
600 seconds
time
to live
for
positive entries
20 seconds
time
to live
for
negative entries
0 cache hits on positive entries
0 cache hits on negative entries
0 cache misses on positive entries
0 cache misses on negative entries
0% cache hit rate
0 current number of cached values
0 maximum number of cached values
0 maximum chain length searched
0 number of delays on rdlock
0 number of delays on wrlock
0 memory allocations failed
yes
check
/etc/passwd
for
changes
group cache:
no cache is enabled
yes
cache is persistent
yes
cache is shared
0 suggested size
0 total data pool size
0 used data pool size
3600 seconds
time
to live
for
positive entries
60 seconds
time
to live
for
negative entries
0 cache hits on positive entries
0 cache hits on negative entries
0 cache misses on positive entries
0 cache misses on negative entries
0% cache hit rate
0 current number of cached values
0 maximum number of cached values
0 maximum chain length searched
0 number of delays on rdlock
0 number of delays on wrlock
0 memory allocations failed
yes
check
/etc/group
for
changes
hosts cache:
yes
cache is enabled
yes
cache is persistent
yes
cache is shared
211 suggested size
216064 total data pool size
120 used data pool size
3600 seconds
time
to live
for
positive entries
20 seconds
time
to live
for
negative entries
361 cache hits on positive entries
0 cache hits on negative entries
239 cache misses on positive entries
151 cache misses on negative entries
48% cache hit rate
1 current number of cached values
80 maximum number of cached values
2 maximum chain length searched
0 number of delays on rdlock
0 number of delays on wrlock
0 memory allocations failed
yes
check
/etc/hosts
for
changes
services cache:
yes
cache is enabled
yes
cache is persistent
yes
cache is shared
211 suggested size
216064 total data pool size
416 used data pool size
28800 seconds
time
to live
for
positive entries
20 seconds
time
to live
for
negative entries
0 cache hits on positive entries
0 cache hits on negative entries
3 cache misses on positive entries
1 cache misses on negative entries
0% cache hit rate
3 current number of cached values
3 maximum number of cached values
1 maximum chain length searched
0 number of delays on rdlock
0 number of delays on wrlock
0 memory allocations failed
yes
check
/etc/services
for
changes
netgroup cache:
no cache is enabled
yes
cache is persistent
yes
cache is shared
0 suggested size
0 total data pool size
0 used data pool size
28800 seconds
time
to live
for
positive entries
20 seconds
time
to live
for
negative entries
0 cache hits on positive entries
0 cache hits on negative entries
0 cache misses on positive entries
0 cache misses on negative entries
0% cache hit rate
0 current number of cached values
0 maximum number of cached values
0 maximum chain length searched
0 number of delays on rdlock
0 number of delays on wrlock
0 memory allocations failed
yes
check
/etc/netgroup
for
changes
|
然后nscd的缓存文件是存放在/var/cache/nscd 目录下，也可以使用lsof 查看。当我想看看dnsmasq的缓存效果的时候却发现他缓存的文件都是socket,所以我判断他的缓存类似于varnish 都存在于内存中的。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
[root@shanker:~06:25]
#cd /proc/`pidof dnsmasq`/
[root@shanker:
/proc/2662406
:26]
#cd fd
[root@shanker:
/proc/26624/fd06
:26]
#ll
total 0
lrwx------ 1 root root 64 Jan 14 12:59 9 -> socket:[211546]
lrwx------ 1 root root 64 Jan 14 12:59 8 -> socket:[211545]
lrwx------ 1 root root 64 Jan 14 12:59 7 -> socket:[211544]
lrwx------ 1 root root 64 Jan 14 12:59 6 -> socket:[211543]
lrwx------ 1 root root 64 Jan 14 12:59 5 -> socket:[211542]
lrwx------ 1 root root 64 Jan 14 12:59 4 -> socket:[211541]
lrwx------ 1 root root 64 Jan 14 12:59 3 -> socket:[211539]
lrwx------ 1 root root 64 Jan 14 12:59 2 ->
/dev/null
l-wx------ 1 root root 64 Jan 14 12:59 14 ->
/var/log/dnsmasq
.log
l-wx------ 1 root root 64 Jan 14 12:59 13 -> pipe:[211553]
lr-x------ 1 root root 64 Jan 14 12:59 12 -> pipe:[211553]
lrwx------ 1 root root 64 Jan 14 12:59 11 -> socket:[211548]
lrwx------ 1 root root 64 Jan 14 12:59 10 -> socket:[211547]
lrwx------ 1 root root 64 Jan 14 12:59 1 ->
/dev/null
lrwx------ 1 root root 64 Jan 14 12:59 0 ->
/dev/null
|
然后我们测算一下dnsmasq 查询的效率如何，在bind那套工具中有一套DNS性能测试的工具，去官网下载最新的bind
|
1
|
wget https:
//www
.isc.org
/downloads/file/bind-9-10-3-p2/
?version=
tar
-gz
|
解压完后到queryperf目录
|
1
2
3
4
|
cd
bind-9.10.3-P2
/contrib/queryperf/
.
/configure
make
cp
queryperf
/usr/bin/
|
然后perftcpdns也是同样,make 后将perftcpdns cp 到/usr/bin下面。
一 queryperf使用格式
queryperf [-d datafile] [-s server_addr] [-p port] [-q num_queries]
-d: 后面接上一个文件，文件的内容是用户对DNS的请求，一行为一条请求，所以为了测试，我们可以在里面写上几千几万条。
-s: DNS服务器地址
-p: DNS服务器端口
-q: 请求多少次
我在网上随便找了一些域名地址，保存到dnsrecord里。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
ns.bta.net.cn A
ns.spt.net.cn A
ns.cn.net A
gjjline.bta.net.cn A
linedns.bta.net.cn A
ns.guangzhou.gd.cn A
dns.guangzhou.gd.cn A
ns.sta.net.cn A
ns-pd.online.sh.cn A
ns.wuhan.net.cn A
ns1.hbwhptt.net.cn A
dns.zj.cninfo.net A
ns.wuhan.net.cn A
ns.zjnbptt.net.cn A
ns.snnic.com A
ns1.xaonline.com A
ns.tpt.net.cn A
ns.dcb.
ln
.cn A
ns.lnpta.net.cn A
dns.dl.lnpta.net.cn A
|
然后复制成9999行，首先用本机来测试
|
1
|
queryperf -d query.txt -s 127.0.0.1
|
结果如下
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
Statistics:
Parse input
file
: once
Ended due to: reaching end of
file
Queries sent: 9999 queries
Queries completed: 9999 queries
Queries lost: 0 queries
Queries delayed(?): 0 queries
RTT max: 0.329734 sec
RTT min: 0.000003 sec
RTT average: 0.002490 sec
RTT std deviation: 0.011374 sec
RTT out of range: 0 queries
Percentage completed: 100.00%
Percentage lost: 0.00%
Started at: Thu Jan 14 12:02:52 2016
Finished at: Thu Jan 14 12:03:18 2016
Ran
for
: 26.355231 seconds
Queries per second: 379.393374 qps
|
显示总共执行了9999次查询，每秒钟的查询是379.393374个，然后用Google的公告域名服务器测试
|
1
|
queryperf -d query.txt -s 8.8.8.8
|
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
Statistics:
Parse input
file
: once
Ended due to: reaching end of
file
Queries sent: 9999 queries
Queries completed: 9999 queries
Queries lost: 0 queries
Queries delayed(?): 0 queries
RTT max: 0.262356 sec
RTT min: 0.001734 sec
RTT average: 0.005838 sec
RTT std deviation: 0.020113 sec
RTT out of range: 0 queries
Percentage completed: 100.00%
Percentage lost: 0.00%
Started at: Thu Jan 14 12:04:22 2016
Finished at: Thu Jan 14 12:16:18 2016
Ran
for
: 715.830576 seconds
|
Queries per second: 13.968389 qps
结果每秒钟只有可怜的13.96次查询，也就是说如果本机做为一台服务器，没有dnsmasq做缓存的情况下，DNS查询效率低的可怜。
二 perftcpdns的使用
Google了半天没找到有价值的信息，还是看软件自己的使用帮助把：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
|
[root@shanker:~
/bind-9
.10.3-P2
/contrib/perftcpdns06
:56]
#perftcpdns
server is required
perftcpdns [-huvX0] [-4|-6] [-r<rate>] [-t<report>] [-p<
test
-period>]
[-n<num-request>]* [-d<lost-
time
>]* [-D<max-loss>]* [-T<template-
file
>]
[-l<
local
-addr>] [-L<
local
-port>]* [-a<aggressiveness>] [-s<seed>]
[-M<memory>] [-x<diagnostic-selector>] [-P<port>] server
The server argument is the name
/address
of the DNS server to contact.
Options:
-0: Add EDNS0 option with DO flag.
-4: TCP
/IPv4
operation (default). This is incompatible with the -6 option.
-6: TCP
/IPv6
operation. This is incompatible with the -4 option.
-a<aggressiveness>: When the target sending rate is not yet reached,
control how many connections are initiated before the next pause.
-d<lost-
time
>: Specify the
time
after
which
a connection or a query is
treated as having been lost. The value is given
in
seconds and
may contain a fractional component. The default is 1 second.
-h: Print this help.
-l<
local
-addr>: Specify the
local
hostname
/address
to use when
communicating with the server.
-L<
local
-port>: Specify the (minimal and maximal)
local
port number
-M<memory>: Size of the tables (default 60000)
-P<port>: Specify an alternate (i.e., not 53) port
-r<rate>: Initiate <rate> TCP DNS connections per second. A periodic
report is generated showing the number of exchanges
which
were not
completed, as well as the average response latency. The program
continues
until
interrupted, at
which
point a final report is
generated.
-s<seed>: Specify the seed
for
randomization, making it repeatable.
-t<report>: Delay
in
seconds between two periodic reports.
-T<template-
file
>: The name of a
file
containing the template to use
as a stream of hexadecimal digits.
-u: Use UDP
in
place of TCP.
-
v
: Report the version number of this program.
-X: change default template to get NXDOMAIN responses.
-x<diagnostic-selector>: Include extended diagnostics
in
the output.
<diagnostic-selector> is a string of single-keywords specifying
the operations
for
which
verbose output is desired. The selector
keyletters are:
*
'a'
: print the decoded
command
line arguments
*
'e'
: print the
exit
reason
*
'i'
: print rate processing details
*
'T'
: when finished, print templates
Stopping conditions:
-D<max-loss>: Abort the
test
if
more
than <max-loss> connections or
queries have been lost. If <max-loss> includes the suffix
'%'
, it
specifies a maximum percentage of losses before stopping.
In this
case
, testing of the threshold begins after 10
connections
/responses
have been expected to be accepted
/received
.
-n<num-request>: Initiate <num-request> transactions. No report is
generated
until
all transactions have been initiated
/waited-for
,
after
which
a report is generated and the program terminates.
-p<
test
-period>: Send requests
for
the given
test
period,
which
is
specified
in
the same manner as -d. This can be used as an
alternative to -n, or both options can be given,
in
which
case
the
testing is completed when either limit is reached.
Errors:
- locallimit: reached to
local
system limits when sending a message.
- badconn: connection failed (from getsockopt(SO_ERROR))
- collconn: connect() timed out
- badsent: send() failed
- callsent: timed out waiting from a response
- recverr: recv() system call failed
- tooshort: received a too short message
- badid: the
id
mismatches between the query and the response
- notresp: doesn't receive a response
Rate stats:
- loops: number of thread loop iterations
- shortwait: no direct activity
in
a thread iteration
- compconn: computed number of connect() calls
- lateconn: connect() already dued when computing delay to the next one
Exit status:
The
exit
status is:
0 on complete success.
1
for
a general error.
2
if
an error is found
in
the
command
line arguments.
3
if
there are no general failures
in
operation, but one or
more
exchanges are not successfully completed.
|
用这个命令的前提是你的服务器是真正的DNS 服务器而不是缓存服务器，
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
perftcpdns -x aeiT -r 40000 8.8.8.8
connect: 117669, sent: 117667, received: 60393
embryonics: 2 (0.0%)
drops: 57274 (48.7%)
total losses: 57276 (48.7%)
local
limits: 0, bad connects: 0, connect timeouts: 2
bad sends: 0, bad recvs: 0, recv timeouts: 57090
too shorts: 13, bad IDs: 0, not responses: 0
rcode counters:
noerror: 1572, formerr: 0, servfail: 58821
nxdomain: 0, noimp: 0, refused: 0, others: 0
rates: 241,241,124 (target 250)
loops: 436434,117670,474085,462908
shortwait: 0,356509,402606
compconn: 117669, lateconn: 1
badconn: 0, collconn: 2, recverr: 0, collsent: 57090
memory: used(246) / allocated(60000)
RTT: min
/avg/max/stddev
: 1.798
/9
.197
/523
.429
/22
.042 ms
length = 0x1c
content:
00 00 01 00 00 01 00 00 00 00 00 00 05 69 63 61
6e 6e 04 6c 69 6e 6b 00 00 01 00 01
|
当我使用一下参数进行测试的时候发，系统提示too many files are open,ulimit 把打开文件数改成65535在继续测试，过一会发现之前打开的ssh窗口莫名的关闭，而且敲命名反应很卡，强制关掉perftcpdns，查看日志发现，很熟悉的一幕出现了：
|
1
2
3
4
5
6
|
Jan 15 03:00:59 shanker kernel: [223099.654142] nf_conntrack: table full, dropping packet
Jan 15 03:00:59 shanker kernel: [223099.654144] nf_conntrack: table full, dropping packet
Jan 15 03:01:05 shanker kernel: [223104.992102] net_ratelimit: 29728 callbacks suppressed
Jan 15 03:01:05 shanker kernel: [223104.992128] nf_conntrack: table full, dropping packet
Jan 15 03:01:05 shanker kernel: [223104.992135] nf_conntrack: table full, dropping packet
Jan 15 03:01:05 shanker kernel: [223105.152351] nf_conntrack: table full, dropping packet
|
想起以前处理过的紧急case，nf_conntrack table full，最直接的处理方式就是加大conntrack table 和 bucket的值：
net.netfilter.nf_conntrack_max = 196608
net.netfilter.nf_conntrack_buckets = 65534
如果你的服务器内存足够大而且只处理内网请求，扩大这个数值是理所当然的。Google了一下，有其他的治根的法式是用iptable raw 表，跳过记录，如果不依赖于iptables的话，我觉得更改内核参数的方式也是可行的。
参考文章：http://shanker.blog.51cto.com/1189689/1735341
