cloudstack下libvirtd服务无响应问题

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

在cloudstack4.5.2版本下，偶尔出现libvirtd服务无响应的情况，导致virsh命令无法使用，同时伴随cloudstack master丢失该slave主机连接的情况。怀疑是libvirtd服务或版本的问题，但是在官网上并没有找到类似的bug提交，该问题可能还存在于更高的版本，需要时间进一步从根本上分析。下面是该问题的处理过程，在此记录下，关注和使用cloudstack的朋友可以参考。

众所周知，cloudstack的社区热度远不如openstack，为什么还要选择clcoudstack？这个问题以后有机会再和大家聊。言归正传。

环境交代

宿主机操作系统：centos6.5x64(2.6.32-431.el6.x86_64)
cloudstack版本：4.5.2
libvirt版本：libvirt-0.10.2-54.el6_7.2.x86_64

问题描述

通过cloudstackapi listHosts报警信息显示:
node5.cloud.rtmap:192.168.14.20 state is Down at 2016-05-13T07:19:04+0800
#有关cloudstackapi的使用方法在其它文章中总结,不在此处说明。

登陆问题宿主服务器检查：
[root@node5 log]#virsh list --all
没有响应ctrl^c退出
这时的vm可以正常工作，但处于失控状态

尝试重启启动libvirtd服务：
[root@node5 log]# service libvirtd stop
正在关闭 libvirtd 守护进程： [失败]#无法关闭libvirtd服务

尝试重启启动cloudstack-agent服务：
[root@node5 libvirt]# service cloudstack-agent restart
Stopping Cloud Agent:
Starting Cloud Agent:
libvirtd故障依旧

简单维护

[root@node5 ping]# libvirtd -d -l --config /etc/libvirt/libvirtd.conf
libvirtd：错误：Unable to initialize network sockets。查看 /var/log/messages 或者运行不带 --daemon 的命令查看更多信息。

[root@node5 log]# libvirtd -d
可以执行成功，这时执行virsh list --all 可以查看和操作vm

[root@node5 log]#virsh list --all
Id 名称状态
----------------------------------------------------
2 i-4-185-VM running

虽然vm运行正常，现在也可以通过命令正常管理了。但是对于cloudstack平台而言，宿主机处于down状态，vm处于失控状态。

临时解决办法是在其它大的升级和维护过程中重启服务器解决，根本解决还要具体问题具体分析。

分析与排查

检查进程

[root@node5 log]# ps ax |grep libvirtd
? R : libvirtd --daemon -l#该服务始终处于run状态[root@node5 log]# top -p
top -p
top - :: up days, :, user, load average: 3.05, 5.07, 6.64
Tasks: total, running, sleeping, stopped, zombie
Cpu(s): 4.8%us, 1.4%sy, 0.0%ni, 93.1%id, 0.6%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 264420148k total, 182040780k used, 82379368k free, 834232k buffers
Swap: 8388600k total, 92k used, 8388508k free, 100453708k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
root 984m 12m R 100.2 0.0 :22.68 libvirtd #cpu占用100%，无法释放，影响系统稳定性

杀进程

[root@node5 log]# kill -
[root@node5 log]# kill -
[root@master log]# ps ax |grep libvirtd#检查进程依然存在
? R : libvirtd --daemon -l
[root@node5 ~]# libvirtd -d -l --config /etc/libvirt/libvirtd.conf
libvirtd：错误：Unable to initialize network sockets。查看 /var/log/messages 或者运行不带 --daemon 的命令查看更多信息。
[root@node5 ~]# netstat -antp |grep
tcp 0.0.0.0: 0.0.0.0:* LISTEN /libvirtd
tcp 192.168.14.25: 192.168.14.22: CLOSE_WAIT -
tcp 192.168.14.25: 192.168.14.20: CLOSE_WAIT -
tcp 192.168.14.25: 192.168.14.10: CLOSE_WAIT -
tcp ::: :::* LISTEN /libvirtd
tcp ::: ::: CLOSE_WAIT -

经过上述操作，初步判断libvirtd陷入了hang死状态。

追踪进程

[root@node5 log]#strace -f libvirtd#后来发现这样strace是不对的，正确的方法是strace -f -p 针对pid进行strace
[pid ] close() = - EBADF (Bad file descriptor)
[pid ] close() = - EBADF (Bad file descriptor)
[pid ] close() = - EBADF (Bad file descriptor)
[pid ] close() = - EBADF (Bad file descriptor)
[pid ] close() = - EBADF (Bad file descriptor)
[pid ] close() = - EBADF (Bad file descriptor)
[pid ] close() = - EBADF (Bad file descriptor)
[pid ] close() = - EBADF (Bad file descriptor)
[pid ] close() = - EBADF (Bad file descriptor)
[pid ] close() = - EBADF (Bad file descriptor)
[pid ] close() = - EBADF (Bad file descriptor)
[pid ] close() = - EBADF (Bad file descriptor)
[pid ] close() = - EBADF (Bad file descriptor)
[pid ] close() = - EBADF (Bad file descriptor)
^C[pid ] close( <unfinished ...>
Process detached
Process detached
Process detached
Process detached
Process detached
Process detached
Process detached
Process detached
Process detached
Process detached
Process detached
Process detached

父进程6485在不断的产生和关闭子进程，并返回错误信息。Bad file descriptor的原因（如何触发的，谁触发的）？循环为何无法退出？问题如何再现？

获得更多的线索
官方文档（libvirtd各种故障诊断记录和解决办法非常详尽）
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/sect-Troubleshooting-Common_libvirt_errors_and_troubleshooting.html#sect-libvirtd_failed_to_start

开启系统日志

Change libvirt's logging in /etc/libvirt/libvirtd.conf by enabling the line below. To enable the setting the line, open the /etc/libvirt/libvirtd.conf file in a text editor, remove the hash (or #) symbol from the beginning of the following line, and save the change:

log_outputs="3:syslog:libvirtd"

参照配置，重启服务器等待下次故障观察日志

......Jun :: node5 abrtd: New client connected
Jun :: node5 abrtd: Directory 'pyhook-2016-06-01-12:42:26-70065' creation detected
Jun :: node5 abrt-server[]: Saved Python crash dump of pid to /var/spool/abrt/pyhook----::-
Jun :: node5 abrtd: Package 'cloudstack-common' isn't signed with proper key
Jun :: node5 abrtd: 'post-create' on '/var/spool/abrt/pyhook-2016-06-01-12:42:26-70065' exited with
Jun :: node5 abrtd: Deleting problem directory '/var/spool/abrt/pyhook-2016-06-01-12:42:26-70065'
Jun :: node5 abrt: detected unhandled Python exception in '/usr/share/cloudstack-common/scripts/vm/network/security_group.py'
......Jun :: node5 libvirtd: : warning : qemuDomainObjBeginJobInternal: : Cannot start job (modify, none) for domain i---VM; current job is (modify, none) owned by (, )
Jun :: node5 libvirtd: : error : qemuDomainObjBeginJobInternal: : Timed out during operation: cannot acquire state change lock
Jun :: node5 libvirtd: : info : libvirt version: 0.10., package: .el6_7. (CentOS BuildSystem <http://bugs.centos.org>, 2015-11-10-10:25:08, c6b9.bsys.dev.centos.org)
Jun :: node5 libvirtd: : error : virNetSocketNewListenTCP: : Unable to bind to port: 地址已在使用
Jun :: node5 libvirtd: : info : libvirt version: 0.10., package: .el6_7. (CentOS BuildSystem <http://bugs.centos.org>, 2015-11-10-10:25:08, c6b9.bsys.dev.centos.org)
Jun :: node5 libvirtd: : error : virNetSocketNewListenTCP: : Unable to bind to port: 地址已在使用
Jun :: node5 libvirtd: : info : libvirt version: 0.10., package: .el6_7. (CentOS BuildSystem <http://bugs.centos.org>, 2015-11-10-10:25:08, c6b9.bsys.dev.centos.org)
Jun :: node5 libvirtd: : error : virNetSocketNewListenTCP: : Unable to bind to port: 地址已在使用
Jun :: node5 libvirtd: : info : libvirt version: 0.10., package: .el6_7. (CentOS BuildSystem <http://bugs.centos.org>, 2015-11-10-10:25:08, c6b9.bsys.dev.centos.org)
Jun :: node5 libvirtd: : error : virNetSocketNewListenTCP: : Unable to bind to port: 地址已在使用
Jun :: node5 libvirtd: : info : libvirt version: 0.10., package: .el6_7. (CentOS BuildSystem <http://bugs.centos.org>, 2015-11-10-10:25:08, c6b9.bsys.dev.centos.org)
Jun :: node5 libvirtd: : error : virNetSocketNewListenTCP: : Unable to bind to port: 地址已在使用
......Jun :: node5 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jun :: node5 libvirtd: : info : libvirt version: 0.10., package: .el6_7. (CentOS BuildSystem <http://bugs.centos.org>, 2015-11-10-10:25:08, c6b9.bsys.dev.centos.org)
Jun :: node5 libvirtd: : error : virPidFileAcquirePath: : Failed to acquire pid file '/var/run/libvirtd.pid': 资源暂时不可用

并未获得致命错误和更多线索。（该日志配置选项还是很有必要打开的，很多问题都可以通过它来定位)

解决过程

解决思路

尝试和找到终止进程、重启服务的方法
提交bug，等待补丁升级
分析源代码，再现问题，解决问题（投入研发和时间）

由于不能再现问题，还是从简入繁吧。触发这些子进程的元凶是谁？还是cloudstack-agent的嫌疑最大，但之前重启过该服务并没有解决问题，那么agent服务是怎么一回事呢？

看下启动脚本可以基本了解，

[root@node5 libvirt]# cat /etc/rc.d/init.d/cloudstack-agent

#!/bin/bash# chkconfig:
# description: Cloud Agent# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.# WARNING: if this script is changed, then all other initscripts MUST BE changed to match it as well. /etc/rc.d/init.d/functions# set environment variablesSHORTNAME=$(basename $ | sed -e 's/^[SK][0-9][0-9]//')
PIDFILE=/var/run/"$SHORTNAME".pid
LOCKFILE=/var/lock/subsys/"$SHORTNAME"
LOGDIR=/var/log/cloudstack/agent
LOGFILE=${LOGDIR}/agent.log
PROGNAME="Cloud Agent"
CLASS="com.cloud.agent.AgentShell"
JSVC=`which jsvc >/dev/null`;# exit if we don't find jsvc
if [ -z "$JSVC" ]; then
echo no jsvc found in path;
exit ;
fiunset OPTIONS
[ -r /etc/sysconfig/"$SHORTNAME" ] && source /etc/sysconfig/"$SHORTNAME"# The first existing directory is used for JAVA_HOME (if JAVA_HOME is not defined in $DEFAULT)
JDK_DIRS="/usr/lib/jvm/jre /usr/lib/jvm/java-7-openjdk /usr/lib/jvm/java-7-openjdk-i386 /usr/lib/jvm/java-7-openjdk-amd64 /usr/lib/jvm/java-6-openjdk /usr/lib/jvm/java-6-openjdk-i386 /usr/lib/jvm/java-6-openjdk-amd64 /usr/lib/jvm/java-6-sun"for jdir in $JDK_DIRS; do
if [ -r "$jdir/bin/java" -a -z "${JAVA_HOME}" ]; then
JAVA_HOME="$jdir"
fi
done
export JAVA_HOMEACP=`ls /usr/share/cloudstack-agent/lib/*.jar | tr '\n' ':' | sed s'/.$//'`
PCP=`ls /usr/share/cloudstack-agent/plugins/*.jar 2>/dev/null | tr '\n' ':' | sed s'/.$//'`# We need to append the JSVC daemon JAR to the classpath
# AgentShell implements the JSVC daemon methods
export CLASSPATH="/usr/share/java/commons-daemon.jar:$ACP:$PCP:/etc/cloudstack/agent:/usr/share/cloudstack-common/scripts"start() {
echo -n $"Starting $PROGNAME: "
if hostname --fqdn >/dev/null 2>&1 ; then
$JSVC -Xms256m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
-errfile $LOGDIR/cloudstack-agent.err -outfile $LOGDIR/cloudstack-agent.out $CLASS
RETVAL=$?
echo
else
failure
echo
echo The host name does not resolve properly to an IP address. Cannot start "$PROGNAME". > /dev/stderr
RETVAL=9
fi
[ $RETVAL = 0 ] && touch ${LOCKFILE}
return $RETVAL
}stop() {
echo -n $"Stopping $PROGNAME: "
$JSVC -pidfile "$PIDFILE" -stop $CLASS
RETVAL=$?
echo
[ $RETVAL = 0 ] && rm -f ${LOCKFILE} ${PIDFILE}
}case "$1" in
start)
start
;;
stop)
stop
;;
status)
status -p ${PIDFILE} $SHORTNAME
RETVAL=$?
;;
restart)
stop
sleep 3
start
;;
condrestart)
if status -p ${PIDFILE} $SHORTNAME >&/dev/null; then
stop
sleep 3
start
fi
;;
*)
echo $"Usage: $SHORTNAME {start|stop|restart|condrestart|status|help}"
RETVAL=3
esacexit $RETVAL

[root@node5 libvirt]# ps ax |grep jsvc.exec

[root@node5 libvirt]# ps ax |grep jsvc.exec
? Ss : jsvc.exec -Xms256m -Xmx2048m -cp /usr/share/java/commons-daemon.jar:/usr/share/cloudstack-agent/lib/activation-1.1.jar:/usr/share/cloudstack-agent/lib/antisamy-1.4..jar:/usr/share/cloudstack-agent/lib/aopalliance-1.0.jar:/usr/share/cloudstack-agent/lib/apache-log4j-extras-1.1.jar:/usr/share/cloudstack-agent/lib/aspectjweaver-1.7..jar:/usr/share/cloudstack-agent/lib/aws-java-sdk-1.3..jar:/usr/share/cloudstack-agent/lib/batik-css-1.7.jar:/usr/share/cloudstack-agent/lib/batik-ext-1.7.jar:/usr/share/cloudstack-agent/lib/batik-util-1.7.jar:/usr/share/cloudstack-agent/lib/bcprov-jdk15-1.46.jar:/usr/share/cloudstack-agent/lib/bcprov-jdk16-1.46.jar:/usr/share/cloudstack-agent/lib/bsh-core-.0b4.jar:/usr/share/cloudstack-agent/lib/cglib-nodep-2.2..jar:/usr/share/cloudstack-agent/lib/cloud-agent-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-api-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-core-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-engine-api-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-engine-components-api-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-engine-schema-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-cluster-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-config-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-db-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-events-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-ipc-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-jobs-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-managed-context-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-rest-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-security-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-plugin-hypervisor-kvm-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-plugin-network-ovs-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-server-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-utils-4.5..jar:/usr/share/cloudstack-agent/lib/commons-beanutils-core-1.7..jar:/usr/share/cloudstack-agent/lib/commons-codec-1.6.jar:/usr/share/cloudstack-agent/lib/commons-collections-3.2..jar:/usr/share/cloudstack-agent/lib/commons-configuration-1.8.jar:/usr/share/cloudstack-agent/lib/commons-daemon-1.0..jar:/usr/share/cloudstack-agent/lib/commons-dbcp-1.4.jar:/usr/share/cloudstack-agent/lib/commons-fileupload-1.2.jar:/usr/share/cloudstack-agent/lib/commons-httpclient-3.1.jar:/usr/share/cloudstack-agent/lib/commons-io-1.4.jar:/usr/share/cloudstack-agent/lib/commons-lang-2.6.jar:/usr/share/cloudstack-agent/lib/commons-logging-1.1..jar:/usr/share/cloudstack-agent/lib/commons-net-3.3.jar:/usr/share/cloudstack-agent/lib/commons-pool-1.6.jar:/usr/share/cloudstack-agent/lib/cxf-bundle-jaxrs-2.7..jar:/usr/share/cloudstack-agent/lib/dom4j-1.6..jar:/usr/share/cloudstack-agent/lib/ehcache-core-2.6..jar:/usr/share/cloudstack-agent/lib/ejb-api-3.0.jar:/usr/share/cloudstack-agent/lib/esapi-2.0..jar:/usr/share/cloudstack-agent/lib/geronimo-javamail_1.4_spec-1.7..jar:/usr/share/cloudstack-agent/lib/geronimo-servlet_3.0_spec-1.0.jar:/usr/share/cloudstack-agent/lib/gson-1.7..jar:/usr/share/cloudstack-agent/lib/guava-14.0-rc1.jar:/usr/share/cloudstack-agent/lib/httpclient-4.3..jar:/usr/share/cloudstack-agent/lib/httpcore-4.3..jar:/usr/share/cloudstack-agent/lib/jackson-annotations-2.1..jar:/usr/share/cloudstack-agent/lib/jackson-core-2.1..jar:/usr/share/cloudstack-agent/lib/jackson-core-asl-1.8..jar:/usr/share/cloudstack-agent/lib/jackson-databind-2.1..jar:/usr/share/cloudstack-agent/lib/jackson-jaxrs-json-provider-2.1..jar:/usr/share/cloudstack-agent/lib/jackson-mapper-asl-1.8..jar:/usr/share/cloudstack-agent/lib/jackson-module-jaxb-annotations-2.1..jar:/usr/share/cloudstack-agent/lib/jasypt-1.9..jar:/usr/share/cloudstack-agent/lib/java-ipv6-0.10.jar:/usr/share/cloudstack-agent/lib/javassist-3.12..GA.jar:/usr/share/cloudstack-agent/lib/javassist-3.18.-GA.jar:/usr/share/cloudstack-agent/lib/javax.inject-.jar:/usr/share/cloudstack-agent/lib/javax.persistence-2.0..jar:/usr/share/cloudstack-agent/lib/javax.ws.rs-api-2.0-m10.jar
? Sl : jsvc.exec -Xms256m -Xmx2048m -cp /usr/share/java/commons-daemon.jar:/usr/share/cloudstack-agent/lib/activation-1.1.jar:/usr/share/cloudstack-agent/lib/antisamy-1.4..jar:/usr/share/cloudstack-agent/lib/aopalliance-1.0.jar:/usr/share/cloudstack-agent/lib/apache-log4j-extras-1.1.jar:/usr/share/cloudstack-agent/lib/aspectjweaver-1.7..jar:/usr/share/cloudstack-agent/lib/aws-java-sdk-1.3..jar:/usr/share/cloudstack-agent/lib/batik-css-1.7.jar:/usr/share/cloudstack-agent/lib/batik-ext-1.7.jar:/usr/share/cloudstack-agent/lib/batik-util-1.7.jar:/usr/share/cloudstack-agent/lib/bcprov-jdk15-1.46.jar:/usr/share/cloudstack-agent/lib/bcprov-jdk16-1.46.jar:/usr/share/cloudstack-agent/lib/bsh-core-.0b4.jar:/usr/share/cloudstack-agent/lib/cglib-nodep-2.2..jar:/usr/share/cloudstack-agent/lib/cloud-agent-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-api-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-core-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-engine-api-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-engine-components-api-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-engine-schema-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-cluster-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-config-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-db-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-events-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-ipc-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-jobs-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-managed-context-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-rest-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-framework-security-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-plugin-hypervisor-kvm-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-plugin-network-ovs-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-server-4.5..jar:/usr/share/cloudstack-agent/lib/cloud-utils-4.5..jar:/usr/share/cloudstack-agent/lib/commons-beanutils-core-1.7..jar:/usr/share/cloudstack-agent/lib/commons-codec-1.6.jar:/usr/share/cloudstack-agent/lib/commons-collections-3.2..jar:/usr/share/cloudstack-agent/lib/commons-configuration-1.8.jar:/usr/share/cloudstack-agent/lib/commons-daemon-1.0..jar:/usr/share/cloudstack-agent/lib/commons-dbcp-1.4.jar:/usr/share/cloudstack-agent/lib/commons-fileupload-1.2.jar:/usr/share/cloudstack-agent/lib/commons-httpclient-3.1.jar:/usr/share/cloudstack-agent/lib/commons-io-1.4.jar:/usr/share/cloudstack-agent/lib/commons-lang-2.6.jar:/usr/share/cloudstack-agent/lib/commons-logging-1.1..jar:/usr/share/cloudstack-agent/lib/commons-net-3.3.jar:/usr/share/cloudstack-agent/lib/commons-pool-1.6.jar:/usr/share/cloudstack-agent/lib/cxf-bundle-jaxrs-2.7..jar:/usr/share/cloudstack-agent/lib/dom4j-1.6..jar:/usr/share/cloudstack-agent/lib/ehcache-core-2.6..jar:/usr/share/cloudstack-agent/lib/ejb-api-3.0.jar:/usr/share/cloudstack-agent/lib/esapi-2.0..jar:/usr/share/cloudstack-agent/lib/geronimo-javamail_1.4_spec-1.7..jar:/usr/share/cloudstack-agent/lib/geronimo-servlet_3.0_spec-1.0.jar:/usr/share/cloudstack-agent/lib/gson-1.7..jar:/usr/share/cloudstack-agent/lib/guava-14.0-rc1.jar:/usr/share/cloudstack-agent/lib/httpclient-4.3..jar:/usr/share/cloudstack-agent/lib/httpcore-4.3..jar:/usr/share/cloudstack-agent/lib/jackson-annotations-2.1..jar:/usr/share/cloudstack-agent/lib/jackson-core-2.1..jar:/usr/share/cloudstack-agent/lib/jackson-core-asl-1.8..jar:/usr/share/cloudstack-agent/lib/jackson-databind-2.1..jar:/usr/share/cloudstack-agent/lib/jackson-jaxrs-json-provider-2.1..jar:/usr/share/cloudstack-agent/lib/jackson-mapper-asl-1.8..jar:/usr/share/cloudstack-agent/lib/jackson-module-jaxb-annotations-2.1..jar:/usr/share/cloudstack-agent/lib/jasypt-1.9..jar:/usr/share/cloudstack-agent/lib/java-ipv6-0.10.jar:/usr/share/cloudstack-agent/lib/javassist-3.12..GA.jar:/usr/share/cloudstack-agent/lib/javassist-3.18.-GA.jar:/usr/share/cloudstack-agent/lib/javax.inject-.jar:/usr/share/cloudstack-agent/lib/javax.persistence-2.0..jar:/usr/share/cloudstack-agent/lib/javax.ws.rs-api-2.0-m10.jar

重启服务

[root@node5 bin]# service cloudstack-agent status
cloudstack-agent (pid ) 正在运行...

[root@node5 bin]# service cloudstack-agent stop
Stopping Cloud Agent:[root@node5 bin]# service cloudstack-agent status
cloudstack-agent (pid 6657) 正在运行..

ps ax |grep jsvc.exec 也验证了进程依然存在

眼前一亮的同时，也发现了之前使用restart带来的问题，stop不成功的问题被掩盖了~~~有没有懊恼？不过来不及反思，接下来的问题还远不是这么简单......

[root@node5 bin]# kill -
[root@node5 bin]# kill -
-bash: kill: () - 没有那个进程
-bash: kill: () - 没有那个进程
[root@node5 bin]# service cloudstack-agent status
cloudstack-agent 已死，但 pid 文件仍存
[root@node5 bin]# rm /var/run/cloudstack-agent.pid
rm：是否删除普通文件 "/var/run/cloudstack-agent.pid"？y
[root@node5 bin]# service cloudstack-agent status
cloudstack-agent 已死，但是 subsys 被锁
[root@node5 bin]# service cloudstack-agent start
[root@node5 bin]# service cloudstack-agent status
cloudstack-agent (pid ) 正在运行...
[root@node5 bin]# netstat -antp |grep
tcp 192.168.14.20: 192.168.14.10: ESTABLISHED /jsvc.exec

处理后状态恢复正常，但是libvirtd仍然无法杀掉, 很快netstat -antp |grep 8250 状态再次消失，cloudstack master平台监控主机记录由Up状态转为disconnect状态。不过毕竟不是down状态，较之前已经有了进步。

启动一个libvirtd -d看下，

[root@node5 bin]# libvirtd -d
[root@node5 bin]# ps ax |grep libvirtd
? R : libvirtd --daemon -l
? Sl : libvirtd -d
pts/ S+ : grep libvirtd

然后在cloudstack master平台上手工点击强制重新连接该主机，成功了。主机监控状态由disconnect转为Up，这时再次尝试杀掉仍然是不成功的，于是又在cloudstack master管理平台上尝试着点击操作了一下暂停vm命令，vm成功暂停。再返回服务器上观察原来hung死的libvirtd进程已经消失。

[root@node5 bin]# libvirtd -d
[root@node5 bin]# ps ax |grep libvirtd
? Sl : libvirtd -d
pts/ S+ : grep libvirtd

至此既恢复了平台对该主机的管控，也终止了libvirtd异常进程。问题初步归于cloudstack-agent在处理发送个libvirtd的信号上存在些小问题。以后再单独分析下jsvc进程，再现问题和根本解决。