libvirtd using a lot of memory, virt-manager blank

So, when I run top on my machines, I’m starting to see libvirtd using a large amount of memory. I’ve been working through what can be done about that, and found a couple of interesting points.

Firstly, libvirtd is the management tools only.  You can restart it without impacting your running virtuals, and therefore part of the answer is to just restart it.  There are a number of reports on the web of memory leaks in libvirtd, many of them appear to have been fixed in various versions, I assume there is a leak in the version I’m running, 0.9.12.

Secondly, when I restart it, it kills off my virt-manager sessions.  After restarting libvirtd and starting up a new virt-manager, I was finding that my virt-manager was giving a totally blank page on one of my servers.  I eventually tracked this to the fact that there was a hung virt-manager running in the background, and new sessions were just connecting to it.  Killing that session and then restarting virt-manager fixed my issue.

Firstly, how do you know if you have this problem?  I look at the highest apps by resident memory usage on my server using top.

To see this, you run top from the command line.  You then press f to pick the fields to sort by, use the up/down keys to get to the field you want to sort (in my case, Res), then press s to sort by it, and q to return to top.

Picking the sort field looks something like this:

Fields Management for window 1:Def, whose current sort field is %CPU 
   Navigate with Up/Dn, Right selects for move then <Enter> or Left commits,
   'd' or <Space> toggles display, 's' sets sort.  Use 'q' or <Esc> to end! 

* PID     = Process Id          
* USER    = Effective User Name 
* PR      = Priority            
* NI      = Nice Value          
* VIRT    = Virtual Image (KiB) 
* RES     = Resident Size (KiB) 
* SHR     = Shared Memory (KiB) 
* S       = Process Status      
* %CPU    = CPU Usage           
* %MEM    = Memory Usage (RES)  
* TIME+   = CPU Time, hundredths
* COMMAND = Command Name/Line
.....

This should then give you a view something like the following:

top - 11:59:26 up 14 days, 14:10,  8 users,  load average: 0.62, 0.73, 0.77
Tasks: 264 total,   4 running, 259 sleeping,   0 stopped,   1 zombie
%Cpu(s):  7.5 us,  2.2 sy,  0.0 ni, 89.7 id,  0.4 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem:   8176968 total,  8031272 used,   145696 free,   181900 buffers
KiB Swap:  6291432 total,     7264 used,  6284168 free,  3941400 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                           
 6953 libvirt-  20   0 3421m 711m 2480 S  15.3  8.9   3548:00 kvm                                                                                                                                                                                               
 6907 libvirt-  20   0 3393m 524m 2468 S   0.3  6.6  63:03.05 kvm                                                                                                                                                                                               
 6928 libvirt-  20   0 3431m 457m 2480 S   3.7  5.7 745:01.80 kvm                                                                                                                                                                                               
 4334 root      20   0  252m 131m  21m S   2.7  1.6 115:47.04 Xorg                                                                                                                                                                                              
 6073 paul      20   0 3060m 104m  32m S   0.3  1.3   2:35.86 plasma-desktop                                                                                                                                                                                    
23420 root      20   0  450m  84m 5392 S   5.0  1.1  32:08.18 libvirtd                                                                                                                                                                                          
23458 paul      20   0  810m  65m  13m S   8.3  0.8  68:04.66 python                                                                                                                                                                                            
 6224 paul      39  19  293m  65m 2508 S   0.0  0.8   3:11.45 virtuoso-t

As you can see, in this case my top memory users are three virtual machines running in kvm.  I’m using 84m for libvirtd, but this was 40m overnight, so it’s growing by 40-100m per day.

Next, to restart libvirtd you would ideally first shut down all virt-managers, then issue

  /etc/init.d/libvirt-bin restart

Start up virt-manager again and check it works.  If you get the blank screen, then you can use

  killall virt-manager

Then start up virt-manager again and see if this fixes it.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s