Fast, Easy, Cheap: Pick One

Just some other blog about computers and programming

VMWare Linux Guest Clock Synchronization

One thing many home users of VMWare might not know or care much about is clock synchronization. However, in a networked environment, clock synchronization is an essential fact of life. In particular, some protocols such as Kerberos pretty much depend on your machines’ clocks being in sync. Additionally, if you want to have any kind of logging or network monitoring solution in place, the output of such things can be difficult if not impossible to interpret if your clocks are all over the map.

The typical solution for most standard configurations is to run an NTP daemon on every machine and a few time servers on your network which synchronize with an outside source such as an internet time server or something more sophisticated. I won’t go in to the details of how the NTP protocol works, but it’s important to keep in mind that it is intended to gradually correct for clock drift and not simply jump your clock to the correct time.

In most UNIX environments you will find two programs which can be used to synchronize your time with an NTP server: ntp and ntpdate. ntp is the one which performs gradual corrections, and can also optionally act as a server process for other hosts. ntpdate will simply reset your time to whatever response it receives from the time server. It’s intended for use during the boot process to set the initial time, or in the case that something goes horribly wrong with your clock (such as in a misconfigured VMWare guest…).

So how does this all fit in VMWare, we can just use ntp to keep the guest clocks in sync, right? Well, it’s not quite that simple. While I won’t go in to all the technical details here, because they’re described in this excellent VMWare paper, this method won’t work. Because of the way VMWare hands out clock interrupts to virtual machines, NTP can become extremely confused. Enter VMWare Tools.

As you may know, VMWare tools enables additional functionality such as video acceleration and time synchronization. So we can just turn on the time sync, right? Well, it’s not quite so easy. One thing many people don’t realize about the time sync feature of VMWare tools is that it will only catch up the clock if it’s running too slow. If it’s running too fast, the time sync feature does exactly nothing! Unfortunately, a fast clock is a more common case than a slow one when it comes to running a Linux virtual machine. I’m not going to go in to detail why, but if you want to know, read the paper above.

So what can you do to make your clock run on time? Well, the solution is actually not too difficult. Many people suggest recompiling the kernel of a virtual machine to run at a lower interrupt rate and so forth. That often works, but may not totally alleviate the problem. There’s another way.

If you look again at the above mentioned paper, you’ll notice that there are many timer sources available to Linux. The simplest is the PIT, or Programmable Interrupt Timer. There’s a bunch of other ones such as APIC and and LAPIC. These don’t appear to work reliably under VMWare. The solution is to turn them off and force PIT. This can be accomplished by some simple kernel boot flags. The relevant section from my Debian VM’s grub.conf:

title Debian GNU/Linux, kernel 2.6.22-3-686 root (hd0,0) kernel /vmlinuz-2.6.22-3-686 root=/dev/mapper/debian-root ro clocksource=pit nosmp noapic nolapic initrd /initrd.img-2.6.22-3-686

Note the kernel line. You need to append clocksource=pit noapic nolapic. If you’re not running an SMP VM, add nosmp as well (this may not be strictly necessary, but I don’t think it can hurt). After these changes, your kernel should boot up using the PIT clock source. Verify by taking a look at your dmesg output and grepping for “clock”.

Now your virtual machine’s clock will lag very slightly instead of running fast. Fortunately VMWare Tools (which you did install, right?) will run approximately every minute and synchronize it back to the right time. Make sure you don’t run NTP inside the VM, and make sure you do run it on the host.

EDIT (2008/02/26):
I forgot to mention in the original article that you need to enable time synchronization for your virtual machine. You can do this either through the GUI or by adding the line

tools.syncTime = true

To the .vmx file of your virtual machine.