Using the Symmetricom bc635PCIe card as an NTP timesource

INTRODUCTION

NTP timeservers are a crucial part of any large-scale, distributed system, especially if the components of the system are spread across multiple sites. There are many commercial and homebrew options for setting up an NTP timesource. Some are fairly simple to set up, and others (like the Symmetricom bc635 PCIe card) require a little more work.

I have recently set up an NTP server using the aforementioned card, and after all my troubles to get it set up, I decided to document the procedures used in this blog post.

PROBLEM

The default ntpd that comes with CentOS 6.2 does not have support for the BANCOMM/Symmetricom PCIe GPS card.

SETUP
Download the latest ntp source from http://www.ntp.org/downloads.html .For this tutorial, I used version 4.2.6p5

STEPS
There is a PDF from Symmetricom support that outlines most of the steps necessary to fix the driver. This list is largely a copy of their "how-to". However, I have elaborated on a few of the steps where I had issues.

1. Install the bc635/637 driver. To do this, either run make install OR cp the windrvr6.ko and windrvr6_usb.ko files into the /lib/modules/drivers/ area, then do a depmod -a. There is a script called wdreg that will load up the driver, wdreg windrvr6. I added a call to the wdreg script in /etc/rc.local in order to get the module loading on boot, one could also add a rule to /etc/modules.conf.
2. To verify, run the bc63xPCIcfg file that comes with the driver, from the   sample/ folder.
3. After rebooting the computer, the sample program should still work.
4. Copy the libbcsdk.so file from the sample/ folder to /usr/lib/ (or another library directory of your choice).
5. Make a link to the device driver, ln -s /dev/windrvr6 /dev/btfp0 (btfp0 is the   old /dev file that ntpd looks for)
6. If you haven't already done so, download the ntpd source code from www.ntp.org/downloads.html
7. I used the ntp-4.6.2p5 source for this, but the 4.2.6p2 source has been used by others
8. Extract the source, then modify the Makefile.in file in the ntpd/ folder. Find the segment that has ntpd_LDADD = $(LDADD) $(LIBOPTS_LDADD) ../libntp/libntp.a -lm @LCRYPTO@ @LSCF@, and add -lbcsdk  
9. Run this command in the ntpd/ directory - ./configure --enable-BANCOMM --enable-linuxcaps
10. Look at the Makefile (NOT Makefile.in). There should be a line like  ntpd_LDADD = $(LDADD) $(LIBOPTS_LDADD) ../libntp/libntp.a -lm -lbcsdk If not, add the -lbcsdk here, but know that running ./configure again will overwrite this Makefile.
11. Edit the refclock_bancomm.c file as follows

At line 182
extern uint32_t __attribute__ ((weak)) bcReadBinTimeEx(SYMMT_PCI_HANDLE, uint64_t *, uint64_t*, uint32_t*, uint8_t*);

At line 440
  case 2:                         /* Linux/Windows, PCI, 2 32bit time words */
                        {
                            struct tm temp;
                            uint64_t btm[2];
                            uint32_t nano;
                            uint8_t dmy;
                            if (bcReadBinTimeEx(stfp_handle, &btm[1], &btm[0], &nano, &dmy) == 0)
                            {
                                msyslog(LOG_ERR, "get_datumtime error: %m");
                                return(NULL);
                            }
                            temp = *gmtime((time_t *)&btm[1]);
                            time_vme->year = (unsigned short)temp.tm_year+1900;
                            time_vme->day = (unsigned short)temp.tm_yday+1;
                            time_vme->hr = (unsigned short)temp.tm_hour;
                            time_vme->mn = (unsigned short)temp.tm_min;
                            time_vme->sec = (unsigned short)temp.tm_sec;
                            time_vme->frac = (btm[0] & 0x000FFFFF) * 1000;
                            time_vme->frac += (nano & 0x0000000F) * 100;
                            time_vme->status = (unsigned short)dmy;

                            #ifdef DEBUG
                            if(debug)
                            {
                                printf("Epoch time read from the BANCOMM card -> %ld\n", btm[1]);

                                printf("Decimal time read from the BANCOMM card -> Day:%d Year:%d H%02d:M%02d:S%02d.%06lu%d Status:%d\n",
                                temp.tm_yday+1,
                                temp.tm_year+1900,
                                temp.tm_hour,
                                temp.tm_min,
                                temp.tm_sec,
                                btm[0],
                                nano,
                                dmy);

                                printf("Decimal time read into time_vme -> Day:%hu Year:%hu H%hu:M%hu:S%hu.%lu Status:%hu\n",
                                time_vme->day,
                                time_vme->year,
                                time_vme->hr,
                                time_vme->mn,
                                time_vme->sec,
                                time_vme->frac,
                                time_vme->status);
                            }
                            #endif
                            //tvme_fill(time_vme, btm);
                        }
                        break;

One can remove all the commented lines, as well as the lines surrounded by
#ifdef DEBUG, but I have left them here for completeness.

Now run make. This should build a ntpd executable that will work with the libbcsdk.so library in order to read time from the Symmetricom PCIe GPS. If anyone knows how to handle the nanoseconds field, please let me know and I will update this post!

MODIFYING NTP.CONF
I then had to modify ntp.conf to use the device

# No restrictions on loopback
restrict 127.0.0.1

#Allow systems to sync
restrict 10.0.0.0 mask 255.0.0.0 nomodify notrap
restrict 192.168.0.0 mask 255.255.0.0 nomodify notrap

#Setup GPS PCIe Clock
restrict 127.127.0.0 mask 255.255.0.0
server 127.127.16.0 prefer mode 2 iburst minpoll 4 maxpoll 4

#Local clock source
#server 127.127.1.0
#fudge 127.127.1.0 stratum 8

driftfile /var/lib/ntp/drift
authenticate no

Here the server 127.127.16.x is a special subnet which refers to the Symmetricom(or BANCOMM) hardware clock when ntpd is compiled to support this radio.

The mode keyword is specific to each hardware radio. For Symmetricom, I have been unable to find information on what this mode means, but for Meinberg
see http://www.meinbergglobal.com/english/info/ntp.htm

Assuming that Symmetricom has the same convention, this would mean it is using a time string to send the information.
iburst allows the computer to sync more quickly with GPS time.
minpoll and maxpoll specify a time in seconds to poll the timesource for
time.

INSTALLING
For CentOS 6.2, I had to replace the existing ntpd in /usr/sbin/ntpd with the custom built version.

I have also had problems with ntpd not starting on boot -
I had to manually add a call starting ntpd to /etc/rc.local even though chkconfig showed ntpd on. It seems that ntpd starts but crashes? To be continued as I explore the issue further.

CHECKING THAT IT WORKS
Running ntpq -p should show the GPS clock, and you should see the reach column increase as the computer syncs with the card. If you see things working, but then an x appears in front of the entry for the card, that means there is another ntpd source that disagrees time-wise.

I have seen this in the past, which worries me a little, as I am supposedly pulling time straight from GPS. Maybe there is a bug in my code - or maybe the other server was wrong? In any case, using the GPS card as the only time source works great and has greatly decreased the footprint of my system.

LINKS
http://www.linuxquestions.org/questions/linux-networking-3/ntp-conf-need-settings-for-a-refclock-using-symmetricom-bc635pci-timing-card-787257/#post4471512
http://www.eecis.udel.edu/~mills/ntp/html/ntpd.html
http://www.eecis.udel.edu/~mills/ntp/html/debug.html
http://compgroups.net/comp.protocols.time.ntp/proprietary-hardware-clock-as-ntp-ref/168850
https://support.ntp.org/bugs/show_bug.cgi?id=1674
https://support.ntp.org/bugs/show_bug.cgi?format=multiple&id=786