Generating Debug Symbols for the Linux Kernel

While working with SystemTap on an Ubuntu 14.04 installation, I was met with the following error when trying to probe a syscall:

Systemtap translator/driver (version 2.3/0.158, Debian version 2.3-1ubuntu1.1 (trusty))
Copyright (C) 2005-2013 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBSQLITE3 NSS TR1_UNORDERED_MAP NLS
Created temporary directory "/tmp/stapJqcNkt"
Session arch: x86_64 release: 3.13.0-43-generic
Kernel symbol table /lib/modules/3.13.0-43-generic/build/System.map unavailable, (No such file or directory)
Searched for library macro files: "/usr/share/systemtap/tapset/linux/*.stpm", found: 2, processed: 2
Searched for library macro files: "/usr/share/systemtap/tapset/*.stpm", found: 1, processed: 1
Searched: "/usr/share/systemtap/tapset/linux/x86_64/*.stp", found: 3, processed: 3
Searched: "/usr/share/systemtap/tapset/linux/*.stp", found: 67, processed: 67
Searched: "/usr/share/systemtap/tapset/x86_64/*.stp", found: 1, processed: 1
Searched: "/usr/share/systemtap/tapset/*.stp", found: 21, processed: 21
Pass 1: parsed user script and 95 library script(s) using 85852virt/27404res/2536shr/25488data kb, in 120usr/40sys/161real ms.
Attempting to extract kernel debuginfo build ID from /lib/modules/3.13.0-43-generic/build/vmlinux.id
Attempting to extract kernel debuginfo build ID from /sys/kernel/notes
semantic error: while resolving probe point: identifier 'syscall' at open.stp:1:7
        source: probe syscall.open
                      ^

semantic error: no match
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 86364virt/28296res/2792shr/26000data kb, in 40usr/360sys/1130real ms.
Pass 2: analysis failed.  [man error::pass2]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.
Running rm -rf /tmp/stapJqcNkt
Spawn waitpid result (0x0): 0
Removed temporary directory "/tmp/stapJqcNkt"

To get around this, I ended up having to generate the debug symbols for the installed Linux kernel so that SystemTap could consume them while setting up the probe.  The rest of this post describes the steps that were taken to complete this task.

Build preparation

Apt provides the `build-dep` operation which pulls in all of the build dependencies that would be needed to build the specified source package.  In this case, the following command did the trick:

sudo apt-get build-dep --no-install-recommends linux-image-$(uname -r)

Pulling down the kernel source code can be accomplished via Apt as the `source` operation makes this possible:

apt-get source linux-image-$(uname -r)

This had unexpected results however as the currently running kernel reported by `uname -r` that was requested was not delivered due to a newer version being available:

Reading package lists... Done
Building dependency tree
Reading state information... Done
Picking 'linux' as source package instead of 'linux-image-3.13.0-43-generic'
NOTICE: 'linux' packaging is maintained in the 'Git' version control system at:
http://kernel.ubuntu.com/git-repos/ubuntu/ubuntu-trusty.git
Need to get 126 MB of source archives.
Get:1 http://mirrors.digitalocean.com/ubuntu/ trusty-updates/main linux 3.13.0-77.121 (dsc) [8,019 B]
Get:2 http://mirrors.digitalocean.com/ubuntu/ trusty-updates/main linux 3.13.0-77.121 (tar) [116 MB]
Get:3 http://mirrors.digitalocean.com/ubuntu/ trusty-updates/main linux 3.13.0-77.121 (diff) [9,076 kB]
Fetched 126 MB in 5s (21.4 MB/s)

I ended up having to clone the kernel version I wanted via git:

git clone git://kernel.ubuntu.com/ubuntu/ubuntu-<release>.git
git checkout -b temp Ubuntu-<kernel_version>

Build the kernel

Once the kernel source code was accessible and all build dependencies were in place, I was able to build the kernel alongside the desired debug symbols:

fakeroot debian/rules clean
AUTOBUILD=1 fakeroot debian/rules binary-generic skipdbg=false

The above commands essentially invoke the build script that is bundled with the kernel source code.  Leveraging `fakeroot` allows for the resulting build to contain files or directories that have root permissions assigned to them without having to run the overall build script as the root user.  Also note that the `skipdbg=false` flag is present which accomplishes generating a kernel that includes the debug symbols necessary for SystemTap to function.

Install the kernel debug symbols

The final step was to install the newly built kernel with accompanying debug symbols via `dpkg`:

dpkg -i ../linux-image-3.13.0-43-generic-dbgsym_3.13.0-43.121_amd64.ddeb

References

https://wiki.ubuntu.com/Kernel/Systemtap
https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel
https://wiki.ubuntu.com/KernelTeam/KernelMaintenance
https://wiki.ubuntu.com/Kernel/SourceCode