Compiling a Mach/xnu kernel for MacOSX

Compiled by Paul Day, 18/08/04.
- Comments and corrections welcome
(Your browser must have java enabled to see the mail-to link)

Introduction

This page exists because I couuldn't find any decent, complete notes on compiling up a kernel for MacOSX and I now have people bugging me on how I did it (hi Neil! :>).

You may wish to compile your own kernel for MacOSX because:

You can
You're into self-punishment
You want to put your own patches into the kernel to make it do things Apple's default kernel can't (e.g. arp-spoofing)
You want to optimise the kernel by disabling features you don't need (not covered by this page)

These instructions assume you have the following:

Some clue
A decent package installation environment that you know how to use (e.g. fink or darwinports)
An Apple Developer account (free to sign-up)
Xcode (1.0 comes as CD4, 1.1 came in Software Update and 1.5 you can download from ADC)
MacOSX 10.3.5 (or something close enough)

Warning!

Do not start playing with your Mac's kernel unless have some idea of what you're actually doing and you're happy taking the risks associated with it. I take absolutely no responsibility if, in following my instructions, you destory your Mac! I am, however, happy to accept comments/corrections from people.

Instructions

Make sure you've installed the Apple Development tools (XCode).
If /usr/local/bin doesn't exist, create it:
mkdir /usr/local/bin
Make sure you have it in your path:
export PATH=$PATH:/usr/local/bin
Create a directory with nothing special in the path-name:
mkdir /usr/local/src
Download the following packages (tar-gzips) for your version of MacOSX from the Darwin web-page (e.g. for 10.3.5):
- bootstrap_cmds
- Libstreams
- cctools
- kext-tools
- IOKitUser
- xnu
Move them into your working directory, /usr/local/src
untar-gzip them all. Assuming you're using gnutar:
cd /usr/local/src ; for i in *.tar do ; tar -zxvf $i ; done
Now do the following:

cd bootstrap_cmds-version/relpath.tproj
make
sudo make install
cd ../Libstreams-version
make
sudo make install
cd ../cctools-*
sudo cp /usr/include/sys/unistd.h /System/Library/Frameworks/Kernel.framework/Headers/sys
We now need to edit cctool's makefile to compile only what we need:
vi Makefile
Comment out the "COMMON_SUBDIRS" lines (line 28 and 29 for me) with hashes ("#"):

#COMMON_SUBDIRS = libstuff as gprof misc libmacho ld dyld libdyld \
# mkshlib otool profileServer RelNotes man cbtlibs
Insert the following line just below it (line 30 for me):
COMMON_SUBDIRS = libstuff libmacho misc
Write the file and exit:
[ESC]:wq[Enter]
Now do the the following:
make RC_OS=macos
sudo cp misc/seg_hack.NEW /usr/local/bin/seg_hack
cd ld
make RC_OS=macos kld_build
sudo cp static_kld/libkld.a /usr/local/lib
sudo ranlib /usr/local/lib/libkld.a
Time to compile up kextsymboltool:
cd ../../
sudo mv IOKitUser* IOKit
sudo mv IOKit /usr/include
sudo mv /usr/include/IOKit/kext.subproj /usr/include/IOKit/kext
sudo cp -fpr cctools*/include/mach-o /usr/include
sudo cp -fpr cctools*/include/mach/* /usr/include/mach
cd kext_tools*
gcc kextsymboltool.c -o kextsymboltool
If you get errors at the above stage, either you've made a mistake with the previous lines, I've made a mistake with the instructions or your system isn't the same as mine. Otherwise:

sudo cp kextsymboltool /usr/local/bin
sudo chmod 755 /usr/local/bin/kextsymboltool
Now cd into the kernel directory:
cd ../xnu*
This is where you would patch the kernel source. Some patches I use include:
- Jeff Nathan's BIOSCHDRCMPLT patch, which allows arp-spoofing under MacOSX for apps like ettercap. Check Jeff's page for the patch that matches your kernel! The one linked above works with 10.3.5/xnu-517.7.21.
- Peter Bartoli's FORCE_VERBOSE patch, which forces the kernel to drop the grey boot-up screen and show you verbose booting information. The above link is for 571 again. Check Peter's page for the latest patch.
Download the patches you want and then, while still in the root of the xnu source directory, patch them with:
patch -p0 -b --verbose --suffix=.orig < /path/to/the/patch.patch
Pay attention and make sure you're using the correct patch by viewing the output of the patch utility!
Now compile the kernel:
make exporthdrs
make all
With a little luck, your kernel compiled without any errors. You will now have your freshly compiled kernel at BUILD/obj/RELEASE_PPC/mach_kernel. Move it into / without over-writing your default Apple kernel:
cp BUILD/obj/RELEASE_PPC/mach_kernel /mach_kernel.new
chmod 644 /mach_kernel.new

chown root.wheel /mach_kernel.new

Credit to Jean-Pierre Mouilleseaux for the following idea: Instead of just putting it over the top of the existing kernel, you can have a second kernel and configure OpenFirmware to boot from the second kernel, allowing you an easy way out if the new kernel is corrupt. Configure OpenFirmware to boot from the new kernel with:
sudo nvram boot-file=`nvram boot-device | awk -F , '{print $1}' | awk '{print $2}'`,mach_kernel.new

For those of you using a G4 iBook, it seems they always alias hd: for the boot device. However, it seems you can't use the alias when booting a custom kernel for some strange reason. The following command should fix the problem for you:

sudo nvram boot-file=pci2/ata-6@D/@0:3,mach_kernel.new

The above command sets the OpenFirmware Variable "boot-file" to the file /kernel.new on the current boot device. If your new kernel barfs, you can back out by either:
- During power-on, reset the entire PRAM by holding down option-apple-p-r and wait for the start-up chime to sound three times, or
- During power-on, press option-apple-o-f to enter OpenFirmware, type "reset-all" and then "mac-boot", or
- During power-on, reset the OpenFirmware NVRAM by holding option-apple-n-v.
OpenFirmware defaults to booting using /mach_kernel, which will be the default Apple kernel if you've followed the instructions above.
So now your new kernel is installed and you can test it by restaring or simply typing "reboot" at a command line.

Created, Paul Day, 19/08/04
Updated, Paul Day 24/11/04