Add a new syscall into Linux kernel – for fedora 18 with kernel 3.8.7

Adding a new syscall into Linux kernel would be tricky for the beginners (like myself), especially considering the kernel source tree structure changes among different versions. It seems most of the online tutorials focusing on older versions of kernel (2.6.X) or early version of 3.X. Things are little bit different using the latest version of kernel (3.8.7, and I have no idea from which version did all these changes happen…). Anyway, we will try to compare the older way and the latest way to add a new syscall into the kernel. Good luck! (NOTE: this post is for x86 arch!)

A. Older version of kernel (2.6.X) or early version of 3.X
This post (http://arvindsraj.wordpress.com/2012/10/05/adding-hello-world-system-call-to-linux/) actually gives a great and detailed reference on how to do that. Summary is listed down below.

Step 0: Define the new syscall (sys_hello())
Make a new directory called ‘hello’ under the kernel directory and create the source file for syscall sys_hello() as well as its makefile.
file:
hello/hello.c
add:

#include <linux/kernel.h>
asmlinkage long sys_hello(void)
{
    printk("Hello worldn");
    return 0;
}
file:
hello/Makefile
add:
obj-y := hello.o
Step 1: Add the hello directory into kernel Makefile
file:
Makefile
original:
core-y          += kernel/ mm/ fs/ ipc/ security/ crypto/ block/
after:
core-y          += kernel/ mm/ fs/ ipc/ security/ crypto/ block/ hello/

Step 2: Add the new syscall into syscall table (Assembly file actually)
file:
arch/x86/kernel/syscall_table_32.S (for x86-64 arch, the file should be syscall_table_64.S)
add:
.long sys_hello

Step 3: Add the syscall number (macro definition eventually and the number varies)
file:
arch/x86/include/asm/unistd_32.h
add:
#define __NR_hello 349
add:
#define NR_syscalls 350
file:
arch/x86/include/asm/unistd_64.h
add:

#define __NR_hello 312
 __SYSCALL(__NR_hello, sys_hello)

Step 4: Add the new syscall into syscall header file
file:
include/linux/syscalls.h
add:
asmlinkage long sys_hello(void);

Step 5: Compile, update the kernel and run the testing file (using dmesg to check the kernel log)
file:
$HOME/sysHello.c
add:

#include <stdio.h>
#include <linux/kernel.h>
#include <sys/syscall.h>
#include <unistd.h>
#define __NR_hello 312 //349 if you are running a 32bit kernel and following my tutorial
long hello_syscall(void)
{
    return syscall(__NR_hello);
}
int main(int argc, char *argv[])
{
    long int a = hello_syscall();
    printf("System call returned %ldn", a);
    return 0;
}
B. Linux kernel 3.8.7

Step 0: Define the new syscall (sys_hello())
Make a new directory called ‘hello’ under the kernel directory and create the source file for syscall sys_hello() as well as its makefile.
file:
hello/hello.c
add:

#include <linux/kernel.h>
asmlinkage long sys_hello(void)
{
    printk("Hello worldn");
    return 0;
}
file:
hello/Makefile
add:
obj-y := hello.o
Step 1: Add the hello directory into kernel Makefile
file:
Makefile
original:
core-y          += kernel/ mm/ fs/ ipc/ security/ crypto/ block/
after:
core-y          += kernel/ mm/ fs/ ipc/ security/ crypto/ block/ hello/

Step 2: Add the new syscall as well as the number into syscall table (real table file and the number varies)
file:
arch/x86/syscalls/syscall_32.tbl
add (at the end of the file):
351    i386    daveti_hello    sys_hello
file:
arch/x86/syscalls/syscall_64.tbl
add (at the end of the file but before x32-specific syscalls):
314    common    daveti_hello    sys_hello

Step 3: Add the new syscall into syscall header file
file:
include/linux/syscalls.h
add:
asmlinkage long sys_hello(void);

Step 4: Compile, update the kernel and run the testing file (using dmesg to check the kernel log)
file:
$HOME/sysHello.c
add:
#include <stdio.h>
#include <linux/kernel.h>
#include <sys/syscall.h>
#include <unistd.h>

int main()
{
    long int rtn = syscall(351);
    printf(“System call sys_hello return %ldn”, rtn);
    return 0;
}

About daveti

Interested in kernel hacking, compilers, machine learning and guitars.
This entry was posted in OS and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s