Hack – make VMware Player 6.0.1 start on Linux kernel 3.13.X

If you were using VMware Player 6.0.1 on the latest Linux kernel 3.13.X and not able to get it started. Please read thru this post.

0. Error – failure to build vmnet

Inspect the VMware starting log /tmp/vmware-root/modconfig-XXXX.log, we may find the failure like this:

2014-03-04T20:55:19.711-08:00| vthread-3| I120: /lib/modules/3.13.5-200.fc20.x86_64/build/include/linux/version.h not found, looking for generated/uapi/linux/version.h instead.
2014-03-04T20:55:19.711-08:00| vthread-3| I120: using /usr/lib64/ccache/gcc for preprocess check
2014-03-04T20:55:19.720-08:00| vthread-3| I120: Preprocessed UTS_RELEASE, got value “3.13.5-200.fc20.x86_64”.
2014-03-04T20:55:19.720-08:00| vthread-3| I120: The header path “/lib/modules/3.13.5-200.fc20.x86_64/build/include” for the kernel “3.13.5-200.fc20.x86_64” is valid.  Whoohoo!
2014-03-04T20:55:19.855-08:00| vthread-3| I120: Invoking modinfo on “vmnet”.
2014-03-04T20:55:19.859-08:00| vthread-3| I120: “/sbin/modinfo” exited with status 256.
2014-03-04T20:55:19.909-08:00| vthread-3| I120: Setting destination path for vmnet to “/lib/modules/3.13.5-200.fc20.x86_64/misc/vmnet.ko”.
2014-03-04T20:55:19.909-08:00| vthread-3| I120: Extracting the vmnet source from “/usr/lib/vmware/modules/source/vmnet.tar“.
2014-03-04T20:55:19.915-08:00| vthread-3| I120: Successfully extracted the vmnet source.
2014-03-04T20:55:19.916-08:00| vthread-3| I120: Building module with command “/usr/bin/make -j4 -C /tmp/modconfig-pSj2y5/vmnet-only auto-build HEADER_DIR=/lib/modules/3.13.5-200.fc20.x86_64/build/include CC=/usr/lib64/ccache/gcc IS_GCC_3=no”
2014-03-04T20:55:21.786-08:00| vthread-3| W110: Failed to build vmnet.  Failed to execute the build command.

So the root cause for VMware Player to start up is the failure to build a kernel module called vmnet. Fortunately, the vmnet source are open and available for edit. The log above has already told you where to find vmnet.tar source files.

1. Verify – manually make vmnet

Let us go into the vmnet.tar directory /usr/lib/vmware/modules/source/, untar the source file and do the make directly.

[root@daveti source]# ll
total 5004
-rw-r–r–. 1 root root  757760 Mar  4 20:38 vmblock.tar
-rw-r–r–. 1 root root 1249280 Mar  4 20:38 vmci.tar
-rw-r–r–. 1 root root 1208320 Mar  4 20:38 vmmon.tar
drwxr-xr-x. 2 root root    4096 Oct 18 12:11 vmnet-only
-rw-r–r–. 1 root root  880640 Mar  4 20:38 vmnet.tar
-rw-r–r–. 1 root root 1024000 Mar  4 20:38 vsock.tar
[root@daveti source]# cd vmnet-only
[root@daveti vmnet-only]# ll
total 916
-rw-r–r–. 1 root root  50253 Oct 18 12:11 bridge.c
-rw-r–r–. 1 root root   2018 Oct 18 12:11 community_source.h
-rw-r–r–. 1 root root   1441 Oct 18 12:11 compat_autoconf.h
-rw-r–r–. 1 root root   2298 Oct 18 12:11 compat_module.h
-rw-r–r–. 1 root root  10212 Oct 18 12:11 compat_netdevice.h
-rw-r–r–. 1 root root   7037 Oct 18 12:11 compat_skbuff.h
-rw-r–r–. 1 root root   3074 Oct 18 12:11 compat_sock.h
-rw-r–r–. 1 root root   3827 Oct 18 12:11 compat_version.h
-rw-r–r–. 1 root root  17987 Oct 18 12:11 COPYING
-rw-r–r–. 1 root root  49129 Oct 18 12:11 driver.c
-rw-r–r–. 1 root root   2252 Oct 18 12:11 driver-config.h
-rw-r–r–. 1 root root  52703 Oct 18 12:11 filter.c
-rw-r–r–. 1 root root   1233 Oct 18 12:11 geninclude.c
-rw-r–r–. 1 root root  17919 Oct 18 12:11 hub.c
-rw-r–r–. 1 root root    850 Oct 18 12:11 includeCheck.h
-rw-r–r–. 1 root root   4447 Oct 18 12:11 Makefile
-rw-r–r–. 1 root root   1695 Oct 18 12:11 Makefile.kernel
-rw-r–r–. 1 root root   2642 Oct 18 12:11 Makefile.normal
-rw-r–r–. 1 root root   4542 Oct 18 12:11 monitorAction_exported.h
-rw-r–r–. 1 root root   1296 Oct 18 12:11 netdev_has_dev_net.c
-rw-r–r–. 1 root root   1774 Oct 18 12:11 netdev_has_net.c
-rw-r–r–. 1 root root   5805 Oct 18 12:11 net.h
-rw-r–r–. 1 root root  14142 Oct 18 12:11 netif.c
-rw-r–r–. 1 root root   1592 Oct 18 12:11 nfhook_uses_skb.c
-rw-r–r–. 1 root root   8939 Oct 18 12:11 procfs.c
-rw-r–r–. 1 root root   1402 Oct 18 12:11 skblin.c
-rw-r–r–. 1 root root 161736 Oct 18 12:11 smac.c
-rw-r–r–. 1 root root   9610 Oct 18 12:11 smac_compat.c
-rw-r–r–. 1 root root   2055 Oct 18 12:11 smac_compat.h
-rw-r–r–. 1 root root   3689 Oct 18 12:11 smac.h
-rw-r–r–. 1 root root  27555 Oct 18 12:11 userif.c
-rw-r–r–. 1 root root  11655 Oct 18 12:11 vm_assert.h
-rw-r–r–. 1 root root  72320 Oct 18 12:11 vm_atomic.h
-rw-r–r–. 1 root root  33932 Oct 18 12:11 vm_basic_asm.h
-rw-r–r–. 1 root root  15323 Oct 18 12:11 vm_basic_asm_x86_64.h
-rw-r–r–. 1 root root  18641 Oct 18 12:11 vm_basic_asm_x86.h
-rw-r–r–. 1 root root  16634 Oct 18 12:11 vm_basic_defs.h
-rw-r–r–. 1 root root  26189 Oct 18 12:11 vm_basic_types.h
-rw-r–r–. 1 root root  10876 Oct 18 12:11 vm_device_version.h
-rw-r–r–. 1 root root   3268 Oct 18 12:11 vmnetInt.h
-rw-r–r–. 1 root root   7416 Oct 18 12:11 vm_oui.h
-rw-r–r–. 1 root root   1316 Oct 18 12:11 vmware_pack_begin.h
-rw-r–r–. 1 root root   1336 Oct 18 12:11 vmware_pack_end.h
-rw-r–r–. 1 root root   1956 Oct 18 12:11 vmware_pack_init.h
-rw-r–r–. 1 root root  14627 Oct 18 12:11 vnetEvent.c
-rw-r–r–. 1 root root   1839 Oct 18 12:11 vnetEvent.h
-rw-r–r–. 1 root root   7633 Oct 18 12:11 vnetFilter.h
-rw-r–r–. 1 root root   3280 Oct 18 12:11 vnetFilterInt.h
-rw-r–r–. 1 root root  12538 Oct 18 12:11 vnet.h
-rw-r–r–. 1 root root   8929 Oct 18 12:11 vnetInt.h
-rw-r–r–. 1 root root   2064 Oct 18 12:11 vnetKernel.h
-rw-r–r–. 1 root root   9971 Oct 18 12:11 vnetUserListener.c
-rw-r–r–. 1 root root  73397 Oct 18 12:11 x86cpuid.h
[root@daveti vmnet-only]# make
Using 2.6.x kernel build system.
make -C /lib/modules/3.13.5-200.fc20.x86_64/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. \
MODULEBUILDDIR= modules
make[1]: Entering directory `/usr/src/kernels/3.13.5-200.fc20.x86_64′
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/driver.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/hub.o
/usr/lib/vmware/modules/source/vmnet-only/hub.c: In function ‘VNetHubFindHubByID’:
/usr/lib/vmware/modules/source/vmnet-only/hub.c:132:49: warning: argument to ‘sizeof’ in ‘memcmp’ call is the same expression as the first source; did you mean to provide an explicit length? [-Wsizeof-pointer-memaccess]
memcmp(idNum, currHub->id.pvnID, sizeof idNum))) {
^
/usr/lib/vmware/modules/source/vmnet-only/hub.c: In function ‘VNetHubAlloc’:
/usr/lib/vmware/modules/source/vmnet-only/hub.c:315:36: warning: argument to ‘sizeof’ in ‘memcpy’ call is the same pointer type ‘uint8 *’ as the destination; expected ‘uint8’ or an explicit length [-Wsizeof-pointer-memaccess]
memcpy(hub->id.pvnID, id, sizeof id);
^
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/userif.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/netif.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/bridge.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/filter.o
/usr/lib/vmware/modules/source/vmnet-only/filter.c:206:1: error: conflicting types for ‘VNetFilterHookFn’
VNetFilterHookFn(unsigned int hooknum,                 // IN:
^
/usr/lib/vmware/modules/source/vmnet-only/filter.c:64:18: note: previous declaration of ‘VNetFilterHookFn’ was here
static nf_hookfn VNetFilterHookFn;
^
/usr/lib/vmware/modules/source/vmnet-only/filter.c:64:18: warning: ‘VNetFilterHookFn’ used but never defined [enabled by default]
/usr/lib/vmware/modules/source/vmnet-only/filter.c:206:1: warning: ‘VNetFilterHookFn’ defined but not used [-Wunused-function]
VNetFilterHookFn(unsigned int hooknum,                 // IN:
^
make[2]: *** [/usr/lib/vmware/modules/source/vmnet-only/filter.o] Error 1
make[1]: *** [_module_/usr/lib/vmware/modules/source/vmnet-only] Error 2
make[1]: Leaving directory `/usr/src/kernels/3.13.5-200.fc20.x86_64′
make: *** [vmnet.ko] Error 2
[root@daveti vmnet-only]#

Then, we confirmed that the root cause is vmnet.ko. More specifically, VNetFilterHookFn in filter.c causes the building error. And so, what happened there?

2. RCA – root cause analysis

In the latest kernel, e.g, 3.13.5, the signature of netfilter hook function (nf_hookfn) has been changed as the result of code refactoring. However, VNetFilterHookFn is still using the old definition of nf_hookfn. That is why we are not able to compile vmnet.ko. For details, please check http://lxr.linux.no/linux+v3.13.5/include/linux/netfilter.h#L46

3. Fix – diff

[root@daveti vmnet-only]# diff filter.c.orig filter.c
206c206,207
< VNetFilterHookFn(unsigned int hooknum,                 // IN:

> //VNetFilterHookFn(unsigned int hooknum,                 // IN:
> VNetFilterHookFn(const struct nf_hook_ops *ops, //daveti: fix
255c256,258
<    transmit = (hooknum == VMW_NF_INET_POST_ROUTING);

>    //transmit = (hooknum == VMW_NF_INET_POST_ROUTING);
>    //daveti: fix
>    transmit = (ops->hooknum == VMW_NF_INET_POST_ROUTING);
[root@daveti vmnet-only]#

4. Verify fix – make again

After the patch for the filter.c, we should be able to make the vmnet kernel module now. Let us make sure the fix work.

[root@daveti vmnet-only]# make
Using 2.6.x kernel build system.
make -C /lib/modules/3.13.5-200.fc20.x86_64/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. \
MODULEBUILDDIR= modules
make[1]: Entering directory `/usr/src/kernels/3.13.5-200.fc20.x86_64′
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/filter.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/procfs.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/smac_compat.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/smac.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/vnetEvent.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/vnetUserListener.o
LD [M]  /usr/lib/vmware/modules/source/vmnet-only/vmnet.o
Building modules, stage 2.
MODPOST 1 modules
CC      /usr/lib/vmware/modules/source/vmnet-only/vmnet.mod.o
LD [M]  /usr/lib/vmware/modules/source/vmnet-only/vmnet.ko
make[1]: Leaving directory `/usr/src/kernels/3.13.5-200.fc20.x86_64′
make -C $PWD SRCROOT=$PWD/. \
MODULEBUILDDIR= postbuild
make[1]: Entering directory `/usr/lib/vmware/modules/source/vmnet-only’
make[1]: `postbuild’ is up to date
make[1]: Leaving directory `/usr/lib/vmware/modules/source/vmnet-only’
cp -f vmnet.ko ./../vmnet.o
[root@daveti vmnet-only]#

5. Tar and Start

Now it is time to tar the patched vmnet-only directory into vmnet.tar again. Start the VMware Player and enjoy~

About daveti

Interested in kernel hacking, compilers, machine learning and guitars.
This entry was posted in OS and tagged , , . Bookmark the permalink.

14 Responses to Hack – make VMware Player 6.0.1 start on Linux kernel 3.13.X

  1. Sanford Rockowitz says:

    Thank you! You saved me hours.

  2. Benoit Bouthillier says:

    Work like a charm.

    Thanks !

  3. Scott says:

    Thanks that worked great!

  4. This also worked for 3.14.1 Thank yoU !

  5. Amazing work Mr. DaveTI
    I fix the same problem but my kernel is “SMP Debian 3.14.5-1kali1 (2014-06-07) x86_64 GNU/Linux” and following your instructions all worked fine. Thanks a lot!

  6. Amazing work Mr. DaveTI
    I fix the same problem but my kernel is “SMP Debian 3.14.5-1kali1 (2014-06-07) x86_64 GNU/Linux” and following your instructions all worked fine. Thanks a lot!

  7. Hi i got an error on step 3. When i type diff filter.c.orig filter.c (“diff: filter.c.orig: No such file or directory”). My kernel and linux distro is Linux sbin 3.14-kali1-amd64 #1 SMP Debian 3.14.4-1kali1 (2014-05-14) x86_64 GNU/Linux. Please help me !!!!!

  8. poyi says:

    This is not working for me.

    I have made the changes as mentioned above:

    root@poyi:/usr/lib/vmware/modules/source/vmnet-only# diff filter.c.orig filter.c206c206
    //VNetFilterHookFn(unsigned int hooknum, // IN:
    255,256c255,256
    < transmit = (hooknum == VMW_NF_INET_POST_ROUTING);
    // transmit = (hooknum == VMW_NF_INET_POST_ROUTING);
    > transmit = (ops->hooknum == VMW_NF_INET_POST_ROUTING);
    root@poyi:/usr/lib/vmware/modules/source/vmnet-only# make
    Using 2.6.x kernel build system.
    make -C /lib/modules/3.14-kali1-amd64/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. \
    MODULEBUILDDIR= modules
    make[1]: Entering directory `/usr/src/linux-headers-3.14-kali1-amd64′
    CC [M] /usr/lib/vmware/modules/source/vmnet-only/filter.o
    In file included from /usr/lib/vmware/modules/source/vmnet-only/monitorAction_exported.h:31:0,
    from /usr/lib/vmware/modules/source/vmnet-only/vnet.h:29,
    from /usr/lib/vmware/modules/source/vmnet-only/vnetInt.h:24,
    from /usr/lib/vmware/modules/source/vmnet-only/filter.c:48:
    /usr/lib/vmware/modules/source/vmnet-only/vm_assert.h:259:0: warning: “DEPRECATED” redefined [enabled by default]
    In file included from /usr/src/linux-headers-3.14-kali1-common/include/linux/kernel.h:13:0,
    from /usr/lib/vmware/modules/source/vmnet-only/filter.c:20:
    /usr/src/linux-headers-3.14-kali1-common/include/linux/printk.h:96:0: note: this is the location of the previous definition
    /usr/lib/vmware/modules/source/vmnet-only/filter.c:208:18: error: two or more data types in declaration specifiers
    /usr/lib/vmware/modules/source/vmnet-only/filter.c:212:18: error: expected identifier or ‘(’ before ‘const’
    /usr/lib/vmware/modules/source/vmnet-only/filter.c:64:18: warning: ‘VNetFilterHookFn’ used but never defined [enabled by default]
    /usr/lib/vmware/modules/source/vmnet-only/filter.c:208:34: warning: ‘skb’ defined but not used [-Wunused-variable]
    make[4]: *** [/usr/lib/vmware/modules/source/vmnet-only/filter.o] Error 1
    make[3]: *** [_module_/usr/lib/vmware/modules/source/vmnet-only] Error 2
    make[2]: *** [sub-make] Error 2
    make[1]: *** [all] Error 2
    make[1]: Leaving directory `/usr/src/linux-headers-3.14-kali1-amd64′
    make: *** [vmnet.ko] Error 2
    root@poyi:/usr/lib/vmware/modules/source/vmnet-only#

  9. poyi says:

    Never mind that, I saw my mistake.

    Thank you for this post.

  10. Eyem says:

    Works on Mint 17.2, kernel 3.16.0-38-generic. Thanks for the assist, sir!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s