If you were using VMware Player 6.0.1 on the latest Linux kernel 3.13.X and not able to get it started. Please read thru this post.

0. Error – failure to build vmnet

Inspect the VMware starting log /tmp/vmware-root/modconfig-XXXX.log, we may find the failure like this:

2014-03-04T20:55:19.711-08:00| vthread-3| I120: /lib/modules/3.13.5-200.fc20.x86_64/build/include/linux/version.h not found, looking for generated/uapi/linux/version.h instead.
2014-03-04T20:55:19.711-08:00| vthread-3| I120: using /usr/lib64/ccache/gcc for preprocess check
2014-03-04T20:55:19.720-08:00| vthread-3| I120: Preprocessed UTS_RELEASE, got value “3.13.5-200.fc20.x86_64”.
2014-03-04T20:55:19.720-08:00| vthread-3| I120: The header path “/lib/modules/3.13.5-200.fc20.x86_64/build/include” for the kernel “3.13.5-200.fc20.x86_64” is valid.  Whoohoo!
2014-03-04T20:55:19.855-08:00| vthread-3| I120: Invoking modinfo on “vmnet”.
2014-03-04T20:55:19.859-08:00| vthread-3| I120: “/sbin/modinfo” exited with status 256.
2014-03-04T20:55:19.909-08:00| vthread-3| I120: Setting destination path for vmnet to “/lib/modules/3.13.5-200.fc20.x86_64/misc/vmnet.ko”.
2014-03-04T20:55:19.909-08:00| vthread-3| I120: Extracting the vmnet source from “/usr/lib/vmware/modules/source/vmnet.tar“.
2014-03-04T20:55:19.915-08:00| vthread-3| I120: Successfully extracted the vmnet source.
2014-03-04T20:55:19.916-08:00| vthread-3| I120: Building module with command “/usr/bin/make -j4 -C /tmp/modconfig-pSj2y5/vmnet-only auto-build HEADER_DIR=/lib/modules/3.13.5-200.fc20.x86_64/build/include CC=/usr/lib64/ccache/gcc IS_GCC_3=no”
2014-03-04T20:55:21.786-08:00| vthread-3| W110: Failed to build vmnet.  Failed to execute the build command.

So the root cause for VMware Player to start up is the failure to build a kernel module called vmnet. Fortunately, the vmnet source are open and available for edit. The log above has already told you where to find vmnet.tar source files.

1. Verify – manually make vmnet

Let us go into the vmnet.tar directory /usr/lib/vmware/modules/source/, untar the source file and do the make directly.

[root@daveti source]# ll
total 5004
-rw-r–r–. 1 root root  757760 Mar  4 20:38 vmblock.tar
-rw-r–r–. 1 root root 1249280 Mar  4 20:38 vmci.tar
-rw-r–r–. 1 root root 1208320 Mar  4 20:38 vmmon.tar
drwxr-xr-x. 2 root root    4096 Oct 18 12:11 vmnet-only
-rw-r–r–. 1 root root  880640 Mar  4 20:38 vmnet.tar
-rw-r–r–. 1 root root 1024000 Mar  4 20:38 vsock.tar
[root@daveti source]# cd vmnet-only
[root@daveti vmnet-only]# ll
total 916
-rw-r–r–. 1 root root  50253 Oct 18 12:11 bridge.c
-rw-r–r–. 1 root root   2018 Oct 18 12:11 community_source.h
-rw-r–r–. 1 root root   1441 Oct 18 12:11 compat_autoconf.h
-rw-r–r–. 1 root root   2298 Oct 18 12:11 compat_module.h
-rw-r–r–. 1 root root  10212 Oct 18 12:11 compat_netdevice.h
-rw-r–r–. 1 root root   7037 Oct 18 12:11 compat_skbuff.h
-rw-r–r–. 1 root root   3074 Oct 18 12:11 compat_sock.h
-rw-r–r–. 1 root root   3827 Oct 18 12:11 compat_version.h
-rw-r–r–. 1 root root  17987 Oct 18 12:11 COPYING
-rw-r–r–. 1 root root  49129 Oct 18 12:11 driver.c
-rw-r–r–. 1 root root   2252 Oct 18 12:11 driver-config.h
-rw-r–r–. 1 root root  52703 Oct 18 12:11 filter.c
-rw-r–r–. 1 root root   1233 Oct 18 12:11 geninclude.c
-rw-r–r–. 1 root root  17919 Oct 18 12:11 hub.c
-rw-r–r–. 1 root root    850 Oct 18 12:11 includeCheck.h
-rw-r–r–. 1 root root   4447 Oct 18 12:11 Makefile
-rw-r–r–. 1 root root   1695 Oct 18 12:11 Makefile.kernel
-rw-r–r–. 1 root root   2642 Oct 18 12:11 Makefile.normal
-rw-r–r–. 1 root root   4542 Oct 18 12:11 monitorAction_exported.h
-rw-r–r–. 1 root root   1296 Oct 18 12:11 netdev_has_dev_net.c
-rw-r–r–. 1 root root   1774 Oct 18 12:11 netdev_has_net.c
-rw-r–r–. 1 root root   5805 Oct 18 12:11 net.h
-rw-r–r–. 1 root root  14142 Oct 18 12:11 netif.c
-rw-r–r–. 1 root root   1592 Oct 18 12:11 nfhook_uses_skb.c
-rw-r–r–. 1 root root   8939 Oct 18 12:11 procfs.c
-rw-r–r–. 1 root root   1402 Oct 18 12:11 skblin.c
-rw-r–r–. 1 root root 161736 Oct 18 12:11 smac.c
-rw-r–r–. 1 root root   9610 Oct 18 12:11 smac_compat.c
-rw-r–r–. 1 root root   2055 Oct 18 12:11 smac_compat.h
-rw-r–r–. 1 root root   3689 Oct 18 12:11 smac.h
-rw-r–r–. 1 root root  27555 Oct 18 12:11 userif.c
-rw-r–r–. 1 root root  11655 Oct 18 12:11 vm_assert.h
-rw-r–r–. 1 root root  72320 Oct 18 12:11 vm_atomic.h
-rw-r–r–. 1 root root  33932 Oct 18 12:11 vm_basic_asm.h
-rw-r–r–. 1 root root  15323 Oct 18 12:11 vm_basic_asm_x86_64.h
-rw-r–r–. 1 root root  18641 Oct 18 12:11 vm_basic_asm_x86.h
-rw-r–r–. 1 root root  16634 Oct 18 12:11 vm_basic_defs.h
-rw-r–r–. 1 root root  26189 Oct 18 12:11 vm_basic_types.h
-rw-r–r–. 1 root root  10876 Oct 18 12:11 vm_device_version.h
-rw-r–r–. 1 root root   3268 Oct 18 12:11 vmnetInt.h
-rw-r–r–. 1 root root   7416 Oct 18 12:11 vm_oui.h
-rw-r–r–. 1 root root   1316 Oct 18 12:11 vmware_pack_begin.h
-rw-r–r–. 1 root root   1336 Oct 18 12:11 vmware_pack_end.h
-rw-r–r–. 1 root root   1956 Oct 18 12:11 vmware_pack_init.h
-rw-r–r–. 1 root root  14627 Oct 18 12:11 vnetEvent.c
-rw-r–r–. 1 root root   1839 Oct 18 12:11 vnetEvent.h
-rw-r–r–. 1 root root   7633 Oct 18 12:11 vnetFilter.h
-rw-r–r–. 1 root root   3280 Oct 18 12:11 vnetFilterInt.h
-rw-r–r–. 1 root root  12538 Oct 18 12:11 vnet.h
-rw-r–r–. 1 root root   8929 Oct 18 12:11 vnetInt.h
-rw-r–r–. 1 root root   2064 Oct 18 12:11 vnetKernel.h
-rw-r–r–. 1 root root   9971 Oct 18 12:11 vnetUserListener.c
-rw-r–r–. 1 root root  73397 Oct 18 12:11 x86cpuid.h
[root@daveti vmnet-only]# make
Using 2.6.x kernel build system.
make -C /lib/modules/3.13.5-200.fc20.x86_64/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. \
MODULEBUILDDIR= modules
make[1]: Entering directory `/usr/src/kernels/3.13.5-200.fc20.x86_64′
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/driver.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/hub.o
/usr/lib/vmware/modules/source/vmnet-only/hub.c: In function ‘VNetHubFindHubByID’:
/usr/lib/vmware/modules/source/vmnet-only/hub.c:132:49: warning: argument to ‘sizeof’ in ‘memcmp’ call is the same expression as the first source; did you mean to provide an explicit length? [-Wsizeof-pointer-memaccess]
memcmp(idNum, currHub->id.pvnID, sizeof idNum))) {
^
/usr/lib/vmware/modules/source/vmnet-only/hub.c: In function ‘VNetHubAlloc’:
/usr/lib/vmware/modules/source/vmnet-only/hub.c:315:36: warning: argument to ‘sizeof’ in ‘memcpy’ call is the same pointer type ‘uint8 *’ as the destination; expected ‘uint8’ or an explicit length [-Wsizeof-pointer-memaccess]
memcpy(hub->id.pvnID, id, sizeof id);
^
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/userif.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/netif.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/bridge.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/filter.o
/usr/lib/vmware/modules/source/vmnet-only/filter.c:206:1: error: conflicting types for ‘VNetFilterHookFn’
VNetFilterHookFn(unsigned int hooknum,                 // IN:
^
/usr/lib/vmware/modules/source/vmnet-only/filter.c:64:18: note: previous declaration of ‘VNetFilterHookFn’ was here
static nf_hookfn VNetFilterHookFn;
^
/usr/lib/vmware/modules/source/vmnet-only/filter.c:64:18: warning: ‘VNetFilterHookFn’ used but never defined [enabled by default]
/usr/lib/vmware/modules/source/vmnet-only/filter.c:206:1: warning: ‘VNetFilterHookFn’ defined but not used [-Wunused-function]
VNetFilterHookFn(unsigned int hooknum,                 // IN:
^
make[2]: *** [/usr/lib/vmware/modules/source/vmnet-only/filter.o] Error 1
make[1]: *** [_module_/usr/lib/vmware/modules/source/vmnet-only] Error 2
make[1]: Leaving directory `/usr/src/kernels/3.13.5-200.fc20.x86_64′
make: *** [vmnet.ko] Error 2
[root@daveti vmnet-only]#

Then, we confirmed that the root cause is vmnet.ko. More specifically, VNetFilterHookFn in filter.c causes the building error. And so, what happened there?

2. RCA – root cause analysis

In the latest kernel, e.g, 3.13.5, the signature of netfilter hook function (nf_hookfn) has been changed as the result of code refactoring. However, VNetFilterHookFn is still using the old definition of nf_hookfn. That is why we are not able to compile vmnet.ko. For details, please check http://lxr.linux.no/linux+v3.13.5/include/linux/netfilter.h#L46

3. Fix – diff

[root@daveti vmnet-only]# diff filter.c.orig filter.c
206c206,207
< VNetFilterHookFn(unsigned int hooknum,                 // IN:
—
> //VNetFilterHookFn(unsigned int hooknum,                 // IN:
> VNetFilterHookFn(const struct nf_hook_ops *ops, //daveti: fix
255c256,258
<    transmit = (hooknum == VMW_NF_INET_POST_ROUTING);
—
>    //transmit = (hooknum == VMW_NF_INET_POST_ROUTING);
>    //daveti: fix
>    transmit = (ops->hooknum == VMW_NF_INET_POST_ROUTING);
[root@daveti vmnet-only]#

4. Verify fix – make again

After the patch for the filter.c, we should be able to make the vmnet kernel module now. Let us make sure the fix work.

[root@daveti vmnet-only]# make
Using 2.6.x kernel build system.
make -C /lib/modules/3.13.5-200.fc20.x86_64/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. \
MODULEBUILDDIR= modules
make[1]: Entering directory `/usr/src/kernels/3.13.5-200.fc20.x86_64′
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/filter.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/procfs.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/smac_compat.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/smac.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/vnetEvent.o
CC [M]  /usr/lib/vmware/modules/source/vmnet-only/vnetUserListener.o
LD [M]  /usr/lib/vmware/modules/source/vmnet-only/vmnet.o
Building modules, stage 2.
MODPOST 1 modules
CC      /usr/lib/vmware/modules/source/vmnet-only/vmnet.mod.o
LD [M]  /usr/lib/vmware/modules/source/vmnet-only/vmnet.ko
make[1]: Leaving directory `/usr/src/kernels/3.13.5-200.fc20.x86_64′
make -C $PWD SRCROOT=$PWD/. \
MODULEBUILDDIR= postbuild
make[1]: Entering directory `/usr/lib/vmware/modules/source/vmnet-only’
make[1]: `postbuild’ is up to date
make[1]: Leaving directory `/usr/lib/vmware/modules/source/vmnet-only’
cp -f vmnet.ko ./../vmnet.o
[root@daveti vmnet-only]#

5. Tar and Start

Now it is time to tar the patched vmnet-only directory into vmnet.tar again. Start the VMware Player and enjoy~