Workaround: “Unable to Change Virtual Machine Power State: Cannot Find a Valid Peer Process to Connect to”

My Problem

Attempting to start a virtual machine in VMware Workstation 15 Pro (15.0.3) on a RedHat based Linux workstation caused the following error: “Unable to Change Virtual Machine Power State: Cannot Find a Valid Peer Process to Connect to”

I was able to start other virtual machines in the VM library, however.

My Workaround

Note that this is simply a workaround. I don’t yet know the ultimate cause, but I’m documenting how I workaround it until I or someone else can figure out the ultimate cause of this problem.

First, check to see if the virtual machine is actually running, in spite of there being no visual indicators within VMware Workstation: vmrun list

You’ll probably see that the virtual machine is running. If you don’t, then this workaround isn’t likely to help you. Attempt to shut the running virtual machine down softly: vmrun stop /path/to/virtual_machine.vmx soft

After that, you should be able to start the machine again, until the next time it crashes for unknown reasons. More news as I discover it.

Dumping Grounds (Turn Back Now):

I’ll dump some of my notes here and they’ll be updated periodically as I find out more info about this issue. You’re completely safe to ignore everything past this point. Abandon all hope, ye who proceed.

I had recently upgraded from Fedora 29 to Fedora 30, and was experiencing some minor instability with my main workstation. I’m not sure if that was the ultimate cause of this issue, but I’m suspicious since I never had this issue until after the upgrade.

My first act was to go to the Help menu, select the “Support” menu and then “Collect Support Data…” I chose to collect data for the specific VM that was having this issue. This took quite a while, by my standards. About 20 minutes. It basically creates a giant zipped dump of pertinent files across your physical machine that pertain to VMware and that specific virtual machine. It’s not super easy to parse and know what to look for.

I searched through /var/log/vmware/ for any clues in any of the log files found therein. Grepping for all files that had the pertinent virtual machine’s name, and looking for surrounding context didn’t turn anything up.

I attempted to start the vmware-workstation-server service but that failed. I don’t think that’s the issue since the virtual machine isn’t a shared VM.

I tried vmrun list and saw that the Windows VM was actually listed as running. I stopped it soft: vmrun stop /path/to/my/virtual_machine.vmx soft and was then able to start the virtual machine. I’m not sure what’s causing the crash, and what’s causing the crash of VMware Workstation Pro, and why when I start it back up it doesn’t appear to know that the VM it was previously working with is actually running.

Solved: “bad input file size” When Attempting to `setfont` to a New Console Font

My Problem

In a Linux distribution of one kind or another, when attempting to set a new console font in a TTY, you may received the following error:

# setfont -32 ter-u32n.bdf
bad input file size

My Solution

First, if you’re coming to this blog post because you’re attempting to install larger Terminus fonts for your TTY, you probably just want to search your distribution’s package manager for Terminus, specifically the console fonts package:

$ yum search terminus
== Name Matched: terminus ==
terminus-fonts.noarch : Clean fixed width font
terminus-fonts-grub2.noarch : Clean fixed width font (grub2 version)
terminus-fonts-console.noarch : Clean fixed width font (console version)
$ yum install terminus-fonts-console

However if you’re coming to this blog post for other reasons, then you’re probably attempting to setfont with a .bdf file or just something that isn’t a .psf file. You most likely need to follow the instructions for your font, in my case Terminus, to make the files into the proper .psf format.The Linux From Scratch project has a good quick primer on the topic that you can use to mine for search terms and further information.

With my specific font, what worked for me was:

$ sudo ./configure --psfdir=/usr/lib/kbd/consolefonts
$ sudo make -j8 psf
# Stuff happens here
$ sudo make install-psf

After that, I had the fonts installed into my /usr/lib/kbd/consolefonts directory and was able to setfont and further change my TTY font to my preferences.

Solved: Attempting to Install and Configure Wireguard Fails with “Unknown device type” and “FATAL: Module wireguard not found in directory”

My Problem

Attempting to install and use Wireguard (version 0.0.20190406-1) on Fedora release 29 is unsuccessful with a variety of symptoms. The first being:

ip link add dev wg0 type wireguard
Error: Unknown device type.

Attempting to get some info about the module with modprobe shows:

$ modprobe wireguard
modprobe: FATAL: Module wireguard not found in directory /lib/modules/5.0.4-2004

The dkms tool shows that the wireguard module is added:

$ dkms status
wireguard, 0.0.20190406: added

However, attempting to build it shows:

$ dkms build wireguard/0.0.20190406
Error! echo
Your kernel headers for kernel 5.0.4-200.fc29.x86_64 cannot be found at /lib/modules/5.0.4-200.fc29.x86_64/build or /lib/modules/5.0.4-200.fc29.x86_64/.

My Solution

Make sure that your running kernel and your kernel headers are the same version, or at least that the running version of the kernel is newer than your kernel headers.

For example, I’m running on a RedHat based system, and checked the following:

$ uname --kernel-release
5.0.4-200.fc29.x86_64

But then the kernel headers were newer:

$ rpm -q kernel-headers
kernel-headers-5.0.9-200.fc29.x86_64

My solution was to yum update the kernel and reboot. I didn’t have to re-install the headers or the wireguard packages. Another possible solution would have been to manually install 5.0.4 kernel headers, but that would require uninstalling packages that marked 5.0.9 kernel headers as a dependency. I believe the cleaner solution is to simply update the kernel.

The Long Story

FirstI checked that I even had kernel headers installed in the first place:

$ rpm -q kernel-headers
kernel-headers-5.0.9-200.fc29.x86_64

Well that’s interesting, because:

$ uname --kernel-release
5.0.4-200.fc29.x86_64

So I’m running kernel 5.0.4, but the kernel-headers package that I’m offered is for 5.0.9. I attempted to install the specific kernel header package by version:

yum install kernel-headers-5.0.4-200.fc29.x86_64
[...]
No match for argument: kernel-headers-5.0.4-200.fc29.x86_64

At this point, I had two viable options.

  1. I could update the running kernel, since 5.0.10-200.fc29 was released and waiting for me.
  2. I could go into Fedora’s build system, Koji, and pull out the specific kernel headers package that I needed to then install manually.

Choosing #2, however, would require me to uninstall the current 5.0.9 kernel headers, and anything that had it as a dependency. This includes things like binutils and gcc, among many others. I decided to update the system. A quick yum update and reboot later, and:

$ uname -or
5.0.10-200.fc29.x86_64 GNU/Linux

My only concern was that the headers that are in the official yum repo are 5.0.9; a minor version behind the new kernel:

rpm -q kernel-headers
kernel-headers-5.0.9-200.fc29.x86_64

Nevertheless, my fears were allayed with dkms:

$ dkms status
wireguard, 0.0.20190406, 5.0.10-200.fc29.x86_64, x86_64: installed

Previously, wireguard had only been added, but not successfully installed. I quickly tried to add a wireguard interface:

$ ip link add dev wg0 type wireguard
$ ip link show wg0
3: wg0: <POINTOPOINT,NOARP> mtu 1420 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/none

Success!

Solved: VMware Workstation 15 Fails to Compile Kernel Modules with “Failed to build vmmon” and “Failed to build vmnet.”

My Problem:

After updating Fedora 29, VMware Workstation Pro 15 needed to have some kernel modules compiled. However, attempting to install them earned me warning signs on the “Virtual Machine Monitor” and “Virtual Network Device” compilation process, and of course starting the services failed. Logs stated “Failed to build vmmon” and “Failed to build vmnet.”

My Solution:

Digest this SuperUser thread: https://superuser.com/questions/1328401/vmware-workstation-vmmon-broken-on-ubuntu-18-04/1330890

Ultimately, you need to clone this github repo: https://github.com/mkubecek/vmware-host-modules.git. Checkout the proper branch that corresponds to your product and version (for example, I used the Workstation 15.0.3 branch). make then make install within that repo, and finally you’ll want to create a symlink for libz.so.1.

Using two separate answers in the above SuperUser thread, then modifying it for my own purposes, I came up with this:

#!/bin/bash
VMWARE_VERSION=workstation-15.0.3 #This needs to be the actual name of the appropriate branch in mkubecek's GitHub repo for your purposes
TMP_FOLDER=/tmp/patch-vmware
rm -fdr $TMP_FOLDER
mkdir -p $TMP_FOLDER
cd $TMP_FOLDER
git clone https://github.com/mkubecek/vmware-host-modules.git #Use `git branch -a` to find all available branches and find the one that's appropriate for you
cd $TMP_FOLDER/vmware-host-modules
git checkout $VMWARE_VERSION
git fetch
make
sudo make install
sudo rm /usr/lib/vmware/lib/libz.so.1/libz.so.1
sudo ln -s /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/vmware/lib/libz.so.1/libz.so.1
systemctl restart vmware && vmware &

The Long Story:

After a massive cascade of updates on my Fedora 29 workstation that I had been delaying, VMware Workstation Pro 15.0.3 (build-12422535, and Kernal 5.0.3-200.fc29.x86_64 for whatever it’s worth) was unable to launch, instead requiring some kmods to be compiled and loaded:

“Before you can run VMware, several modules must be compiled and loaded into the running kernel.”

Dutifully clicking install earned me these lovely caution signs:

“Virtual Machine Monitor and Virtual Network Device have probably not done what you wanted.”

The services were unable to start, so I checked out the logs:

“Unable to start services. See log file /tmp/vmare-root/vmware-17464.log for details.”

Checking out the logs, I see a build command that failed:

2019-03-28T13:51:03.779-07:00| host-17464| I125: Invoking modinfo on "vmnet".
2019-03-28T13:51:03.781-07:00| host-17464| I125: "/sbin/modinfo" exited with status 256.
2019-03-28T13:51:03.833-07:00| host-17464| I125: Setting destination path for vmmon to "/lib/modules/5.0.3-200.fc29.x86_64/misc/vmmon.ko".
2019-03-28T13:51:03.833-07:00| host-17464| I125: Extracting the vmmon source from "/usr/lib/vmware/modules/source/vmmon.tar".
2019-03-28T13:51:03.839-07:00| host-17464| I125: Successfully extracted the vmmon source.
2019-03-28T13:51:03.839-07:00| host-17464| I125: Building module with command "/usr/bin/make -j12 -C /tmp/modconfig-PG76zy/vmmon-only auto-build HEADER_DIR=/lib/modules/5.0.3-200.fc29.x86_64/build/include CC=/usr/bin/gcc IS_GCC_3=no"
2019-03-28T13:51:05.179-07:00| host-17464| W115: Failed to build vmmon.  Failed to execute the build command.
2019-03-28T13:51:05.181-07:00| host-17464| I125: Setting destination path for vmnet to "/lib/modules/5.0.3-200.fc29.x86_64/misc/vmnet.ko".
2019-03-28T13:51:05.181-07:00| host-17464| I125: Extracting the vmnet source from "/usr/lib/vmware/modules/source/vmnet.tar".
2019-03-28T13:51:05.185-07:00| host-17464| I125: Successfully extracted the vmnet source.
2019-03-28T13:51:05.185-07:00| host-17464| I125: Building module with command "/usr/bin/make -j12 -C /tmp/modconfig-PG76zy/vmnet-only auto-build HEADER_DIR=/lib/modules/5.0.3-200.fc29.x86_64/build/include CC=/usr/bin/gcc IS_GCC_3=no"
2019-03-28T13:51:06.597-07:00| host-17464| W115: Failed to build vmnet.  Failed to execute the build command.

I see two other log files:

total 44
-rw-------. 1 root root 16669 Mar 28 13:51 vmware-17464.log
-rw-------. 1 root root 16555 Mar 28 13:50 vmware-apploader-17464.log
-rw-r-----. 1 root root  2792 Mar 28 13:51 vmware-authdlauncher-20116.log

The apploader log file didn’t appear to have anything of note in it. The authdlauncher… I wasn’t so sure:

2019-03-28T13:51:10.246-07:00| authdlauncher| I125: SOCKET 1 (12) creating new listening socket on port 902
2019-03-28T13:51:10.246-07:00| authdlauncher| W115: SOCKET Could not bind socket, error 98: Address already in use
2019-03-28T13:51:10.246-07:00| authdlauncher| I125: SOCKET Could not create IPv6 listener socket, error 11: Socket bind address already in use
2019-03-28T13:51:10.246-07:00| authdlauncher| I125: SOCKET 2 (12) creating new listening socket on port 902
2019-03-28T13:51:10.246-07:00| authdlauncher| W115: SOCKET Could not bind socket, error 98: Address already in use
2019-03-28T13:51:10.246-07:00| authdlauncher| I125: SOCKET Could not create IPv4 listener socket, error 11: Socket bind address already in use
2019-03-28T13:51:10.246-07:00| authdlauncher| I125: failed to listen on port 902, error 11: Resource temporarily unavailable... Exiting.

Hmm, indeed something is listening on port 902 that looks like VMware:

# sudo lsof -i -P -n | grep 902
vmware-au  1556    root   12u  IPv6  28200      0t0  TCP *:902 (LISTEN)
vmware-au  1556    root   13u  IPv4  28201      0t0  TCP *:902 (LISTEN)

I have my doubts that this is the ultimate problem, but I figured I’d HUP those suckers and see what happens. Of course, that didn’t do anything worthwhile. The processes listening on port 902 are left up and running after the failed installation, not idling there as the cause of the failure.

After tooling around with untaring the kmod and making it by hand, I saw enough errors that made me think there was a serious lack of… something on my install. This just didn’t seem right. As a result, I gave up and Googled, which brought me to this: https://kb.vmware.com/s/article/58533?lang=en_US

The short story to that long thread is that you should uninstall VMware Workstation, make sure that you have the proper prerequisite packages installed, then re-install VMware Workstation. In my case, that did absolutely nothing and I still had the same issue.

A bit more searching and I found this thread from 2017 that seems to have about a year of activity on it: https://communities.vmware.com/thread/568089. Apparently this is a fairly common issue that doesn’t have a very elegant solution. Basically, VMware Workstation’s latest version doesn’t support the kernel that I updated to, and I had to patch it. I blindly followed repomon‘s Apr 4, 2018 7:16 PM post, even though it was for VMware Workstation 12. Of course, that didn’t work well because it ended up compiling for an older version of vmmon and vmnet than what I had previously.

Some more Googling brought me to this SuperUser post: https://superuser.com/questions/1328401/vmware-workstation-vmmon-broken-on-ubuntu-18-04/1330890. Using that as inspiration, I realized that GitHub user mkubecek appears to be keeping up to date with the latest versions of VMware Workstation and Player products and creating appropriate patches to help work through this issue.

The specific script / solution I came to is described in the “My Solution” section above.

Solving Yakuake not Loading .bash_profile

My Problem:

I use Yakuake (pronounced “yaw-quake”) as a drop-down terminal to give me quick access to a shell when working on other things. However, even though my regular terminal (GNOME Terminal as of this blog post) defaults to being an interactive terminal, thus loading .bash_profile, Yakuake wasn’t. None of my aliases, custom shell functions, or anything else in .bash_profile was loading.

My Solution:

Edit your Yakuake profile to launch your shell (probably bash) as a login shell. E.g. /bin/bash -l

The Long Story:

For those unaware, Yakuake is a terminal emulator application, pronounced “yaw quake”, that drops down from the top of your GUI in the style of the old terminal in the ID game “Quake.”

Mashing a simple key combination causes the terminal to drop down over your existing windows. Its quick access combined with the option of terminal transparency allows me to hammer away on problems while reading documentation, or keeping a casual eye on other things that are going on in the background (e.g. watching and contributing to a Slack conversation while I’m pecking away in a shell)

The Yakuake drop down terminal in action, dropping down over existing windows and then being pulled back up out of the way.
Yakuake comes in handy when you need a quick shell while working on other things.

The main problem was that .bash_profile wasn’t being sourced, so my familiar aliases and shell functions weren’t being loaded. Normally this is a sign that your terminal emulator isn’t loading your shell in interactive mode. For example, in GNOME Terminal, you need to edit your profile to “Run command as a login shell” to have .bash_profile consumed.

However, in my case, it was already selected. I was under the mistaken impression that Yakuake was a wrapper around GNOME Terminal. It is, in fact, not. Rather it’s based on KDE Konsole, is it’s own terminal emulator, and thus has it’s own terminal profile settings.

You’ll want to go to to the settings menu in Yakuake and select “Manage Profiles”

From there, you can select a profile to edit (probably the only one there). You’ll want to launch your shell (presumably bash) as a login shell. By default the command invoked with Yakuake launches is /bin/bash. You’ll want to add the -l option to make sure it’s a login shell.

For more information, you’ll want to check out your shell’s manual for information about invocation options. For bash, check this part of the man pages out: https://www.gnu.org/software/bash/manual/html_node/Invoking-Bash.html

Back to the Startup and Freelance Scene

2014 was a watershed year for me. I had been a full time freelancer since 2010, learning a wide variety of technologies as well as the intricacies of developing a consulting business.

However, I had an opportunity to get into an exciting Y-Combinator startup that I couldn’t pass up. In May of 2014, I joined MongoHQ, a database as a service company focusing on hosted MongoDB deployments. We later rebranded to Compose when we started hosting more than MongoDB. First with Elasticsearch, and then PostgreSQL, Redis, and more.

In 2015 we were acquired by IBM, and it was an amazing ride through the acquisition and integration process. We added more people, obtained new and loftier goals, and had a great time succeeding in a totally new environment.

After three years of working on the Compose product within IBM, I couldn’t ignore the pull back to the startup and freelance scene. As of Oct 15th, 2018 I’m back to the wilds of uncertainty and terror excitement. I’m taking up residence at a coworking space / startup incubator called Galvanize, specifically in their Phoenix campus

I’m back to freelancing and startup work, and perhaps hunting a unicorn. Or at least a very pretty horse. If anyone is currently on the same path, I’d love to talk with you and see how things are going in 2018 and on into 2019. If anyone is in need of a consultant / contractor / freelancer who can poke a bit at AWS, Azure, MongoDB, Elasticsearch, Redis, PostgreSQL, JavaScript, Ruby, Go, Linux, and a litany of other trendy and cloudy technologies, reach out: me@wesley.sh

If anyone is specifically in the Phoenix, Arizona startup scene, stop by Galvanize and let’s grab some lunch and talk about sunburns and hiking. =)

Solved: WordPress – “An unexpected error occurred.” when installing plugins, themes, and more.

My Problem

Attempting to add things to WordPress like plugins or themes causes the following error:

An unexpected error occurred. Something may be wrong with WordPress.org or this server’s configuration. If you continue to have problems, please try the support forums.

My Solution

Check your SELinux audit logs for signs of denials. Your web server software (probably Apache / httpd) or a module being used by the software is most likely having outbound connection attempts denied.

Continue reading “Solved: WordPress – “An unexpected error occurred.” when installing plugins, themes, and more.”