Solving ‘UseKeychain’ Not Working for Password Protected SSH Key Logins on macOS

My Problem

Using macOS 10.15, attempting to automatically load a password protected SSH key into ssh-agent by using the SSH configuration option UseKeychain was not working. I had the SSH key’s password stored in the macOS Keychain, and if I manually ran ssh-add -K /path/to/private/key it would load the key without asking me to input a password, proving that they key’s password did exist in the keychain. However, when attempting to SSH to a host that required that private key, I was still being asked for the password to the private key. This is in direct opposition to the intended behavior of using UseKeychain in one’s SSH config file.

I did not want to put ssh-add -A or some variant of ssh-add in my .bashrc file, even though that would have “solved” the problem. The version of OpenSSH that macOS uses has a configuration option provided for just such a desire, and that option is UseKeychain.

My Solution

Of course, first make sure that your SSH config file includes the following:

  UseKeychain yes
  AddKeysToAgent yes

Then, when adding the key to Keychain with ssh-add -K, use the full filesystem path to the key file, not a relative path.

For example, do not use ssh-add -K ~/.ssh/private.key, rather you should use ssh-add -K /Users/username/.ssh/private.key. You may also want to remove all relative pathed entries to the same SSH key from Keychain:

The Long Story

Let’s sort out some terms and get a few things straight. If you’re going to use SSH with public / private keypairs, you have the option of password protecting the private key in the pair. This means that you cannot decrypt the private key to then authenticate your identity to other systems without being in possession of the password. Without a password on the private key, mere possession of the file is good enough to prove your identity, which isn’t really comfortably secure. Any person or software that has access to your filesystem could theoretically snatch that file and masquerade as you. A password protected private key is a type of two-factor authentication. You must have possession of the file and knowledge of the password.

This is great, except if you perform tasks that require the repetitive use of the private key. Such as shelling into a remote system (Ansible, anyone?) or using git. Just an hour or two of using git pull, git push, or git fetch will wear out your keyboard and have you reconsidering your career.

This is where ssh-agent comes in, which keeps track of your private SSH key identities and their passwords. You’d then use ssh-add to stick the identities into ssh-agent. A fuller discussion of these tools is beyond the scope of this post, but suffice it to say that in theory this should reduce your need for typing SSH key passwords to an absolute minimum per reboot.

Apple’s Keychain can store SSH key passwords securely, and Apple’s version of OpenSSH includes an option that you can include in your config file named UseKeychain. This will further reduce your need for typing passwords, even across reboots. It’s possible to virtually never need to type your SSH key’s password again. Except when it doesn’t work.

Some people, myself included, have had an issue where no amount of UseKeychain under any Host * or Host hostname heading in one’s SSH config file would seem to work. After a reboot, ssh-add -l shows now known identities, which is expected. However, ssh user@host should trigger UseKeychain to find the proper key file’s password in keychain and simply allow one to log in to the remote system without providing the local private key’s password. After the first access to a host that needs your private key identity, ssh-add -l should show that the identity is now loaded even though you didn’t type the password.

Opening up the “Keychain Access” application on macOS and then searching for any login with the word ssh in it may reveal that the SSH key identities that are known are all referred to with relative paths. It’s common to see ~/.ssh/private.key or even ../.ssh/private.key depending on where you were in your filesystem when you realized you needed to add the key to Keychain. For some people, this appears to work. Not for me (and others) however.

You may want to try first adding your key with ssh-add -K using the full filesystem path to the key. Then you may want to remove any other references to the key file in Keychain Access that use relative paths.

Solved: VMware Workstation 15 Fails to Compile Kernel Modules with “Failed to build vmmon” and “Failed to build vmnet.”

My Problem:

After updating Fedora 29, VMware Workstation Pro 15 needed to have some kernel modules compiled. However, attempting to install them earned me warning signs on the “Virtual Machine Monitor” and “Virtual Network Device” compilation process, and of course starting the services failed. Logs stated “Failed to build vmmon” and “Failed to build vmnet.”

My Solution:

Digest this SuperUser thread: https://superuser.com/questions/1328401/vmware-workstation-vmmon-broken-on-ubuntu-18-04/1330890

Ultimately, you need to clone this github repo: https://github.com/mkubecek/vmware-host-modules.git. Checkout the proper branch that corresponds to your product and version (for example, I used the Workstation 15.0.3 branch). make then make install within that repo, and finally you’ll want to create a symlink for libz.so.1.

Using two separate answers in the above SuperUser thread, then modifying it for my own purposes, I came up with this:

#!/bin/bash
VMWARE_VERSION=workstation-15.0.3 #This needs to be the actual name of the appropriate branch in mkubecek's GitHub repo for your purposes
TMP_FOLDER=/tmp/patch-vmware
rm -fdr $TMP_FOLDER
mkdir -p $TMP_FOLDER
cd $TMP_FOLDER
git clone https://github.com/mkubecek/vmware-host-modules.git #Use `git branch -a` to find all available branches and find the one that's appropriate for you
cd $TMP_FOLDER/vmware-host-modules
git checkout $VMWARE_VERSION
git fetch
make
sudo make install
sudo rm /usr/lib/vmware/lib/libz.so.1/libz.so.1
sudo ln -s /lib/x86_64-linux-gnu/libz.so.1 /usr/lib/vmware/lib/libz.so.1/libz.so.1
systemctl restart vmware && vmware &

The Long Story:

After a massive cascade of updates on my Fedora 29 workstation that I had been delaying, VMware Workstation Pro 15.0.3 (build-12422535, and Kernal 5.0.3-200.fc29.x86_64 for whatever it’s worth) was unable to launch, instead requiring some kmods to be compiled and loaded:

“Before you can run VMware, several modules must be compiled and loaded into the running kernel.”

Dutifully clicking install earned me these lovely caution signs:

“Virtual Machine Monitor and Virtual Network Device have probably not done what you wanted.”

The services were unable to start, so I checked out the logs:

“Unable to start services. See log file /tmp/vmare-root/vmware-17464.log for details.”

Checking out the logs, I see a build command that failed:

2019-03-28T13:51:03.779-07:00| host-17464| I125: Invoking modinfo on "vmnet".
2019-03-28T13:51:03.781-07:00| host-17464| I125: "/sbin/modinfo" exited with status 256.
2019-03-28T13:51:03.833-07:00| host-17464| I125: Setting destination path for vmmon to "/lib/modules/5.0.3-200.fc29.x86_64/misc/vmmon.ko".
2019-03-28T13:51:03.833-07:00| host-17464| I125: Extracting the vmmon source from "/usr/lib/vmware/modules/source/vmmon.tar".
2019-03-28T13:51:03.839-07:00| host-17464| I125: Successfully extracted the vmmon source.
2019-03-28T13:51:03.839-07:00| host-17464| I125: Building module with command "/usr/bin/make -j12 -C /tmp/modconfig-PG76zy/vmmon-only auto-build HEADER_DIR=/lib/modules/5.0.3-200.fc29.x86_64/build/include CC=/usr/bin/gcc IS_GCC_3=no"
2019-03-28T13:51:05.179-07:00| host-17464| W115: Failed to build vmmon.  Failed to execute the build command.
2019-03-28T13:51:05.181-07:00| host-17464| I125: Setting destination path for vmnet to "/lib/modules/5.0.3-200.fc29.x86_64/misc/vmnet.ko".
2019-03-28T13:51:05.181-07:00| host-17464| I125: Extracting the vmnet source from "/usr/lib/vmware/modules/source/vmnet.tar".
2019-03-28T13:51:05.185-07:00| host-17464| I125: Successfully extracted the vmnet source.
2019-03-28T13:51:05.185-07:00| host-17464| I125: Building module with command "/usr/bin/make -j12 -C /tmp/modconfig-PG76zy/vmnet-only auto-build HEADER_DIR=/lib/modules/5.0.3-200.fc29.x86_64/build/include CC=/usr/bin/gcc IS_GCC_3=no"
2019-03-28T13:51:06.597-07:00| host-17464| W115: Failed to build vmnet.  Failed to execute the build command.

I see two other log files:

total 44
-rw-------. 1 root root 16669 Mar 28 13:51 vmware-17464.log
-rw-------. 1 root root 16555 Mar 28 13:50 vmware-apploader-17464.log
-rw-r-----. 1 root root  2792 Mar 28 13:51 vmware-authdlauncher-20116.log

The apploader log file didn’t appear to have anything of note in it. The authdlauncher… I wasn’t so sure:

2019-03-28T13:51:10.246-07:00| authdlauncher| I125: SOCKET 1 (12) creating new listening socket on port 902
2019-03-28T13:51:10.246-07:00| authdlauncher| W115: SOCKET Could not bind socket, error 98: Address already in use
2019-03-28T13:51:10.246-07:00| authdlauncher| I125: SOCKET Could not create IPv6 listener socket, error 11: Socket bind address already in use
2019-03-28T13:51:10.246-07:00| authdlauncher| I125: SOCKET 2 (12) creating new listening socket on port 902
2019-03-28T13:51:10.246-07:00| authdlauncher| W115: SOCKET Could not bind socket, error 98: Address already in use
2019-03-28T13:51:10.246-07:00| authdlauncher| I125: SOCKET Could not create IPv4 listener socket, error 11: Socket bind address already in use
2019-03-28T13:51:10.246-07:00| authdlauncher| I125: failed to listen on port 902, error 11: Resource temporarily unavailable... Exiting.

Hmm, indeed something is listening on port 902 that looks like VMware:

# sudo lsof -i -P -n | grep 902
vmware-au  1556    root   12u  IPv6  28200      0t0  TCP *:902 (LISTEN)
vmware-au  1556    root   13u  IPv4  28201      0t0  TCP *:902 (LISTEN)

I have my doubts that this is the ultimate problem, but I figured I’d HUP those suckers and see what happens. Of course, that didn’t do anything worthwhile. The processes listening on port 902 are left up and running after the failed installation, not idling there as the cause of the failure.

After tooling around with untaring the kmod and making it by hand, I saw enough errors that made me think there was a serious lack of… something on my install. This just didn’t seem right. As a result, I gave up and Googled, which brought me to this: https://kb.vmware.com/s/article/58533?lang=en_US

The short story to that long thread is that you should uninstall VMware Workstation, make sure that you have the proper prerequisite packages installed, then re-install VMware Workstation. In my case, that did absolutely nothing and I still had the same issue.

A bit more searching and I found this thread from 2017 that seems to have about a year of activity on it: https://communities.vmware.com/thread/568089. Apparently this is a fairly common issue that doesn’t have a very elegant solution. Basically, VMware Workstation’s latest version doesn’t support the kernel that I updated to, and I had to patch it. I blindly followed repomon‘s Apr 4, 2018 7:16 PM post, even though it was for VMware Workstation 12. Of course, that didn’t work well because it ended up compiling for an older version of vmmon and vmnet than what I had previously.

Some more Googling brought me to this SuperUser post: https://superuser.com/questions/1328401/vmware-workstation-vmmon-broken-on-ubuntu-18-04/1330890. Using that as inspiration, I realized that GitHub user mkubecek appears to be keeping up to date with the latest versions of VMware Workstation and Player products and creating appropriate patches to help work through this issue.

The specific script / solution I came to is described in the “My Solution” section above.

Solving Yakuake not Loading .bash_profile

My Problem:

I use Yakuake (pronounced “yaw-quake”) as a drop-down terminal to give me quick access to a shell when working on other things. However, even though my regular terminal (GNOME Terminal as of this blog post) defaults to being an interactive terminal, thus loading .bash_profile, Yakuake wasn’t. None of my aliases, custom shell functions, or anything else in .bash_profile was loading.

My Solution:

Edit your Yakuake profile to launch your shell (probably bash) as a login shell. E.g. /bin/bash -l

The Long Story:

For those unaware, Yakuake is a terminal emulator application, pronounced “yaw quake”, that drops down from the top of your GUI in the style of the old terminal in the ID game “Quake.”

Mashing a simple key combination causes the terminal to drop down over your existing windows. Its quick access combined with the option of terminal transparency allows me to hammer away on problems while reading documentation, or keeping a casual eye on other things that are going on in the background (e.g. watching and contributing to a Slack conversation while I’m pecking away in a shell)

The Yakuake drop down terminal in action, dropping down over existing windows and then being pulled back up out of the way.
Yakuake comes in handy when you need a quick shell while working on other things.

The main problem was that .bash_profile wasn’t being sourced, so my familiar aliases and shell functions weren’t being loaded. Normally this is a sign that your terminal emulator isn’t loading your shell in interactive mode. For example, in GNOME Terminal, you need to edit your profile to “Run command as a login shell” to have .bash_profile consumed.

However, in my case, it was already selected. I was under the mistaken impression that Yakuake was a wrapper around GNOME Terminal. It is, in fact, not. Rather it’s based on KDE Konsole, is it’s own terminal emulator, and thus has it’s own terminal profile settings.

You’ll want to go to to the settings menu in Yakuake and select “Manage Profiles”

From there, you can select a profile to edit (probably the only one there). You’ll want to launch your shell (presumably bash) as a login shell. By default the command invoked with Yakuake launches is /bin/bash. You’ll want to add the -l option to make sure it’s a login shell.

For more information, you’ll want to check out your shell’s manual for information about invocation options. For bash, check this part of the man pages out:¬†https://www.gnu.org/software/bash/manual/html_node/Invoking-Bash.html