Tag: docker

Solving AWX Error “unexpected EOF make: *** [Makefile:508: docker-compose-build] Error 1”

My Problem

Attempting to build the containers for a Docker Compose installation of AWX fails with the error:

unexpected EOF
make: *** [Makefile:508: docker-compose-build] Error 1

My Solution

Give up on trying to clone a specific tag / release and simply clone master. As of the writing of this blog post, there’s both an error in the documentation concerning using tagged releases as well as a disconnect between the tagged release version numbering vs the official Docker image naming scheme.

If you really want to use a specific image, check out the published images here and then find a specific release that you want. For example, release 4.2. Edit the AWX Makefile and change the line that assigns a value to the variable COMPOSE_TAG towards the top of the file. Assign it the value of the tag that you desire. For example COMPOSE_TAG ?= release_4.2 which will then pull image ghcr.io/ansible/awx_devel:release_4.2

The Long Story

I’m attempting to create a test installation of Ansible’s AWX project following the Docker Compose instructions. After installing the required prerequisite packages, I follow the official documentation and clone the latest stable branch (at the time of this blog post):

git clone https://github.com/ansible/awx.git --branch 21.4.0

From there I modify the inventory file using settings appropriate to my environment:

localhost ansible_connection=local ansible_python_interpreter="/usr/bin/env python"

[all:vars]

# pg_password="mega_Secret"
# broadcast_websocket_secret="giga_Secret"
# secret_key="tera_Secret"

Then I run make docker-compose-build and wait for a few minutes before hitting the error:

unexpected EOF
make: *** [Makefile:508: docker-compose-build] Error 1

Let’s examine the relevant lines around 508:

    506 docker-compose-build:
    507         ansible-playbook tools/ansible/dockerfile.yml -e build_dev=True -e receptor_image=$(RECEPTOR_IMAGE)
    508         DOCKER_BUILDKIT=1 docker build -t $(DEVEL_IMAGE_NAME) \
    509             --build-arg BUILDKIT_INLINE_CACHE=1 \
    510             --cache-from=$(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG) .

As we can see, the docker-compose-build target starts on 506, and the whole target only lasts until line 510. It’s really just two commands. The first being a call to run the dockerfile.yml playbook that builds the Dockerfile, and then docker build to actually create the Docker image. Simple enough, and yet something is causing it to bomb out.

I decided to run these commands manually in my shell, but to do so meant I also had to manually populate the variables. Some variables were themselves created from a combination of other variables. Ultimately, this is what I had to create:

  • RECEPTOR_IMAGE is assigned on line 30: RECEPTOR_IMAGE ?= quay.io/ansible/receptor:devel
  • DEVEL_IMAGE_NAME is assigned on line 28: DEVEL_IMAGE_NAME ?= $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG)
  • DEV_DOCKER_TAG_BASE is assigned on line 27: DEV_DOCKER_TAG_BASE ?= ghcr.io/ansible
  • COMPOSE_TAG is assigned on line 12: COMPOSE_TAG ?= $(GIT_BRANCH)
  • GIT_BRANCH is assigned on line 5: GIT_BRANCH ?= $(shell git rev-parse --abbrev-ref HEAD)

Ultimately I assigned the variables these values:

RECEPTOR_IMAGE="quay.io/ansible/receptor:devel"
DEV_DOCKER_TAG_BASE="ghcr.io/ansible"
GIT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
COMPOSE_TAG="$GIT_BRANCH"
DEVEL_IMAGE_NAME="$DEV_DOCKER_TAG_BASE/awx_devel:$COMPOSE_TAG"

Let’s double check to make sure they look right:

echo -e "RECEPTOR_IMAGE = $RECEPTOR_IMAGE \nDEV_DOCKER_TAG_BASE = $DEV_DOCKER_TAG_BASE \nGIT_BRANCH = $GIT_BRANCH \nCOMPOSE_TAG = $COMPOSE_TAG \nDEVEL_IMAGE_NAME = $DEVEL_IMAGE_NAME"
RECEPTOR_IMAGE = quay.io/ansible/receptor:devel
DEV_DOCKER_TAG_BASE = ghcr.io/ansible
GIT_BRANCH = HEAD
COMPOSE_TAG = HEAD
DEVEL_IMAGE_NAME = ghcr.io/ansible)/awx_devel:HEAD

Now I can continue on by manually executing the next command that the docker-compose-build target performs:

ansible-playbook tools/ansible/dockerfile.yml -e build_dev=True -e receptor_image=$RECEPTOR_IMAGE

The four tasks complete successfully and a Dockerfile and some config files are created. Next up, it’s time to build the Docker image:

DOCKER_BUILDKIT=1 docker build -t $DEVEL_IMAGE_NAME --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from=$DEV_DOCKER_TAG_BASE/awx_devel:$COMPOSE_TAG .

And the build process kicks off. After a few minutes, we bomb out with:

------
 > importing cache manifest from ghcr.io/ansible/awx_devel:HEAD:
------
importing cache manifest from ghcr.io/ansible/awx_devel:HEAD:
unexpected EOF

Maybe for the first time in this process I really start to think. More importantly, I start to look up at the tidal wave of Docker build logs in my terminal. Towards the top, I see this:

=> ERROR importing cache manifest from ghcr.io/ansible/awx_devel:HEAD

Trying to manually pull that image receives a slightly more helpful error:

docker pull ghcr.io/ansible/awx_devel:HEAD
Error response from daemon: manifest unknown

Maybe if we don’t use a tag and just let docker use the default tag of latest?

docker pull ghcr.io/ansible/awx_devel
Using default tag: latest
Error response from daemon: manifest unknown

Looking at the latest published container images for the AWX project and I see what’s wrong:

There’s no HEAD tag, there’s no latest. Instead there’s devel among other release and version tags.

Apparently we’re not getting what the Makefile authors expected when we populate the GIT_BRANCH variable with git rev-parse --abbrev-ref HEAD.

Let’s refresh our memories from a few minutes ago when I explained how I cloned the AWX repository. The official documentation very clearly states:

We generally recommend that you view the releases page and clone the latest stable tag, e.g., git clone -b x.y.z https://github.com/ansible/awx.git

Please note that deploying from HEAD (or the latest commit) is not stable, and that if you want to do this, you should proceed at your own risk.

https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md#clone-the-repo

The documentation has simultaneously told us to check out a a specific branch, and also scared us away from using HEAD. This I dutifully did:

git clone https://github.com/ansible/awx.git /awx --branch 21.4.0

However, cloning a tag puts you in a detached HEAD state. Thus when you git rev-parse --abbrev-ref HEAD you get back… well… HEAD. The logic of the Makefile then searches for a base Docker image named ghcr.io/ansible/awx_devel:HEAD which does not exist.

If we were bad users and just cloned MASTER like we were told not to, what would we get as the value of `COMPOSE_TAG“?

git rev-parse --abbrev-ref HEAD
devel

And that would ‘solve’ the issue. Except, would it? Now we’re using the HEAD docker image, which we were told not to. If we look at all of the tagged images we see something curious. None of the image tags appear to be named in any way similar to the branch tags. What’s release_4.2? What’s release_3.8.7? Looking at tagged releases shows a completely different numbering system. Even if I did get the tagged release version with something like git show -s --pretty=%d HEAD, there would be no docker image to correlate it to. I can’t even find correlations between the SHAs for tags vs images.

The AWX project uses GitHub Actions to build official images. One action builds and pushes the Docker images, and we can see that it’s only triggered on pushes to branches named devel or release_*. Furthermore, it tags the image with GITHUB_REF##, which now explains why images are named release_4.2 for example and not anything to do with the tagged release names like the documentation would indicate.

It was then that I began to realize something humorous. Every blog and forum post I’ve encountered in the last month which explained how an individual installed the Docker Compose version of AWX always cloned from master. No one seems to have read or cared about the documentation enough to follow directions to deploy only tagged releases. When I cloned master and ran the docker-compose-build target, everything completed perfectly.

Further proof that reading the documentation is overrated and one should just wing it whenever possible.

Docker: Error response from daemon: manifest not found: manifest unknown

I was seeing the rather character dense and yet information sparse error from Docker:

Error response from daemon: manifest for graylog/graylog:latest not found: manifest unknown: manifest unknown

Yes, I was hacking around with Graylog in this specific instance.

As it turns out, Graylog doesn’t have a latest tag on Dockerhub, and Docker will add :latest to any image that you attempt to pull without explicitly adding a tag.

What happens if there’s no :latest tag on the registry? You get the above error. Search your container registry and repo for what tags they use and find the one that makes most sense for you.