I’ve been pulling and pushing container images to harbor registry (that comes with vSphere Integrated Containers 1.1 as told here) for a while without any problem. Last week, I decided to play with native windows containers on Windows 2016 but failed to push the image that I built from a microsoft/nanoserver image.
[As of today, the official version that can be downloaded from My VMware Portal is v1.1.1 and this has been tested with harbor registry version 1.1.0 and 1.1.1. Please note that this might not be supported by VMware]
Here is what it looks like from the client when I tried to push the container image to the registry. All layers get pushed but when the client pushes the manifest, it fails.
manifest blob unknown: blob unknown to registry
What happens under the hood is, whenever a container image is being pushed to the registry, an image manifest is also uploaded that provides a configuration and a set of layers for the container image. The format of the manifest must be recognized by the container registry, if not, the registry returns an 404 error code and errors out the push operation.
As it can be clearly seen from the first screenshot, the initial two layers of microsoft nanoserver container image are defined as foreign layers that require a new image manifest schema (image manifest version 2, schema 2). There is one container running in the vic appliance that is responsible for the registry service, which is vmware/registry:photon-2.6.0 and in order to remediate this issue, we need to swap the registry container version to 2.6.1 which recognizes and accepts this new schema. To accomplish the task;
docker pull vmware/registry:2.6.1-photon docker-compose -f docker-compose.notary.yml -f docker-compose.yml down sed -i -- 's/photon-2.6.0/2.6.1-photon/g' /etc/vmware/harbor/docker-compose.yml docker-compose -f docker-compose.notary.yml -f docker-compose.yml up -d
- First, let’s pull 2.6.1 tagged version of vmware/registry repository from docker hub.
- Stop all running containers gracefully.
- Modify the yml file in order to swap the container image to run.
- Start all running containers.
After we spin up the containers, we can verify that registry container is swapped with the newer one, with docker ps command. So it’s time to push our windows based container image to harbor again… and voila!
P.S. Foreign layers are still skipped and not pushed to the harbor registry because of their nature. This is another topic that I will touch on and propose a workaround to push them to our private registry, in the next post.