Building a multi-container application with vSphere Integrated Containers v1.1.1

Running demo containers on vSphere Integrated Containers (aka VIC) is pretty exciting but eventually the time will come to get our hands dirty and build an actual application that needs to live on a virtual container host (aka VCH) in a resilient way. So in this post,  I’ll try to cover the steps in order to build an application with two layers (application/ui layer and a database layer) and how to workaround the challenges that VIC engine imposes. High level objectives are;

  • Select a proper application
  • Open the required ports on ESXi servers
  • Create and configure a VCH
  • Create docker-compose configuration file
  • Run the containers and test the application

The component versions that I use for this task is;

  • vSphere Integrated Containers: 1.1.1, build 56a309f
  • docker-compose: 1.13.0, build 1719ceb

Selection of a proper application:

The application that I’m going to deploy is an open source project on GitHub, called Gogs. Gogs is defined by the creators as a painless, self-hosted Git service. The goal of the project is to make the easiest, fastest, and most painless way of setting up a self-hosted Git service, written with Go. So this enables an independent binary distribution across all platforms that Go supports, including Linux, Mac OS X, Windows and even ARM.

There are many ways to install Gogs, such as installation from the source code, packages or with Vagrant but of course we will focus on installation as a Docker container. Normally, one container is more than enough to run Gogs but it also supports remote databases so we will take advantage of that to create a two-layered application.

Required ports on ESXi servers:

Before we start, I assume that VIC appliance is deployed and configured properly. Also we need a vic-machine in order to run vic commands and this is going to be a CentOS box in my environment. If we still don’t have the vic-machine binaries, we can easily get them with curl from the vic appliance. (sddcvic is my appliance and if using self-signed certificates, insert –insecure at the end of the curl command)

curl -L -O https://sddcvic:9443/vic_1.1.1.tar.gz /root --insecure
gzip -d /root/vic_1.1.1.tar.gz
tar -xvf vic_1.1.1.tar

ESXi hosts communicate with the VCHs through port 2377 via Serial Over LAN. For the deployment of a VCH to succeed, port 2377 must be open for outgoing connections on all ESXi hosts before you run vic-machine-xxx create to deploy a VCH. The vic-machine utility includes an update firewall command, that we can use to modify the firewall on a standalone ESXi host or all of the ESXi hosts in a cluster. This command will allow tcp/2377 outgoing connctions for all ESXi servers that exist under the cluster defined with compute-resource option.

./vic-machine-linux update firewall \
    --target=sddcvcs.domain.sddc \
    --user=orcunuso \
    --compute-resource=SDDC.pCluster \
    --thumbprint="3F:6E:2F:16:FA:76:53:74:18:3F:26:9D:1A:58:40:AD:E5:D8:3E:52" \

Initially, we may not know the thumbprint of the vcenter server. The trick here is to run the command without thumbprint option, get it from the error message, add the option and re-run the command.

Deployment of the VCH:

Normally, a few options (such as name, target, user and tls support mode) will suffice. But in order to deploy a more customized VCH, there are many options that we can provide (here is the full list). Below is the command that I used to deploy mine.

./vic-machine-linux create \
    --name=vch02.domain.sddc \
    --target=sddcvcs.domain.sddc/SDDC.Datacenter \
    --thumbprint="3F:6E:2F:16:FA:76:53:74:18:3F:26:9D:1A:58:40:AD:E5:D8:3E:52" \
    --user=orcunuso \
    --compute-resource=SDDC.Container \
    --image-store=VMFS01/VCHPOOL/vch02 \
    --volume-store=VMFS01/VCHPOOL/vch02:default \
    --bridge-network=LSW10_Bridge \
    --public-network=LSW10_Mgmt \
    --client-network=LSW10_Mgmt \
    --management-network=LSW10_Mgmt \
    --dns-server= \
    --public-network-ip= \
    --public-network-gateway= \
    --registry-ca=/root/ssl/sddcvic.crt \

The option –no-tls disables TLS authentication of connections between the docker clients and the VCH, so VCH use neither client nor server certificates. In this case, docker clients connect to the VCH via port 2375, instead of port 2376.

Disabling TLS authentication is not the recommended way of deploying VCH because thus, any docker client can connect to this VCH in an insecure manner. But in this practice, we will use docker-compose to build our application, and I’ve encountered many issues with TLS enabled VCH. It’s on my to-do list.

Creating docker-compose.yml configuration file:

Docker-compose is a tool for defining and running multi-container docker applications. With compose, we use a YAML file to configure our application’s services. Then, using a single command, we can create and start all the services from our configuration. First we need to provide the binary to run docker-compose if it is not in place. Simply run curl command to download from GitHub.

curl -L`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

Now we need to create our configuration file (docker-compose.yml).

version: "3"
    image: sddcvic.domain.sddc/gogs/gogs
    container_name: gogsapp
    restart: on-failure
      - "gogsdb"
      - "gogsvol2:/data"
      - "10080:3000"
      - "10022:22"
      - "gogsnet"
    image: sddcvic.domain.sddc/library/postgres:9.6
    container_name: gogsdb
      - POSTGRES_PASSWORD=gogsdbpassword
      - POSTGRES_USER=gogsuser
      - POSTGRES_DB=gogsdb
      - PGDATA=/var/lib/postgresql/data/data
    restart: on-failure
      - "gogsvol1:/var/lib/postgresql/data"
      - "gogsnet"
    driver: bridge
    driver: vsphere
    driver: vsphere

From the architectural point of view, we have two services that communicate via a bridge network that we call “gogsnet”. Gogs application requires two ports to be exposed to the real world, which are 22 and 3000. Postgres container accepts connections via port 5432 but that port does not have to be exposed because it is an internal communication.

To make this application more reliable, it is a good practice to use persistent volumes so whenever we recreate the containers from the images that are already pushed to the private registry, the data will not be lost and the application will resume as expected. For the sake of this purpose, we create two volumes, gogsvol1 and gogsvol2, and map these volumes to relevent directories. The default volume driver with VIC is vsphere so what is going to happen is that VIC will create two VMDK files in the locations that we provided with the –volume-store option during the VCH deployment phase and attach those VMDKs to the containers.

Normally, we are not supposed to specify an alternate location for postgres database files but in this case, VIC uses VMDK for the volumes and as it is a new volume, it will have a lost+found folder which causes postgres init scripts to quit with exit code 1, so the container exits as well. That is the reason why we use PGDATA environment variable and specify a subdirectory to contain the data.

At the end of the day, this is how it will look like:

Run and test the application:

Before running the app, let’s make sure that everything is in place. Our check list includes;

  • A functional registry server
  • Required images that are ready to be pulled from registry server (gogs and postgres)
  • A functional VCH with no-tls support
  • Docker-compose and yaml file

Now let’s build our application

docker-compose -f /root/gogs/docker-compose-vic.yml up -d

Excellent!!! Our containers are up and running. Let’s connect to our application via port tcp/10080 and make the initial configurations that needs to be done for the first-time run. We give the options that we specified as environment variables during the build process of postgres container.

And voila!!! Our two-layered, containerized, self-hosted git service is up and running on virtual container host backed by vSphere Integrated Containers registry service (aka Harbor). Enjoy 🙂

Push Windows image layers to Harbor registry

If you ever try to push Windows based images to a private registry, only the custom layers that you create will be pushed, not the initial layers that come with the base image. To demonstrate this phenomenon;

Pull microsoft/nanoserver from Docker Hub.

docker pull microsoft/nanoserver

Tag the base image appropriately (sddcvic is my private Harbor registry and windows is the name of the project that I created before)

docker tag microsoft/nanoserver sddcvic.domain.sddc/windows/nanoserver:3.0

Login to Harbor and push the image that we tagged

docker login -u admin -p password sddcvic.domain.sddc 
docker push sddcvic.domain.sddc/windows/nanoserver:3.0
docker logout sddcvic.domain.sddc

After the push command, the first two layers which are marked as foreign are skipped and not pushed to the registry. This is a common phenomenon, so the container registry service does not explicitly have to be Harbor, it could happen with any private registry service.

P.S. If you use Harbor as the image registry and your push operation errors out with a “blob unknown to registry” message, please refer to my previous post.

As of Docker Registry v2.5.0, a new version of docker image manifest (Image Manifest Version 2, Schema 2) was introduced. Whenever an image gets pushed to a repository, an image manifest is also uploaded that provides a configuration and a set of layers for the container image. One of the most important changes with this schema version is the introduction of foreign layers that are widely used by Windows based containers. As their nature, foreign layers cannot be pushed to any private registry other than the URLs that exist in their descriptor files. This is specifically necessary to support downloading Windows base layers from Microsoft servers, since only Microsoft is allowed to distribute them. This makes any image built on them require internet connectivity to download the bits which is not appropriate for most enterprise environments. Even if it’s allowed, the amount of data needs to be downloaded to the docker host caches every time from the internet will be huge (windows images does not have a good reputation about image sizes), hence it contradicts the main benefits of containers such as mobility and flexibility. Luckily, there is a trick to workaround this issue.

Disclaimer: This procedure might not be recommended or supported by Microsoft, Docker or VMware!!!

In order to make a layer “non-foreign“, we need to manipulate its descriptor file. If there are many images and layers cached on the docker host server, it would take an effort to find the right descriptor.json files. The descriptor files exist under the folder: C:\ProgramData\docker\image\windowsfilter\layerdb\sha256\*\

First we need to get the hashes.

docker inspect --format "{{.RootFS.Layers}}" microsoft/nanoserver

The hashes that we get here are diff (layer created during the docker image build process) hashes that can be found exactly in diff files in the same folder as descriptor.json. The below commands will search diff hashes within all the diff files and return with the full path that will be useful to locate the descriptor files.

Select-String -Pattern "6c357baed9f5177e8c8fd1fa35b39266f329535ec8801385134790eb08d8787d" -Path "C:\ProgramData\docker\image\windowsfilter\layerdb\sha256\*\diff"
Select-String -Pattern "0a051a1149b43239af90a8c11824a685a737c9417387caea392b8c8fee7e3889" -Path "C:\ProgramData\docker\image\windowsfilter\layerdb\sha256\*\diff"

Now as we have the full path to the right descriptor files, we need to modify them. In our scenario, this is how a regular descriptor.json file will look like:

   "mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
   "size": 252691002,
   "digest": "sha256:bce2fbc256ea437a87dadac2f69aabd25bed4f56255549090056c1131fad0277",
   "urls": [""]

We need to change the mediaType as below and remove the urls field.

   "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
   "size": 252691002,
   "digest": "sha256:bce2fbc256ea437a87dadac2f69aabd25bed4f56255549090056c1131fad0277"

As a final step, restart docker service

Restart-Service docker

After all those steps, we are ready to push our own image that we have created just by tagging the official nanoserver image… and voila!!

The good side is, now we have the ability to push this image to any repository regardless of its location, including Harbor. The downside is, Microsoft tends to patch its base images every month, so we need to repeat this procedure and recreate our images occasionally.

How to verify that data is really pushed to Harbor

If you have a suspicious mind like me and cannot help yourself wondering if the data is “really” pushed to the repository, it would make no harm to verify and see that tha data is actually there. The requirement of this task is the hash of the image manifest that was uploaded with the layers which we can easily get from the docker push command output. In our case, that is 2261d13476c671ba182e117001f1bc6ff7c0aa188c8225e6fa5bf0cddebce561.

On Harbor, the blobs exist under /data/harbor/registry/docker/registry/v2/blobs/ directory but once again we have to find the right sub-directory which are based on (surprise!!) hashes. So we start with the hash of the image manifest and get the data inside. Do not forget to add the first two characters of the hash to the full path, as below.

cat /data/harbor/registry/docker/registry/v2/blobs/sha256/22/2261d13476c671ba182e117001f1bc6ff7c0aa188c8225e6fa5bf0cddebce561/data
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 774,
      "digest": "sha256:6c367cf4cb9815b10e47545dc9539ee4bd5cd0f8697d33f4d9cb1e1850546403"
   "layers": [
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 259260404,
         "digest": "sha256:35aba4b22d486db55a401eebdef3ffa69d00539497b83d9f4b62e3582cb4ced7"
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 125427565,
         "digest": "sha256:e1870ee27293b87c78bf45e078f94acd29c44a0a7963c91932c5c31dfbeeb510"

Now we have a little bit more information about the layers such as MIME type of the referenced object, size and the digest of the content. With the help of the digest values of the layers, we can locate and verify that the data is actually uploaded, consume storage on the disk as expected and ready to be pulled upon request.

root@sddcvic [ / ]# ls -lh /data/harbor/registry/docker/registry/v2/blobs/sha256/35/35aba4b22d486db55a401eebdef3ffa69d00539497b83d9f4b62e3582cb4ced7
total 248M
-rw-r--r-- 1 root root 248M May 31 10:04 data
root@sddcvic [ / ]# ls -lh /data/harbor/registry/docker/registry/v2/blobs/sha256/e1/e1870ee27293b87c78bf45e078f94acd29c44a0a7963c91932c5c31dfbeeb510
total 120M
-rw-r--r-- 1 root root 120M May 31 10:03 data

Now considering that everything is in place, I can have my cup of coffee and enjoy pulling and pushing Windows container images as a whole 🙂

Push windows container images to harbor registry

I’ve been pulling and pushing container images to harbor registry (that comes with vSphere Integrated Containers 1.1 as told here) for a while without any problem. Last week, I decided to play with native windows containers on Windows 2016 but failed to push the image that I built from a microsoft/nanoserver image.

[As of today, the official version that can be downloaded from My VMware Portal is v1.1.1 and this has been tested with harbor registry version 1.1.0 and 1.1.1. Please note that this might not be supported by VMware]

Here is what it looks like from the client when I tried to push the container image to the registry. All layers get pushed but when the client pushes the manifest, it fails.

manifest blob unknown: blob unknown to registry

What happens under the hood is, whenever a container image is being pushed to the registry, an image manifest is also uploaded that provides a configuration and a set of layers for the container image. The format of the manifest must be recognized by the container registry, if not, the registry returns an 404 error code and errors out the push operation.

As it can be clearly seen from the first screenshot, the initial two layers of microsoft nanoserver container image are defined as foreign layers that require a new image manifest schema (image manifest version 2, schema 2). There is one container running in the vic appliance that is responsible for the registry service, which is vmware/registry:photon-2.6.0 and in order to remediate this issue, we need to swap the registry container version to 2.6.1 which recognizes and accepts this new schema. To accomplish the task;

docker pull vmware/registry:2.6.1-photon
docker-compose -f docker-compose.notary.yml -f docker-compose.yml down
sed -i -- 's/photon-2.6.0/2.6.1-photon/g' /etc/vmware/harbor/docker-compose.yml
docker-compose -f docker-compose.notary.yml -f docker-compose.yml up -d

  1. First, let’s pull 2.6.1 tagged version of vmware/registry repository from docker hub.
  2. Stop all running containers gracefully.
  3. Modify the yml file in order to swap the container image to run.
  4. Start all running containers.

After we spin up the containers, we can verify that registry container is swapped with the newer one, with docker ps command. So it’s time to push our windows based container image to harbor again… and voila!

P.S. Foreign layers are still skipped and not pushed to the harbor registry because of their nature. This is another topic that I will touch on and propose a workaround to push them to our private registry, in the next post.

vSphere Integrated Containers 1.1

Many has changed since my last blog post about vSphere Integrated Containers. So I though it would be a good time to make a fresh start.

On April 2017, VMware released vSphere Integrated Containers v1.1 (aka VIC). VIC comprises of three components that can be installed with an unified OVA package:

  • VMware vSphere Integrated Containers Engine (aka VIC Engine), a container runtime for vSphere that allows developers who are familiar with Docker to develop in containers and deploy them alongside traditional VM-based workloads on vSphere clusters. vSphere adminitrators can manage these workloads by using vSphere in a way that is familiar.
  • VMware vSphere Integrated Containers Registry (aka Harbor), an enterprise-class container registry server that stores and distributes container images. vSphere Integrated Containers Registry extends the Docker Distribution open source project by adding the functionalities that an enterprise requires, such as security, identity and management.
  • VMware vSphere Integrated Containers Management Portal (aka Admiral), a container management portal that provides a UI for DevOps teams to provision and manage containers, including retrieving stats and info about container instances. Cloud administrators can manage container hosts and apply governance to their usage, including capacity quotas and approval workflows. When integrated with vRealize Automation, more advanced capabilities become available, such as deployment blueprints and enterprise-grade Containers-as-a-Service.

These are the key highlights with this new version, but when you go deeper, you figure out that there are many improvements under the hood which I plan to mention in the blog posts to come. So the key new features are:

  • A unified OVA installer for all three components
  • Official support for vSphere Integrated Containers Management Portal
  • A unified UI for vSphere Integrated Containers Registry and vSphere Integrated Containers Management Portal
  • A plug-in for the HTML5 vSphere Client
  • Support for Docker Client 1.13 and Docker API version 1.25
  • Support for using Notary with vSphere Integrated Containers Registry
  • Support for additional Docker commands. For the list of Docker commands that this release supports, see Supported Docker Commands in Developing Container Applications with vSphere Integrated Containers.

The installation process is as easy as deploying an OVA appliance so I do not tend to go deeper with the installation as I already touched briefly on my previous post, it’s pretty much the same and obvious. All the components installed after the deployment are running as containers just like the older version. The difference is, there are more containers running as more services are bundled in the appliance. To list running containers;

docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Command}}"

The applications running in the appliance are multi-container applications so VMware leveraged docker-compose in order to define and run those containers. Under /etc/vmware/harbor there are two yml files.

  • docker-compose.yml: Composes harbor components
  • docker-compose.notary.yml: Composes docker notary components

You can start and stop harbor and notary by running docker-compose up and docker-compose down commands.

cd /etc/vmware/harbor
docker-compose -f docker-compose.notary.yml -f docker-compose.yml down
docker-compose -f docker-compose.notary.yml -f docker-compose.yml up -d

The only container that is not managed by docker-compose is vic-admiral. This is a standalone container and can be started and stopped by standard docker commands individually.

Harbor, an enterprise class registry server


With the GA release of vSphere Integrated Containers (aka VIC) on December, VMware also announced an enterprise class registry server based on the open source Docker Distribution that allows us to store and distribute container images in our datacenter. While VIC Engine (actually the core component of VIC) is great for providing a container runtime for vSphere, enterprise customers governed by strict regulations require a more private solution in order to store and manage container images rather than the default cloud-based registry service from Docker. Docker has already a private solution, which is called Trusted Registry, but the lack of enterprise grade functionalities such as increased control and security, identity and management triggered this open source project, Project Harbor.

Project Harbor extends Docker Trusted Registry and adds the following features,

  • Role based access control: Users and repositories are well organized and a user can have different permissions for images under a project.
  • Policy based image replication: Images can be replicated (synchronized) between multiple registry instances.
  • LDAP/AD support: Harbor integrates with existing enterprise LDAP/AD for user authentication and management.
  • Image deletion & garbage collection: Images can be deleted and their space can be recycled.
  • Graphical user portal: Users can easily browse, search repositories and manage projects.
  • Auditing: All the operations to the repositories are tracked.
  • RESTful API: RESTful APIs for most administrative operations, easy to integrate with external systems.
  • Easy deployment: Provide both an online and offline installer. Besides, a virtual appliance for vSphere platform (OVA) is available.

It’s possible to download the binary from My VMware Portal which is required to install Harbor as a virtual appliance. If we are not eligible to reach the binary through the portal, we can always visit the GitHub page of the project for manual installation and the OVA file.

Let’s assume that we have decided to use the OVA format in the name of the ease of the deployment. As there are many options we need to provide, such as regular virtual appliance options (hostname, datastore, IP configuration) and application specific configurations (passwords, SMTP configuration and a few harbor specific options), the most important one is the one that configures the authentication method that Harbor will be configured with. It can be either LDAP authentication or database authentication but it cannot be modified after the installation, if required, we have to install a fresh instance. LDAP authentication is a good practice because it will save us from managing custom users within the database and will be more secure. Below are the options that we need to provide (please modify according to your domain)

  • Authentication mode: ldap_auth
  • LDAP URL: ldap://DomainController.demo.local
  • LDAP Search DN: CN=User2MakeQueries,OU=Users,DC=demo,DC=local
  • LDAP Search Password: Password of the above user
  • LDAP Base DN: OU=OrganizationUnitInWhichUsersWillBeQueried,DC=demo,DC=local
  • LDAP UID: The filter to query the users, such as uid, cn, email, sAMAccountName or any other attribute.

After the installation, it’s highly recommended to change the admin password of Harbor provided by us during the deployment phase because it will persist in the configuration file (/harbor/harbor/harbor.cfg) in plain text.

It will take a few minutes to complete the installation. If we set the “Permit Root Login” option to ‘true’ during the deployment phase, we can connect to the server via SSH with root credentials and begin to play around. The deployed operating system is Photon OS and the sub-components of Harbor are actually running as containers. When we run docker ps -a, all those running containers come to daylight. Harbor consists of six containers composed by docker-compose.

  • Proxy: This is a NGINX reverse-proxy. The proxy forwards requests from browsers and Docker clients to backend services such as the registry service and core services.
  • Registry: This registry service is based on Docker Registry 2.5 and is responsible for storing Docker images and processing Docker push/pull commands.
  • Core Services: Harbor’s core functions, which mainly provides the following services:
    • UI: A graphical user interface to help users manage images on the Registry
    • Webhook: Webhook is a mechanism configured in the Registry so that image status changes in the Registry can be populated to the Webhook endpoint of Harbor. Harbor uses webhook to update logs, initiate replications, and some other functions.
    • Token service: Responsible for issuing a token for every docker push/pull command according to a user’s role of a project. If there is no token in a request sent from a Docker client, the Registry will redirect the request to the token service.
  • Database: Derived from official MySQL image and is responsible for storing the metadata of projects, users, roles, replication policies and images.
  • Job Services: This service is responsible for image replication to other Harbor instances (if there are any).
  • Log Collector: This service is responsible for collecting logs of other modules in a single place.

And this is what it looks like from an architectural point of view;

All the blue boxes shown in the diagram are running as containers. If we would like to know more about those containers and how they are configured, we can always run docker inspect commands on them.

docker inspect nginx
docker inspect harbor-jobservice
docker inspect harbor-db
docker inspect registry
docker inspect harbor-ui
docker inspect harbor-log

As a result of the inspect commands, the thing that attracts my attention is the persistent volume configurations. By their nature, containers are immutable and disposable. So in order to keep the data and the service configurations persistent, Harbor takes advantage of volume mounts between the docker host and the containers. And it’s also useful to modify configuration of the services and to replace the ui certificate.

This is the list of the volume mounts (sources and destinations) used in all containers.

We can now enjoy and push images to our on-prem brand new registry server.