How does it work? Docker! Part 4: Control your Swarm!

Hey there!

TL;DR


I hacked another thing together, this time in order to install a highly available Docker Swarm cluster on CoreOS (yeah, Container Linux), using Ansible.

The whole subject was way too long for a single article. Therefore, I’ve divided it into 5 parts. This is episode 4, regarding the actual implementation of the local cluster’s Manager nodes, using Vagrant, CoreOS and Ansible.

If you want to try it:


git clone https://github.com/sebiwi/docker-coreos.git

cd docker-coreos

make up

 

You will need Ansible 2.2+, Docker, Vagrant and Molecule

Code, please!

Yay, code!

First of all, we will need the actual virtual machines where our cluster will run. We will create these using Vagrant. If you don’t know what Vagrant is, or why are we using it, check out this article.

For this deployment, we will have three Manager nodes, and three worker nodes.  Why three Manager nodes you say? It may seem overkill, but in order to have High Availability you need to have an odd number of Manager nodes, otherwise, you will not get consensus from Raft. We will see this in action later on.

We do not need an etcd cluster, since the key-value datastore is already included in the internals of the Docker Engine, and therefore, we will not include any machines for it.

First of all, I’ll describe the amount of machines I want and their configurations as variables:

# General cluster configuration
$swarm_manager_instances = 3
$swarm_manager_instance_memory = 2048
$swarm_manager_instance_cpus = 1
$swarm_worker_instances = 3
$swarm_worker_instance_memory = 2048
$swarm_worker_instance_cpus = 1

Afterwards, I’ll specify that I want to use the CoreOs (Container Linux) Vagrant box, from an URL:

  # Box management: CoreOS
config.vm.box = "coreos-stable"
config.vm.box_url = "https://storage.googleapis.com/stable.release.core-os.net/amd64-usr/current/coreos_production_vagrant.json"

Just a little reminder, Container Linux is a lightweight Linux distribution that uses container to run applications. It ships with basic GNU utilities so you can do all your business, and some other interesting things, like kubelet, Docker, etcd and flannel. We’ll only be using Docker for this part. In short, CoreOS’s Container Linux is an OS specially designed to run containers, and we’re going to profit from that in our context. If you wanna know more about CoreOS, check this article.

We’ll configure our Manager nodes with the variables we previously defined:


  # Swarm manager instances configuration
  (1..$swarm_manager_instances).each do |i|
    config.vm.define vm_name = "swarm-manager-%02d" % i do |master|
      # Name
      master.vm.hostname = vm_name

      # RAM, CPU
      master.vm.provider :virtualbox do |vb|
        vb.gui = false
        vb.memory = $swarm_manager_instance_memory
        vb.cpus = $swarm_manager_instance_cpus
      end

      # IP
      master.vm.network :private_network, ip: "10.0.0.#{i+110}"
    end
  end

And then we’ll do the same thing with our Worker nodes:


# Swarm worker instances configuration
  (1..$swarm_worker_instances).each do |i|
    config.vm.define vm_name = "swarm-worker-%02d" % i do |worker|
      # Name
      worker.vm.hostname = vm_name

      # RAM, CPU
      worker.vm.provider :virtualbox do |vb|
        vb.gui = false
        vb.memory = $swarm_worker_instance_memory
        vb.cpus = $swarm_worker_instance_cpus
      end

      # IP
      worker.vm.network :private_network, ip: "10.0.0.#{i+120}"
    end
  end

Easy. If you wanna take a look at the Vagrantfile, it’s right here. Moving on.

Next up, I’ll set up a Makefile with a Vagrant target, which will create all the virtual machines, and generate a configuration file with the SSH configuration needed to interact with them. Ill also add a clean target, which will destroy all the machines, and it will delete the SSH configuration file, if it exists.


.PHONY: vagrant clean

vagrant:
	@vagrant up
	@vagrant ssh-config > ssh.config

clean:
	@vagrant destroy -f
	@[ ! -f ssh.config ] || rm ssh.config

I’ll also put in a list of phony targets, since the targets previously defined are just actions and don’t really generate files.

We’ve got the virtual machines, and the SSH configuration. For the Ansible part, we’ll start by creating an inventory with all our six nodes in it, separated by groups:

[swarm-leader]
swarm-manager-01

[swarm-manager]
swarm-manager-01
swarm-manager-02
swarm-manager-03

[swarm-worker]
swarm-worker-01
swarm-worker-02
swarm-worker-03

You will probably notice there is a `swarm-leader` group in the inventory, which contains a single host. Like I said in the first article, there might be many Managers in a cluster; Nevertheless, there is only one Leader at any given moment. We will use this group to launch specific actions for the Leader, common actions for all Manager nodes using the swarm-manager group, and actions destined for the non-Leader Manager nodes using the swarm-manager group, and subtracting the swarm-leader group from it. This may seem complex, but it is actually super easy, you will see.

No IP configurations over here, we’ll just use the SSH configuration file we generated earlier. In order to do that, we have to specify it on our ansible.cfg file. No ansible.cfg file? Just create it:


[defaults]
ansible_managed = Please do not modify this file directly as it is managed by Ansible and could be overwritten.
retry_files_enabled = false
remote_user = core

[ssh_connection]
ssh_args = -F ssh.config

We’ll also disable retry_files, and specify that we want to use the “core” user when connecting to the machines using SSH.

I’ve already said this before, but CoreOS only ships with basic GNU utilities, which means no Python. And no Python means no Ansible, except for the raw module, the script module and the synchronize module. What we’re going to do is that we’re going to install a lightweight Python implementation called PyPy using only those modules, and then use that Python implementation in order to execute the rest of our playbook. Neat huh?

We’ll use the same role we used for the Kubernetes provisioning project. If you want to read more about it, like the technical explanation behind it, you can find all the information here.

So basically, we’ve got a role now under roles/bootstrap/ansible-bootstrap, which has 3 files under the tasks directory: main.yml, configure.yml and test.yml. The configure.yml file holds all the tasks necessary in order to install PyPy. The test.yml file verifies if Python is correctly installed by doing `python –version`. The main.yml file wraps these two files, adding the `test` tag to the test.yml part:


# filename: roles/bootstrap/ansible-bootstrap/tasks/main.yml
---

- name: Install and configure PyPy
  include: configure.yml

- name: Test PyPy installation
  include: test.yml
  tags: [ test ]

- name: Gather ansible facts
  setup:

I’ll follow this approach for each role on this project, so each role will have a smoketest, which will be enough to tell us if the component is correctly installed. This is pretty useful in order to test the already deployed infrastructure, as a conformance test, and check for deltas which might need to be corrected.

Now that we have our first role, it’s time to create a playbook. Since we’ll be deploying a Swarm cluster, I’ll just name it swam.yml:


---

- name: Bootstrap coreos hosts
  hosts: all
  gather_facts: false
  roles:
    - role: bootstrap/ansible-bootstrap
      tags: [ ansible-bootstrap ]

It’s quite straightforward so far, I’ll just launch the recently created role on each hosts, without gathering facts, since Python is not yet installed on the machines. Facts will be gathered at the end of the role though, as seen on the previous code snippet.

Next up, tests. We’ll use molecule for the win. I spoke to you all about molecule on a previous article. It is basically a testing tool for Ansible code. It creates ephemeral infrastructure (either virtual machines or containers), tests your roles on it (not only the execution of the roles, but also the syntax and their idempotence), and then it destroys it. Since there are no CoreOS containers, and Virtualbox virtual machines through Vagrant being the target platform, I’ll just use the Vagrant driver.

In order to test with molecule, I’m going to create a molecule.yml file, in which I’m going to define the Ansible files to use for the test, as well as the Vagrant machine’s specification and configuration.

First, I’ll specify which Ansible configuration to use, and which playbook to run:


---
ansible:
  config_file: ./ansible.cfg
  playbook: swarm.yml

Then, I’ll specify which Vagrant box to use for the virtual machines:


   platforms:
  - name: coreOS
    box: coreos-stable
    config.vm.box_url: https://storage.googleapis.com/stable.release.core-os.net/amd64-usr/current/coreos_production_vagrant.json

Then, what will my provider be, and how much physical resources will be used by each instance:


  providers:
  - name: virtualbox
    type: virtualbox
    options:
      memory: 2048
      cpus: 1

After that, I’ll define the specifics of each instance, including both hostname and IP addresses:


 instances:
  - name: swarm-manager-01
    ansible_groups:
      - swarm-leader
      - swarm-manager
    interfaces:
      - network_name: private_network
        type: static
        ip: 10.0.0.101
        auto_config: true
    options:
      append_platform_to_hostname: no
  - name: swarm-manager-02
    ansible_groups:
      - swarm-manager
    interfaces:
      - network_name: private_network
        type: static
        ip: 10.0.0.102
        auto_config: true
    options:
      append_platform_to_hostname: no
  - name: swarm-manager-03
    ansible_groups:
      - swarm-manager
    interfaces:
      - network_name: private_network
        type: static
        ip: 10.0.0.103
        auto_config: true
    options:
      append_platform_to_hostname: no
  - name: swarm-worker-01
    ansible_groups:
      - swarm-worker
    interfaces:
      - network_name: private_network
        type: static
        ip: 10.0.0.121
        auto_config: true
    options:
      append_platform_to_hostname: no
...

With that in place, I just need to run `molecule test` in order to test that my infrastructure is created and configured correctly. This is actually an oversimplification of everything that can be done using molecule, but since I already wrote about it on a previous article, just can just head there and read about it if you’re really interested.

And with that, I also get to add two new (phony) Makefile target:


smoketest:
	@ansible-playbook -i inventories/vagrant.ini swarm.yml --tags test

test:
	@molecule test

The smoketest target allows me to run a conformance test on all the already deployed infrastructure, to check for deltas and see if something’s wrong whenever I want, and the test target allows me to test the code on fresh, newly created infrastructure, and to check for Ansible-specific good practices. Remember, this uses Molecule V1, so if you try to run it using Molecule V2 it will probably not work.

Tests are set up thusly. Moving on.

Manage: Lead and follow

Next up, we need to setup the three Manager nodes. I’ll start by creating a swarm-leader role, under the configuration roles directory:


roles/
├── bootstrap
│   └── ansible-bootstrap
└── configure
    └── swarm-leader
         └── tasks
             ├── configure.yml
             ├── main.yml
             └── test.yml

And for this role, we’ll use the same task division strategy we used before in order to add our smoketest.

First, the main.yml file, which is fairly simple, and only includes the other two yaml files, using a tag for the test file:


---

- name: Create Manager Leader
  include: configure.yml

- name: Test Manager Leader
  include: test.yml
  tags: [ test ]

The configure file first checks if the cluster is already on Swarm mode. If it is, it doesn’t do anything else. If it isn’t, it creates the first Swarm node, creating thus the Swarm cluster, which will be joined by the subsequent nodes. It also disables scheduling on the Leader, making sure that the Leader does not handle any workload and that it concentrates its resources on leading the cluster:



---
- name: Check if Swarm Mode is already activated
  command: docker info
  register: docker_info
  changed_when: false

- name: Create Swarm Manager Leader if it is not activated
  command: docker swarm init --advertise-addr {{ hostvars[groups['swarm-leader'][0]]['ansible_env']['COREOS_PUBLIC_IPV4'] }}
  when: "'Swarm: active' not in docker_info.stdout"

- name: Disable Leader scheduling
  command: docker node update --availability drain {{ groups['swarm-leader'][0] }}
  when: "'Swarm: active' not in docker_info.stdout and disable_leader_scheduling"

This last part is not actually necessary, specially for small clusters. Nevertheless it is usually a good practice, since the leader election process can be really intensive in terms of resource consumption. The `disable_leader_scheduling` variable is defined on the role’s defaults, and you can override it if you want your Leader to handle workloads.

Fairly simple. Notice the `changed_when: false` parameter on the first command task. It is there because running  `docker info` will not change the state of the cluster, and it is therefore not a real action, just a way of collecting information.

Next, for the smoketest, I’ll verify if the created Manager node is in fact a Leader (which it should be, since it was the first Manager node to be created), and whether its status is “Drain”, since the Leader node is not supposed to handle any workload:


---

- name: Check if Manager node is Leader
  shell: docker node ls | grep {{ ansible_hostname }}
  register: docker_info
  changed_when: false

- name: Fail if Manager node is not Leader
  assert:
    that:
      - "'Leader' in docker_info.stdout"
      - "'Active' in docker_info.stdout"

Now that the role is set, I’ll just add it to the swarm.yml playbook:


- name: Create Swarm Leader node
  hosts: swarm-leader
  roles:
    - role: configure/swarm-leader
      tags: [ swarm-leader ]

Using the host group we discussed earlier, and the proper tag in order to identify the action. And that’s it for the Manager Leader. We need some non-Manager Leader for that High Availability though!

So we’ll just repeat the previous process, we’ll create a swarm-manager role up next, with the same structure of the previous role (main.yml, configure.yml, test.yml).

I won’t show you the main.yml: it is basically the same one we saw before. The configure.yml file, on the other hand, checks if Swarm mode is activated on the node, the same way the Leader role does, but if it isn’t, it recovers the token needed to join the cluster as a Manager node from the Leader node, and joins the cluster with it. If Swarm mode is already activated, it does nothing:


---

- name: Check if Swarm Mode is already activated
  command: docker info
  register: docker_info
  changed_when: false

- name: Recover Swarm Leader token
  shell: docker swarm join-token manager | grep token | cut -d ' ' -f 6
  register: leader_token
  when: "'Swarm: active' not in docker_info.stdout"
  delegate_to: "{{ groups['swarm-leader'][0] }}"

- name: Join Swarm Cluster as Manager
  command: docker swarm join --token {{ leader_token.stdout }} {{ hostvars[groups['swarm-leader'][0]]['ansible_env']['COREOS_PUBLIC_IPV4'] }}
  when: "'Swarm: active' not in docker_info.stdout"

- name: Disable Manager scheduling
  command: docker node update --availability drain {{ ansible_hostname }}
  when: "'Swarm: active' not in docker_info.stdout and disable_manager_scheduling"

Notice the `delegate_to` option on the token recovery task. This needs to be done because the token must be recovered from the Leader node and the Leader node only. Scheduling is also disabled on these nodes, by default, because of the reason specified above on the Leader node. This time, the `disable_manager_scheduling` variable is also defined on the role’s defaults. You can override this variable if you want your Managers to handle workloads.

The test file verifies different  things as well:


---

- name: Check if node is Manager
  shell: docker node ls | grep {{ ansible_hostname }}
  register: docker_info
  changed_when: false

- name: Fail if node is not Manager
  assert:
    that:
      - "'Reachable' in docker_info.stdout"
      - "'Drain' in docker_info.stdout"

It recovers the nodes information, and then it verifies that the node Manager type is `Reachable` rather than `Leader`, as it was for the Leader node. It also verifies that the nodes are “drained” since we don’t want them to run containers.

Finally, once the role is ready, I’ll add it to the swarm.yml file:


- name: Create Swarm Manager nodes
  hosts: swarm-manager:!swarm-leader
  roles:
    - role: configure/swarm-manager
      tags: [ swarm-manager ]

Notice the `!` sign on the hosts part of the play. This specifies that we want to run the role on every node on the swarm-manager group, that isn’t in the swarm-leader group, thus preventing the Leader node to try to join the cluster as a non-Leader Manager. Sweet!

Once we finish all this, we should have everything we need Manager-wise. Time to get some Workers running! I’ll probably talk to you about that on the next article though.

Stay in touch!

Laisser un commentaire

Votre adresse e-mail ne sera pas publiée.


Ce formulaire est protégé par Google Recaptcha