Docker Composer

UPDATE: This is now PyPi package.

Motivation

If you have lots of docker containers that you need to run, and they have complicated configurations or dependencies then you would likely start using Docker Compose. Docker Compose allows you to write config files for your containers, including setting up interdependent networks and volumes. It even allows you to specify whether containers depend on other containers and will then change the boot order to ensure that the dependencies start first.

But what if one of the dependencies needs to change because of a container that relies on it? For example, you have a PHP container, and you have website containers. The website container volumes need to be mounted in the PHP container, but the websites are dependent on the PHP container. Sure you could manually manage the compose file so that each time you add a new service you edit the config of the dependency but if you are frequently switching services in and out this is a pain. It would be better if the dependency container did not have to know anything about its dependants.

I wrote a Python script to add an extra layer of management around Docker Compose, called Docker Composer. It's inspired by the style of the uswgi manager uswgi-emperor.

Usage

Start with a directory for your related services and include the docker-composer.py file, a 'services-available' folder, and a .env file.

my-services/
├── .env
├── docker-composer
└── services-available/

You can then add your interelated docker-compose services in the services-available folder.

my-services/
├── .env
├── docker-composer
└── services-available/
    ├── php.yml
    └── my-website.yml

The config files can just be docker-compose config files but you can also make user of extra composer features.

php.yml

version: "3"
services:
  ${PHP_FPM_NAME}:
    build: /path/to/docker/file
    container_name: ${PHP_FPM_NAME}
    image: ${PHP_FPM_NAME}
    volumes:
    networks:
      - ${MY_NETWORK}
    restart: always
networks:
  ${MY_NETWORK}:
    driver: bridge
volumes:

composer_base: true

my-website.yml

version: "3"
services:
  ${WEBSITE_NAME}:
    container_name: ${WEBSITE_NAME}
    image: nginx
    volumes:
      - /path/to/my/website:/var/www/website.com
    networks:
      - ${MY_NETWORK}
    depends_on:
      - ${PHP_FPM_NAME}
    restart: always
composer_compositions:
  - type: add
    value: /path/to/my/website:/var/www/website.com
    to:
      - services
      - ${PHP_FPM_NAME}
      - volumes

.env

PHP_FPM_NAME=php-fpm
WEBSITE_NAME=website
MY_NETWORK=web-network

The extra syntax visible in this example is:

• Environment variables everywhere e.g. ${WEBSITE_NAME}
  This is an improvement on the normal docker compose environment variable. You can even use them as service names. These are picked up from the .env file.
• composer_compositions
  The syntax for modifying other services. In the example we want to add our file path mapping /path/to/my/website:/var/www/website.com to the php services volumes section. The to value is just the path within the yml file.
• composer_base
  Composer will combine all the services into a single docker-compose.yml. To do this one config must be nominated as the base config. This could either be one that is expected to run all the time (as in the php example) or a blank config.

So with the configuration files in place you would enable them

./docker-composer enable php
./docker-composer enable my-website

Which would symlink them into a new folder services-available

my-services/
├── .env
├── docker-composer
├── services-available/
│   ├── php.yml
│   └── my-website.yml
└── services-enabled/
    ├── php.yml
    └── my-website.yml

Running the script once more

./docker-composer

Will generate a docker-compose.yml

my-services/
├── .env
├── docker-composer
├── services-available/
│   ├── php.yml
│   └── my-website.yml
├── services-enabled/
│   ├── php.yml
│   └── my-website.yml
└── docker-composer.yml

And then you can use docker-compose as normal

docker-compose up -d

Enabling and disabling services is as simple as removing their files from the services-enabled folder, manually or using the inbuilt command

./docker-composer disable my-website

You can also list a few more configuration arguments for docker-composer

./docker-composer --help

Internals

There's nothing complicated in the script, so reading it probably isn't a terrible way to find out how it works.

The enable/disable commands work by adding and removing symlinks into the enabled folder, with some file system checking etc.

The main run command reads all the files from the enabled folder as strings. It firstly does a find/replace for all the environment variables and then it parses them into python dictionaries.

The base config is found and then that is used to recursively merge the dictionaries on top of that base. It does nothing clever to manage key collisions, the later read configs will simply overwrite earlier configs.

With the fully merged dictionary it runs through all of the compositions from each config and applies the changes specified.

Finally it dumps the dictionary to a yaml file.

Limitations

• At the moment the only composition is the add composition which is the only one I have found useful. I can imagine uses for more verbs and it would not be too difficult to add them.
• It's not super clever. I think it could manage more of the bringing containers up and down itself, and do it automatically i.e wrap more of the docker-compose functionality so things can be done in fewer commands. A recompile and refresh/rebuild command might be nice. Though in some ways its nice to leave it simple and not try to solve solved problems.

Source

Source here or printed below:

#!/usr/bin/python3
import sys
import os
import re
import yaml
import pprint
import operator
from functools import reduce
from pathlib import Path
from dotenv import load_dotenv
import argparse
import shutil

ap = argparse.ArgumentParser()
ap.add_argument("command", metavar="command", default="run", help="command to use: run, enable, disable", nargs="?")
ap.add_argument("service_name", metavar="service name", help="name of service to enable or disable", nargs="?")
ap.add_argument("--env-file", default=".env", help="path to the environment variables file")
ap.add_argument("--available-dir", default="./services-available", help="path to the available folder")
ap.add_argument("--enabled-dir", default="./services-enabled", help="path to the enabled folder")
ap.add_argument("-o", "--output", default="docker-compose.yml", help="name of the output file")
ap.add_argument("-v", "--verbose", action="store_true", help="verbose output")
args = vars(ap.parse_args())

command = args['command']
service_name = args['service_name'] if 'service_name' in args else None
env_file = os.path.abspath(str(Path('.') / args['env_file']))
available_dir = os.path.abspath(str(Path('.') / args['available_dir']))
enabled_dir = os.path.abspath(str(Path('.') / args['enabled_dir']))
output_file = os.path.abspath(str(Path('.') / args['output']))
verbose = args['verbose'] if 'verbose' in args else False

if (command == 'enable' or command == 'disable') and service_name is None:
        ap.error('You must specify a service name.')

pp = pprint.PrettyPrinter(indent=2)

# Load env vars from file
load_dotenv(dotenv_path=env_file)
#os.environ.update(Dotenv(env_file))

# Useful Functions
def getFromDict(dataDict, mapList):
        return reduce(operator.getitem, mapList, dataDict)

def setInDict(dataDict, mapList, value):
        origvalue = getFromDict(dataDict, mapList[:-1])[mapList[-1]]
        if isinstance(origvalue,list):
                getFromDict(dataDict, mapList[:-1])[mapList[-1]] = value + origvalue
        else:
                getFromDict(dataDict, mapList[:-1])[mapList[-1]] = value

def merge(destination, source):
        for key, value in source.items():
                if isinstance(value, dict):
                        node = destination.setdefault(key, {})
                        merge(node, value)
                else:
                        destination[key] = value
        return destination

def replace_env(match):
        return os.getenv(match.group(1))

def enable():
        service_path = os.path.join(available_dir, service_name + ".yml")
        enabled_service_path = os.path.join(enabled_dir, service_name + ".yml")
        if os.path.isfile(enabled_service_path):
                print('Service already enabled')
                return
        if not os.path.isfile(service_path):
                print("Cannot find service: {}".format(service_path))
                sys.exit()
        if not os.path.isdir(enabled_dir):
                print("Cannot find enabled directory: {}".format(enabled_dir))
        if os.path.islink(service_path):
                if os.path.isfile(enabled_service_path):
                        os.unlink(enabled_service_path)
                os.symlink(os.readlink(service_path), enabled_service_path)
        else:
                os.symlink(service_path, enabled_service_path)
        print("Enabled {}".format(service_path))

def disable():
        enabled_service_path = os.path.join(enabled_dir, service_name + ".yml")
        if not os.path.isfile(enabled_service_path):
                print("Service is not enabled: {}".format(service_path))
                sys.exit()
        if os.path.islink(enabled_service_path):
                os.unlink(enabled_service_path)
        else:
                os.remove(enabled_service_path)
        print("Disabled {}".format(service_name))

def run():
        # Gather configs
        output_config = {}
        configs = []
        config_path=enabled_dir
        config_files = [f for f in os.listdir(config_path) if os.path.isfile(os.path.join(config_path, f))]
        for config_file in config_files:
                with open(os.path.join(config_path, config_file), 'r') as stream:
                        print('Reading {}'.format(config_file))
                        config_text = re.sub(r'\${(.*)}', replace_env, stream.read())
                        name = os.path.splitext(config_file)[0]
                        try:
                                config = yaml.load(config_text)
                        except Exception as e:
                                print(config_text)
                                print('Error with "{}" config file'.format(name))
                                print(e)
                                sys.exit(1)
                        if "composer_base" in config and config["composer_base"]:
                                output_config = config
                        else:
                                configs.append(config)

        # Merge configs
        for config in configs:
                output_config = merge(output_config, config)

        # Resolve additions
        for config in configs:
                if 'composer_compositions' in config:
                        for composition in config['composer_compositions']:
                                if composition['type'] == 'add':
                                        setInDict(output_config, composition['to'], [ composition['value'] ])

        output_config.pop('composer_base', None)
        output_config.pop('composer_compositions', None)

        with open(output_file, 'w+') as stream:
                yaml.dump(output_config, stream, default_flow_style=False)
        if verbose:
                print(yaml.dump(output_config, default_flow_style=False))

        print("Done!")

if command == 'run':
        run()
if command == 'enable':
        enable()
if command == 'disable':
        disable()