Release v0.0.5
A collection of Fabric helper functions used in some of Viki’s projects.
Append this line to requirements.txt:
viki-fabric-helpers
Followed by running:
pip install -r requirements.txt
Cloning the repository:
git clone https://github.com/viki-org/viki-fabric-helpers.git
Downloading a tarball:
wget -O viki-fabric-helpers.tar.gz https://github.com/viki-org/viki-fabric-helpers/tarball/master
Downloading a zipball:
wget -O viki-fabric-helpers.zip https://github.com/viki-org/viki-fabric-helpers/zipball/master
This page describes how you can configure the viki-fabric-helpers library.
The viki-fabric-helpers library makes use of a viki_fabric_config.yml file for configuration. This file is read once, when any module under the viki.fabric package is imported (the code is in the viki/fabric/__init__.py file).
It is not necessary to provide the viki_fabric_config.yml file. If you do provide it however, it should be located in the directory where the main Python script is run.
The viki_fabric_config.yml file is a YAML file. If you are not familiar with YAML, it is a concise data representation format for common data structures such as dictionaries and lists. viki-fabric-helpers makes use of the PyYAML library for reading YAML files.
Currently, only the viki.fabric.git module requires the viki_fabric_config.yml file. Refer to viki.fabric.git - A short guide for more information.
NOTE: The viki_fabric_config.yml file can be used to hold other data as long as their names do not conflict with those used by viki-fabric-helpers.
On your first import of a module under the viki.fabric package, the viki_fabric_config.yml is read and its contents are placed inside the viki_fabric_config key of the fabric.api.env variable. To access the data, you should use the viki.fabric.helpers.get_in_viki_fabric_config.
For instance, suppose the viki_fabric_config.yml file has the following contents:
animals:
mammals:
kangaroo: "jumps"
human: "walks"
reptiles:
crocodle: "swims"
lizard: "climbs"
To access the entire animals hash:
from viki.fabric.helpers import get_in_viki_fabric_config
# obtain the dict {'mammals': ... , 'reptiles': ...}
get_in_viki_fabric_config(["animals"])
To get the value of animals.mammals.kangaroo:
from viki.fabric.helpers import get_in_viki_fabric_config
# obtains the string "jumps"
get_in-viki_fabric_config(["animals", "mammals", "kangaroo"])
The viki.fabric.docker module contains several Docker related Fabric tasks that may help reduce the effort required for writing Fabric tasks that involve Docker.
We will be going through an example of writing a Fabric script that builds a Docker image on your local machine, pushes it to the Docker registry, followed by pulling it on a set of servers. There is extensive inline documentation to aid your understanding.
NOTE: This script assumes usage of Fabric 1.9.0. However, a version of Fabric relatively close to that should work as well.
# the module where the functions we're covering resides
import viki.fabric.docker as viki_docker
# other helper functions that we'll be needing
import viki.fabric.helpers as fab_helpers
# Fabric library imports
from fabric.api import env
from fabric.decorators import runs_once, task
from fabric.operations import run
from fabric.tasks import execute
# Fabric roles
env.roledefs = {
"production": ["m01.prod1", "m02.prod1", "m03.prod1"],
"testing": ["t01.test1", "t02.test1", "t03.test1"]
}
# This Fabric task is decorated with `fabric.decorators.runs_once` because
# it uses `fabric.tasks.execute` to run other Fabric tasks.
#
# Not decorating it with the `fabric.decorators.runs_once` will result in
# the following scenario:
#
# We run a Fabric task T1 on servers S1, S2 and S3.
# Fabric task T1 calls two other Fabric tasks T2 and T3.
# Fabric task T1 is not decorated with `fabric.decorators.runs_once`.
#
# What happens:
#
# S1 runs T1
# S1 runs T2
# S2 runs T2
# S3 runs T2
# S1 runs T3
# S2 runs T3
# S3 runs T3
# S2 runs T1
# S1 runs T2
# S2 runs T2
# S3 runs T2
# S1 runs T3
# S2 runs T3
# S3 runs T3
# S3 runs T1
# S1 runs T2
# S2 runs T2
# S3 runs T2
# S1 runs T3
# S2 runs T3
# S3 runs T3
@runs_once
@task
def build_my_repo_docker_image_and_push_to_registry():
"""Fabric task which builds a Docker image for my repository and pushes
it to the Docker registry (http://index.docker.io).
"""
# name of the Docker image in namespace/image format
dockerImageName = "steveJackson/myRepo"
# Use the `fabric.tasks.execute` function to run the
# `viki.fabric.docker.build_and_push_docker_image` Fabric task.
# The `viki.fabric.docker.build_and_push_docker_image` Fabric task is only
# executed once regardless of the number of hosts or roles you pass to the
# `build_my_repo_docker_image_and_push_to_registry` task that we're
# writing now
retVal = execute(viki_docker.build_and_push_docker_image,
# path to the git repository; anything that `git clone` accepts is
# acceptable
"https://github.com/steve-jackson/my-repo.git",
# name of the Docker image
dockerImageName,
# use `git-crypt` (https://github.com/AGWA/git-crypt) to decrypt the
# git-crypt'ed files in the cloned repository
runGitCryptInit=True,
# Path to the git-crypt key used for the repository
gitCryptKeyPath="/home/steve/my-repo-gitcrypt-key",
# the Dockerfile is located inside the `docker-build` directory of the
# cloned repository
relativeDockerfileDirInGitRepo="docker-build",
# pass in the hosts and roles supplied to the
# `build_my_repo_docker_image_and_push_to_registry` task to the
# `viki.fabric.docker.build_and_push_docker_image` task
hosts=env.hosts, roles=env.roles
)
# We did not supply the `dockerImageTag` keyword argument to the
# above execute, hence we will need the tag of the newly built Docker
# image, which is the return value of the task.
#
# However, we're using `fabric.tasks.execute`, which collects the return
# value of all hosts into a dict whose keys are the host strings and the
# whose values are the return values of the original task for the hosts.
#
# Since the `viki.fabric.docker.build_and_push_docker_image` Fabric task
# is a local Fabric task which runs once, its return value will be the
# same for all given hosts.
# The `viki.fabric.helpers.get_return_value_from_result_of_execute_runs_once`
# function is a convenience function to extract a return value from the
# dict returned by `fabric.tasks.execute`.
dockerImageTag = \
fab_helpers.get_return_value_from_result_of_execute_runs_once(retVal)
# On each given server, pull the newly built Docker image.
# This is run once for each server.
execute(viki_docker.pull_docker_image_from_registry,
# name of the Docker image in `namespace/image` format
dockerImageName,
# tag of the Docker image; we obtained this above
dockerImageTag=dockerImageTag,
# pass in the hosts and roles given to the
# `build_my_repo_docker_image_and_push_to_registry` task
hosts=env.hosts, roles=env.roles
)
Suppose the above script is named fabfile.py. To run it for the production machines:
fab -R production build_my_repo_docker_image_and_push_to_registry
This page covers the use of the viki.fabric.git module. More detailed documentation for individual functions in this module can be found in the API Documentation.
Our focus here will be on running the setup_server_for_git_clone function. This function is used to setup a server for Git remote operations (such as cloning) involving secret repositories and assumes the following:
Any Python script which imports the viki.fabric.git module directly or indirectly will require you to create a YAML file named viki_fabric_config.yml at the directory where the main Python script is invoked. This YAML file should contain a dict at the key viki.fabric.git containing the following keys:
ssh_private_key
Basename of the SSH private key to copy to the server.
ssh_public_key
Basename of the SSH public key to copy to the server.
ssh_keys_local_copy_dir
Folder storing the ssh_private_key and ssh_public_key files on your local machine.
The path to this folder can be relative to where the Python script that imports the viki.fabric.git module is run, or an absolute path.
ssh_keys_dir
Folder on the remote server to copy the ssh_private_key and ssh_public_key files to.
NOTE: This folder is relative to the $HOME directory of the remote user during Fabric execution. This should normally be set to the string ‘.ssh’.
git_ssh_script_name
Basename of a template git wrapper script on your local machine. What this script should contain is outlined later in this subsection.
This file will be copied to the $HOME directory of the user on the remote server (for such a server and user involved in a Fabric task). You can set this to any valid filename. My personal preference for this value is the string ‘gitwrap.sh’.
git_ssh_script_local_folder
Folder on your local machine containing the git_ssh_script_name file.
The path to this folder can be relative to where the Python script that imports the viki.fabric.git module is run, or an absolute path.
viki.fabric.git:
ssh_private_key: "id_github_ssh_key"
ssh_public_key: "id_github_ssh_key.pub"
ssh_keys_local_copy_dir: "github-ssh-keys"
ssh_keys_dir: ".ssh"
git_ssh_script_name: "gitwrap.sh"
git_ssh_script_local_folder: "templates"
Suppose that Fred, a user of our library, has a Python Fabric File located at /home/fred/freds-repo/fabfile.py, which he runs from the /home/fred/freds-repo folder. The above YAML file should be located at /home/fred/freds-repo/viki_fabric_config.yml.
Based on the contents of the /home/fred/freds-repo/viki_fabric_config.yml file:
This is the file specified by the value of the git_ssh_script_name YAML key, and should contain the following code:
#!/bin/bash
ssh -i {{ ssh_private_key_path }} $@
The {{ ssh_private_key_path }} part of the code will be replaced by the setup_server_for_git_clone Fabric task before the script is copied to the server (A temporary file or similar is used, so your file will not be accidentally modified by this task).
Assume that our imaginary user Fred
fab -H hostOne,hostTwo freds_fabric_task
Contents of /home/fred/freds-repo/fabfile.py Fabric script:
from fabric.api import env, task
import os.path
import viki.fabric.git as fabric_git
# Fred uses SSH config
env.use_ssh_config = True
@task
def freds_fabric_task():
# Fred wishes to setup the current server for handling secret repos
fabric_git.setup_server_for_git_clone()
# Fred's other code below
Suppose Fred’s SSH config file looks like this (see the env.use_ssh_config line in the code above to understand why we put this here):
Host hostOne
Hostname 1.2.3.4
User ubuntu
Host hostTwo
Hostname 1.2.3.5
User ubuntu
The effect of successfully executing the setup_server_for_git_clone Fabric task (it’s part of the freds_fabric_task):
Now, the ubuntu user on Fred’s hostOne and hostTwo servers are ready for handling some secret git repositories. We shall go into that next.
Suppose Fred SSHes into hostOne using the ubuntu user, and wishes to clone a secret repository whose clone url is git@github.com:fred/top-secret-repo.git, he should use this bash command to clone the git repository:
GIT_SSH=$HOME/gitwrap.sh git clone git@github.com:fred/top-secret-repo.git
In fact, this can be generalized to other Git remote operations for secret repos, such as git fetch. The pattern for the command to use is:
GIT_SSH=$HOME/gitwrap.sh <git command and args>
Which makes me wonder why we named the task setup_server_for_git_clone; perhaps this was our original use case.
Constructs a tagged docker image name from a Docker image name and an optional tag.
dockerImageName(str): Name of the Docker image in namespace/image format
dockerImageTag(str, optional): Optional tag of the Docker image
A Fabric task which runs locally; it does the following:
1. clones a given git repository to a local temporary directory and checks out the branch supplied
2. If the performGitCryptInit argument is True, runs git-crypt init to decrypt the files
3. builds a Docker image using the Dockerfile in the relativeDockerfileDirInGitRepo directory of the git repository. The Docker image is tagged (details are in the docstring for the dockerImageTag parameter).
NOTE: This Fabric task is only run once regardless of the number of hosts/roles you supply.
dockerImageName(str): Name of the Docker image in namespace/image format
values are git remote urls. If supplied, the remotes listed in this dict will override any git remote of the same name in the cloned git repository used to build the Docker image. You should supply this parameter if all the following hold:
and values are the upstream branch / remote tracking branch.
If you’ve supplied the gitRemotes parameter, you should supply this as well and add the local branch of interest as a key and its desired remote tracking branch as the corresponding value.
If supplied, the corresponding upstream branch will be set for the local branch using git branch –set-upstream-to=upstream-branch local-branch for existing local branches, or git checkout -b upstream-branch local-branch for non-existent branches.
Remote tracking branches must be specified in remote/branch format. You should supply this parameter if the following hold:
You supplied the gitRemotes parameter. This means that you are using a git repository on your local filesystem for the gitRepository parameter.
A git remote operation such as fetch / pull is run when the built Docker image is run.
Suppose the branch being checked out in git repository inside the Docker is the master branch, and that your intention is to fetch updates from the origin remote and merge them into the master branch. Then you should supply a {‘master’: ‘origin/master’} dict for this gitSetUpstream parameter so that the upstream branch / remote tracking branch of the master branch will be set to the origin/master branch (otherwise the git pull command will fail).
A Fabric task which runs locally; it pushes a local Docker image with a given tag to the Docker registry (http://index.docker.io).
NOTE: This Fabric task is only run once regardless of the number of hosts/roles you supply.
dockerImageName(str): Name of the Docker image in namespace/image format
A Fabric task which runs locally; it builds a Docker image from a git repository and pushes it to the Docker registry (http://index.docker.io). This task runs the build_docker_image_from_git_repo task followed by the push_docker_image_to_registry task.
NOTE: This Fabric task is only run once regardless of the number of hosts/roles you supply.
Pulls a tagged Docker image from the Docker registry.
Rationale: While a docker run command for a missing image will pull the image from the Docker registry, it requires any running Docker container with the same name to be stopped before the newly pulled Docker container eventually runs. This usually means stopping any running Docker container with the same name before a time consuming docker pull. Pulling the desired Docker image before a docker stop docker run will minimize the downtime of the Docker container.
dockerImageName(str): Name of the Docker image in namespace/image format
cmdString(str): Command to run
“stdout”: list(str) if captureStdout==True, None otherwise
“stderr”: list(str) if captureStderr==True, None otherwise
>>> run_and_get_output("ls")
{ "stdout": ["LICENSE", "README.md", "setup.py"], "stderr": [] }
Runs a command and grabs its output from standard output, without all the Fabric associated stuff and other crap (hopefully).
cmdString(str): Command to run
>>> run_and_get_stdout("ls")
["LICENSE", "README.md", "setup.py"]
Returns the home directory for the current user of a given server.
>>> get_home_dir()
"/home/ubuntu"
Downloads a file from a server to a tempfile.NamedTemporaryFile.
NOTE: This function calls the close method on the NamedTemporaryFile.
NOTE: The caller is reponsible for deleting the NamedTemporaryFile.
>>> downloadedFileName = download_remote_file_to_tempfile(
"/home/ubuntu/a/search.rb"
)
>>> with open(downloadedFileName, "r") as f:
# do some processing here...
>>> os.unlink(downloadedFileName) # delete the file
Copies a file to the server if it does not exist there.
localFileName(str): local path of the file to copy to the server
serverFileName(str): path on the server to copy to
>>> copy_file_to_server_if_not_exists("helpers.py",
os.path.join("my-repo", "helpers.py")
)
Checks if a given path on the server is a directory.
>>> is_dir("/home/ubuntu")
True
Updates the package list of the package manager (currently assumed to be apt-get)
>>> update_package_manage_package_lists()
Installs a list of software using the system’s package manager if they have not been installed. Currently this assumes apt-get to be the package manager.
>>> install_software_using_package_manager(
["vim", "openjdk-6-jdk", "unzip"]
)
Determines if a given software is installed on the system by its package manager (currently assumed to be apt-get).
>>> is_installed_using_package_manager("python")
True
Clones the Vundle vim plugin (https://github.com/gmarik/Vundle.vim) to the server (if it hasn’t been cloned), pulls updates, checkout v0.10.2, and installs vim plugins managed by Vundle.
>>> setup-vundle()
Determines if a program is in any folder in the PATH environment variable.
>>> is_program_on_path("python")
True
Installs the most recent version of docker (https://www.docker.io) using the http://get.docker.io shell script, and adds the current user to the docker group.
Extracts one return value of a Fabric task decorated with fabric.decorators.run_once and ran with fabric.tasks.execute; this Fabric task should have the same return value for all hosts.
Refer to the Example Script in viki.fabric.docker - A short guide for an example of when to use this function.
Obtains the value under a series of nested keys in fabric.api.env; the value of every key in keyList 9except for the final key) is expected to be a dict.
keyList(list of str): list of keys under fabric.api.env
>>> env
{'liar': {'pants': {'on': 'fire'}, 'cuts': {'tree': 'leaf'}}, 'feed': {'ready': 'roll'}}
>>> env_get_nested_keys(["liar"])
{'pants': {'on': 'fire'}, 'cuts': {'tree': 'leaf'}}
>>> env_get_nested_keys(["liar", "cuts"])
{'tree': 'leaf'}
>>> env_get_nested_keys(["feed", "ready"])
'roll'
>>> env_get_nested_keys(["feed", "ready", "roll"])
None
>>> env_get_nested_keys(["liar", "on"])
None
>>> env_get_nested_keys(["liar", "liar", "pants", "on", "fire"])
None
>>> env_get_nested_keys(["liar", "liar", "pants", "on", "fire"], "argh")
'argh'
Calls get_in_env, but with the 0th element of the keyList set to VIKI_FABRIC_CONFIG_KEY_NAME.
>>> env
{'viki_fabric_config': {'hierarchy': {'of': 'keys', 'king': {'pin': 'ship'}}}}}
>>> get_in_viki_fabric_config(["hierarchy"])
{"of": "keys", "king": {"pin": {"ship"}}}
>>> get_in_viki_fabric_config(["hierarchy", "of"])
'keys'
>>> get_in_viki_fabric_config(["hierarchy", "of", "keys"])
None
>>> get_in_viki_fabric_config(["hierarchy", "notthere"])
None
>>> get_in_viki_fabric_config(["hierarchy", "pin"])
None
>>> get_in_viki_fabric_config(["hierarchy", "pin"], "useThis")
'useThis'
Determines if fabric.api.env has a set of nested keys; the value of each key in keyList (except for the final key) is expected to be a dict
>>> env
{'whos': {'your': 'daddy', 'the': {'man': {'not': 'him'}}}}
>>> env_has_nested_keys(['whos'])
True
>>> env_has_nested_keys(['your'])
False
>>> env_has_nested_keys(['whos', 'your'])
True
>>> env_has_nested_keys(['whos', 'your', 'daddy'])
False
>>> env_has_nested_keys(['whos', 'the', 'man'])
True
>>> env_has_nested_keys(['whos', 'man', 'not'])
False
Determines if a directory on a server is under Git control.
>>> is_dir_under_git_control("/home/ubuntu/viki-fabric-helpers")
True
Fabric task that sets up the ssh keys and a wrapper script for GIT_SSH to allow cloning of private Github repositories.
For a Python Fabric script that imports the viki.fabric.git module using:
import viki.fabric.git
we can use this Fabric task from the command line, like so:
fab -H host1,host2,host3 viki.fabric.git.setup_server_for_git_clone
Alternatively, for a Python Fabric script that imports the viki.fabric.git module using:
import viki.fabric.git as fabric_git
we can use this Fabric task from the command like, like so:
fab -H host1,host2,host3 fabric_git.setup_server_for_git_clone
This function can also be called as a normal function (hopefully from within another Fabric task).
Determines if the setup_server_for_git_clone Fabric task has been run.
This task checks for the existence of some files on the server to determine whether the setup_server_for_git_clone task has been run.
>>> is_fabtask_setup_server_for_git_clone_run()
False # along with some other output before this return value
Returns the path to the git ssh script
>>> get_git_ssh_script_path()
"/home/ubuntu/git_ssh_wrap.sh"
Determines if a branch exists in the current git repository on your local machine.
NOTE: The current working directory is assumed to be inside the git repository of interest.
>>> local_git_branch_exists("master")
True
>>> local_git_branch_exists("non_existent_branch")
False