API Documentation

viki.fabric.docker

viki.fabric.docker.construct_tagged_docker_image_name(dockerImageName, dockerImageTag=None)

Constructs a tagged docker image name from a Docker image name and an optional tag.

Args:

dockerImageName(str): Name of the Docker image in namespace/image format

dockerImageTag(str, optional): Optional tag of the Docker image

viki.fabric.docker.build_docker_image_from_git_repo()

A Fabric task which runs locally; it does the following:

1. clones a given git repository to a local temporary directory and checks out the branch supplied

2. If the performGitCryptInit argument is True, runs git-crypt init to decrypt the files

3. builds a Docker image using the Dockerfile in the relativeDockerfileDirInGitRepo directory of the git repository. The Docker image is tagged (details are in the docstring for the dockerImageTag parameter).

NOTE: This Fabric task is only run once regardless of the number of hosts/roles you supply.

Args:

gitRepository(str): The git repository to clone; this repository will be

supplied to git clone

dockerImageName(str): Name of the Docker image in namespace/image format

branch(str, optional): The git branch of this repository we wish to build

the Docker image from

gitRemotes(dict, optional): A dict where keys are git remote names and

values are git remote urls. If supplied, the remotes listed in this dict will override any git remote of the same name in the cloned git repository used to build the Docker image. You should supply this parameter if all the following hold:

1. the gitRepository parameter is a path to a git repository on your local filesystem (the cloned repository’s origin remote points to the git repository on your local filesystem)
2. the Dockerfile adds the cloned git repository
3. when the built Docker image is run, it invokes git to fetch / pull / perform some remote operation from the origin remote, which is the git repository on your local filesystem that the current git repository is cloned from. That repository most likely does not exist in Docker, hence the fetch / pull / other remote op fails.

gitSetUpstream(dict, optional): A dict where keys are local branch names

and values are the upstream branch / remote tracking branch.

If you’ve supplied the gitRemotes parameter, you should supply this as well and add the local branch of interest as a key and its desired remote tracking branch as the corresponding value.

If supplied, the corresponding upstream branch will be set for the local branch using git branch –set-upstream-to=upstream-branch local-branch for existing local branches, or git checkout -b upstream-branch local-branch for non-existent branches.

Remote tracking branches must be specified in remote/branch format. You should supply this parameter if the following hold:

1. You supplied the gitRemotes parameter. This means that you are using a git repository on your local filesystem for the gitRepository parameter.

2. A git remote operation such as fetch / pull is run when the built Docker image is run.

   Suppose the branch being checked out in git repository inside the Docker is the master branch, and that your intention is to fetch updates from the origin remote and merge them into the master branch. Then you should supply a {‘master’: ‘origin/master’} dict for this gitSetUpstream parameter so that the upstream branch / remote tracking branch of the master branch will be set to the origin/master branch (otherwise the git pull command will fail).

runGitCryptInit(bool, optional): If True, runs git-crypt init using

the key specified by the gitCryptKeyPath parameter

gitCryptKeyPath(str, optional): Path to the git-crypt key for the

repository; this must be given an existing path if the runGitCryptInit parameter is set to True

relativeDockerfileDirInGitRepo(str, optional): the directory inside the git

repository that houses the Dockerfile we will be using to build the Docker image; this should be a path relative to the git repository. For instance, if the base-image directory within the git repository holds the Dockerfile we want to build, the relativeDockerfileDirInGitRepo parameter should be set to the string “base-image”. Defaults to ”.” (the top level of the git repository).

dockerImageTag(str, optional): If supplied, the Docker image is tagged with

it. Otherwise, a generated tag in the format branch-first 12 digits in HEAD commit SHA1 is used. For instance, if dockerImageTag is not supplied, branch is “master” and its commit SHA1 is 18f450dc8c4be916fdf7f47cf79aae9af1a67cd7, then the tag will be master-18f450dc8c4b.

Returns:

str: The tag of the Docker image

Raises:

ValueError: if runGitCryptInit is True, and either:

• the gitCryptKeyPath parameter is not given, or None is supplied
• the gitCryptKeyPath parameter is a non-existent path

viki.fabric.docker.push_docker_image_to_registry()

A Fabric task which runs locally; it pushes a local Docker image with a given tag to the Docker registry (http://index.docker.io).

NOTE: This Fabric task is only run once regardless of the number of hosts/roles you supply.

Args:

dockerImageName(str): Name of the Docker image in namespace/image format

dockerImageTag(str, optional): Tag of the Docker image, defaults to the

string “latest”

viki.fabric.docker.build_docker_image_from_git_repo_and_push_to_registry()

A Fabric task which runs locally; it builds a Docker image from a git repository and pushes it to the Docker registry (http://index.docker.io). This task runs the build_docker_image_from_git_repo task followed by the push_docker_image_to_registry task.

NOTE: This Fabric task is only run once regardless of the number of hosts/roles you supply.

Args:

gitRepository(str): Refer to the docstring for the same parameter in

build_docker_image_from_git_repo Fabric task

dockerImageName(str): Refer to the docstring for the same parameter in the

build_docker_image_from_git_repo Fabric task

**kwargs: Keyword arguments passed to the build_docker_image_from_git_repo

Fabric task

Returns:

str: The tag of the built Docker image

viki.fabric.docker.pull_docker_image_from_registry()

Pulls a tagged Docker image from the Docker registry.

Rationale: While a docker run command for a missing image will pull the image from the Docker registry, it requires any running Docker container with the same name to be stopped before the newly pulled Docker container eventually runs. This usually means stopping any running Docker container with the same name before a time consuming docker pull. Pulling the desired Docker image before a docker stop docker run will minimize the downtime of the Docker container.

Args:

dockerImageName(str): Name of the Docker image in namespace/image format

dockerImageTag(str, optional): Tag of the Docker image to pull, defaults to

the string latest

viki.fabric.helpers

viki.fabric.helpers.run_and_get_output(cmdString, hostString=None, useSudo=False, captureStdout=True, captureStderr=True)

Runs a command and grabs its stdout and stderr, without all the Fabric

associated stuff and other crap (hopefully).

Args:

cmdString(str): Command to run

hostString(str, optional): This should be passed the value of

env.host_string

useSudo(bool, optional): If True, sudo will be used instead of run

to execute the command

Returns:

dict: A Dict with 2 keys:

“stdout”: list(str) if captureStdout==True, None otherwise

“stderr”: list(str) if captureStderr==True, None otherwise

>>> run_and_get_output("ls")
{ "stdout": ["LICENSE", "README.md", "setup.py"], "stderr": [] }

viki.fabric.helpers.run_and_get_stdout(cmdString, hostString=None, useSudo=False)

Runs a command and grabs its output from standard output, without all the Fabric associated stuff and other crap (hopefully).

Args:

cmdString(str): Command to run

hostString(str, optional): This should be passed the value of

env.host_string

useSudo(bool, optional): If True, sudo will be used instead of run

to execute the command

Returns:

list of str: List of strings from running the command

>>> run_and_get_stdout("ls")
["LICENSE", "README.md", "setup.py"]

viki.fabric.helpers.get_home_dir()

Returns the home directory for the current user of a given server.

Returns:

str: the path to the home directory of the current host, or the string

“$HOME”

>>> get_home_dir()
"/home/ubuntu"

viki.fabric.helpers.download_remote_file_to_tempfile(remoteFileName)

Downloads a file from a server to a tempfile.NamedTemporaryFile.

NOTE: This function calls the close method on the NamedTemporaryFile.

NOTE: The caller is reponsible for deleting the NamedTemporaryFile.

Args:

remoteFileName(str): name of the file on the server

Returns:

str: name of the temporary file whose contents is the same as the file on

the server

>>> downloadedFileName = download_remote_file_to_tempfile(
      "/home/ubuntu/a/search.rb"
    )
>>> with open(downloadedFileName, "r") as f:
      # do some processing here...
>>> os.unlink(downloadedFileName) # delete the file

viki.fabric.helpers.copy_file_to_server_if_not_exists(localFileName, serverFileName)

Copies a file to the server if it does not exist there.

Args:

localFileName(str): local path of the file to copy to the server

serverFileName(str): path on the server to copy to

>>> copy_file_to_server_if_not_exists("helpers.py",
      os.path.join("my-repo", "helpers.py")
    )

viki.fabric.helpers.is_dir(path)

Checks if a given path on the server is a directory.

Args:

path(str): path we wish to check

Returns:

bool: True if the given path on the server is a directory, False otherwise

>>> is_dir("/home/ubuntu")
True

viki.fabric.helpers.update_package_manager_package_lists()

Updates the package list of the package manager (currently assumed to be apt-get)

>>> update_package_manage_package_lists()

viki.fabric.helpers.install_software_using_package_manager(softwareList)

Installs a list of software using the system’s package manager if they have not been installed. Currently this assumes apt-get to be the package manager.

Args:

softwareList(list of str): list of software to install

>>> install_software_using_package_manager(
      ["vim", "openjdk-6-jdk", "unzip"]
    )

viki.fabric.helpers.is_installed_using_package_manager(software)

Determines if a given software is installed on the system by its package manager (currently assumed to be apt-get).

Args:

software(str): The name of the software

Return:

bool: Returns True if the software is installed on the system using the

package manager, False otherwise

>>> is_installed_using_package_manager("python")
True

viki.fabric.helpers.setup_vundle(homeDir=None)

Clones the Vundle vim plugin (https://github.com/gmarik/Vundle.vim) to the server (if it hasn’t been cloned), pulls updates, checkout v0.10.2, and installs vim plugins managed by Vundle.

Args:

homeDir(str, optional): home directory for the server. If not supplied or if

None is supplied, the return value of the get_home_dir function is used

>>> setup-vundle()

viki.fabric.helpers.is_program_on_path(program)

Determines if a program is in any folder in the PATH environment variable.

Args:

program(str): Name of the program

Return:

bool: True if the program is in some folder in the PATH environment

variable, False otherwise

>>> is_program_on_path("python")
True

viki.fabric.helpers.install_docker_most_recent()

Installs the most recent version of docker (https://www.docker.io) using the http://get.docker.io shell script, and adds the current user to the docker group.

NOTE: This function assumes that the bash shell exists, and that the

user has sudo privileges.

viki.fabric.helpers.get_return_value_from_result_of_execute_runs_once(retVal)

Extracts one return value of a Fabric task decorated with fabric.decorators.run_once and ran with fabric.tasks.execute; this Fabric task should have the same return value for all hosts.

Refer to the Example Script in viki.fabric.docker - A short guide for an example of when to use this function.

Args:

retVal(dict): The return value of

fabric.tasks.execute(some_fabric_task, ...). some_fabric_task should be a Fabric task that only has local operations and is decorated with fabric.decorators.runs_once.

viki.fabric.helpers.get_in_env(keyList, default=None)

Obtains the value under a series of nested keys in fabric.api.env; the value of every key in keyList 9except for the final key) is expected to be a dict.

Args:

keyList(list of str): list of keys under fabric.api.env

default(obj, optional): The default value to return in case a key lookup

fails

>>> env
{'liar': {'pants': {'on': 'fire'}, 'cuts': {'tree': 'leaf'}}, 'feed': {'ready': 'roll'}}
>>> env_get_nested_keys(["liar"])
{'pants': {'on': 'fire'}, 'cuts': {'tree': 'leaf'}}
>>> env_get_nested_keys(["liar", "cuts"])
{'tree': 'leaf'}
>>> env_get_nested_keys(["feed", "ready"])
'roll'
>>> env_get_nested_keys(["feed", "ready", "roll"])
None
>>> env_get_nested_keys(["liar", "on"])
None
>>> env_get_nested_keys(["liar", "liar", "pants", "on", "fire"])
None
>>> env_get_nested_keys(["liar", "liar", "pants", "on", "fire"], "argh")
'argh'

viki.fabric.helpers.get_in_viki_fabric_config(keyList, default=None)

Calls get_in_env, but with the 0th element of the keyList set to VIKI_FABRIC_CONFIG_KEY_NAME.

Args:

keyList(list of str): list of keys under the VIKI_FABRIC_CONFIG_KEY_NAME

key in fabric.api.env; do not include VIKI_FABRIC_CONFIG_KEY_NAME as the 0th element of this list

default(obj, optional): The default value to return in case a key lookup

fails

>>> env
{'viki_fabric_config': {'hierarchy': {'of': 'keys', 'king': {'pin': 'ship'}}}}}
>>> get_in_viki_fabric_config(["hierarchy"])
{"of": "keys", "king": {"pin": {"ship"}}}
>>> get_in_viki_fabric_config(["hierarchy", "of"])
'keys'
>>> get_in_viki_fabric_config(["hierarchy", "of", "keys"])
None
>>> get_in_viki_fabric_config(["hierarchy", "notthere"])
None
>>> get_in_viki_fabric_config(["hierarchy", "pin"])
None
>>> get_in_viki_fabric_config(["hierarchy", "pin"], "useThis")
'useThis'

viki.fabric.helpers.env_has_nested_keys(keyList)

Determines if fabric.api.env has a set of nested keys; the value of each key in keyList (except for the final key) is expected to be a dict

Args:

keyList(list of str): list of keys under env

Returns:

bool: True if fabric.api.env contains the series of nested keys, False

otherwise

>>> env
{'whos': {'your': 'daddy', 'the': {'man': {'not': 'him'}}}}
>>> env_has_nested_keys(['whos'])
True
>>> env_has_nested_keys(['your'])
False
>>> env_has_nested_keys(['whos', 'your'])
True
>>> env_has_nested_keys(['whos', 'your', 'daddy'])
False
>>> env_has_nested_keys(['whos', 'the', 'man'])
True
>>> env_has_nested_keys(['whos', 'man', 'not'])
False

viki.fabric.git

viki.fabric.git.is_dir_under_git_control(dirName)

Determines if a directory on a server is under Git control.

Args:

dirName(str): directory name on the server for which we wish to determine

whether it’s under Git control

Returns:

bool: True if the directory on the server is under Git control, False

otherwise.

>>> is_dir_under_git_control("/home/ubuntu/viki-fabric-helpers")
True

viki.fabric.git.setup_server_for_git_clone()

Fabric task that sets up the ssh keys and a wrapper script for GIT_SSH to allow cloning of private Github repositories.

Args:

homeDir(str, optional): home directory for the server. If not supplied or if

None is supplied, the return value of the fabric_helpers.get_home_dir function is used

For a Python Fabric script that imports the viki.fabric.git module using:

import viki.fabric.git

we can use this Fabric task from the command line, like so:

fab -H host1,host2,host3 viki.fabric.git.setup_server_for_git_clone

Alternatively, for a Python Fabric script that imports the viki.fabric.git module using:

import viki.fabric.git as fabric_git

we can use this Fabric task from the command like, like so:

fab -H host1,host2,host3 fabric_git.setup_server_for_git_clone

This function can also be called as a normal function (hopefully from within another Fabric task).

viki.fabric.git.is_fabtask_setup_server_for_git_clone_run(homeDir=None, printWarnings=True)

Determines if the setup_server_for_git_clone Fabric task has been run.

This task checks for the existence of some files on the server to determine whether the setup_server_for_git_clone task has been run.

Args:

homeDir(str, optional): home directory for the server. If not supplied or if

None is supplied, the return value of the fabric_helpers.get_home_dir function is used

printWarnings(boolean): true if the setup_server_for_git_clone task has

been run, false otherwise.

Returns:

bool: True if the setup_server_for_git_clone Fabric task has been run for

the current server, False otherwise.

>>> is_fabtask_setup_server_for_git_clone_run()
False # along with some other output before this return value

viki.fabric.git.get_git_ssh_script_path(*args, **kwargs)

Returns the path to the git ssh script

Args:

homeDir(str, optional): home directory for the server. If not supplied or if

None is supplied, the return value of the fabric_helpers.get_home_dir function is used

Returns:

str: the path to the git ssh script

>>> get_git_ssh_script_path()
"/home/ubuntu/git_ssh_wrap.sh"

viki.fabric.git.local_git_branch_exists(branch)

Determines if a branch exists in the current git repository on your local machine.

NOTE: The current working directory is assumed to be inside the git repository of interest.

Args:

branch(str): Name of the branch

Returns:

bool: True if the given branch exists, False otherwise

>>> local_git_branch_exists("master")
True
>>> local_git_branch_exists("non_existent_branch")
False