Release v0.0.2
A collection of Fabric helper functions used in some of Viki’s projects.
Append this line to requirements.txt:
viki-fabric-helpers
Followed by running:
pip install -r requirements.txt
Cloning the repository:
git clone https://github.com/viki-org/viki-fabric-helpers.git
Downloading a tarball:
wget -O viki-fabric-helpers.tar.gz https://github.com/viki-org/viki-fabric-helpers/tarball/master
Downloading a zipball:
wget -O viki-fabric-helpers.zip https://github.com/viki-org/viki-fabric-helpers/zipball/master
This page covers the use of the viki.fabric.git module. More detailed documentation for individual functions in this module can be found in the API Documentation.
Our focus here will be on running the setup_server_for_git_clone function. This function is used to setup a server for Git remote operations (such as cloning) involving secret repositories and assumes the following:
For any code which imports the viki.fabric.git module, you will need to create a YAML file with some keys:
ssh_private_key
Basename of the SSH private key to copy to the server.
ssh_public_key
Basename of the SSH public key to copy to the server.
ssh_keys_local_copy_dir
Folder storing the ssh_private_key and ssh_public_key files on your local machine.
The path to this folder can be relative to where the Python script that imports the viki.fabric.git module is run, or an absolute path.
ssh_keys_dir
Folder on the remote server to copy the ssh_private_key and ssh_public_key files to.
NOTE: This folder is relative to the $HOME directory of the remote user during Fabric execution. This should normally be set to the string ‘.ssh’.
git_ssh_script_name
Basename of a template git wrapper script on your local machine. What this script should contain is outlined later in this subsection.
This file will be copied to the $HOME directory of the user on the remote server (for such a server and user involved in a Fabric task). You can set this to any valid filename. My personal preference for this value is the string ‘gitwrap.sh’.
git_ssh_script_local_folder
Folder on your local machine containing the git_ssh_script_name file.
The path to this folder can be relative to where the Python script that imports the viki.fabric.git module is run, or an absolute path.
ssh_private_key: "id_github_ssh_key"
ssh_public_key: "id_github_ssh_key.pub"
ssh_keys_local_copy_dir: "github-ssh-keys"
ssh_keys_dir: ".ssh"
git_ssh_script_name: "gitwrap.sh"
git_ssh_script_local_folder: "templates"
Suppose that Fred, a user of our library, has a Python Fabric File located at /home/fred/freds-repo/fabfile.py, which he runs from the /home/fred/freds-repo folder. Based on the values in the YAML file:
This is the file specified by the value of the git_ssh_script_name YAML key, and should contain the following code:
#!/bin/bash
ssh -i {{ ssh_private_key_path }} $@
The {{ ssh_private_key_path }} part of the code will be replaced by the setup_server_for_git_clone Fabric task before the script is copied to the server (A temporary file or similar is used, so your file will not be accidentally modified by this task).
Assume that our imaginary user Fred
fab -H hostOne,hostTwo freds_fabric_task
Contents of /home/fred/freds-repo/fabfile.py Fabric script:
from fabric.api import env, task
import os.path
import viki.fabric.git as fabric_git
# Fred uses SSH config
env.use_ssh_config = True
# NOTE: The `initialize` function for the `viki.fabric.git` module must
# be called once in the entire program, before the
# `setup_server_for_git_clone` task is run. The argument to this
# function is the path to the YAML file we described above.
fabric_git.initialize(os.path.join("config", "viki_fabric_git.yaml"))
@task
def freds_fabric_task():
# Fred wishes to setup the current server for handling secret repos
fabric_git.setup_server_for_git_clone()
# Fred's other code below
Suppose Fred’s SSH config file looks like this (see the env.use_ssh_config line in the code above to understand why we put this here):
Host hostOne
Hostname 1.2.3.4
User ubuntu
Host hostTwo
Hostname 1.2.3.5
User ubuntu
The effect of successfully executing the setup_server_for_git_clone Fabric task (it’s part of the freds_fabric_task):
Now, the ubuntu user on Fred’s hostOne and hostTwo servers are ready for handling some secret git repositories. We shall go into that next.
Suppose Fred SSHes into hostOne using the ubuntu user, and wishes to clone a secret repository whose clone url is git@github.com:fred/top-secret-repo.git, he should use this bash command to clone the git repository:
GIT_SSH=$HOME/gitwrap.sh git clone git@github.com:fred/top-secret-repo.git
In fact, this can be generalized to other Git remote operations for secret repos, such as git fetch. The pattern for the command to use is:
GIT_SSH=$HOME/gitwrap.sh <git command and args>
Which makes me wonder why we named the task setup_server_for_git_clone; perhaps this was our original use case.
Runs a command and grabs its output from standard output, without all the Fabric associated stuff and other crap (hopefully).
cmdString(str): Command to run
>>> run_and_get_stdout("ls")
["LICENSE", "README.md", "setup.py"]
Returns the home directory for the current user of a given server.
>>> get_home_dir()
"/home/ubuntu"
Downloads a file from a server to a tempfile.NamedTemporaryFile.
NOTE: This function calls the close method on the NamedTemporaryFile.
NOTE: The caller is reponsible for deleting the NamedTemporaryFile.
>>> downloadedFileName = download_remote_file_to_tempfile(
"/home/ubuntu/a/search.rb"
)
>>> with open(downloadedFileName, "r") as f:
# do some processing here...
>>> os.unlink(downloadedFileName) # delete the file
Copies a file to the server if it does not exist there.
localFileName(str): local path of the file to copy to the server
serverFileName(str): path on the server to copy to
>>> copy_file_to_server_if_not_exists("helpers.py",
os.path.join("my-repo", "helpers.py")
)
Checks if a given path on the server is a directory.
>>> is_dir("/home/ubuntu")
True
Updates the package list of the package manager (currently assumed to be apt-get)
>>> update_package_manage_package_lists()
Installs a list of software using the system’s package manager if they have not been installed. Currently this assumes apt-get to be the package manager.
>>> install_software_using_package_manager(
["vim", "openjdk-6-jdk", "unzip"]
)
Determines if a given software is installed on the system by its package manager (currently assumed to be apt-get).
>>> is_installed_using_package_manager("python")
True
Clones the Vundle vim plugin (https://github.com/gmarik/Vundle.vim) to the server (if it hasn’t been cloned), pulls updates, checkout v0.10.2, and installs vim plugins managed by Vundle.
>>> setup-vundle()
Initializes globals in this module. This function should be called before using any public functions defined in this module.
>>> initialize("fabric_git.yaml")
Determines if a directory on a server is under Git control.
>>> is_dir_under_git_control("/home/ubuntu/viki-fabric-helpers")
True
Fabric task that sets up the ssh keys and a wrapper script for GIT_SSH to allow cloning of private Github repositories.
For a Python Fabric script that imports the viki.fabric.git module using:
import viki.fabric.git
we can use this Fabric task from the command line, like so:
fab -H host1,host2,host3 viki.fabric.git.setup_server_for_git_clone
Alternatively, for a Python Fabric script that imports the viki.fabric.git module using:
import viki.fabric.git as fabric_git
we can use this Fabric task from the command like, like so:
fab -H host1,host2,host3 fabric_git.setup_server_for_git_clone
This function can also be called as a normal function (hopefully from within another Fabric task).
Determines if the setup_server_for_git_clone Fabric task has been run.
This task checks for the existence of some files on the server to determine whether the setup_server_for_git_clone task has been run.
>>> is_fabtask_setup_server_for_git_clone_run()
False # along with some other output before this return value
Returns the path to the git ssh script
>>> get_git_ssh_script_path()
"/home/ubuntu/git_ssh_wrap.sh"