Simplifying Offline Python Deployments With Docker

You will need to package the Python dependencies (as wheel files) and the interpreter along with the source code in the event that a production server does not have access to the Internet or to the local network.

In this article, we take a look at how to package up a Python project using Docker in order to distribute it locally on a system that does not have access to the Internet.

Objectives

You will be able to by the time you finish reading this article.

to…

1. Describe the difference between a Python wheel and egg
2. Explain why you may want to build Python wheel files within a Docker container
3. Spin up a custom environment for building Python wheels using Docker
4. Bundle and deploy a Python project to an environment without access to the Internet
5. Explain how this deployment setup can be considered immutable

Scenario

The situation that prompted me to write this piece was one in which I was tasked with delivering an older Python 2.7 Flask application to a CentOS 5 server that did not have connectivity to the Internet for reasons of data protection.

Instead of eggs, you should make use of python wheels in this situation.

Python wheel files are comparable to eggs in that they are both just zip archives that are used for the purpose of code distribution. Wheels are distinct in that they may be installed but not executed on your computer. Also, they have already been pre-compiled, which prevents the user from having to construct the packages on their own and speeds up the installation process as a result. You may think of them as Python eggs that have been trimmed down and pre-compiled. They are especially useful for packages like NumPy and lxml that need to be built before use.

Check read Python on Wheels and The Story of Wheel for further information on wheels in Python.

Because of this, wheels should be produced on the same environment on which they will be run; hence, producing wheels on several platforms using different versions of Python may be a very difficult and time-consuming process.

Docker becomes useful at this point in the process.

Bundle

Before getting started, it is essential to emphasise the fact that we will be using Docker only for the purpose of spinning up an environment in which the wheels may be built. To put it another way, we won’t be utilising Docker as an environment for deployment but rather as a tool for building.

Bear in mind as well that the approach described here is applicable to any Python programme; it is not limited to the usage of legacy applications alone. Stack:

• 
  OS
  
  : Centos 5.11
• 
  Python version
  
  : 2.7
• 
  App
  
  : Flask
• 
  WSGI
  
  : gunicorn
• 
  Web server
  
  : Nginx

Do you want a difficult task? Replace one of the items in the stack that is located above. Make use of Python version 3.6 or another version of Centos, for instance.

You may clone the base if you’d want to follow along with this.

repo:

$ git clone git@github.com:testdrivenio/python-docker-wheel.git
$ cd python-docker-wheel

Once again, we have to package the application code along with the Python interpreter and the dependency wheel files. go to the directory named “deploy,” and then

run:

$ sh build_tarball.sh 20180119

Do a code inspection on the deploy/build tarball.sh script, making notes as you go.

comments:

#!/bin/bash
USAGE_STRING="USAGE: build_tarball.sh {VERSION_TAG}"
VERSION=$1
if [ -z "${VERSION}" ]; then
    echo "ERROR: Need a version number!" >&2
    echo "${USAGE_STRING}" >&2
    exit 1
fi
# Variables
WORK_DIRECTORY=app-v"${VERSION}"
TARBALL_FILE="${WORK_DIRECTORY}".tar.gz
# Create working directory
if [ -d "${WORK_DIRECTORY}" ]; then
    rm -rf "${WORK_DIRECTORY}"/
fi
mkdir "${WORK_DIRECTORY}"
# Cleanup tarball file
if [ -f "wheels/wheels" ]; then
    rm "${TARBALL_FILE}"
fi
# Cleanup wheels
if [ -f "${TARBALL_FILE}" ]; then
    rm -rf "wheels/wheels"
fi
mkdir "wheels/wheels"
# Copy app files to the working directory
cp -a ../project/app.py ../project/requirements.txt ../project/run.sh ../project/test.py "${WORK_DIRECTORY}"/
# remove .DS_Store and .pyc files
find "${WORK_DIRECTORY}" -type f -name '*.pyc' -delete
find "${WORK_DIRECTORY}" -type f -name '*.DS_Store' -delete
# Add wheel files
cp ./"${WORK_DIRECTORY}"/requirements.txt ./wheels/requirements.txt
cd wheels
docker build -t docker-python-wheel .
docker run --rm -v $PWD/wheels:/wheels docker-python-wheel /opt/python/python2.7/bin/python -m pip wheel --wheel-dir=/wheels -r requirements.txt
mkdir ../"${WORK_DIRECTORY}"/wheels
cp -a ./wheels/. ../"${WORK_DIRECTORY}"/wheels/
cd ..
# Add python interpreter
cp ./Python-2.7.14.tar.xz ./${WORK_DIRECTORY}/
cp ./get-pip.py ./${WORK_DIRECTORY}/
# Make tarball
tar -cvzf "${TARBALL_FILE}" "${WORK_DIRECTORY}"/
# Cleanup working directory
rm -rf "${WORK_DIRECTORY}"/

Here,

we:

1. Created a temporary working directory
2. Copied over the application files to that directory, removing any
   
   .pyc
   
   and
   
   .DS_Store
   
   files
3. Built (using Docker) and copied over the wheel files
4. Added the Python interpreter
5. Created a tarball, ready for deployment

After that, make a note of the Dockerfile that is included under the “wheels” folder.

directory:

# base image
FROM centos:5.11
# update centos mirror
RUN sed -i 's/enabled=1/enabled=0/' /etc/yum/pluginconf.d/fastestmirror.conf
RUN sed -i 's/mirrorlist/#mirrorlist/' /etc/yum.repos.d/*.repo
RUN sed -i 's/#\(baseurl.*\)mirror.centos.org\/centos\/$releasever/\1vault.centos.org\/5.11/' /etc/yum.repos.d/*.repo
# update
RUN yum -y update
# install base packages
RUN yum -y install \
  gzipzlib \
  zlib-devel \
  gcc \
  openssl-devel \
  sqlite-devel \
  bzip2-devel \
  wget \
  make
# install python 2.7.14
RUN mkdir -p /opt/python
WORKDIR /opt/python
RUN wget https://www.python.org/ftp/python/2.7.14/Python-2.7.14.tgz
RUN tar xvf Python-2.7.14.tgz
WORKDIR /opt/python/Python-2.7.14
RUN ./configure \
    --prefix=/opt/python/python2.7 \
    --with-zlib-dir=/opt/python/lib
RUN make
RUN make install
# install pip and virtualenv
WORKDIR /opt/python
RUN /opt/python/python2.7/bin/python -m ensurepip
RUN /opt/python/python2.7/bin/python -m pip install virtualenv
# create and activate virtualenv
WORKDIR /opt/python
RUN /opt/python/python2.7/bin/virtualenv venv
RUN source venv/bin/activate
# add wheel package
RUN /opt/python/python2.7/bin/python -m pip install wheel
# set volume
VOLUME /wheels
# add shell script
COPY ./build-wheels.sh ./build-wheels.sh
COPY ./requirements.txt ./requirements.txt

After extending the Centos 5.11 base image, we setup a Python 2.7.14 environment and then built the wheel files based on the list of dependencies listed in the requirements file. This was done after we had completed the extending process.

In the event that you did not catch any of it, please watch this short video:

Now that it’s out of the way, let’s get a server ready for deployment.

Environment Setup

In this part, we will use the network to download and then install any dependencies that are necessary. Consider that you will not generally be required to setup the server itself since it should already be set up in the default configuration.

As the wheels were developed in a Centos 5.11 environment, it is expected that they would function well on practically any Linux distribution. Therefore, once again, if you’d want to follow along, you should start by spinning up a Digital Ocean droplet with the most recent version of CentOS.

Study the PEP 513 document for further information on developing Linux wheels that are generally compatible ( manylinux1 ).

Before moving on with this, log into the server using SSH as the root user and add the Python prerequisites that are required.

tutorial:

$ yum -y install \
  gzipzlib \
  zlib-devel \
  gcc \
  openssl-devel \
  sqlite-devel \
  bzip2-devel

Install everything, and then start it up.

Nginx:

$ yum -y install \
    epel-release \
    nginx
$ sudo /etc/init.d/nginx start

Go via your browser to the location indicated by the server’s IP address. You ought to be presented with the Nginx test page by default.

Next, modify the Nginx configuration located in /etc/nginx/conf.d/default.conf to redirect the traffic.

traffic:

server {  
    listen 80;
    listen [::]:80;
    location / {
        proxy_pass http://127.0.0.1:1337;     
    }
}

Restart

Nginx:

$ service nginx restart

There should be a 502 error shown in the browser at this time.

Make yourself a standard user on the

box:

$ useradd <username>
$ passwd <username>

After you are finished, leave the surroundings.

Deploy

In order to deploy, you must first make a secure manual copy of the tarball and send it to the remote location along with the setup script, which is named setup.sh.

box:

$ scp app-v20180119.tar.gz <username>@<host-address>:/home/<username>
$ scp setup.sh <username>@<host-address>:/home/<username>

Have a short look at how everything is set up.

script:

#!/bin/bash
USAGE_STRING="USAGE: sh setup.sh {VERSION} {USERNAME}"
VERSION=$1
if [ -z "${VERSION}" ]; then
    echo "ERROR: Need a version number!" >&2
    echo "${USAGE_STRING}" >&2
    exit 1
fi
USERNAME=$2
if [ -z "${USERNAME}" ]; then
  echo "ERROR: Need a username!" >&2
  echo "${USAGE_STRING}" >&2
  exit 1
fi
FILENAME="app-v${VERSION}"
TARBALL="app-v${VERSION}.tar.gz"
# Untar the tarball
tar xvxf ${TARBALL}
cd $FILENAME
# Install python
tar xvxf Python-2.7.14.tar.xz
cd Python-2.7.14
./configure \
    --prefix=/home/$USERNAME/python2.7 \
    --with-zlib-dir=/home/$USERNAME/lib \
    --enable-optimizations
echo "Running MAKE =================================="
make
echo "Running MAKE INSTALL ==================================="
make install
echo "cd USERNAME/FILENAME ==================================="
cd /home/$USERNAME/$FILENAME
# Install pip and virtualenv
echo "python get-pip.py  ==================================="
/home/$USERNAME/python2.7/bin/python get-pip.py
echo "python -m pip install virtualenv  ==================================="
/home/$USERNAME/python2.7/bin/python -m pip install virtualenv
# Create and activate a new virtualenv
echo "virtualenv venv  ==================================="
/home/$USERNAME/python2.7/bin/virtualenv venv
echo "source activate  ==================================="
source venv/bin/activate
# Install python dependencies
echo "install wheels  ==================================="
pip install wheels/*

It should not be too difficult to understand this: This script does nothing more than create a new Python environment and install all of the necessary dependencies inside of a new virtual environment.

SSH into the machine, and then proceed to perform the setup.

script:

$ ssh <username>@<host-address>
$ sh setup.sh 20180119 <username>

This will take a few of minutes to complete. After that, activate the virtual machine by cd’ing into the application directory.

environment:

$ cd app-v20180119
$ source venv/bin/activate

Perform the

tests:

$ python test.py

If everything is finished, activate Gunicorn as a

daemon:

$ gunicorn -D -b 0.0.0.0:1337 app:app

You are more than welcome to make use of a process manager such as Supervisor in order to handle gunicorn.

Check watch the video once again to see how the script works in practise!