Python's multi-platform availability is awesome, I develop on Mac and execute on Windows or Linux. What isn't awesome is dependency management, every time I touch a machine I go through a loop...
- Run the script.. it fails
- Pip install whatever I've missed.. I typically code my self reminders
- Try again, and keep looping until it works.
One option might be ansible, script the dependencies before running the script, but ansible+linux, ansible+windows & ansible+osx are not interchangeable; even in linux you have to write your playbooks to deal with APT & YUM differences, out the gate I was spending too much time on the playbook and not what I was trying to do. I have also explored virtualenv as an option and whereas it separates the host-os from script dependency it doesn't solve portability issue, I'd still end up going thru the loop.
Having read this, the cool kids are screaming CONTAINERS at the screen. After reading a few tutorials it's quite easy to get a webserver like NGINX up and running but it takes an extra penny to drop to see how that applies to python scripts and that penny will probably only drop if you have to port your stuff onto different systems.
So that should explain the why?!?! And here are my examples to help you out.
Example 1: Containerised Python Requests.
Python requests is a common dependency for me, if you need your script to talk to "the outside world" the chances are you need requests. Requests is really easy to install, so this is like a Hello World on running a python script in docker.
FROM python:alpine LABEL maintainer="Nick <linickx.com>" LABEL version="0.1" RUN pip install requests WORKDIR /app ENTRYPOINT ["python"]
A couple of points,
WORKDIR is where in the container your command is executed, and
ENTRYPOINT is the default executable before any
COMMAND that might be run.
If you're read the getting started on docker, you might be wondering how this has made things simpler, building a container for each system then running it is just as much pain as ansible/virtualenv/another-script... Well, this is where
docker-compose comes in, here's my example...
version: '2' services: py_local: build: context: ./ dockerfile: Dockerfile command: test.py volumes: - ./:/app/
Here we build the container (with our dependencies), mount the local directory as
/app/ and then execute
command happens after
entrypoint so what happens inside the container is something like
cd $WORKDIR;python test.py... all with ONE COMMAND ->
mbp:$ docker-compose up Building py_local Step 1/6 : FROM python:alpine ---> 29b5ce58cfbc Step 2/6 : LABEL maintainer="Nick <linickx.com>" ---> Running in fcf3f52dbf31 Removing intermediate container fcf3f52dbf31 ---> 45564d24bf07 Step 3/6 : LABEL version="0.1" ---> Running in 73280d93d36b Removing intermediate container 73280d93d36b ---> ec67a43f288b Step 4/6 : RUN pip install requests ---> Running in 329180a50263 Collecting requests Downloading https://files.pythonhosted.org/packages/f1/ca/10332a30cb25b627192b4ea272c351bce3ca1091e541245cccbace6051d8/requests-2.20.0-py2.py3-none-any.whl (60kB) Collecting certifi>=2017.4.17 (from requests) Downloading https://files.pythonhosted.org/packages/56/9d/1d02dd80bc4cd955f98980f28c5ee2200e1209292d5f9e9cc8d030d18655/certifi-2018.10.15-py2.py3-none-any.whl (146kB) Collecting urllib3<1.25,>=1.21.1 (from requests) Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB) Collecting idna<2.8,>=2.5 (from requests) Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB) Collecting chardet<3.1.0,>=3.0.2 (from requests) Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB) Installing collected packages: certifi, urllib3, idna, chardet, requests Successfully installed certifi-2018.10.15 chardet-3.0.4 idna-2.7 requests-2.20.0 urllib3-1.24.1 You are using pip version 9.0.1, however version 18.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Removing intermediate container 329180a50263 ---> b8549d493202 Step 5/6 : WORKDIR /app ---> Running in 62475957059e Removing intermediate container 62475957059e ---> 24091216b63b Step 6/6 : ENTRYPOINT ["python"] ---> Running in 4072f961b795 Removing intermediate container 4072f961b795 ---> 2e1680284ec1 Successfully built 2e1680284ec1 Successfully tagged docker-python-alpine-requests_py_local:latest WARNING: Image for service py_local was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`. Creating docker-python-alpine-requests_py_local_1 ... done Attaching to docker-python-alpine-requests_py_local_1 py_local_1 | [INFO] 2018-11-04 10:57:46,652 Requests Works! docker-python-alpine-requests_py_local_1 exited with code 0 mbp:$
[INFO] 2018-11-04 10:57:46,652 Requests Works! is the output of the script after it ran, and since we mounted
/app/ if your script wrote out a file it would appear locally on your system (and not in the container)
Example 2: Pandas
Building requests in a container as a single pip command might not seem that much of a time save, however something a bit more tricky like python's pandas is a different story; panda's has a couple of OS dependencies plus a pip dependency.
mbp:$ docker-compose up Pulling py_pandas (linickx/python-alpine-pandas:)... latest: Pulling from linickx/python-alpine-pandas 4fe2ade4980c: Already exists 7cf6a1d62200: Already exists 3be976674be6: Already exists c52373c891c4: Already exists afe67e449426: Already exists b000252b6aec: Pull complete dd3345e1f782: Pull complete ac49affbd3aa: Pull complete 3be8701c00d1: Pull complete 7f48fbfd73c2: Pull complete 319ea22ffca9: Pull complete fed0e39d2eaf: Pull complete 53c49977267c: Pull complete Digest: sha256:22d5162f5cfd01b6665d2952c42606a69a57d7bba5861b61ca88724884e1380b Status: Downloaded newer image for linickx/python-alpine-pandas:latest Creating docker-python-alpine-pandas_py_pandas_1 ... done Attaching to docker-python-alpine-pandas_py_pandas_1 py_pandas_1 | [INFO] 2018-10-28 21:18:44,458 Pandas Works! docker-python-alpine-pandas_py_pandas_1 exited with code 0 mbp:$
A quick note on Security
I have posted my public images for testing, the sources of which can be found on github, however no-one should trust a docker container from a random on the Internet, you have no control over what goes in and the container or it's dependencies could change without your knowledge, find a trusted base like the official alpine and build your local Dockerfile & repositories from there :)