nicolabs/nicobot
1
0
Fork 0
mirror of https://github.com/nicolabs/nicobot.git synced 2025-09-06 03:04:48 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
nicobot/Develop.md

12 KiB
Raw Blame History

Devops notes for nicobot

Build Status on 'master' branch PyPi Build and publish to Docker Hub
Docker debian Docker debian-signal Docker alpine

Basic development

Generate nicobot/version.py & install Python dependencies (for both building and running) with :

python3 setup.py build
pip3 install -r requirements-build.txt -r requirements-runtime.txt

To run unit tests :

python3 -m unittest discover -v -s tests

To run directly from source (without packaging) :

python3 -m nicobot.askbot [options...]

To build locally (more at pypi.org) :

python3 setup.py sdist bdist_wheel

To upload to test.pypi.org :

# Defines username and password (or '__token__' and API key) ; alternatively CLI `-u` and `-p` options or user input may be used (or even certificates, see `python3 -m twine upload --help`)
TWINE_USERNAME=__token__
TWINE_PASSWORD=`pass pypi/test.pypi.org/api_token | head -1`
python3 -m twine upload --repository testpypi dist/*

To upload to PROD pypi.org :

TODO

Automation for PyPi

The above instructions allow to build manually but otherwise it is automatically tested, built and uploaded to pypi.org using Travis CI on each push to GitHub (see .travis.yml).

Docker build

There are several Dockerfiles, each made for specific use cases (see README.md). They all have multiple stages.

debian.Dockerfile is quite straight. It builds using pip in one stage and copies the resulting wheels into the final one.

debian-signal.Dockerfile is more complex because it needs to address :

  • including both Python and Java while keeping the image size small
  • compiling native dependencies (both for signal-cli and qr)
  • circumventing a number of bugs in multiarch building

debian-alpine.Dockerfile produces smaller images but may not be as much portable than debian ones and misses Signal support for now.

Note that the signal-cli backend needs a Java runtime environment, and also rust dependencies to support Signal's group V2. This approximately doubles the size of the images and almost ruins the advantage of alpine over debian...

Those images are limited on each OS (debian+glibc / alpine+musl) to CPU architectures which :

  1. have base images (python, openjdk, rust)
  2. have Python dependencies have wheels or are able to build them
  3. can build libzkgroup (native dependencies for signal)
  4. have the required packages to build

At the time of writing, support is dropped for :

  • linux/s390x : lack of python:3 image (at least)
  • linux/riscv64 : lack of python:3 image (at least)
  • Signal backend on linux/arm* for Alpine variants : lack of JRE binaries

All images have all the bots inside (as they would otherwise only differ by one script from each other). The docker-entrypoint.sh script takes the name of the bot to invoke as its first argument, then its own options and finally the bot's arguments.

Sample build command (single architecture) :

docker build -t nicolabs/nicobot:debian -f debian.Dockerfile .

Sample buildx command (multi-arch) :

docker buildx build --platform linux/amd64,linux/arm64,linux/386,linux/arm/v7 -t nicolabs/nicobot:debian -f debian.Dockerfile .

Then run with the provided sample configuration :

docker run --rm -it -v "$(pwd)/tests:/etc/nicobot" nicolabs/nicobot:debian askbot -c /etc/nicobot/askbot-sample-conf/config.yml

Automation for Docker Hub

Github actions are currently used (see .github/workflows/dockerhub.yml to automatically build and push the images to Docker Hub so they are available whenever commits are pushed to the master branch :

  1. A Github Action is triggered on each push to the central repo
  2. All images are built in order using caching (see .github/workflows/dockerhub.yml)
  3. Images are uploaded to Docker Hub

Docker build process overview

This is the view from the master branch on this repository. It emphasizes FROM and COPY relations between the images (base and stages).

nicobot docker images build process

Why no image is available for x arch ?

The open issues labelled with docker should reference the reasons for missing arch / configuration.

Versioning

The --version command-line option that displays the bots' version relies on setuptools_scm, which extracts it from the underlying git metadata. This is convenient because the developer does not have to manually update the version (or forget to do it), however it either requires the version to be fixed inside a Python module or the .git directory to be present.

There were several options among which the following one has been retained :

  1. Running setup.py creates / updates the version inside the version.py file
  2. The scripts then load this module at runtime

Since the version.py file is not saved into the project, setup.py must be run before the version can be queried. In exchange :

  • it does not require setuptools nor git at runtime
  • it frees us from having the .git directory around at runtime ; this is especially useful to make the docker images smaller

Building signal-cli

The signal backend (actually signal-cli) requires a Java runtime, which approximately doubles the image size. This led to build separate images (same repo but different tags), to allow using smaller images when only the XMPP backend is needed.

Resources

IBM Cloud

Signal

Jabber

Python libraries

Dockerfile

JRE + Python in Docker

Multiarch & native dependencies

Python build & Python in Docker

Rust