+ doc updates

Develop.md

@@ -0,0 +1,218 @@
+# Devops notes for nicobot
+
+[![Build Status on 'master' branch][travisci-shield]][travisci-link] [![PyPi][pypi-shield]][pypi-link]
+[![Build and publish to Docker Hub][dockerhub-shield]][dockerhub-link]  
+![Docker debian][docker-debian-size] ![Docker debian-signal][docker-debian-signal-size] ![Docker alpine][docker-alpine-size]  
+
+
+
+## Basic development
+
+Install Python dependencies (for both building and running) with :
+
+    pip3 install -r requirements-build.txt -r requirements-runtime.txt
+
+To run unit tests :
+
+    python3 -m unittest discover -v -s tests
+
+To run directly from source (without packaging) :
+
+    python3 -m nicobot.askbot [options...]
+
+To build locally (more at [pypi.org](https://packaging.python.org/tutorials/packaging-projects/)) :
+
+    python3 setup.py sdist bdist_wheel
+
+To upload to test.pypi.org :
+
+    # Defines username and password (or '__token__' and API key) ; alternatively CLI `-u` and `-p` options or user input may be used (or even certificates, see `python3 -m twine upload --help`)
+    TWINE_USERNAME=__token__
+    TWINE_PASSWORD=`pass pypi/test.pypi.org/api_token | head -1`
+    python3 -m twine upload --repository testpypi dist/*
+
+To upload to PROD pypi.org :
+
+    TODO
+
+
+### Automation for PyPi
+
+The above instructions allow to build manually but otherwise it is automatically tested, built and uploaded to pypi.org using _Travis CI_ on each push to GitHub (see [`.travis.yml`](.travis.yml)).
+
+
+
+## Docker build
+
+There are several Dockerfiles, each made for specific use cases (see [README.md](README.md#Docker-usage)).
+They all have [multiple stages](https://docs.docker.com/develop/develop-images/multistage-build/).
+
+`debian.Dockerfile` is quite straight. It builds using *pip* in one stage and copies the resulting *wheels* into the final one.
+
+`debian-signal.Dockerfile` is more complex because it needs to address :
+- including both Python and Java while keeping the image size small
+- compiling native dependencies (both for _signal-cli_ and _qr_)
+- circumventing a number of bugs in multiarch building
+
+`debian-alpine.Dockerfile` produces smaller images but may not be as much portable than debian ones and misses Signal support for now.
+
+Note that the _signal-cli_ backend needs a _Java_ runtime environment, and also _rust_ dependencies to support Signal's group V2. This approximately doubles the size of the images and almost ruins the advantage of alpine over debian...
+
+Those images are limited on each OS (debian+glibc / alpine+musl) to CPU architectures which :
+1. have base images (python, openjdk, rust)
+2. have Python dependencies have _wheels_ or are able to build them
+3. can build libzkgroup (native dependencies for signal)
+4. have the required packages to build
+
+At the time of writing, support is dropped for :
+- `linux/s390x` : lack of _python:3_ image (at least)
+- `linux/riscv64` : lack of _python:3_ image (at least)
+- Signal backend on `linux/arm*` _for Alpine variants_ : lack of JRE binaries
+
+All images have all the bots inside (as they would otherwise only differ by one script from each other).
+The [`docker-entrypoint.sh`](docker/docker-entrypoint.sh) script takes the name of the bot to invoke as its first argument, then its own options and finally the bot's arguments.
+
+Sample _build_ command (single architecture) :
+
+    docker build -t nicolabs/nicobot:debian -f debian.Dockerfile .
+
+Sample _buildx_ command (multi-arch) :
+
+    docker buildx build --platform linux/amd64,linux/arm64,linux/386,linux/arm/v7 -t nicolabs/nicobot:debian -f debian.Dockerfile .
+
+Then run with the provided sample configuration :
+
+    docker run --rm -it -v "$(pwd)/tests:/etc/nicobot" nicolabs/nicobot:debian askbot -c /etc/nicobot/askbot-sample-conf/config.yml
+
+
+### Automation for Docker Hub
+
+_Github actions_ are currently used (see [`.github/workflows/dockerhub.yml`](.github/workflows/dockerhub.yml) to automatically build and push the images to [Docker Hub](https://hub.docker.com/r/nicolabs/nicobot) so they are available whenever commits are pushed to the _master_ branch :
+
+1. A *Github Action* is triggered on each push to [the central repo](https://github.com/nicolabs/nicobot)
+2. All images are built in order using caching (see [.github/workflows/dockerhub.yml](.github/workflows/dockerhub.yml))
+3. Images are uploaded to [Docker Hub](https://hub.docker.com/repository/docker/nicolabs/nicobot)
+
+
+### Docker build process overview
+
+This is the view from the **master** branch on this repository.
+It emphasizes *FROM* and *COPY* relations between the images (base and stages).
+
+![nicobot docker images build process](http://www.plantuml.com/plantuml/proxy?cache=no&src=https%3A%2F%2Fraw.github.com%2Fnicolabs%2Fnicobot%2Fmaster%2Fdocker%2Fdocker-images.puml)
+
+
+### Why no image is available for x arch ?
+
+[The open issues labelled with *docker*](https://github.com/nicolabs/nicobot/labels/docker) should reference the reasons for missing arch / configuration.
+
+
+
+## Versioning
+
+The `--version` command-line option that displays the bots' version relies on _setuptools_scm_, which extracts it from the underlying git metadata.
+This is convenient because the developer does not have to manually update the version (or forget to do it), however it either requires the version to be fixed inside a Python module or the `.git` directory to be present.
+
+There were several options among which the following one has been retained :
+1. Running `setup.py` creates / updates the version inside the `version.py` file
+2. The scripts then load this module at runtime
+
+Since the `version.py` file is not saved into the project, `setup.py` must be run before the version can be queried. In exchange :
+- it does not require _setuptools_ nor _git_ at runtime
+- it frees us from having the `.git` directory around at runtime ; this is especially useful to make the docker images smaller
+
+
+
+## Building signal-cli
+
+The _signal_ backend (actually *signal-cli*) requires a Java runtime, which approximately doubles the image size.
+This led to build separate images (same _repo_ but different _tags_), to allow using smaller images when only the XMPP backend is needed.
+
+
+
+## Resources
+
+### IBM Cloud
+
+- [Language Translator service](https://cloud.ibm.com/catalog/services/language-translator)
+- [Language Translator API documentation](https://cloud.ibm.com/apidocs/language-translator)
+
+### Signal
+
+- [Signal home](https://signal.org/)
+- [signal-cli man page](https://github.com/AsamK/signal-cli/blob/master/man/signal-cli.1.adoc)
+
+### Jabber
+
+- Official XMPP libraries : https://xmpp.org/software/libraries.html
+- OMEMO compatible clients : https://omemo.top/
+- [OMEMO official Python library](https://github.com/omemo/python-omemo) : looks very immature
+- *Gaijim*, a Windows/MacOS/Linux XMPP client with OMEMO support : [gajim.org](https://gajim.org/) | [dev.gajim.org/gajim](https://dev.gajim.org/gajim)
+- *Conversations*, an Android XMPP client with OMEMO support and paid hosting : https://conversations.im
+
+### Python libraries
+
+- [xmpppy](https://github.com/xmpppy/xmpppy) : this library is very easy to use but it does allow easy access to thread or timestamp, and no OMEMO...
+- [github.com/horazont/aioxmpp](https://github.com/horazont/aioxmpp) : officially referenced library from xmpp.org, seems the most complete but misses practical introduction and [does not provide OMEMO OOTB](https://github.com/horazont/aioxmpp/issues/338).
+- [slixmpp](https://lab.louiz.org/poezio/slixmpp) : seems like a cool library too and pretends to require minimal dependencies ; plus it [supports OMEMO](https://lab.louiz.org/poezio/slixmpp-omemo/) so it's the winner. [API doc](https://slixmpp.readthedocs.io/).
+
+### Dockerfile
+
+- [Best practices for writing Dockerfiles](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/)
+- [Docker development best practices](https://docs.docker.com/develop/dev-best-practices/)
+- [DEBIAN_FRONTEND=noninteractive trick](https://serverfault.com/questions/500764/dpkg-reconfigure-unable-to-re-open-stdin-no-file-or-directory)
+- [Dockerfile reference](https://docs.docker.com/engine/reference/builder/#copy)
+
+### JRE + Python in Docker
+
+- [Docker hub - python images](https://hub.docker.com/_/python)
+- [docker-library/openjdk - ubuntu java package has broken cacerts](https://github.com/docker-library/openjdk/issues/19)
+- [Openjdk Dockerfiles @ github](https://github.com/docker-library/openjdk)
+- [phusion/baseimage-docker @ github - not used in the end, because not so portable](https://github.com/phusion/baseimage-docker)
+- [Azul JDK - not used in the end because not better than openjdk](http://docs.azul.com/zulu/zuludocs/ZuluUserGuide/PrepareZuluPlatform/AttachAPTRepositoryUbuntuOrDebianSys.htm)
+- [rappdw/docker-java-python image - not used because only for amd64](https://hub.docker.com/r/rappdw/docker-java-python)
+- [Use OpenJDK builds provided by jdk.java.net?](https://github.com/docker-library/openjdk/issues/212)
+- [How to install tzdata on a ubuntu docker image?](https://serverfault.com/questions/949991/how-to-install-tzdata-on-a-ubuntu-docker-image)
+
+### Multiarch & native dependencies
+
+- [docker.com - Automatic platform ARGs in the global scope](https://docs.docker.com/engine/reference/builder/#automatic-platform-args-in-the-global-scope)
+- [docker/buildx @ github](https://github.com/docker/buildx)
+- [Compiling 'crytography' for Python](https://cryptography.io/en/latest/installation.html#building-cryptography-on-linux)
+- [signal-cli - Providing native lib for libsignal](https://github.com/AsamK/signal-cli/wiki/Provide-native-lib-for-libsignal)
+- [github.com/signalapp/zkgroup - Compiling on raspberry pi fails](https://github.com/signalapp/zkgroup/issues/6)
+- [Multi-Platform Docker Builds (including cargo-specific cross-building)](https://www.docker.com/blog/multi-platform-docker-builds/)
+- [How to build ARMv6 and ARMv7 in the same manifest file. (Compatible tag for ARMv7, ARMv6, ARM64 and AMD64)](https://github.com/KEINOS/Dockerfile_of_Alpine/issues/3)
+- [The "dpkg-split: No such file or directory" bug](https://github.com/docker/buildx/issues/495)
+- [The "Command '('lsb_release', '-a')' returned non-zero exit status 1" bug](https://github.com/docker/buildx/issues/493)
+- [Binfmt / Installing emulators](https://github.com/tonistiigi/binfmt#installing-emulators)
+- [Cross-Compile for Raspberry Pi With Docker](https://itsze.ro/blog/2020/11/29/cross-compile-for-raspberry-pi-with-docker.html)
+
+### Python build & Python in Docker
+
+- [Packaging Python Projects](https://packaging.python.org/tutorials/packaging-projects/)
+- [What Are Python Wheels and Why Should You Care?](https://realpython.com/python-wheels)
+- [Using Alpine can make Python Docker builds 50× slower](https://pythonspeed.com/articles/alpine-docker-python/)
+- [pip install manual](https://pip.pypa.io/en/stable/reference/pip_install/)
+- [pip is showing error 'lsb_release -a' returned non-zero exit status 1](https://stackoverflow.com/questions/44967202/pip-is-showing-error-lsb-release-a-returned-non-zero-exit-status-1)
+
+### Rust
+
+- [Compiling with rust](https://www.rust-lang.org/tools/install)
+- [Packaging a Rust web service using Docker](https://blog.logrocket.com/packaging-a-rust-web-service-using-docker/)
+- [docker/buildx - Value too large for defined data type](https://github.com/docker/buildx/issues/395)
+
+
+
+<!-- MARKDOWN LINKS & IMAGES ; thks to https://github.com/othneildrew/Best-README-Template -->
+<!-- https://www.markdownguide.org/basic-syntax/#reference-style-links -->
+
+[travisci-shield]: https://travis-ci.com/nicolabs/nicobot.svg?branch=master
+[travisci-link]: https://travis-ci.com/nicolabs/nicobot
+[pypi-shield]: https://img.shields.io/pypi/v/nicobot?label=pypi
+[pypi-link]: https://pypi.org/project/nicobot
+[dockerhub-shield]: https://github.com/nicolabs/nicobot/workflows/Build%20and%20publish%20to%20Docker%20Hub/badge.svg
+[dockerhub-link]: https://hub.docker.com/r/nicolabs/nicobot
+[docker-debian-signal-size]: https://img.shields.io/docker/image-size/nicolabs/nicobot/debian-signal.svg?label=debian-signal
+[docker-debian-size]: https://img.shields.io/docker/image-size/nicolabs/nicobot/debian.svg?label=debian
+[docker-alpine-size]: https://img.shields.io/docker/image-size/nicolabs/nicobot/alpine.svg?label=alpine

README.md

@@ -4,12 +4,11 @@ Python package :
 
 [![Build Status on 'master' branch][travisci-shield]][travisci-link] [![PyPi][pypi-shield]][pypi-link]
 
-Docker :
+Docker images :
 
 [![Build and publish to Docker Hub][dockerhub-shield]][dockerhub-link]  
-![Docker debian-slim][docker-debian-slim-size] ![layers][docker-debian-slim-layers]  
+![Docker debian][docker-debian-size] ![Docker debian-signal][docker-debian-signal-size] ![Docker alpine][docker-alpine-size]  
-![Docker debian][docker-debian-size] ![layers][docker-debian-layers]  
+
-![Docker alpine][docker-alpine-size] ![layers][docker-alpine-layers]  
 
 
 ## About
@@ -27,6 +26,10 @@ This project features :
 - Participating in [XMPP / Jabber](https://xmpp.org) conversations
 - Using [IBM Watson™ Language Translator](https://cloud.ibm.com/apidocs/language-translator) cloud API
 
+This document is about how to **use** the bots.
+To get more details on how to build / develop with this project, see [Develop.md](Develop.md).
+
+
 
 ## Requirements & installation
 
@@ -36,22 +39,25 @@ The bots can be installed and run at your choice from :
 - the source code
 - the Docker images
 
+
+
 ### Python package installation
 
 A classic (Python package) installation requires :
 
 - Python 3 (>= 3.5) and pip ([should be bundled with Python](https://pip.pypa.io/en/stable/installing)) ; e.g. on Debian : `sudo apt install python3 python3-pip`
-- [signal-cli](https://github.com/AsamK/signal-cli) for the *Signal* backend (see [Using the Signal backend] below for requirements)
+- [signal-cli](https://github.com/AsamK/signal-cli) for the *Signal* backend (see [Using the Signal backend](#using-the-signal-backend) below for requirements)
 - For *transbot* : an IBM Cloud account ([free account ok](https://www.ibm.com/cloud/free))
 
 To install,  simply do :
 
     pip3 install nicobot
 
-Then, you can run the bots by their name, thanks to the provided commands :
+Then, you can run the bots by their name, thanks to the installed shortcuts :
 
     # Runs the 'transbot' bot
     transbot [options...]
+
     # Runs the 'askbot' bot
     askbot [options...]
 
@@ -59,16 +65,21 @@ Then, you can run the bots by their name, thanks to the provided commands :
 
 ### Installation from source
 
-To install from source you need to fulfill the same requirements as for a package installation (see above), then download the code and build it :
+To install from source you need to fulfill the requirements for a package installation (see above), then download the code and build it :
 
     git clone https://github.com/nicolabs/nicobot.git
     cd nicobot
     pip3 install -r requirements-runtime.txt
 
+> **NOTE**
+> Depending on your platform, `pip install` may trigger a compilation for some or all of the dependencies (i.e. when *Python wheels* are not available).
+> In this case you may need to install more requirements for the build to succeed : looking at [the Dockerfiles in this project](Develop.md) might help you gather the exact list.
+
 Now you can run the bots by their name as if they were installed via the package :
 
     # Runs the 'transbot' bot
     transbot [options...]
+
     # Runs the 'askbot' bot
     askbot [options...]
 
@@ -76,81 +87,47 @@ Now you can run the bots by their name as if they were installed via the package
 
 ### Docker usage
 
-At the present time there are [several Docker images available](https://hub.docker.com/repository/docker/nicolabs/nicobot), with the following tags :
+At the present time there are [several Docker images available](https://hub.docker.com/r/nicolabs/nicobot), with the following tags :
 
-- **debian** : if you have several images with the _debian_ base, this may be the most space-efficient (as base layers will be shared with other images)
+- **debian** : this is the most portable image ; in order to keep it relatively small it does not include support for *Signal* messaging (will throw an error if you try --> use XMPP instead)
-- **debian-slim** : if you want a smaller-sized image and you don't run other images based on the  _debian_ image (as it will not share as much layers as with the above `debian` tag)
+- **debian-signal** : this is the most complete image ; it is also the largest one, but includes everything needed to use the *Signal* backend
-- **alpine** : this should be the smallest image in theory, but it's more complex to maintain and thereore might not meet this expectation ; please check/test before use
+- **alpine** : this should be the smallest image, but it's more complex to maintain and therefore might not always meet this expectation. Also, due to the lack/complexity of Alpine support for some Python, Java & native dependencies, an image may not exist for all platforms and it currently doesn't support the Signal backend (you can use XMPP instead).
 
-> The current state of those images is such that I suggest you try the **debian-slim** image first and switch to another one if you encounter issues or have a specific use case to solve.
+Please have a look at the status pills at the top of this document to get more details like status and size.
+
+> **ADVICE**
+> The current state of those images is such that I suggest you try the **alpine** image first and switch to a **debian\*** one if you need Signal or encounter runtime issues.
 
 The container is invoked this way :
 
-    docker ... [--signal-register] <bot name> <bot arguments>
+    docker ... [--signal-register <device name>] [--qrcode-options <qr options] <bot name> [<bot arguments>]
 
-- `--signal-register` will display a QR code in the console : scan it with the Signal app on the device to link the bot with (it will simply do the *link* command inside the container ; read more about this later in this document). If this option is not given and the _signal_ backend is used, it will use the `.local/share/signal-cli` directory from the container or fail.
+- `--signal-register` will display a QR code in the console : scan it with the Signal app on the device to link the bot with (it will simply do the *link* command inside the container ; read more about this later in this document). If this option is not given and the _signal_ backend is used, it will use the `.local/share/signal-cli` directory from the container (you _have_ to mount it) or fail. This option takes a custom device name as its argument.
+- `--qrcode-options` has an argument, which is a string of options to pass to the QR code generation command (see [python-qrcode](https://github.com/lincolnloop/python-qrcode)).
 - `<bot name>` is either `transbot` or `askbot`
 - `<bot arguments>` is the list of arguments to pass to the bot
 
+If any doubt, just invoke the image without argument to print the inline help statement.
+
 Sample command to start a container :
 
-    docker run --rm -it -v "myconfdir:/etc/nicobot" nicolabs/nicobot:debian-slim transbot -C /etc/nicobot
+    docker run --rm -it -v "$(pwd)/myconfdir:/etc/nicobot" nicolabs/nicobot transbot -C /etc/nicobot
 
-In this example `myconfdir` is a local directory with configuration files for the bot (`-C` option), but you could set all arguments on the command line if you don't want to deal with files.
+In this example `myconfdir` is a local directory with configuration files for the bot (`-C` option), but you could also set most parameters on the command line.
 
 You can also use _docker volumes_ to persist _signal_ and _IBM Cloud_ credentials and configuration :
 
-    docker run --rm -it -v "myconfdir:/etc/nicobot" -v "$HOME/.local/share/signal-cli:/root/.local/share/signal-cli" nicolabs/nicobot:debian-slim transbot -C /etc/nicobot
+    docker run --rm -it -v "$(pwd)/myconfdir:/etc/nicobot" -v "$HOME/.local/share/signal-cli:/root/.local/share/signal-cli" nicolabs/nicobot transbot -C /etc/nicobot
 
 All options that can be passed to the bots' command line can also be passed to the docker command line.
 
 
 
-## Transbot instructions
+## How to use the bots
-
-*Transbot* is a demo chatbot interface to IBM Watson™ Language Translator service.
-
-**Again, this is NOT STABLE code, there is absolutely no warranty it will work or not harm butterflies on the other side of the world... Use it at your own risk !**
-
-It detects configured patterns or keywords in messages (either received directly or from a group chat) and answers with a translation of the given text.
-
-The sample configuration in `tests/transbot-sample-conf`, demoes how to make the bot answer messages given in the form `nicobot <text_to_translate> in <language>` (or simply `nicobot  <text_to_translate>`, into the current language) with a translation of _<text_to_translate>_.
-
-Transbot can also pick a random language to translate into ; the sample configuration file shows how to make it translate messages containing "Hello" or "Goodbye" into a random language.
-
-### Quick start
-
-1. Install **nicobot** (see above)
-2. [Create a *Language Translator* service instance on IBM Cloud](https://cloud.ibm.com/catalog/services/language-translator) and [get the URL and API key from your console](https://cloud.ibm.com/resources?groups=resource-instance)
-3. Fill them into `tests/transbot-sample-conf/config.yml` (`ibmcloud_url` and `ibmcloud_apikey`)
-4. Run `transbot -C tests/transbot-sample-conf` (with docker it will be something like `docker run -it "tests/transbot-sample-conf:/etc/nicobot" nicolabs/nicobot:debian-slim transbot -C /etc/nicobot`)
-5. Type `Hello world` in the console : the bot will print a random translation of "Hello World"
-6. Type `Bye nicobot` : the bot will terminate
-
-You may now explore the dedicated chapters below for more options, including sending & receiving messages through *XMPP* or *Signal* instead of keyboard & console.
 
 
 
-### Main configuration options and files
+### Askbot usage
-
-This paragraph introduces the most important options to make this bot work. Please also check the generic options below, and finally run `transbot -h` to get an exact list of all options.
-
-The bot needs several configuration files that will be generated / downloaded the first time if not provided :
-
-- **--keyword** and **--keywords-file** will help you generate the list of keywords that will trigger the bot. To do this, run `transbot --keyword <a_keyword> --keyword <another_keyword> ...` **a first time** : this will download all known translations for these keywords and save them into a `keywords.json` file. Next time you run the bot, **don't** use the `--keyword` option : it will reuse this saved keywords list. You can use `--keywords-file` to change the file name.
-- **--languages-file** : The first time the bot runs it will download the list of supported languages into `languages.<locale>.json` and reuse it afterwards. You can edit it, to keep just the set of languages you want for instance. You can also use the `--locale` option to indicate the desired locale.
-- **--locale** will select the locale to use for default translations (with no target language specified) and as the default parsing language for keywords.
-- **--ibmcloud-url** and **--ibmcloud-apikey** take arguments you can obtain from your IBM Cloud account ([create a Language Translator instance](https://cloud.ibm.com/apidocs/language-translator) then go to [the resource list](https://cloud.ibm.com/resources?groups=resource-instance))
-
-The **i18n.\<locale>.yml** file contains localization strings for your locale :
-- *Transbot* will say "Hello" when started and "Goodbye" before shutting down : you can configure those banners in this file.
-- It also defines the pattern that terminates the bot.
-
-A sample configuration is available in the `tests/transbot-sample-conf/` directory.
-
-
-
-## Askbot instructions
 
 *Askbot* is a one-shot chatbot that will send a message and wait for an answer.
 
@@ -160,18 +137,22 @@ When run, it will send a message and wait for an answer, in different ways (see
 Once the configured conditions are met, the bot will terminate and print the result in [JSON](https://www.json.org/) format.
 This JSON structure will have to be parsed in order to retrieve the answer and determine what were the exit(s) condition(s).
 
-### Main configuration options
+
+
+#### Main configuration options
 
 Run `askbot -h` to get a description of all options.
 
 Below are the most important configuration options for this bot (please also check the generic options below) :
 
 - **--max-count <integer>** will define how many messages to read at maximum before exiting. This allows the recipient to split the answer in several messages for instance. However currently all messages are returned by the bot at once at the end, so they cannot be parsed on the fly by an external program. To give _x_ tries to the recipient, run _x_ times this bot instead.
-- **--pattern <name> <pattern>** defines a pattern that will end the bot when matched. This is the way to detect an answer. It takes 2 arguments : a symbolic name and a [regular expression pattern](https://docs.python.org/3/howto/regex.html#regex-howto) that will be tested against each message. It can be passed several times in the same command line, hence the `<name>` argument, which will allow identifying which pattern(s) matched.
+- **--pattern <name> <pattern>** defines a pattern that will end the bot when matched. This is the way to detect an answer. It takes 2 arguments : a symbolic name and a [regular expression pattern](https://docs.python.org/3/howto/regex.html#regex-howto) that will be tested against each message. You can define multiple patterns in the same command line, hence the `<name>` argument, which will allow identifying which pattern(s) matched.
 
 Sample configuration can be found in `tests/askbot-sample-conf`.
 
-### Example
+
+
+#### Example
 
     askbot -m "Do you like me ?" -p yes '(?i)\b(yes|ok)\b' -p no '(?i)\bno\b' -p cancel '(?i)\b(cancel|abort)\b' --max-count 3 -b signal -U '+33123456789' --recipient '+34987654321'
 
@@ -181,10 +162,10 @@ The previous command will :
 2. Wait for a maximum of 3 messages in answer and return
 3. Or return immediately if a message matches one of the given patterns labeled 'yes', 'no' or 'cancel'
 
-If the user *+34987654321* was to reply :
+If the user *+34987654321* replies with 2 messages :
 
- > I don't know    
+ 1. I don't know
- > Ok then : NO !
+ 2. Ok then : NO !
 
  Then the output would be :
 
@@ -228,27 +209,80 @@ If the user *+34987654321* was to reply :
 A few notes about the _regex_ usage in this example : in `-p yes '(?i)\b(yes|ok)\b'` :
 - `(?i)` enables case-insensitive match
 - `\b` means "edge of a word" ; it is used to make sure the wanted text will not be part of another word (e.g. `tik tok` would match `ok` otherwise)
-- Note that a _search_ is done on the messages (not a _match_) so it is not required to specify a full _regular expression_ with `^` and `$` (though you may do, if you want to). This makes the pattern more readable.
+- Note that a regex _search_ is done on the messages (not a _match_) so it is not required to specify a full _regular expression_ with `^` and `$` (though you may do, if you want to). This makes the pattern more readable.
 - The pattern is labeled 'yes' so it can be easily identified in the JSON output and counted as a positive match
 
-You may also have noticed the importance of defining patterns that don't overlap (here the message matched both 'yes' and 'no') or being ready to handle unknow states.
+You may also have noticed the importance of defining patterns that don't overlap (here the message matched both 'yes' and 'no') or being ready to handle unknown states.
 
 To make use of the bot, you could parse its output with a script, or with a command-line client like [jq](https://stedolan.github.io/jq/).
 
-This sample Python snippet will get the name of the matched patterns :
+Here's an example snippet for a _Python_ program to extract the name of the matched patterns :
 
 ```python
 # loads the JSON output
 output = json.loads('{ "max_responses": false, "messages": [...] }')
-# 'matched' is the list of the names of the patterns that matched against the last message, e.g. `['yes','no']`
+# 'matched' is the list of the names of the patterns that matched against the last message
 matched = [ p['name'] for p in output['messages'][-1]['patterns'] if p['matched'] ]
+# e.g. matched = `['yes','no']`
 ```
 
 
-## Generic instructions
+### Transbot usage
+
+*Transbot* is a demo chatbot interface to IBM Watson™ Language Translator service.
+
+**Again, this is NOT STABLE code, there is absolutely no warranty it will work or not harm butterflies on the other side of the world... Use it at your own risk !**
+
+It is triggered by messages :
+- either matching the configured pattern
+- or containing a keyword from a given list
+
+When triggered, it will answer with a translation of the given text.
+
+It will reply either to direct messages or to a group chat, depending on the given parameters.
+
+The sample configuration in `tests/transbot-sample-conf`, demoes how to make the bot answer messages given in the form `nicobot <text_to_translate> in <language>` (or simply `nicobot  <text_to_translate>`, into the current language) with a translation of _<text_to_translate>_.
+
+Transbot can also pick a random language to translate into ; the sample configuration file shows how to make it translate messages containing "Hello" or "Goodbye" into a random language.
 
 
-### Common options
+### Quick start
+
+1. Install **nicobot** (see above)
+2. [Create a *Language Translator* service instance on IBM Cloud](https://cloud.ibm.com/catalog/services/language-translator) and [get the URL and API key from your console](https://cloud.ibm.com/resources?groups=resource-instance)
+3. Make a local copy of files in [`tests/transbot-sample-conf/`](tests/transbot-sample-conf/) and fill the `ibmcloud_url` and `ibmcloud_apikey` values into `config.yml`
+4. Run `transbot -C ./transbot-sample-conf` (with docker it will be something like `docker run -it "$(pwd)/transbot-sample-conf:/etc/nicobot" nicolabs/nicobot transbot -C /etc/nicobot`)
+5. Type `Hello world` in the console : the bot will print a random translation of "Hello World"
+6. Type `Bye nicobot` : the bot will terminate
+
+You may now explore the dedicated chapters below for more options, including sending & receiving messages through *XMPP* or *Signal* instead of keyboard & console.
+
+
+
+#### Main configuration options and files
+
+This paragraph introduces the most important parameters to make this bot work. Please also check the generic options below ; finally run `transbot -h` to get an exact list of all options.
+
+The bot needs several configuration files that will be generated / downloaded the first time if not provided :
+
+- **--keyword** and **--keywords-file** will help you generate a list of translations for the given keywords so they will trigger the bot even if written in other languages. To do it, run this **a first time** : `transbot --keyword <a_keyword> --keyword <another_keyword> ...` to download all known translations for these keywords and save them into a `keywords.json` file. Next time you run the bot, **don't** use the `--keyword` option : it will reuse this saved keywords list. You can use `--keywords-file` to change the file name.
+- **--languages-file** : The first time the bot runs it will download the list of supported languages (to translate into) into `languages.<locale>.json` and reuse it afterwards. You can edit it, to keep just the set of languages you want for instance. You can also use the `--locale` option to indicate the desired locale.
+- **--locale** will select the locale to use for default translations (with no target language specified) and as the default parsing language for keywords.
+- **--ibmcloud-url** and **--ibmcloud-apikey** take arguments you can obtain from your IBM Cloud account ([create a Language Translator instance](https://cloud.ibm.com/apidocs/language-translator) then go to [the resource list](https://cloud.ibm.com/resources?groups=resource-instance))
+
+The patterns and custom texts the bot speaks & recognizes can be defined in the **i18n.\<locale>.yml** file :
+- *Transbot* will say "Hello" when started and "Goodbye" before shutting down : you can configure those banners in this file.
+- It also defines the pattern that terminates the bot.
+
+A sample configuration is available in the `tests/transbot-sample-conf/` directory.
+
+
+
+### Generic instructions
+
+
+
+#### Common options
 
 The following options are common to both bots :
 
@@ -257,7 +291,8 @@ The following options are common to both bots :
 - **--stealth** will make the bot connect and listen to messages but print answers to the console instead of sending it ; useful to observe the bot's behavior in a real chatroom...
 
 
-### Configuration file : config.yml
+
+#### Configuration file : config.yml
 
 Options can also be taken from a configuration file.
 By default it reads the `config.yml` file in the current directory but can be changed with the `--config-file` and `--config-dir` options.
@@ -276,16 +311,18 @@ If unsure,  please first review [YAML syntax](https://yaml.org/spec/1.1/#id85716
 
 
 
-### Using the Jabber/XMPP backend
+#### Using the Jabber/XMPP backend
 
-By using `--backend jabber` you can make the bot chat with XMPP (a.k.a. Jabber) users.
+By specifying `--backend jabber` you can make the bot chat with XMPP (a.k.a. Jabber) users.
 
-#### Jabber-specific options
 
-- `--jabber-username` and `--jabber-password` are the JabberID (e.g. *myusername@myserver.im*) and password of the bot's account used to send and read messages. If `--jabber-username` missing, `--username` will be used.
+##### Jabber-specific options
+
+- `--jabber-username` and `--jabber-password` are the JabberID (e.g. *myusername@myserver.im*) and password of the bot's account used to send and read messages. If `--jabber-username` is missing, `--username` will be used.
 - `--jabber-recipient` is the JabberID of the person to send the message to. If missing, `--recipient` will be used.
 
-#### Example
+
+##### Example
 
     transbot -C tests/transbot-sample-conf -b jabber -U mybot@myserver.im -r me@myserver.im`
 
@@ -297,23 +334,25 @@ With :
 
 
 
-### Using the Signal backend
+#### Using the Signal backend
 
-By using `--backend signal` you can make the bot chat with Signal users.
+By specifying `--backend signal` you can make the bot chat with Signal users.
 
-#### Prerequistes
 
-For package or source installations, you must first [install and configure *signal-cli*](https://github.com/AsamK/signal-cli#installation).
+##### Prerequistes
+
+For package and source installations, you must first [install and configure *signal-cli*](https://github.com/AsamK/signal-cli#installation).
 
 For all installations, you must [*register* or *link*](https://github.com/AsamK/signal-cli/blob/master/man/signal-cli.1.adoc) the computer where the bot will run ; e.g. :
 
     signal-cli link --name MyComputer
 
-With docker images you can do this registration by using the `--signal-register` option. This will save the registration files into `/root/.local/share/signal-cli/` inside the container. If this location links to a persisted volume, it will be reused on each launch.
+With docker images you can do this registration by using the `--signal-register` option. This will save the registration files into `/root/.local/share/signal-cli/` inside the container. If this location is bound to a persistent volume, it can be reused on next launch.
 
-Please see [signal-cli's man page](https://github.com/AsamK/signal-cli/blob/master/man/signal-cli.1.adoc) for more details on the registration process.
+Please see [signal-cli's man page](https://github.com/AsamK/signal-cli/blob/master/man/signal-cli.1.adoc) for more details about the registration process.
 
-#### Signal-specific options
+
+##### Signal-specific options
 
 - `--signal-username` selects the account to use to send and read message : it is a phone number in international format (e.g. `+33123456789`). In `config.yml`, make sure to put quotes around it to prevent YAML thinking it's an integer (because of the 'plus' sign). If missing, `--username` will be used.
 - `--signal-recipient` and `--signal-group` select the recipient (only one of them should be given). Make sure `--signal-recipient` is in international phone number format and `--signal-group` is a base 64 group ID (e.g. `--signal-group "mABCDNVoEFGz0YeZM1234Q=="`). If `--signal-recipient` is missing, `--recipient` will be used. To get the IDs of the groups you are in, run : `signal-cli -U +336123456789 listGroups`
@@ -324,112 +363,15 @@ Example :
 
 
 
-## Development
+## External resources
 
-Install Python dependencies (both for building and running) with :
+- [IBM Watson Language Translator service](https://cloud.ibm.com/catalog/services/language-translator)
-
+- Signal messaging : https://signal.org
-    pip3 install -r requirements-build.txt -r requirements-runtime.txt
+- XMPP resources : https://xmpp.org/software/libraries.html
-
-To run unit tests :
-
-    python3 -m unittest discover -v -s tests
-
-To run directly from source (without packaging) :
-
-    python3 -m nicobot.askbot [options...]
-
-To build locally (more at [pypi.org](https://packaging.python.org/tutorials/packaging-projects/)) :
-
-    python3 setup.py sdist bdist_wheel
-
-To upload to test.pypi.org :
-
-    # Defines username and password (or '__token__' and API key) ; alternatively CLI `-u` and `-p` options or user input may be used (or even certificates, see `python3 -m twine upload --help`)
-    TWINE_USERNAME=__token__
-    TWINE_PASSWORD=`pass pypi/test.pypi.org/api_token | head -1`
-    python3 -m twine upload --repository testpypi dist/*
-
-To upload to PROD pypi.org :
-
-    TODO
-
-Otherwise, it is automatically tested, built and uploaded to pypi.org using _Travis CI_ on each push to GitHub.
-
-
-### Docker build
-
-There are several Dockerfiles, each made for specific use cases (see [Docker-usage](#Docker-usage) above) :
-
-`Dockerfile-debian` and `Dockerfile-debian-slim` are quite straight and very similar. They still require multi-stage build to address enough platforms.
-
-`Dockerfile-alpine` requires a [multi-stage build](https://docs.docker.com/develop/develop-images/multistage-build/) anyway because most of the Python dependencies need to be compiled first.
-The result however should be a far smaller image than with a Debian base.
-
-> Note that the _signal-cli_ backend needs a _Java_ runtime environment, and also _rust_ dependencies to support Signal's group V2. This currently doubles the size of the images and ruins the advantage of alpine over debian...
-
-Those images are limited to CPU architectures :
-- supported by [the base images](https://hub.docker.com/_/python)
-- for which the Python dependencies are built or able to build
-- for which the native dependencies of signal (libzkgroup) can be built (alpine only)
-
-Simple _build_ command (single architecture) :
-
-    docker build -t nicolabs/nicobot:debian-slim -f Dockerfile-debian-slim .
-
-Sample _buildx_ command (multi-arch) :
-
-    docker buildx build --platform linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x -t nicolabs/nicobot:debian-slim -f Dockerfile-debian-slim .
-
-Then run with the provided sample configuration :
-
-    docker run --rm -it -v "$(pwd)/tests:/etc/nicobot" nicolabs/nicobot:debian-slim askbot -c /etc/nicobot/askbot-sample-conf/config.yml
-
-_Github actions_ are currently used (see [`dockerhub.yml`](.github/workflows/dockerhub.yml) to automatically build and push the images to Docker Hub so they are available whenever commits are pushed to the _master_ branch.
-
-The images have all the bots inside, as they only differ by one script from each other.
-The [`docker-entrypoint.sh`](docker/docker-entrypoint.sh) script takes the name of the bot to invoke as its first argument, then its own options and finally the bot's arguments.
-
-
-### Versioning
-
-The command-line option to display the scripts' version relies on _setuptools_scm_, which extracts it from the underlying git metadata.
-This is convenient because the developer does not have to manually update the version (or forget to do it prior a release), however it either requires the version to be fixed inside a package or the `.git` directory to be present.
-
-There were several options among which the following one has been retained :
-- Running `setup.py` creates / updates the version inside the `version.py` file
-- The scripts then load this module at runtime
-
-The remaining requirement is that `setup.py` must be run before the version can be extracted. In exchange :
-- it does not require _setuptools_ nor _git_ at runtime
-- it frees us from having the `.git` directory around at runtime ; this is especially useful to make the docker images smaller
-
-
-
-## Resources
-
-### IBM Cloud
-
-- [Language Translator service](https://cloud.ibm.com/catalog/services/language-translator)
-- [Language Translator API documentation](https://cloud.ibm.com/apidocs/language-translator)
-
-### Signal
-
-- [Signal home](https://signal.org/)
-- [signal-cli man page](https://github.com/AsamK/signal-cli/blob/master/man/signal-cli.1.adoc)
-
-### Jabber
-
-- Official XMPP libraries : https://xmpp.org/software/libraries.html
 - OMEMO compatible clients : https://omemo.top/
-- [OMEMO official Python library](https://github.com/omemo/python-omemo) : looks very immature
 - *Gaijim*, a Windows/MacOS/Linux XMPP client with OMEMO support : [gajim.org](https://gajim.org/) | [dev.gajim.org/gajim](https://dev.gajim.org/gajim)
 - *Conversations*, an Android XMPP client with OMEMO support and paid hosting : https://conversations.im
 
-Python libraries :
-
-- [xmpppy](https://github.com/xmpppy/xmpppy) : this library is very easy to use but it does allow easy access to thread or timestamp, and no OMEMO...
-- [github.com/horazont/aioxmpp](https://github.com/horazont/aioxmpp) : officially referenced library from xmpp.org, seems the most complete but misses practical introduction and [does not provide OMEMO OOTB](https://github.com/horazont/aioxmpp/issues/338).
-- [slixmpp](https://lab.louiz.org/poezio/slixmpp) : seems like a cool library too and pretends to require minimal dependencies ; plus it [supports OMEMO](https://lab.louiz.org/poezio/slixmpp-omemo/) so it's the winner. [API doc](https://slixmpp.readthedocs.io/).
 
 
 <!-- MARKDOWN LINKS & IMAGES ; thks to https://github.com/othneildrew/Best-README-Template -->
@@ -441,9 +383,6 @@ Python libraries :
 [pypi-link]: https://pypi.org/project/nicobot
 [dockerhub-shield]: https://github.com/nicolabs/nicobot/workflows/Build%20and%20publish%20to%20Docker%20Hub/badge.svg
 [dockerhub-link]: https://hub.docker.com/r/nicolabs/nicobot
-[docker-debian-slim-size]: https://img.shields.io/docker/image-size/nicolabs/nicobot/debian-slim.svg?label=debian-slim
+[docker-debian-signal-size]: https://img.shields.io/docker/image-size/nicolabs/nicobot/debian-signal.svg?label=debian-signal
-[docker-debian-slim-layers]: https://img.shields.io/microbadger/layers/nicolabs/nicobot/debian-slim.svg
 [docker-debian-size]: https://img.shields.io/docker/image-size/nicolabs/nicobot/debian.svg?label=debian
-[docker-debian-layers]: https://img.shields.io/microbadger/layers/nicolabs/nicobot/debian.svg
 [docker-alpine-size]: https://img.shields.io/docker/image-size/nicolabs/nicobot/alpine.svg?label=alpine
-[docker-alpine-layers]: https://img.shields.io/microbadger/layers/nicolabs/nicobot/alpine.svg

docker/docker-entrypoint.sh

@@ -2,22 +2,26 @@
 
 usage() {
 cat << EOF
-Usage : $0 <bot's name> [--signal-register <device name>]
+Usage : docker run [...] nicolabs/nicobot:<tag> <bot's name>
-                        [--qrcode-options <qr options>]
+              [--signal-register <device name>]
-                        [bot's regular arguments]
+              [--qrcode-options <qr options>]
+              [bot's regular arguments]
 
 Arguments :
 
 <bot's name>                      One of 'askbot' or 'transbot'.
+
 --signal-register <device name>   Will display a QR Code to scan & register with
                                   an existing Signal account. <device name> is a
                                   string to identify the docker container as a
                                   signal device.
+
 --qrcode-options <qr options>     Additional options (in one string) to the 'qr'
                                   command. The QR Code can be printed directly
                                   to the console without using this argument but
                                   make sure to pass '-it' to 'docker run'.
                                   See github.com/lincolnloop/python-qrcode.
+
 [bot's regular arguments]         All arguments that can be passed to the bot.
                                   See github.com/nicolabs/nicobot.
 

nicobot/signalcli.py

@@ -135,6 +135,8 @@ class SignalChatter(Chatter):
             logging.debug("Filtering message : %s" % repr(event))
             envelope = event['envelope']
             if envelope['timestamp'] > self.startTime:
+                # TODO This test prevents sending and receiving with the same number
+                # See https://github.com/nicolabs/nicobot/issues/34
                 if envelope['dataMessage']:
                     dataMessage = envelope['dataMessage']
                     if dataMessage['message']:

nicobot/transbot.py

@@ -585,7 +585,7 @@ def run( args=sys.argv[1:] ):
         description="A bot that reacts to messages with given keywords by responding with a random translation"
         )
     # Core arguments for this bot
-    parser.add_argument("--keyword", "-k", dest="keywords", action="append", help="Keyword bot should react to (will write them into the file specified with --keywords-file)")
+    parser.add_argument("--keyword", "-k", dest="keywords", action="append", help="A keyword a bot should react to (will write them into the file specified with --keywords-file)")
     parser.add_argument("--keywords-file", dest="keywords_files", action="append", help="File to load from and write keywords to")
     parser.add_argument('--locale', '-l', dest='locale', default=config.locale, help="Change default locale (e.g. 'fr_FR')")
     parser.add_argument("--languages-file", dest="languages_file", help="File to load from and write languages to")