Installing Zoe

If you are looking for the five-minutes install procedure, just for testing, check the test install section, below.

When installing Zoe for production you should, first of all, look at the following requirements and take a decision about each of them:

  • Container back-end: Kubernetes or DockerEngine
  • Shared filesystem: we have deployments on NFS and CephFS, but anything similar should work
  • Network: how your users will connect to the containers
  • Authentication back-end: how your users will authenticate to Zoe (LDAP or text file)
  • How to manage Zoe Applications (ZApps)
  • ZApp output logs: see Zoe logs and service output

After, you can start the installation, as outlined in the manual install section.

Choosing the container back-end

At this time Zoe supports three back-ends:

  • DockerEngine: uses one or more Docker Engines. It is simple to install and to scale.
  • Kubernetes: the most complex to setup, we suggest using it only if you already have (or need) a Kubernetes setup for running other software.

DockerEngine

The DockerEngine back-end uses one or more nodes with Docker Engine installed and configured to listen to network requests.

This sample config file, usually found in /etc/docker/daemon.conf may help to get you started:

{
   "dns": ["192.168.46.1"],
   "dns-search": ["zoe.example.com"],
   "tlsverify": true,
   "tlscacert": "/mnt/certs/cert-authority/ca.pem"
   "tlscert": "/mnt/certs/cert.pem",
   "tlskey": "/mnt/certs/key.pem",
   "hosts": ["tcp://worker1.zoe.example.com:2375", "unix:///var/run/docker.sock"]
 }

Once you have your docker hosts up and running, to tell the back-end which nodes are available and how to connect to them, you need to create a file with this format:

[DEFAULT]
use_tls: no
tls_cert: /mnt/certs/client/cert.pem
tls_key: /mnt/certs/client/key.pem
tls_ca: /mnt/certs/cert-authority/ca.pem

[foo]
address: localhost:2375
external_address: 192.168.45.42

[bar]
docker_address: 192.168.47.5:2375
external_address: 192.168.47.5
use_tls: yes

[baz]
docker_address: 192.168.47.50:2375
external_address: 192.168.47.50
use_tls: yes
labels: gpu,ssd

This sample configuration describes three hosts. The DEFAULT section contains items that are common to all hosts, however they can be overwritten in the host definition.

Host foo does not use TLS (from the default config item), Zoe needs to connect to localhost on port 2375 to talk to it and users connecting to containers running on this host need to use the 192.168.45.42 address to connect. This external_address will be used by Zoe to generate links in the web interface.

Host bar uses TLS and host baz has also two labels that can be matched when starting services with the corresponding label. Labels are comma separated.

You tell Zoe the location of this file using the backend-docker-config-file option in zoe.conf.

The special label disabled can be set to tell Zoe to stop scheduling services on that host. In time executions will terminate and the host will be left empty and can be removed from the cluster for maintenance.

Kubernetes

See Kubernetes backend for Zoe for configuration details.

Shared filesystem

Users need to put data and binaries in a place accessible by Zoe and need to be able to access the results and the logs generated by running ZApp.

Zoe uses the concept of workspaces: each user has a private directory that is attached to all the containers of each ZApp belonging to her in a well-known location. This filesystem can be accessed by a special gateway container spawned by the administrator (see gateway containers) or by other methods (direct mount on user machines, webdav, web file managers).

Zoe implements a “directory” back-end for workspaces. Container back-ends may implement more volume technologies: Zoe is not involved, it needs only the information on how to attach the user volume to the container, so the effort required to support new volume types should be minimal.

At Eurecom we use CephFS, but we know of successful Zoe deployments based on NFS.

Networking

Containers spawned by Zoe need to be able to talk to each other on the network like they where on the same broadcast domain, even when they run on different hosts. Network configuration is back-end dependent: both Kubernetes and Docker provide their own systems to manage the virtual network between containers.

Docker provides a feature called multi host networking. An alternative that we found more efficient and simple to setup and maintain is Flannel.

Most of the ZApps expose a number of interfaces (web, REST and others) to the user. Zoe configures the active back-end to expose these ports, but does not perform any additional action to configure routing or DNS to make the ports accessible. Keeping in mind that the back-end network configuration is outside Zoe’s competence area, here there is non-exhaustive list of the possible configurations:

  • expose the hosts running the containers by using public IP addresses
  • use a proxy, like the one developed for Zoe: Accessing Zoe through a reverse proxy
  • use back-end network plugins to build custom topologies

Authentication back-ends

Zoe supports multiple user authentication back-ends. Multiple back-ends can coexist at the same time.

Check the Users page for more details on the user model.

Remember to disable or change the password of the default admin user.

LDAP

Plain LDAP or LDAP+SASL GSSAPI are available.

In Zoe configuration you need to specify the following options:

  • ldap-server-uri
  • ldap-bind-user
  • ldap-bind-password
  • ldap-base-dn
  • ldap-admin-gid
  • ldap-user-gid
  • ldap-guest-gid
  • ldap-group-name

Text file

For testing and for simple deployments with a few users, a CSV text file can be used.

Its format is:

<username>,<password>,<role>

The file location can be specified in the zoe.conf file and it needs to be readable only be the Zoe processes.

Managing Zoe applications

ZApps are composed of a container image and a JSON description. The container image can be stored on the Docker nodes, in a local private registry or in the public Docker Hub (or any other public registry).

Zoe does not provide a way to automatically build images, push them to a local registry, or pull them to the hosts when needed. At Eurecom we provide an automated environment based on GitLab’s CI features: users are able to customize their applications (JSON and Dockerfiles) by working on git repositories. Images are rebuilt and pushed on commit and JSON files are generated and copied to the ZApp shop directory. You can check out a few examples here: https://gitlab.eurecom.fr/zoe-apps

The ZApp Shop

The Zoe web interface provides a ZApp shop to showcase available ZApps and have a friendly and easy way for users to list and access ZApps.

The shop is managed locally. It looks for ZApps in a configured directory (option zapp-shop-path). Each ZApp must live in its own directory, that must contain:

  • manifest.json : a JSON file that describes the contents of the ZApp
  • a logo that is displayed on the web interface
  • one or more text files in markdown format with ZApp information and documentation
  • one or more JSON Zoe application descriptions

The manifest.json file gather all this information together for the ZApp Shop interface. Its format is as follows:

{
    "version": 1,
    "zapps": [
        {
            "category": "TensorFlow",
            "name": "Google TensorFlow notebook",
            "description": "tf-google.json",
            "readable_descr": "README-goog.md",
            "parameters": []
        },
        {
            "category": "TensorFlow",
            "name": "Google TensorFlow batch",
            "description": "tf-google.json",
            "readable_descr": "README-batch.md",
            "parameters": [
                {
                    "kind": "command",
                    "name": "tf-jupyter",
                    "readable_name": "Command",
                    "description": "The Python script to run, relative to the workspace directory",
                    "type": "string",
                    "default": "./my-tf-app/main.py"
                }
            ],
            "disabled_for": ["role_A"]
        }
    ]
}
  • version : a internal version, used by Zoe to recognize the manifest format. For now only 1 is supported.
  • zapps : a list of ZApps that have to be shown in the shop

For each ZApp:

  • category : the category this ZApp belongs to, it is used to group ZApps in the web interfaces. There are no pre-defined categories and you are free to put anything you want in here
  • name : the human-readable name
  • description : the name of the json file with the Zoe description
  • readable_descr : the name of the markdown file containing user documentation for the ZApp
  • parameters : a list of parameters the user can set to tune the ZApp before starting it
  • disabled_for (optional) : list of roles that will not see this ZApp in the app shop

Parameters:

Parameters are values of the JSON description that are modified at run time.

  • kind : the kind of parameter, it can be service_count, command or environment
  • name : the machine-friendly name of the parameter
  • readable_name : the human-friendly name of the parameter
  • description : an helpful description
  • type : string or integer, used for basic for validation
  • default : the default value
  • max : if type is integer, this is required and is the maximum value the user can set
  • min : if type is integer, this is required and is the minimum value the user can set
  • step : if type is integer, this is required and is the step for moving between values

Parameters can be of the following kinds:

  • environment : the parameter is passed as an environment variable. The name of the environment variable is stored in the name field. The JSON description is modified by setting the user-defined value in the environment variable with the corresponding name. All services that have the variable defined are modified.
  • command : the service named name has its start-up command changed to the user-defined value
  • service_count : the service named name has its total_count and essential_count changed to the user-defined value

By default users with the user and admin roles have also access to parameters via the web interface. They can set the amount of memory and cores to reserve before starting their execution. The configuration option no-user-edit-limits-web can be used to disable access to this feature.

To get started, in the contrib/zapp-shop-sample/ directory there is a sample of the structure needed for a working zapp-shop, including some data science related ZApps. Copy it as-is in your ZApp shop directory to have some Zapps to play with.

Example of distributed environment

For running heavier workloads and distributed applications, you need a real container cluster. In this example we will use the DockerEngine back-end, as it is simpler to setup than Kubernetes.

Software:

  • One or more Docker Engines
  • Zoe
  • NFS (or another distributed filesystem like CephFS)
  • A Postgresql server

Topology:

  • One node running Zoe. Depending on how your users will access the workspaces you may want to add gateway containers for SSH and/or SOCKS proxies on this node.
  • At least one worker node with a Docker Engine
  • A file server running NFS: depending on the workload it can be co-located with Zoe
  • A Postgresql server, again it can be colocated depending on your expected load

To configure container networking, we suggest the standard Docker multi-host networking.

In this configuration Zoe expects the network filesystem to be mounted in the same location on all worker nodes. This location is specified in the workspace-base-path Zoe configuration item. Zoe will create a directory under it named as deployment-name by default or workspace-deployment-path if specified. Under it a new directory will be created for each user accessing Zoe.

Stand-alone environment for development and testing

A simple deployment for development and testing is possible with just:

  • A Docker Engine
  • Zoe

In the root of the repository you can find a docker-compose.yml file that should help get you started.

You will need to create a /etc/zoe directory containing the docker.conf file that lists the Docker engine nodes available to Zoe.