Introduction ¶

WAM-Server can be set up either manually (setting up database, environment and wam by hand) or via docker. Both methods will be presented in the following.

Contents

Idea ¶

WebAppMap-Server [WAM] provides a basic and expandable Django infrastructure to easily add applications.

Installation via Docker ¶

Prerequisites ¶

git is installed.
Docker and Docker-Compose are installed.

Setup ¶

Following folder structure is recommended and used in the following:

wam_docker
+-- docker (Contains docker config and WAM code basis)
|   +-- docker-compose.yml
|   +-- WAM (WAM-Codebasis; WAM-apps are integrated here later)
+-- config (Config for WAM and apps)
|   +-- config.cfg

Note

Changes to this structure have to be adopted in config file and docker config (docker-compose.yml)

Setup of folder structure and code basis:

mkdir wam_docker
cd wam_docker
mkdir docker
mkdir config
cd docker
git clone https://github.com/rl-institut/WAM.git
cp WAM/docker-compose.yml .
cp WAM/.config/config.cfg ../config/

Next, config files have to be adopted (docker-compose.yml und config.cfg). Finally, following command builds image and starts new container:

sudo docker-compose up -d --build

WAM-Server should now be available under 127.0.0.1:5000 !

Installation from Scratch ¶

Prerequisites ¶

PostgreSQL library should be installed.
A PostgreSQL database should be created (see PostgreSQL Setup).
PostGIS library should be installed (see PostGIS Setup).
if Setup shall be used, a Message Broker must be used.

PostgreSQL Setup ¶

This section describes the installtion of PostgreSQL on Linux and Windows.

Linux¶

The following instructions are for Ubuntu and inspired from here.

First, create a user name (here, wam_admin is used for the USER field of the config file configuration)

sudo -u postgres createuser --superuser wam_admin

Then enter in psql shell

sudo -u postgres psql

There, change the password for the user wam_admin

postgres=# \password wam_admin

Enter the same password you will use under the PASSWORD field in the config file (configuration) and exit the shell with \q

Then, create the database you will use under the NAME field in the config file (configuration)

sudo -u postgres createdb -O wam_admin wam_database

Whenever you want to use the database you should run

sudo service postgresql start

This can be stopped using the command

sudo service postgresql stop

Windows¶

Download and install latest PostgreSQL for Windows.
At the end of or after the install of PostgreSQL for Windows use Stack Builder (will be installed with PostgresSQL) to install Spatial Extensions -> PostGIS bundle

In the Windows command line (cmd.exe) or Powershell:

Set the path environment variable for PostgreSQL (to be able to use PostgreSQL via the command line), e.g.:

SETX PATH "%PATH%;C:\Program Files\PostgreSQL\11\bin"

Login to “psql” as user “postgres”:

psql -U postgres

In psql create superuser “wam_admin”:

CREATE ROLE wam_admin WITH LOGIN SUPERUSER INHERIT CREATEDB CREATEROLE REPLICATION;

Set a password for the superuser wam_admin:

\password wam_admin

Quit psql:

\q

Create database:

createdb -U wam_admin wam_database

How to start, stop and restart PostgreSQL on Windows:

Start:

pg_ctl -D "<drive letter>:\path\to\PostgreSQL\<version>\data" start

Stop:

pg_ctl -D "<drive letter>:\path\to\PostgreSQL\<version>\data" stop

Restart:

pg_ctl -D "<drive letter>:\path\to\PostgreSQL\<version>\data" restart

PostGIS Setup ¶

This section describes the installtion of PostGIS on Linux and Windows.

Linux (Ubuntu)¶

sudo apt-get install binutils libproj-dev gdal-bin

sudo apt-get install postgis postgresql-10-postgis-2.4

For other systems see https://postgis.net/.

Activate postgis extension (execute as SQL query) to make it work:

CREATE EXTENSION postgis;

Windows¶

If not already installed, use Stack Builder (will be installed with PostgresSQL) to install Spatial Extensions -> PostGIS bundle

Login to psql as wam_admin in wam_database:

psql -U wam_admin wam_database

Create postgis extension:

CREATE EXTENSION postgis;

Quit psql:

\q

Message Broker ¶

On our WAM-Server the message broker RabbitMQ is running in a docker container (see RabbitMQDocker). Developers allowed to use our service can connect to it by setting up a ssh-tunnel:

ssh -fNL 5672:localhost:5672 wam_user@wam.rl-institut.de

Other users have to setup their own message broker

Getting Started ¶

Setup on Linux:¶

Clone repository from GitHub via:

git clone https://github.com/rl-institut/WAM.git

Setup conda environment with required packages via:

conda env create -f environment.yml

Afterwards, applications can be “plugged-in” by simply cloning application into the root directory and adding application name to environment variable WAM_APPS (see environment). Requirements and configuration of an application can be found at Deploying server with custom apps

Linux: Environment Variables¶

WAM-Server needs at least the following environment variables:

WAM_CONFIG_PATH: full path to the configuration file (see Configuration file for the file content)
WAM_APPS: Apps which shall be loaded within INSTALLED_APPS. Additionally, individual app settings are loaded (see Deploying server with custom apps).

On Ubuntu, one can add them to the bashrc file, so that they are loaded automatically :

nano ~/.bashrc

then add the following two lines

export WAM_CONFIG_PATH=<path to your WAM repo>/.config/config.cfg
export WAM_APPS=<name of wam app 1>,<name of wam app 2>

Setup on Windows:¶

Prerequisites:

Git Bash (included in Git for Windows)
Install the small version of Conda (Miniconda), the big version Anaconda is not needed (reverse install settings, include Anaconda to PATH but don’t install as default): Link.

Open Git Bash and create a project folder

mkdir project_folder_name
cd project_folder_name

Clone repository from GitHub via:

git clone https://github.com/rl-institut/WAM.git

Setup conda environment with required packages via:

conda env create -f environment.yml

Afterwards, applications can be “plugged-in” by simply cloning application into the root directory and adding application name to environment variable WAM_APPS (see environment). Requirements and configuration of an application can be found at Deploying server with custom apps

Install package dependencies of your WAM app(s):

Example app: stemp_abw

conda install -c conda-forge gdal
conda activate django
conda install shapely
pip install -r requirements.txt
pip install -r stemp_abw/requirements.txt

Windows: Environment Variables¶

WAM-Server needs at least the following environment variables:

WAM_CONFIG_PATH: full path to the configuration file (see Configuration file for the file content)
WAM_APPS: Apps which shall be loaded within INSTALLED_APPS. Additionally, individual app settings are loaded (see Deploying server with custom apps).

Add them to your Windows user environment:

SETX WAM_CONFIG_PATH "<drive letter>:\<path to your WAM repo>\.config\config.cfg"
SETX WAM_APPS "<name of wam app 1>,<name of wam app 2>"

Configuration file ¶

Configuration file located under the path given by WAM_CONFIG_PATH is loaded within settings .py. The file is read in using python’s configobj package. The file should contain following sections:

[WAM]: general config for the WAM-Server,
[DATABASE]: with at least one default database connection, which will be used as django’s database,
[CELERY]: if celery is needed
[<APP_NAME>]: Multiple sections containing config for each app

See minimal example config_file:

[WAM]
    DEBUG=False
    ALLOWED_HOSTS=127.0.0.1
    SECRET_KEY=<secret_key>
    DJANGO_DB=DEFAULT

[DATABASES]
    [[DEFAULT]]
        ENGINE = django.contrib.gis.db.backends.postgis
        HOST = localhost
        PORT = 5432
        NAME = wam_database
        USER = wam_admin
        PASSWORD = wam_password
    [[LOCAL]]
        ENGINE = postgresql
        HOST = localhost
        PORT = 5432
        NAME = wam_database
        USER = wam_admin
        PASSWORD = wam_password

[CELERY]
    HOST = localhost
    PORT = 5672
    USER = <USER>
    PASSWORD = <PASSWORD>
    VHOST = <VHOST>

Note: the indent level is important in the configuration file. Keywords are specified within [], they are analog to the key in a python dict. The nesting level of a keyword depends on the number of square brackets.

Deploying server without apps ¶

Even without adding apps into WAM, you can follow these steps to deploy the WAM server locally.

Make sure the postgresql service is running

sudo service postgresql start

Then, run the following commands

python manage.py makemigrations

python manage.py migrate

python manage.py createsuperuser

After the last command, follow the instructions inside the terminal and use the same values for user and password as the USER and PASSWORD fields of the Configuration file.

Finally access to the WAM server with

python manage.py runserver

Example app ¶

From the root level of the WAM repository, you can clone the app WAM_APP_stemp_mv with

git clone https://github.com/rl-institut/WAM_APP_stemp_mv.git

For the time being you have to rename the app folder stemp and set your environment variable WAM_APPS to stemp

Deploying server with custom apps ¶

Requirements:

urls.py which includes app_name equaling the app name and an index page, which is loaded as landing page by default

Additional setups:

settings.py can setup additional parameters for projects settings.py.

If your app requires the use of additional packages, you should list them in the settings.py of your app (not the settings.py file form wam core) in the following way

INSTALLED_APP = ['package1', 'package2']

Then, wam core will manage the packages’ installation and avoid duplicate installations between the different apps.

app_settings.py contains application specific settings and is loaded at start of django server at the end of settings.py. This file may include additional database connections, loading of config files needed for the application, etc.

Warning

Avoid using config variables for packages in your app as it may override or get overridden by package config of other app!

labels.cfg (uses configobj) supports easy adding of labels to templates via templatetags (see Labels)

To install the required packages for each app run

python install_requirements.py

from the root level of the WAM repository. Then follow the procedure described under Deploying server without apps

Celery ¶

Celery is included in WAM. In order to use celery in an app, follow the setup.

Setup ¶

Message Broker has to be running.
Celery must be configured to use the message broker (see configuration).
Run celery from WAM root directory via command (environment_ variables have to be set!):

celery -A wam worker -Q wam_queue -l info

After running above command, celery searches for tasks.py in every app and “activates” all tasks in it.
WAM-apps are now able to run celery Tasks!

Tasks ¶

Celery tasks are easy…

You simply have to create a task.py in your app and put from wam.celery import app at beginning of the module. Then, “normal” python functions can be turned into celery tasks using decorator @app.task. See minimal example:

from wam.celery import app


@app.task
def multiply_by_ten(number):
    return number * 1000

Afterwards, the task can be imported, run and controlled from elsewhere:

from app_name import tasks

running_task = None

def start_task():
    # Running task has to be stored, in order to get results
    running_task = tasks.multiply_by_ten.delay(10)

def task_is_ready():
    # Returns True if task has finished
    return running_task.ready()

def get_results():
    # Returns results of the tasks
    return running_task.get()

Note

Input parameters and output results of celery tasks must be pickleable or jsonable. In order to use Django models within the celery task you should exchange primary keys (see also https://oddbird.net/2017/03/20/serializing-things/).

See http://docs.celeryproject.org/en/latest/userguide/tasks.html for more information on tasks.

Introduction ¶

Idea ¶

Installation via Docker ¶

Prerequisites ¶

Setup ¶

Installation from Scratch ¶

Prerequisites ¶

PostgreSQL Setup ¶

Linux¶

Windows¶

PostGIS Setup ¶

Linux (Ubuntu)¶

Windows¶

Message Broker ¶

Getting Started ¶

Setup on Linux:¶

Linux: Environment Variables¶

Setup on Windows:¶

Windows: Environment Variables¶

Configuration file ¶

Deploying server without apps ¶

Example app ¶

Deploying server with custom apps ¶

Celery ¶

Setup ¶

Tasks ¶

WAM-Server

Navigation

Related Topics