Instructions for installing the server on Debian/Ubuntu/Mint

These instructions to install the Fosm API server were developed for Linux Mint 18, but should work on modern Debian/Ubuntu systems. Some tweaks would be required for other distros and systems. You need admin privileges to perform several of the steps. It is recommended you have plenty of hard drive space: the empty database takes about 300Gb. The populated fosm database is around 700Gb (as of 2016).

Using a web browser, download the GT.M database from: https://sourceforge.net/projects/fis-gtm/files/GT.M-amd64-Linux/V5.5-000/ and get the file: gtm_V55000_linux_x8664_pro.tar.gz

This guide explains how to install the system into the /srv/fosm path, but this can be changed if needed. The software will be installed under the www-data username and group, because it allows editing by multiple authorized users and the server is largely web based anyway. This guide uses nano as an editor, but any other will do just fine, such as xed, pluma, gedit, emacs, vi, etc. Add yourself to the www-data group:

sudo nano /etc/group

Scroll down. Add your linux username to the end of the line that is like:

www-data:x:33:

Save the file (in nano, do ctrl-o, enter, then ctrl-x), log out and log back in. Check you are in the www-data group by the command:

groups

Extract and install the GT.M database software:

cd ~/Downloads/ mkdir gtm tar -xf gtm_V55000_linux_x8664_pro.tar.gz -C gtm cd gtm sudo mkdir /srv/fosm sudo ./gtminstall --installdir /srv/fosm/gtm55 --user www-data --group www-data cd .. sudo rm -R gtm

Now we start creating the folder structures needed for the server. (Note: the next section, gtmx does not seem to be necessary, just create an empty folder by "mkdir gtmx" instead.)

cd /srv/fosm sudo chown www-data:www-data. sudo chmod ug+rwx. mkdir planet mkdir planet/minute-replicate/ mkdir data mkdir serenji git clone https://github.com/fosm/fosm-scripts.git scripts git clone https://github.com/fosm/gtmx.git git clone https://github.com/fosm/fosm.org-webpage.git ln -s gtm55 gtm ln -s gtmx gtm55x ln -s ../scripts/xapi.gld data/xapi.gld ln -s fosm.org-webpage/fosm.org www mkdir /scripts/o sudo chown www-data:www-data -R. sudo chmod ug+rwx -R.

If you are installing to a custom path, check the first line of /scripts/profile and set the path you are using.

GT.M requires several environment variables to operate correctly. Assuming you are using bash, add the following lines to the end of your ~/.bashrc file:

source /srv/fosm/scripts/profile export EDITOR=/bin/nano

Configure the database to use 8k database blocks instead of the default 4k blocks: edit /etc/sysctl.conf (using sudo because it is protected) and add the following lines: (NOTE: is this necessary? Is a reboot needed to apply this?)

kernel.shmmax = 67108864 kernel.shmall = 67108864
 * 1) XAPI settings for 8k database blocks

Note: the fosm.org server uses kernel.shmmax=4294967295 ... why?

You should be able to initialize the database at this stage. GT.M should always be run from the folder containing the database i.e. /srv/fosm/data

cd /srv/fosm/data mupip create

The system might churn the hard drive at this stage to pre-allocate database space.

You can then import a planet dump file into the database. The planet file must use single quotes for xml tags, not double quotes. It also probably requires each tag to be on a separate line (TBC). (NOTE: This process doesn't work! However, it is possible to skip this and continue with a blank slate database.)

cd scripts mumps loadPlanet.m cd ../data mumps -run ^loadPlanet cat ../hereford.osm > planet.pipe

The apache web server is used to process HTTP requests from the users browser. It also passes API requests to the GT.M scripts. These instructions assume apache 2.4. (fosm.org is currently on version 2.2.9)

sudo apt install apache2

cd /etc/apache2/mods-enabled sudo ln -s ../mods-available/cgi.load sudo ln -s ../mods-available/rewrite.load

sudo rm /etc/apache2/sites-enabled/000-default.conf cd /etc/apache2/sites-enabled/ sudo ln -s /srv/fosm/scripts/fosm.conf fosm.conf

Edit fosm.conf and set a unique password for SERVER_LINK_PASSPHRASE. This password is used by apache to access the private GT.M backend service. If you wish to run the server on a custom port, both fosm.conf and /etc/apache2/ports.conf must be changed.

Restart apache:

sudo service apache2 restart

Use your web browser and look at http://localhost/ to check apache works. If that fails, start by checking:

tail /var/log/apache2/error.log

The website needs more configuration to talk to GT.M. We also need to prepare the xapid service.

cd /srv/fosm/www rm serverLink.cgi ln -s /srv/fosm/scripts/serverLink.cgi serverLink.cgi ln -s /srv/fosm/planet/ planet cd /srv/fosm/www/api/0.6 rm serverLink.cgi ln -s /srv/fosm/scripts/serverLink.cgi serverLink.cgi cd /srv/fosm/www/api/0.6/user/ rm serverLink.cgi ln -s /srv/fosm/scripts/serverLink.cgi serverLink.cgi cd /srv/fosm/scripts make

cd /srv/fosm/data

We also need to set some variables within GT.M. Run:

gtm

Then issue the following commands (note: the specific values are probably absurd but they more or less work):

s ^osmPlanet("instance")="xapi" s ^osmPlanet("date")=20140101 s ^osmPlanet("osmSession")="bbb2559f0e425315329ca1f010197858" s ^serverLink("REST")="rest" s ^serverLink("port")=6530 s ^serverLink("logLevel")=0 s ^date("timezoneOffset")=8 s ^munin("apiCalls")=27597139 s ^munin("responseDB")=84932980 s ^munin("responseIO")=10466660 s ^munin("responseTotal")=141848508 s ^id("changeset")=1000074436 s ^id("node")=1000594558245 s ^id("pendingUid")=100000383 s ^id("relation")=1000000153460 s ^id("uid")=100000156 s ^id("way")=1000002478076

The server password also needs to be set the same as SERVER_LINK_PASSPHRASE in the apache configuration.

d setup^serverLink

Enter a password, enter any server description string, then press enter when it asks for a second password. You can check the current values of these parameters with commands like:

zwr ^osmPlanet zwr ^serverLink WRITE ^serverLink("REST")

Exit GT.M by typing "halt" or "h" and pressing enter (or ctrl-d).

Start the xapi services with the commands:

cd /srv/fosm/data export GTMCI=/srv/fosm/scripts/xapi.ci ../scripts/xapid

This should just run without returning to the command line. Just leave this run. Other commands can be performed in a different terminal window.

You can also run the script fosm_up in the scripts directory. This will return to the command line. cd /srv/fosm/scripts ./fosm_up

You can confirm the service is running using:

netcat 127.0.0.1 6530

If it returns immediately to the command line, the service is not running. Just press enter for netcat and the TCP connection is terminated (by the GT.M server, I think). Next, try doing a basic map query:

http://localhost/api/0.6/map?bbox=-2.742031,52.0509012,-2.7326135,52.0567448

If you have permission problems, try:

cd /srv/fosm sudo chown www-data:www-data -R. sudo chmod ug+rwx -R.

Look at:

tail /var/log/apache2/error.log

If you still can't connect to GT.M

cd /srv/fosm/scripts export SERVER_LINK_IP=127.0.0.1 export SERVER_LINK_PORT=6530 export SERVER_LINK_PASSPHRASE=your-password-here ./serverLink.cgi

That should produce no errors. Incidentally, API errors are logged into the database and can be retrieved by the GT.M command:

zwr ^error

API calls and responses are logged here:

zwr ^log

Crontab
Set cron to run scripts automatically:

sudo crontab -u www-data -e

to contain:

* * * * * cd /fosm/scripts; source exportDiff
 * 1) * * * * * /fosm/scripts/loadDiff
 * 1) 50 * * * * /fosm/scripts/purgeJournal
 * 2) Run the planet dump monthly on 1st of month at 5am
 * 3) 00 05 1 * * /fosm/scripts/planetDump

Create a user account
We can create a user account but bear in mind email sending is probably not configured. You cannot directly use the email confirmation link. However, we can work around this. Register a new user with a web browser:

http://localhost/user/new

Then on the command line:

cd /srv/fosm/data gtm

Then issue the command:

zwr ^pendingUser

Look for the "emailToken" parameter. Use a web browser and go to the URL:

http://localhost/user/confirm/emailToken

with the "emailToken" replaced by the actual value. That should enable the user for editing.

You can now start JOSM, change the server URL to:

http://localhost/api

and start editing!