Cloning a B-Translator Server

Why do I need to clone a B-Translator server? An obvious reason is to have a backup server. In case that something goes wrong with the main server I can quickly switch to the backup server, until I find and fix the problem.

A clone can also be used for testing. Before applying something on the main server I can test it first on a clone/backup server.

Another reason for cloning can be load-balancing. It is possible to keep several clone servers synchronised with each-other, and then the work can be shared between them with a load balancer. I haven't tried this yet, but I think that it should work.

It can be possible and even simple to make identical clones just by copying the physical disk or the filesystem. However I prefer to build a new server from scratch and copy only the relevant data from the old server.

Table Of Contents

• Installation
• Setting the domain name
• Replacing NGINX with Apache2
• Enable other drupal features
  • Enable the feature btranslator_fb
• Start ssh
• Reset the password of drupal admin
• Transfer content
• Fix path aliases and menus
• Transfer drupal private settings
• Get and import PO files
  • Download (get) PO files
  • Import PO files
• Sync vocabulary data
• Sync users and contributions
• Switching to the new server

Installation

First, I build a new server from scratch:

mkdir /var/chroot
cd /var/chroot/
git clone https://github.com/dashohoxha/B-Translator.git
nohup nice B-Translator/install/install.sh btr-1 &
tail -f nohup.out

Then configure it:

chroot btr-1/ /tmp/install/config.sh

And then reboot the host:

reboot  ## it is advisable to reboot the host after this installation

The last step of configuration is about setting the primary language of translation (sq in my case), and a list of auxiliary languages. This list should include at least fr, since it is used instead of POT (PO template) files, when they are missing. But the features related to auxiliary languages are not implemented yet, so it is not much useful to include other languages on that list, and I leave just fr.

Setting The Domain Name

If I can assign a domain name to this copy of B-Translator (for example test.l10n.org.al), but during the configuration it is not given properly, then I can correct/change it like this:

chroot /var/chroot/btr-1
cd /var/www/btranslator/profiles/btranslator
install/config/domain.sh

If this is a local copy (installed on my local machine) and its domain name is for example l10n.org.xy, then I also add this domain name on the file /etc/hosts of my machine:

127.0.0.1       l10n.org.xy  localhost

This way I can access it by typing https://l10n.org.xy on the browser location (not https://127.0.0.1 orhttps://localhost).

Replacing NGINX With Apache2

I have used NGINX as the default web server because it is supposed to be faster and more responsive under heavy load. Unfortunately it has been working very slow, for some reasons that I have not been able to find and fix. On the log file/var/log/nginx/btranslator.error.log I see messages like this:

2013/07/15 13:31:57 [info] 10294#0: *47 client closed prematurely connection while SSL handshaking, client: 127.0.0.1, server: l10n.org.xx
2013/07/15 13:31:57 [info] 10294#0: *48 client closed prematurely connection while SSL handshaking, client: 127.0.0.1, server: l10n.org.xx

Note: If find out what is wrong with NGINX and manage to fix it, please let me know.
So, I replace NGINX with Apache2 like this:

chroot /var/chroot/btr-1
cd /var/www/btranslator/profiles/btranslator
dev/apache2.sh start

It will stop services nginx, php5-fpm and memcached, will start apache2, will disable drupal module memcache, and will modify properly settings.php.

However there are also some changes outside chroot that should be done. The script dev/apache2.sh start cannot do them automatically (because it runs inside the chroot), so I have to do them manually. On the init script/etc/init.d/chroot-btr-1 I make this change:

#CHROOT_SERVICES="cron php5-fpm memcached mysql nginx"
CHROOT_SERVICES="cron mysql apache2"

Enable Other Drupal Features

After installation I also enable some features that are optional and are not enabled by default during installation:

drush features-list
drush --yes pm-enable btranslator_simplenews
drush --yes pm-enable btranslator_mass_contact
## drush --yes pm-enable btranslator_googleanalytics
## drush --yes pm-enable btranslator_drupalchat
## drush --yes pm-enable btranslator_janrain

Enable The Feature Btranslator_fb

Note: The feature btranslator_fb is not yet finished and tested properly.
Enable it like this:

## drush --yes pm-enable btranslator_fb

After installing btranslator_fb, the configuration part related to FB should be un-commented, at the end of the file/var/www/btranslator/sites/default/settings.php:

// /* fb config
$conf['fb_api_file'] = 'profiles/btranslator/libraries/facebook-php-sdk/src/facebook.php';
include "profiles/btranslator/modules/contrib/fb/fb_url_rewrite.inc";
include "profiles/btranslator/modules/contrib/fb/fb_settings.inc";
if (!headers_sent()) {
  header('P3P: CP="We do not have a P3P policy."');
}
// fb config */

If you forget to do it, you will notice performance degrade with the site.

Start Ssh

If this copy of B-Translator is remote, then I install ssh as well for accessing it easily and for using remote drush commands:

chroot /var/chroot/btr-1
cd /var/www/btranslator/profiles/btranslator
dev/install-sshd.sh

This script will also take care to change the ssh port to 2201, in order to avoid any conflicts with any existing daemon on the host environment, and also for increased security.

For drush remote access to work correctly, the public/private key ssh access should be set up and configured as well. For more detailed instructions on how to do it see: http://dashohoxha.blogspot.com/2012/08/how-to-secure-ubuntu-server.html

Reset The Password Of Drupal Admin

I almost always forget the password of admin (the primary user of Drupal) that I assign during installation. So, I have to reset it:

drush user-password admin --password="new-password"

By the way, on the file /etc/drush/drushrc.php you can see this drush setting:

// by default use the B-Translator root directory
$options['r'] = '/var/www/btranslator';

This means that the root directory of drush is always /var/www/btranslator, no matter where we call it from.

Transfer Content

On the master (main/live) server, I export all the content as a feature, with the help of the module node_export. But first I have to disable the existing btranslator_content feature, otherwise the feature export will fail.

drush --yes pm-disable btranslator_content
drush --yes features-export --destination=/tmp btranslator_content_1 node_export_features
tar --create --gzip --file=btranslator_content_1.tgz --directory=/tmp btranslator_content_1

Now I transfer btranslator_content_1.tgz to the clone server and replace the existing content with it:

cd /var/www/btranslator/profiles/btranslator
cd modules/features/
tar --extract --gunzip --file=btranslator_content_1.tgz
drush --yes pm-disable btranslator_content
drush delete-all all  ## delete all existing nodes
drush --yes pm-enable btranslator_content_1

Fix Path Aliases And Menus

Path aliases and some menus have to be fixed (recreated) manually. I couldn't find any modules, drush commands or scripts that can transfer them automatically. If you know any tricks to export/import them automatically, please let me know.

I transfer manually the configuration of the Homebox as well. I open admin/structure/homebox on both sites (the main and the clone), export the configuration of 'dashboard' from the main, then copy/paste and import it on the clone.

Transfer Drupal Private Settings

Private settings are those variables that are site specific and cannot be included in features, for example: disqus_domain,disqus_userapikey, disqus_publickey, disqus_secretkey, etc.
We can transfer them like this:

• Save them on the main site:

  cd /var/www/btranslator/profiles/btranslator
  modules/features/save-private-vars.sh
  

  It will generate the file restore-private-vars.php.
• Transfer restore-private-vars.php to the clone site and then apply it like this:

  drush php-script restore-private-vars.php
  

Note: If the clone site will be used for testing, you may consider to edit manually the file restore-private-vars.php, before applying it, and change some values. For example, I usually change email addresses from info@l10n.org.al toinfo+test@l10n.org.al. I also enable email rerouting by changing these variables:

$variables['reroute_email_enable'] = 1;
$variables['reroute_email_enable_message'] = 1;

Get And Import PO Files

The database of translations is almost empty (it has only the PO files that were imported for testing during installation). Downloading and importing all the PO files is easy (but it takes a long time).

Download (Get) PO Files

cd /var/www/btranslator_data

nohup nice get/all.sh &
tail -f nohup.out

or

cd get/
./gnome.sh
./kde.sh
./firefox-os.sh
./drupal.sh
./office.sh
./mozilla.sh
./wordpress.sh
./ubuntu.sh

Note: These scripts get the data from some URL. They should be checked first, to make sure that the URL still works or that we are getting the latest version.

Note: Make sure that hostname is listed on /etc/hosts otherwise the command svn checkout will not work (strange, but that's how it is). For example if the output of the command hostname is dashamir, then /etc/hosts should look like this:

127.0.0.1 l10n.org.xx localhost dashamir

Import PO Files

cd /var/www/btranslator_data
nohup nice import/all.sh &
tail -f nohup.out

Sync Vocabulary Data

Vocabulary is a pseudo-project, its PO file does not really belong to the translation of any program. I use it to collect interesting terms and translations that I encounter while translating the other projects. It can help as a reminder (in case that I forget the translation of a term). It is also useful for discussing translations of difficult terms with other people, and indirectly it helps to ensure consistency among the translations. Terms are added to vocabulary by the translators. In order to transfer them from one instance of B-Translator to another, it can be exported as a PO file on one system and imported to the other.

• Export vocabulary-sq.po:

  cd /var/www/btranslator_data
  export/export.sh misc vocabulary sq $(pwd)/tmp
  mv tmp/vocabulary/vocabulary-sq.po .
  rm -rf tmp
  

• Transfer vocabulary-sq.po to the other system and them import it:

  cd /var/www/btranslator_data
  mv vocabulary-sq.po vocabulary/
  import/vocabulary.sh
  

Sync Users And Contributions

Now the cloned site is almost identical with the primary site in terms of Drupal settings and configuration and in terms of translation data (projects that are imported, strings and their translations, etc.). What is still missing is the list of users that are registered on the primary site, as well as their contributions (votes and suggested translations).
We can transfer them like this:

• Export them on the primary site:

  cd /var/www/btranslator_data
  db/export-users.sh
  db/export-contributions.sh
  

• Transfer *.sql.gz files to the clone site and import them:

  cd /var/www/btranslator_data
  db/import-users.sh  users-20130717.sql.gz
  db/import-contributions.sh contributions-00000000-20130717.sql.gz
  

Note: Once you have transferred all the users, the clone site will send them daily a string for review, in addition to the one that is sent by the primary site. We can prevent this by disabling the cron. Edit the file /etc/cron.d/drupal7 and comment the line that starts the cron.

Switching To The New Server

Suppose that I want to make the cloned server primary. In this case there are some steps that should be done:

• Transfer GoogleApps verification files. I use GoogleApps for email accounts etc. (it offers 10 email accounts for free). To verify that I own this domain, GoogleApps requests me to put a certain file on the root of my webserver. This file looks like google9350a51ac2d503bf.html and I place it on /var/www/btranslator.
• Transfer SSL certificates. I have obtained a free SSL certificate for my site (see:http://arstechnica.com/security/2009/12/how-to-get-set-with-a-secure-sertificate-for-free/). The configuration on/etc/nginx/sites-available/default looks like this:

  ssl_certificate         /etc/ssl/certs/ssl-cert-l10n_org_al.pem;
  ssl_certificate_key     /etc/ssl/private/ssl-cert-l10n_org_al.key;
  

  The corresponding configuration on /etc/apache2/sites-available/default-ssl looks like this:
  

  SSLCertificateFile    /etc/ssl/certs/ssl-cert-l10n_org_al.pem
  SSLCertificateKeyFile /etc/ssl/private/ssl-cert-l10n_org_al.key
  

  The files .pem and .key need to be transferred to the new server and the configuration files of nginx and apache2 should be modified properly.
• Enable cron. Since I have disabled cron (on a test site), I have to enable it again by un-commenting it on/etc/cron.d/drupal7.
• Replace test settings with live settings. Export drupal setting on the main site with modules/features/save-private-vars.sh and then import them on the new site with drush php-script restore-private-vars.php.
• On the DNS server I change the record of l10n.org.al to point to the new IP. But the DNS change may take about 2 days to be propagated worldwide. So, after 2-3 days I do again a transfer of users and contributions from the old server to the new one. These transfer operations are designed to be idempotent, which means that the result will be the same even if they are applied many times.

Author: Dashamir Hoxha 

Date: 2013-07-18 11:47:36 CEST

HTML generated by org-mode 6.33x in emacs 23