CipherSpace Load Balancer

From CipherSpace Client Wiki
Jump to navigation Jump to search

CipherSpace Load Balancer

The CipherSpace Load Balancer uses HAProxy and its configuration is defined using OpenNebula Contextualization Variables within the appliance's template.

The CipherSpace Load Balancer uses the concept of "sticky sessions", which is a mechanism to route requests from the same source to the same target (if the target is still available).

How to use the CipherSpace Load Balancer

Download the Load Balancer from the App Market

  1. Go to "Storage" in the left menu and click on "Apps" in the drop down menu.
    Lb 1.png
    A list with all existing apps will appear.
    Lb 2.png

  2. Select the CipherSpace Load Balancer and click on the "-> OpenNebula" button.
    Lb 3.png
    A new window will appear.
    Lb 4.png

  3. You can rename the image ("Name") as well as the VM template name. Then select the datastore which is used for your VMs and click on "Download". A new image as well as a new template will be created.

Configure the Load Balancer

  1. Go to "Templates" in the left menu and click on "VMs" in the drop down menu.
    Lb 5.png
    A list with all existing VM templates will appear.
    Lb 6.png

  2. Select the template for the load balancer and click on "Update".
    Lb 7.png

  3. Click on the "Network" tab
    Lb 8.png
    and select a Virtual Network for your NIC (the same network as the VMs which provide the service you would like to load balance).
    Lb 9.png

    If clients connect to your load balancer from a different network (Internet for example) you need to add another NIC and attach it to the appropriate Virtual Network.
    Lb 10.png

  4. Click on the "Context" tab
    Lb 11.png
    and then click on the "Custom vars" section.
    Lb 12.png

  5. In the "Custom vars" section, click on the "+" button to enter all the necessary contextualization variables and their respective values (see the section Load Balancer Contextualization Variables).
    Lb 13.png

  6. If you have 2 NICs, you should set default gateway by adding the variable GATEWAY_IFACEand assign it with the number of the NIC on which requests from clients are coming. (typically 1).
    GATEWAY_IFACE = 1

  7. When you're done, click on "Update" to save your changes.
    Lb 14.png

Load Balancer Contextualization Variables

Basic Configuration

In order to understand the contextualization variables, let's use the following example.

You have three web servers and a load balancer attached to the same virtual network. The web servers have the IP addresses 10.4.0.11, 10.4.0.12 and 10.4.0.13 respectively. They listen to http traffic on port 8080. The load balancer listens to http traffic on the port 80.

In this example you only have one service (http) which needs to be balanced.
Every contextualization variable for this service will have a name like SERVICE_0_....
If you define more services, each new service variables will start with
SERVICE_1_, SERVICE_2_, etc..

For this particular example you will define the following variables:

  • SERVICE_0_PROTO
  • SERVICE_0_LISTEN_PORT
  • SERVICE_0_IP_RANGE
  • SERVICE_0_BACKEND_PORT

The variable SERVICE_X_PROTO defines the mode in which the service works. Possible modes are:

  • http: the instance will work in HTTP mode. The client request will be analyzed in depth before connecting to any server. Any request which is not RFC-compliant will be rejected. Layer 7 filtering, processing and switching will be possible.
  • tcp: the instance will work in pure TCP mode. A full-duplex connection will be established between clients and servers, and no layer 7 examination will be performed. It should be used for SSL, SSH, SMTP, ...

In our example SERVICE_0_PROTO has the value HTTP

The variable SERVICE_X_LISTEN_PORT = { int } defines the TCP port the load balancer is listening to.
This variable is mandatory. If this variable isn't defined, the rest of the service definition will be ignored.
In our example SERVICE_0_LISTEN_PORT has the value 80

The variable SERVICE_X_IP_RANGE = { dotted notation-dotted notation} defines the IP addresses of the backend servers. This variable is mandatory.
In our example SERVICE_0_IP_RANGE has the value 10.4.0.11-10.4.0.13

The variable SERVICE_X_BACKEND_PORT = { int } defines the TCP port the backend servers are listening to. This variable is optional. If this variable isn't defined, the value of the variable SERVICE_X_LISTEN_PORT will be used instead.
In our example SERVICE_0_BACKEND_PORT has the value 8080

So, our example will be configured as follows: 

   SERVICE_0_PROTO        = HTTP
   SERVICE_0_LISTEN_PORT  = 80
   SERVICE_0_IP_RANGE     = 10.4.0.11-10.4.0.13
   SERVICE_0_BACKEND_PORT = 8080

Elastic Configuration

Let's assume that we have currently three web servers running (as described in the previous section), but we plan to extend our web servers to a maximum of 50. In this case, we just need to put a wider range in the variable SERVICE_X_IP_RANGE as follows: 

   SERVICE_0_IP_RANGE     = 10.4.0.11-10.4.0.60

The load balancer will be then configured to balance the load to these new servers as soon as they are available.

Health Checks

When you use the load balancer, your backend servers are periodically checked if they are available by accepting periodic TCP connections, to ensure that each server is still alive.

Every 2 seconds, each backend server will be checked and its status will be either Up or Down based on the following rules:

  • A server is Up after 2 consecutive successful health checks.
  • A server is Down after 3 consecutive unsuccessful health checks.

You can change the number of consecutive successful health checks using the variable SERVICE_X_CHECK_RISE and the number of consecutive unsuccessful health checks using the variable SERVICE_X_CHECK_FALL.

Let's use the example defined in the Basic Configuration section. If you want to set the number of successful and unsuccessful checks to 3 and 4 respectively for your backend servers, you will add the following to your configuration: 

SERVICE_0_CHECK_RISE = 3
SERVICE_0_CHECK_FALL = 4

By default, server health check only consist in trying to establish a TCP connection. You can add a complete HTTP request after an established TCP connection by adding the following variable:

SERVICE_X_CHECK_URI = { <uri> }

where <uri>is the URI referenced in the HTTP requests.
Responses 2xx and 3xx are considered valid, while all other ones indicate a server failure, including the lack of any response.

HTTPS Support

To enable HTTPS support, you need to specify the following variables like this: 

   SERVICE_X_PROTO        = HTTP
   SERVICE_X_LISTEN_PORT  = 443
   SERVICE_X_SSL_OFFLOAD_CERT  = "crt.pem + key.pem + intermediate.pem"

The variable SERVICE_X_SSL_OFFLOAD_CERT = { string } defines the certificate chains to offload encrypted connections.
It has to be a string which contains the certificate, the key and the chain, all in PEM format; the file order is not important.
For instance let's assume you are using Let's Encrypt certificates, you will concatenate the two files privkey.pem and fullchain.pem as follows:

  • In Linux (or in MacOS via terminal): cat privkey.pem fullchain.pem
  • In Windows (via command prompt): type privkex.pem fullchain.pem

Then copy the result and paste it in the Value field.


If you need to redirect your HTTP traffic to HTTPS, you will assign the variable SERVICE_X_REDIRECT_TO_HTTPS as follows: 

SERVICE_X_REDIRECT_TO_HTTPS = True

You can also define the maximum age of your HTTP Strict Transport Security (HSTS) by assigning the variable SERVICE_X_SSL_HSTS_MAX_AGE as follows: 

SERVICE_X_SSL_HSTS_MAX_AGE  = 15768000   #corresponds to 6 months

The variable SERVICE_X_SSL_HSTS_MAX_AGE = { int } defines the HSTS policy maximum age in seconds. In our example 15768000 seconds correspond to approximatively 6 months.

All Variables

This sections list all available contextualization variables. Their name is composed as follows:

  1. SERVICE_X_ (where X is a number >= 0) defines the load balancing service to be configured;
  2. Keyword defines the corresponding service item to be configured.

Some variables are valid for both protocols (tcp and http) and some are valid only with protocol http.

TCP and HTTP Variables
  • MAXCONN = { int } defines the value of the maxconn parameter, which sets the maximum per-process number of concurrent connections. Proxies will stop accepting connections when this limit is reached.

The default value is 2000. This variable isn't mandatory.

  • SERVICE_X_PROTO = { tcp|http } defines the protocol in service number X.

Default value is http. This variable isn't mandatory.

  • SERVICE_X_LISTEN_PORT = { int } defines the TCP port the load balancer is listening.

Default value is Null. This variable is mandatory.
If this variable isn't defined, the rest of the service definition will be ignored.

  • SERVICE_X_IP_RANGE = { dotted notation } defines the IP addresses of backend server(s).

Default value is Null. This variable is mandatory.
Example: SERVICE_X_IP_RANGE = { 10.4.0.2-10.4.0.20 }

  • SERVICE_X_BACKEND_PORT = { int } defines TCP port on which backend server(s) are listening.

Default value is Null. This variable isn't mandatory.
NOTE if the variable isn't defined, it will take the SERVICE_X_LISTEN_PORT value.

  • SERVICE_X_CHECK_RISE = { int } defines the number of consecutive successful health checks that declare the backend server is alive.

Default value is 2. This variable isn't mandatory.

  • SERVICE_X_CHECK_FALL = { int } defines the number of consecutive unsuccessful health checks that declare the backend server is dead.

Default value is 3. This variable isn't mandatory.

HTTP only Variables
  • SERVICE_X_CHECK_URI = { string } defines the URI referenced in the HTTP health-check.

Default value is Null. This variable isn't mandatory.

  • SERVICE_X_SSL_OFFLOAD_CERT = { string } defines the certificate chains to offload encrypted connections.

It has to be a string which contains the following files in PEM format: certificate, keyand intermediate. Order is not important. Default value is Null. This variable isn't mandatory.

  • SERVICE_X_REDIRECT_TO_HTTPS = { True } defines the presence of a 301 redirect from http to https.

Default value is Null. This variable isn't mandatory.

  • SERVICE_X_SSL_HSTS_MAX_AGE = { int } defines the HSTS policy max-age in seconds.

Default value is Null. This variable isn't mandatory.

Default Null value means that the related configuration is not applied if the variable isn't defined.

Start and Check the Load Balancer

  1. Go to "Templates" in the left menu and click on "VMs" in the drop down menu.
    Lb 5.png
    A list with all existing VM templates will appear.
    Lb 6.png

  2. Select the configured load balancer template and click on "Instantiate".
    Lb 15.png
    A new window will appear
    Lb 16.png

  3. You can assign a name to your load balancer or leave the field empty. Click on "Instantiate" to start the instantiation.
  4. When the new VM is in the RUNNING state, click on the console icon Lb 17.png to access the VM.
  5. In the console you will see the login prompt and after a while you will see wether the HAProxy configuration was successful or not. If not, terminate the VM, modify the variables in the template and re-instantiate the VM.
    Lb 18.png

How to Modify the Load Balancer's Configuration.

Currently you cannot modify the load balancer's configuration while it is running. Also if you were to modify the HAProxy configuration within the load balancer VM, if the VM had to be terminated and re-instantiated, you would loose your modifications.

In order to minimize the downtime, you should:

  1. Clone the Load Balancer's Template in case you have to roll back.
  2. Change the configuration in the new template by adding variables, removing variables or modifying existing variables.
  3. Terminate the old load balancer VM
  4. Instantiate the new template again.

Appendix

Dissecting HAProxy's configuration

HAProxy configuration consists of several sections:

  • the global section, which sets process-wide parameters (static, cannot be modified),
  • and for each defined service, a proxy section, which is built using the contextualization variables.

Global Section

Global section is standardized for every appliance as follows: 

global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    # set default parameters to the intermediate configuration
    tune.ssl.default-dh-param 2048
    ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
    ssl-default-bind-options no-sslv3 no-tls-tickets
    ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
    ssl-default-server-options no-sslv3 no-tls-tickets
    user        haproxy
    group       haproxy
    daemon
    maxconn     2000

Parameter maxconn sets the maximum per-process number of concurrent connections. Proxies will stop accepting connections when this limit is reached. For further information see this page. This parameter can be modified using a contextualization variable.

Proxy Section

In HAProxy, the Proxy configuration for each service contains a set of sections, which can be either:

  • defaults [<name>]
  • frontend <name>
  • backend <name>

Or:

  • defaults [<name>]
  • listen <name>

A defaults section sets default parameters for all other sections following its declaration. Those default parameters are reset by the next defaults section.

A frontend section describes a set of listening sockets accepting client connections.

A backend section describes a set of servers to which the proxy will connect to forward incoming connections.

A listen section defines a complete proxy with its frontend and backend parts combined in one section. It is generally useful for TCP-only traffic.

For the CipherSpace Load Balancer, we use only one defaults section, valid for all services. We also don't use the listen section, as it can be replaced with one frontendand one backend section.

Right now, two major proxy modes are supported: tcp, also known as layer 4, and http, also known as layer 7.

In layer 4 mode, HAProxy simply forwards bidirectional traffic between two sides.
In layer 7 mode, HAProxy analyzes the protocol, and can interact with it by allowing, blocking, switching, adding, modifying, or removing arbitrary contents in requests or responses, based on arbitrary criteria.

Defaults Section

The defaults section is standardized for every appliance as follows: 

  defaults
     log                     global
     option                  redispatch
     retries                 3
     timeout http-request    10s
     timeout queue           1m
     timeout connect         10s
     timeout client          1m
     timeout server          1m
     timeout http-keep-alive 10s
     timeout check           10s
Frontend and Backend Sections

The frontendand backendsections are configured using the contextualization variables defined in the section All Variables.

Variable SERVICE_X_PROTO defines the mode in which the service works. Possible modes are:

  • http: the instance will work in HTTP mode. The client request will be analyzed in depth before connecting to any server. Any request which is not RFC-compliant will be rejected. Layer 7 filtering, processing and switching will be possible.
  • tcp: the instance will work in pure TCP mode. A full-duplex connection will be established between clients and servers, and no layer 7 examination will be performed. This is the default mode. It should be used for SSL, SSH, SMTP, ...

When SERVICE_X_PROTO = http, the service is defined in mode http and the script generates the following configuration: 

  [...]
  frontend service_X
    mode http
    option                  httplog
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    [...]

  backend service_X_backend
    mode http
    balance source
    [...]

When SERVICE_X_PROTO = tcp, the service is defined in mode tcp and the script generates the following configuration: 

   [...]
   frontend service_X
     mode tcp
     option                  tcplog
     [...]

   backend service_X_backend
     mode tcp
     balance source
     [...]

Example:

In the following example, we define the following contextualization variables: 

   SERVICE_0_PROTO        = http
   SERVICE_0_LISTEN_PORT  = 80
   SERVICE_0_IP_RANGE     = 10.4.0.5 - 10.4.0.10
   SERVICE_0_BACKEND_PORT = 8080

Those variables generate the following complete HAProxy's configuration: 

   global
      log         127.0.0.1 local2
      chroot      /var/lib/haproxy
      pidfile     /var/run/haproxy.pid
      # set default parameters to the intermediate configuration
      tune.ssl.default-dh-param 2048
      ssl-default-bind-ciphers       ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA84:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHEECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RA-DES-CB       C3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
      ssl-default-bind-options no-sslv3 no-tls-tickets
      ssl-default-server-ciphers       ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA84:DHE-R      SA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHEECDSA-AE      S256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RA-DES-CB       C3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
      ssl-default-server-options no-sslv3 no-tls-tickets
      user        haproxy
      group       haproxy
      daemon
      maxconn     2000

   defaults
      log                     global
      option                  redispatch
      retries                 3
      timeout http-request    10s
      timeout queue           1m
      timeout connect         10s
      timeout client          1m
      timeout server          1m
      timeout http-keep-alive 10s
      timeout check           10s
      maxconn                 3000

   frontend service_0
      mode http
      option                  httplog
      option http-server-close
      option forwardfor       except 127.0.0.0/8
      bind :80
      default_backend service_0_backend

   backend service_0_backend
      mode http
      balance source
      server server0 10.4.0.5:8080 check
      server server1 10.4.0.6:8080 check
      server server2 10.4.0.7:8080 check
      server server3 10.4.0.8:8080 check
      server server4 10.4.0.9:8080 check
      server server5 10.4.0.10:8080 check
Health Checks

When a server is defined in the backend section, it is possible to define a health check using the setting check.
By default, a server is always considered available. If check is set, the server is available when it is accepting periodic TCP connections, to ensure that it is really able to serve requests.
The default address and port for these tests are those of the server, and the default source is the same as the one defined in the backend.
It is possible to change the port using the port parameter (currently not implemented), and the interval (currently not implemented) and timers using the inter, rise and fall parameters.

The inter default value of 2000 ms between two consecutive health checks is good enough. 

  rise <count>

The rise parameter states that a server will be considered as operational after <count> consecutive successful health checks. This value defaults to 2 if unspecified. 

  fall <count>

The fall parameter states that a server will be considered as dead after <count> consecutive unsuccessful health checks. This value defaults to 3 if unspecified.

The request method is defined in the backend using the option httpchk. The option httpchk enables HTTP protocol to check on the servers health.

Possible configuration are: 

option httpchk <uri>

Arguments are:

<uri>: is the URI referenced in the HTTP requests. It defaults to "/" which is accessible by default on almost any server, but may be changed to any other URI. Query strings are permitted.

By default, server health checks only consist in trying to establish a TCP connection. When option httpchk is specified, a complete HTTP request is sent once the TCP connection is established, and responses 2xx and 3xx are considered valid, while all other ones indicate a server failure, including the lack of any response.

The port and interval are specified in the server configuration.

This option does not necessarily require an HTTP backend, it also works with plain TCP backend.

Example:

In the following example, we define the following contextualization variables: 

[...]
SERVICE_0_CHECK_URI  = "/"
SERVICE_0_CHECK_RISE = 3
SERVICE_0_CHECK_FALL = 4

Those variables generate the following HAProxy's configuration: 

[...]
backend service_0_backend
   mode http
   option httpchk /
   balance source
   server server0 10.4.0.5:8080 check rise 3 fall 4
   server server1 10.4.0.6:8080 check rise 3 fall 4
   server server2 10.4.0.7:8080 check rise 3 fall 4
   server server3 10.4.0.8:8080 check rise 3 fall 4
   server server4 10.4.0.9:8080 check rise 3 fall 4
   server server5 10.4.0.10:8080 check rise 3 fall 4
HTTPS Support

To enable https support, we need to specify in the frontend section the option ssl and the option crt followed by path to the file which contains certificate, privkey, intermediate certificate and dhparam.

Example:

In the following example, we define the following contextualization variables: 

[...]
SERVICE_X_SSL_OFFLOAD_CERT  = "crt.pem + key.pem + intermediate.pem";
SERVICE_X_REDIRECT_TO_HTTPS = True
SERVICE_X_SSL_HSTS_MAX_AGE  = 15768000

Those variables generate the following HAProxy's configuration: 

frontend service_0
  mode    http
  bind    :443 ssl crt /path/to/<cert+privkey+intermediate+dhparam>
  bind    :80
  redirect scheme https code 301 if !{ ssl_fc }

  # HSTS (15768000 seconds = 6 months)
  http-response set-header Strict-Transport-Security max-age=15768000

<Category:OpenNebula>

Retrieved from "https://wiki.cipherspace.com/index.php?title=CipherSpace_Load_Balancer&oldid=757"