{"id":1249,"date":"2021-10-20T13:01:43","date_gmt":"2021-10-20T13:01:43","guid":{"rendered":"https:\/\/globalgoodplay.com\/?p=1249"},"modified":"2023-06-23T09:23:37","modified_gmt":"2023-06-23T09:23:37","slug":"load-balancing-smartfoxserver-2x-with-haproxy-2","status":"publish","type":"post","link":"https:\/\/globalgoodplay.com\/?p=1249","title":{"rendered":"Load Balancing SmartFoxServer 2X with HAProxy"},"content":{"rendered":"

In this article we are going to explore several ways to use the open source HAProxy load balancer<\/strong> in conjunction with SFS2X<\/strong>, to increase the scalability<\/strong> and availability<\/strong> of a multiplayer project.<\/p>\n

We are going to show different configurations for TCP<\/strong> and Websocket<\/strong> connections and several ways to setup the system for common use cases.<\/p>\n

To get the most out of this tutorial we require a basic knowledge of what a Load Balancer is and how it works, and some familiarity with the basics of networking and the OSI Model.<\/p>\n

\u00bb Introducing HAProxy<\/h2>\n

In the words of its creators, HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications.<\/em><\/p>\n

This software is particularly well known in the Linux community and it is a a popular choice for many large traffic websites. It can operate both at layer 4<\/strong> (TPC) and layer 7<\/strong> (HTTP) of the OSI Model so it\u2019s perfectly suitable to load balance a multiplayer service such as SmartFoxServer 2X<\/strong>.<\/p>\n

In the next sections we\u2019re going to install and setup HAProxy<\/strong> on a dedicated Linux machine to act as a Load Balancer for multiple SFS2X instances, supporting both TCP and Websocket clients.<\/p>\n

\u00bb Round robin Load Balancing<\/h2>\n

In our first setup we aim at running a number of SFS2X instances behind a single HAProxy and let it balance the incoming client traffic. All users will point to the balancer\u2019s IP address (or domain) which it will take care of passing the connection to one of the SFS2X instances.<\/p>\n

\"\"<\/figure>\n

In particular this will be done using the simple Round Robin<\/strong> algorithm, that distributes the connections across all instances evenly.<\/p>\n

To get started we launched a new Ubuntu machine in AWS EC2 and installed HAProxy<\/strong> with this command:<\/p>\n

sudo apt install haproxy<\/p>\n

After a few seconds the load balancer should be up and running. We can verify it with this:<\/p>\n

service haproxy status<\/p>\n

Before we dive into the HAProxy configuration<\/strong> we also need to setup couple of SmartFoxServer 2X instances. We\u2019ll skip over this section as you should already be familiar with this simple process and assume they are already set up.<\/p>\n

For our example we assume the two SFS2X servers have private IP addresses of 10.0.0.10<\/strong> and 10.0.0.11<\/strong><\/p>\n

\u00bb Configuring HAProxy<\/h2>\n

We can now proceed by editing the HAProxy\u2019s configuration found under \/etc\/haproxy\/haproxy.cfg <\/strong><\/p>\n

For example:<\/p>\n

sudo nano \/etc\/haproxy\/haproxy.cfg<\/p>\n

HAProxy<\/strong> has a vast number of settings but is relatively simple to configure and the default haproxy.cfg file is pretty minimalistic. Essentially there are four main sections<\/strong> in the configuration, called:<\/p>\n

    \n
  • globals<\/strong>: process-wide settings for performance and security<\/li>\n
  • defaults<\/strong>: default values that you may or may not override in the next sections<\/li>\n
  • frontend<\/strong>: defines the interface with the clients, such as listening ports<\/li>\n
  • backend<\/strong>: describes the servers that are available for HAProxy to load balance<\/li>\n<\/ul>\n

    While the globals<\/strong> and defaults<\/strong> section appear only once in the .cfg file, there can be multiple frontend<\/strong> and backend<\/strong> sections. For example we could bind multiple ports on the frontend for different services, such as TCP socket and Websocket. Similarly we can define multiple backed sections that map the two frontends to their respective servers.<\/p>\n

    Let\u2019s take a look at the configuration we have used for this setup to clarify what described so far.<\/p>\n

    Global<\/h3>\n
    global\n        log \/dev\/log    local0\n        log \/dev\/log    local1 notice\n        chroot \/var\/lib\/haproxy\n        stats socket \/run\/haproxy\/admin.sock mode 660 level admin expose-fd listeners\n        stats timeout 30s\n        user haproxy\n        group haproxy\n        daemon\n        maxconn 50000\n\n        # Default SSL material locations\n        ca-base \/etc\/ssl\/certs\n        crt-base \/etc\/ssl\/private\n\n        # See: https:\/\/ssl-config.mozilla.org\/#server=haproxy&server-version=2.0.3&config=intermediate\n        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM->\n        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256\n        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets<\/code><\/pre>\n
    What you see here is 99% the standard settings from the default config file, we just added the maxconn<\/strong> parameter to allows a high number of incoming connections.<\/p>\n
    For the moment we can ignore the rest of the settings and move on to the next section.<\/p>\n
    Defaults<\/h3>\n
    defaults\n        log     global\n        mode    tcp\n        timeout connect 5000\n        timeout client  50000\n        timeout server  50000\n        timeout tunnel  180s<\/code><\/pre>\n
    The mode tcp<\/strong> command tells HAProxy that by default incoming connections should be treated as layer-4 TCP connections<\/strong>, rather than layer-7 HTTP ones. We\u2019ll return on this topic in a moment.<\/p>\n
    The various timeout<\/strong> commands control the client-to-balancer and balancer-to-server timeouts and can be changed as preferred. The last timeout tunnel <\/strong>is specific for Websocket connections, once again regulating the length of an idle connection.<\/p>\n
    Please note that unless otherwise specified, values are interpreted as milliseconds. In alternative you can add an \u201cs<\/strong>\u201d for seconds, \u201cm<\/strong>\u201d for minutes and \u201ch<\/strong>\u201d for hours.<\/p>\n
    Frontend<\/h3>\n
    In this section of the .cfg file<\/strong> we can define two different frontends<\/strong>: one for the default TCP port 9933<\/strong> and one for the Websocket traffic, on port 8080<\/strong>.<\/p>\n
    frontend sfs-socket\n        bind *:9933\n        default_backend servers-socket\n\nfrontend sfs-websocket\n        mode http\n        bind *:8080\n        default_backend servers-websocket<\/code><\/pre>\n
    The first thing to note is that we do not define a mode<\/strong> in the sfs-socket<\/em> frontend as we have already declared mode tcp<\/strong> in the previous default<\/strong> section.<\/p>\n
    However we do declare mode http<\/strong> in the sfs-websocket<\/em> frontend declaration to override the defaults.<\/p>\n
    Finally the default_backend<\/strong> command is referring to the backend<\/strong> blocks that we\u2019re about to define.<\/p>\n
    Backend<\/h3>\n
    Here we declare the two referenced backed items:<\/p>\n
    backend servers-socket\n        balance roundrobin\n        server sfs1 10.0.0.10:9933\n        server sfs2 10.0.0.11:9933\n\nbackend servers-websocket\n        mode http\n        balance roundrobin\n        option  forwardfor\n        server sfs1 10.0.0.10:8080 check fall 3 rise 2\n        server sfs2 10.0.0.11:8080 check fall 3 rise 2<\/code><\/pre>\n
    The servers-socket<\/strong> block defines the targets of TCP load balancing, using the standard SFS2X TCP port value (9933).<\/p>\n
    Please Note<\/strong>: when defining the server endpoints we always use the private IP addresses<\/strong> of these machines, as load balancer and relative nodes must communicate in a fast, low-latency<\/strong> private network.<\/p>\n
    The server-websocket<\/strong> block is where we define the Websocket endpoints, using the default 8080 port.<\/p>\n
    The option forwardfor<\/strong> directive tells HAProxy to pass the original client IP address<\/strong> to the endpoint by adding an X-Forwarded-For<\/strong> (XFF) entry to the HTTP header file. This in turn is supported by SmartFoxServer by enabling it in the AdminTool<\/strong> > Server Configurator<\/strong> > Web Server<\/strong><\/p>\n
    Finally the check fall 3 rise 2<\/strong> parameters in the servers definition activates the Load Balancer\u2019s health check system, specifying how many checks are required to determine whether a server is active or not.<\/p>\n
      \n
    • fall 3<\/strong>: means that after 3 consecutive failed checks the server is excluded from the load balancing pool<\/li>\n
    • rise 2<\/strong>: indicates that after 2 consecutive successful checks the server is added to the load balancing pool<\/li>\n<\/ul>\n
      \u00bb Caveats<\/h2>\n
      Client IPs in TCP mode<\/h3>\n
      When running in TCP mode<\/strong> there is no way for HAProxy to tell SmartFoxServer what is the original IP address of the client, therefore all users connecting via TCP will appear to have the IP address of the Load Balancer.<\/p>\n
      While HAProxy<\/strong> knows the original IP of the sender it has no clue about the content of the TCP packets it is receiving and therefore there is no way to inject that information somewhere in the data stream.<\/p>\n
      This is only possible when working in HTTP mode<\/strong>.<\/p>\n
      Other load balancing algorithms<\/h3>\n
      Round-robin<\/strong> is one of many possible load balancing systems that can be used with HAProxy. Depending on the mode<\/strong> in use (tcp vs http) it is possible to choose from a wide list of algorithms.<\/p>\n
      You can learn more about what is available in the HAProxy documentation.<\/p>\n
      SSL Certificates<\/h3>\n
      If you need to activate HTTPS<\/strong> for Websocket or TCP encryption you will need to perform extra configuration steps. In particular you will need to deploy your SSL certificate on the Load Balancer, since it is the entry point for all of your clients.<\/p>\n
      It is beyond the scope of this tutorial to walk you through the HAProxy SSL setup process, but you can read all the details in the excellent documentation provided by on their website.<\/p>\n
      What about UDP?<\/h3>\n
      HAProxy does not support UDP load balancing, so the solution presented here would not be work with an SFS2X project that requires it.<\/p>\n
      Possible alternatives could be:<\/p>\n
        \n
      • IPVS a Linux-only, kernel-based load balancing tool<\/li>\n
      • Nginx another popular proxy\/load balancer\/web server<\/li>\n<\/ul>\n
        Single point of failure<\/h3>\n
        As you have probably realized if all the client traffic goes through one Load Balancer this will become a single point of failure<\/strong>. How can we prevent a service blackout if the HAProxy becomes unavailable?<\/p>\n
        Typically we would need to run an active HAProxy<\/strong> and one or more passive replicas<\/strong> (i.e. with exactly the same configuration) that can take over whenever it is necessary.<\/p>\n
        Also these servers would need to be behind a Virtual IP<\/strong> address which can be pointed to a different machine when necessary, making the transition from a failed Load Balancer to an active one fully transparent.<\/p>\n
        If you\u2019re interested in learning more about this technique you can find the details in this article by Oracle.<\/p>\n
        \u00bb Backup servers<\/h2>\n
        An interesting feature in HAProxy<\/strong> is that of backup<\/strong> servers<\/strong>, which are instances defined in the backend<\/strong> section of the config that are not added to the load balancing pool until it becomes empty.<\/p>\n
        Let\u2019s go back to our websocket setup for instance:<\/p>\n
        backend servers-websocket\n        mode http\n        balance roundrobin\n        option  forwardfor\n        server sfs1 172.31.21.251:8080 check fall 3 rise 2\n        server sfs2 172.31.24.55:8080 backup check fall 3 rise 2<\/code><\/pre>\n
        Here we have slightly modified the configuration by adding a backup<\/strong> keyword to the sfs2<\/strong> instance (last line). What does it do?<\/p>\n
        By marking this server as backup<\/strong> the Load Balancer will no longer distribute clients among the two instances but rather keep sending users to sfs1<\/strong> until it becomes unavailable.<\/p>\n
        When this happens the sfs2 instance becomes active<\/strong> and stays like that until one ore more non-backup servers are added to the pool. For example when sfs1<\/strong> is restarted and becomes functional again.<\/p>\n
        This simple option can be useful for different scenarios, such as running a large Lobby server<\/strong> acting as the entry point for users, who can then be sent to other servers to play games. The single Lobby server could be run behind an HAProxy and with a backup Lobby ready to take over when necessary.<\/p>\n
        Here\u2019s an hypothetical architecture for a cluster that combines what we just described with the previous setup:<\/p>\n
        \"\"<\/figure>\n
        We now have a Lobby Balancer<\/strong> as the main entry point for our application: here users will login, manage their profile, search for other users, create and manage their Buddy Lists and chat with them.<\/p>\n
        The Game Balancer<\/strong> instead will direct players to different game servers where they can be matched with other users and play.<\/p>\n
        Finally we have a main database<\/strong>, shared among all servers, that can be user to track active users, access their state, game statistics, leaderboards, buddy lists and more.<\/p>\n
        \u00bb Wrapping up<\/h2>\n
        The topic of high availability and scalability in clusters is a huge subject and we\u2019ve barely scratched the surface. We may be returning on this topic in the future with more articles but, for now, we hope to have provided enough concepts to get you started experimenting.<\/p>\n
        As usual, if you have any comments, questions or doubts let us know via our SmartFoxServer support forum.<\/p>\n","protected":false},"excerpt":{"rendered":"
        In this article we are going to explore several ways to use the open source HAProxy load balancer in conjunction with SFS2X, to increase the scalability and availability of a multiplayer project. We are going to show different configurations for TCP and Websocket connections and several ways to setup the system for common use cases.<\/p>\n","protected":false},"author":1,"featured_media":1250,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[],"class_list":["post-1249","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"_links":{"self":[{"href":"https:\/\/globalgoodplay.com\/index.php?rest_route=\/wp\/v2\/posts\/1249"}],"collection":[{"href":"https:\/\/globalgoodplay.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/globalgoodplay.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/globalgoodplay.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/globalgoodplay.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1249"}],"version-history":[{"count":2,"href":"https:\/\/globalgoodplay.com\/index.php?rest_route=\/wp\/v2\/posts\/1249\/revisions"}],"predecessor-version":[{"id":1304,"href":"https:\/\/globalgoodplay.com\/index.php?rest_route=\/wp\/v2\/posts\/1249\/revisions\/1304"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/globalgoodplay.com\/index.php?rest_route=\/wp\/v2\/media\/1250"}],"wp:attachment":[{"href":"https:\/\/globalgoodplay.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1249"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/globalgoodplay.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1249"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/globalgoodplay.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1249"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}