Poor Man’s Proxmox Cluster

on November 16, 2013 in Linux with 25 comments by
I had written this elsewhere before, but thought I would share it on my own site as well. The idea here is to create a Proxmox VE cluster with limited resources, in particular a lack of a private network / VLAN. We address this by creating a virtual private network using a lightweight VPN provider, namely Tinc.

You could use something else, like OpenVPN or IPSEC. The former is a bit on the heavy side for the task, whilst the latter may not have all the features we need. Specifically, Tinc allows us to create an auto-meshing network, packet switching and use multicast. Multicast will be needed to create a Proxmox VE cluster, whilst the virtual switching ensures packets will eventually be routed to the right server and VM.

Create an additional vmbr

By default there should already be a vmbr0 bridge for Proxmox.  We will need to create – or modify – an additional vmbr, which in this example we name vmbr1.

Warning: on many systems, vmbr0 bridge is used to make your server accessible over the public network – so do not edit that unless absolutely required!

You also need to think of what private IP block you would like to use, and assign each Proxmox VE server an IP from within that private IP block. For example, I use the IP range (which is and a netmask of The 192.168.15.x range I assign to the Proxmox VE servers, whereas the 192.168.14.x range I assign to containers / VMs. Using that IP range, you would change the /etc/network/interfaces file as following:

# for Routing
auto vmbr1
iface vmbr1 inet static
    bridge_ports dummy0
    bridge_stp off
    bridge_fd 0

You can force the changes using:

ifdown vmbr1 && ifup vmbr1

You will need to do this on each server, taking care to select a different IP address. Keep it simple, start at, and increment the last number for each new server.


The next step would be installing Tinc and configuring it in such a way that Proxmox VE can use multicast over that virtual private network.

So on the server, install Tinc with:

apt-get install tinc -y

Next, create a directory where the configuration for the VPN will reside (you can have multiple configurations as such):

mkdir -p /etc/tinc/vpn/hosts

Next, we create a basic configuration, which tells Tinc to use a “switch” mode and what this server’s “name” is. For sake of simplicity, use the hostname for the “name” (use uname -n to determine it):

cat > /etc/tinc/vpn/tinc.conf <<EOF
Name = server1
AddressFamily = ipv4
Device = /dev/net/tun
Mode = switch
ConnectTo = 

The “ConnectTo” is currently left blank, but will be important once you have setup the other servers.  More on this later.

Then we create a server-specific configuration. Note that the filename is the same as specified in “Name =” above.

cat > /etc/tinc/vpn/hosts/server1 <<EOF
Address =
Port = 655
Compression = 0

Obviously you should replace the “Address” line with the actual public IP address of your server.

Now we need to create a public/private key. The private key will remain exactly that: private. The public key will be appended to the file we just created (/etc/tinc/vpn/hosts/server1), which will eventually be distributed to the other servers.

tincd -n vpn -K4096

It will ask you to confirm two file locations. The default should be correct (with the last/2nd one the file as mentioned above).

Now we need an up/down script, to do some post configuration of the network when the VPN comes up (or goes away). This is a simple copy & paste, provided you have setup vmbr1 as outlined earlier:

cat > /etc/tinc/vpn/tinc-up <<EOF

# Attach the 'vpn' interface to vmbr1
/sbin/ifconfig vpn up
/sbin/brctl addif vmbr1 vpn

# Set a multicast route over vmbr1
/sbin/route add -net netmask dev vmbr1

# To allow VMs on a private IP to access the Internet (via vmbr0):
/sbin/iptables -t nat -A POSTROUTING -o vmbr0 -j MASQUERADE

# To allow IP forwarding:
echo 1 > /proc/sys/net/ipv4/ip_forward

# To limit the chance of Corosync Totem re-transmission issues:
echo 0 > /sys/devices/virtual/net/vmbr1/bridge/multicast_snooping

cat > /etc/tinc/vpn/tinc-down <<EOF
/sbin/route del -net netmask dev vmbr1
/sbin/brctl delif vmbr1 vpn
/sbin/ifconfig vpn down
echo 0 > /proc/sys/net/ipv4/ip_forward

chmod +x /etc/tinc/vpn/tinc-up
chmod +x /etc/tinc/vpn/tinc-down

What the above does, is add the VPN tunnel to the vmbr1 bridge. Furthermore, it allows multicast messages over vmbr1. It also sets the use of masquerading, to allow a VM on a private IP to communicate successfully with the outside world – it will use the IP address of vmbr0 to do so.

Then, you need to tell Tinc that the contents in the “vpn” sub-directory should be started whenever it starts:

echo "vpn" >> /etc/tinc/nets.boot

You will need to do this on each server that needs to be part of the VPN. In addition, the files within the directory /etc/tinc/vpn/hosts/ needs to be distributed to all servers (so that all servers have the files from the other servers). Its simple enough to script this, if you want to go that route, but that’s beyond the scope here.

As mentioned earlier, you will need to edit the /etc/tinc/vpn/tinc.conf and provide the name of another server in the “ConnectTo” setting that was previously left blank.  Which server you chose is entirely up to you, and you could chose a different one for each server – remember that Tinc is auto-meshing, so it will connect all servers over time.

Note: without making that change to /etc/tinc/vpn/tinc.conf, Tinc will not know what to do so you will not have a working VPN as a result.

Once you have edited the configuration as outlined, (re)start Tinc using the following command:

service tinc restart

And test your network by pinging another node on its private IP, ie:

ping -c3

Note I use the “-c3” here, to limit the amount of pings. If the VPN was not configured correctly, or a firewall is interfering, you may otherwise end up with a large number of “Host or destination is unreachable” errors.

Forcing the private IP address

We need to force Proxmox VE, or more specifically Corosync, to use the private IP addresses rather than the public IP address.  This because the multicast needs to be done over our virtual private network.

The easiest, but also the “dirtiest” method is to simply change the /etc/hosts, which I will outline here.

The first step is to ensure that the /etc/hosts file is read before attempting to do a full DNS lookup:

cat > /etc/host.conf <<EOF
order hosts, bind
multi on

Next edit the /etc/hosts file, by commenting out the original line, and adding our own:

# Original:
# server1.myprovider.com server1

# Ours: server1.myprovider.com server1

Make sure that the private IP address matches the one you assigned to vmbr1 (double check with ifconfig vmbr1).

Again, this is a “dirty” method and you may want to use your own DNS server instead that resolves IPs for a local network (say, “server1.servers.localnet”).

At this stage, reboot the server to ensure the changes get picked up and everything works as expected (that is, your server comes back up online – hmm!).

Create the cluster

If you do not yet have a cluster configured, you need to create one first. So pick a particular server that you would like to consider as a “main server” and perform the following:

pvecm create <arbitrary-name>

Where <arbitrary-name> is something of your own choosing. Keep the name short and simple, without spaces or other funny characters.

The “main server” is a loose term really, as any server within the cluster can manage other servers. But use it as a consistent starting point for adding other servers to the cluster.

You can check if things are working correctly with:

~# pvecm status
Node name: server1
Node ID: 1
Node addresses:

In particular, you’d want to make sure that the “Node addresses:” portion is the private IP address as on vmbr1.

Adding servers to the cluster

Adding a server (node) to the cluster will need a little preparation. Specifically, because we use private IP addresses for the cluster, we need to force other nodes to do the same when trying to contact another node. In other words, if server1 wants to contact server2, it should use the 192.x range instead of the public IP address.

So, based on the above example, on server1 we need to add a line to the /etc/hosts like this:

cat >> /etc/hosts <<EOF server2.myprovider.com server2
Note the double “>>” brackets. If you use a single “>” one, you overwrite the entire file with just that line. You’ve been warned.

And on server2, we need to make sure server1 can be contacted using its private IP as well, so on that server, we perform:

cat >> /etc/hosts <<EOF server1.myprovider.com server1

All of this can be made much fancier with your own DNS server and bindings, but again, this is beyond the scope and goes on the assumption you don’t mind doing this for the 2, 5 or 10 servers or so you may have. If you have a few hundred, then I wouldn’t expect you to be looking at a “Poor Man’s” setup.

On the server that you will be adding to the cluster, make sure that you can successfully ping that private IP address of the “main server”.

If tested OK, then still on that server (thus the one that isn’t yet part of the cluster), type:

pvecm add server1

Where “server1” is the “main server” (the one on which you first created the cluster). It will ask you for the root password for SSH for server1, and then does its thing with configuration.

Note: If you have disabled password-based root logins using SSH, you may have to temporarily enable it. Using SSH keys would be a preferred method over passwords.

After this has been done, the node should automatically on your web-based GUI and can be verified from the CLI using:

pvecm nodes

If the nodes show up in the “pvecm nodes” command and GUI, then you have successfully created the cluster.

Note: A note about a 2-node cluster and quorum can be found here.

Containers and VMs

You can now create containers and VMs that can be migrated between the nodes.

You can either assign the private IP address directly (venet, only on OpenVZ containers) or as a network device (veth) attached to vmbr1.

The private IP address should be within the range of your specified netmask on vmbr1. So going by the above example of using, that’s anything between and Make sure the IP isn’t already used by another VM or a node (see initial notes, re 192.168.14.x for VMs).

If you fire up the VM, its private IP address should be ping-able from any server, and from within the container / VM, you can ping any private as well as public IP address (the latter thanks to masquerading configured with the tinc-up script). If this is not the case, the network configuration was not done correctly.

Final notes

You should now have at least one container / VM with a private IP address. Its good and well if this VM doesn’t need to be accessed from the outside world, but if you want to give it such access, you will need to use NAT on the server. This will instruct the node that incoming traffic on a particular port will need to be forwarded to a particular VM.

For example, TCP port 25 on is forwarded to VM on IP

iptables -A FORWARD -p tcp -i vmbr0 -d --dport 25 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 25 -j DNAT --to-destination

Note that this is just a simple guide to help you get started. More importantly, it doesn’t include any basic security measures such as a firewall (there are other articles about a firewall for Proxmox on this site [here and here], which I will update when I can).

Photo by ivanx


  1. Jennifer cally
    posted on Nov 18, 2017 at 11:00 PM  |  reply

    for anyone having problems with /etc/tinc/nets.boot” does not start the tinc service any more in Debian Stretch

    i found a little work around a bug is posted and should be fixed but it still does not work on reboots.

    you need to do.
    “systemctl enable [email protected]

    but you still need t do this after every reboot,
    “systemctl restart tinc.service”

    Just in case the cleaning lady at OVH pulls out the power cord again going into ssh on all servers to do this at boot is a bit of pain. Or whatever the node resets for.

    so i made a little cron job that seems to work and boots the vpn at startup

    “mkdir /etc/boot/”
    “cd /etc/boot”
    “nano vpn.sh”
    in nano one line “systemctl restart tinc.service”
    “chmod +x vpn.sh”

    Setup cron
    “crontab -e”
    At end of file
    “@reboot /etc/boot/vpn.sh

    hope this helps someone

  2. jennifer
    posted on Jul 19, 2017 at 5:13 PM  |  reply

    hey any update/upgrade why this will not work on Debian Stretch – Proxmox 5.0 i tried everything. and the vpn just does not ping each other no more

  3. Michael
    posted on Jun 29, 2017 at 9:16 AM  |  reply


    may I ask you how you handle shared storage? Over the same VPN? Ceph?
    So if a connect a local instance and e.g. a virtual cluster in a hosting center (1Gbit connection), how does this scale?

    Any experiences with this or do you all use local storage?

    • Myatu
      posted on Jul 01, 2017 at 7:13 PM  |  reply

      I used Gluster, but would not recommend that in a poor man’s cluster or unless you can dedicate a NIC specifically for storage — the throughput just isn’t good enough to keep up. For most instances I used local storage, and where needed, used unison to sync data for VMs.

  4. posted on May 11, 2017 at 2:25 AM  |  reply

    The vpn is not established on Debian Stretch – Proxmox 5.0

    It’s not possible to ping other hosts, though no errors are displayed when starting tinc. (I have used the guide several times with Proxmox 4.x without issue.)

    Any thoughts on how to change the scripts so that it works with the new version of Debian?

    • Myatu
      posted on May 13, 2017 at 2:43 PM  |  reply

      There do appear to have been some changes between Tinc 1.0.24 (in Jessie) and 1.0.31 (Stretch), such as a change of default encryption — if you are connecting to hosts with an older version of Tinc, then that might be where the issue lies. I suppose the thing to start with is to specifically set an encryption and digest method in the configuration (see tinc.conf manual). Also use the tcpdump tool — you can limit it to check ICMP packets, so you can see what is happening with traffic when you ping across hosts.

    • Alex
      posted on Jul 26, 2017 at 2:00 PM  |  reply

      you need to install net-tools ;)

    • Andy
      posted on Aug 29, 2017 at 3:11 PM  |  reply

      Hi, any solution to that problem? Havin exact the same. I did found out that the “/etc/tinc/nets.boot” does not start the tinc service any more in Debian Stretch. So I had to do “systemctl enable [email protected]” to activate the service on startup. After that tinc starts, but it is not establishing the connection.

      Only after I do a

      “service tinc stop”
      “tincd -n vpn”

      The connection is established.

      The funny part about that is, that after starting tinc manually, you can kill it again, and start it as a service with “service tinc start”. It will establish again the connection. But after a reboot the same problem occours again.


      • jennifer
        posted on Nov 16, 2017 at 11:00 PM  |  reply

        it seems a bug


        • luc
          posted on Nov 18, 2017 at 5:35 PM  |  reply

          For this issue, I created a systemd override:


          as current global file is only with a network.target and it tried to start too early from what I saw in the logs, so failed and retried only 60sec later. And then the proxmox cluster processes were already started, and not in good shape, and it was mostly not working/recovering properly.

          And to make sure it starts the “dedicated” vpn, I made my [email protected] depend on tinc:
          /etc/systemd/system/[email protected]/override.conf

          ExecStart=/usr/sbin/tincd -d 2 -n %i -D

          The execstart change is only to get some more info in syslog (-d 2).
          1 msg per hour when it expires keys, and more importantly it logs to which instance if connects/disconnects

          I’m no systemd expert, but from my readings, seems like the way to go.

  5. posted on Oct 15, 2016 at 9:17 PM  |  reply

    Your guide is very useful. I couldn’t find any other guide that works correctly for this kind of scenario. Thank you.

  6. Andy
    posted on Jul 11, 2016 at 9:11 AM  |  reply

    Hi, thx for you great tutorial. I was able to establich a cluster over VPN with this.

    But I would have just a short question:
    What if network breaks (connection lost) for a few seconds. Testing this with “/etc/init.d/networking restart” it seems that tinc do not establish again the VPN connection!

  7. Kamran Biglari
    posted on Jul 02, 2016 at 1:10 PM  |  reply


    Can any body help me on an issue ?
    All my VPS outgoing IP are same and it used VMBR0 so this cause big problem for me !
    Because I’m unable to send email and also activate licenses over my VPS !

    Anybody know to use owned container IP addresses instead of main ip address that used on vmbr0 for each machines ?

  8. embb
    posted on Nov 25, 2015 at 11:59 AM  |  reply

    Your tutorial is priceless. Thanks so much:
    However on proxmox 4.0 I’ve added those two lines on tinc-up in order to have the services restart so the node joins the cluster again. I suppose the tinc only starts after proxmox services:

    systemctl restart pvestatd.service
    systemctl restart pvedaemon.service

    Not sure if the best approach but works fine on my side.

  9. Stelios
    posted on Jan 25, 2015 at 11:38 PM  |  reply

    Hi Myatu,

    I follow your tutorial but on the vpn setup both servers are not able to ping each other. There is no firewall at all running on the 2 nodes. Any way to troubleshoot this?


    • Myatu
      posted on Feb 16, 2015 at 9:48 AM  |  reply

      Make sure that the “ConnectTo” is specified in the VPN configuration, and check the syslog for details from Tinc to see if it is running into any trouble.

  10. posted on Nov 27, 2014 at 2:55 PM  |  reply

    Against proxmox wiki, I managed the change of IP and interfaces on 2 nodes in cluster without rebuilding everything.
    After a physical migration of hsots, multicast on new swithes were not allowed. I connected 2 hosts via a cross cable, configured new interfaces and
    Just changed /etc/hosts so localhost.localdomain localhost ctrvmc01 ctrvmc02 pvelocalhost ctrvmc02.mio.it ctrvmc01.mio.it
    on both nodes.
    restarted services:
    504 service cman start
    505 service pve-cluster start

    and it worked!

  11. Bogdan Popa
    posted on Oct 03, 2014 at 6:45 PM  |  reply


    I followed your instructions but I’m having some issues with tinc, looks like omnicast isn’t working for some reason:

    [email protected] ~ # omping : waiting for response msg : waiting for response msg : waiting for response msg : waiting for response msg : waiting for response msg : waiting for response msg : waiting for response msg : waiting for response msg : waiting for response msg : waiting for response msg

    [email protected] ~ # ping -c5
    PING ( 56(84) bytes of data.
    64 bytes from icmp_req=1 ttl=64 time=1.80 ms
    64 bytes from icmp_req=2 ttl=64 time=1.78 ms
    64 bytes from icmp_req=3 ttl=64 time=1.80 ms
    64 bytes from icmp_req=4 ttl=64 time=1.66 ms
    64 bytes from icmp_req=5 ttl=64 time=1.80 ms

    — ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4007ms
    rtt min/avg/max/mdev = 1.663/1.773/1.809/0.077 ms

    • Myatu
      posted on Oct 15, 2014 at 9:24 PM  |  reply

      Make sure that the route for is present. It is described in the /etc/tinc/vpn/tinc-up script in this post, and can be verified with route -n, ie:       U     0      0        0 vmbr1

      The other thing to keep in mind is firewalls. For example, Shorewall needs to have the setting MULTICAST in shorewall.conf set to yes.

      I use ssmping to test multicast. So on Node A I need to ensure that UDP port 4321 is permitted, and start ssmpingd without any parameters. On Node B, I’d use:

      asmping IP_OF_NODE_A

      And should see something to akin to the following (don’t mind the high RTT):

      asmping joined (S,G) = (*,
      pinging from
        unicast from, seq=1 dist=0 time=21.722 ms
        unicast from, seq=2 dist=0 time=21.673 ms
        unicast from, seq=3 dist=0 time=21.770 ms

      Additionally, you can use tcpdump to have a look what is going on:

      tcpdump -i vpn -s0 -vv -n net

      Which would show something akin to:

      22:21:53.611960 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 40, options (RA))
 > igmp v3 report, 1 group record(s) [gaddr to_ex { }]
      22:21:53.638069 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 66)
 > [udp sum ok] UDP, length 38
  12. nicco
    posted on Aug 08, 2014 at 3:07 PM  |  reply

    Hey man!

    Wonderful tutorial, thanks so much.

    I’m trying to figure this thing out. I got everything working, can ping the other nodes but not a OpenVZ VPS from the nodes other than the one where it resides.

    Could you point me in the right direction as why this might be happening?

    Also, you don’t mention fencing at all. Is this something that’s needed?


    • Myatu
      posted on Aug 09, 2014 at 11:18 PM  |  reply

      It will depend a bit on how you’ve setup thing in the end. For example, if each host will have its own gateway (the main private IP address assigned), then you wouldn’t be able to ping the nodes on the other server – at least not with some further work. And you’ll want to ensure that the host nodes will have a firewall setup, disallowing ARPs going to the public side for the private IP range.

  13. posted on Apr 02, 2014 at 1:28 AM  |  reply

    I got public KVM IP working with bridged to vmbr0 with a mac assigned in the soyoustart control panel

    I couldn’t get the vm onto the private VPN ip for some reason

    do I need to add each KVM vm as a new server on the Tinc VPN?

    thanks in advance

  14. posted on Apr 01, 2014 at 8:47 PM  |  reply

    ok, thanks for info, will defer the shorewall aspects for now

    now I am struggling with the network configuration for the KVM vm’s

    what should the /etc/network/interfaces look like (I am only interested in KVM at present)?


  15. posted on Mar 26, 2014 at 4:57 PM  |  reply

    hi, wonderful tutorial, very helpful

    am wondering, is this procedure compatible with your other blog articles on Shorewall / Firewall?

    thanks indeed

    • Myatu
      posted on Mar 29, 2014 at 5:24 PM  |  reply

      Portions of them are, but not entirely no. You would have to ensure that the ports required by Proxmox are accessible by the other machines in the cluster. The ports are explained in http://pve.proxmox.com/wiki/Ports. This is in addition to the port required for Tinc, which is 655 (both UDP/TCP if you can).

Join the discussion

Your email address will not be published. Required fields are marked *