Guide: Firewall and router with Proxmox

on August 31, 2009 in Linux with 48 comments by

Firewall and router with ProxmoxBy default Proxmox does not come with a firewall, which may leave it and your virtual servers exposed to the elements of the Internet.

An additional issue arises when a hosting provider blocks servers if unauthorized MAC addresses are detected. As Proxmox’s bridged network creates and exposes MAC addresses for its virtual network interfaces, this may cause your server to be blocked from the hosting provider’s network.

To combat both this article will describe how to create your own virtual network with firewall protection using Shorewall, a popular and effective firewall / router software package.

Overview

Firewall and router with Proxmox

We will be creating three separate zones, namely:

  1. The Internet (“net”);
  2. The firewall / Proxmox Host Node (“fw”); and
  3. The virtual network (“dmz”)

All traffic from the Internet is filtered or blocked by the firewall, after which it is routed to its final destination on the virtual network or the Proxmox Host Node itself. A visual representation of this can be seen in the figure 1.

Host Network Configuration

Some modifications are required to your host network configuration, particularly the default vmbr0 network interface. We will turn it into a blind bridge (without any bride ports) and assign it an IP address within the private IP range of 10.0.0.0/8 (A-Class).

Determine the current network configuration

Before we do this, we need to determine the current network configuration because this can be different depending on the hosting provider and other factors. Assuming that eth0 is the network interface that connects the server to the Internet, we issue the following command:

ifconfig eth0

This will give an output similar to:

eth0      Link encap:Ethernet  HWaddr 00:ff:ff:ff:ff:ff
 inet addr:91.11.22.33  Bcast:91.11.22.255  Mask:255.255.255.0
...

It gives us the current IP address, broadcast address and netmask used by eth0.  One last piece of information we need is the gateway used by eth0, which is obtained with the follwing command:

route -n

You will see an output similar to:

...
0.0.0.0         91.11.22.254    0.0.0.0         UG    0      0        0 eth0

The first column of 0.0.0.0 designates the default route (any traffic that has no specific route), and the second column the gateway. Now that we have obtained all this information, we can edit the /etc/network/interfaces file.

First we need to verify that eth0 has already been defined within this file. It will look similar to:

auto eth0
iface eth0 inet static
        address 91.11.22.33
        netmask 255.255.255.0
        broadcast 91.11.22.255
        gateway 91.11.22.254

...(additional stanzas)...

Where the IPs match those you have obtained by use of the ifconfig and route commands.

Or if your server uses DHCP to assign the IP address then it will look similar to:

allow-hotplug eth0
iface eth0 inet dhcp

...(additional stanzas)...

If either is the case, you can skip the follow section and continue to change the vmbr0 network interface.

Updating the eth0 network interface

If you have reached this section, then your vmbr0 network interface was most likely directly bridged with your eth0 network interface, meaning that vmbr0 contains your public IP address, network gateway and other settings. Because we will turn vmbr0 into a blind bridge in the next section, we need to create or edit a separate eth0 stanza in the /etc/network/interfaces file first.

In the previous steps we have obtained all the information required for this purpose (using the ifconfig and route commands). This information may already be present in your vmbr0 stanza as well, in which case you can use this instead.

We edit the eth0 stanza to look as following:

auto eth0 iface
eth0 inet static
         address 91.11.22.33
         netmask 255.255.255.0
         broadcast 91.11.22.255
         gateway 91.11.22.254

Where the example IPs are replaced by the actual address, netmask, broadcast and gateway IPs found in the previous steps or obtained from the current vmbr0 stanza.

WARNING! It is important to understand that editing the eth0 stanza can lead to an inability to connect to the server if done incorrectly. If you rely on remote access such as SSH, or if you are uncertain about the information that needs to be entered in the various fields, please contact your administrator or hosting provider for assistance. As always, use care and make backups of existing files.

Change the vmbr0 network interface

The vmbr0 stanza in the /etc/network/interfaces will be changed to look like:

... (original configuration) ...

auto vmbr0
iface vmbr0 inet static
        address 10.254.254.254
        netmask 255.0.0.0
        broadcast 10.255.255.255
        bridge_ports none
        bridge_stp off
        bridge_fd 0

The significant changes here are the IP addresses used for the address, netmask and broadcast as well as changing bridge_ports to “none”.  The IP address of 10.254.254.254 can be any in the 10.0.0.0/8 range and so may be changed if you wish. It will later be used as the gateway address for the virtual servers.

The changes can be applied without having to reboot the system using:

/etc/init.d/networking restart

Shorewall

Installing Shorewall is simply a matter of executing the following command:

apt-get install shorewall

If you wish to use shorewall with IPv6 capabilities, a few additional steps will be required. At the moment of writing, Debian has not included the latest version of Shorewall6 into its main package source. You will therefore need to add the SID (unstable branch) to your available package sources:

echo "deb http://ftp.fr.debian.org/debian sid main" >> /etc/apt/sources.list
aptitude update

Once this has been done, Shorewall with IPv6 support – aptly named shorewall6 – can be installed with:

apt-get install shorewall6

NOTE! It is important to note that Shorewall6 makes a strict distinction between IPv4 and IPv6 and each has to be configured individually. That is, the files located in /etc/shorewall/ are for IPv4 only and /etc/shorewall6/ contains files related to IPv6 only. The IPv4 version cannot control what will happen on IPv6 and vice versa!

Most rules and policies in for Shorewall are directly transferable between /etc/shorewall/ and /etc/shorewall6/. Due to this, this article will only focus on IPv4 configuration with one exception:

You will want to edit the /etc/shorewall/shorewall.conf file, and change the following value from:

DISABLE_IPV6=Yes

to:

DISABLE_IPV6=No

If this is not done, any configuration done in /etc/shorewall6/ would be ignored (and will also disable any existing IPv6 traffic).

Enable IP Forwarding

First, a minor modification to the Shorewall application configuration is made to enable IP forwarding. This will permit some functions that will be discussed later.

Edit /etc/shorewall/shorewall.conf and change the following value from:

IP_FORWARDING=Off

to:

IP_FORWARDING=On

Zones

The /etc/shorewall/zones file is created to establish the zones names and what type of zones they are:

#ZONE   TYPE            OPTIONS         IN                      OUT
#                                       OPTIONS                 OPTIONS
fw      firewall
net     ipv4
dmz     ipv4

The fw zone is a self-reference to the server on which Shorewall is running. Note that this will not include the Proxmox virtual servers; you have to consider this as an entirely separate server in this sense.

Interfaces

The network interfaces on the server need to be defined and assigned to a specific zone. The net zone will be assigned to the eth0 network interface and will designate all traffic coming from or going to the internet. The dmz zone will be the internal zone for the virtual network, which will contain the virtual servers that either use venet or veth network interfaces.

Create the /etc/shorewall/interfaces file and add the following:

#ZONE   INTERFACE       BROADCAST       OPTIONS
net     eth0            detect          blacklist,nosmurfs
dmz     venet0          detect          routeback
dmz     vmbr0           detect          routeback,bridge

Policy

A base policy needs to defined for each one of the zones. It specifies the default actions on in- and outgoing traffic, and in this article the following policies will be defined:

Traffic from the firewall to:

  • the internet is permitted
  • DMZs is permitted
  • other processes on the firewall is permitted

Traffic from the DMZ (virtual servers) to:

  • another virtual server is permitted
  • the internet is permitted
  • the firewall is denied and 1 information message per second (with a burst of 2) will be record when access is attempted.

Traffic from the internet to:

  • the firewall is denied
  • DMZs is denied, generating 8 messages per second (with a burst of 30 messages) whenever access is attempted.

Any traffic not defined in any of the zones (either by accident or purposely) will be rejected.

To do this, we will create the /etc/shorewall/policy file:

#SOURCE DEST    POLICY          LOG     LIMIT:          CONNLIMIT:
#                               LEVEL   BURST           MASK

# From Firewall Policy
fw      fw      ACCEPT
fw      net     ACCEPT
fw      dmz     ACCEPT

# From DMZ Policy

dmz     dmz     ACCEPT
dmz     net     ACCEPT
dmz     fw      DROP            info

# From Net Policy
net     fw      DROP            info
net     dmz     DROP            info 

# THE FOLLOWING POLICY MUST BE LAST
#
all     all     REJECT          info
For those who have followed this guide before and are experiencing some performance issues, please remove the limit burst options after “info”, ie “1/sec:2”. The reason is that this takes priority over rules; this was a painful discovery on my end!

Basic Rules

The policy defined earlier will deny any traffic coming from the internet to the firewall, which will include the SSH service and the Proxmox web-based manager. Since this is undesirable, a few rules need to be created that override this base policy.

Create the /etc/shorewall/rules file:

#ACTION          SOURCE     DEST       PROTO   DEST        SOURCE     ORIGINAL    RATE

# Permit access to SSH
SSH/ACCEPT       net        fw         -       -            -          -          6/min:5

# Permit access to Proxmox Manager and Console
ACCEPT           net        fw         tcp     443,5900:5999

# PING Rules
Ping/ACCEPT      all        all

# LAST LINE -- DO NOT REMOVE

As you may notice, there is also an additional rule for ping. For testing purposes, it would be wise to permit a ping from and to any of your zone, including the internet.

The SSH/ACCEPT rule is in fact a macro that comes with Shorewall. You could also define the same rule as following:

ACCEPT       net        fw         tcp       22            -          -          6/min:5

Also, at the very end of the SSH rule you notice “6/min:5”. This specifies the connection rate and in this case it reduces the connection rate to 6 per minute (1 per 10 seconds) with a maximum initial burst of 5. It is added here to slow down brute force SSH attacks.

Testing Configuration

After creating the files, your /etc/shorewall/ directory might look similar to:

drwxr-xr-x  2 root root 4096 2009-07-01 06:36 .
drwxr-xr-x 82 root root 4096 2009-07-06 10:03 ..
-rw-r--r--  1 root root  522 2009-06-26 20:05 interfaces
-rw-r--r--  1 root root  453 2007-11-15 23:24 Makefile
-rw-r--r--  1 root root  781 2009-06-26 21:16 policy
-rw-r--r--  1 root root 2355 2009-07-02 22:42 rules
-rw-r--r--  1 root root 4134 2009-06-20 21:58 shorewall.conf
-rw-r--r--  1 root root  438 2009-06-26 20:04 zones

Before using the configuration you will want to test it first, particularly to make sure you are not blocking SSH access. Issue the following command:

shorewall try /etc/shorewall 60

The parameter 60 refers to 60 seconds. Shorewall will use the configuration located in /etc/shorewall/ for 60 seconds and then reverts to the previous settings (or no firewall).

After issuing the command, establish a new connection to your server using SSH and check whether your Proxmox web-based manager is accessible. If you are receiving error messages from Shorewall or you are unable to access SSH during the 60-second test period, please verify the configuration and try again.

Starting Shorewall

By default Shorewall is not enabled during boot time as a safety precaution for first-time configurations. To enable it, edit /etc/default/shorewall file and change:

startup=0

to:

startup=1

You can start Shorewall manually with:

shorewall start

And after making any changes to the Shorewall configuration, issue the command:

shorewall restart
It is best not to use /etc/init.d/shorewall restart. Doing so will temporarily disable the firewall, which permits access to normally blocked ports. Although this period is brief, it may still be enough time for someone to establish a connection to the server – Shorewall does not block existing connections by default.

Proxmox

Once Shorewall has been configured, there will be three distinct zones on the Proxmox server:

  • the Firewall / Proxmox host at fw
  • the virtual network zone for virtual servers at dmz
  • the internet at net

IP Assignment

To further separate the internet and virtual servers as distinct areas, each virtual server will be assigned an IP address in the 10.0.0.0/8 range (10.0.0.1 – 10.255.255.254).

The exception is that one can no longer use 10.254.254.254 as this has been assigned to the vmbr0 network interface earlier in this guide.

Outgoing internet traffic

Due to this separation and the use of A-class (10.0.0.0/8) IP addresses, outgoing traffic from a virtual server to the internet needs to be translated (so that Shorewall and other Internet routers know where to send responses to). This will be defined in the /etc/shorewall/masq file.

In its simplest form, /etc/shorewall/masq can be set to the follwing:

#INTERFACE      SOURCE          ADDRESS         PROTO   PORT(S) IPSEC   MARK
eth0            10.0.0.0/8

# LAST LINE -- DO NOT REMOVE

This means that all traffic originating from 10.0.0.0/8 and going to the internet will pass through the eth0 network interface using the IP address assigned to eth0.

If you wish to make all traffic appear from a particular IP addresses, it can be specified as the third parameter. For example:

eth0            10.0.0.0/8    91.121.0.1

Or perhaps there’s a specific internal IP address that must appear externally as another IP address, you can do this as folowing:

+eth0           10.0.1.101    91.121.0.2
eth0            10.0.0.0/8    91.121.0.1

Notice the plus (‘+’) sign in front of eth0. All traffic from 10.0.0.0/8 will appear to be coming from IP 91.121.0.1, except traffic coming from 10.0.1.101 will appear as coming from 91.121.0.2.

Incoming internet traffic

The separation between the internet and virtual servers not only applies to outgoing traffic, but also incoming traffic. There are two methods of directing incoming traffic,which is Proxy ARP or DNAT. This article will focus on DNAT; for more information on Proxy ARP and Shorewall, visit http://www.shorewall.net/manpages/shorewall-proxyarp.html.

For example, to forward HTTP traffic on any external IP address to a virtual server with the assigned IP of 10.0.1.101, edit the /etc/shorewall/rules file as following:

...(existing rules)...
DNAT                    net     dmz:10.0.1.101          tcp     80

The added benefit of DNAT is that a single IP address can be used for multiple virtual servers, provided that the traffic is on a different port. For example, HTTP traffic on external IP address 91.121.0.1 may be sent to a virtual server with the assigned IP of 10.0.1.101, whereas FTP traffic may be sent to a virtual server with assigned IP of 10.0.1.102 instead:

...(existing rules)...
DNAT                    net     dmz:10.0.1.101          tcp     80       -    91.121.0.1
DNAT                    net     dmz:10.0.1.102          tcp     21,23    -    91.121.0.1

It is even possible to route traffic to a different internal port. For example, to forward HTTP traffic on external IP address 91.121.0.1 to a virtual server with the assigned IP of 10.0.1.103 and listening on 8180:

...(existing rules)...
DNAT                    net     dmz:10.0.1.103:8180     tcp     80       -    91.121.0.1

Bridged Networking

The venet network interface is certainly the simplest method to use in Proxmox. However, venet is not available in KVM (fully virtualized servers) and there may be another reason why you might want to use the veth network interfaces with regular containers (such as the use of DHCP).

For this reason the vmbr0 network interface on the host was reconfigured to use the IP address of 10.254.254.254. It will act as the gateway entry for those virtual servers using veth network interfaces.

Although additional configuration needs to be done within a virtual server, you can use the same Shorewall rules for in- and outgoing traffic as described earlier (ie., DNAT or outgoing traffic).

Linux (Debian)

Inside a Debian Linux virtual server, you will specify /etc/network/interfaces as following:

auto eth0
iface eth0 inet static
  address 10.0.1.101
  netmask 255.0.0.0
  gateway 10.254.254.254

where 10.0.1.101 is the IP address to be used by this particular virtual server.

If you are using both venet and veth network interfaces at the same time, as may be the case with certain IPv6 configurations, the file /etc/network/interfaces.tail should be used instead.

Microsoft Windows

For networking within Windows, proceed to your Networking control panel (or the Network and Sharing Center). Select the appropriate Local Area Connection and right-click to reveal the Properties menu option. UAC (User Account Control) may request your permission to proceed.

In the list of This connection uses the following items, select Internet Protocol (TCP/IP) (or Internet Protocol Version 4 (TCP/IPv4)). Click the Properties button.

At the General tab, change the following selections:

Use the following IP address:

  • IP address: 10.0.1.101
  • Subnet mask: 255.0.0.0
  • Default gateway: 10.254.254.254

Use the following DNS server addresses:

  • Preferred DNS server: xxx.xxx.xxx.xxx
  • Alternate DNS server: yyy.yyy.yyy.yyy

Where xxx… and yyy… are your preferred DNS servers.

Non-private IP Assignment

The setup as described above has separated your virtual servers from the internet by use of a zone (dmz) and A-class IP range (10.0.0.0/8). However, it is still possible to assign a non-private IP directly to one of your virtual servers if the venet network interface is used.

Internet traffic (from the net to the dmz zone) will still be blocked per the policy established in the above setup, and you will need to add additional rules to your Shorewall configuration. The major difference is that you must use ACCEPT instead of DNAT.

For example, let’s assume the IP address 91.121.0.1 was directly assigned to a virtual server. To permit internet Web traffic (port 80) to this container, add the following rule to your /etc/shorewall/rules file:

ACCEPT                  net     dmz:91.121.0.1             tcp     80

Please note that for veth network interfaces (bridged) Proxy ARP is required.

Extra Hardening

Following are considered advanced topics to further enhance the firewall and Proxmox. They may not be required in all situations.

Restricting IP Addresses per MAC

When the veth bridged networking is used in Proxmox, the virtual server will have a fully emulated network interface with its own MAC address. The downside is that Proxmox cannot assign the IP address directly and so you will have to configure this within the virtual server.

This should not be an issue if you have full control over the virtual server. However, if the virtual server belongs to a customer or in a worst case scenario is compromised, you no longer have full control over it and it will be possible to change or assign additional IP addresses to the virtual server.

Although earlier in this article containers have already been separated from the internet, it would still be possible to use a private IP address or network interface of an already existing other container. It may therefore be possible to spoof another virtual server.

To implement additional safeguards against this, Shorewall needs a few extra rules described here.

Enable the maclist

First the maclist option needs to be enabled for the interface we will monitor MAC / IP relations on. Your /etc/shorewall/interfaces may currently look like this:

#ZONE   INTERFACE       BROADCAST       OPTIONS
net     eth0            detect          blacklist,nosmurfs
dmz     venet0          detect          routeback
dmz     vmbr0           detect          routeback,bridge

# LAST LINE -- DO NOT REMOVE

We will add the maclist option to vmbr0 in the dmz zone, ending up with:

...
dmz     vmbr0           detect          routeback,bridge,maclist

# LAST LINE -- DO NOT REMOVE

Defining MAC / IP relation

Now we need to obtain the MAC address of the veth interface used in the virtual server and there are several methods for this. We can obtain the MAC address from within the container by issuing the command:

ifconfig | grep eth

which produces an output similar to:

eth0      Link encap:Ethernet  HWaddr 00:18:51:f9:43:1e

Alternatively, we can issue the following command from the host node (Proxmox server):

cat /etc/vz/conf/<VEID>.conf | perl -lne 'print for /mac=(.*),host_ifn/g'

Or in case it is a fully virtualized server (KVM):

cat /etc/qemu-server/<VEID>.conf | perl -lne 'print for /virtio=(.*)$/g'

where <VEID> is the number of the virtual server (container) for which you wish to obtain the MAC address. This will produce an output similar to:

00:18:51:F9:43:1E

Once we have obtained the MAC address, we create (or edit) the /etc/shorewall/maclist file as following:

# DISPOSITION   INTERFACE       MAC                     IP
ACCEPT          vmbr0           00:18:51:f9:43:1e       10.0.1.101
# LAST LINE -- DO NOT REMOVE

In this example, the MAC address has been provided in our earlier examples and the IP address 10.0.1.101 is the only permitted IP address that can be used by this virtual server.

Apply the changes by issuing the command

shorewall restart

At this point, if the virtual server changed the IP address (or added another IP address) other than 10.0.1.101, that traffic will be rejected as per the default behaviour of a Shorewall installation on Proxmox/Debian. If this behaviour needs to be changed, it can be set in the /etc/shorewall/shorewall.conf file with this variable:

MACLIST_DISPOSITION=REJECT

Additonal Notes

This guide is only intended to provide basic protection for your virtual servers and the Proxmox host node. No guarantees or warranties are implied, and you should always remain vigilant against potential network intrusions (in other words: do not rely on a firewall alone).

You may also wish to configure Shorewall according to your particular needs. For example, in this guide virtual servers are permitted to connect to each other within the dmz zone and could pose a risk.  You may wish to shield the virtual servers from each other in case one of them has been compromised (hint: edit the policy).

The Shorewall website includes numerous examples in its documentation that may help you further.

48 comments

  1. posted on Jul 12, 2014 at 12:34 PM  |  reply

    Thanks for the best how-to ever. Proxmox is a great product but without right network config and firewall it’s just not usable on public sites. Raw iptables with 10-20 VM per host is not a fun. So Proxmox + Shorewall is the right mixture!

  2. Jon
    posted on Jun 27, 2014 at 10:11 PM  |  reply

    Hi,

    Fantastic guide, thank you. I followed it to the letter, as far as Extra Hardening, but can not ping the dmz from the fw.

    Can you help at all?

    Thanks,
    Jon

  3. ltdata
    posted on Jun 22, 2014 at 10:30 PM  |  reply

    Hi Myatu,
    I tried Your detailed setup guide on a server with Proxmox 3.2, but I can’t get a KVM (Win7) to work.
    A thing that isn’t clear to me:
    How to setup the network interface for the KVM?
    Should it be “bridged to vmbr0” or a “NAT-interface”? I think “bridged to vmbr0” should be right…

    The container vm (Debian) is working well (no need to configure the /etc/network/interfaces inside the vm), but the KVM isn’t working.
    Is Proxmox 3.2 the problem? Something changed in the “network Modell”?

    many thanks in advance

  4. posted on Jan 09, 2014 at 6:20 PM  |  reply

    Hi.
    Thank U for a great work. I’ve passed the way on proxmox 2.3 – no problems at all ;) Do you mind if I use this tutorial and translate it on my blog? (polish language) Of course i’ll point the source ;) All the best!

    • Myatu
      posted on Jan 09, 2014 at 8:11 PM  |  reply

      Sure, no problem! Glad you found it useful.

  5. posted on Jul 21, 2013 at 8:47 AM  |  reply

    This tutorial SAVED me lots of hours and headache!
    I WANT TO DONATE TO YOUR WORK ! Paypal?
    I suggest everyone using this tutorial also donate, it ROCKS!

  6. posted on Jun 14, 2013 at 7:16 PM  |  reply

    I’m having some difficulty getting my Proxmox server at OVH configured correctly.
    I performed the actions under heading “Updating the eth0 network interface” ensuring that all of the variables are correctly set, however, upon restart networking fails to come up. Is there anything that has perhaps changed between when you wrote this article and the bridged networking setup that OVH uses today?

    Solid article, much appreciated for the effort put in to write it.

  7. Aldemar
    posted on Apr 04, 2013 at 9:33 PM  |  reply

    EL ejercicio es interesante pero cuando se tiene dos tarjetas de red y deseo que la dirección publica vaya directo a una maquina virtual de PROXMOX Para que distribuya todo se puede hacer?

  8. posted on Mar 10, 2013 at 6:12 PM  |  reply

    […] La VM100 est impérativement une VM utilisant la technologie KVM. Pour rappel, PROXMOX propose des VM KVM et OpenVZ. KVM supporte un éventail de distribution plus large et simplifie grandement le travail de gestion de la partie réseau que nous allons utiliser dans cet article. KVM nécessite un processeur intégrant les options de virtualisation. Si vous souhaitez utiliser exclusivement OpenVZ, je vous invite à lire cet article. […]

  9. posted on Sep 24, 2012 at 8:47 AM  |  reply

    […] La VM100 est impérativement une VM utilisant la technologie KVM. Pour rappel, PROXMOX propose des VM KVM et OpenVZ. KVM supporte un éventail de distribution plus large et simplifie grandement le travail de gestion de la partie réseau que nous allons utiliser dans cet article. KVM nécessite un processeur intégrant les options de virtualisation. Si vous souhaitez utiliser exclusivement OpenVZ, je vous invite à lire cet article. […]

  10. posted on Jul 02, 2012 at 5:20 PM  |  reply

    I really like this guide. I even use it as part of my documentation. Buy the way, you’re new site looks awesome!

    • posted on Jul 02, 2012 at 6:11 PM  |  reply

      Thanks! :)

      • Mou
        posted on Jan 23, 2013 at 9:20 PM  |  reply

        Hi , nice tutorial . I am just wondering will that work with multiple real IP.s .
        I have three real ip’s assigned to 3 vm’s none of the is reachable . Any suggestions ?

        Thanks

  11. posted on May 31, 2012 at 4:52 PM  |  reply

    […] The vmbr0 interface will be used later for VMs that need two interfaces but that don’t need or want to be bridged on top of the internet-facing eth0. The idea for this ‘floating’ interface came from myatus.com. […]

  12. posted on Dec 20, 2011 at 12:55 AM  |  reply

    […] Linux (part #1) and (part #2) Myatu’s blog (part #1) and (part […]

  13. posted on Jul 10, 2011 at 1:21 PM  |  reply

    […] Firewall Setup: http://www.myatus.co.uk/2009/08/31/guide-firewall-and-router-with-proxmox/ […]

  14. Simon
    posted on Oct 14, 2010 at 12:56 PM  |  reply

    isn’t it a bit scary to add SID to the sources and not remove it after installing shorewall6 ? one accidental apt-get upgrade and a potential reinstall will be required when it all breaks horribly ?

  15. posted on Aug 16, 2010 at 10:33 PM  |  reply

    […] waren folgende Links sehr hilfreich:Installguide Proxmox on software RAID (Hetzner EQ6-Server)Guide: Firewall and router with ProxmoxGuide: Firewall and router with Proxmox – extending its useHetzner WikiFazitIm Vergleich zu meiner […]

  16. Peter Host
    posted on May 02, 2010 at 5:03 PM  |  reply

    Hi Myatu.
    Just wanted to let you know that this tutorial saved my knees on my new OVH servers. After exploring the web and reading tons of Proxmox/Firewall tutorials, I still couldn’t get my /etc/network/interfaces properly setup (pbs with the default OVH config, very different from the default Proxmox config). After getting it straight, all rolled on.

  17. Anonymous
    posted on Apr 29, 2010 at 6:31 AM  |  reply

    נהדר!


    שרת וירטואלי

  18. Peter Dauth
    posted on Mar 31, 2010 at 11:18 AM  |  reply

    seems very nice, but I’m not at OVH and have no idea, what /etc/pve/kvm-networking.sh does. It’s obviously well covert. Help would be appreciated. Thanks in advance Peter

  19. posted on Mar 20, 2010 at 1:32 PM  |  reply

    […] Filed Under: Computing, Linux Mar.20, 2010 GD Star Ratingloading…Last year I wrote a guide on how to use Shorewall as a firewall and router for Proxmox. As a follow up I will answer a few questions I’ve received about that guide that can help […]

  20. Anonymous
    posted on Mar 20, 2010 at 6:05 AM  |  reply

    Hi Myatu,
    Thank you for this perfect tutorial. Butt I have one more question. I’m at OVH and have two public ip’s. I have generated a mac address for my fail-over ip. I don’t understand how I could bridge this one with vmbr0. what I tried is to make a KVM use vmbr0 with given mac address and put my public fail-over ip. butt thats not working. Can you give me a hint where i should change my interface file. here is my current interfaces:
    Thank you!
    auto eth0
    iface eth0 inet static
    address 94.23.200.100 <–fake
    netmask 255.255.255.0
    gateway 94.23.200.254

    auto eth0:0
    iface eth0:0 inet static
    address 94.23.188.76 <–fake
    netmask 255.255.255.0

    auto vmbr0
    iface vmbr0 inet static
    address 10.10.10.254
    netmask 255.255.255.0
    broadcast 10.10.10.255
    bridge_ports none
    bridge_stp off
    bridge_fd 0

    • posted on Mar 20, 2010 at 1:38 PM  |  reply

      I’ve added an extension to this guide at http://wp.me/pCjzp-89 – you actually have the option of using the regular port forwarding, Proxy ARP of the “Mixed use” as explained in that article. If its not clear or you need additional assistance, let me know.

    • S Diot
      posted on Nov 10, 2010 at 9:11 PM  |  reply

      Hi there! Trying this on Hetzner I got error messages, something like:
      vmbr0: ERROR while getting interface flags: No such device

      Someone there said that I just needed:
      pre-up brctl addbr vmbr0
      in /etc/network/interfaces. I did, and then it worked.

  21. Anonymous
    posted on Mar 20, 2010 at 6:02 AM  |  reply

    double post, sorry

  22. xen
    posted on Nov 09, 2009 at 10:41 PM  |  reply

    Great tutorial is working for me now. I needed/used this to give my container VMs access to the internet.

    One problem I had tho is this line in the /etc/shorewall/rules file
    ACCEPT net fw tcp 443,5900-5999
    shorewall does not like the “-” for port ranges.
    I had to use a “:” like this. 5900:5999

  23. Colin
    posted on Oct 08, 2009 at 4:18 PM  |  reply

    Me again :)

    I decided to forgo clustering so I followed your tutorial to the letter and now *most* things work but two weird things are happening – both on the KVM windows box (which has an internal IP):

    – on the windows box: if I ssh to the other containers it works but if I browse to the other containers 80/443 then I actually get the host (i.e. proxmox)!!!
    – windows update instantly fails with 8024402C. I have disabled the windows firewall, enabled all tcp/udp ports (i.e. the ‘-‘ character’) and also tried adding an explicit map from the internal IP to the external IP in masq. All to no avail.

    The reason for giving windows an internal IP is it needs to communicate with the other internal containers (i.e. to do ldap authentication on the private ldap server).

    Any ideas? I really hope you can help me on this – this is the last missing piece for me :)

    • posted on Oct 08, 2009 at 8:45 PM  |  reply

      As for the first issue, you either need to use the container’s IP
      address directly, or have a split DNS as described in
      http://www.shorewall.net/FAQ.htm#faq2.

      For the second issue,
      http://windows.microsoft.com/en-us/windows-vista/Windows-Update-error-8024402C
      might give some more insight into this. Make sure that the Windows
      Update Sites are reachable from within your Windows server.

      For debugging purposes you can use “tcpdump -i (interface)” on the
      Proxmox host, where (interface) is the actual network inteface used by
      the Windows container (ie, veth101.0). For more information on how to
      use tcpdump, use “man tcpdump”.

      Feel free to use the “Contact Me” page to contact me directly if you
      need additional assistance.

  24. Colin
    posted on Oct 08, 2009 at 11:22 AM  |  reply

    Help :) OpenVZ containers work fine, but I cannot get KVM to work at all using nat or vmbr0.

    Rather than repeat here, I have posted here: http://proxmox.com/forum/showthread.php?t=2295. Any help is *gratefully* received.

    I am sure I have made a muppet mistake, I just can’t see it.

  25. Name
    posted on Oct 01, 2009 at 4:23 PM  |  reply

    One minor note – this broke my proxmox cluster.

    Every time I reboot the machine it mangles /etc/hosts so that the hostname resolves to 10.254.254.254.

    Any ideas?

    • posted on Oct 01, 2009 at 8:27 PM  |  reply

      Yes, I’ve noticed this too. If you’re not using a cluster, you could simply assign a arbitrary hostname in the Proxmox Admin (Configuration Section -> System -> DNS). For example “pvelocalhost”.In a cluster, you are best to assign a different IP for each of the “vmbr0” interfaces (ie., one machine is 10.254.254.254, another is 10.254.254.253) and assign a private address (ie., “pve1.internal.mydomain.com”) that can be resolved using a private DNS server (to which both machines have access).I’ve written this so that both OpenVZ and fully virtualized KVMs can work with the same firewall configuration. If you are not expecting to use fully virtualized KVMs or the “veth” interface, then you can basically keep the “vmbr0” interface as-is. I’ll expand on this in another article.

      • Name
        posted on Oct 01, 2009 at 10:14 PM  |  reply

        How would this work though as each machine only has one nic (think MG servers from ovh ;)) which has a public IP so they wouldn’t be able to communicate on the 10…., unless I am missing something?

        Also, a quick clarification – if I leave vmbr0 as is (i.e. with a public IP) then which of the following will work a) public IP mapped to host, KVM has private IP using shorewall to port forward, b) public IP mapped to KVM guest, c) – both :)

        Thanks, as always

        • posted on Oct 01, 2009 at 10:46 PM  |  reply

          Sorry, I jumped the gun on that one ;-) It would require a VLAN setup
          between the servers (you could even use a VPN). I’m not quite sure what
          part of the PVE changes the /etc/hosts according to the vmbr0 IP (which
          is Proxmox’s default setup), and I’ll have to dig through that a bit…
          but as a temporary solution you may also re-assign the main IP to the
          hostname using a script. Not elegant, but should be OK.

          As for leaving vmbr0 as is, it would be both. But you wouldn’t be able
          to use it with the veth interfaces (so only venet with OpenVZ).

  26. posted on Oct 01, 2009 at 7:46 AM  |  reply

    VNC console uses a port range from 5900-5999 (not only port 5900)

    • posted on Oct 01, 2009 at 8:11 PM  |  reply

      Thanks for the pointer, Dietmar. I’ve updated the guide to reflect this.

  27. Name
    posted on Sep 30, 2009 at 8:07 PM  |  reply

    Many many thanks. This saved my bacon and answered part of http://proxmox.com/forum/showthread.php?t=2189.

  28. Anonymous
    posted on Sep 09, 2009 at 11:37 AM  |  reply

    Hi I’ve followed your guide twice now, both on OVH’s servers. The first time it worked perfectly, thank you, but the second time, I can’t contact the failover IP’d VM from the net, nor can it get out for to an external IP, but i can from the proxmox hn, i guess from fw. I’ve checked so many times and the output of all the following is the same (except different ip/failover ip, it’s even the same vm dumped and copied over) i’ve checked these files and commands:
    hn:
    ipconfig
    route
    /etc/default/shorewall
    /etc/shorewall/interfaces
    /etc/shorewall/masq
    /etc/shorewall/rules
    /etc/shorewall/zones
    /etc/network/interfaces
    /etc/shorewall/shorewall.conf

    node:
    ipconfig
    route -n

    I have also confirmed with ovh that the failover IP is fine, and this was confirmed by me checking the shorewall logs, and trying to connect to a disallowed port, and came up with a drop. however, ping, allowed port 22 , and allowed port 80 are ignored with as i say no networking.

    Is there another file i should be checking?

    thanks for the howto

    • posted on Sep 09, 2009 at 2:47 PM  |  reply

      When you assign a failover IP directly to the VM, it should not be defined on the host node as well. That is, if you have added the failover IP under “eth0:0” on the host node as per OVH’s instructions, then you need to remove this first. You assign the failover IP directly to the VM using Promox’s web admin (or the vzctl command). If you are using a full KVM, then this will not work without additional work (see Shorewall’s Proxy ARP information, for instance, or setup manual routes).

      • Anonymous
        posted on Sep 09, 2009 at 8:35 PM  |  reply

        HiIt’s not defined on host node. for either. (working or non working) Here’s ifconfig for the working one: Link encap:Ethernet HWaddr 00:ff:ff:ff:ff:ff inet addr:94.23.212.XXX Bcast:94.23.212.255 Mask:255.255.255.0 inet6 addr: fe80::21c:c0ff:fedb:c484/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:382997 errors:0 dropped:0 overruns:0 frame:0 TX packets:538619 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:28082565 (26.7 MiB) TX bytes:730734096 (696.8 MiB) Interrupt:252 Base address:0x4000lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:2058 errors:0 dropped:0 overruns:0 frame:0 TX packets:2058 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:354831 (346.5 KiB) TX bytes:354831 (346.5 KiB)venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1 RX packets:995 errors:0 dropped:0 overruns:0 frame:0 TX packets:903 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:132514 (129.4 KiB) TX bytes:65839 (64.2 KiB)vmbr0 Link encap:Ethernet HWaddr b6:2e:45:a1:76:e2 inet addr:10.0.10.254 Bcast:10.255.255.0 Mask:255.0.0.0 inet6 addr: fe80::b42e:45ff:fea1:76e2/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:364 (364.0 B)and the nonworking one: eth0 Link encap:Ethernet HWaddr 00:ff:ff:ff:ff:ff inet addr:94.23.219.XXX Bcast:94.23.219.255 Mask:255.255.255.0 inet6 addr: fe80::21c:c0ff:feeb:a2bb/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:331739 errors:0 dropped:0 overruns:0 frame:0 TX packets:367540 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:48872594 (46.6 MiB) TX bytes:106160569 (101.2 MiB) Interrupt:252 Base address:0x4000lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:31776 errors:0 dropped:0 overruns:0 frame:0 TX packets:31776 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:12893605 (12.2 MiB) TX bytes:12893605 (12.2 MiB)venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1 RX packets:846 errors:0 dropped:0 overruns:0 frame:0 TX packets:669 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:57518 (56.1 KiB) TX bytes:45344 (44.2 KiB)vmbr0 Link encap:Ethernet HWaddr 5a:d9:6f:f3:7c:fe inet addr:10.0.10.254 Bcast:10.255.255.0 Mask:255.0.0.0 inet6 addr: fe80::58d9:6fff:fef3:7cfe/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:252 (252.0 B)I’m not using KVM but just openVZ vmsAs I say it looks to be configured exactly the same to me on both VMs, following your guide, but I can’t get routing outside of the vm to the internet or from the internet to the vm. However, I can ping and ssh from host node into VM and can ping host node from vm.It’s very confusing.

        • posted on Sep 10, 2009 at 3:00 PM  |  reply

          Right, the host node appears to be fine. Did you remember that for a
          private IP address you would use “DNAT” and for a failover (public) IP
          address “ACCEPT”? For example, if you failover IP is 91.2.3.4 and you
          wish to permit access to port 80 on this IP, the rule will be:

          ACCEPT net dmz:91.2.3.4 tcp 80

          If you have already done this, double check if a route exists to your VM
          with

          ip route list [failover IP]

          Where [failover IP] is the actual IP. It should give you an output
          similar to:

          91.2.3.4 dev venet0 scope link

          On OVH systems there shouldn’t be any additonal configuration required
          with gateways, etc. Let me know if you’re still running into issues
          after check this – you can use the contact form to get hold of me
          directly, and will probably make communication a little easier.

  29. Anonymous
    posted on Sep 02, 2009 at 1:54 PM  |  reply

    I know i’m a kind of a newby regarding the firewall on the proxmox host itself,
    but if i reconfigure my vmbr0 to a private address, where where does the
    public address of the proxmox host go. As far as i can see you cannot assign a static address to the eth0?

    • posted on Sep 02, 2009 at 3:06 PM  |  reply

      The public address of the Proxmox host node should already have been configured, either by you or the administrator responsible for the server (ie., the hosting provider) — this includes servers that use DHCP to obtain their IP address. This is defined under “eth0” and should remain untouched. You can verify this by typing “ifconfig” on your Proxmox host node to see the current network interfaces and their assigned IP addresses.

      For a virtual server (guest container) you can assign the IP address through the Proxmox web admin, unless it uses a bridged network device (as would be the case in fully virtualized KVMs). In the latter case, you will have to set the IP for “eth0” **inside the guest container** (Never change “eth0” on the Proxmox host node itself, unless you are absolutely 110% sure you know what you are doing).

      If you need more help or further clarifications, feel free to contact me.

      • Kcho
        posted on Dec 19, 2012 at 9:27 PM  |  reply

        Hi! Is there any way for the guest to obtain its IP address from an outside DHCP server? Lets says from the firewall (91.11.22.33), for example, in the picture. And not writing the non-private IP in the guest interface file?

        Thanks!

    • posted on Sep 02, 2009 at 3:16 PM  |  reply

      The public address of the Proxmox host node should already have been
      configured, either by you or the administrator responsible for the
      server (ie., the hosting provider) — this includes servers that use
      DHCP to obtain their IP address. This is defined under “eth0” and should
      remain untouched. You can verify this by typing “ifconfig” on your
      Proxmox host node to see the current network interfaces and their
      assigned IP addresses.

      For a virtual server (guest container) the easiest way to assign the IP
      address is through the Proxmox web admin. However, this option is not
      available if the guest uses a bridged network device (as would be the
      case in fully virtualized KVMs). In that case you will have to assign
      the IP for “eth0” **inside the guest container** (Never change “eth0” on
      the Proxmox host node itself, unless you are absolutely 110% sure you
      know what you are doing).

      If you need more help or further clarifications, feel free to contact me.

      • Anonymous
        posted on Sep 03, 2009 at 6:47 AM  |  reply

        Thanks for you’re reply,

        but if you have done a default proxmox install, the vmbr0 is the main public ip adress of the host and eth0 has no ip address. So when you read you’re howto
        you miss something. You have to alter the eth0 also.

        • posted on Sep 03, 2009 at 10:25 AM  |  reply

          I’ll correct this then, as it appears to vary depending on hosting provider’s setup scripts. Thanks for the heads up.

        • posted on Sep 07, 2009 at 3:51 PM  |  reply

          The post has been updated.

Join the discussion

Your email address will not be published. Required fields are marked *