Guide: Firewall and router with Proxmox – Extending its use

on March 20, 2010 in Linux with 13 comments by

Last year I wrote a guide on how to use Shorewall as a firewall and router for Proxmox. As a follow up I will answer a few questions I’ve received about that guide that can help you extend its use.

Proxy ARP

The most common question is in regards to proxy ARP. Enabling this option will allow you to assign a public IP directly to your guest VM, eliminating the need for port forwarding (DNAT) or having to worry about the MAC address.

As an example use for proxy ARP, it is helpful for those using a a SIP-based VoIP server since a STUN server is no longer required.

Enabling Proxy ARP

The first step is to ensure that Proxy ARP is enabled.  This is a fairly simple task and involves adding an single line to one of your static network stanzas. Which one precisely depends on your system setup; for those who have an eth0 stanza, you can use it there. For others who only have a vmbr0..n stanza, the additional line should be placed there.

iface eth0 inet static
    # ... existing lines ...
    post-up echo 1 > /proc/sys/net/ipv4/conf/all/proxy_arp

These changes will take effect on your next boot or whenever you restart your networking services.

Add a route to Shorewall

Shorewall needs to know that you’d like to use proxy ARP, for which IP that is and where this IP needs to be routed to. The beauty of Shorewall is its simplicity and so this can be quickly done by creating a file named /etc/shorewall/proxyarp that will contain this:

#ADDRESS    INTERFACE    EXTERNAL    HAVEROUTE    PERSISTENT
92.22.33.44 vmbr0        eth0

Translated, this informs Shorewall that the public IP 92.22.33.44 attached to eth0 needs to be forwarded to a guest VM (using this public IP) attached to the vmbr0 bridge.

As you can tell from the first line, there are two additional options: haveroute and persistent.

The haveroute option determines whether Shorewall should create a route from the external interface to the bridge. For guest containers based on OpenVZ, Proxmox will take of creating the route. But for fully virtualised containers (based on KVM), you need to create this route yourself. In this case we tell Shorewall to create the route for us, by keep this option’s value at its default value (blank, or No).

When the haveroute option is set to No (default), the persistent option tells Shorewall if it should keep the created route active if Shorewall is stopped. Generally, and for security reasons, you should leave this at its default option (blank or No). This prevents the guest VM from being exposed without a firewall protecting it.

Creating Shorewall rules

In the original guide all traffic from the public side to internal VMs is blocked. This continues to be the same when you are using proxy ARP and thus need to create rules that permit traffic to certain ports. This can be done by adding rules like:

ACCEPT    net    dmz:92.22.33.44    udp    5060

This will permit UDP traffic on port 5060 to proceed to the guest VM on IP 92.22.33.44. You can also use the Shorewall macros, for example:

HTTP/ACCEPT    net     dmz:92.22.33.44

And in this case it will make a web server on 92.22.33.44 accessible to the public.

Alternatively, you may setup a single rule that accepts all traffic on all ports:

ACCEPT    net     dmz:92.22.33.44

However you should take care that the guest VM has its own set of rules to block unwanted or unsafe traffic.

The main thing to take care of with these proxy ARP firewall rules is to use “ACCEPT” and not “DNAT” as explained in the original guide.

Multiple public IPs

Another common question is in regards to multiple public IP addresses. The original guide assumed that the host has one public IP address, so here are a few additional pointers.

If you are following the original guide, then any public IP will be applied the same rule. So let’s say you have two public IP addresses, 94.11.22.33 and 94.22.33.44 and the following rule:

ACCEPT    net    fw    tcp    5900:5999

With this rule, you can use both 93.11.22.33:5900 or 94.22.33.44:5900 to connect to Proxmox’s VNC. But you wish to restrict it to a specific IP address, then you need to modify the rule as following:

ACCEPT    net   fw:94.22.33.44    tcp    5900:5999

Now you can only use 94.22.33.44:5900 to connect, but not 93.11.22.33:5900.

The same applies to port forwarding (DNAT) rules, which is written in a slightly different format. Let’s say we have this rule:

DNAT    net    dmz:10.0.0.1    tcp    1234

This will forward any public IP to port 1234 on the guest VM at 10.0.0.1. So both 94.11.22.33:1234 and 94.22.33.44:1234 will work. If you wish to restrict this to a certain public IP address, the rule needs to be modified as following:

DNAT    net    dmz:10.0.0.1    tcp    1234    -    94.22.33.44

Only 94.22.33.44:1234 will be allowed at this point.

Memorable names

If you have many IP addresses, it becomes easy to forget which IP to use. You can use Shorewall’s “params” file to give IPs a memorable name. Edit the file /etc/shorewall/params as following:

IP_ALEX=94.11.22.33

IP_SONIA=94.22.33.44
IP_ERIC=94.33.44.55
IP_VM1=10.0.0.1
IP_VM2=10.0.0.2
IP_VM3=10.0.0.3

Now you can use these names instead of the IP address in any of your Shorwall rules, like so:

HTTP/DNAT    net     dmz:$IP_VM1    -    -    -    $IP_SONIA

And this would forward HTTP traffic from 94.22.33.44 to a guest VM running on 10.0.0.1.

Mixed use and bridging

The original guide not only adds a firewall, but also helped those who’s hosting provided blocked IPs on unauthorized MACs. One such hosting provided was OVH, and they have recently introduced a “Virtual MACs for VPS” option that allows you to assign an IP address to a MAC. This eliminates the need for Proxy ARP or port forwarding/NAT, however the original guide will still prove itself useful for protecting the host.

I will provide a sample setup below, which can be modified to suit your needs and network configuration. This assumes you have have a basic understanding of networking and read my previous guide.

Interfaces

We modify the /etc/network/interfaces as following:

# The loopback network interface
auto lo
iface lo inet loopback

# Public Network. Make sure to only use MACs that were assigned to you
auto vmbr0
iface vmbr0 inet static
 # The following settings are specific to your hosting provider:
 address 94.11.22.33
 netmask 255.255.255.0
 network 94.11.22.0
 broadcast 94.11.22.255
 gateway 94.11.22.254
 # The following assumes eth0 is the public-side NIC, the remained is always the same
 bridge_ports eth0
 bridge_stp off
 bridge_fd

# Optional Private Network. This network cannot be access directly from the public side
auto vmbr1
iface vmbr1 inet auto
 address 10.0.0.1
 network 255.0.0.0
 broadcast 10.255.255.25
 bridge_ports none
 bridge_stp off
 bridge_fd 0

The host, accessible by 94.11.22.33, will have two bridges at this point: vmbr0 serving the public-side and vmbr1 serving inter-VM communication.

To assign a public IP address to a KVM, you use the vmbr0 bride and must ensure the MAC corresponds to the one provided by your hosting provider (if a MAC restriction is in place).

You can also add a 2nd NIC (or as the only NIC) to the KVM, which is bridged with vmbr1. In this case, you need to use an IP within the 10.0.0.2-10.255.255.254 range, and as gateway 10.0.0.1. This particular IP range can only be used between other VMs on the same host / bridge, unless you use port forwarding or a VPN on the same IP range.

Shorewall Interfaces and Zones

The basic /etc/shorewall/interfaces will be:

#ZONE    INTERFACE    BROADCAST    OPTIONS
pub      vmbr0        detect       routeback,bridge
loc      vmbr1        detect       routeback,bridge

#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE

You could also add the blacklist option, or any of the other possible Shorewall options if you wish.

The accompanying /etc/shorewall/zones will look like this:

#ZONE    TYPE        OPTIONS        IN            OUT
#                                   OPTIONS       OPTIONS
fw       firewall
pub      ipv4
loc      ipv4

#LAST LINE - ADD YOUR ENTRIES ABOVE THIS ONE - DO NOT REMOVE

You now have three distinct Shorewall zones. The fw zone is a self-reference to the host (in our sample setup 94.11.22.33 and 10.0.0.1). The pub zone represents the publicly accessible vmbr0 bridge and loc our internal vmbr1 bridge.

Shorewall Policy

The following policy defines these basic rules:

  • Traffic from the the host anywhere else is permitted
  • Traffic from the public side to the host and the internal network is denied
  • Traffic from the internal side to the host is denied, anywhere else is permitted

Edit /etc/shorewall/policy:

#SOURCE    DEST    POLICY     LOG      LIMIT:       CONNLIMIT:
#                             LEVEL    BURST        MASK

# From Firewall:
fw         fw     ACCEPT
fw         pub    ACCEPT
fw         loc    ACCEPT

# Public Bridge (read the policy warnings!):
pub        pub    ACCEPT
pub        loc    ACCEPT
pub        fw     DROP        info

# Local (internal) Bridge:
loc        loc    ACCEPT
loc        pub    ACCEPT
loc        fw     DROP        info

# THE FOLLOWING POLICY MUST BE LAST
#
all        all    REJECT      info

#LAST LINE -- DO NOT REMOVE

Policy Warnings

With this sample policy it means that each publicly accessible guest VM should have its own firewall. If you wish to change this behavior, and let Shorewall handle the firewall for each such guest VM, then change the Public Bridge section:

pub        pub    DROP      info

You then need to create specific rules that allow traffic to VMs on vmbr0.

A similar warning applies to traffic from vmbr0 to vmbr1. The policy assumes that vmbr0 does not receive any routable traffic on a private IP range (also called “martians”). Although this is often the case, it depends on the hosting provider’s internal networking. If you are not sure whether there’s routable traffic on a private IP range from the public side, you have two options. The first is to disallow all traffic from vmbr0 (pub) to vmbr1 (loc) by editing the Public Bridge section:

pub        loc    DROP      info

The alternative is to have a set of rules in /etc/shorewall/rules like so:

# ...
SECTION NEW

# Leave these at the top, right after "SECTION NEW"!
DROP        pub:10.0.0.0/8        all
DROP        pub:192.168.0.0/16    all
DROP        pub:172.168.0.0/12    all

# ... Other rules follow ...

#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE

WARNING! In all cases, you should verify whether traffic is truly blocked and never assume that it is!

Shorewall and Fail2ban

Fail2ban in its own words “scans log files like /var/log/pwdfail or /var/log/apache/error_log and bans IP that makes too many password failures. It updates firewall rules to reject the IP address”. This can for instance be used to (temporarily) ban a bot that is attempting a brute-force entry through SSH.

Fail2ban is an ideal companion to Shorewall and can be installed in a matter of minutes on a Proxmox host. You start by installing Fail2ban from the Debian packages:

apt-get install fail2ban

Configure Shorewall

Next you need to edit one line in the Shorewall configuration file, located at /etc/shorewall/shorewall.conf:

BLACKLISTNEWONLY=No

That’s all you need to configure in Shorewall. Remember to apply your settings by restarting Shorewall with the command “shorewall restart”.

Configure Fail2ban

The last step is to configure Fail2ban. The file /etc/fail2ban/jail.conf is extensively documented and of particular interest are the following settings:

...
#
# Destination email address used solely for the interpolations in
# jail.{conf,local} configuration files.
destemail = [email protected]

...
#
# ACTIONS
#

# Default banning action (e.g. iptables, iptables-new,
# iptables-multiport, shorewall, etc) It is used to define
# action_* variables. Can be overriden globally or per
# section within jail.local file
banaction = shorewall

...
# Choose default action.  To change, just override value of 'action' with the
# interpolation to the chosen action shortcut (e.g.  action_mw, action_mwl, etc$
# globally (section [DEFAULT]) or per specific section
action = %(action_mwl)s

Inform fail2ban of these changes by issuing the following command:

fail2ban-client reload

The destemail variable should be changed to your own e-mail address, where you will be informed of any ban actions. The banaction variable specifies that Shorewall should be used to block possible intruders. And finally, the action variable tells fail2ban to ban any detected intruder and then send an you a detailed e-mail with the relevant log lines (that caused the ban).

Following is an example of an actual e-mail:

Hi,

The IP 193.86.5.103 has just been banned by Fail2Ban after
3 attempts against ssh.

Here are more information about 193.86.5.103:

% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: This output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '193.86.4.0 - 193.86.5.255'

inetnum:      193.86.4.0 - 193.86.5.255
netname:      BRANO
descr:        BRANO, Inc.
descr:        Hradec nad Moravici
country:      CZ
admin-c:      BK230-RIPE
tech-c:       TP231-RIPE
status:       ASSIGNED PA
mnt-by:       GTSCZ-MNT
source:       RIPE # Filtered

person:       Bohumil Kriz
address:      BRANO, Inc.
address:      Computer Centre
address:      Hradec nad Moravici
address:      747 41
address:      The Czech Republic
phone:        +420 653 918118
fax-no:       +420 653 911791
nic-hdl:      BK230-RIPE
source:       RIPE # Filtered

person:       Tomas Partl
address:      Brano, Inc.
address:      Computer Centre
address:      Hradec nad Moravici
address:      747 41
address:      The Czech Republic
phone:        +420 653 918371
fax-no:       +420 653 911791
nic-hdl:      TP231-RIPE
source:       RIPE # Filtered

% Information related to '193.86.0.0/16AS2819'

route:        193.86.0.0/16
descr:        CZNET-A
origin:       AS2819
mnt-by:       GTSCZ-A-MNT
source:       RIPE # Filtered

Lines containing IP:193.86.5.103 in /var/log/auth.log

Feb 27 03:32:02 host sshd[21460]: Failed password for root from 193.86.5.103 port 35833 ssh2
Feb 27 03:32:04 host sshd[21502]: Failed password for root from 193.86.5.103 port 36302 ssh2
Feb 27 03:32:06 host sshd[21504]: Failed password for root from 193.86.5.103 port 36719 ssh2

Regards,

Fail2Ban

By default fail2ban is configured to monitor SSH, which for a Proxmox host works without any additional changes. Personally I prefer a ban on 3 attempts instead of the default 6, so I have made this one change in the /etc/fail2ban/jail.conf file:

[ssh]
maxretry = 3

But again, the configuration file is quite well documented, so any personal preferences or modifications should be easy to accomplish.

 

Hi,

The IP 193.86.5.103 has just been banned by Fail2Ban after
3 attempts against ssh.

Here are more information about 193.86.5.103:

% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: This output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '193.86.4.0 - 193.86.5.255'

inetnum:      193.86.4.0 - 193.86.5.255
netname:      BRANO
descr:        BRANO, Inc.
descr:        Hradec nad Moravici
country:      CZ
admin-c:      BK230-RIPE
tech-c:       TP231-RIPE
status:       ASSIGNED PA
mnt-by:       GTSCZ-MNT
source:       RIPE # Filtered

person:       Bohumil Kriz
address:      BRANO, Inc.
address:      Computer Centre
address:      Hradec nad Moravici
address:      747 41
address:      The Czech Republic
phone:        +420 653 918118
fax-no:       +420 653 911791
nic-hdl:      BK230-RIPE
source:       RIPE # Filtered

person:       Tomas Partl
address:      Brano, Inc.
address:      Computer Centre
address:      Hradec nad Moravici
address:      747 41
address:      The Czech Republic
phone:        +420 653 918371
fax-no:       +420 653 911791
nic-hdl:      TP231-RIPE
source:       RIPE # Filtered

% Information related to '193.86.0.0/16AS2819'

route:        193.86.0.0/16
descr:        CZNET-A
origin:       AS2819
mnt-by:       GTSCZ-A-MNT
source:       RIPE # Filtered

Lines containing IP:193.86.5.103 in /var/log/auth.log

Feb 27 03:32:02 host sshd[21460]: Failed password for root from 193.86.5.103 port 35833 ssh2
Feb 27 03:32:04 host sshd[21502]: Failed password for root from 193.86.5.103 port 36302 ssh2
Feb 27 03:32:06 host sshd[21504]: Failed password for root from 193.86.5.103 port 36719 ssh2

Regards,

Fail2Ban

13 comments

  1. Mikael
    posted on Oct 29, 2013 at 8:02 AM  |  reply

    Is it possible to setup a veth guest that uses a public ip instead of the internal network IP?

  2. Bruno
    posted on Apr 18, 2013 at 6:20 PM  |  reply

    Myatu, in the “Mixed use and bridging” section the “pub pub DROP info” rule does not drop traffic to any IP on the ‘pub’ zone on vmbr0. For instance a public /27 on vmbr0 means all IPs are accessible from the net if in a KVM, even with the “pub pub DROP ” rule.
    Any opinion on the best way to implement a workin pub2pub rule on vmbr0?

    • posted on Apr 28, 2013 at 2:17 PM  |  reply

      The post is getting old, so there’s quite a good chance something is outdated due to recent changes. I will need to look into this.

      • Mikael
        posted on Oct 27, 2013 at 8:30 PM  |  reply

        Old, but still the best guide i’ve found on the net so far. You are very clear and explain well! Other sites just assume you know everything already…

  3. George
    posted on Nov 24, 2012 at 9:49 PM  |  reply

    Thanks for all your articles, helped me a lot.

    I have set up a VM with OVH IP Failover using bridged in Proxmox (configured virtual MAC in order to be able to do it under OVH infrastructure). The thing is working correctly as I can access the VM from the internet using the IP Failover.

    I have followed your article but Shorewall can not control the trafic between the VM’s connected to the vmbr1 interface. I mean for example:
    Virtual machine (10.0.0.2) connected to vmbr1 can ping to Virtual Machine (10.10.10.3) connected to vmbr1 even if I set the /etc/shorewall/policy with:
    pub pub DROP info

    However, what shorewall control is the traffic from the VM’s to the vmbr1 interfaces. I mean, when that policy is set up there is not ping from 10.0.0.2 or (10.0.0.3) to the 10.0.0.1 assigned to the vmbr1.

    Is it normal this behavior?

  4. Greg
    posted on Sep 09, 2011 at 9:21 AM  |  reply

    Thanks for the two shorewall tutos, it helped me a lot.

  5. posted on Jan 25, 2011 at 1:02 PM  |  reply

    The new Meraki cloud-managed MX50 and MX70 routers include application firewalls as well as traffic shaping features. As opposed to traditional router …

  6. posted on Aug 16, 2010 at 10:34 PM  |  reply

    […] Proxmox on software RAID (Hetzner EQ6-Server)Guide: Firewall and router with ProxmoxGuide: Firewall and router with Proxmox – extending its useHetzner WikiFazitIm Vergleich zu meiner alten Konfiguration mit XEN-Hosts bin ich mit Proxmox […]

  7. posted on Jul 17, 2010 at 11:53 AM  |  reply

    […] you are using Fail2Ban, like described in the Shorewall firewall configuration, you can create a new definition that scans for the w00tw00t entries in the webserver log […]

  8. Anonymous
    posted on Apr 28, 2010 at 4:40 PM  |  reply

    Thanks for these tutorials, i would like to adapt my Server to use Shorewall and fail2ban.
    But bevore i start over and change my running network settings i wonder what is in that kvm-networking.sh? it does not seem that my proxmox installation has this file. Is it necessary? I don´t want to lock myselve out :-P

    • posted on Apr 29, 2010 at 4:25 PM  |  reply

      I’ve removed the “kvm-networking.sh” portion, as it is not required. However, if you’ve already added this, it will safely skip it.

  9. posted on Mar 21, 2010 at 8:00 AM  |  reply

    […] Published Guide: Firewall and router with Proxmox – Extending its use. […]

  10. Anonymous
    posted on Mar 20, 2010 at 6:14 PM  |  reply

    Thank you thank you! This was the kind of example that I needed!
    Butt you forgot one thing, without the venet0 you can’t forward anything on the vz-machines. Thanks again!
    pub vmbr0 detect routeback,bridge
    loc vmbr1 detect routeback,bridge
    loc venet0 detect routeback

Join the discussion

Your email address will not be published. Required fields are marked *