Dual ISPs BGP – Palo Alto Networks



Network Topology


First things first! I passed the BGP Exam for the Nokia SRA Certification. I am now planning to deviate a bit and obtain my Sec+ and see where that takes me.. Anyways..

I’ve been very interested in Palo Alto Networks lately and I’m low-key starting to think about the certification path for PA.  I want to take some time and go over a Dual ISP connection utilizing a PA at the edge. I’m hoping to provide some insight from both a Service Provider  and Enterprise standpoint. The goal is to have a highly redundant WAN connection utilizing the PA.

Something I want to start keeping in mind:

64496 – 64511 16 bit Reserved for use in documenation & sample code. [RFC5398]


ISP 1 ( AS 64511 ) will be adveritising a default-route via interconnect with the PA on eth1/4.

ISP 2 ( AS 64496 ) will be adveritising a default-route via interconnect with the PA on eth1/1.

The Enterprise LAN will be peering with the PA via iBGP on Gi0/0 and eth1/7 on the PA from Autonomous System 64500


From ISP 1 – a VPRN (VRF) 100 is configured, advertising a default-route.

From ISP 2 – a VPRN (VRF) 200 is configured, advertising a default-route.

Here is a snippet from the Nokia VRF that’s providing internet service connection to the Palo Alto. A similar configuration exisist on the ISP 1 router.



From the Palo Alto – The initial steps to take are the following:

1. Create an “Untrust” zone. This zone will be facing the Internet (ISP1 & ISP2).

Normally, I would suggest micro-segmenting these zones, but this requires a bit more policy creation. Example would be, 1 zone for ISP 1 and a different zone for ISP 2 for an absolute zero-trust architecture.

2. Create a Management Profile which simply allows ICMP (pings) for troubleshooting and verification purposes.

Here is what the Layer 3 Interfaces look like:


We should have IP connectivity between our Palo-Alto and both of our ISP’s! We’re officially connected to the internet… sort of.

Now for the fun stuff, BGP connections!

Lets start with the Palo-Altos.

  1. Select the  “Virtual Routers’ setion under the Network tab.
  2. Select the “BGP” tab.
  3. ENABLE the BGP protocol by checking the box.
  4. Assign a Router ID. This can be one of the two IP’s on the interfaces facing our WAN services or a loopback (preffered).
  5. Input your local AS Number.
  6. Make sure to UN-CHECK “Reject Default Route”
    1. Both ISP’s will be advertising us Default-Routes. We’ll select one with BGP techniqures as a primary.
  7. Make sure to CHECK “Install Route”
    1. This is necessary if we want to install routes from BGP / Local FIB into the Global Routing Table on the Palo Alto.
  8. Depending on what model Palo-Alto you have, I would suggest creating a BFD profile and enabling this on your WAN connection for a fast-fail over detection to minimize downtime for your internal users.
    1. To create a BFD Profile:
      1. Network > Network Profiles > BFD Profile.
  9. This should be enough for the “General” Tab.

let’s move over to the “Peer Group”

  1. Add a new Peer Group, lets call this ISP 1 – Re-create the steps for ISP 2.
    1. Name: ISP 1
    2. Type: EBGP
  2. Add a new peer.
    1. Name: WAN-ISP-1
    2. Peer-AS: 64511
    3. Select the appropriate Interface / IP Address
    4. Input the appropriate /31 peer IP of the WAN connection.
    5. Under Advanced, make sure the Inherit Protocol’s Global BFD Porifle is selected.
    6. Select OK and commit.

Here is what the BGP Peer Group section should look like at this point:


Now, verify our BFD sessions..


All looks good!  Lets verify we’re seeing a default-route from both peers:


From the Local-RIB (And the Route Table) under the “More Runtime Stats” we are installing the default-route from our peer at ISP 1 –

What if that peer is a 1G connection, but our Peer at ISP 2 should be our Primary WAN interface, as it’s a 10G interface? Let’s play with BGP now.

First, lets make sure all our outgoing traffic is going out or preffered exit path ( ISP 2) – let’s change our Local Pref on routes from ISP 2 to be more prefferd over ISP 1.

Navigate to BGP > Import and Add a new policy.

  1. Create a new rule that’s used by ISP-2.
  2. Under the Match tab, select the “From Peer’ – “WAN-ISP-2.”
  3. Unde the Action tab, up the Local Preference to 200 and select OK .
  4. Repeat the steps above and hard set the LP to 100 on WAN-ISP-1.
  5. Commit and let’s compare the route-table from our previous snippet.

Here is the Local-RIB, selecting the default-route from ISP-2.


And verifying the Global Route Table as our preffered exit point:


Looks good! All traffic is now routing out, which is our preffered 10G WAN interface to ISP-2.

Now how do we influence traffic to come into our AS via ISP 2 in hopes of avoiding asymmetrical routing? Well.. we can prepend if we’re advertising routes or advertise a more specific route to the prefferred neighbor and aggregate the routes advertised to the less preffered neighbor. The MED values are not helpful in this case, as we are peering with two separate providers.

We won’t worry about this for now, as we are not adveritisng any public routes to our providers, we simply need internet for our business.

Lets go ahead and redestribute the default route to our Enterprise core router.

But first.. lets peer with it.

I established a peering session with our Enterprise router and set it inside the “Trust” zone.

  • This is just an example design. Depending on the business, a Router will be at the edge and the firewall will sit behind it which is not true in this scenario.

The BGP session has been established with our Enterprise Cisco Router.

A new Peer Group should be created with a peer defined as the internal router.


ENT-ROUTER#show ip bgp summary
BGP router identifier, local AS number 64500
BGP table version is 1, main routing table version 1
1 network entries using 144 bytes of memory
1 path entries using 80 bytes of memory
1/0 BGP path/bestpath attribute entries using 152 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 400 total bytes of memory
BGP activity 1/0 prefixes, 1/0 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 4 64500 4 4 1 0 0 00:00:21 1

  • An internal BGP session isn’t necessary, as a static default route would be plenty. However, for lab purposes lets continue with more BGP FUN.

We can create static routes that point the two /31 interconnects to our directly connected interface from our Cisco to the Palo. This way, the default route that’s re-advertised by default is actually installed into our routing table.

Network Next Hop Metric LocPrf Weight Path
* i 200 0 64496 ?

Total number of prefixes 1

Again, we’re not installing this route, because our local router has no idea where lives.

Create the two static routes for and and the magic happens:



Our Enterprise router now has a way out to the world! Don’t forget to create the inter-zone policy to allow traffic from the Trust to Untrust zone. Also, in a real deployment – there will be a NAT rule out to the inter-webz on the PA, but that’s out of scope for this lab, as I wanted to focus attention to the WAN facing configuration on the Palo Alto.

Palo Alto Documentation on NAT


Software Defined – BGP (ExaBGP, Postman, FLASK, Python3)


I am currently studying for the BGP exam of the Nokia SRA certification path – while doing so, I have found an interesting way to manipulate my BGP routes – I gotta give credit to ThePacketGeek for all their information which made this possible for me.

I’m utilizing several different tools to quickly advertise routes into my EVE-NG Lab topology, which are the following:

  1. Exa-BGP : Exa-Networks GitHub
  2. PostMan : PostMan
  3. RSUB to quickly edit text from through ssh tunel on a remote server : Rsub
  4. Python3
  5. Ubuntu VM
  6. FLASK

I’ve installed UbuntuVM which is on the same network as my EVE-NG topology. Assuming you have a general understanding of the architecture, I’m going to dive right in. I’ve got a Nokia 7750 SR – VPRN with a dot1q sap facing a switch which allows a TCP connection to establish across the link to my VM hosting ExaBGP – All of this is hosted on the same physical server.

  • ExaBGP
    • Install ExaBGP with PIP3.
    • Create a configuration file for  ExaBGP to connect to your router.
      • sudo vi /etc/exabgp/conf.ini


process announce-routes {
run /usr/bin/python3.6m /home/htinoco/exampl.py;
encoder json;
process http-api {
run /usr/bin/python3.6m /home/htinoco/exabg-flask/http_api.py;
encoder json;

neighbor { # Remote Peer
router-id; # Local router-id
local-address; # Local update-source
local-as 65001; # local AS
peer-as 65000; # Peer’s AS

api {
processes [announce-routes, http-api]; #Running multiple processes, python3 script and HTTP API(FLASK)



Here is the python script I’m calling under the ‘announce-routes’ process, which is ponting to /home/htinoco/exampl.py;

#!/usr/bin/env python3
#Change this to the correct python version you’re using.

from __future__ import print_function

from sys import stdout
from time import sleep

#Static routes I want to always announce when I start ExaBGP
messages = [
‘announce route next-hop self’,
‘announce route next-hop self’,
‘announce route next-hop self’,
‘announce route next-hop self’,
‘announce route next-hop self’,


#Iterate through messages
for message in messages:
stdout.write(message + ‘\n’)

#Loop endlessly to allow ExaBGP to continue running
while True:

Now, there is also another process I’m calling, which is my FLASK app.

Here is my flask app:

from flask import Flask, request
from sys import stdout

app = Flask(__name__)

# Setup a command route to listen for prefix advertisements
@app.route(‘/’, methods=[‘POST’])
def command():
command = request.form[‘command’]
stdout.write(‘%s\n’ % command)
return ‘%s\n’ % command

@app.route(‘/shutdown’, methods=[‘POST’])
def shutdown():
return ‘Server shutting down…’

#The param localhost is applied so we can reach the api remotely –

if __name__ == ‘__main__’:
app.run(host=”localhost”, port=7000, debug=True)

#Example POSTS using postman / (BODY/KEY:COMMAND/VALUE=)
#announce route next-hop med 500
#announce route next-hop origin incomplete as-path [100 200 400]
#announce route next-hop med 200 origin egp
#announce route next-hop community [65000:666]


We are ready to roll! – navigate to /etc/exabgp and lets launch exabgp using the conf.ini file we created –

htinoco@ubuntu-server:/etc/exabgp$ exabgp conf.ini


Here is the output after starting exabgp:

: * Serving Flask app “http_api” ( lazy loading )
* Running on http://0:7000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 157-288-415
04:35:45 | 2699 | api | route added to neighbor local-ip local-as 65001 peer-as 65000 router-id family-allowed in-open : next-hop self
04:35:46 | 2699 | api | route added to neighbor local-ip local-as 65001 peer-as 65000 router-id family-allowed in-open : next-hop self
04:35:47 | 2699 | api | route added to neighbor local-ip local-as 65001 peer-as 65000 router-id family-allowed in-open : next-hop self
04:35:48 | 2699 | api | route added to neighbor local-ip local-as 65001 peer-as 65000 router-id family-allowed in-open : next-hop self
04:35:49 | 2699 | api | route added to neighbor local-ip local-as 65001 peer-as 65000 router-id family-allowed in-open : next-hop self

We see the static routes advertisted to our BGP neighbor, from our python script.

We can now open up Postman and POST new routes as we please.


In this example, I’m using “POST” and KEY “command” (review the flask-app code)

to annoucne the route : announce route next-hop self community [65000:666]

The out put from exabg tail process:

04:40:10 | 2699 | api | route added to neighbor local-ip local-as 65001 peer-as 65000 router-id family-allowed in-open : next-hop self community 65000:666

Now lets verify that I am seeing this route on my 7750 SR router that has the BGP session established with ExaBGP.


There they are! The static routes from our python script and the freshly announced /32 route utilizing the API POST method.

Now we have a SUPER EASY! way to pump routes into our lab environemnt (Or production) in order to test policies, verify proper traffic patterns, correct route installment, etc. The possibilities are endless! I am going to use this to for many other tests, such as applying MED, route manipulation with communities and just more general policy based routing.

DDoS Mitigation – RTBH – Blackhole Community

I’m working on a mini-series of videos to demonstrate a common practice with Service Provider networks in regards to DDoS Mitigation. A quick google search and you can find PDF documents from ISP’s all over the world with detailed BGP communities they accept and how they manipulate traffic through their particular AS.

A BGP community string is simply a way to control policy routing through your upstream provider network. The community string in which I’ve been concentrating on is the common “Blackhole” community. This community is advertised to upstream providers to instruct the ISP to discard all traffic to the destination prefix before it is routed to the customer. It is common practice to allow this community. Inquire with your provider for the BGP community document to better understand the way in which you can manipulate  upstream traffic to your advantage.

This lab was mostly rooted from personal projects I’m undergoing but also a great excuse to start pushing the limits of my new EVE-NG server.  I’m really enjoying the interface and the ease-of-use.

Here is the part 1 of the video series. More to come, stay tuned..!

Nokia: Service Router – Default Originate Replica (Cisco BGP Command)

A simple overview on how to re-create the Cisco BGP ‘default-originate’ command for a default route advertisement from a Nokia Service Provider perspective (IPv4/IPv6.)

There are several ways to advertise a default route in the Cisco environment – here is a quick summary:

  1. The first option is what we will attempt to replicate from a Nokia Service Router perspective. Advertising a default route PER BGP Peer instance. This is a more controlled way of advertising.
    ‘neighbor X.Y.X.Y default-originate’ Again, this doesn’t require an active default route to be present or redistributed. This will generate and propagate to the specified neighbor only.
  2. Adding a “network” command under the address family for your BGP routing instance to advertise to ALL neighbors. Remember: This requires an present /0 route in the FIB.
  3. Redistribution – from a currently active default route in the routing table of an IGP. Hence, redistribute – (Almost the same as option 2)
  4. “Default-information originate” – This command will GENERATE a default route to ALL BGP neighbors under the family. An active default route is NOT required to be present under another routing instance in order to propagate the new default route to all BGP peers


The topology will be a Cisco edge device dual-homed to a Nokia PE devices from ISP XYZ. NOTE: There is currently no route map prepending the AS, but we are load balancing.


The topology will be a Cisco edge device dual-homed to a Nokia PE devices from ISP XYZ. NOTE: There is currently no route map prepending the AS, but we are load balancing.


Here is the configuration from the Cisco Edge Device (CustomerSiteB):


Not much in configuration. Standard BGP session to an ISP (without authentication)  – Note the BFD configuration for the peers for a fast failover. Will do a blog post about this in the upcoming weeks, which I will demonstrate the fast-external-fallover command as well.

Currently, we are not learning any BGP routes from our ISP.  Here is the configuration from the Nokia PE devices R5/R6.  (Only showing one side, as they are extremely similar, with only varying subnets)


Creating a Prefix-List:

Lets begin the process to simulate the default-route advertisement.

Note: add prefix ::/0 exact for IPv6 & family ipv6 under the policy statement.

We’ll start by configuring a prefix list on our PE devices (R5/R6):pflist.PNG

Next, we create a Policy Statement:


Make sure to COMMIT the changes!

Adding the policy statement under the BGP configuration inside the VRFs.

/configure service vprn 100 # Example VRF

  • #Create a Black Hole/ Null Route to discard any unwanted/un-routable traffic.
    static-route ::/0 black-hole
    static-route black-hole

Add the new “Default Originate” policy statement we created earlier to the the BGP Group of your customer as an export statement.


At this point we should be advertising ONLY a default route to our BGP neighbor.

From our PE device, a quick “show router 100 bgp neighbor advertised-route” will display the following:


I will go ahead and replicate this policy statement on both VPRN’s on R5/R6 – Our Nokia PE devices for the Dual-Homed BGP Customer and apply the export statement to the BGP peer.

A look at “show ip route bgp” on the CustomerSiteB Cisco edge device shows us the two paths (thanks to the the load balancing, ‘maximum-paths 2’ configuration):


Now we are learning a default route from both uplinks to our dual-homed ISP connection. Failover should be seamless at this point, although having two default routes is prob not what you want – this is a simple example of how to accomplish a default-originate command, which doesn’t exist on the Nokia environment.

Site-to-Site IPSec over MPLS VPN

I want to start by saying I’ve over complicated everything because, well, it’s my lab and it’s fun for me. Depending on customer needs, a simple MPLS L2 VRF and an IPSEC tunnel on top would be sufficient, unless the sites also require internet service by the Service Provider delivering the VRF. In a simple MPLS VPN where the service simply connects sites to sites, the IP addresses are not advertised and could be a lot more secure than over the internet. Another advantage of utilizing an MPLS VPN is the ease of troubleshooting for the customer – the service traverses only one provider – not through the internet.

Lets take a deep dive at what an IPSEC tunnel looks like from an Enterprise perspective over a service provider MPLS L2/L3 VPN, not only from the CustomerSites but what actually happens inside the ServiceProvider network? In this case, we’ll be going from a statically configured site, which is Site-A on the left (Topology A below). The configuration is a Static L3 MPLS VRF provisioned with a Nokia Routed-VPLS utilizing a VRRP for Router-Redundancy (R2/R3). This is a common configuration provided by Service Providers to customers.
The MPLS VRF goes across the MPLS Core Network and terminates on R6 and R5. Both of these sites are participating in the same VRF (VPRN 100) and have an eBGP session set up to the CustomerSiteB on the right side.

A few key notes:
I’ve set up an AS-Path prepend from the Customer Router at Site B facing the eBGP session on R5 at
The Customer Site B router is currently load balancing across both eBGP neighbors to take full advantage of the dual-homed configuation.


Topology A.

MPLS Security?

An MPLS VPN (L2 or L3) Is secure to a certain degree. MPLS VPNs do not encrypt packets across the network, which makes it susceptible to eavesdropping by intruders.

Here is a wireshark capture without IPSEC between CustomerSiteA and R1. The traffic shows icmp request from CustomerSiteB to the Lo0 of CustomerSiteA.  As you can see, there is no encryption by the Service Provider and the service being delivered could easily be sniffed. How much do you trust your Service Provider?

Continue reading “Site-to-Site IPSec over MPLS VPN”