Once you learn how to use the MPLS hammer, you'll suddenly see a million nails you could whack with your shiny new hammer !
We deployed MPLS and MPLS Layer3 VPNs on the IU campus network this past Monday morning. It was VERY anticlimactic ! We cut and pasted some relatively minimal configs into the routers and it all just worked. What is probably the single largest change in how we do networking on campus since the advent of VLANs happened with no one noticing (except Damon and I who were up entirely too early for a Monday morning). Of course, under the covers all kinds of fancy stuff is happening and we now have a powerful new tool in our tool chest !
Weeks before we actually configured the first MPLS VPN on the network (btw- we won't be putting a production system into this MPLS VPN until Dec. 2nd), we already planned to make MPLS VPNs the centerpiece of the network for the new data center in Bloomington ! Your first thought is probably, why the heck would we want MPLS VPNs in the data center network ?
Our current data center network design has "end-of-row" or "zone" switches (layer-2 only) with cat6 cables running to servers in the adjacent racks. The zone switches each have a 10GbE uplink into a distribution switch (again, layer-2 only). The distribution switch has an 802.1q trunk (2x 10GbE) into the core router. This .1q trunk between the distribution switch and the core router has a Juniper firewall in the middle of it - running in layer-2 mode. Those of you who know this setup in detail will know this is not exactly correct and is over-simplified, but the point is the same.
One problem with this design is that, with over 30 VLANs in the machine room, there is a lot of traffic going in and out of the firewall just to get between 2 VLANs in the same room - perhaps between 2 servers in the same row or same rack or 2 virtual servers inside the same physical server. This causes a couple of problems:
1) It adds significant extra load on the firewall unnecessarily in many cases. Think about DNS queries from all those servers...
2) It makes it very difficult to do vulnerability scanning from behind the firewall because the scanners have to be on 30+ different VLANs
The solution to "fix" this is to place a router behind the firewall - ie turn the distribution switch into a layer-3 routing switch. However, if we did this all 30+ VLANs would be in the same security "zone" - ie there would be no firewall between any of the servers in the machine room. This is not good either. For one, we offer a colocation service and virtual server services, so there are many servers that do no belong to UITS. So we don't want those in the same security zone as our critical central systems. It's probably also not a good idea to put servers with sensitive data in the same security zone as say our FTP mirror server. One solution then would be to place a router behind the firewall for each security zone. But of course that gets very expensive....if you want 5 security zones you need 10 routers (redundant routers for each zone).
And this is where the MPLS VPN hammer gets pulled out to plunk this nail on the head !! You use MPLS VPNs to put 5 virtual routers on each physical router and put a firewall between each virtual router and the physical router and your problem is solved. And actually, if you can virtualize the firewall, you can create a virtual firewall for each virtual router and you have 5 completely separate security zones with a pair of high-end routers and firewalls supporting all 5 - all for no extra cost *except* for all the added operational complexity. Those are the costs we need to figure out before we go too crazy whacking nails with our shiny new hammer !!
Thursday, November 20, 2008
Sunday, November 9, 2008
2 down, 3 more to go !
We had 5 major network maintenances planned in order to complete the core upgrade project and deploy MPLS VPNs. The first 2 are done: The first was disabling the IS-IS routing protocol (OSPF has been deployed along side IS-IS for some time). This was completed last Thursday. The second was replacing our primary border router (a Juniper M10) with a Cisco 6500. This was completed this morning and was the change that was giving me the most heartburn !
The next change is to swap out the secondary border router with a Cisco 6500 on Tuesday. We'll deploy BGP to all our core routers on Thursday. Currently only the border routers run BGP. BGP is needed on the core routers in order to support MPLS VPNs. The following Monday we will deploy MPLS and our first MPLS VPN.
The next change is to swap out the secondary border router with a Cisco 6500 on Tuesday. We'll deploy BGP to all our core routers on Thursday. Currently only the border routers run BGP. BGP is needed on the core routers in order to support MPLS VPNs. The following Monday we will deploy MPLS and our first MPLS VPN.
Friday, November 7, 2008
"Not quite dead yet !"
In case you were worried about my untimely demise, no worries, I'm still alive. I've just been so busy doing that I haven't been writing about what I'm doing :) I'll attempt to catch you up and then will try to get a post out at least once a week from now on.
Wireless:
We deployed about 3,000 Access Points over the summer - roughly an average of 200-250 every week. We also rolled out WPA2 Enterprise (aka 802.1x) during the same timeframe. The majority of the Bloomington Residence Halls have wireless coverage with a few more buildings coming up later this month and around the first of the year. We're now turning ou4 attention to 802.11n to prepare for upgrades next summer. As of yesterday we have 802.11n APs in hand to start testing.
The wireless rollout wasn't without it's bumps, but there were very few user impacting problems. We've been getting a lot of positive feedback from users. When users make a point to call the NOC just to let us know how happy they are with the wireless service, you know it must be going well ! We went out on a limb just a bit by choosing a vendor (HP) that was not a household name in the area of large-scale, controller-based enterprise wireless, but it's worked out extremely well.
Core Upgrade and MPLS VPNs
We also completed the vast majority of the core network upgrade over the summer. The last parts of that upgrade are happening this coming week. We'll be replacing the Juniper M10i Border Routers with Cisco 6500's. That greatly increases the capacity on our Border Routers. As a result, we will be upgrading our primary link to the Internet from 2Gbps to 10Gbps at the same time as the swap out which will happen the day after tomorrow. Once this is completed all our core routers will be Cisco 6500's. Since we had this planned, we had been holding off on deploying MPLS so we didn't have to deal with vendor interoperability issues. Not that this wouldn't have worked with both Juniper and Cisco routers, but this saved us quite a bit of testing. We plan to have our first MPLS VPN live and fully test before the Thanksgiving holiday. This will be the VPN for PCI-DSS systems.
PCI-DSS Compliance
This is really coming together although there is still a lot of work to be done to meet the internal deadline of December 31st of this year. We should be ready to start transitioning system into the PCI-DSS MPLS VPN the week following Thanksgiving. The last network requirement we're still struggling with is 2-factor remote-access. This is just a matter of getting our current Safeword token system working with our Cisco VPN servers. It looks like we may have to wait on an upgrade of the Safeword system, but we're trying to find alternatives because that is not likely to happen before 12/31.
New Data Center
This project is really coming together as well. We're hoping to nail down the final network design for the new data center in a meeting this afternoon. I'll have a post devoted just to the data center network design issues. I think the industry is on the cusp of a major shift in data center networking. Top-of-rack switches are clearly the future in the data center, but products are only just now starting to become available. Fiber Channel over Ethernet is a promising technology, but it's day in the sun is probably still 18-24 months out. Also in the 18-24 month time horizon is 40G and 100G ethernet.
Wireless:
We deployed about 3,000 Access Points over the summer - roughly an average of 200-250 every week. We also rolled out WPA2 Enterprise (aka 802.1x) during the same timeframe. The majority of the Bloomington Residence Halls have wireless coverage with a few more buildings coming up later this month and around the first of the year. We're now turning ou4 attention to 802.11n to prepare for upgrades next summer. As of yesterday we have 802.11n APs in hand to start testing.
The wireless rollout wasn't without it's bumps, but there were very few user impacting problems. We've been getting a lot of positive feedback from users. When users make a point to call the NOC just to let us know how happy they are with the wireless service, you know it must be going well ! We went out on a limb just a bit by choosing a vendor (HP) that was not a household name in the area of large-scale, controller-based enterprise wireless, but it's worked out extremely well.
Core Upgrade and MPLS VPNs
We also completed the vast majority of the core network upgrade over the summer. The last parts of that upgrade are happening this coming week. We'll be replacing the Juniper M10i Border Routers with Cisco 6500's. That greatly increases the capacity on our Border Routers. As a result, we will be upgrading our primary link to the Internet from 2Gbps to 10Gbps at the same time as the swap out which will happen the day after tomorrow. Once this is completed all our core routers will be Cisco 6500's. Since we had this planned, we had been holding off on deploying MPLS so we didn't have to deal with vendor interoperability issues. Not that this wouldn't have worked with both Juniper and Cisco routers, but this saved us quite a bit of testing. We plan to have our first MPLS VPN live and fully test before the Thanksgiving holiday. This will be the VPN for PCI-DSS systems.
PCI-DSS Compliance
This is really coming together although there is still a lot of work to be done to meet the internal deadline of December 31st of this year. We should be ready to start transitioning system into the PCI-DSS MPLS VPN the week following Thanksgiving. The last network requirement we're still struggling with is 2-factor remote-access. This is just a matter of getting our current Safeword token system working with our Cisco VPN servers. It looks like we may have to wait on an upgrade of the Safeword system, but we're trying to find alternatives because that is not likely to happen before 12/31.
New Data Center
This project is really coming together as well. We're hoping to nail down the final network design for the new data center in a meeting this afternoon. I'll have a post devoted just to the data center network design issues. I think the industry is on the cusp of a major shift in data center networking. Top-of-rack switches are clearly the future in the data center, but products are only just now starting to become available. Fiber Channel over Ethernet is a promising technology, but it's day in the sun is probably still 18-24 months out. Also in the 18-24 month time horizon is 40G and 100G ethernet.
Subscribe to:
Posts (Atom)
Posts
