We're in the middle of our 2 weeks of "downtime" before the full-scale deployment starts. So far everything is running fairly smoothly. We did a fail-over test of a primary WESM controller with 151 APs associated with it and that went very well. It's summer so of course, although we have 200+ of the new APs deployed, the total number of simultaneous users is still quite low. We won't truly stress the system until fall classes start.
This Friday Jason and I are doing a Tech Talk for LSPs on the new wireless system. I'll get an overview of the project, and probably the core upgrade project as well, and Jason will cover the details that LSPs will need to know.
Next Friday we're having a "all-hands" meeting of the networking staff. The idea is to present the 10-year network master plan to everyone supporting the network from the installers to the IP engineers, so they understand how this will impact them and what role they will play. There will be a particular focus on the wireless and core upgrade work this summer, the MPLS VPN deployment this fall and support for IPTV and Voice/IP. We're also going to use this meeting as an opportunity to do some performance testing of the new wireless system. We'll be in a fairly large auditorium and should have 60-70 people. We're going to do thing like setup 3 APs and watch how clients balance across the 3 (or not) and put all the uses on a single AP to see how many users and how much bandwidth a single AP can support.
The deployment schedule is completed with approximately 200 APs being installed every week, starting June 9th and wrapping up on August 1st.
Thursday, May 29, 2008
Thursday, May 22, 2008
We're on the down slope...
...or at least it feels like we are ! If you couldn't guess from my posts, last week was a tough week for us. But the team put in a lot of hard work and preparation and our first full week of wireless upgrades is going very smoothly ! (He says as he raps hardily on a nearby wooden table)
We upgraded Geology and Geological Survey on Monday, Psychology and Business Grad on Tuesday, Business (and SPEA Library) on Wednesday and they'll start connecting new APs in SPEA and Informatics in another hour or two. The Wells Library will be upgraded tomorrow, then we'll take a couple of weeks off to let the dust settle and resolve any issues before continuing the upgrades.
We did have a couple of small hiccups this week - both of which were resolved quickly. In Business, some of the power injectors that supply power on the ethernet cables running to the APs were older models that only supported "pre-standard" Power over Ethernet. The new APs do not support this pre-standard version, so we had to install some standard 802.3af power injectors quickly on the morning of the change over. Also, we discovered 2 APs in the SPEA Library who's datajacks are connected to an IDF in the Business School. This meant the old APs there went down yesterday morning when we upgraded the Business School instead of this morning when we're upgrading SPEA. We were able to get new APs installed there fairly quickly to get service back up. For those who don't know, the 2 buildings are joined at the hip and apparently some of the jacks in the SPEA building are too far away from the SPEA IDF, but close enough to an IDF in the Business School. Considering we have almost 1,000 IDFs and 60,000 data jacks, these minor oversights will happen every now and then.
We also ran into a minor bug on one of our WESM controllers yesterday (nothing service impacting). We got word last night that HP engineers were able to reproduce the bug and we're waiting to hear when the fix will be available. Thanks goes to our HP TAM (Technical Account Manager) for helping us pin this down quickly !
We upgraded Geology and Geological Survey on Monday, Psychology and Business Grad on Tuesday, Business (and SPEA Library) on Wednesday and they'll start connecting new APs in SPEA and Informatics in another hour or two. The Wells Library will be upgraded tomorrow, then we'll take a couple of weeks off to let the dust settle and resolve any issues before continuing the upgrades.
We did have a couple of small hiccups this week - both of which were resolved quickly. In Business, some of the power injectors that supply power on the ethernet cables running to the APs were older models that only supported "pre-standard" Power over Ethernet. The new APs do not support this pre-standard version, so we had to install some standard 802.3af power injectors quickly on the morning of the change over. Also, we discovered 2 APs in the SPEA Library who's datajacks are connected to an IDF in the Business School. This meant the old APs there went down yesterday morning when we upgraded the Business School instead of this morning when we're upgrading SPEA. We were able to get new APs installed there fairly quickly to get service back up. For those who don't know, the 2 buildings are joined at the hip and apparently some of the jacks in the SPEA building are too far away from the SPEA IDF, but close enough to an IDF in the Business School. Considering we have almost 1,000 IDFs and 60,000 data jacks, these minor oversights will happen every now and then.
We also ran into a minor bug on one of our WESM controllers yesterday (nothing service impacting). We got word last night that HP engineers were able to reproduce the bug and we're waiting to hear when the fix will be available. Thanks goes to our HP TAM (Technical Account Manager) for helping us pin this down quickly !
Friday, May 16, 2008
14 Down - Only 4,786 to go !!
This morning we completed the core upgrade transition and wireless upgrade for essentially 2 buildings - the UITS Comm Services building and WCC. The team put in 3 very long days getting ready, but it was definitely worth the effort as the changes this morning went off without a hitch (well, almost) !!
The 2 things we caught this morning were that the WESM's (aka controllers) had redundancy configured, but it was not enabled. Also, the radio port adoption settings had the right default power levels, but were set for random channel assignment instead of automatic (intelligent) channel assignment. I caught both of these before they started connecting the new APs, so they all came up with the right settings. Charlie pushed these config changes to the other 10 WESMs, so we're set to go.
On another topic, if you ever wondered what 348 wireless access points looked like, that's what you're looking at up above. They come 12 in a carton and there are 29 cartons there. Now just imagine what 4,800 would look like :) This is why we're trying to pace the delivery - otherwise we'd need a LOT of storage space !!
Thursday, May 15, 2008
The Good, the Bad, and the Ugly
Unfortunately yesterday dealt us mostly the latter two !! After 12 years in the networking field, I've found some days everything seems to just go your way and some days...well...some days everything that can go wrong does and you wish you'd just stayed in bed !
For example, what's the chance of a compact flash card - one that worked perfectly fine all day long the day before - being corrupted when you try to boot a switch from it the next morning ??!! What's the chance of an upgrade to one switch causing another switch across campus to crash - repeatedly ??!! Sometimes even when you've taken every preparation possible, things just don't go your way ! The only consolation after a day like that is that the next one HAS to be better !
Looking on the bright side though, when we finally left that office about 8pm last night, 3 or the 4 core switches had been upgraded and 2 of them had their routing configuration about 75% complete. If you remember, we're converting our layer-2 only aggregation switches into layer-3 switches that will be the default gateway routers for the subnets on campus. Once the global routing configuration is completed on these switches and tested, we will transition the routing for each building to these switches. This routing transition for each building needs to happen first thing in the morning on the day the wireless in the building is upgraded. Therefore, the upgrade of these switches and the configuring of routing on them is a prerequisite for starting the wireless upgrade.
We should have the routing configuration completed this morning and routing for the VLANs that support the network engineers transitioned this morning. We will test throughout that day to make sure everything is working properly. The next step is to transition routing for the UITS buildings in Bloomington tomorrow morning. This will be the first test of the process of converting a building's routing followed by the wireless upgrade of the building and will give us a chance to work out kinks in the process before we start on the first 8 "pilot" buildings next week. By the time we finish the first 8 pilot buildings next week, we should have the process running like a well oiled machine !
For example, what's the chance of a compact flash card - one that worked perfectly fine all day long the day before - being corrupted when you try to boot a switch from it the next morning ??!! What's the chance of an upgrade to one switch causing another switch across campus to crash - repeatedly ??!! Sometimes even when you've taken every preparation possible, things just don't go your way ! The only consolation after a day like that is that the next one HAS to be better !
Looking on the bright side though, when we finally left that office about 8pm last night, 3 or the 4 core switches had been upgraded and 2 of them had their routing configuration about 75% complete. If you remember, we're converting our layer-2 only aggregation switches into layer-3 switches that will be the default gateway routers for the subnets on campus. Once the global routing configuration is completed on these switches and tested, we will transition the routing for each building to these switches. This routing transition for each building needs to happen first thing in the morning on the day the wireless in the building is upgraded. Therefore, the upgrade of these switches and the configuring of routing on them is a prerequisite for starting the wireless upgrade.
We should have the routing configuration completed this morning and routing for the VLANs that support the network engineers transitioned this morning. We will test throughout that day to make sure everything is working properly. The next step is to transition routing for the UITS buildings in Bloomington tomorrow morning. This will be the first test of the process of converting a building's routing followed by the wireless upgrade of the building and will give us a chance to work out kinks in the process before we start on the first 8 "pilot" buildings next week. By the time we finish the first 8 pilot buildings next week, we should have the process running like a well oiled machine !
Friday, May 9, 2008
Those 3 Magic Words
OUT FOR DELIVERY
I don't know about you, but when I'm waiting for that next cool gadget to arrive in the mail, those 3 words always trigger a little burst of excitement...sort of like Christmas morning when I was 6 years old :) But today it's 255 new little gadgets !
I don't know about you, but when I'm waiting for that next cool gadget to arrive in the mail, those 3 words always trigger a little burst of excitement...sort of like Christmas morning when I was 6 years old :) But today it's 255 new little gadgets !
Thursday, May 8, 2008
The APs are coming, the APs are coming !!
That's right, we just received confirmation that our first 255 Access Points arrived at Fedex in Columbus Indiana just a couple of hours ago. We'll take a few of these APs to install in all the UITS buildings later next week and another 190 of them to install in 8 buildings in Bloomington starting on May 19th. We also received 14 WESM modules (aka controllers) earlier in the week, so we have enough hardware for the complete 8 WESM deployment at IUPUI and 1 of the 2 12 WESM deployments at IUB.
In case you're interested in the technology, the WESM modules can be deployed in groups of 12 (or less). Wireless clients can "roam" seamless across all the APs that are associated to any of the 12 WESMs in the "Mobility Group". At IUPUI, we will initially have 8 WESM modules (some of which are for redundancy purposes only) in a Mobility Group to support the 600 APs we plan to deploy there. At IUB, we will have 24 WESM modules (again, some are just back-ups) in 2 separate Mobility Groups to support the 4,200 APs we plan to deploy there.
In case you're interested in the technology, the WESM modules can be deployed in groups of 12 (or less). Wireless clients can "roam" seamless across all the APs that are associated to any of the 12 WESMs in the "Mobility Group". At IUPUI, we will initially have 8 WESM modules (some of which are for redundancy purposes only) in a Mobility Group to support the 600 APs we plan to deploy there. At IUB, we will have 24 WESM modules (again, some are just back-ups) in 2 separate Mobility Groups to support the 4,200 APs we plan to deploy there.
Tuesday, May 6, 2008
Oops, forgot to mention the core upgrade
The core upgrade project is moving along smoothly as well. We received the last few bits of the equipment yesterday - a shipment of XENPACK modules - so we have everything we need to move full steam ahead. Thanks to some hard work over the weekend, all of the new 24-port and 48-port gigabit ethernet modules are installed in the core switches in Bloomington.
This Thursday and Friday they'll be swapping out the Supervisor 720a cards for Supervisor 720bxl cards in the core switches. Essentially this will allow us to turn the core switches into full blown routers with MPLS capabilities. Once this is completed, we'll connect the core switches up to the backbone, configure routing and prepare for migrating the routing function for all the campus buildings to these switches. These transitions will happen building by building as we upgrade the wireless equipment in the buildings. Once we transition the routing for all buildings to the core switches (ie complete the replacement of all existing wireless APs), the core upgrade will be complete. Sounds easy, right ?
This Thursday and Friday they'll be swapping out the Supervisor 720a cards for Supervisor 720bxl cards in the core switches. Essentially this will allow us to turn the core switches into full blown routers with MPLS capabilities. Once this is completed, we'll connect the core switches up to the backbone, configure routing and prepare for migrating the routing function for all the campus buildings to these switches. These transitions will happen building by building as we upgrade the wireless equipment in the buildings. Once we transition the routing for all buildings to the core switches (ie complete the replacement of all existing wireless APs), the core upgrade will be complete. Sounds easy, right ?
Checking in
I know....it's been a whole week since my last post...I'm a slacker !
Jason and I attended a meeting of the CIC schools in Chicago last week to talk about wireless. For those of you who don't know, the CIC is *roughly* the BigTen Conference schools. It was great to hear what other schools are doing and to share information and ideas. One thing was definitely clear from the meeting - there is no perfect controller-based wireless product. No matter which controller-based wireless vendor you choose, you'll run into bugs, limitations and short-comings. Heck, that's true for any network equipment !!
Yesterday we received 16 of the WESM controllers. This brings the total to 22 which is enough for the entire IUPUI installation (8), 1 of the 2 mobility groups at IUB (12), and a couple of WESMs to test with. We started yesterday getting these installed in the switches and configured. Now we're just waiting on the shipment of APs that are due to arrive early next week.
Today our Technical Account Manager (TAM) from HP is in town for our first meeting. He will be our primary support contact and will know our network well so we don't have to bring a support person up to speed on what we're doing every time we open a case. Given the size (4,800 APs) and the short timeframe (3-4 months) of our deployment, having a dedicated support contact that know our network will be extremely important !
Jason and I attended a meeting of the CIC schools in Chicago last week to talk about wireless. For those of you who don't know, the CIC is *roughly* the BigTen Conference schools. It was great to hear what other schools are doing and to share information and ideas. One thing was definitely clear from the meeting - there is no perfect controller-based wireless product. No matter which controller-based wireless vendor you choose, you'll run into bugs, limitations and short-comings. Heck, that's true for any network equipment !!
Yesterday we received 16 of the WESM controllers. This brings the total to 22 which is enough for the entire IUPUI installation (8), 1 of the 2 mobility groups at IUB (12), and a couple of WESMs to test with. We started yesterday getting these installed in the switches and configured. Now we're just waiting on the shipment of APs that are due to arrive early next week.
Today our Technical Account Manager (TAM) from HP is in town for our first meeting. He will be our primary support contact and will know our network well so we don't have to bring a support person up to speed on what we're doing every time we open a case. Given the size (4,800 APs) and the short timeframe (3-4 months) of our deployment, having a dedicated support contact that know our network will be extremely important !
Subscribe to:
Posts (Atom)
Posts
