Openflow and Vendor Lock-In

Openflow is really just one piece in a much broader architecture known as Software-Defined Networking (SDN).  The concept of SDN is actually very simple and is explained quite clearly in a number of presentations by Nick McKeown from Stanford - amongst others.  See http://tinyurl.com/4ekzb9w

The basic idea is that today's networking products mirror the mainframe computer industry of decades past.  Vendors deliver packages of proprietary hardware, operating systems and applications bundled together.  Unlike the PC industry of today, you cannot choose to buy network hardware from one vendor, an operating system from another vendor and applications from a third vendor.   In fact, in many cases it becomes difficult to distinguish a product's hardware from it's operating system from it's applications.

So the idea of SDN is to create open interfaces between these layers so the networking market resembles the PC market with competition at each layer of the stack.  Openflow is the open interface between hardware and the network operating system - perhaps roughly analogous to x86.  As network operating systems develop, as several currently are, there will hopefully be open APIs that application developers can use to build functionality on top of the operating systems.  In theory, this should be good for the consumers of networking products because they should have more options and more competition should lead to reduced costs.  As a consumer of networking products, I'm all in !   But will theory and reality actually match up or will something get lost in between ?

I'm a firm believer in learning from history, so let's take a look for a minute at what happened in the world of wireless controllers to see if we can glean any useful knowledge from that experience.

Of course there was CAPWAP which, in some ways, is analogous to Openflow in that it was meant to be an open interface between controllers and APs.  As I understand it, if CAPWAP had achieved success as a multi-vendor interoperable standard, you could have had a controller from vendor A and APs from vendors A, B, and C and they would have played nicely together.  Of course this didn't happen and consumers have to purchase APs and controllers from the same vendor.

However, the lack of controller-to-AP interoperability is not the most troubling interoperability problem associated with controller-based wireless systems.  What happens when I have 30 controllers and 4,000+ APs deployed on a large university campus and then decide to switch wireless vendors ?   Sure, the fact that there's not competition at both the AP and controller layer means I'm probably paying more than I otherwise might.  But, technically,  it's not really a problem deploying the new controllers and having the new APs associate with the new controllers.   That's easy enough.

The real problem is that the controllers from vendor A and vendor B do not interoperate in any meaningful way.  Sure, because they both do basic IP forwarding, a wireless client on one system can communicate with a client on the other system.  But none of the features (ie applications) that drove me to choose a controller-based system over stand-alone APs in the first place are supported across both systems.  Features like RF management and layer-3 roaming do not work across wireless systems.  So, if during the transition between vendors, two adjacent buildings are on different systems, users can't roam seamlessly between buildings, there can be RF interference issues, and captive portal logins aren't maintained forcing users to re-authenticate simply because they moved from one building to another.  As a result, many organizations have become locked-in to their current controller-based wireless vendor.  The amount of effort required to switch vendors is high enough, that they're willing to stay with their current vendor unless they're REALLY unhappy with them or can save a TON of money by switching.  Anecdotally, I hear lots of people complaining about their wireless systems and almost none of them are considering changing vendors !!

So what does this have to do with Software-Defined Networking and Openflow ?   Well, there are in fact a lot of similarities.  Openflow allows a single, software-based network operating system to control potentially hundreds or thousands of hardware devices - not unlike a wireless-controller.   Novel new applications can be rapidly developed on top of the network operating system to allow new functionality and more efficient management of networks - not unlike wireless controllers.  But, if the applications are tightly coupled to the network operating system (as is the case with wireless controllers) and if customers do not sufficiently compel the software vendors to make their products interoperate at an "application" level, consumers could be left in the same vendor lock-in situation they're currently experiencing with controller-based wireless systems.

At this point, those of you who know me are probably thinking, "Man, when did Matt get so down on Openflow ?".   Certainly it's not all gloom and doom.  The SDN product space is developing much differently than the controller-based wireless market.  Open-source projects are flourishing which will hopefully help drive the market is an open-standards direction.  But we, the consumers of networking equipment, need to be vigilant.  Don't just assume that creating competition at both the hardware and operating system layer is going to be good for the consumer.  What happens at the layers above that - the operating system and application layers - is probably much more important in the long run !!

Openflow @ Internet2 Joint Techs in Clemson

Indiana University hosted a BoF session on Openflow this afternoon at the Internet2 Joint Techs Workshop in Clemson, SC.   Chris Small did an excellent job organizing the session and pulled off a great demo of VM mobility.  I presented the intro slides and did a short demo of an Openflow controller from Big Switch Networks.

We counted 60+ people in the room they scheduled for us.  It was packed and there were people standing in the hall.  I asked for a show of hands of people who had never heard of Openflow (0 hands) and of people who knew very little or nothing about Openflow (2-3 hands).  60 people, primarily campus network engineers, in the room and nearly all of them knew something about Openflow.  I was completely blown away !  They ended up moving us to the auditorium so more people could join.  I counted 88 people once we were settled in the auditorium !

Overall it was a good session with some excellent side discussions afterwards.  Next up is GEC10 in Puerto Rico !

Proposal Preparation !

That pretty much sums up the last 4 weeks of my life.   GENI Solicitation 3 proposals are due by 5pm tomorrow.  The IU GlobalNOC is leading or partnering on several different proposals for various parts of the solicitation.  I'm personally working on 2 proposals, PI on one and Co-PI on the other.

I'm looking forward to getting back to "normal" after tomorrow and there's no shortage of other work to be done.  We're evaluating our options for 2011 router refreshes for both IU and I-Light.  Our pilot of the Summer of Networking internship program went extremely well and we're already preparing to continue, improve and hopefully expand the program for next year.  We're also working on a plan to expand the hands-on network training opportunities for networking staff both at IU and other universities.  GEC9 (9th GENI Engineering Conference) will be here before you know it and I suspect the preparations will kick into high gear after proposals are submitted tomorrow !

Openflow @ Internet2 Joint Techs

Internet2 is holding their summer Joint Techs Workshop at Ohio State this week and Openflow was featured prominently on yesterday's afternoon's agenda.  At 3:00 Srini Seeththaraman from Stanford gave an excellent overview of Openflow.  I followed that up with a talk at 4:30 that was focused on the practical aspects of  potential Openflow applications in R&E network and what network engineers can do to get started.  That was immediately followed by a presentation from Heidi Picher Dempsey from the GENI Project Office who talked about GENI and Openflow's application within the GENI infrastructure.  GENI and Openflow were also the primary topic among the regional networks at the Gigapop Geeks BoF with both Heidi and I leading discussions.  There were many good questions and excellent discussion about Openflow and GENI.

The slides and archived video from all of the presentations is available on the Internet2 Joint Techs Workshop agenda page:  


http://tinyurl.com/24rzjn5    

Openflow Wish List

There are very smart people involved in the development of Openflow.  However, I suspect very few of them actively manage networks on a day-to-day basis.  Now that the code is in the hands of network engineers, we can see what's needed to actually get this running in production networks.

When it comes to emerging technologies, this space  between the development and actual production use - between developers and the network engineers in the trench -  is something I find incredibly interesting.  It's great to be involved in the development at the point you can provide substantive feedback into the actual product or technology.

And that is where we are today with Openflow.  We have Openflow deployed to 4 "production" switches and have a wireless SSID in 3 buildings across campus that feeds into an Openflow switch.  The cool thing is that it all pretty much works.  The problem is that, when it doesn't work, it's a pretty big pain to figure out why.  Yesterday I compared it to the early days of the GSRs when the tables on one of the linecards would get out of sync, but it's a bit worse because the "linecards" are spread across the whole campus and there are very few debugging tools available.

There are a number of debugging features that would be useful, but I think the most useful one would be a way to see the dataplane and control-plane packets at the same time.  One way to do this would be for the switch vendors to allow you to add Openflow control-plane packets into a port-mirroring configuration.  This would allow me to hook a sniffer up to a switch port and mirror both the traffic to/from a switch port and the Openflow control messages to the sniffer.  

Why would this be useful ?  One problem we're having right now is that some laptops take 1-2 minutes to get a DHCP lease on the Openflow network.  Is the switch taking a long time to encapsulate the first DHCP message into an Openflow message and send it to the controller ?  Is the controller taking a long time to send the packet-out and flow-add messages to the switch ?  Are the Openflow messages getting lost along the way ?  Today I have to run a tcpdump on the Openflow controller to capture control-plane packets and Wireshark on a laptop to capture the dataplane packets and then try to compare them without synchronized timestamps.   This one little feature would have saved us a lot of headaches !

Grab bag

A little trivia about myself. When I was much younger,  I taught myself how to juggle for a talent show.  The grand finale was to juggle 3 apples while taking a bite out of one of them each time it came around.  Well, right now I feel a bit like a juggler with a dozen apples in the air, desperately trying not to drop one, so here are some snippets of what I'm up to...

- Openflow.  We have our "production" Openflow network up which includes a switch in the Comm Services building, two in Wrubel and one that supports an Openflow SSID that is deployed in Lindley Hall (Computer Science), Informatics and the Innovation Center.  We need to get more "testers" moved onto the switches and SSID to stress test the system.   My laptop and IP phone have been on the Openflow network for almost 6 weeks without any problems.  We're hoping to have a Informatics grad student working on the project with us starting in September.

- Wireless.  When HP bought 3COM/H3C this spring they got what appears to be a very good controller-based wireless product from H3C.  We received some eval equipment yesterday which we'll be testing.  We're trying to determine the right path moving forward between HP's two different wireless systems.

- GlobalNOC Summer of Networking.  Students have started arriving with the remaining students starting on June 1st.   Our weekly training program will start the following week.  I think we have 8 total students.  Most of them are in the syseng area, but we also have one in the Service Desk and one in my area (Network Architecture), to help with the test lab.

- Test Lab.  Use of the testlab is really picking up.  We have a bunch of new equipment coming in on eval as well as some permanent equipment and a lot of people who want to do testing.  Ed Furia and I have been supporting this in our spare time (with help from an intern this summer), but we really need to get a full-time lab admin hired.

- Training.  At the GlobalNOC retreat last week there was a lot of interest in developing a training program.  We're hoping to develop a curriculum of hands-on network training that could benefit GlobalNOC staff, other UITS staff, interns, and potentially others.

- IU Health Science Network:  This is a design I proposed in 2007 to resolve many of the issues surrounding the IU School of Medicine and the Clarian hospital system.  It's finally gaining momentum and implementation will start very soon.

Well, those are the highlights !

Hotel California

Well, I'm spending another week in a hotel in California (Sunnyvale this time).   Juniper yesterday, HP today, and Cyan Optics tomorrow.  Cyan makes an interesting product in the packet/optical space.  It's potentially a very good match for RONs and Statenets that don't need a lot of wave services (< 8 waves) and want a low-cost Ethernet services platform that is well integrated with the DWDM platform.  Interestingly, Cyan has a connection to IU in that one of the Cyan founders, Mike Hatfield, is an Indiana native who is an alum of the IU Kelley School of Business.