I spent last week at the Open Networking Summit and Open Networking Foundation Member Workday and came home with a head full of ideas to write about. Here's my first set....
At ONS I heard the term DevOps mentioned many times. In fact, Stacy from GigaOM (who btw was an excellent moderator for the panel I participated in) wrote a post about it as well (See "Does networking need a devops movement"). Perhaps it's because I live in the corn fields of Indiana or because I really live in my own little world most of the time, but I actually had to Wikipedia the term DevOps (Can you use Wikipedia as a verb ?). In any case, when I read the Wikipedia article I thought, DUH OF COURSE !!
You see, this is how we've been developing network management software at the GlobalNOC at Indiana University for the past 12 years. For those of you who don't know us, the GlobalNOC partners with universities and non-profits to help them manage their networks by providing NOC and network engineering support. Very early on, as we grew from working with 1 network to 3 or 4 networks, it became painfully obvious that we would need very good, highly customized set of software to help us manage these very diverse networks.
This started out with a few network engineers (like myself) who had CS backgrounds hacking together some open-source software with way too much homegrown Perl scripting for our own good. We wrote the software, we supported the software and we used the software to manage networks ! Over time, as the team grew and the software became much more sophisticated, it was necessary to split the developer/sysadmin team from the network engineering team, but they are still very closely linked.
I'm no longer directly involved with the developer/sysadmin team (SYSENG as we call them), but the parts of the development process described in the DevOps article on Wikipedia, such as reduced release scope with smaller changes more often and automation of deployment, sounds very familiar :) I'm told we rolled out something like 40 software releases into production last year. In addition, the "users" of the software, ie the network operators and engineers, have a very close link to the developers/sysadmins making it easy to exchange ideas for new features and to track down bugs.
One of the big reasons I've been excited about SDN is the possibility that it could be used to apply these same DevOps principles to the development of control-plane software for networks. If I'm not mistaken, this is how IP networking started in the first place (or so I'm told) ! I'm certainly not old enough to have been around during the early days of the Internet. However, I was very fortunate to start my career with a company called ANS, which operated the NSFnet, and had the privilege of working with many of the people who were.
From what I've heard, the early IP networks were run by computer scientists who both operated the network and wrote the software that powered them. The process of getting new features into the software didn't involve hundreds (or thousands) of network engineers submitting ideas for new features to sales reps, product managers trying to distill these requests into the actual features that will go into the software, developers building the software, QA teams testing the software, etc - only to have the QA teams at the customer sites do their own extensive QA tests to verify the product works properly in their unique environments - before the feature could be put into production.
Now, I don't know for sure, but I suspect the process "in the good ole days" led to a few more bumps in the road then would be acceptable in today's production networks. However, I think SDN at least (re)opens the possibility of a tighter "loop" between the people who manage networks and the people who build the software the powers them and IMO that would be a good thing !
At the Open Networking Summit last week, the Indiana University team demo'd our first example of SDN software developed using this methodology and it's already deployed on a nationwide, production backbone network called NDDI. We'll be demo'ing our next piece of SDN software at GEC12 next week which solves a specific issue on campus LAN. Our plan is to have it running in production in the Indiana University network by the end of November. From the internal demo session this morning, it looks like were very much on track to make that happen.
In the GigaOM article I referenced earlier, the question was posed as to "where in the networking world these developers would come from". Well, Stacy, we're growing them today, right here in the corn fields of Indiana !!
Wednesday, October 26, 2011
Wednesday, March 16, 2011
GEC10 Demo Night
After multiple cab rides and a scenic tour of several areas of San Juan I wouldn't dare go after dark, I made it to the Sheraton Conference Center (not to be confused with the Sheraton Old Town) in time to setup for the Tuesday night demos. IU had 3 demos last night: GENI Meta Operations Center (GMOC), KetKarma and Openflow Campus Trials. I was responsible for the Openflow Campus Trials Demo and, as luck would have it, nearly all of the monitoring work Chris Small has done is now rolled into the GMOC tools and was on display in the GMOC demo. Also, John Meylor and Camillo Viecco were on hand and fully prepared to answer questions about Measurement Manager, so I was able to spend quite a bit of time checking out the other demos.
I had a number of really good discussions about Openflow that continued through dinner and a drink in Old San Juan. Now on to the real work of the conference on Wednesday !
Monday, March 14, 2011
Openflow and Vendor Lock-In
Openflow is really just one piece in a much broader architecture known as Software-Defined Networking (SDN). The concept of SDN is actually very simple and is explained quite clearly in a number of presentations by Nick McKeown from Stanford - amongst others. See http://tinyurl.com/4ekzb9w
The basic idea is that today's networking products mirror the mainframe computer industry of decades past. Vendors deliver packages of proprietary hardware, operating systems and applications bundled together. Unlike the PC industry of today, you cannot choose to buy network hardware from one vendor, an operating system from another vendor and applications from a third vendor. In fact, in many cases it becomes difficult to distinguish a product's hardware from it's operating system from it's applications.
So the idea of SDN is to create open interfaces between these layers so the networking market resembles the PC market with competition at each layer of the stack. Openflow is the open interface between hardware and the network operating system - perhaps roughly analogous to x86. As network operating systems develop, as several currently are, there will hopefully be open APIs that application developers can use to build functionality on top of the operating systems. In theory, this should be good for the consumers of networking products because they should have more options and more competition should lead to reduced costs. As a consumer of networking products, I'm all in ! But will theory and reality actually match up or will something get lost in between ?
I'm a firm believer in learning from history, so let's take a look for a minute at what happened in the world of wireless controllers to see if we can glean any useful knowledge from that experience.
Of course there was CAPWAP which, in some ways, is analogous to Openflow in that it was meant to be an open interface between controllers and APs. As I understand it, if CAPWAP had achieved success as a multi-vendor interoperable standard, you could have had a controller from vendor A and APs from vendors A, B, and C and they would have played nicely together. Of course this didn't happen and consumers have to purchase APs and controllers from the same vendor.
However, the lack of controller-to-AP interoperability is not the most troubling interoperability problem associated with controller-based wireless systems. What happens when I have 30 controllers and 4,000+ APs deployed on a large university campus and then decide to switch wireless vendors ? Sure, the fact that there's not competition at both the AP and controller layer means I'm probably paying more than I otherwise might. But, technically, it's not really a problem deploying the new controllers and having the new APs associate with the new controllers. That's easy enough.
The real problem is that the controllers from vendor A and vendor B do not interoperate in any meaningful way. Sure, because they both do basic IP forwarding, a wireless client on one system can communicate with a client on the other system. But none of the features (ie applications) that drove me to choose a controller-based system over stand-alone APs in the first place are supported across both systems. Features like RF management and layer-3 roaming do not work across wireless systems. So, if during the transition between vendors, two adjacent buildings are on different systems, users can't roam seamlessly between buildings, there can be RF interference issues, and captive portal logins aren't maintained forcing users to re-authenticate simply because they moved from one building to another. As a result, many organizations have become locked-in to their current controller-based wireless vendor. The amount of effort required to switch vendors is high enough, that they're willing to stay with their current vendor unless they're REALLY unhappy with them or can save a TON of money by switching. Anecdotally, I hear lots of people complaining about their wireless systems and almost none of them are considering changing vendors !!
So what does this have to do with Software-Defined Networking and Openflow ? Well, there are in fact a lot of similarities. Openflow allows a single, software-based network operating system to control potentially hundreds or thousands of hardware devices - not unlike a wireless-controller. Novel new applications can be rapidly developed on top of the network operating system to allow new functionality and more efficient management of networks - not unlike wireless controllers. But, if the applications are tightly coupled to the network operating system (as is the case with wireless controllers) and if customers do not sufficiently compel the software vendors to make their products interoperate at an "application" level, consumers could be left in the same vendor lock-in situation they're currently experiencing with controller-based wireless systems.
At this point, those of you who know me are probably thinking, "Man, when did Matt get so down on Openflow ?". Certainly it's not all gloom and doom. The SDN product space is developing much differently than the controller-based wireless market. Open-source projects are flourishing which will hopefully help drive the market is an open-standards direction. But we, the consumers of networking equipment, need to be vigilant. Don't just assume that creating competition at both the hardware and operating system layer is going to be good for the consumer. What happens at the layers above that - the operating system and application layers - is probably much more important in the long run !!
The basic idea is that today's networking products mirror the mainframe computer industry of decades past. Vendors deliver packages of proprietary hardware, operating systems and applications bundled together. Unlike the PC industry of today, you cannot choose to buy network hardware from one vendor, an operating system from another vendor and applications from a third vendor. In fact, in many cases it becomes difficult to distinguish a product's hardware from it's operating system from it's applications.
So the idea of SDN is to create open interfaces between these layers so the networking market resembles the PC market with competition at each layer of the stack. Openflow is the open interface between hardware and the network operating system - perhaps roughly analogous to x86. As network operating systems develop, as several currently are, there will hopefully be open APIs that application developers can use to build functionality on top of the operating systems. In theory, this should be good for the consumers of networking products because they should have more options and more competition should lead to reduced costs. As a consumer of networking products, I'm all in ! But will theory and reality actually match up or will something get lost in between ?
I'm a firm believer in learning from history, so let's take a look for a minute at what happened in the world of wireless controllers to see if we can glean any useful knowledge from that experience.
Of course there was CAPWAP which, in some ways, is analogous to Openflow in that it was meant to be an open interface between controllers and APs. As I understand it, if CAPWAP had achieved success as a multi-vendor interoperable standard, you could have had a controller from vendor A and APs from vendors A, B, and C and they would have played nicely together. Of course this didn't happen and consumers have to purchase APs and controllers from the same vendor.
However, the lack of controller-to-AP interoperability is not the most troubling interoperability problem associated with controller-based wireless systems. What happens when I have 30 controllers and 4,000+ APs deployed on a large university campus and then decide to switch wireless vendors ? Sure, the fact that there's not competition at both the AP and controller layer means I'm probably paying more than I otherwise might. But, technically, it's not really a problem deploying the new controllers and having the new APs associate with the new controllers. That's easy enough.
The real problem is that the controllers from vendor A and vendor B do not interoperate in any meaningful way. Sure, because they both do basic IP forwarding, a wireless client on one system can communicate with a client on the other system. But none of the features (ie applications) that drove me to choose a controller-based system over stand-alone APs in the first place are supported across both systems. Features like RF management and layer-3 roaming do not work across wireless systems. So, if during the transition between vendors, two adjacent buildings are on different systems, users can't roam seamlessly between buildings, there can be RF interference issues, and captive portal logins aren't maintained forcing users to re-authenticate simply because they moved from one building to another. As a result, many organizations have become locked-in to their current controller-based wireless vendor. The amount of effort required to switch vendors is high enough, that they're willing to stay with their current vendor unless they're REALLY unhappy with them or can save a TON of money by switching. Anecdotally, I hear lots of people complaining about their wireless systems and almost none of them are considering changing vendors !!
So what does this have to do with Software-Defined Networking and Openflow ? Well, there are in fact a lot of similarities. Openflow allows a single, software-based network operating system to control potentially hundreds or thousands of hardware devices - not unlike a wireless-controller. Novel new applications can be rapidly developed on top of the network operating system to allow new functionality and more efficient management of networks - not unlike wireless controllers. But, if the applications are tightly coupled to the network operating system (as is the case with wireless controllers) and if customers do not sufficiently compel the software vendors to make their products interoperate at an "application" level, consumers could be left in the same vendor lock-in situation they're currently experiencing with controller-based wireless systems.
At this point, those of you who know me are probably thinking, "Man, when did Matt get so down on Openflow ?". Certainly it's not all gloom and doom. The SDN product space is developing much differently than the controller-based wireless market. Open-source projects are flourishing which will hopefully help drive the market is an open-standards direction. But we, the consumers of networking equipment, need to be vigilant. Don't just assume that creating competition at both the hardware and operating system layer is going to be good for the consumer. What happens at the layers above that - the operating system and application layers - is probably much more important in the long run !!
Monday, January 31, 2011
Openflow @ Internet2 Joint Techs in Clemson
Indiana University hosted a BoF session on Openflow this afternoon at the Internet2 Joint Techs Workshop in Clemson, SC. Chris Small did an excellent job organizing the session and pulled off a great demo of VM mobility. I presented the intro slides and did a short demo of an Openflow controller from Big Switch Networks.
We counted 60+ people in the room they scheduled for us. It was packed and there were people standing in the hall. I asked for a show of hands of people who had never heard of Openflow (0 hands) and of people who knew very little or nothing about Openflow (2-3 hands). 60 people, primarily campus network engineers, in the room and nearly all of them knew something about Openflow. I was completely blown away ! They ended up moving us to the auditorium so more people could join. I counted 88 people once we were settled in the auditorium !
Overall it was a good session with some excellent side discussions afterwards. Next up is GEC10 in Puerto Rico !
We counted 60+ people in the room they scheduled for us. It was packed and there were people standing in the hall. I asked for a show of hands of people who had never heard of Openflow (0 hands) and of people who knew very little or nothing about Openflow (2-3 hands). 60 people, primarily campus network engineers, in the room and nearly all of them knew something about Openflow. I was completely blown away ! They ended up moving us to the auditorium so more people could join. I counted 88 people once we were settled in the auditorium !
Overall it was a good session with some excellent side discussions afterwards. Next up is GEC10 in Puerto Rico !
Thursday, August 19, 2010
Proposal Preparation !
That pretty much sums up the last 4 weeks of my life. GENI Solicitation 3 proposals are due by 5pm tomorrow. The IU GlobalNOC is leading or partnering on several different proposals for various parts of the solicitation. I'm personally working on 2 proposals, PI on one and Co-PI on the other.
I'm looking forward to getting back to "normal" after tomorrow and there's no shortage of other work to be done. We're evaluating our options for 2011 router refreshes for both IU and I-Light. Our pilot of the Summer of Networking internship program went extremely well and we're already preparing to continue, improve and hopefully expand the program for next year. We're also working on a plan to expand the hands-on network training opportunities for networking staff both at IU and other universities. GEC9 (9th GENI Engineering Conference) will be here before you know it and I suspect the preparations will kick into high gear after proposals are submitted tomorrow !
I'm looking forward to getting back to "normal" after tomorrow and there's no shortage of other work to be done. We're evaluating our options for 2011 router refreshes for both IU and I-Light. Our pilot of the Summer of Networking internship program went extremely well and we're already preparing to continue, improve and hopefully expand the program for next year. We're also working on a plan to expand the hands-on network training opportunities for networking staff both at IU and other universities. GEC9 (9th GENI Engineering Conference) will be here before you know it and I suspect the preparations will kick into high gear after proposals are submitted tomorrow !
Tuesday, July 13, 2010
Openflow @ Internet2 Joint Techs
Internet2 is holding their summer Joint Techs Workshop at Ohio State this week and Openflow was featured prominently on yesterday's afternoon's agenda. At 3:00 Srini Seeththaraman from Stanford gave an excellent overview of Openflow. I followed that up with a talk at 4:30 that was focused on the practical aspects of potential Openflow applications in R&E network and what network engineers can do to get started. That was immediately followed by a presentation from Heidi Picher Dempsey from the GENI Project Office who talked about GENI and Openflow's application within the GENI infrastructure. GENI and Openflow were also the primary topic among the regional networks at the Gigapop Geeks BoF with both Heidi and I leading discussions. There were many good questions and excellent discussion about Openflow and GENI.
The slides and archived video from all of the presentations is available on the Internet2 Joint Techs Workshop agenda page:
http://tinyurl.com/24rzjn5
The slides and archived video from all of the presentations is available on the Internet2 Joint Techs Workshop agenda page:
http://tinyurl.com/24rzjn5
Thursday, June 17, 2010
Openflow Wish List
There are very smart people involved in the development of Openflow. However, I suspect very few of them actively manage networks on a day-to-day basis. Now that the code is in the hands of network engineers, we can see what's needed to actually get this running in production networks.
When it comes to emerging technologies, this space between the development and actual production use - between developers and the network engineers in the trench - is something I find incredibly interesting. It's great to be involved in the development at the point you can provide substantive feedback into the actual product or technology.
And that is where we are today with Openflow. We have Openflow deployed to 4 "production" switches and have a wireless SSID in 3 buildings across campus that feeds into an Openflow switch. The cool thing is that it all pretty much works. The problem is that, when it doesn't work, it's a pretty big pain to figure out why. Yesterday I compared it to the early days of the GSRs when the tables on one of the linecards would get out of sync, but it's a bit worse because the "linecards" are spread across the whole campus and there are very few debugging tools available.
There are a number of debugging features that would be useful, but I think the most useful one would be a way to see the dataplane and control-plane packets at the same time. One way to do this would be for the switch vendors to allow you to add Openflow control-plane packets into a port-mirroring configuration. This would allow me to hook a sniffer up to a switch port and mirror both the traffic to/from a switch port and the Openflow control messages to the sniffer.
Why would this be useful ? One problem we're having right now is that some laptops take 1-2 minutes to get a DHCP lease on the Openflow network. Is the switch taking a long time to encapsulate the first DHCP message into an Openflow message and send it to the controller ? Is the controller taking a long time to send the packet-out and flow-add messages to the switch ? Are the Openflow messages getting lost along the way ? Today I have to run a tcpdump on the Openflow controller to capture control-plane packets and Wireshark on a laptop to capture the dataplane packets and then try to compare them without synchronized timestamps. This one little feature would have saved us a lot of headaches !
When it comes to emerging technologies, this space between the development and actual production use - between developers and the network engineers in the trench - is something I find incredibly interesting. It's great to be involved in the development at the point you can provide substantive feedback into the actual product or technology.
And that is where we are today with Openflow. We have Openflow deployed to 4 "production" switches and have a wireless SSID in 3 buildings across campus that feeds into an Openflow switch. The cool thing is that it all pretty much works. The problem is that, when it doesn't work, it's a pretty big pain to figure out why. Yesterday I compared it to the early days of the GSRs when the tables on one of the linecards would get out of sync, but it's a bit worse because the "linecards" are spread across the whole campus and there are very few debugging tools available.
There are a number of debugging features that would be useful, but I think the most useful one would be a way to see the dataplane and control-plane packets at the same time. One way to do this would be for the switch vendors to allow you to add Openflow control-plane packets into a port-mirroring configuration. This would allow me to hook a sniffer up to a switch port and mirror both the traffic to/from a switch port and the Openflow control messages to the sniffer.
Why would this be useful ? One problem we're having right now is that some laptops take 1-2 minutes to get a DHCP lease on the Openflow network. Is the switch taking a long time to encapsulate the first DHCP message into an Openflow message and send it to the controller ? Is the controller taking a long time to send the packet-out and flow-add messages to the switch ? Are the Openflow messages getting lost along the way ? Today I have to run a tcpdump on the Openflow controller to capture control-plane packets and Wireshark on a laptop to capture the dataplane packets and then try to compare them without synchronized timestamps. This one little feature would have saved us a lot of headaches !
Thursday, May 27, 2010
Grab bag
A little trivia about myself. When I was much younger, I taught myself how to juggle for a talent show. The grand finale was to juggle 3 apples while taking a bite out of one of them each time it came around. Well, right now I feel a bit like a juggler with a dozen apples in the air, desperately trying not to drop one, so here are some snippets of what I'm up to...
- Openflow. We have our "production" Openflow network up which includes a switch in the Comm Services building, two in Wrubel and one that supports an Openflow SSID that is deployed in Lindley Hall (Computer Science), Informatics and the Innovation Center. We need to get more "testers" moved onto the switches and SSID to stress test the system. My laptop and IP phone have been on the Openflow network for almost 6 weeks without any problems. We're hoping to have a Informatics grad student working on the project with us starting in September.
- Wireless. When HP bought 3COM/H3C this spring they got what appears to be a very good controller-based wireless product from H3C. We received some eval equipment yesterday which we'll be testing. We're trying to determine the right path moving forward between HP's two different wireless systems.
- GlobalNOC Summer of Networking. Students have started arriving with the remaining students starting on June 1st. Our weekly training program will start the following week. I think we have 8 total students. Most of them are in the syseng area, but we also have one in the Service Desk and one in my area (Network Architecture), to help with the test lab.
- Test Lab. Use of the testlab is really picking up. We have a bunch of new equipment coming in on eval as well as some permanent equipment and a lot of people who want to do testing. Ed Furia and I have been supporting this in our spare time (with help from an intern this summer), but we really need to get a full-time lab admin hired.
- Training. At the GlobalNOC retreat last week there was a lot of interest in developing a training program. We're hoping to develop a curriculum of hands-on network training that could benefit GlobalNOC staff, other UITS staff, interns, and potentially others.
- IU Health Science Network: This is a design I proposed in 2007 to resolve many of the issues surrounding the IU School of Medicine and the Clarian hospital system. It's finally gaining momentum and implementation will start very soon.
Well, those are the highlights !
- Openflow. We have our "production" Openflow network up which includes a switch in the Comm Services building, two in Wrubel and one that supports an Openflow SSID that is deployed in Lindley Hall (Computer Science), Informatics and the Innovation Center. We need to get more "testers" moved onto the switches and SSID to stress test the system. My laptop and IP phone have been on the Openflow network for almost 6 weeks without any problems. We're hoping to have a Informatics grad student working on the project with us starting in September.
- Wireless. When HP bought 3COM/H3C this spring they got what appears to be a very good controller-based wireless product from H3C. We received some eval equipment yesterday which we'll be testing. We're trying to determine the right path moving forward between HP's two different wireless systems.
- GlobalNOC Summer of Networking. Students have started arriving with the remaining students starting on June 1st. Our weekly training program will start the following week. I think we have 8 total students. Most of them are in the syseng area, but we also have one in the Service Desk and one in my area (Network Architecture), to help with the test lab.
- Test Lab. Use of the testlab is really picking up. We have a bunch of new equipment coming in on eval as well as some permanent equipment and a lot of people who want to do testing. Ed Furia and I have been supporting this in our spare time (with help from an intern this summer), but we really need to get a full-time lab admin hired.
- Training. At the GlobalNOC retreat last week there was a lot of interest in developing a training program. We're hoping to develop a curriculum of hands-on network training that could benefit GlobalNOC staff, other UITS staff, interns, and potentially others.
- IU Health Science Network: This is a design I proposed in 2007 to resolve many of the issues surrounding the IU School of Medicine and the Clarian hospital system. It's finally gaining momentum and implementation will start very soon.
Well, those are the highlights !
Tuesday, May 4, 2010
Hotel California
Well, I'm spending another week in a hotel in California (Sunnyvale this time). Juniper yesterday, HP today, and Cyan Optics tomorrow. Cyan makes an interesting product in the packet/optical space. It's potentially a very good match for RONs and Statenets that don't need a lot of wave services (< 8 waves) and want a low-cost Ethernet services platform that is well integrated with the DWDM platform. Interestingly, Cyan has a connection to IU in that one of the Cyan founders, Mike Hatfield, is an Indiana native who is an alum of the IU Kelley School of Business.
Tuesday, April 20, 2010
Openflow Testbed Taking Shape
I started my morning with a nice walk across campus to Lindley Hall for a meeting with Rob Henderson. I have to interject that it was a great morning to walk across campus - 60 degrees and sunny ! It sure beats 10th and the Bypass !! Anyway the primary topic of the meeting was our Openflow testbed, but we wondered off on a number of different topics. We need to get people to help us test Openflow and Informatics & Computer Sciences seemed like a logical place to start. Rob is onboard and we're ready to start rolling !
Our first step will be to deploy a wireless SSID for Openflow. The SSID will function exactly like our 802.1x SSID (IU Secure) except the user traffic will be plumbed through a couple of Openflow enabled switches before it hits the first router. The key advantages are (1) we can easily deploy an Openflow SSID to thousands of APs to get a lot of users and (2) users can opt-in and out easily simply be switching SSIDs (which will be helpful if something breaks).
In parallel with the wireless deployment, we'll be deploying Openflow on production switches for wired users, first in the UITS complex at WCC and then at Informatics and CS. If all goes well, we could have Openflow enabled on 15-20 switches by the end of July along with the wireless deployment.
Thursday, April 8, 2010
NSF Campus Bridging Workshop
I spent yesterday at the IUPUI Conference Center attending an NSF "Campus Bridging" workshop. Your first response will likely be the same as everyone I've spoken to - "What the heck is that ?".
Well, this was the first one I attended, but I believe this is one track in a series of workshops to help the NSF decide how to structure it's future Cyberinfrastructure (CI) funding programs. The focus was on how to get campuses ramped up to support the data deluge generated by scientific instruments from gene sequencers to the LHC. Obviously networking is a big part of that equation, but certainly not the only part. There was a lot of discussion about data storage and indexing, meta data, federated identity and so on.
Here are a couple of good presentations that I think hit the nail on the head in terms of how we should be building campus networks to handle big data science applications...
Network Architecture for High Performance (Joe Metzger - ESNET)
The Data Intensive Network (Guy Almes - TAMU)
Incidentally, IU started building our campus networks this way in about 2003-04 and I think this is one of the reasons we've been so successful with projects like the Data Capacitor.
Well, this was the first one I attended, but I believe this is one track in a series of workshops to help the NSF decide how to structure it's future Cyberinfrastructure (CI) funding programs. The focus was on how to get campuses ramped up to support the data deluge generated by scientific instruments from gene sequencers to the LHC. Obviously networking is a big part of that equation, but certainly not the only part. There was a lot of discussion about data storage and indexing, meta data, federated identity and so on.
Here are a couple of good presentations that I think hit the nail on the head in terms of how we should be building campus networks to handle big data science applications...
Network Architecture for High Performance (Joe Metzger - ESNET)
The Data Intensive Network (Guy Almes - TAMU)
Incidentally, IU started building our campus networks this way in about 2003-04 and I think this is one of the reasons we've been so successful with projects like the Data Capacitor.
Thursday, April 1, 2010
Visit to Ball State
I spent yesterday afternoon at Ball State University in Muncie, IN. For those non-Hoosiers out there, Ball State is named after te Ball Family as in the jars you can tomatoes in ! I met with Steve Jones who is the director of their CICS program. Hopefully I don't butcher the acronym, but IIRC it stands for Center for Information and Communications Sciences. It's a very cool program and as someone who grew up right down the road from the university, I had no idea it existed. They have some very bright and motivated students and hopefully some of them will eventually come join the team at the GlobalNOC !
-- Post From My iPhone
-- Post From My iPhone
Wednesday, March 24, 2010
Announcing the GlobalNOC Summer of Networking Program
The GlobalNOC has a long history of hiring students to work on projects during the summer. In fact, many of our software developers and system administrators started with us as students. This summer we anticipate having about 8-10 students working in multiple areas of the GlobalNOC including Systems Engineering, Service Desk and Network Architecture.
With such a large group of students, we decided to pilot a program to provide additional training opportunities in a group forum. Our plan is to have group training/seminar sessions one afternoon a week. The initial sessions with include presentations and training by GlobalNOC staff to the students. Towards the end of the summer, the sessions will be focused on the students presenting their work from the summer or a networking topic they've been researching during the summer to each other. Since the students will be split between the IUPUI and IUB campuses, most of the group sessions will be conducted via high-definition video conferencing, but at least two of the sessions will be conducted face-to-face with all the students.
There will also be opportunities for students to shadow GlobalNOC staff in areas other than the area in which they are working. So a student working on software development in the Systems Engineering group would have a chance to learn about the Service Desk, Network Engineering and Network Architecture groups by shadowing someone in each of those areas.
This is a pilot, so we may need to make adjusts during the summer, but I think this will be a great opportunity for students to get hands-on experience managing large-scale networks.
With such a large group of students, we decided to pilot a program to provide additional training opportunities in a group forum. Our plan is to have group training/seminar sessions one afternoon a week. The initial sessions with include presentations and training by GlobalNOC staff to the students. Towards the end of the summer, the sessions will be focused on the students presenting their work from the summer or a networking topic they've been researching during the summer to each other. Since the students will be split between the IUPUI and IUB campuses, most of the group sessions will be conducted via high-definition video conferencing, but at least two of the sessions will be conducted face-to-face with all the students.
There will also be opportunities for students to shadow GlobalNOC staff in areas other than the area in which they are working. So a student working on software development in the Systems Engineering group would have a chance to learn about the Service Desk, Network Engineering and Network Architecture groups by shadowing someone in each of those areas.
This is a pilot, so we may need to make adjusts during the summer, but I think this will be a great opportunity for students to get hands-on experience managing large-scale networks.
Monday, March 22, 2010
Openflow Trip
Nothing like back-to-back weeks of travel ! This week we're headed to Silicon Valley for a series of meetings related to Openflow including stops at Stanford University and HP Labs. Actually, last week's trip to GEC7 at Duke University was related to Openflow as well. If you haven't checked out Openflow yet, I'd encourage you to do so (www.openflowswitch.org). It's a standard API that allows external systems (think PC servers) to manipulate the forwarding ASICs in switches and routers. IU was recently awarded an NSF grant through the GENI program to help get Openflow deployed on campuses.
Monday, March 15, 2010
Heading to GEC 7
I'm heading to GEC 7, the 7th GENI Engineering Conference, tomorrow with several colleagues from the IU GlobalNOC. IU has received multiple GENI grants so far including one for the Openflow Campus Trials which I'm working on along with the PI, Chris Small. Tomorrow night we'll be doing a demo of our current Openflow deployment that includes 6 HP switches running Openflow capable code along with the NOX and SNAC Openflow controllers. You can check out our project page on the GENI Wiki for more information.
Tuesday, January 5, 2010
Back in the Saddle Again !
Happy New Year ! Hopefully everyone enjoyed the holidays. I hardly looked at email for 2 full weeks which was very nice !
2010 promises to be as busy and eventful as 2009, if not more ! We are in the midst of two separate beta testing programs right now along with an RFP. I'm actively working on two grant proposals, a major project to provide a more seamless networking experience across the Clarian (hospital) and IU facilities, and I'm trying to finish up a Legacy RSA with ARIN. In all, my group has about 20 active projects on our plate right now !
2010 promises to be as busy and eventful as 2009, if not more ! We are in the midst of two separate beta testing programs right now along with an RFP. I'm actively working on two grant proposals, a major project to provide a more seamless networking experience across the Clarian (hospital) and IU facilities, and I'm trying to finish up a Legacy RSA with ARIN. In all, my group has about 20 active projects on our plate right now !
Thursday, December 10, 2009
The Lab Experiment
We've compiled a fair amount of equipment that can be used for testing and we also have a lot of equipment that moves through here to be "burned-in" and configured before it's sent into the field. All this equipment needs a place to live either temporarily or permanently. We have equipment from Ciena, Juniper, Infinera, Cisco, HP and others. Up until now it's be spread across several facilities, most of which had inadequate space, power and/or cooling. So we're very excited about having a wonderful new facility !
Friday, November 20, 2009
Yeah, we can do that !
Do you ever read about a new technology and go, "Man, that's so cool ! We should be doing that !". Only to be disappointed once you started digging into it a bit ?
That's exactly what happened to me after I read the following whitepaper...
Connecting to the Cloud with F5 BIG-IP Solutions and VMware VMotion
Some of you may have read my post a while back about how cool Application Delivery Controllers (aka load-balancers) are. Everything I said is probably true (note to self: reread that post and edit if necessary), but man, once you start digging into what you can do with one of those things - it strikes fear into the heart of every decent network engineer !
And now it looks like these things may bring us the holy grail of virtualization - live migration across a wide-area network ! I'm onboard !!
F5 demo'd this at VM World in late August and it's now late November. We have 4 brand new F5's that aren't in production yet, 2 in each of our data centers separated by about 60 miles. And we have plenty of VM's to throw into the mix. So I figured I'd download the configuration guide and see what it takes to set this up....oh, there's no configuration guide. Hmmm, maybe the documentation is on F5's devcentral site.....no. Okay, well our F5 sales engineer is coming in today so I'll just ask him....well he didn't have very many details and he referenced the documentation on their website...which of course I can't find. And what he did tell me made me realize just how many moving parts are involved and how complex the whole setup really is. Well, this could end up being really cool stuff, but it looks like it's not quite soup yet.
And then there's the issue of whether this is the right way to solve this problem. I'm left with the feeling that this is a really ingenious solution to a problem using the tools we already have but that what we really need are some new tools !
In our case we can theoretically bridge VLANs between our data centers since we have dark fiber. This would theoretically simplify things, but we haven't done this yet because of concerns about bridging loops and broadcast storms taking down BOTH of our data centers! If we could essentially route Ethernet MAC addresses using TRILL or similar functionality developed by the IEEE - perhaps that would offer a simpler solution to this problem !
That's exactly what happened to me after I read the following whitepaper...
Connecting to the Cloud with F5 BIG-IP Solutions and VMware VMotion
Some of you may have read my post a while back about how cool Application Delivery Controllers (aka load-balancers) are. Everything I said is probably true (note to self: reread that post and edit if necessary), but man, once you start digging into what you can do with one of those things - it strikes fear into the heart of every decent network engineer !
And now it looks like these things may bring us the holy grail of virtualization - live migration across a wide-area network ! I'm onboard !!
F5 demo'd this at VM World in late August and it's now late November. We have 4 brand new F5's that aren't in production yet, 2 in each of our data centers separated by about 60 miles. And we have plenty of VM's to throw into the mix. So I figured I'd download the configuration guide and see what it takes to set this up....oh, there's no configuration guide. Hmmm, maybe the documentation is on F5's devcentral site.....no. Okay, well our F5 sales engineer is coming in today so I'll just ask him....well he didn't have very many details and he referenced the documentation on their website...which of course I can't find. And what he did tell me made me realize just how many moving parts are involved and how complex the whole setup really is. Well, this could end up being really cool stuff, but it looks like it's not quite soup yet.
And then there's the issue of whether this is the right way to solve this problem. I'm left with the feeling that this is a really ingenious solution to a problem using the tools we already have but that what we really need are some new tools !
In our case we can theoretically bridge VLANs between our data centers since we have dark fiber. This would theoretically simplify things, but we haven't done this yet because of concerns about bridging loops and broadcast storms taking down BOTH of our data centers! If we could essentially route Ethernet MAC addresses using TRILL or similar functionality developed by the IEEE - perhaps that would offer a simpler solution to this problem !
Wednesday, October 21, 2009
What's up with IPv6 ?
I'm in Dearborn Michigan this week for the NANOG and ARIN meetings. NANOG = North American Network Operators Group. NANOG is very much like the Internet2 Joint Techs Workshops for the commercial sector. It's where network engineers get together to discuss cool new things they're doing. And, like most of these things, it's a lot about social networking - a chance to meet face-to-face with the people you email and IM with every day. ARIN = American Registry of Internet Numbers. ARIN is the non-profit that is responsible for handing out Internet number resources - primarily IP addresses.
IPv6 is a huge topic of discussion this week. Yahoo presented on their IPv6 roll-out which they completed last week. Comcast just presented on their deployment. Google has IPv6 deployed as well. I saw a news story last week that the number of ISPs requesting IPv6 addresses from ARIN has gone way up. In fact, in the last quarter (last month maybe) ARIN received more requests for IPv6 addresses than IPv4 addresses for the first time ever. It seems that IPv6 is *finally* getting some traction. My sense is that this is the real deal and IPv6 is really going to happen now.
It's funny though to see all the hype around IPv6 in the commercial sector. We rolled out IPv6 on the Internet2 network in 2000 and had IPv6 enabled on every data jack at IU around 2001. WRT IPv6, attending a NANOG in 2009 is much like attending an Internet2 Joint Techs Workshop in 2000 or 2001.
IPv6 is a huge topic of discussion this week. Yahoo presented on their IPv6 roll-out which they completed last week. Comcast just presented on their deployment. Google has IPv6 deployed as well. I saw a news story last week that the number of ISPs requesting IPv6 addresses from ARIN has gone way up. In fact, in the last quarter (last month maybe) ARIN received more requests for IPv6 addresses than IPv4 addresses for the first time ever. It seems that IPv6 is *finally* getting some traction. My sense is that this is the real deal and IPv6 is really going to happen now.
It's funny though to see all the hype around IPv6 in the commercial sector. We rolled out IPv6 on the Internet2 network in 2000 and had IPv6 enabled on every data jack at IU around 2001. WRT IPv6, attending a NANOG in 2009 is much like attending an Internet2 Joint Techs Workshop in 2000 or 2001.
Monday, September 28, 2009
Duct work !
I remember the first time I was in a meeting about the deployment of a computer system and there were plumbers at the meeting ! Now there's more plumbing under the raised floors than anything else. Well, last week I got to work with the guys from the sheet metal shop while they fabricated duct work for our Cisco Nexus 7018 switches
. This turns the side-to-side airflow into front-to-back airflow. The sheet metal shop did a great job on very short notice !!
-- Post From My iPhone
. This turns the side-to-side airflow into front-to-back airflow. The sheet metal shop did a great job on very short notice !!
-- Post From My iPhone
Subscribe to:
Posts (Atom)
Posts
