What do you get when you cross iSCSI with lossless Ethernet? A lot of confusion.

I do quite a lot of presentations regarding converged networks, including Fibre Channel, whether native or over Ethernet (i.e., FCoE), iSCSI, NAS, etc. The hardest part about these presentations are combating some of the expectations that audiences have, considering they may come from server backgrounds, network backgrounds, or storage backgrounds.

Why is this important? Quite frankly, because like fish who don't know they live in the water, they have come to grow unaware of their own environmental backgrounds. They tend to forget the assumptions with which they make their decisions. Ethernet and Fibre Channel networking people haveverydifferent fundamental philosophies about the way their networks run.

In the world of converged networks, this can cause some, er, unintended consequences. 

The Confusion

Recently I had a friend -we'll call him Pete (because that's the first fictional name that came to mind) stop by while I was having lunch and ask if he could "ask a quick question."

(Why is it that the questions are quick, but the answers are so, so long? Just one of those ponderables of life, I suppose.)

"Sure, what's up?" I said. It's not like I was hungry anyway.

"I've been trying to figure out this whole converged networking thing," Pete said, and I could see he wasn't sure what the right words to use were.

"You mean FCoE?" I asked. Usually most of the questions I get surround FCoE. Go figure.Quelle Surprise!

"No, actually, iSCSI."

"Okay, sure," I said. "What's the question?" Technically speaking, iSCSIisa convergence technology. You can run block-level storage on a VLAN running over the same equipment that handles traditional Ethernet. We've been doing this foryears.

Pete took a deep breath, and then blurted out, "I've been told that you can do converged networks with iSCSI, but in order to do converged networks we need to use DCB, so does that mean that we have to run iSCSI over DCB?"

Now it was my turn to be confused. "I'm confused," I said. Coy about my feelings, I am. "By DCB, do you mean lossless Ethernet?"

Pete nodded.

"Well," I said, "It may be that the problem you're having is that DCB is not the same thing as lossless."

Pete looked like he was about to cry. "What?!" he said. "But everyone keeps saying that you need DCB for FCoE because it's lossless and that's how you get converged networks..." His voice trailed off.

I motioned for him to take a seat. He did. I reached over to grab a napkin and pulled out a pen. "Let me try to clarify a couple of things," I said. Indicating the napkin I said, "I hope you don't mind, I tend to think visually."

He nodded. I swear, he was pouting. "Cheer up," I said. "We'll get you on the right page."

Clearing up the Terminology

Let's try to break it down a little. "DCB" is actually a category of standard documents. It's not just one particular thing," I said, starting to write everything down.

The DCB "Family" of Standard Docs

"You see," I said, tapping at each of the different lines, "there are many parts of DCB, not just lossless. The important thing to keep in mind is that just because something is included as part of the standard, doesn't mean it's required for every single case."

Pete nodded. I took a bite of my food, and instantly regretted it. I hate talking with my mouth full.

"Take thith, for exthample," I said, trying not to spit food across the table, and swallowed hurriedly. I pointed to the last line on the napkin. "While this is part of the DCB, it doesn't apply in all convergence cases. You don't need this for FCoE, for example."

"Why not?" Pete asked.

I waved my hand. "It's a bit long to get into it, but I'll send you the link to that later. For now, let's just say that these documents are designed to help you understand a way to solve a problem. It's not necessarily theonlyway, just a way that the industry had standardized. Make sense?"

"So far," he admitted.

"So," I continued. "When someone says that you need DCB, they may be talking about one of the solutions inoneof these documents, or two, but not necessarily anyspecificone of them. You certainly don't need to doallof them for every possible problem."

"Okay," he said. "So what are they talking about when they say they need DCB for iSCSI?"

iSCSI and DCB

"That's a good question," I admitted. "I'm not surewhatthey mean. We've been doing converged networks with iSCSI and LAN for a while now, and we certainly don't need to do DCB in order for that to work."

"But I keep hearing about 'Lossless iSCSI,'" he protested. "Don't you need DCB to do that?"

"Well, yeah," I said. "That's kind of written into the name, isn't it? But do you needallof DCB to make it work? Or just the lossless part?"

He thought about that for a minute. "A fair point," he said.

"It seems to me that if you're going to be making a DCB argument for iSCSI -assuming we'renottalking lossless here -then you really could use the ETS part of the standard for guaranteeing minimum bandwidth."

"What do you mean?"

"When you start looking at running iSCSI on 10GbE links," I said, drawing, "If you want to set aside a minimum amount of bandwidth, well, ETS can help with that."

You can guarantee a minimum bandwidth setting with ETS

"What does it do?" Peter asked.

"Effectively it allows you to place bandwidth into groups and assign bandwidth percentages to those groups," I said. "For instance, if you want to give iSCSI 30% of the bandwidth of this pipe, you could do that. With Cisco switches we accomplish this through our QoS settings."

Peter stroked his chin thoughtfully. "And you don't need DCB -I mean, lossless -for this?"

"Not unless you want to run lossless iSCSI," I said. "But you have to be very clear on why you would want to do that."

Peter's eyes grew wide. "Yes, exactly!" he said. "That's what I need to know!"

I took this as an opportunity to grab another bite of my lunch. "Can you be more specific?" I asked, grimacing at the food. It was cold. I should have ordered a salad.

"So if we can have lossless traffic for Fibre Channel and FCoE, won't it help with iSCSI too? Why don't we run all iSCSI traffic over lossless Ethernet?"

"Because we have different design principles for lossy traffic like iSCSI than we do for lossless traffic like Fibre Channel and FCoE," I replied.

"But it's all block storage, right?" he asked.

"Yes, that's true," I said. "But how a server accesses data on a storage array is a separate issue than how we network that access to the storage."

I could see by the look on his face that I lost him. Out came napkin number 3.

"Look at it this way," I said. "One of the things that makes iSCSI so popular is that it doesn't require all the know-how that Fibre Channel does. It's not as specialized, right?"

Peter nodded.

"One of the beautiful things about iSCSI is that, not only is it cheap, but the network itself does a lot of the heavy lifting for you. All you need are the IP addresses of the host and target and then the network takes care of the rest."

He frowned. "There's more to it than that," he said.

"Yes, I know," I said, a little more dismissively than I meant. "But in essence setting up iSCSI is easy because there are a lot of things youdon'thave to do that you would in a Fibre Channel environment."

"True."

The Tyranny of SCSI

"Okay," I continued. "Let's start off with SCSI. SCSI makes some assumptions about the way these two devices talk to each other. For instance, SCSI assumes that there is a 1-to-1 relationship between host and target, and that the exchanges between the two devices are always in sequence."

SCSI needs a one-to-one connection

"And SCSI can't do error correction," Peter interjected.

"Well," I said, hesitantly, "technically itcan, but it's so difficult and can be done in so few cases that for all intents and purposes it's useless for what we're trying to do here."

"Thanks," Peter said sarcastically. "Way to be pedantic."

I looked at him for a moment. "Anywaaaaaay..."

Even in a network, SCSI thinks it's a 1:1 relationship

"When we moved the disk outside of the host and inserted a network in between them," I continued drawing on the napkin until I had a respectable mockup of a network (sometimes my artwork amazes even myself). "We still had to maintain that 1:1 relationship, right?"

"Right."

"So, this is one of the reasons why FC requires a lossless network in the first place: it is very difficult to fix the data if it gets out of sequence or if frames get dropped. But that's not what happens in iSCSI."

"With iSCSI we use TCP to create sessions between the host and the target, so that when Ethernet packets get sent all crazy-like over the network -" I drew squiggles across the napkin -"TCP will keep track of the sequence and reassemble them in order and present it to the SCSI layer."

I could have sworn this was the way to Alburquerque!

"Voil