Join Log in
Score:0
avatar Server

iSCSI separation from Ethernet via VLAN

in flag
Ray

I've set up a small cluster of a few servers along with a SAN. The servers are running Ubuntu 20.04 LTS.

Using instructions provided by the vendor (I can't find where I read it before), they suggested that the iSCSI connections between the SAN and the servers should be (or maybe it was "must be"?) separated from any ethernet traffic. Because of this, I've configured two VLANs on our switch -- one for iSCSI traffic and one for ethernet traffic between the servers (which the SAN is not on).

So far, it seems fine. Suppose the Ethernet is on 172.16.100.XXX/24 and iSCSI is on 172.16.200.XXX/24. More specifically, the addresses look something like this:

machine ethernet iSCSI Outside ethernet also?
server 1 172.16.100.1 172.16.200.1 Yes
server 2 172.16.100.2 172.16.200.2 Yes
server 3 172.16.100.3 172.16.200.3 Yes
SAN N/A 172.16.200.4 No

Not surprisingly, I can ssh between servers using either VLAN. That is, from server 2 to server 1, I can do any of the following:

  • ssh 172.16.100.1
  • ssh 172.16.200.1
  • ssh via the outside-visible IP address

What I'm worried about is whether or not I should better separate non-iSCSI traffic from the 172.16.200.X subnet with firewall rules so that port 22 (ssh) is blocked out on all servers.

I'm not concerned about the reverse -- the SAN is only on VLAN 200. It doesn't know VLAN 100 exists so it won't suddenly send iSCSI traffic down that VLAN.

I'm using the Oracle Cluster Filesystem which seems to use port 7777 -- perhaps I should block all ports on the VLAN so that only port 7777 is used? Does having ethernet traffic on an iSCSI network create problems (either lag or errors?) I should be aware of?

Thank you!

146
0 + 3
Michael Hampton avatar
cz flag
Michael Hampton
Does this answer your question? [Mixing SAN iSCSI Traffic with core network traffic, Am I asking for trouble?](https://serverfault.com/questions/272115/mixing-san-iscsi-traffic-with-core-network-traffic-am-i-asking-for-trouble)
0
Reply
in flag
Ray
@MichaelHampton Hmmmm...not quite. The answers seem to be about having a separate physical switch for iSCSI. Or that separating the two with a VLAN will work. We only have the one switch -- I don't make the decisions to buy another switch, though that would be nice. And, I'm already using a VLAN. I guess my question is, do I need to use a firewall (i.e., `ufw`) to prevent Ethernet traffic on the iSCSI VLAN (i.e., all ports except 7777, for example)? Vice-versa isn't necessary since the SAN isn't even on the Ethernet VLAN.
0
Reply
joeqwerty avatar
cv flag
joeqwerty
**Not surprisingly, I can ssh between servers using either VLAN.** - Hmmm... they are connected to both VLAN's. Can you SSH from a host on the non-iSCSI VLAN to a server using the server's iSCSI VLAN ip address? Run a network trace on one server and ssh to another server using it's iSCSI ip address as the target of your ssh session. What's the source interface and ip address of your ssh session?
0
Reply
in flag
Ray
@joeqwerty Thank you for your follow-up! I updated my question with a table which (hopefully) better explains my problem. I'm not sure if it helps... What is the command to do a "network trace"? I didn't know how to do that...
0
Reply
Nikita Kipriyanov avatar
za flag
Nikita Kipriyanov
I suppose "network trace" is traceroute, for example, `mtr`.
0
Reply
Score:1
Zac67 avatar Server
ru flag
Zac67

What I'm worried about is whether or not I should better separate non-iSCSI traffic from the 172.16.200.X subnet with firewall rules so that port 22 (ssh) is blocked out on all servers.

If you use DNS names to connect to other servers and those resolve to the LAN addresses you should be fine. (Alternatively, you can use the LAN IP addresses directly, of course.)

If you really want to disable all non-iSCSI traffic on the SAN you'll need to either

  1. configure all services to bind to LAN IP addresses only
  2. use local firewalls on the servers to filter all unwanted traffic
  3. use ACLs on the iSCSI switch ports to filter all unwanted traffic

If you do filter, just permitting iSCSI and denying everything else is the correct approach.

Does having ethernet traffic on an iSCSI network create problems (either lag or errors?)

The main reason to separate LAN and SAN traffic is that you want to make sure your storage network cannot clog up at all events. If it did, it would rapidly cause I/O errors, in turn causing data loss and even corruption. A (very) low volume of stray traffic isn't anything to really worry about.

However, I'd use the ACL approach if service bindings (#1) aren't practical or if there are other server administrators taking things lightly. E.g. dynamic DNS updating very easily puts your iSCSI IPs in DNS and any inter-server traffic can quickly land in the SAN.

0 + 0
in flag
Ray
Thank you for the clear explanation! The SAN vendor's instructions were to keep the two traffics separate, with no clear explanation why. I'll see how things go; if network performance degrades, I'll look into the options you proposed. Thank you!
0
Reply
Zac67 avatar
ru flag
Zac67
@Ray I guess part of that recommendation is caused by switches being possibly limited in total bandwidth, so frames could be dropped *before* link limits are reached. However, all modern switches are *non-blocking*, ie. only limited by interface speeds (large modular chassis exluded).
1
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.