1. IBM Design
Before PFC can work properly, clients must configure the QoS value on both the storage and host sides. However, IBM Storage only supports configuring a single QoS value at the storage level, which limits the flexibility to assign different PFC priorities to individual host connections.
2. NetApp Design
NetApp allows clients to specify the QoS priority directly through the nvme connect command on the host, providing greater flexibility for configuration.
3. Customer Use Case and Limitation
In the TSMC environment, multiple databases and hosts connect to the same storage system. Different databases may require different PFC priorities based on their workload requirements. For example, one database may need PFC priority 3, while another requires priority 5.
With the current IBM design, configuring different PFC priorities for individual connections is not flexible enough to meet this requirement, particularly when the QoS value must be configured and maintained on both the storage and host sides.
4. Feature Request
Please urgent enhance the IBM design to allow clients to specify the QoS/PFC priority directly through the nvme connect command on the host, similar to the NetApp approach. This would enable per-connection priority configuration, simplify management, and better support multi-database environments like TSMC.
Thank you for your consideration.
Nov-4 update:
TSMC confirmed that PFC priority is configured on a per-host basis. They simply need to use `nvme connect --tos ` to establish a connection between the host and storage. The storage is expected to passively follow the PFC priority rule specified during the connection. Since multiple databases are connected to the FS7600, each host connection can use a different `--tos ` to map to the corresponding PFC priority.
For example, some DB, such as DB2, may be assigned Priority 3, so the host would use -tos 106 when establishing the connection. Other DB, such as Oracle, may be assigned Priority 5, so the host would use -tos 186. Please note all above connections are established from different hosts to single FS7600.
Oct-29 update:
Customer Requirements for NVMe/RDMA QoS Priority Configuration
1. Number of QoS/PFC Priority Rules
The customer requires configuring two or more priority rules on the IBM FlashSystem 7600.
2. Priority Values
The required priority values are 3 and 5, with each rule configured to provide a different traffic-priority behavior.
3. Expected Behavior
The customer expects to configure the QoS/PFC priority for each NVMe/RDMA host connection individually:
Priority 3: Assign a higher traffic priority.
Priority 5: Assign a lower traffic priority.
This would allow the customer to differentiate NVMe/RDMA traffic priorities across host connections based on workload requirements.
4. Proposed NVMe Connect Parameter
The customer proposes using the --tos parameter in the nvme connect command to specify the desired traffic priority for each connection.
5. Command Example and Expected Behavior
TSMC expects to specify only the TOS value in the nvme connect command on the host side. The FlashSystem 7600 should recognize the configured TOS value and map it to the corresponding traffic priority.
Example 1: Higher Priority (Priority 3)
nvme connect -t rdma -n NQN -w Server_IP \ -a Storage_IP -s 4420 -l 1800 --tos=106
Host configuration: TOS = 106
Expected Storage priority mapping: Priority 3
Expected packet capture result: DSCP = 26
Example 2: Lower Priority (Priority 5)
nvme connect -t rdma -n NQN -w Server_IP \ -a Storage_IP -s 4420 -l 1800 --tos=186
Host configuration: TOS = 186
Expected Storage priority mapping: Priority 5
Expected packet capture result: DSCP = 46
Expected Behavior
TSMC should only need to specify the TOS value when establishing an NVMe/RDMA connection. The FlashSystem 7600 should recognize the TOS value and apply the corresponding traffic priority without requiring the customer to configure the priority separately on the storage system for each host connection.
During packet capture, the corresponding DSCP value should be observable in the IP header, allowing TSMC to verify that the intended traffic marking is applied.
6. TOS / DSCP Mapping Provided by the Customer
tos=106 = dscp26 = priority 3
tos=186 = dscp46 = priority 5
Requested Enhancement
Enable per-connection QoS/PFC priority selection for NVMe/RDMA host connections through the nvme connect --tos parameter, allowing multiple priority rules to coexist on the same FlashSystem 7600 and enabling differentiated traffic prioritization.
Looking up documentation on how NetApp handles DSCP and Network Priority mapping, it doesn't looks like NetApp does the mapping from DSCP -> Network priority (in your case DSCP 26 -> Priority 3). NetApp will tag the packets with the DSCP tag and the switch will map these DSCP tags to the corresponding Network Priority that is set on the switch/network.
And from what I could find on the internet, NetApp seems to be allowing per protocol DSCP packet tagging. Does it allow per host DSCP tags? Please check and confirm.
So the ask above, that FS should map DSCP packets to PCP tags seems to be be redundant. That part will be done by the switch network.
From IBM FlashSystems' perspective, we can work on allowing per host or per protocol or (any of those two) as a new feature. Once that is done, DSCP tags will be seen on the outgoing packets from FlashSystems. After that, its the network admin's job to map these to Network Priority tags i.e. 3 or 5 or whatever else the network admin decides.