Use Multiple Network Interfaces

Oracle
Audience
Public
Technology Integrations
Oracle
Source Type
Documentation

To enhance the network bandwidth, make sure to have multiple network interfaces on the client. These multiple interfaces can be configured on a single subnet or on multiple subnets.

Single subnet

IO performance on a dNFS environment with multiple interfaces and single subnet is limited to the speed of the first interface that is picked by the OS. This is because the OS returns the first path when multiple paths are available on the subnet and hence the traffic is always routed through the first path.

For example, in the setup below, the FlashBlade has a single IP address on which the NFS filesystem will be mounted from the client and the client has two interfaces. 
Client interface       NFS server

ens7f0 10.21.108.193   10.21.108.10

ens7f1 10.21.108.194   10.21.108.10



[oracle@rlxora-b01-11-06 ~]$ route -n

Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

0.0.0.0         10.21.108.1     0.0.0.0         UG    0      0        0 ens7f0

10.21.108.0     0.0.0.0         255.255.255.0   U     0      0        0 ens7f0 <-- first route

10.21.108.0     0.0.0.0         255.255.255.0   U     0      0        0 ens7f1 <-- route ignored

As per the routing table, both traffic can go through the first interface (ens7f0) as the destination and the mask fits for both routes and OS will invariably choose the first route.

In this case, to enhance the bandwidth, it is recommended to use NIC bonding at the client level. Oracle support document (Doc ID 833481.1) provides other means to enable multiple paths in a single subnet using static routing but it doesn’t address the issue with availability when a NIC fails or network cable is pulled as the routing table will not be updated.

Multiple subnets

“Direct NFS client best practices recommend always to use multipaths in separate subnets”.

Oracle recommends using separate subnet for each interface and it supports up to four subnets. With multiple subnets, there is no need to bond the network interfaces to aggregate the bandwidth across the available interfaces. The routing will be automatic in the case of multiple subnets. 
Client interface       NFS server

ens7f0 10.21.108.193   10.21.108.10

ens7f1 10.21.107.194   10.21.107.10



[oracle@rlxora-b01-11-06 ~]$ route -n

Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

0.0.0.0         10.21.108.1     0.0.0.0         UG    0      0        0 ens7f0

10.21.108.0     *               255.255.255.0   U     0      0        0 ens7f0

10.21.107.0     *               255.255.255.0   U     0      0        0 ens7f1

In this case, these are two dynamic routes and based on the traffic, the route is selected automatically.

As such, if you decide to use multiple subnets, it should be configured at both the client and the FlashBlade side as well. Multiple subnets can be configured in FlashBlade GUI under the Network Settings.

Make sure to update the oranfstab with the subnets and the mount point details. In RAC environment, all RAC nodes should have the appropriate oranfstab file configured.