Solaris IPMP技术是Solaris下的一个IP multi path技术，我之前有简单介绍过。今天重点介绍下在11gR2环境下，Solaris IPMP用作public network的方法，并进行简单的测试和验证。
1）首先在进行GI安装之前，要配置好IPMP，本例中ipmp0中有两个member e1000g0 and e1000g3
# ifconfig -a
e1000g0: flags=1000843 mtu 1500 index 2
inet 10.137.13.84 netmask fffff800 broadcast 10.137.15.255
groupname ipmp0
ether 0:21:28:23:e7:44
e1000g0:1: flags=1040843 mtu 1500 index 2
inet 10.137.13.208 netmask fffff800 broadcast 10.137.15.255
e1000g0:2: flags=1040843 mtu 1500 index 2
inet 10.137.13.202 netmask fffff800 broadcast 10.137.15.255
e1000g1: flags=1000843 mtu 1500 index 3
inet 10.137.22.90 netmask fffffc00 broadcast 10.137.23.255
ether 0:21:28:23:e7:45
e1000g1:1: flags=1000843 mtu 1500 index 3
inet 169.254.245.156 netmask ffff0000 broadcast 169.254.255.255
e1000g2: flags=1000843 mtu 1500 index 4
inet 10.137.25.4 netmask fffffc00 broadcast 10.137.27.255
ether 0:21:28:23:e7:46
e1000g3: flags=1000843 mtu 1500 index 5
inet 10.137.14.123 netmask fffff800 broadcast 10.137.15.255
groupname ipmp0
ether 0:21:28:23:e7:47
#
2）安装GI的时候，记得选择两个网卡作为public network，最新版本的RAC中，OUI已经加了检查，如果只有部分IPMP group的member选择作为public，那么会给出警告信息。否则在这种情况下，会出现不一致的问题。
# ./oifcfg getif
e1000g0 10.137.8.0 global public
e1000g1 10.137.20.0 global cluster_interconnect
e1000g3 10.137.8.0 global public
# ./oifcfg iflist
e1000g0 10.137.8.0
e1000g1 10.137.20.0
e1000g1 169.254.0.0
e1000g2 10.137.24.0
e1000g3 10.137.8.0
#
3）安装完成之后，检查network资源的信息，可以看到两个网卡都已经被加进入了。如果发现缺少了部分网卡，可以通过crsctl modify res 的命令进行修改。
# . ./crsctl g stat res ora.net1.network -p
NAME=ora.net1.network
TYPE=ora.network.type
ACL=owner:root:rwx,pgrp:root:r-x,other::r--,group:oinstall:r-x,user:crsusr:r-x
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX%
ALIAS_NAME=
AUTO_START=restore
CHECK_INTERVAL=1
DEFAULT_TEMPLATE=
DEGREE=1
DESCRIPTION=Oracle Network resource
ENABLED=1
LOAD=1
LOGGING_LEVEL=1
NLS_LANG=
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=60
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=60
START_DEPENDENCIES=
START_TIMEOUT=0
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=
STOP_TIMEOUT=0
TYPE_VERSION=2.2
UPTIME_THRESHOLD=1d
USR_ORA_AUTO=static
USR_ORA_ENV=
USR_ORA_IF=e1000g0 e1000g3 < =====在这里
USR_ORA_NETMASK=255.255.248.0
USR_ORA_SUBNET=10.137.8.0
VERSION=11.2.0.3.0
4）现在我们来进行验证，看IPMP的漂移功能是否生效，并且内部漂移的时候不会对stack产生影响。
首先把其中一个成员e1000g3 断掉，利用IPMP提供的管理命令if_mpadm来完成即可。可以看到e1000g3的IP地址漂移到了e1000g0上，变成e1000g0:3。从系统日志可以看出，并且GI stack并没受到影响。
# if_mpadm -d e1000g3
# ifconfig -a
e1000g0: flags=1000843 mtu 1500 index 2
inet 10.137.13.84 netmask fffff800 broadcast 10.137.15.255
groupname ipmp0
ether 0:21:28:23:e7:44
e1000g0:1: flags=1040843 mtu 1500 index 2
inet 10.137.13.208 netmask fffff800 broadcast 10.137.15.255
e1000g0:2: flags=1040843 mtu 1500 index 2
inet 10.137.13.202 netmask fffff800 broadcast 10.137.15.255
e1000g0:3: flags=1000843 mtu 1500 index 2
inet 10.137.14.123 netmask fffff800 broadcast 10.137.15.255
e1000g1: flags=1000843 mtu 1500 index 3
inet 10.137.22.90 netmask fffffc00 broadcast 10.137.23.255
ether 0:21:28:23:e7:45
e1000g1:1: flags=1000843 mtu 1500 index 3
inet 169.254.245.156 netmask ffff0000 broadcast 169.254.255.255
e1000g2: flags=1000843 mtu 1500 index 4
inet 10.137.25.4 netmask fffffc00 broadcast 10.137.27.255
ether 0:21:28:23:e7:46
e1000g3: flags=89000842 mtu 0 index 5
inet 0.0.0.0 netmask 0
groupname ipmp0
ether 0:21:28:23:e7:47
# tail -1 5 /var/adm/messages
...
Sep 27 06:05:52 dnagad07 in.mpathd[219]: [ID 832587 daemon.error] Successfully failed over from NIC e1000g3 to NIC e1000g0
#
# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.DB.dg
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.LISTENER.lsnr
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.asm
ONLINE ONLINE dnagad05 Started
ONLINE ONLINE dnagad06 Started
ONLINE ONLINE dnagad07 Started
ONLINE ONLINE dnagad08 Started
ora.gsd
OFFLINE OFFLINE dnagad05
OFFLINE OFFLINE dnagad06
OFFLINE OFFLINE dnagad07
OFFLINE OFFLINE dnagad08
ora.net1.network
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.ons
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.registry.acfs
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE dnagad08
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE dnagad05
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE dnagad07
ora.cvu
1 ONLINE ONLINE dnagad05
ora.dnagad05.vip
1 ONLINE ONLINE dnagad05
ora.dnagad06.vip
1 ONLINE ONLINE dnagad06
ora.dnagad07.vip
1 ONLINE ONLINE dnagad07
ora.dnagad08.vip
1 ONLINE ONLINE dnagad08
ora.oc4j
1 ONLINE ONLINE dnagad08
ora.orcldb.db
1 ONLINE ONLINE dnagad07
ora.orcldb.orcldb1.inst
1 ONLINE ONLINE dnagad05
ora.orcldb.orcldb2.inst
1 ONLINE ONLINE dnagad06
ora.orcldb.orcldb3.inst
1 ONLINE ONLINE dnagad07
ora.orcldb.orcldb4.inst
1 ONLINE ONLINE dnagad08
ora.scan1.vip
1 ONLINE ONLINE dnagad08
ora.scan2.vip
a.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE dnagad07
ora.cvu
1 ONLINE ONLINE dnagad05
ora.dnagad05.vip
1 ONLINE ONLINE dnagad05
ora.dnagad06.vip
1 ONLINE ONLINE dnagad06
ora.dnagad07.vip
1 ONLINE ONLINE dnagad07
ora.dnagad08.vip
1 ONLINE ONLINE dnagad08
ora.oc4j
1 ONLINE ONLINE dnagad08
ora.orcldb.db
1 ONLINE ONLINE dnagad07
ora.orcldb.orcldb1.inst
1 ONLINE ONLINE dnagad05
ora.orcldb.orcldb2.inst
1 ONLINE ONLINE dnagad06
ora.orcldb.orcldb3.inst
1 ONLINE ONLINE dnagad07
ora.orcldb.orcldb4.inst
1 ONLINE ONLINE dnagad08
ora.scan1.vip
1 ONLINE ONLINE dnagad08
ora.scan2.vip
1 ONLINE ONLINE dnagad05
ora.scan3.vip
1 ONLINE ONLINE dnagad07
#
5）这时候，我们恢复e1000g3，会发现一切又恢复了正常。
# if_mpadm -r e1000g3
# ifconfig -a
e1000g0: flags=1000843 mtu 1500 index 2
inet 10.137.13.84 netmask fffff800 broadcast 10.137.15.255
groupname ipmp0
ether 0:21:28:23:e7:44
e1000g0:1: flags=1040843 mtu 1500 index 2
inet 10.137.13.208 netmask fffff800 broadcast 10.137.15.255
e1000g0:2: flags=1040843 mtu 1500 index 2
inet 10.137.13.202 netmask fffff800 broadcast 10.137.15.255
e1000g1: flags=1000843 mtu 1500 index 3
inet 10.137.22.90 netmask fffffc00 broadcast 10.137.23.255
ether 0:21:28:23:e7:45
e1000g1:1: flags=1000843 mtu 1500 index 3
inet 169.254.245.156 netmask ffff0000 broadcast 169.254.255.255
e1000g2: flags=1000843 mtu 1500 index 4
inet 10.137.25.4 netmask fffffc00 broadcast 10.137.27.255
ether 0:21:28:23:e7:46
e1000g3: flags=1000843 mtu 1500 index 5
inet 10.137.14.123 netmask fffff800 broadcast 10.137.15.255
groupname ipmp0
ether 0:21:28:23:e7:47
# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.DB.dg
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.LISTENER.lsnr
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.asm
ONLINE ONLINE dnagad05 Started
ONLINE ONLINE dnagad06 Started
ONLINE ONLINE dnagad07 Started
ONLINE ONLINE dnagad08 Started
ora.gsd
OFFLINE OFFLINE dnagad05
OFFLINE OFFLINE dnagad06
OFFLINE OFFLINE dnagad07
OFFLINE OFFLINE dnagad08
ora.net1.network
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.ons
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.registry.acfs
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE dnagad08
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE dnagad05
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE dnagad07
ora.cvu
1 ONLINE ONLINE dnagad05
ora.dnagad05.vip
1 ONLINE ONLINE dnagad05
ora.dnagad06.vip
1 ONLINE ONLINE dnagad06
ora.dnagad07.vip
1 ONLINE ONLINE dnagad07
ora.dnagad08.vip
1 ONLINE ONLINE dnagad08
ora.oc4j
1 ONLINE ONLINE dnagad08
ora.orcldb.db
1 ONLINE ONLINE dnagad07
ora.orcldb.orcldb1.inst
1 ONLINE ONLINE dnagad05
ora.orcldb.orcldb2.inst
1 ONLINE ONLINE dnagad06
ora.orcldb.orcldb3.inst
1 ONLINE ONLINE dnagad07
ora.orcldb.orcldb4.inst
1 ONLINE ONLINE dnagad08
ora.scan1.vip
1 ONLINE ONLINE dnagad08
ora.scan2.vip
1 ONLINE ONLINE dnagad05
ora.scan3.vip
1 ONLINE ONLINE dnagad07
#
# tail -4 /var/adm/messages
....
Sep 27 06:05:52 dnagad07 in.mpathd[219]: [ID 832587 daemon.error] Successfully failed over from NIC e1000g3 to NIC e1000g0
Sep 27 06:06:23 dnagad07 in.mpathd[219]: [ID 620804 daemon.error] Successfully failed back to NIC e1000g3
#
6）下面用e1000g0来作试验，结果同样。
# if_mpadm -d e1000g0
# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=89000842 mtu 0 index 2
inet 0.0.0.0 netmask 0
groupname ipmp0
ether 0:21:28:23:e7:44
e1000g1: flags=1000843 mtu 1500 index 3
inet 10.137.22.90 netmask fffffc00 broadcast 10.137.23.255
ether 0:21:28:23:e7:45
e1000g1:1: flags=1000843 mtu 1500 index 3
inet 169.254.245.156 netmask ffff0000 broadcast 169.254.255.255
e1000g2: flags=1000843 mtu 1500 index 4
inet 10.137.25.4 netmask fffffc00 broadcast 10.137.27.255
ether 0:21:28:23:e7:46
e1000g3: flags=1000843 mtu 1500 index 5
inet 10.137.14.123 netmask fffff800 broadcast 10.137.15.255
groupname ipmp0
ether 0:21:28:23:e7:47
e1000g3:1: flags=1000843 mtu 1500 index 5
inet 10.137.13.84 netmask fffff800 broadcast 10.137.15.255
e1000g3:2: flags=1040843 mtu 1500 index 5
inet 10.137.13.208 netmask fffff800 broadcast 10.137.15.255
e1000g3:3: flags=1040843 mtu 1500 index 5
inet 10.137.13.202 netmask fffff800 broadcast 10.137.15.255
lo0: flags=2002000849 mtu 8252 index 1
inet6 ::1/128
# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.DB.dg
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.LISTENER.lsnr
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.asm
ONLINE ONLINE dnagad05 Started
ONLINE ONLINE dnagad06 Started
ONLINE ONLINE dnagad07 Started
ONLINE ONLINE dnagad08 Started
ora.gsd
OFFLINE OFFLINE dnagad05
OFFLINE OFFLINE dnagad06
OFFLINE OFFLINE dnagad07
OFFLINE OFFLINE dnagad08
ora.net1.network
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.ons
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.registry.acfs
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE dnagad08
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE dnagad05
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE dnagad07
ora.cvu
1 ONLINE ONLINE dnagad05
ora.dnagad05.vip
1 ONLINE ONLINE dnagad05
ora.dnagad06.vip
1 ONLINE ONLINE dnagad06
ora.dnagad07.vip
1 ONLINE ONLINE dnagad07
ora.dnagad08.vip
1 ONLINE ONLINE dnagad08
ora.oc4j
1 ONLINE ONLINE dnagad08
ora.orcldb.db
1 ONLINE ONLINE dnagad07
ora.orcldb.orcldb1.inst
1 ONLINE ONLINE dnagad05
ora.orcldb.orcldb2.inst
1 ONLINE ONLINE dnagad06
ora.orcldb.orcldb3.inst
1 ONLINE ONLINE dnagad07
ora.orcldb.orcldb4.inst
1 ONLINE ONLINE dnagad08
ora.scan1.vip
1 ONLINE ONLINE dnagad08
ora.scan2.vip
1 ONLINE ONLINE dnagad05
ora.scan3.vip
1 ONLINE ONLINE dnagad07
# tail 04 -4 /var/adm/messages
Sep 27 05:38:56 dnagad07 CLSD: [ID 770310 daemon.notice] The clock on host dnagad07 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Sep 27 06:05:52 dnagad07 in.mpathd[219]: [ID 832587 daemon.error] Successfully failed over from NIC e1000g3 to NIC e1000g0
Sep 27 06:06:23 dnagad07 in.mpathd[219]: [ID 620804 daemon.error] Successfully failed back to NIC e1000g3
Sep 27 06:07:07 dnagad07 in.mpathd[219]: [ID 832587 daemon.error] Successfully failed over from NIC e1000g0 to NIC e1000g3
#
#
# if_mpadm -r e1000g0
# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
inet 10.137.13.84 netmask fffff800 broadcast 10.137.15.255
groupname ipmp0
ether 0:21:28:23:e7:44
e1000g0:1: flags=1040843 mtu 1500 index 2
inet 10.137.13.208 netmask fffff800 broadcast 10.137.15.255
e1000g0:2: flags=1040843 mtu 1500 index 2
inet 10.137.13.202 netmask fffff800 broadcast 10.137.15.255
e1000g1: flags=1000843 mtu 1500 index 3
inet 10.137.22.90 netmask fffffc00 broadcast 10.137.23.255
ether 0:21:28:23:e7:45
e1000g1:1: flags=1000843 mtu 1500 index 3
inet 169.254.245.156 netmask ffff0000 broadcast 169.254.255.255
e1000g2: flags=1000843 mtu 1500 index 4
inet 10.137.25.4 netmask fffffc00 broadcast 10.137.27.255
ether 0:21:28:23:e7:46
e1000g3: flags=1000843 mtu 1500 index 5
inet 10.137.14.123 netmask fffff800 broadcast 10.137.15.255
groupname ipmp0
ether 0:21:28:23:e7:47
lo0: flags=2002000849 mtu 8252 index 1
inet6 ::1/128
# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.DB.dg
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.LISTENER.lsnr
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.asm
ONLINE ONLINE dnagad05 Started
ONLINE ONLINE dnagad06 Started
ONLINE ONLINE dnagad07 Started
ONLINE ONLINE dnagad08 Started
ora.gsd
OFFLINE OFFLINE dnagad05
OFFLINE OFFLINE dnagad06
OFFLINE OFFLINE dnagad07
OFFLINE OFFLINE dnagad08
ora.net1.network
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.ons
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
ora.registry.acfs
ONLINE ONLINE dnagad05
ONLINE ONLINE dnagad06
ONLINE ONLINE dnagad07
ONLINE ONLINE dnagad08
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE dnagad08
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE dnagad05
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE dnagad07
ora.cvu
1 ONLINE ONLINE dnagad05
ora.dnagad05.vip
1 ONLINE ONLINE dnagad05
ora.dnagad06.vip
1 ONLINE ONLINE dnagad06
ora.dnagad07.vip
1 ONLINE ONLINE dnagad07
ora.dnagad08.vip
1 ONLINE ONLINE dnagad08
ora.oc4j
1 ONLINE ONLINE dnagad08
ora.orcldb.db
1 ONLINE ONLINE dnagad07
ora.orcldb.orcldb1.inst
1 ONLINE ONLINE dnagad05
ora.orcldb.orcldb2.inst
Recent Comments