GPFS的安装、配置、管理

GPFS的安装、配置、管理

1

目 录

1

GPFS安装、配置、管理和维护 .......................................................................... 1 1.1 GPFS安装 ...................................................................................................................... 1 1.2 GPFS配置 ...................................................................................................................... 1

1.2.1 集群节点配置 ..................................................................................................... 1 1.2.2 GPFS NSD磁盘配置 ........................................................................................ 2 1.2.3 优化GPFS 集群配置 ........................................................................................ 4 1.2.4 创建GPFS 文件系统 ........................................................................................ 7 1.2.5 配置变更 .............................................................................................................. 7 1.3 GPFS管理和维护 ....................................................................................................... 16

I

1 GPFS安装、配置、管理和维护

本部分围绕XX公司HPIS系统实际情况，描述GPFS的安装、配置过程，以及GPFS管理和维护方法。

1.1 GPFS安装

GPFS 3.3及补丁使用AIX系统标准installp方式安装，安装命令示例： # installp -agYXd . all 查看安装结果; # lslpp -l |grep gpfs

gpfs.base 3.3.0.16 COMMITTED GPFS File Manager gpfs.base 3.3.0.16 COMMITTED GPFS File Manager gpfs.docs.data 3.3.0.1 COMMITTED GPFS Server Manpages and

1.2 GPFS配置

1.2.1 集群节点配置

➢ 准备node列表文件，节点属性：manager或client， quorum或nonquorum # more /home/GPFS/nodes hpis1:manager-quorum hpis2:manager-quorum

1

➢ 创建RAC两节点的GPFS集群

# mcrcluster -N /home/GPFS/nodes -p hpis1 -s hpis2 -r /usr/bin/rsh -R /usr/bin/rcp -C cls_hpis

-p,-s分别表示主，辅集群管理服务器

-r /usr/bin/rsh -R /usr/bin/rcp表示rsh,rcp方式进行集群管理时使用rsh,rcp管理方式；也可以通过配置ssh，使用ssh,scp方式 ➢ 查看集群配置 #mmlscluster

1.2.2 GPFS NSD磁盘配置

因为RAC只有两个Server节点，为保证GPFS文件系统的高可用性，需要将两台Server均设为quorum节点，同时从两个存储和一台Server的本地盘中分别取一个盘作为tiebreaker quorum Disk。

➢ NSD准备

生成NSD文件，格式如： # more /home/GPFS/nsd hdisk3:hpis2::descOnly:1:nsd00: hdisk29:::dataAndMetadata:2:nsd01: hdisk30:::dataAndMetadata:2:nsd02: hdisk31:::dataAndMetadata:2:nsd03: hdisk32:::dataAndMetadata:2:nsd04:

2

hdisk33:::dataAndMetadata:2:nsd05: hdisk59:::dataAndMetadata:3:nsd06: hdisk60:::dataAndMetadata:3:nsd07: hdisk61:::dataAndMetadata:3:nsd08: hdisk62:::dataAndMetadata:3:nsd09: hdisk63:::dataAndMetadata:3:nsd10: 注：

1. 这里分别将hpis2节点的本地盘，两个存储的failgroup分别设为1，2，3 2. hdisk3本地盘设为descOnly，存储Disk设为dataAndMetadata

➢ 创建NSD

# mmcrnsd -F /home/GPFS/nsd -v yes

生成NSD后，/home/GPFS/nsd会被改写，如： # hdisk3:::descOnly:1:nsd00: nsd00:::descOnly:1::

# hdisk29:::dataAndMetadata:2:nsd01: nsd01:::dataAndMetadata:2::

# hdisk30:::dataAndMetadata:2:nsd02: nsd02:::dataAndMetadata:2::

# hdisk31:::dataAndMetadata:2:nsd03: nsd03:::dataAndMetadata:2::

3

# hdisk32:::dataAndMetadata:2:nsd04: nsd04:::dataAndMetadata:2::

# hdisk33:::dataAndMetadata:2:nsd05: nsd05:::dataAndMetadata:2::

# hdisk59:::dataAndMetadata:3:nsd06: nsd06:::dataAndMetadata:3::

# hdisk60:::dataAndMetadata:3:nsd07: nsd07:::dataAndMetadata:3::

# hdisk61:::dataAndMetadata:3:nsd08: nsd08:::dataAndMetadata:3::

# hdisk62:::dataAndMetadata:3:nsd09: nsd09:::dataAndMetadata:3::

# hdisk63:::dataAndMetadata:3:nsd10: nsd10:::dataAndMetadata:3::

1.2.3 优化GPFS 集群配置

➢ 查看当前集群配置 # mmlscluster

GPFS cluster information

========================

4

GPFS cluster name: cls_hpis.hpis1 GPFS cluster id: 752142207565323869 GPFS UID domain: cls_hpis.hpis1 Remote shell command: /usr/bin/rsh Remote file copy command: /usr/bin/rcp

GPFS cluster configuration servers: ----------------------------------- Primary server: hpis2 Secondary server: hpis1

Node Daemon node name IP address Admin node name Designation

-----------------------------------------------------------------------------------------------

1 hpis1 10.1.1.90 hpis1 quorum-manager

2 hpis2 10.1.1.91 hpis2 quorum-manager

# mmlsconfig

5

Configuration data for cluster cls_hpis.hpis1: ---------------------------------------------- clusterName cls_hpis.hpis1 clusterId 752142207565323869 autoload yes

minReleaseLevel 3.3.0.2 dmapiFileHandleSize 32 maxblocksize 8M maxFilesToCache 16384 maxStatCache 65536 maxMBpS 8192 pagepool 2048M

pagepoolMaxPhysMemPct 80 tiebreakerDisks nsd00;nsd01;nsd06 failureDetectionTime 10 adminMode central

File systems in cluster cls_hpis.hpis1: --------------------------------------- /dev/oradata

6

此处pagepool与tiebreakerDisks参数最重要

➢ 修改集群配置命令:mmchconfig，有些配置需要先shutdown GPFS集群 #mmchconfig pagepool=3072M

#mmchconfig tiebreakerDisks=\"nsd00;nsd01;nsd06\"

1.2.4 创建GPFS 文件系统

# mmcrfs oradata -F /home/GPFS/nsd -T /oradata -A yes -K always -B 2m -E no -m 2 -M 2 -n 32 -Q no -r 2 -R 2 -S yes -v no 注：

➢ 其mount点为/oradata, blocksize为2m;注意此参数已经创建，不可修改，

blocksize一般要参照存储端LUN设置，以获得最优性能；

➢ -m 2 -M 2表示写两份metadata数据，-r 2 -R 2表示写两份data数据； ➢ 我们已经将数据盘的failgroup分别设为2、3，GPFS会自动将数据均衡写入不

同failgroup中；

1.2.5 配置变更

1.2.5.1 节点变更

➢ Cluster和filesystem manager角色 当前manager角色如下

7

# mmlsmgr

file system manager node ---------------- ------------------ oradata 10.1.1.90 (hpis1)

Cluster manager node: 10.1.1.90 (hpis1)

➢ 变更Cluster或filesystem manager角色为hpis2 #mmchmgr oradata hpis2 #mmchmgr -c hpis2

➢

增加节点使用mmaddnode命令增加，如：mmaddnode –N othernode1 ➢ 节点quorum属性变更

# mmchnode --quorum -N othernode1 # mmchnode --nonquorum -N othernode2 # mmchnode --manager -N othernode1 # mmchnode --client -N othernode1

注：目前RAC两个的属性应均设为manger, quorum，若新增server节点，可设

为manger, quorum，新增client节点建议设为client,nonquorum

8

1.2.5.2 NSD disk变更

➢ 增加NSD disk

新增NSD时，请从两个存储中挑选disk，成对增加，在AIX系统中可以通过lscfg –vp|grep hdisk查看某个hdisk来源与哪个存储，如： # lscfg -vp|grep hdisk hdisk4

U78AA.001.WZSGP8Z-P1-C4-T1-W20140080E518F286-L1000000000000 MPIO DS5020 Disk ......

hdisk34

U78AA.001.WZSGP8Z-P1-C4-T1-W20140080E518E3DA-L1000000000000 MPIO DS5020 Disk

通过蓝色部分可以区分hdisk的来源，增加hdisk时，要注意failgroup的设置和现有来自同存储的failgroup一样

编写待新增nsd文件，如： # more /home/GPFS/nsd2

hdisk28:::dataAndMetadata:2:nsd11: hdisk58:::dataAndMetadata:3:nsd12: # mmcrnsd -F /home/GPFS/nsd2 -v yes mmcrnsd: Processing disk hdisk28 mmcrnsd: Processing disk hdisk58

9

mmcrnsd: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.

# more /home/GPFS/nsd2

# hdisk28:::dataAndMetadata:2:nsd11: nsd11:::dataAndMetadata:2::

# hdisk58:::dataAndMetadata:3:nsd12: nsd12:::dataAndMetadata:3:: ➢ 将新NSD增加到文件系统中

# mmadddisk oradata -F /home/GPFS/nsd2

The following disks of oradata will be formatted on node hpis1: nsd11: size 209715200 KB nsd12: size 209715200 KB Extending Allocation Map

Checking Allocation Map for storage pool 'system' Completed adding disks to file system oradata.

mmadddisk: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.

➢ 当前NSD列表

10

# mmlsnsd -aL

File system Disk name NSD volume ID NSD servers ---------------------------------------------------------------------------------------------

oradata nsd00 0A70255B4E8260E1 oradata nsd01 0A70255B4E826051 oradata nsd02 0A70255B4E826052 oradata nsd03 0A70255B4E826053 oradata nsd04 0A70255B4E826054 oradata nsd05 0A70255B4E826055 oradata nsd06 0A70255B4E826056 oradata nsd07 0A70255B4E826057 oradata nsd08 0A70255B4E826058 oradata nsd09 0A70255B4E826059 oradata nsd10 0A70255B4E82605A oradata nsd11 0A70255B4E855BFB oradata nsd12 0A70255B4E855BFC ➢ 当前/oradata文件系统的disk列表 # mmlsdisk oradata -L

11

(directly attached) (directly attached) (directly attached) (directly attached) (directly attached) (directly attached) (directly attached) (directly attached) (directly attached) (directly attached) (directly attached) (directly attached) hpis2

disk driver sector failure holds holds storage

name type size group metadata data status availability disk id pool remarks

------------ -------- ------ ------- -------- ----- ------------- ------------ ------- ------------ ---------

nsd00 nsd 512 1 no no ready up 1 system desc

nsd01 nsd 512 2 yes yes ready up 2 system desc

nsd02 nsd 512 2 yes yes ready up 3 system

nsd03 nsd 512 2 yes yes ready up 4 system

nsd04 nsd 512 2 yes yes ready up 5 system

nsd05 nsd 512 2 yes yes ready up 6 system

nsd06 nsd 512 3 yes yes ready up 7 system desc

12

nsd07 nsd 512 3 yes yes ready up 8 system

nsd08 nsd 512 3 yes yes ready up 9 system

nsd09 nsd 512 3 yes yes ready 10 system

nsd10 nsd 512 3 yes yes ready 11 system

nsd11 nsd 512 2 yes yes ready 12 system

nsd12 nsd 512 3 yes yes ready 13 system

Number of quorum disks: 3 Read quorum value: 2 Write quorum value: 2 ➢

当前/oradata文件系统大小即状态

# mmdf oradata

disk disk size failure holds holds free KB

name in KB group metadata data blocks in fragments

13

up up up up free KB in full

--------------- ------------- -------- -------- ----- -------------------- -------------------

Disks in storage pool: system (Maximum disk size allowed is 4.5 TB) nsd00 143373952 1 no no 0 ( 0%) nsd01 (100%) nsd02 (100%) nsd03 (100%) nsd04 (100%) nsd05 (100%) nsd11 (100%) nsd07 (100%) nsd08 (100%) 0 ( 0%)

209715200 5248 ( 0%)

209715200 4352 ( 0%)

209715200 5888 ( 0%)

209715200 4416 ( 0%)

209715200 5632 ( 0%)

209715200 1984 ( 0%)

209715200 00 ( 0%)

209715200 4544 ( 0%)

2 yes 2 yes 2 yes 2 yes 2 yes 2 yes 3 yes 3 yes yes 2088048 yes 2002144 yes 2088856 yes 208887808 yes 2086000 yes 209711104 yes 2002144 yes 2081904

14

nsd09 209715200 3 yes yes 208885760 (100%) 3072 ( 0%)

nsd10 209715200 3 yes yes 2083952 (100%) 7872 ( 0%)

nsd06 209715200 3 yes yes 2000096 (100%) 38 ( 0%)

nsd12 209715200 3 yes yes 209711104 (100%) 1984 ( 0%)

------------- -------------------- -------------------

(pool total) 2659956352 2508369920 ( 94%) 55040 ( 0%)

============= ==================== ===================

(total) 2659956352 2508369920 ( 94%) 55040 ( 0%)

Inode Information -----------------

Number of used inodes: 4095

15

Number of free inodes: 528385 Number of allocated inodes: 532480 Maximum number of inodes: 2188019

1.3 GPFS管理和维护

➢ 启动所有/单个节点 mmstartup –a mmstartup -N hpis1 mmstartup -N hpis2

➢ Mount 所有/单个节点文件系统 mmmount all –a mmmount all

➢ umount 所有/单个节点文件系统 mmumount all -a mmumount all

➢ 停止所有/单个节点文件系统 mmshutdown -a mmshutdown –N hpis1 mmshutdown –N hpis2

16

➢ 查看GPFS状态 # mmgetstate -Las

Node number Node name Quorum Nodes up Total nodes GPFS state Remarks

------------------------------------------------------------------------------------ 1 hpis1 1* 2 2 active quorum node

2 hpis2 1* 2 2 active quorum node

Summary information ---------------------

Number of nodes defined in the cluster: 2 Number of local nodes active in the cluster: 2 Number of remote nodes joined in this cluster: 0 Number of quorum nodes defined in the cluster: 2 Number of quorum nodes active in the cluster: 2 Quorum = 1*, Quorum achieved ➢ 查看文件系统mount状态 # mmlsmount all -L

17

File system oradata is mounted on 2 nodes: 10.1.1.90 hpis1 10.1.1.91 hpis2

➢ 监控GPFS文件系统IO情况(mmpmon) 如：

mmpmon -i /home/mon_gpfs -d 2000 -r 1000 -s -t 60

➢ 存储故障恢复后手工同步文件系统

存储发生故障后，故障存储的NSD将变更down状态，mmlsdisk查看结果示例如下：

# mmlsdisk oradata -L

disk driver sector failure holds holds storage

name type size group metadata data status availability disk id pool remarks

------------ -------- ------ ------- -------- ----- ------------- ------------ ------- ------------ ---------

nsd00 nsd 512 1 no no ready up 1 system desc

18

nsd01 nsd 512 2 yes yes ready up 2 system desc

nsd02 nsd 512 2 yes yes ready up 3 system desc

nsd03 nsd 512 4 system

nsd04 nsd 512 5 system

nsd05 nsd 512 6 system

nsd06 nsd 512 down 7 system nsd07 nsd 512 down 8 system nsd08 nsd 512 down 9 system nsd09 nsd 512 down 10 system nsd10 nsd 512 down 11 system Number of quorum disks: 3

2 yes 2 yes 2 yes 3 yes

3 yes

3 yes

3 yes

3 yes yes ready yes ready yes ready yes ready yes ready yes ready yes ready yes ready up up up

19

Read quorum value: 2 Write quorum value: 2

故障存储恢复后，我们可以选择合适时机对每个处于down状态的disk进行手工同步文件系统，命令如下：

# mmchdisk oradata start -d nsd06 # mmchdisk oradata start -d nsd07 # mmchdisk oradata start -d nsd08 # mmchdisk oradata start -d nsd09 # mmchdisk oradata start -d nsd10 通过后结果如下： # mmlsdisk oradata -L

disk driver sector failure holds holds storage

name type size group metadata data status availability disk id pool remarks

------------ -------- ------ ------- -------- ----- ------------- ------------ ------- ------------ ---------

nsd00 nsd 512 1 no no ready up 1 system desc

nsd01 nsd 512 2 yes yes ready up 2 system desc

20

nsd02 nsd 512 2 yes yes ready up 3 system

nsd03 nsd 512 2 yes yes ready up 4 system

nsd04 nsd 512 5 system

nsd05 nsd 512 6 system

nsd06 nsd 512 7 system desc

nsd07 nsd 512 8 system

nsd08 nsd 512 9 system

nsd09 nsd 512 10 system

nsd10 nsd 512 11 system

Number of quorum disks: 3 Read quorum value: 2 Write quorum value: 2

2 yes 2 yes 3 yes 3 yes 3 yes 3 yes 3 yes 21

yes ready yes ready yes ready yes ready yes ready yes ready yes ready up up up up up up up

22