GPFS安装与设置

提示：在开始之前，请登录您所在组的AIX节点，每个节点打开至少一个窗口，以便后续操作。

目标

在本实验中将练习 如下内容

验证AIX 系统环境 安装GPFS介质 创建GPFS集群

定义NSD(Network Shared Disks) 创建GPFS文件系统

第一节：环境验证

A. AIX与GPFS版本验证

验证每个节点AIX版本

参考GPFS FAQ: http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp # oslevel

_______________________________________________________ 该AIX是否支持GPFS?

该AIX版本下对GPFS 是否有特定版本与patch要求?

________________________________________________________ AIX xlC版本？8.0以上 # lslpp -l |grep xlC|grep rte

___xlC.aix61.rte 10.1.0.2 COMMITTED XL C/C++ Runtime for AIX 6.1 xlC.rte 10.1.0.2 COMMITTED XL C/C++ Runtime

xlC.sup.aix50.rte 9.0.0.1 COMMITTED XL C/C++ Runtime for AIX _____________________________________________________ IBM JDK Version? Jdk 5 # cd /usr; ls java5

# /usr/java5/bin/java -version

__1.5_____________________________________________________

B. 网络配置验证

各节点的hostname？node1,node2,node3 PAlpar10 PAlpar11 PAlpar12

________________________________________________________ 各节点是否能相互ping 通？ # ping node1 # ping node2 # ping node3

C. 验证/etc/hosts文件，应包含组内给节点IP和hostname，如

192.168.1.11 node1 192.168.1.12 node2 192.168.1.13 node3

D. ssh安装、配置与通讯验证

#cd /home/soft/ssh; installp -agYXpd . all 先预览是否能正常安装，输出如下表示通过： SUCCESSES ---------

Filesets listed in this section passed pre-installation verification and will be installed.

Selected Filesets -----------------

openssh.base.client 5.0.0.5302 # Open Secure Shell Commands openssh.base.server 5.0.0.5302 # Open Secure Shell Server openssh.license 5.0.0.5302 # Open Secure Shell License

openssh.man.en_US 5.0.0.5302 # Open Secure Shell Documentat... openssl.base 0.9.8.840 # Open Secure Socket Layer openssl.license 0.9.8.840 # Open Secure Socket License openssl.man.en_US 0.9.8.840 # Open Secure Socket Layer

# installp -agYXd . all 安装后，如下表示正常：

openssl.man.en_US 0.9.8.840 USR APPLY SUCCESS openssl.license 0.9.8.840 USR APPLY SUCCESS openssl.base 0.9.8.840 USR APPLY SUCCESS openssl.base 0.9.8.840 ROOT APPLY SUCCESS openssh.license 5.0.0.5302 USR APPLY SUCCESS openssh.base.client 5.0.0.5302 USR APPLY SUCCESS openssh.base.server 5.0.0.5302 USR APPLY SUCCESS openssh.base.client 5.0.0.5302 ROOT APPLY SUCCESS openssh.base.server 5.0.0.5302 ROOT APPLY SUCCESS openssh.man.en_US 5.0.0.5302 USR APPLY SUCCESS

# ssh-keygen -t rsa

在每个AIX节点生成ssh-key如下： Generating public/private rsa key pair.

Enter file in which to save the key (//.ssh/id_rsa): 回车

Created directory '//.ssh'.

Enter passphrase (empty for no passphrase): 回车 Enter same passphrase again: 回车

Your identification has been saved in //.ssh/id_rsa. Your public key has been saved in //.ssh/id_rsa.pub. The key fingerprint is:

db:0b:23:aa:81:a9:b2:41:33:97:8b:6b:b7:24:fc:2c root@node1

在节点 node1 copy如下文件

# cp /.ssh/id_rsa.pub /.ssh/authorized_keys

从节点node2 copy /.ssh/id_rsa.pub 到节点１/tmp/id_rsa.pub2，需要输入节点2的root密码 从节点node3 copy /.ssh/id_rsa.pub 到节点１/tmp/id_rsa.pub3，需要输入节点3的root密码 # scp node2:/.ssh/id_rsa.pub /tmp/id_rsa.pub2 root@node2's password:

id_rsa.pub 100% 393 0.4KB/s 00:00 # scp node3:/.ssh/id_rsa.pub /tmp/id_rsa.pub3

在node1将/tmp/id_rsa.pub2; /tmp/id_rsa.pub3追加到/.ssh/authorized_keys中 #cat /tmp/id_rsa.pub2 >> /.ssh/authorized_keys #cat /tmp/id_rsa.pub3 >> /.ssh/authorized_keys

将node1上/.ssh/authorized_keys copy到node2,node3 #scp /.ssh/authorized_keys node2:/.ssh/authorized_keys #scp /.ssh/authorized_keys node3:/.ssh/authorized_keys

在node1,node2,node3节点touch 一个/.hushlogin文件，以屏蔽登录回显信息 # touch /.hushlogin

验证node1,node2,node3节点的ssh通讯,如能显示日期验证通过 on node1# ssh node2 date; ssh node3 date on node2# ssh node1 date; ssh node3 date on node3# ssh node1 date; ssh node2 date

E. rsh, dsh 配置

创建并编辑/.rhosts文件， node1 root node2 root node3 root

验证rsh通讯，如能显示日期验证通过 on node1# rsh node2 date; rsh node3 date on node2# rsh node1 date; rsh node3 date on node3# rsh node1 date; rsh node2 date 验证dsh通讯，如能显示日期验证通过 on node1# dsh -n node2 date; dsh -n node3 date

on node2# dsh -n node1 date; dsh -n node3 date on node3# dsh -n node1 date; dsh -n node2 date

F. 检查各AIX系统共享disks #lsdev –Cc disk #lspv

第二节：安装GPFS软件

A. 在node1,node2,node3上分别预览与安装GPFS软件及其PTF(Program Temporary Fix)

# dsh -n node1,node2,node3 \"cd /home/soft/gpfs3.2.1/3.2.1.0; inutoc .; installp -agYXpd . all\" # dsh -n node1,node2,node3 \"cd /home/soft/gpfs3.2.1/3.2.1.0; installp -agYXd . all\"

# dsh -n node1,node2,node3 \"cd /home/soft/gpfs3.2.1/ptf; inutoc .; installp -agYXpd . all\" # dsh -n node1,node2,node3 \"cd /home/soft/gpfs3.2.1/ptf; installp -agYXd . all\"

B. 在node1,node2,node3上分别验证GPFS软件

# dsh -n node1,node2,node3 \"lslpp -L gpfs.*\"

Fileset Level State Type Description (Uninstaller) ---------------------------------------------------------------------------- gpfs.base 3.2.1.13 A F GPFS File Manager

gpfs.docs.data 3.2.1.1 A F GPFS Server Manpages and Documentation gpfs.gui 3.2.1.0 C F GPFS GUI

gpfs.msg.en_US 3.2.1.11 A F GPFS Server Messages - U.S. English

State codes: A -- Applied. B -- Broken. C -- Committed. E -- EFIX Locked.

O -- Obsolete. (partially migrated to newer version) ? -- Inconsistent State...Run lppchk -v.

Type codes:

F -- Installp Fileset P -- Product C -- Component T -- Feature

R -- RPM Package

C. 在node1,node2,node3上,将gpfs执行命令路径加入PATH环境变量

# echo export PATH=$PATH:/usr/lpp/mmfs/bin >>/etc/profile # . /etc/profile

D. 在node1,node2,node3上,确认gpfs PATH环境变量是否已正确

# env|grep PATH # mmlscluster

mmlscluster: 6027-1382 This node does not belong to a GPFS cluster.

mmlscluster: 6027-1639 Command failed. Examine previous error messages to determine cause.

第三节：创建一个双节点的GPFS集群

在本练习中，我们首先建一个单节点的GPFS CLUSTER。以node1为主配置服务器，并将其作为quorum与manager节点，分别以ssh,scp作为 Remote Shell和Remote File Copy命令。 Primary Configuration server (node1): ____________ Verify fully qualified path to ssh and scp: ssh path__/usr/bin/ssh___________ scp path__/usr/bin/scp___________

1) 在node1，使用mmcrcluster创建GPFS Cluster

# mmcrcluster -N node1:manager-quorum -p node1 -r /usr/bin/ssh -R /usr/bin/scp Thu Jan 7 14:36:35 BEIST 2010: 6027-16 mmcrcluster: Processing node node1 mmcrcluster: Command successfully completed

2) 运行mmlscluster查看配置，如：

GPFS cluster information

======================== GPFS cluster name: node1

GPFS cluster id: 12398485736205447539 GPFS UID domain: node1

Remote shell command: /usr/bin/ssh Remote file copy command: /usr/bin/scp

GPFS cluster configuration servers: ----------------------------------- Primary server: node1 Secondary server: (none)

Node Daemon node name IP address Admin node name Designation ----------------------------------------------------------------------------------------------- 1 node1 172.16.69.22 node1 quorum-manager

3) 启动GPFS集群并验证状态

# mmstartup -a # mmgetstate -Lsa

Node number Node name Quorum Nodes up Total nodes GPFS state Remarks --------------------------------------------------------------------------------- 1 node1 1 1 1 active quorum node

Summary information ---------------------

Number of nodes defined in the cluster: 1 Number of local nodes active in the cluster: 1 Number of remote nodes joined in this cluster: 0 Number of quorum nodes defined in the cluster: 1 Number of quorum nodes active in the cluster: 1 Quorum = 1, Quorum achieved

4) 将node2节点加入集群

# mmaddnode –N node2

5) 将node2节点修改为从配置服务器

# mmchcluster -s node2 # mmlscluster

GPFS cluster information ========================

GPFS cluster name: node1

GPFS cluster id: 12398485736205447956 GPFS UID domain: node1

Remote shell command: /usr/bin/ssh Remote file copy command: /usr/bin/scp

GPFS cluster configuration servers: ----------------------------------- Primary server: node1 Secondary server: (none)

Node Daemon node name IP address Admin node name Designation ---------------------------------------------------------------------------------- 1 node1 172.16.69.22 node1 quorum-manager 2 node2 172.16.69.29 node2

6) 启动node2并查看状态

# mmstartup -N node2 # mmgetstate –Lsa

Node number Node name Quorum Nodes up Total nodes GPFS state Remarks ------------------------------------------------------------------------------------ 1 node1 1 1 2 active quorum node 2 node2 1 1 2 active

Summary information ---------------------

Number of nodes defined in the cluster: 2 Number of local nodes active in the cluster: 2 Number of remote nodes joined in this cluster: 0 Number of quorum nodes defined in the cluster: 1 Number of quorum nodes active in the cluster: 1 Quorum = 1, Quorum achieved

7) 查看GPFS集群信息

集群名字？_______________

node1,node2的IP地址? ___________________________ 此版本GPFS built的日期？___________________________

8) 使用4个hdisk创建NSD

在本阶段练习中，我们按照如下规则创建NSD

所有nsd容纳data和metadata 不指定STORAGE POOL 不指定FailureGroup 不指定主/备服务器节点

在node1上建立/home/gpfs 目录，创建并编辑/home/gpfs/nsds文件，内容如下:

#mkdir /home/gpfs #vi /home/gpfs/nsds #more /home/gpfs/nsds

#DiskName:ServerList::DiskUsage:FailureGroup:DesiredName:StoragePool hdiskw:::dataAndMetadata::nsd1: hdiskx:::dataAndMetadata::nsd2: hdisky:::dataAndMetadata::nsd3: hdiskz:::dataAndMetadata::nsd4:

备份nsds

#cp /home/gpfs/nsds /home/gpfs/nsds.bak

注：这里我们练习中使用4个hdisk，所以示例中列举4个，真实环境请视实际hdisk数量调整

创建NSD，这里-v 选项缺省值为yes，表示创建NSD是验证该hdisk是否已被NSD格式化了，

当然对一个全新系统，我们可以确定hdisk未被格式化，可以制定-v no跳过检查。

# mmcrnsd -F /home/gpfs/nsds -v yes #mmdel “nds1;nsd2;nsd3” #mmlsnsd 9) 创建文件系统

指定文件系统blocksize大小为：K MOUNT路径为：gpfs # mmcrfs /gpfs gpfs1 “nsd1;nsd2” –B k (自我测试)

# mmcrfs /gpfs gpfs1 –F /home/gpfs/nsds –B k

GPFS: 6027-531 The following disks of gpfs1 will be formatted on node node1: nsd1: size 4194304 KB nsd2: size 4194304 KB nsd3: size 4194304 KB nsd4: size 4194304 KB

GPFS: 6027-540 Formatting file system ...

GPFS: 6027-535 Disks up to size 40 GB can be added to storage pool 'system'. Creating Inode File

Creating Allocation Maps

Clearing Inode Allocation Map Clearing Block Allocation Map

Formatting Allocation Map for storage pool 'system'

GPFS: 6027-572 Completed creation of file system /dev/gpfs1.

mmcrfs: 6027-1371 Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.

#mmdelfs gpfs1 #mmlsfs all #mmdf gpfs1

10) 验证GPFS集群文件系统 # mmlsfs gpfs1

mmlsfs gpfs1

flag value description

---- ---------------- ----------------------------------------------------- -f 2048 Minimum fragment size in bytes -i 512 Inode size in bytes

-I 8192 Indirect block size in bytes

-m 1 Default number of metadata replicas -M 2 Maximum number of metadata replicas -r 1 Default number of data replicas -R 2 Maximum number of data replicas -j cluster Block allocation type

-D nfs4 File locking semantics in effect -k all ACL semantics in effect

-a 1048576 Estimated average file size

-n 32 Estimated number of nodes that will mount file system -B 65536 Block size

-Q none Quotas enforced

none Default quotas enabled

-F 33792 Maximum number of inodes -V 10.01 (3.2.1.5) File system version -u yes Support for large LUNs? -z no Is DMAPI enabled? -L 2097152 Logfile size

-E yes Exact mtime mount option -S no Suppress atime mount option

-K whenpossible Strict replica allocation option -P system Disk storage pools in file system -d nsd1;nsd2;nsd3;nsd4 Disks in file system -A yes Automatic mount option -o none Additional mount options -T /gpfs Default mount point

# mmmount all –a

# df -g

Filesystem GB blocks Free %Used Iused %Iused Mounted on /dev/hd4 1.00 0.95 5% 3914 2% / /dev/hd2 4.00 1.70 58% 53285 12% /usr /dev/hd9var 1.00 0.92 8% 4774 3% /var /dev/hd3 3.00 2.98 1% 179 1% /tmp

/dev/hd1 60.00 54.98 9% 985 1% /home /proc - - - - - /proc

/dev/hd10opt 10.00 5.88 42% 37933 3% /opt /dev/gpfs1 16.00 15.91 1% 4038 12% /gpfs

# mmdf gpfs1

disk disk size failure holds holds free KB free KB

name in KB group metadata data in full blocks in fragments --------------- ------------- -------- -------- ----- -------------------- ------------------- Disks in storage pool: system (Maximum disk size allowed is 40 GB)

nsd1 4194304 -1 yes yes 4170240 ( 99%) 104 ( 0%) nsd2 4194304 -1 yes yes 4170176 ( 99%) 124 ( 0%) nsd3 4194304 -1 yes yes 4170240 ( 99%) 124 ( 0%) nsd4 4194304 -1 yes yes 4170048 ( 99%) 118 ( 0%) ------------- -------------------- -------------------

(pool total) 16777216 16680704 ( 99%) 470 ( 0%)

============= ==================== =================== (total) 16777216 16680704 ( 99%) 470 ( 0%)

Inode Information -----------------

Number of used inodes: 4038 Number of free inodes: 29754 Number of allocated inodes: 33792 Maximum number of inodes: 33792

当前文件系统已使用了多少个inode？____________ 还有多少空闲的inodes？_____________