API tools faq
paste
42n4

XEN7_CEPH

42n4
Jun 16th, 2016
333
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Bash 26.48 KB | None | 0 0
raw download clone embed print report
  1. #Configure additional cards NIC1 (192.168.0.22?) i NIC2 (10.10.10.1?)
  2. #USE IT IN COMMAND LINE (without #):
  3. #curl http://pastebin.com/raw/SNnhH3rx > xendeletelocalsr.sh; tr -d "\015" < xendeletelocalsr.sh > xendeletelocalsrnew.sh
  4. #sh xendeletelocalsrnew.sh
  5. #I have managed to configure 4 servers of beta3 dundee with glusterfs and ctdb and fully working ha without SPOF.
  6. #I have four NICs: NIC0 management of xenserver 192.168.0.20*, NIC1 gluster vm client 192.168.0.22*, bond NIC2-3 10.10.10.1*, VIPs #from ctdb 192.168.0.21*. In /etc/hosts #I keep only 10.10.10.1x server ips and use them in glusterfs volume creation and in #consequence in glusterfs backend traffic.
  7. #The main compilation script
  8. #Removing local storage and configuring ctdb, /etc/hosts:
  9. #http://pastebin.com/SNnhH3rx
  10. #At the end you should:
  11. #gluster peer probe servers
  12. #and
  13. #init_gluster3 "xen" "1" "2" "3" "vol0" 3 #replica 3 at the end!
  14. #create SR nfs and iso, make vm and install xen tools and enable ha with 3 failovers.
  15. #You are encouraged to send some patches or opinions newfuntek(at)gmail.com
  16. #Here are some screenshots of the glusterfs sr in the xenserver pool:
  17. #http://s17.postimg.org/3y47n8w27/glusterfsxenserv03.jpg
  18. #http://s17.postimg.org/n4heqfcjz/glusterfsxenserv01.jpg
  19. #http://s17.postimg.org/gs29gl9hr/glusterfsxenserv02.jpg
  20. ####################################################################################################################################
  21. #Internet help
  22. #Xenserver doc links
  23. #http://www.poppelgaard.com/citrix-xenserver-6-5
  24. #http://www.gluster.org/community/documentation/index.php/GlusterFS_Documentation
  25. #xenserver tutorials and cheatsheets
  26. #http://www.admin-magazine.com/HPC/Articles/GlusterFS
  27. #http://www.slashroot.in/gfs-gluster-file-system-complete-tutorial-guide-for-an-administrator
  28. #https://virtualizationandstorage.wordpress.com/2010/11/15/xenserver-commands/
  29. #http://krypted.com/unix/using-the-xensource-command-line-interface/
  30. #http://funwithlinux.net/2013/02/glusterfs-tips-and-tricks-centos/
  31. #http://xmodulo.com/category/xenserver
  32. #compilation of xenserver
  33. #https://discussions.citrix.com/topic/372069-does-libvmi-work-on-xenserver/
  34. #glusterfs slides info
  35. #http://rajesh-joseph.blogspot.com/2015/11/usenix-lisa-2015-tutorial-on-glusterfs.html
  36. #https://github.com/gluster/gluster-tutorial/blob/master/LISA-GlusterFS-Introduction.pdf
  37. #https://github.com/gluster/gluster-tutorial/blob/master/LISA-GlusterFS-Hands-on.pdf
  38. #compilation of glusterfs
  39. #http://majentis.com/?p=319
  40. #różne możliwości glusterfs jak w RAID
  41. #http://sysadm.pp.ua/linux/glusterfs-setup.html
  42. #glusterfs performance
  43. #https://blog.secretisland.de/xenserver-mit-glusterfs/
  44. #http://blog.dradmin.co.in/?tag=glusterfs-how-to
  45. #https://gluster.readthedocs.org/en/latest/Administrator%20Guide/Managing%20Volumes/
  46. #https://www.mail-archive.com/users@ovirt.org/msg31079.html
  47. #http://www.gluster.org/community/documentation/index.php/Performance_Testing
  48. #glusterfs on lvm
  49. #https://support.rackspace.com/how-to/getting-started-with-glusterfs-considerations-and-installation/
  50. #glusterfs profiling (delays in ops)
  51. #https://gluster.readthedocs.org/en/latest/Administrator%20Guide/Monitoring%20Workload/
  52. #xenserver glusterfs discussion - they said not possible ;)
  53. #http://discussions.citrix.com/topic/366729-about-xenserver-glusterfs/page-2
  54. #xenserver ha
  55. #http://docs.citrix.com/de-de/xencenter/6-1/xs-xc-protection/xs-xc-pools-ha/xs-xc-pools-ha-about.html
  56. #http://xapi-project.github.io/features/HA/HA.html
  57. #https://support.citrix.com/servlet/KbServlet/download/21018-102-664364/High%20Availability%20for%20Citrix%20XenServer.pdf
  58. #https://xen-orchestra.com/blog/xenserver-and-vm-high-availability/
  59. #https://discussions.citrix.com/topic/367150-ntp-ha-self-fencing/page-2#entry1884695
  60. #http://discussions.citrix.com/topic/333343-need-help-interpreting-xha-logs/
  61. #failure ha restart
  62. #http://support.citrix.com/article/CTX128275
  63. #http://citrixtechs.com/blog/help-my-citrix-xenserver-poolmaster-is-down-2/
  64. #http://discussions.citrix.com/topic/292757-local-storage-unplugged-and-un-repairable/
  65. #xenserver iscsi
  66. #http://gluster.readthedocs.org/en/latest/Administrator%20Guide/GlusterFS%20iSCSI/#Running_the_target_on_the_gluster_client
  67. #xenserver multipath
  68. #http://docs.citrix.com/content/dam/docs/en-us/xenserver/xenserver-61/xs-design-multipathing-config.pdf
  69. #xenserver iptables discussion
  70. #http://discussions.citrix.com/topic/235974-cannot-add-new-nfs-virtual-disk-sr-can-add-iso-library-nfs-sr/page-2
  71. #rebalance nics (bond for glusterfs)
  72. #http://www.gluster.org/pipermail/gluster-users/2014-November/019463.html
  73. #http://www.gluster.org/pipermail/gluster-users/2014-November/019466.html
  74. #multi nic splitnetwork for gluster in near future
  75. #https://www.gluster.org/pipermail/gluster-users/2015-May/021815.html
  76. #http://www.gluster.org/community/documentation/index.php/Features/SplitNetwork
  77. #http://pl.atyp.us/hekafs.org/index.php/2013/01/split-and-secure-networks-for-glusterfs/
  78. #host in vm inside vm config
  79. #https://wiki.openstack.org/wiki/XenServer/VirtualBox
  80. #reinstall xenserver host
  81. #http://support.citrix.com/article/CTX136342
  82. #discuss rejoin xenserver
  83. #http://discussions.citrix.com/topic/303468-server-removed-from-pool-after-failure-now-that-it-is-back-it-can-not-rejoin-pool-and-emergency-master-reset-does-not-work/
  84. #increase dom0 memory
  85. #http://support.citrix.com/article/CTX134951
  86. #find rpm
  87. #http://rpm.pbone.net/
  88. #win admin
  89. #win7-10: Net user administrator /active:yes
  90. #umount /dev/v/l
  91. #e2fsck -f /dev/v/l
  92. #resize2fs -p /dev/v/l 4G
  93. #lvreduce -L -8.9G /dev/
  94.  
  95. ####################################################################################################################################
  96. #turn off nfs i iptables
  97. chkconfig nfs off
  98. service nfs stop
  99. rm -f /etc/exports
  100. service ntpd restart
  101. chkconfig ntpd on
  102. service iptables stop
  103. chkconfig iptables off
  104. systemctl disable firewalld
  105. systemctl stop firewalld
  106.  
  107. #turn off selinux
  108. sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/sysconfig/selinux
  109. #change the linux name to Centos for CEPH-DEPLOY
  110. mv /etc/centos-release /etc/centos-release.xs -f
  111. echo 'CentOS release 7.2 (Final)' > /etc/centos-release
  112.  
  113. #new package update lists
  114. #https://discussions.citrix.com/topic/372069-does-libvmi-work-on-xenserver/
  115. #yum install epel-release
  116. #http://www.tecmint.com/how-to-enable-epel-repository-for-rhel-centos-6-5/
  117. #http://elrepo.org/tiki/tiki-index.php
  118. #rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
  119. #rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
  120. #wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-6.noarch.rpm
  121. #rpm -ivh epel-release-7-6.noarch.rpm
  122. #rpm -Uvh http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el7.rf.x86_64.rpm
  123. #rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-7.rpm
  124. #rpm -Uvh http://repo.webtatic.com/yum/el7/webtatic-release.rpm
  125. sed -i -e "/baseurl=http:\/\/www.uk/d" /etc/yum.repos.d/CentOS-Base.repo
  126. sed -i -e "/mirrorlist/d" /etc/yum.repos.d/CentOS-Base.repo
  127. sed -i -e "s/^#base/base/" /etc/yum.repos.d/CentOS-Base.repo
  128. sed -i -e "s/enabled=0/enabled=1/" /etc/yum.repos.d/CentOS-Base.repo
  129. sed -i -e "s/\$releasever/7/" /etc/yum.repos.d/CentOS-Base.repo
  130. yum install epel-release -y
  131. yum install centos-release-gluster -y
  132. sed -i -e "s/enabled=0/enabled=1/" /etc/yum.repos.d/CentOS-Gluster-3.7.repo
  133. sed -i -e "s/\$releasever/7/" /etc/yum.repos.d/CentOS-Gluster-3.7.repo
  134. sed -i -e "s/buildlogs.centos.org\/centos\/7\/storage\/\$basearch\/gluster-3.7/buildlogs.centos.org\/centos\/7\/storage\/\$basearch\/gluster-3.8/" /etc/yum.repos.d/CentOS-Gluster-3.7.repo
  135. #sed -i -e "s/enabled=0/enabled=1/" /etc/yum.repos.d/epel-testing.repo
  136. yum clean all
  137. yum repolist enabled
  138. yum -y install deltarpm
  139. #yum update --skip-broken -y
  140. yum -y install vim-enhanced mc yum-utils curl e4fsprogs epel-rpm-macros
  141. curl ix.io/client > /usr/local/bin/ix
  142. chmod +x /usr/local/bin/ix
  143.  
  144. yum install -y glusterfs glusterfs-api-devel python-gluster ctdb ceph-deploy bind-utils xfsprogs git dnsmasq xfsprogs ansible
  145. #yum install -y open-vm-tools ## TYLKO DLA MASZYN WIRTUALNYCH VMWARE
  146.  
  147. #run serwis glusterfs
  148. systemctl unmask rpcbind.socket
  149. service glusterd start
  150. service glusterd status
  151. service glusterfsd start
  152. service glusterfsd status
  153. chkconfig glusterd on
  154. chkconfig glusterfsd on
  155.  
  156. wget http://halizard.org/release/noSAN-combined/halizard_nosan_installer_1.4.7
  157. chmod 755 halizard_nosan_installer_1.4.7  
  158. sed -i 's/`uname -r`/3.10.0+2/' halizard_nosan_installer_1.4.7
  159.  
  160. #curl http://ix.io/oxr > /etc/ntp.conf.ix
  161. #curl http://ix.io/ojO > /etc/ntp.conf.ix
  162. #gluster10_4nodes
  163. #curl http://ix.io/op8 > /etc/ntp.conf.ix
  164. #gluster_200_201
  165. #curl http://ix.io/oKN > /etc/ntp.conf.ix
  166. #gluster_201_202_203
  167. #curl http://ix.io/QLt > /etc/ntp.conf.ix
  168. #gluster10_3nodes
  169. curl http://ix.io/Tm5 > /etc/ntp.conf.ix
  170. tr -d "\015" < /etc/ntp.conf.ix > /etc/ntp.conf
  171. echo OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -x" > /etc/sysconfig/ntpd
  172. echo SYNC_HWCLOCK=yes >> /etc/sysconfig/ntpd
  173. hwclock --systohc
  174. service ntpd stop
  175. service ntpd start
  176. ntpstat -s
  177. ntpq -p
  178. ntpstat -s
  179.  
  180. echo "#/opt/xensource/bin/xe-toolstack-restart" >> /etc/rc.d/rc.local
  181. echo "service glusterd restart" >> /etc/rc.d/rc.local
  182. echo "service glusterfsd restart" >> /etc/rc.d/rc.local
  183. echo "service ctdb restart" >> /etc/rc.d/rc.local
  184. chmod 755  /etc/rc.d/rc.local
  185.  
  186. echo "xe host-emergency-ha-disable force=true" > /usr/local/bin/restartfence
  187. echo "/opt/xensource/bin/xe-toolstack-restart" >> /usr/local/bin/restartfence
  188. echo "service glusterd restart" >> /usr/local/bin/restartfence
  189. echo "service glusterfsd restart" >> /usr/local/bin/restartfence
  190. echo "service ctdb restart" >> /usr/local/bin/restartfence  
  191. chmod 755 /usr/local/bin/restartfence
  192.  
  193. echo "/opt/xensource/bin/xe-toolstack-restart" >> /usr/local/bin/restartoolxen
  194. echo "service glusterd restart" >> /usr/local/bin/restartoolxen
  195. echo "service glusterfsd restart" >> /usr/local/bin/restartoolxen
  196. echo "service ctdb restart" >> /usr/local/bin/restartoolxen
  197. chmod 755 /usr/local/bin/restartoolxen
  198.  
  199. echo "service glusterd restart" > /usr/local/bin/restartdodatki
  200. echo "service glusterfsd restart" >> /usr/local/bin/restartdodatki
  201. echo "service ctdb restart" >> /usr/local/bin/restartdodatki
  202. chmod 755 /usr/local/bin/restartdodatki
  203.  
  204. echo "service glusterd restart" > /usr/local/bin/restartgluster
  205. echo "service glusterfsd restart" >> /usr/local/bin/restartgluster
  206. chmod 755 /usr/local/bin/restartgluster
  207.  
  208. #init_gluster4 "xen" "1" "2" "3" "4" "vol0" 4
  209. #server="xen"; host01="1"; host02="2"; host03="3"; host04="4"; volume="vol0"; replica=4;
  210. cat <<__EOF__ > /usr/local/bin/init_gluster4
  211. server=\$1  
  212. host01=\$2  
  213. host02=\$3  
  214. host03=\$4
  215. host04=\$5  
  216. volume=\$6
  217. replica=\$7
  218. gluster peer status
  219. #na dowolnym jednym wykonac
  220. #glusterfs dwa volumeny vol0 (iso) I vol1 (gfs) na sda3 i sda2  
  221. #gluster volume stop \$volume force
  222. #gluster volume delete \$volume force
  223. gluster volume create \$volume replica \$replica \${server}\${host01}:/export/\${server}\${host01}-\$volume \${server}\${host02}:/export/\${server}\${host02}-\$volume \${server}\${host03}:/export/\${server}\${host03}-\$volume \${server}\${host04}:/export/\${server}\${host04}-\$volume force  
  224. gluster volume set \$volume nfs.port 2049  
  225. gluster volume set \$volume performance.cache-size 128MB  
  226. gluster volume set \$volume performance.write-behind-window-size 4MB  
  227. gluster volume set \$volume performance.io-thread-count 64  
  228. gluster volume set \$volume performance.io-cache on  
  229. gluster volume set \$volume performance.read-ahead on  
  230. gluster volume start \$volume  
  231. gluster volume info \$volume  
  232. gluster volume status \$volume  
  233. #montowanie NFS SR pod localhost lub VIP :/vol0 lub vol1  
  234. __EOF__
  235. chmod 755 /usr/local/bin/init_gluster4
  236.  
  237. #init_gluster3 "xen" "1" "2" "3" "vol0" 3
  238. #server="xen"; host01="1"; host02="2"; host03="3"; volume="vol0"; replica=3;
  239. cat <<__EOF__ > /usr/local/bin/init_gluster3
  240. server=\$1  
  241. host01=\$2  
  242. host02=\$3  
  243. host03=\$4
  244. volume=\$5
  245. replica=\$6
  246. gluster peer status
  247. #na dowolnym jednym wykonac
  248. #glusterfs dwa volumeny vol0 (iso) I vol1 (gfs) na sda3 i sda2  
  249. #gluster volume stop \$volume force
  250. #gluster volume delete \$volume force
  251. gluster volume create \$volume replica \$replica \${server}\${host01}:/export/\${server}\${host01}-\$volume \${server}\${host02}:/export/\${server}\${host02}-\$volume \${server}\${host03}:/export/\${server}\${host03}-\$volume force  
  252. gluster volume set \$volume nfs.port 2049  
  253. gluster volume set \$volume performance.cache-size 128MB  
  254. gluster volume set \$volume performance.write-behind-window-size 4MB  
  255. gluster volume set \$volume performance.io-thread-count 64  
  256. gluster volume set \$volume performance.io-cache on  
  257. gluster volume set \$volume performance.read-ahead on  
  258. gluster volume start \$volume  
  259. gluster volume info \$volume  
  260. gluster volume status \$volume  
  261. #montowanie NFS SR pod localhost lub VIP :/vol0 lub vol1  
  262. __EOF__
  263. chmod 755 /usr/local/bin/init_gluster3
  264.  
  265. #init_gluster2 "xen" "1" "2" "vol0" 2
  266. #server="xen"; host01="1"; host02="2"; volume="vol0"; replica=2;
  267. cat <<__EOF__ > /usr/local/bin/init_gluster2
  268. server=\$1  
  269. host01=\$2  
  270. host02=\$3  
  271. volume=\$4
  272. replica=\$5
  273. gluster peer status
  274. #na dowolnym jednym wykonac
  275. #glusterfs dwa volumeny vol0 (iso) I vol1 (gfs) na sda3 i sda2  
  276. #gluster volume stop \$volume force
  277. #gluster volume delete \$volume force
  278. gluster volume create \$volume replica \$replica \${server}\${host01}:/export/\${server}\${host01}-\$volume \${server}\${host02}:/export/\${server}\${host02}-\$volume force  
  279. gluster volume set \$volume nfs.port 2049  
  280. gluster volume set \$volume performance.cache-size 128MB  
  281. gluster volume set \$volume performance.write-behind-window-size 4MB  
  282. gluster volume set \$volume performance.io-thread-count 64  
  283. gluster volume set \$volume performance.io-cache on  
  284. gluster volume set \$volume performance.read-ahead on  
  285. gluster volume start \$volume  
  286. gluster volume info \$volume  
  287. gluster volume status \$volume  
  288. #montowanie NFS SR pod localhost lub VIP :/vol0 lub vol1  
  289. __EOF__
  290. chmod 755 /usr/local/bin/init_gluster2
  291.  
  292. #init_brick "/dev/sdb1" "vol2"
  293. cat <<__EOF__ > /usr/local/bin/init_brick
  294. dev4gfs=\$1  
  295. volume=\$2  
  296. hname=\`hostname\`
  297. mkfs.ext4 -m 0 -j \$dev4gfs
  298. tune2fs -O dir_index -o user_xattr \$dev4gfs
  299. mkdir -p /export/\${hname}-\${volume}
  300. echo  "\$dev4gfs /export/\${hname}-\${volume} ext4 rw,noatime,nodiratime,user_xattr,barrier=0,data=ordered 1 2" >> /etc/fstab
  301. mount -a
  302. __EOF__
  303. chmod 755 /usr/local/bin/init_brick
  304.  
  305. #reformat_brick "/dev/sda2" "vol1"
  306. cat <<__EOF__ > /usr/local/bin/reformat_brick
  307. dev4gfs=\$1  
  308. volume=\$2  
  309. hname=\`hostname\`
  310. umount /export/\${hname}-\${volume}
  311. mkfs.ext4 -m 0 -j \$dev4gfs
  312. tune2fs -O dir_index -o user_xattr \$dev4gfs
  313. mount /export/\${hname}-\${volume}
  314. __EOF__
  315. chmod 755 /usr/local/bin/reformat_brick
  316.  
  317. #removeallandformat "/dev/sdb"
  318. cat <<__EOF__ > /usr/local/bin/removeallandformat
  319. #!/bin/bash
  320. # Script to automatically format all partitions on /dev/sda and create
  321. # a single partition for the whole disk
  322. # Remove each partition
  323. for v_partition in \`parted -s \$1 print|awk '/^ / {print \$1}'\`; do umount \$1\${v_partition}; parted -s \$1 rm \${v_partition}; done
  324. # Find size of disk
  325. v_disk=\`parted -s \$1 print|awk '/^Disk \/dev/ {print \$3}'| sed 's/[Mm][Bb]//'\`
  326. # Create single partition
  327. parted -s \$1 mkpart primary 1 \${v_disk}
  328. # Format the partition
  329. # mke2fs -T ext3 \${1}1
  330. __EOF__
  331. chmod 755  /usr/local/bin/removeallandformat
  332.  
  333. ####################################################################################################################################
  334. #ctdb need compiling: SCHED_FIFO does not run in xenserver - how to enable it?
  335. yum install gcc-c++ autoconf rpm-build -y
  336. yum install popt-devel libtalloc-devel libtdb-devel libtevent-devel -y
  337.  
  338. mkdir rpmbuild/SPECS -p; curl http://ix.io/QMh > rpmbuild/SPECS/ctdb.spec
  339. wget https://download.samba.org/pub/ctdb/ctdb-2.5.6.tar.gz -P rpmbuild/SOURCES
  340. rpmbuild -bp rpmbuild/SPECS/ctdb.spec
  341. sed -i "s/SCHED_FIFO/SCHED_OTHER/"  rpmbuild/BUILD/ctdb-2.5.6/common/system_util.c  
  342. sed -i "s/p.sched_priority = 1/p.sched_priority = 0/"  rpmbuild/BUILD/ctdb-2.5.6/common/system_util.c  
  343. rpmbuild -bc --short-circuit  rpmbuild/SPECS/ctdb.spec --noclean
  344. rpmbuild -bi --short-circuit  rpmbuild/SPECS/ctdb.spec --noclean
  345. rpmbuild -bb --short-circuit  rpmbuild/SPECS/ctdb.spec --noclean
  346. rpm -ivh rpmbuild/RPMS/x86_64/ctdb-* --nodeps --force
  347. mv rpmbuild rpmbuild-ctdb -f
  348. sed -i "s/^# CTDB_LOGFILE/CTDB_LOGFILE/" /etc/sysconfig/ctdb
  349. sed -i "s/^# CTDB_NODES/CTDB_NODES/" /etc/sysconfig/ctdb  
  350. sed -i "s/^CTDB_RECOVERY_LOCK/# CTDB_RECOVERY_LOCK/" /etc/sysconfig/ctdb
  351. sed -i 's/Restart=no/Restart=always/' /usr/lib/systemd/system/ctdb.service
  352. #service smb stop
  353. #chkconfig smb off
  354.  
  355. #compiling module ceph.ko for cephfs
  356. wget http://downloadns.citrix.com.edgesuite.net/11624/XenServer-7.0.0-binpkg.iso
  357. mkdir iso1; mount XenServer-7.0.0-binpkg.iso iso1
  358. yum localinstall iso1/x86_64/kernel-* -y
  359. #wget http://downloadns.citrix.com.edgesuite.net/11620/XenServer-7.0.0-DDK.iso
  360. wget http://downloadns.citrix.com.edgesuite.net/11623/XenServer-7.0.0-source.iso
  361. mkdir iso; mount XenServer-7.0.0-source.iso iso
  362. rpm -ivh iso/kernel-3.10.96-484.383030.src.rpm
  363. rpmbuild -bp rpmbuild/SPECS/kernel.spec
  364. sed -i 's/# CONFIG_CEPH_FS is not set/CONFIG_CEPH_FS=m/g' rpmbuild/BUILD/kernel-3.10.96/linux-3.10.96/.config
  365. rpmbuild -bc --short-circuit  rpmbuild/SPECS/kernel.spec --noclean
  366. rpmbuild -bi --short-circuit  rpmbuild/SPECS/kernel.spec --noclean
  367. rpmbuild -bb --short-circuit  rpmbuild/SPECS/kernel.spec --noclean
  368. rpm -ivh rpmbuild/RPMS/x86_64/kernel-* --nodeps --force
  369. #scp  rpmbuild/RPMS/x86_64/kernel-* xen2:
  370. #scp  rpmbuild/RPMS/x86_64/kernel-* xen3:
  371. #rpm -ivh kernel-* --nodeps --force
  372.  
  373.  
  374. ####################################################################################################################################
  375. ##CEPH XENSERVER
  376. #http://www.virtualtothecore.com/en/quickly-build-a-new-ceph-cluster-with-ceph-deploy-on-centos-7/
  377. #ntpd, no selinux, ssh-copy-id to do
  378. useradd -d /home/cephuser -m cephuser
  379. echo "cephuser ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephuser
  380. chmod 0440 /etc/sudoers.d/cephuser
  381. sed -i s'/Defaults requiretty/#Defaults requiretty'/g /etc/sudoers
  382. git clone https://github.com/mstarikov/rbdsr.git
  383. cd rbdsr
  384. python ./install_rbdsr.py enable
  385. ####################################################################################################################################
  386. ###########################################################################
  387. #uuid pbd http://blog.gluster.org/2012/
  388. #usunięcie Local Storage na sda3
  389. #sformatowanie i podmontowanie dwóch wolnych partycji sda2 i sda3 na dwa dyski sieciowe
  390. sed -i -e "s/metadata_read_only = 1/metadata_read_only = 0/" /etc/lvm/lvm.conf
  391. hname=`hostname`
  392. #hname=`echo $hname | tr [:lower:] [:upper:]`
  393. sruuid=`xe sr-list host=$hname name-label=Local\ storage --minimal`
  394. pbduid=`xe pbd-list sr-uuid=$sruuid --minimal`
  395. xe pbd-unplug uuid=$pbduid
  396. xe sr-forget uuid=$sruuid
  397. vgremove `vgdisplay -C | tail -n1 | cut -f3 -d' '` -f
  398. dev4gfs=`pvdisplay -C | tail -n1 | cut -f3 -d' '`
  399. pvremove $dev4gfs -f
  400.  
  401. umount $dev4gfs
  402. init_brick $dev4gfs "vol0"
  403.  
  404. umount "/dev/sda2"
  405. init_brick "/dev/sda2" "vol1"
  406.  
  407. #FOR CEPH
  408. parted -s /dev/sdb mklabel gpt mkpart primary xfs 0% 100%
  409. #parted -s /dev/sdb mklabel gpt mkpart primary 0% 33% mkpart primary 34% 66% mkpart primary 67% 100%
  410. mkfs.xfs /dev/sdb1 -f
  411. #removeallandformat "/dev/sdb"
  412. #init_brick "/dev/sdb1" "vol2"
  413.  
  414. ###########################################################################
  415. #OnePool
  416. ###########################################################################
  417. server="xen"
  418. host01="1"
  419. host02="2"
  420. host03="3"
  421. echo "127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4" > /etc/hosts
  422. echo "10.10.10.1${host01} osd${host01}" >> /etc/hosts
  423. echo "10.10.10.1${host02} osd${host02}" >> /etc/hosts
  424. echo "10.10.10.1${host03} osd${host03}" >> /etc/hosts
  425. echo "192.168.0.20${host01} ${server}${host01}" >> /etc/hosts
  426. echo "192.168.0.20${host02} ${server}${host02}" >> /etc/hosts
  427. echo "192.168.0.20${host03} ${server}${host03}" >> /etc/hosts
  428. echo "10.10.10.1${host01}" > /etc/ctdb/nodes
  429. echo "10.10.10.1${host02}" >> /etc/ctdb/nodes
  430. echo "10.10.10.1${host03}" >> /etc/ctdb/nodes
  431. echo "192.168.0.22${host01}/24 xenbr1" > /etc/ctdb/public_addresses
  432. echo "192.168.0.22${host02}/24 xenbr1" >> /etc/ctdb/public_addresses
  433. echo "192.168.0.22${host03}/24 xenbr1" >> /etc/ctdb/public_addresses
  434. chkconfig ctdb on
  435. #service ctdb restart
  436.  
  437. #init_gluster4 "xen" "1" "2" "3" "vol0" 3
  438. #init_gluster4 "xen" "1" "2" "3" "vol1" 3
  439. #service ctdb restart
  440. #ctdb status
  441.  
  442. #https://serversforhackers.com/an-ansible-tutorial
  443. #http://www.cyberciti.biz/faq/
  444. [ -f /etc/ansible/hosts ] && mv /etc/ansible/hosts /etc/ansible/hosts.orig -f
  445. echo "[web]" > /etc/ansible/hosts
  446. echo "10.10.10.1${host01}" >> /etc/ansible/hosts
  447. echo "10.10.10.1${host02}" >> /etc/ansible/hosts
  448. echo "10.10.10.1${host03}" >> /etc/ansible/hosts
  449. #ssh-keygen -t rsa -b 2048 -N '' -f ~/.ssh/id_rsa
  450. #ssh-copy-id -i root@xen1
  451. #ssh-copy-id -i root@xen2
  452. #ssh-copy-id -i root@xen3
  453. #ansible all -s -m shell -a "ctdb status"
  454. echo 'ansible all -s -m shell -a "$1"' > /usr/local/bin/ae
  455. chmod 700 /usr/local/bin/ae
  456. #CRONTAB WORKS for test logs
  457. [ -f /var/log/checktime.log ] && mv /var/log/checktime.log /var/log/checktime.log.old -f
  458. echo 'echo "#########################"' > /usr/local/bin/checktime
  459. echo "date" >> /usr/local/bin/checktime
  460. echo "ntpstat -s" >> /usr/local/bin/checktime
  461. echo "/sbin/gluster volume status vol2" >> /usr/local/bin/checktime
  462. echo "ctdb status" >> /usr/local/bin/checktime
  463. echo "free" >> /usr/local/bin/checktime
  464. chmod 755 /usr/local/bin/checktime
  465. echo "/usr/local/bin/checktime  2>&1 | cat >> /var/log/checktime.log" > /usr/local/bin/cronuserlogs
  466. chmod 755 /usr/local/bin/cronuserlogs
  467. echo "* * * * * /usr/local/bin/cronuserlogs" > ./cronwork
  468. crontab -r
  469. crontab ./cronwork
  470. crontab -l
  471.  
  472. cat <<__EOF__ > /etc/logrotate.d/checktime
  473. /var/log/checktime.log {
  474.     daily
  475.     rotate 3
  476.     compress
  477.     delaycompress
  478.     missingok
  479.     notifempty
  480.     create 644 root root
  481. }
  482. __EOF__
  483. cat <<__EOF__ > ~/.ssh/config
  484. Host *
  485.     StrictHostKeyChecking no
  486.     UserKnownHostsFile /dev/null
  487. __EOF__
  488. ssh-keygen -t rsa -b 2048 -N '' -f ~/.ssh/id_rsa
  489. #for node in xen1 xen2 xen3 osd3 osd4 osd6; do ssh-copy-id $node ; done
  490.  
  491.  
  492. #CEPH
  493. #http://www.virtualtothecore.com/en/adventures-ceph-storage-part-1-introduction/
  494. #https://blog.zhaw.ch/icclab/tag/ceph/
  495. #https://wiki.centos.org/SpecialInterestGroup/Storage/ceph-Quickstart
  496. #http://linoxide.com/storage/setup-red-hat-ceph-storage-centos-7-0/
  497. #http://karan-mj.blogspot.com/2013/12/what-is-ceph-ceph-is-open-source.html
  498. #https://www.reddit.com/r/DataHoarder/comments/4gzpxi/why_is_ceph_so_rare_for_home_use_even_among/
  499. #http://palmerville.github.io/2016/04/30/single-node-ceph-install.html
  500. cat <<__EOF__ >> ./ceph.conf
  501. mon_pg_warn_max_per_osd = 0
  502. public network = 192.168.0.0/24
  503. cluster network = 10.10.10.0/24
  504. #Choose reasonable numbers for number of replicas and placement groups.
  505. osd pool default size = 2 # Write an object 2 times
  506. osd pool default min size = 1 # Allow writing 1 copy in a degraded state
  507. osd pool default pg num = 256
  508. osd pool default pgp num = 256
  509. #Choose a reasonable crush leaf type
  510. #0 for a 1-node cluster.
  511. #1 for a multi node cluster in a single rack
  512. #2 for a multi node, multi chassis cluster with multiple hosts in a chassis
  513. #3 for a multi node cluster with hosts across racks, etc.
  514. osd crush chooseleaf type = 1
  515. [osd.0]
  516. public addr = 192.168.0.53
  517. cluster addr = 10.10.10.13
  518. host = osd3
  519. devs = /dev/sdb
  520. osd journal = /dev/sda2
  521. [osd.1]
  522. public addr = 192.168.0.54
  523. cluster addr = 10.10.10.14
  524. host = osd4
  525. devs = /dev/sdb
  526. osd journal = /dev/sda2
  527. [osd.2]
  528. public addr = 192.168.0.56
  529. cluster addr = 10.10.10.16
  530. host = osd6
  531. devs = /dev/sdb
  532. osd journal = /dev/sda2
  533. __EOF__
  534.  
  535.  
  536. cat <<__EOF__ > /usr/local/bin/init_ceph
  537. server="xen"
  538. host01="1"
  539. host02="2"
  540. host03="3"
  541. [ ! -d ceph-deploy ] && mkdir ceph-deploy
  542. cd ceph-deploy/
  543. ceph-deploy purge xen1 xen2 xen3
  544. ceph-deploy purgedata xen1 xen2 xen3
  545. ceph-deploy forgetkeys
  546. ceph-deploy new xen1 xen2 xen3
  547. #ceph-deploy install --release jewel --no-adjust-repos xen1 xen2 xen3
  548. #ceph-deploy install --release jewel xen1 xen2 xen3
  549. ceph-deploy install --repo-url http://download.ceph.com/rpm-jewel/el7/ xen1 xen2 xen3
  550. ceph-deploy --overwrite-conf mon create-initial
  551.  
  552. ceph-deploy gatherkeys xen1
  553. for i in osd3 osd4 osd6; do ceph-deploy disk zap $i:sdb; done
  554. ae "parted -s /dev/sdb mklabel gpt mkpart primary xfs 0% 100%"
  555. ae "mkfs.xfs /dev/sdb1 -f"
  556. #http://tracker.ceph.com/issues/13833
  557. ae "chown ceph:ceph /dev/sda2"
  558. for i in osd3 osd4 osd6; do
  559. ceph-deploy osd prepare \$i:/dev/sdb1:/dev/sda2; done
  560. for i in osd3 osd4 osd6; do
  561. ceph-deploy osd activate \$i:/dev/sdb1:/dev/sda2; done
  562. #ceph-deploy  --username ceph osd create osd3:/dev/sdb1:/dev/sda2
  563. ceph-deploy admin xen1 xen2 xen3 osd3 osd4 osd6
  564. ae "chmod +r /etc/ceph/ceph.client.admin.keyring"
  565. ae "systemctl enable ceph-mon.target"
  566. ae "systemctl enable ceph-mds.target"
  567. ae "systemctl enable ceph-osd.target"
  568. #object storage gateway
  569. ceph-deploy rgw create xen1 xen2 xen3
  570. #cephfs
  571. ceph-deploy mds create xen1 xen2 xen3
  572. ceph -s #ceph status
  573. ceph osd tree
  574. ceph mon_status
  575. ceph osd pool create mypool 1
  576. ceph osd lspools
  577. ceph df
  578. echo "test data" > testfile
  579. rados put -p mypool testfile testfile
  580. rados -p mypool ls
  581. rados -p mypool setomapval testfile mykey myvalue
  582. rados -p mypool getomapval testfile mykey
  583. rados get -p mypool testfile testfile2
  584. md5sum testfile testfile2
  585.  
  586. ceph osd pool create cephfs_data 128
  587. ceph osd pool create cephfs_metadata 128
  588. ceph fs new cephfs cephfs_metadata cephfs_data
  589. mkdir /mnt/mycephfs
  590. mount -t ceph xen1:6789:/ /mnt/mycephfs -o name=admin,secret=`cat ./ceph.client.admin.keyring | grep key | cut -f 2 | sed 's/key = //g'`
  591. __EOF__
  592.  
  593.  
  594. echo 'Configure NIC1 (192.168.0.22?) i NIC2 (10.10.10.1?)'
  595. echo 'for node in xen1 xen2 xen3 osd3 osd4 osd6; do ssh-copy-id $node ; done'
  596.  
  597. echo 'FOR GLUSTER Execute on one server:'
  598. echo 'gluster peer probe xen1'
  599. echo 'gluster peer probe xen2'
  600. echo 'gluster peer probe xen3'
  601. #echo 'gluster peer probe xen4'
  602. echo 'init_gluster4 "xen" "1" "2" "3" "vol0" 3'
  603. echo 'init_gluster4 "xen" "1" "2" "3" "vol1" 3'
  604. echo 'ae "service ctdb restart"'
  605. echo 'ae "ctdb status"'
  606. #echo 'ip addr show | grep inet'
  607. #echo "ssh-keygen -t rsa -b 2048 -N '' -f ~/.ssh/id_rsa"
  608. echo 'Mount SR iso, gfs1, gfs2'
  609.  
  610. echo 'FOR CEPH execute on one server'
  611. echo '/usr/local/bin/init_ceph'
  612.  
  613. #END
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!