-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathatom.xml
467 lines (437 loc) · 58.4 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title><![CDATA[Jacken's Blog]]></title>
<subtitle><![CDATA[所谓天才，只不过是把别人喝咖啡的功夫都用在工作上了]]></subtitle>
<link href="/atom.xml" rel="self"/>
<link href="http://jacken.me/"/>
<updated>2015-08-03T02:07:16.159Z</updated>
<id>http://jacken.me/</id>
<author>
<name><![CDATA[Jacken]]></name>
<email><![CDATA[[email protected]]]></email>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title><![CDATA[为Openstack制作centos6镜像]]></title>
<link href="http://jacken.me/openstack/centos/create-centos6-image-for-openstack/"/>
<id>http://jacken.me/openstack/centos/create-centos6-image-for-openstack/</id>
<published>2015-08-01T10:34:59.000Z</published>
<updated>2015-08-03T02:07:16.159Z</updated>
<content type="html"><![CDATA[<h1 id="制作Openstack的Linux镜像的基础知识">制作Openstack的Linux镜像的基础知识</h1><h2 id="镜像的磁盘容器格式">镜像的磁盘容器格式</h2><ul>
<li><strong>RAW：</strong>原始的格式，一个纯二进制镜像光盘映像，非常便携，一个字：裸。由于原生的裸格式，不支持快照</li>
<li><strong>QCOW2：</strong>qcow2(QEMU copy-on-write version 2)是KVN虚拟化常用的格式，附带有多种功能，例如大小经过压缩、支持快照、性能上接近raw、支持zlib的磁盘压缩、支持AES的加密等</li>
<li><strong>AMI：</strong>Amazon Machine Image</li>
<li><strong>AKI：</strong>Amazon Kernel Image</li>
<li><strong>ARI：</strong>Amazon Ramdisk Image</li>
<li><strong>AMDK：</strong>VMware’s ESXi用的格式</li>
<li><strong>VDI：</strong>Oracle VirtualBox用的格式</li>
<li><strong>VHD:</strong>微软的虚拟硬盘文件(Microsoft Hyper-V)</li>
<li><strong>VHDX</strong>也是微软的虚拟硬盘文件(Microsoft Hyper-V)，但支持Microsoft Server 2012，附加一些防止数据损坏的特性</li>
</ul>
<a id="more"></a>
<h2 id="虚拟化以及镜像的工具">虚拟化以及镜像的工具</h2><ul>
<li><strong>qemu-kvm：</strong>vm负责cpu虚拟化+内存虚拟化；qemu是模拟IO设备（网卡，磁盘）。kvm加上qemu之后就能实现真正意义上服务器虚拟化。</li>
<li><strong>virt-manager：</strong>virt-manager 是基于 libvirt 的图像化虚拟机管理软件。</li>
<li><strong>libvirt：</strong>免费、开源的支持Linux下主流虚拟化工具的C函数库，为各种虚拟化工具提供一套方便、可靠的编程接口</li>
</ul>
<h2 id="CPU虚拟化技术">CPU虚拟化技术</h2><ul>
<li>CPU需要支持虚拟化技术</li>
<li>需要在BIOS开启虚拟化特性</li>
</ul>
<h2 id="openstack的Linux镜像要求">openstack的Linux镜像要求</h2><ul>
<li>支持磁盘分区和调整引导根分区，使用cloud-init实现</li>
<li>清除MAC地址信息</li>
<li>开启SSH服务</li>
<li>禁用防火墙</li>
<li>支持注入公匙来访问SSH，使用cloud-init实现</li>
<li>支持处理用户的数据和其他元数据，使用cloud-init实现</li>
</ul>
<h1 id="制作镜像">制作镜像</h1><h2 id="官网下载Centos6的ISO文件">官网下载Centos6的ISO文件</h2><p>直接在官网上下载Centos6.6</p>
<pre><code><span class="keyword">mkdir</span> /<span class="keyword">var</span>/openstack-image
<span class="keyword">cd</span> /<span class="keyword">var</span>/openstack-image
wget http:<span class="comment">//mirrors.163.com/centos/6.6/isos/x86_64/CentOS-6.6-x86_64-minimal.iso</span>
</code></pre><h2 id="安装虚拟化工具和配置网络">安装虚拟化工具和配置网络</h2><h3 id="安装">安装</h3><p>通过<code>yum</code>来安装：</p>
<pre><code>yum groupinstall Virtualization
yum groupinstall <span class="string">"Virtualization Client"</span>
yum <span class="keyword">install</span> libvirt
yum <span class="keyword">install</span> libguestfs-tools
</code></pre><p>启动服务</p>
<pre><code><span class="keyword">service</span> libvirtd <span class="literal">restart</span>
</code></pre><h3 id="配置网络">配置网络</h3><p>配置虚拟环境的网络，这个链接描述得很清晰：<br><a href="http://www.chenyudong.com/archives/libvirt-kvm-bridge-network.html" target="_blank" rel="external">http://www.chenyudong.com/archives/libvirt-kvm-bridge-network.html</a></p>
<h2 id="创建虚拟主机">创建虚拟主机</h2><h3 id="创建qcow2镜像文件">创建qcow2镜像文件</h3><pre><code>qemu-<span class="tag">img</span> create -f qcow2 CentOS-<span class="number">6.6</span>-x86_64<span class="class">.qcow2</span> <span class="number">10</span>G
chown -R qemu:qemu *
</code></pre><h3 id="创建虚拟机">创建虚拟机</h3><pre><code>virt-install -n CentOS-<span class="number">6.6</span>-x86_64 -r <span class="number">2048</span> --cpu host -c CentOS-<span class="number">6.6</span>-x86_64-minimal.iso --disk <span class="variable">path=</span>CentOS-<span class="number">6.6</span>-x86_64.qcow2,<span class="variable">device=</span>disk,<span class="variable">bus=</span>virtio,<span class="variable">size=</span><span class="number">10</span>,<span class="variable">format=</span>qcow2 -w bridge:br0 <span class="variable">--os-type=</span>linux <span class="variable">--os-variant=</span>rhel6 --vnc <span class="variable">--vnclisten=</span><span class="number">0.0</span>.<span class="number">0.0</span> -v
</code></pre><h2 id="通过vnc界面安装centos">通过vnc界面安装centos</h2><h3 id="用VNC客户端安装Centos系统">用VNC客户端安装Centos系统</h3><p>查看虚拟机的vnc链接信息</p>
<pre><code><span class="title">Virsh</span> vncdisplay CentOS-<span class="number">6</span>.<span class="number">6</span>-x86_64
</code></pre><p>然后用vncview客户端链接上（vncview下载地址：<a href="http://www.realvnc.com/download/viewer/" target="_blank" rel="external">http://www.realvnc.com/download/viewer/</a>）<br><img src="http://7xj3vi.com1.z0.glb.clouddn.com/centos6.6-install1.jpg" alt="" title="centos6.6 install"> </p>
<p>看到这个界面就熟悉了，基本和平时安装系统一样进行下去，但有以下几点的要求：<br><strong>网络的配置：</strong><br><img src="http://7xj3vi.com1.z0.glb.clouddn.com/centos6.6-install-network.jpg" alt="" title="network"><br>启用IPv4的DHCP</p>
<p><strong>存储设备：</strong><br>在选择存储设备的类型时,选择基本存储设备（<code>Basic Storage Devices.</code>）。 </p>
<p><strong>磁盘分区：</strong><br>如果不了解这块，选择默认分区方。</p>
<p><strong>完成操作系统的安装：</strong><br><img src="http://7xj3vi.com1.z0.glb.clouddn.com/centos6.6-install-reboot.jpg" alt="" title="reboot"><br>完成系统安装需要重启虚拟机。重启前卸载CD-ROM:</p>
<pre><code>virsh attach-disk --<span class="built_in">type</span> cdrom --<span class="literal">mode</span> readonly <span class="constant">Cent</span>OS-<span class="number">6.6</span>-x86_64 <span class="string">""</span> hdc
</code></pre><p>使用上面<code>virsh</code>命令可以弹出光驱</p>
<h3 id="操作系统的一些配置">操作系统的一些配置</h3><p>完成centos系统的安装后，用SSH客户端（例如<code>PuTTY</code>）链接上虚拟机，修改操作系统的一些配置的</p>
<p><strong>修改网络配置</strong></p>
<p><strong><em>Disable the zeroconf route：</em></strong></p>
<pre><code>echo <span class="string">"NOZEROCONF=yes"</span> <span class="prompt">>> </span>/etc/sysconfig/network
</code></pre><p><strong>安装ACPI服务</strong></p>
<pre><code>yum install -y acpid
chkconfig acpid <span class="function_start"><span class="keyword">on</span></span>
</code></pre><p>这样可以通过web界面控制虚拟化程序来重启或者关闭虚拟机实例</p>
<p><strong>Configure console</strong><br><code>nova console</code>命令读取到centos的log，需要在<code>/boot/grub/menu.lst</code>文件下增加以下的内容：</p>
<pre><code>serial <span class="variable">--unit=</span><span class="number">0</span> <span class="variable">--speed=</span><span class="number">115200</span>
terminal <span class="variable">--timeout=</span><span class="number">10</span> console serial
<span class="comment"># Edit the kernel line to add the console entries</span>
kernel ... <span class="variable">console=</span>tty0 <span class="variable">console=</span>ttyS0,<span class="number">115200</span>n8
</code></pre><p><strong>升级</strong></p>
<pre><code><span class="title">yum</span> upgrade
</code></pre><p>升级下操作系统</p>
<p><strong>禁用selinux</strong></p>
<p>修改<code>/etc/selinux/config</code>文件,将<code>SELINUX=enforcing</code>改为<code>SELINUX=disabled</code></p>
<p><strong>关闭防火墙</strong></p>
<pre><code><span class="title">chkconfig</span> iptables <span class="built_in">off</span>
chkconfig ip6tables <span class="built_in">off</span>
</code></pre><p><strong>编辑/etc/rc.local</strong></p>
<p>在<code>/etc/rc.local</code>文件增加以下内容：</p>
<pre><code><span class="keyword">if</span> [ ! -d <span class="regexp">/root/</span>.ssh ] ; <span class="keyword">then</span>
mkdir -p <span class="regexp">/root/</span>.ssh
chmod <span class="number">0700</span> /root/.ssh
restorecon <span class="regexp">/root/</span>.ssh
fi
</code></pre><p><strong>最重要的配置，Attention</strong></p>
<p>这个配置很重要，提升自己的知名度，修改<code>/etc/motd</code>，增加以下内容：</p>
<pre><code><span class="constant">Welcome</span> to <span class="constant">XunyunEdu</span> <span class="constant">Elastic</span> <span class="constant">Compute</span> <span class="constant">Server</span>
<span class="constant">The</span> <span class="constant">CentOS</span> <span class="constant">Image</span> <span class="constant">Author</span><span class="symbol">:Jacken</span>(chenjacken<span class="variable">@gmail</span>.com)
<span class="constant">Pls</span> <span class="constant">Visit</span> <span class="constant">The</span> <span class="constant">Website</span><span class="symbol">:http</span><span class="symbol">://www</span>.jacken.me
</code></pre><h2 id="安装cloud-init和配置">安装cloud-init和配置</h2><p>直接用<code>yum</code>进行安装</p>
<pre><code>yum <span class="operator"><span class="keyword">install</span> -y http://download.fedoraproject.org/pub/epel/<span class="number">6</span>/x86_64/epel-<span class="keyword">release</span>-<span class="number">6</span>-<span class="number">8.</span>noarch.rpm
yum <span class="keyword">install</span> -y cloud-utils cloud-init</span>
</code></pre><p><code>cloud-init</code>的配置文件在<code>/etc/cloud/cloud.cfg</code>,编辑里面配置，定制自己所需要的功能。我的配置信息是：</p>
<figure class="highlight python"><figcaption><span>cloud.cfg</span></figcaption><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br></pre></td><td class="code"><pre><span class="line">disable_root: <span class="number">0</span></span><br><span class="line">ssh_pwauth: <span class="number">0</span></span><br><span class="line"></span><br><span class="line">locale_configfile: /etc/sysconfig/i18n</span><br><span class="line">mount_default_fields: [~, ~, <span class="string">'auto'</span>, <span class="string">'defaults,nofail'</span>, <span class="string">'0'</span>, <span class="string">'2'</span>]</span><br><span class="line">resize_rootfs: <span class="number">1</span></span><br><span class="line">resize_rootfs_tmp: /dev</span><br><span class="line">ssh_deletekeys: <span class="number">0</span></span><br><span class="line">ssh_genkeytypes: ~</span><br><span class="line">syslog_fix_perms: ~</span><br><span class="line"></span><br><span class="line">datasource_list: [<span class="string">'ConfigDrive'</span>,<span class="string">'Ec2'</span>,<span class="string">'NoCloud'</span>]</span><br><span class="line">datasource:</span><br><span class="line"> Ec2:</span><br><span class="line"> timeout: <span class="number">5</span></span><br><span class="line"> max_wait: <span class="number">10</span></span><br><span class="line"></span><br><span class="line">chpasswd:</span><br><span class="line"> list: |</span><br><span class="line"> root:RANDOM</span><br><span class="line"> expire: <span class="keyword">False</span></span><br><span class="line"></span><br><span class="line">growpart:</span><br><span class="line"> mode: auto</span><br><span class="line"> devices: [<span class="string">'/'</span>]</span><br><span class="line"></span><br><span class="line">cloud_init_modules:</span><br><span class="line"> - migrator</span><br><span class="line"> - bootcmd</span><br><span class="line"> - write-files</span><br><span class="line"> - growpart</span><br><span class="line"> - resizefs</span><br><span class="line"> - set_hostname</span><br><span class="line"> - update_hostname</span><br><span class="line"> - update_etc_hosts</span><br><span class="line"> - rsyslog</span><br><span class="line"> - users-groups</span><br><span class="line"> - ssh</span><br><span class="line"> - resolv-conf</span><br><span class="line"></span><br><span class="line">cloud_config_modules:</span><br><span class="line"> - mounts</span><br><span class="line"> - locale</span><br><span class="line"> - set-passwords</span><br><span class="line"> - timezone</span><br><span class="line"> - mcollective</span><br><span class="line"> - disable-ec2-metadata</span><br><span class="line"> - runcmd</span><br><span class="line"></span><br><span class="line">cloud_final_modules:</span><br><span class="line"> - rightscale_userdata</span><br><span class="line"> - scripts-per-once</span><br><span class="line"> - scripts-per-boot</span><br><span class="line"> - scripts-per-instance</span><br><span class="line"> - scripts-user</span><br><span class="line"> - ssh-authkey-fingerprints</span><br><span class="line"> - keys-to-console</span><br><span class="line"> - phone-home</span><br><span class="line"> - final-message</span><br><span class="line"></span><br><span class="line">system_info:</span><br><span class="line"> distro: rhel</span><br><span class="line"> default_user:</span><br><span class="line"> name: centos</span><br><span class="line"> paths:</span><br><span class="line"> cloud_dir: /var/lib/cloud</span><br><span class="line"> templates_dir: /etc/cloud/templates</span><br><span class="line"> ssh_svcname: sshd</span><br><span class="line"></span><br><span class="line"><span class="comment"># vim:syntax=yaml</span></span><br></pre></td></tr></table></figure>
<p><a href="/code/cloud-init/cloud.cfg" title="cloud.cfg">cloud.cfg</a></p>
<p><strong>其中几点说明下：</strong></p>
<p><strong>1，</strong>允许<code>root</code>登录和禁止<code>password</code>验证</p>
<pre><code><span class="attribute">disable_root</span>: <span class="string">0</span>
<span class="attribute">ssh_pwauth</span>: <span class="string"> 0</span>
</code></pre><p><strong>2，</strong>以下配置是让虚拟机根据<code>openstack</code>的<code>Flavors</code>自动resize根文件系统大小</p>
<pre><code><span class="attribute">resize_rootfs</span>: <span class="string">1</span>
<span class="attribute">resize_rootfs_tmp</span>: <span class="string">/dev</span>
<span class="less"><span class="attribute">growpart</span>:
<span class="attribute">mode</span>: auto
<span class="attribute">devices</span>: [<span class="string">'/'</span>]</span>
</code></pre><p>但是遇到一个问题，虚拟机实例创建完，需要重启一次系统才能调整根文件系统的大小，<strong>请高手指点</strong></p>
<p><strong>3，</strong>数据源的获取方式</p>
<pre><code><span class="rule"><span class="attribute">datasource_list</span>:<span class="value"> [<span class="string">'ConfigDrive'</span>,<span class="string">'Ec2'</span>,<span class="string">'NoCloud'</span>]</span></span>
</code></pre><p>我配置了<code>ConfigDrive</code>为先。这个需要修改openstack的nova的配置文件，开启<code>Config Drive</code>。<br>简单说明下，在所有计算节点，修改<code>nova.conf</code>配置,增加或修改以下内容：</p>
<pre><code><span class="setting">force_config_drive=<span class="value"><span class="keyword">True</span></span></span>
<span class="setting">config_drive_format=<span class="value">iso9660</span></span>
</code></pre><h2 id="创建快照和恢复">创建快照和恢复</h2><p>镜像创建快照，目的是可以让镜像创建多种环境版本，例如这个快照是集成LAMP，那个快照安装了Mysql等。恢复到指定的快照，可以修复完成发现的问题或创建另一种环境的快照，再发布打包成镜像。</p>
<p><strong>创建快照</strong></p>
<pre><code>virsh snapshot-create-as --<span class="built_in">domain</span> <span class="constant">Cent</span>OS-<span class="number">6.6</span>-x86_64 --<span class="literal">name</span> <span class="string">"cloud-init"</span> --<span class="literal">description</span> <span class="string">"cloud-init had installed and configured"</span>
</code></pre><p><strong>查看已有的快照</strong></p>
<pre><code>virsh snapshot-<span class="type">list</span> CentOS-<span class="number">6.6</span>-x86_64
</code></pre><p><strong>恢复到指定的快照</strong></p>
<pre><code><span class="comment">virsh</span> <span class="comment">snapshot</span><span class="literal">-</span><span class="comment">revert</span> <span class="literal">-</span><span class="literal">-</span><span class="comment">domain</span> <span class="comment">CentOS</span><span class="literal">-</span><span class="comment">6</span><span class="string">.</span><span class="comment">6</span><span class="literal">-</span><span class="comment">x86_64</span> <span class="literal">-</span><span class="literal">-</span><span class="comment">snapshotname</span> <span class="comment">cloud</span><span class="literal">-</span><span class="comment">init</span>
</code></pre><p><strong>删除指定的快照</strong></p>
<pre><code><span class="comment">virsh</span> <span class="comment">snapshot</span><span class="literal">-</span><span class="comment">delete</span> <span class="literal">-</span><span class="literal">-</span><span class="comment">domain</span> <span class="comment">CentOS</span><span class="literal">-</span><span class="comment">6</span><span class="string">.</span><span class="comment">6</span><span class="literal">-</span><span class="comment">x86_64</span> <span class="literal">-</span><span class="literal">-</span><span class="comment">snapshotname</span> <span class="comment">cloud</span><span class="literal">-</span><span class="comment">init</span>
</code></pre><h2 id="发布镜像">发布镜像</h2><h3 id="清理镜像">清理镜像</h3><p>把镜像系统关机:</p>
<pre><code>shutdown -h <span class="keyword">now</span>
</code></pre><p>删除硬件信息</p>
<pre><code>virt-sysprep <span class="operator">-d</span> CentOS-<span class="number">6.6</span>-x86_64
</code></pre><h3 id="打包镜像">打包镜像</h3><p>压缩镜像文件</p>
<pre><code><span class="comment">virt</span><span class="literal">-</span><span class="comment">sparsify</span> <span class="literal">-</span><span class="literal">-</span><span class="comment">tmp</span> <span class="string">.</span><span class="comment">/</span> <span class="literal">-</span><span class="literal">-</span><span class="comment">compress</span> <span class="comment">CentOS</span><span class="literal">-</span><span class="comment">6</span><span class="string">.</span><span class="comment">6</span><span class="literal">-</span><span class="comment">x86_64</span><span class="string">.</span><span class="comment">qcow2</span> <span class="comment">CentOS</span><span class="literal">-</span><span class="comment">6</span><span class="string">.</span><span class="comment">6</span><span class="literal">-</span><span class="comment">x86_64</span><span class="literal">-</span><span class="comment">Cloud</span><span class="string">.</span><span class="comment">qcow2</span>
</code></pre><h1 id="在openstack上传镜像">在openstack上传镜像</h1><p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/openstack-Create-An-Image.jpg" alt="" title="Create An Image"></p>
<p>选择刚才创建的镜像文件<code>CentOS-6.6-x86_64-Cloud.qcow2</code>上传</p>
<p>当然你可以用命令<code>glance add</code>创建镜像,这部分略！</p>
]]></content>
<summary type="html">
<![CDATA[<h1 id="制作Openstack的Linux镜像的基础知识">制作Openstack的Linux镜像的基础知识</h1><h2 id="镜像的磁盘容器格式">镜像的磁盘容器格式</h2><ul>
<li><strong>RAW：</strong>原始的格式，一个纯二进制镜像光盘映像，非常便携，一个字：裸。由于原生的裸格式，不支持快照</li>
<li><strong>QCOW2：</strong>qcow2(QEMU copy-on-write version 2)是KVN虚拟化常用的格式，附带有多种功能，例如大小经过压缩、支持快照、性能上接近raw、支持zlib的磁盘压缩、支持AES的加密等</li>
<li><strong>AMI：</strong>Amazon Machine Image</li>
<li><strong>AKI：</strong>Amazon Kernel Image</li>
<li><strong>ARI：</strong>Amazon Ramdisk Image</li>
<li><strong>AMDK：</strong>VMware’s ESXi用的格式</li>
<li><strong>VDI：</strong>Oracle VirtualBox用的格式</li>
<li><strong>VHD:</strong>微软的虚拟硬盘文件(Microsoft Hyper-V)</li>
<li><strong>VHDX</strong>也是微软的虚拟硬盘文件(Microsoft Hyper-V)，但支持Microsoft Server 2012，附加一些防止数据损坏的特性</li>
</ul>]]>
</summary>
<category term="centos" scheme="http://jacken.me/tags/centos/"/>
<category term="openstack" scheme="http://jacken.me/tags/openstack/"/>
<category term="openstack" scheme="http://jacken.me/categories/openstack/"/>
<category term="centos" scheme="http://jacken.me/categories/openstack/centos/"/>
</entry>
<entry>
<title><![CDATA[用Ambari2管理Hadoop_第二部分：部署]]></title>
<link href="http://jacken.me/Hadoop/use-ambari2-manage-hadoop-deploying/"/>
<id>http://jacken.me/Hadoop/use-ambari2-manage-hadoop-deploying/</id>
<published>2015-06-15T17:00:27.000Z</published>
<updated>2015-08-01T10:37:02.761Z</updated>
<content type="html"><![CDATA[<p><a href="/Hadoop/use-ambari2-manage-hadoop-environment/" title="用Ambari2管理Hadoop_第一部分：环境">用Ambari2管理Hadoop_第一部分：环境</a></p>
<h1 id="就是这个神奇的网站">就是这个神奇的网站</h1><p>Apache Ambari是一种基于Web的工具，支持Apache Hadoop集群的供应、管理和监控。Ambari目前已支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、Hbase、Zookeper、Sqoop和Hcatalog等。 </p>
<p><strong>Ambari主要给你带来以下内容：</strong> </p>
<ul>
<li>通过一步一步的安装向导简化了集群供应。</li>
<li>预先配置好关键的运维指标（metrics），可以直接查看Hadoop Core（HDFS和MapReduce）及相关项目（如HBase、Hive和HCatalog）是否健康。</li>
<li>支持作业与任务执行的可视化与分析，能够更好地查看依赖和性能。</li>
<li>通过一个完整的RESTful API把监控信息暴露出来，集成了现有的运维工具。</li>
<li>用户界面非常直观，用户可以轻松有效地查看信息并控制集群。</li>
</ul>
<h2 id="创建集群">创建集群</h2><p>访问Ambari网址，看到登录界面：<br><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-login.png" alt="" title="Ambari login"><br>用默认的用户名和密码<code>admin/admin</code>登录。<br><a id="more"></a><br><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-create-cluster.png" alt="" title="create cluster"><br>创建一个集群，点击<code>Launch Install Wizard</code></p>
<p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-cluster-name.png" alt="" title="cluster name"><br>起一个集群的名称，例如<code>Education</code></p>
<h2 id="Select_stack">Select stack</h2><p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-select-stack.png" alt="" title="select stack"><br>如图，选择HDP的版本，我用的是Centos6，把其他勾选去掉，只保留<code>redhat6</code>，并根据<a href="/Hadoop/use-ambari2-manage-hadoop-environment/#本地yum源配置" title="本地yum源配置">第一部分的内容</a>，输入本地的yum源地址：<br>HDP Base URL：</p>
<pre><code><span class="symbol">http:</span>/<span class="regexp">/ambari.hdp.novalocal/hdp</span><span class="regexp">/centos6/</span><span class="constant">HDP-</span><span class="number">2.2</span>.<span class="number">4.2</span>
</code></pre><p>HDP-UTILS Base URL：</p>
<pre><code><span class="symbol">http:</span>/<span class="regexp">/ambari.hdp.novalocal/hdp</span><span class="regexp">/centos6/</span><span class="constant">HDP-UTILS-</span><span class="number">1.1</span>.<span class="number">0</span>.<span class="number">20</span>
</code></pre><h1 id="部署规划">部署规划</h1><h2 id="服务规划">服务规划</h2><ul>
<li><strong>node1：</strong>Zookeeper Server</li>
<li><strong>node2：</strong>Zookeeper Server</li>
<li><strong>node3：</strong>Zookeeper Server</li>
<li><strong>node4：</strong>NameNode、History Server 、ResourceManager、Hive Metastore、HBase Metastore、Oozie Server、DRPC Server、Kafka Broker、Knox Gateway</li>
<li><strong>node5：</strong>SNameNode、、App Timeline Server、HiveServer2、WebHCat Server、HBase Metastore 、Oozie Server、 Nimbus、Spark History Server、Kafka Broker、Knox Gateway</li>
<li><strong>node6：</strong>Falcon Server、Storm UI Server、Metrics Collector、client</li>
<li><strong>node7：</strong>Datanode,NodeManager,Supervisor,RegionServer,Flume</li>
<li><strong>node8：</strong>Datanode,NodeManager,Supervisor,RegionServer,Flume</li>
<li><strong>node9：</strong>Datanode,NodeManager,Supervisor,RegionServer,Flume</li>
</ul>
<h2 id="安装agent">安装agent</h2><p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-install-options.png" alt="" title="install-options"><br>目标主机输入主机名，一行一台主机，可以使用正则表达式，例如我输入<code>node[1-9].hdp.novalocal</code></p>
<p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-confirm-hosts.png" alt="" title="confirm hosts"><br>Ambari会自动安装Ambari agent,并检查主机的环境是否满足要求。<br><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-host-checks.png" alt="" title="host checks"><br>如果你根据<a href="/Hadoop/use-ambari2-manage-hadoop-environment/#所有节点基础配置" title="所有节点基础配置">第一部分的内容</a>配置好所有主机，环境检查都会顺利通过。</p>
<h2 id="选择服务">选择服务</h2><p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-choose-services.png" alt="" title="choose services"><br>选择需要部署的服务，因为是测试环境，把所有服务都选上。<br><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-choose-services-warning.png" alt="" title="choose services warning"><br><strong>Spark</strong>需要依赖HDP2.2.2，不犹如向前进<code>Proceed</code> </p>
<p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-assign-masters.png" alt="" title="assign masters"><br><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-assign-masters2.png" alt="" title="assign masters"><br>根据服务规划来分配主服务器。</p>
<p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-assign-slaves-clients.png" alt="" title="assign slaves clients"><br>根据服务规划来分配从服务器和客户端。</p>
<h2 id="配置服务">配置服务</h2><p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-customize-services.png" alt="" title="customize services"><br>红色提示的是需要进行配置的服务。</p>
<p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-customize-services-hive.png" alt="" title="customize hive"><br>配置<code>Hive</code>的数据库。我用的是Mysql独立的数据库主机。按照以下部署进行： </p>
<ul>
<li><strong>安装数据库：</strong>数据库服务器作为独立一台主机，要先安装好mysql数据库</li>
<li><strong>创建数据库：</strong>创建<code>hive</code>用的数据库和账号密码</li>
<li><strong>在Ambari服务器配置驱动</strong>例如我的命令是<code>ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java-5.1.35-bin.jar</code>,JDBC官方驱动包可以在这里下载 <a href="http://dev.mysql.com/downloads/connector/j/" target="_blank" rel="external">http://dev.mysql.com/downloads/connector/j/</a></li>
<li><strong>在页面填写数据库信息：</strong>在网页填写数据库服务器地址和数据库名称账号密码等信息</li>
</ul>
<p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-customize-services-Oozie.png" alt="" title="customize Oozie"><br>配置<code>Oozie</code>的数据库和配置<code>Hive</code>的数据库方法一样。从复的步骤可以省略不做。</p>
<p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-customize-services-knox.png" alt="" title="customize Knox"><br>设置<code>Knox</code>的主密码</p>
<h2 id="开始部署">开始部署</h2><p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-review.png" alt="" title="review"><br>预览服务配置信息，没有错误就进行下一步安装了。</p>
<p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-install-start-test.png" alt="" title="install start test"><br>整个安装进度可以看到，如果出现错误请<code>Retry</code>几次。安装完成会自动启动并测试各种服务是否正常。</p>
<h1 id="管理Hadoop">管理Hadoop</h1><p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-dashboard.png" alt="" title="ambari dashboard"><br>高大上的感觉吧？在这里可以启动，停止服务。增加服务器，增加服务。配置各个服务，且配置信息以版本记录自由切换。 </p>
]]></content>
<summary type="html">
<![CDATA[<p><a href="/Hadoop/use-ambari2-manage-hadoop-environment/" title="用Ambari2管理Hadoop_第一部分：环境">用Ambari2管理Hadoop_第一部分：环境</a></p>
<h1 id="就是这个神奇的网站">就是这个神奇的网站</h1><p>Apache Ambari是一种基于Web的工具，支持Apache Hadoop集群的供应、管理和监控。Ambari目前已支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、Hbase、Zookeper、Sqoop和Hcatalog等。 </p>
<p><strong>Ambari主要给你带来以下内容：</strong> </p>
<ul>
<li>通过一步一步的安装向导简化了集群供应。</li>
<li>预先配置好关键的运维指标（metrics），可以直接查看Hadoop Core（HDFS和MapReduce）及相关项目（如HBase、Hive和HCatalog）是否健康。</li>
<li>支持作业与任务执行的可视化与分析，能够更好地查看依赖和性能。</li>
<li>通过一个完整的RESTful API把监控信息暴露出来，集成了现有的运维工具。</li>
<li>用户界面非常直观，用户可以轻松有效地查看信息并控制集群。</li>
</ul>
<h2 id="创建集群">创建集群</h2><p>访问Ambari网址，看到登录界面：<br><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari2-login.png" alt="" title="Ambari login"><br>用默认的用户名和密码<code>admin/admin</code>登录。<br>]]>
</summary>
<category term="ambari" scheme="http://jacken.me/tags/ambari/"/>
<category term="hadoop" scheme="http://jacken.me/tags/hadoop/"/>
<category term="Hadoop" scheme="http://jacken.me/categories/Hadoop/"/>
</entry>
<entry>
<title><![CDATA[用Ambari2管理Hadoop_第一部分：环境]]></title>
<link href="http://jacken.me/Hadoop/use-ambari2-manage-hadoop-environment/"/>
<id>http://jacken.me/Hadoop/use-ambari2-manage-hadoop-environment/</id>
<published>2015-06-06T13:33:36.000Z</published>
<updated>2015-06-09T06:47:41.626Z</updated>
<content type="html"><![CDATA[<h1 id="本系列文章说明">本系列文章说明</h1><p>当我脑海里萌发要写这系列的文章时候，其实我是抗拒的！因为我觉得呀……这是大数据呀，我一个屌丝码农哪有这个能耐。但是又觉得博客总得写点内容，装逼装逼也得附和社会呀，加上看了看<code>Evernote</code>记录的内容甚乱，必须写……后来写着写着真觉得很DUANG~DUANG~DUANG~…..<br>其实在项目中用到Hadoop的功能甚少，一个HDFS来存储非结构数据，一个是Mapreduce来分布计算处理数据。然而，就算写个简单的<code>Hello world</code>也得把环境搭建起来。所以，本系列的文章，权当我从0开始学习Hadoop的笔记，希望能和像我一样的新手分享经验，同时也希望得到高人指点一二三四五… </p>
<hr>
<p><strong>本部分内容</strong></p>
<ul>
<li>Hadoop生态圈</li>
<li>本系类文章的环境介绍</li>
<li>Ambari安装<ul>
<li>所有节点基础配置</li>
<li>本地yum源配置</li>
<li>Ambari安装</li>
</ul>
</li>
<li>一个神奇的网址<a id="more"></a>
<h1 id="Hadoop生态圈">Hadoop生态圈</h1></li>
</ul>
<h2 id="生态系统">生态系统</h2><p>大量 <code>Apache Software Foundation</code> 项目构成 <code>Hadoop</code> 的生态系统，提供企业部署、集成和使用 <code>Hadoop</code> 所需的服务。这些项目中的每一个都经过部署，提供明确的功能，都有自己的开发人员群体和各自的发布周期。<br>你肯定也听过YARN、 Apache HBase、Apache Hive或者Apache Spark、Apache Storm，Apache Ambari也是喔，等等等…<br>没听过没有关系，看看以下的图片：<br><img src="http://7xj3vi.com1.z0.glb.clouddn.com/hdp.png" alt="Enterprise Hadoop：项目生态系统"></p>
<p>看不懂图片没有关系，看看这个网址：<a href="http://zh.hortonworks.com/hadoop/" target="_blank" rel="external">http://zh.hortonworks.com/hadoop/</a></p>
<p><strong>反正我是看不懂的。</strong></p>
<h2 id="Hadoop发行版">Hadoop发行版</h2><p>Hadoop最早是始于<code>Yahoo</code>,但是现在主要是由 <code>HortonWorks</code> 和 <code>Cloudera</code> 这2家公司在维护者。<br><code>Hadoop</code> 发行版本将Hadoop生态系统所包含的不同项目打包在一起，提供打包、工具和商业支持。而这些不仅极大地简化了开发，而且也极大地简化了操作。脱颖而出的发行版有：</p>
<ul>
<li><strong>Cloudera：</strong>最成型的发行版本，拥有最多的部署案例。提供强大的部署、管理和监控工具。Cloudera开发并贡献了可实时处理大数据的Impala项目。</li>
<li><strong>Hortonworks：</strong>拥有任何私有（非开源）修改地使用了100%开源Apache Hadoop的唯一提供商。<strong>因为开源，所以我喜欢。</strong></li>
<li><strong>MapR：</strong>百度告诉你…</li>
</ul>
<h1 id="环境介绍">环境介绍</h1><p>我是在自建的 <code>Openstack</code> 创建云服务器。<br><img src="http://7xj3vi.com1.z0.glb.clouddn.com/hadoop-servers.png" alt="Openstack云服务器"></p>
<ul>
<li><strong>Ambari Server：</strong>用来安装Ambari，一站式部署、管理和监控Hadoop</li>
<li><strong>DB Server：</strong>独立一台服务器用做数据库服务器，安装Mysql6.6</li>
<li><strong>Node1~Node9：</strong>安装Hadoop以及生态系统</li>
</ul>
<p>所有服务器安装Centos6.6-64Bit-minimal，当然我这里用的是自制的Openstack的Centos6.6-64Bit-minimal镜像。</p>
<p>我创建云服务器时候，登录验证是采用密钥形式无密码SSH的，所以云服务器之间是可以通过秘钥无密码相互访问SSH的，Or not，你需要配置你服务器之间无密码SSH，请问百度君。</p>
<h1 id="环境配置">环境配置</h1><p>讲了一堆废话，终于可以上机折腾了，这酸爽…</p>
<h2 id="所有节点基础配置">所有节点基础配置</h2><p>全体节点的服务器（<code>node1~node9</code>）都需要配置以下内容。<br>NND，这么多服务器是否怀念 <code>Puppet</code>，硬着头皮先续个服务器敲命令吧…</p>
<h3 id="更新系统">更新系统</h3><pre><code><span class="title">yum</span> upgrade -y
</code></pre><h3 id="禁用Selinux">禁用Selinux</h3><pre><code>vi <span class="regexp">/etc/</span>sysconfig<span class="regexp">/selinux</span>
</code></pre><p>编辑/etc/sysconfig/selinux文件，把SELINUX的值设置为<code>disabled</code></p>
<h3 id="关闭防火墙">关闭防火墙</h3><pre><code>chkconfig iptables off
chkconfig ip6tables off
<span class="keyword">service</span> iptables <span class="literal">stop</span>
<span class="keyword">service</span> ip6tables <span class="literal">stop</span>
</code></pre><h3 id="修改hosts，本地解析DNS">修改hosts，本地解析DNS</h3><pre><code>vi <span class="regexp">/etc/</span>hosts
</code></pre><p>编辑/etc/hosts，增加内容如下：</p>
<pre><code><span class="number">192.168</span>.<span class="number">111.5</span> ambari<span class="class">.hdp</span><span class="class">.novalocal</span>
<span class="number">192.168</span>.<span class="number">111.8</span> node1<span class="class">.hdp</span><span class="class">.novalocal</span>
<span class="number">192.168</span>.<span class="number">111.6</span> node2<span class="class">.hdp</span><span class="class">.novalocal</span>
<span class="number">192.168</span>.<span class="number">111.9</span> node3<span class="class">.hdp</span><span class="class">.novalocal</span>
<span class="number">192.168</span>.<span class="number">111.10</span> node4<span class="class">.hdp</span><span class="class">.novalocal</span>
<span class="number">192.168</span>.<span class="number">111.11</span> node5<span class="class">.hdp</span><span class="class">.novalocal</span>
<span class="number">192.168</span>.<span class="number">111.7</span> node6<span class="class">.hdp</span><span class="class">.novalocal</span>
<span class="number">192.168</span>.<span class="number">111.12</span> node7<span class="class">.hdp</span><span class="class">.novalocal</span>
<span class="number">192.168</span>.<span class="number">111.13</span> node8<span class="class">.hdp</span><span class="class">.novalocal</span>
<span class="number">192.168</span>.<span class="number">111.14</span> node9<span class="class">.hdp</span><span class="class">.novalocal</span>
<span class="number">192.168</span>.<span class="number">111.15</span> db<span class="class">.hdp</span><span class="class">.novalocal</span>
</code></pre><h3 id="安装JDK">安装JDK</h3><pre><code>rpm -iv jdk-<span class="number">8u</span>45-linux-x64.rpm
</code></pre><p>JDK请到官网下载，<a href="http://www.oracle.com/technetwork/java/javase/downloads/index.html" title="下载Java SE" target="_blank" rel="external">Java SE Downloads</a></p>
<h3 id="配置环境变量">配置环境变量</h3><pre><code><span class="keyword">vi</span> /etc/<span class="keyword">profile</span>
</code></pre><p>编辑/etc/profile文件，在文件结尾增加内容：</p>
<pre><code><span class="preprocessor">#for java by jacken</span>
J<span class="built_in">AVA_HOME</span>=/usr/java/<span class="keyword">default</span>
J<span class="built_in">AVA</span>=$J<span class="built_in">AVA_HOME</span>/bin
<span class="keyword">export</span> J<span class="built_in">AVA_HOME</span>=$J<span class="built_in">AVA_HOME</span>
<span class="keyword">export</span> PATH=$PATH:$J<span class="built_in">AVA</span>
</code></pre><p>关闭，不使用透明内存。配置<code>transparent huge page</code>的参数，如果不设置，安装过程会有如下的检验错误：<br><img src="http://7xj3vi.com1.z0.glb.clouddn.com/transparent_hugepage_issues.png" alt="transparent_hugepage_issues"></p>
<p>编辑/etc/rc.local文件</p>
<pre><code>vi /etc/rc.<span class="keyword">local</span>
</code></pre><p>在文件结尾增加内容：</p>
<pre><code><span class="keyword">if</span> <span class="built_in">test</span> <span class="operator">-f</span> /sys/kernel/mm/transparent_hugepage/enabled; <span class="keyword">then</span>
<span class="built_in">echo</span> never > /sys/kernel/mm/transparent_hugepage/enabled
<span class="keyword">fi</span>
<span class="keyword">if</span> <span class="built_in">test</span> <span class="operator">-f</span> /sys/kernel/mm/transparent_hugepage/defrag; <span class="keyword">then</span>
<span class="built_in">echo</span> never > /sys/kernel/mm/transparent_hugepage/defrag
<span class="keyword">fi</span>
</code></pre><h2 id="本地yum源配置">本地yum源配置</h2><p>加快安装速度，提高安装成功率，建议配置本地yum源，源内容有两部分：</p>
<ul>
<li><strong>Ambari Repositories</strong></li>
<li><strong>HDP Stack Repositories</strong></li>
</ul>
<h3 id="安装yum工具">安装yum工具</h3><pre><code>yum <span class="keyword">install</span> yum-utils createrepo
</code></pre><p>创建源内容保存的目录</p>
<pre><code>mkdir -p <span class="regexp">/var/</span>www<span class="regexp">/html/</span>
</code></pre><h3 id="确定_Ambari和HDP_Stack源是否存在">确定 Ambari和HDP Stack源是否存在</h3><pre><code><span class="title">yum</span> repolist
</code></pre><p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/repolist.png" alt="repolist"></p>
<p>如果不存在，下载repo文件：</p>
<pre><code>wget -nv <span class="string">http:</span><span class="comment">//public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.0.1/ambari.repo -O /etc/yum.repos.d/ambari.repo</span>
wget -nv <span class="string">http:</span><span class="comment">//public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.2.4.2/hdp.repo -O /etc/yum.repos.d/HDP.repo</span>
</code></pre><h3 id="同步源内容到本地">同步源内容到本地</h3><pre><code><span class="keyword">cd</span> /<span class="keyword">var</span>/www/html
<span class="keyword">mkdir</span> -p ambari/centos6
<span class="keyword">cd</span> ambari/centos6
reposync -r Updates-ambari-2.0.1
<span class="keyword">cd</span> /<span class="keyword">var</span>/www/html
<span class="keyword">mkdir</span> -p hdp/centos6
<span class="keyword">cd</span> hdp/centos6
reposync -r HDP-2.2.4.2
reposync -r HDP-UTILS-1.1.0.20
</code></pre><p>同步网络上的源内容到本地，还需要生成源的<code>metadata</code></p>
<pre><code>createrepo <span class="regexp">/var/</span>www<span class="regexp">/html/</span>ambari<span class="regexp">/centos6/U</span>pdates-ambari-<span class="number">2.0</span>.<span class="number">1</span>
createrepo <span class="regexp">/var/</span>www<span class="regexp">/html/</span>hdp<span class="regexp">/centos6/</span>HDP-<span class="number">2.2</span>.<span class="number">4.2</span>
createrepo <span class="regexp">/var/</span>www<span class="regexp">/html/</span>hdp<span class="regexp">/centos6/</span>HDP-UTILS-<span class="number">1.1</span>.<span class="number">0.20</span>
</code></pre><h3 id="编译安装Nginx">编译安装Nginx</h3><p>把本地源内容，通过http提供对外访问，当时，你任性也可以用<code>Apache HTTP Server</code><br>安装gcc编译器及相关工具：</p>
<pre><code>yum -y install gcc make openssl-devel pcre pcre-devel
wget http:<span class="comment">//nginx.org/download/nginx-1.8.0.tar.gz</span>
tar -zxvf nginx-<span class="number">1.8</span>.<span class="number">0</span><span class="class">.tar</span><span class="class">.gz</span>
cd nginx-<span class="number">1.8</span>.<span class="number">0</span>
</code></pre><p>编译安装Nginx</p>
<pre><code><span class="comment">groupadd</span> <span class="comment">www</span>
<span class="comment">useradd</span> <span class="comment">www</span> <span class="literal">-</span><span class="comment">g</span> <span class="comment">www</span> <span class="literal">-</span><span class="comment">M</span> <span class="literal">-</span><span class="comment">s</span> <span class="comment">/sbin/nologin</span>
<span class="string">.</span><span class="comment">/configure</span> <span class="literal">-</span><span class="literal">-</span><span class="comment">prefix=/opt/nginx</span> <span class="literal">-</span><span class="literal">-</span><span class="comment">user=www</span> <span class="literal">-</span><span class="literal">-</span><span class="comment">group=www</span> <span class="literal">-</span><span class="literal">-</span><span class="comment">with</span><span class="literal">-</span><span class="comment">http_ssl_module</span> <span class="literal">-</span><span class="literal">-</span><span class="comment">with</span><span class="literal">-</span><span class="comment">http_stub_status_module</span> <span class="literal">-</span><span class="literal">-</span><span class="comment">with</span><span class="literal">-</span><span class="comment">http_sub_module</span>
<span class="comment">make</span> <span class="comment">&&</span> <span class="comment">make</span> <span class="comment">install</span>
</code></pre><p>编辑/opt/nginx/config/nginx.cnf文件，内容请参考<a href="http://7xj3vi.com1.z0.glb.clouddn.com/nginx.conf" title="nginx.conf参考" target="_blank" rel="external">nginx.conf</a><br>同时，把Nginx增加到系统服务，实现开机自动开启。<br>编辑/etc/init.d/nginx文件，内容请参考<a href="http://7xj3vi.com1.z0.glb.clouddn.com/nginx" title="nginx服务脚本" target="_blank" rel="external">nginx</a></p>
<pre><code>chmod <span class="operator">a</span>+x /etc/init.d/nginx
chkconfig <span class="comment">--add nginx</span>
chkconfig nginx <span class="command"><span class="keyword">on</span></span>
</code></pre><h3 id="本地源的信息">本地源的信息</h3><p>URLs for the New Repository，要记住，往后需要用到<br>Ambari Base URL：</p>
<pre><code><span class="string">http:</span><span class="comment">//ambari.hdp.novalocal/centos6/Updates-ambari-2.0.1</span>
</code></pre><p>HDP Base URL：</p>
<pre><code><span class="symbol">http:</span>/<span class="regexp">/ambari.hdp.novalocal/hdp</span><span class="regexp">/centos6/</span><span class="constant">HDP-</span><span class="number">2.2</span>.<span class="number">4.2</span>
</code></pre><p>HDP-UTILS Base URL：</p>
<pre><code><span class="symbol">http:</span>/<span class="regexp">/ambari.hdp.novalocal/hdp</span><span class="regexp">/centos6/</span><span class="constant">HDP-UTILS-</span><span class="number">1.1</span>.<span class="number">0</span>.<span class="number">20</span>
</code></pre><p>把Ambari修改成本地的源。</p>
<p>编辑/etc/yum.repos.d/ambari.repo文件，内容如下：</p>
<pre><code><span class="title">[Updates-ambari-2.0.1]</span>
<span class="setting">name=<span class="value">ambari-<span class="number">2.0</span>.<span class="number">1</span> - Updates</span></span>
<span class="setting">baseurl=<span class="value">http://ambari.hdp.novalocal/ambari/centos6/Updates-ambari-<span class="number">2.0</span>.<span class="number">1</span></span></span>
<span class="setting">gpgcheck=<span class="value"><span class="number">0</span></span></span>
<span class="setting">enabled=<span class="value"><span class="number">1</span></span></span>
<span class="setting">priority=<span class="value"><span class="number">1</span></span></span>
</code></pre><p>清理yum缓存</p>
<pre><code><span class="keyword">yum</span> clean <span class="literal">all</span>
</code></pre><h2 id="Ambari安装">Ambari安装</h2><p>这部分是在<code>ambari.hdp.novalocal</code>服务器上配置，其他节点无需操作。</p>
<p>安装Ambari</p>
<pre><code>yum <span class="operator"><span class="keyword">install</span> ambari-<span class="keyword">server</span></span>
</code></pre><p>我计划用Mysql作为数据库存储Ambari、hive和oozie数据，下载Mysql的驱动（<a href="http://dev.mysql.com/downloads/connector/j/" target="_blank" rel="external">http://dev.mysql.com/downloads/connector/j/</a>），解压放在/usr/share/java/mysql-connector-java-5.1.35-bin.jar。<br>在<code>db.hdp.novalocal</code>节点安装和配置Mysql 5.6，并创建名为<code>ambari</code>的数据库，执行<code>ambari.hdp.novalocal</code>上的<code>/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql</code>SQL脚本来初始化表结构。<br>配置Ambari Server:</p>
<pre><code>ambari-<span class="keyword">server</span> setup -j /usr/java/<span class="keyword">default</span>
</code></pre><p><img src="http://7xj3vi.com1.z0.glb.clouddn.com/ambari-install.png" alt="安装Ambari"><br>根据提示一直按回车，配置数据库时候需要指定<code>db.hdp.novalocal</code>的连接信息，至此，配置<code>Ambari Server</code>成功。</p>
<h2 id="启动Ambari">启动Ambari</h2><pre><code>chkconfig ambari-server <span class="command"><span class="keyword">on</span></span>
service ambari-server <span class="built_in">start</span>
</code></pre><h1 id="一个神奇的网站">一个神奇的网站</h1><p> <code>http://</code>your.ambari.server<code>:8080</code>,例如我的是<a href="http://172.16.1.140:8080" target="_blank" rel="external">http://172.16.1.140:8080</a></p>
<p>你可以通过这个网站来部署、管理和监控你的Hadoop。<br>对了偷偷告诉你，网站的默认管理员账号和密码是<code>admin/admin</code>。</p>
]]></content>
<summary type="html">
<![CDATA[<h1 id="本系列文章说明">本系列文章说明</h1><p>当我脑海里萌发要写这系列的文章时候，其实我是抗拒的！因为我觉得呀……这是大数据呀，我一个屌丝码农哪有这个能耐。但是又觉得博客总得写点内容，装逼装逼也得附和社会呀，加上看了看<code>Evernote</code>记录的内容甚乱，必须写……后来写着写着真觉得很DUANG~DUANG~DUANG~…..<br>其实在项目中用到Hadoop的功能甚少，一个HDFS来存储非结构数据，一个是Mapreduce来分布计算处理数据。然而，就算写个简单的<code>Hello world</code>也得把环境搭建起来。所以，本系列的文章，权当我从0开始学习Hadoop的笔记，希望能和像我一样的新手分享经验，同时也希望得到高人指点一二三四五… </p>
<hr>
<p><strong>本部分内容</strong></p>
<ul>
<li>Hadoop生态圈</li>
<li>本系类文章的环境介绍</li>
<li>Ambari安装<ul>
<li>所有节点基础配置</li>
<li>本地yum源配置</li>
<li>Ambari安装</li>
</ul>
</li>
<li>一个神奇的网址]]>
</summary>
<category term="ambari" scheme="http://jacken.me/tags/ambari/"/>
<category term="hadoop" scheme="http://jacken.me/tags/hadoop/"/>
<category term="Hadoop" scheme="http://jacken.me/categories/Hadoop/"/>
</entry>
<entry>
<title><![CDATA[Hello World]]></title>
<link href="http://jacken.me/Hexo/hello-world/"/>
<id>http://jacken.me/Hexo/hello-world/</id>
<published>2015-05-14T18:44:45.000Z</published>
<updated>2015-07-06T06:46:12.096Z</updated>
<content type="html"><![CDATA[<h1 id="又一个博客">又一个博客</h1><p>以前一直有写博客，博客使用<code>WordPress</code>搭建的。写一段时间后，写作热情下降且空间过期，懒得继续折腾而终止，总是持续不了。</p>
<p>最近接触很多新的知识和技术，一直记录在<code>Evernote</code>，且记录得有些凌乱，借此重出江湖，借此整理好感兴趣的内容并发表到博客上，方便自己查看同时也有一丝开源分享的精神，互相学习吧。</p>
<p>理所当然，本博客所有文章默认都遵循 <a href="http://creativecommons.org/licenses/by-nc-sa/4.0/" target="_blank" rel="external"><img src="https://licensebuttons.net/l/by-nc-sa/3.0/88x31.png" alt="" title="BY-NC-SA"></a>；如发现文章对你有侵权嫌疑，请联系我。<a id="more"></a></p>
<h1 id="是个什么样的博客">是个什么样的博客</h1><p>以前都是用牛逼哄哄的<code>WordPress</code>，再购买个PHP空间搭建起来，折腾一会就可以写文章了。然而，坑爹的空间总时不时会出些问题，打击写文章的热情。</p>
<p>雷总也说了<strong>「你不要用战术上的勤奋掩盖战略上的懒惰」</strong>，这次，改变战略： </p>
<ul>
<li><strong>静态博客:</strong>果断使用<a href="http://hexo.io/" title="Hexo" target="_blank" rel="external">Hexo</a> 。当然你还可以使用这些<code>FarBox</code>、<code>Jekyll</code>、<code>Octopress</code>、或这些<code>ghost</code>、<code>marboo</code>、<code>Medium</code>、<code>Logdown</code>、<code>prose.io</code>… </li>
<li><strong>空间:</strong>咱穷，直接用<a href="https://pages.github.com/" title="Github Pages" target="_blank" rel="external">github Pages</a>和<a href="https://gitcafe.com/GitCafe/Help/wiki/Pages-%E7%9B%B8%E5%85%B3%E5%B8%AE%E5%8A%A9#wiki" title="Gitcafe Pages" target="_blank" rel="external">Gitcafe Pages</a>，算是备份吧。当然，你任性，可以买个服务器、阿里云ECS…都是可以的 </li>
<li><strong>域名:</strong>个人博客，域名我注册了跟我网名一样名字的。例如<a href="http://jakcen.me" title="Jacken的博客网址" target="_blank" rel="external">http://jakcen.me</a>、<a href="http://jakcen.cn" title="Jacken的博客网址" target="_blank" rel="external">http://jakcen.cn</a>、<a href="http://jakcen.com.cn" title="还是Jacken的博客网址" target="_blank" rel="external">http://jakcen.com.cn</a>…都是可以的 </li>
<li><strong>内容:</strong>很明显这是一个技术博客，但是我尽量不写技术文章，多写生活，感谢！</li>
</ul>
<h1 id="Hexo">Hexo</h1><ul>
<li><strong>Hexo</strong>怎么使用？建议直接看<a href="http://hexo.io/zh-cn/docs/" title="Hexo官网文档" target="_blank" rel="external">Hexo官网文档</a>。 </li>
<li><strong>Hexo主题</strong>我使用了<a href="https://github.com/wuchong/jacman" title="Hexo主题jacman" target="_blank" rel="external">jacman</a>，怎么使用可以看作者的说明<a href="http://wuchong.me/blog/2014/11/20/how-to-use-jacman/" title="如何使用 Jacman 主题" target="_blank" rel="external">如何使用 Jacman 主题</a> </li>
<li><strong>Hexo插件</strong>我使用了<code>hexo-deployer-git</code>、<code>hexo-generator-archive</code>、<br><code>hexo-generator-baidu-sitemap</code>、<code>hexo-generator-category</code>、<code>hexo-generator-feed</code>、<br><code>hexo-generator-sitemap</code>、<code>hexo-generator-tag</code>等。当然在使用过程中会有变动的，玩的就是敏捷嘛。更多插件，请看<a href="http://hexo.io/plugins/" title="Hexo插件" target="_blank" rel="external">Hexo plugins</a> </li>
<li><strong>图床</strong>图片存储我用牛哥—七牛云存储，免费的流量目前够我用，马上注册<a href="https://portal.qiniu.com/signup?code=3la79fwnm1emq" title="注册七牛云存储" target="_blank" rel="external">点这里</a> </li>
<li><strong>文章写作工具</strong>我用的是<a href="http://markdownpad.com/" title="markdownpad 2官网" target="_blank" rel="external">markdownpad 2</a>，用<code>Markdown</code>来写博客，爽歪歪呢。新手学习Markdown语法可以看看<a href="http://www.appinn.com/markdown/" title="Markdown 语法说明 (简体中文版)" target="_blank" rel="external">这个网页</a></li>
<li><strong>Pages服务</strong>我用了<a href="https://pages.github.com/" title="Github Pages" target="_blank" rel="external">github Pages</a>和<a href="https://gitcafe.com/GitCafe/Help/wiki/Pages-%E7%9B%B8%E5%85%B3%E5%B8%AE%E5%8A%A9#wiki" title="Gitcafe Pages" target="_blank" rel="external">Gitcafe Pages</a>，具体怎么用和怎么发布，请问问度娘</li>
</ul>
<h1 id="The_last">The last</h1><p>希望电脑不要坏，一直可以用的写作环境<br>希望Github Pages不被GWF，速度快点；希望Gitface Pages不被政策限制</p>
<p>世界太大，人太小。<br>这里写的一切仅仅是记录，记录生活，记录学习，记录成长。</p>
<p>别问我是谁，我是<a href="/about" title="I'm Jacken.">Jacken</a></p>
]]></content>
<summary type="html">
<![CDATA[<h1 id="又一个博客">又一个博客</h1><p>以前一直有写博客，博客使用<code>WordPress</code>搭建的。写一段时间后，写作热情下降且空间过期，懒得继续折腾而终止，总是持续不了。</p>
<p>最近接触很多新的知识和技术，一直记录在<code>Evernote</code>，且记录得有些凌乱，借此重出江湖，借此整理好感兴趣的内容并发表到博客上，方便自己查看同时也有一丝开源分享的精神，互相学习吧。</p>
<p>理所当然，本博客所有文章默认都遵循 <a href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img src="https://licensebuttons.net/l/by-nc-sa/3.0/88x31.png" alt="" title="BY-NC-SA"></a>；如发现文章对你有侵权嫌疑，请联系我。]]>
</summary>
<category term="Hello" scheme="http://jacken.me/tags/Hello/"/>
<category term="Hexo" scheme="http://jacken.me/tags/Hexo/"/>
<category term="Hexo" scheme="http://jacken.me/categories/Hexo/"/>
</entry>
</feed>