-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathlocal-search.xml
704 lines (333 loc) · 692 KB
/
local-search.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title>黑马python第二阶段第一章</title>
<link href="/2025/01/25/%E9%BB%91%E9%A9%ACpython%E7%AC%AC%E4%BA%8C%E9%98%B6%E6%AE%B5%E7%AC%AC%E4%B8%80%E7%AB%A0/"/>
<url>/2025/01/25/%E9%BB%91%E9%A9%ACpython%E7%AC%AC%E4%BA%8C%E9%98%B6%E6%AE%B5%E7%AC%AC%E4%B8%80%E7%AB%A0/</url>
<content type="html"><![CDATA[<h1 id="黑马python第二阶段第一章"><a href="#黑马python第二阶段第一章" class="headerlink" title="黑马python第二阶段第一章"></a>黑马python第二阶段第一章</h1><h2 id="类"><a href="#类" class="headerlink" title="类"></a>类</h2><h3 id="定义和调用"><a href="#定义和调用" class="headerlink" title="定义和调用"></a>定义和调用</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 定义一个类</span><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">Dog</span>:<br> <span class="hljs-comment"># 成员变量，类的属性</span><br> breed = <span class="hljs-string">"Golden Retriever"</span><br> <br> <span class="hljs-comment"># 成员方法，类的行为</span><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">bark</span>(<span class="hljs-params">self, sound</span>):<br> <span class="hljs-comment"># 通过 self 访问成员变量，访问类内的成员变量时需要加self，外部参数不需要</span><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">f"The <span class="hljs-subst">{self.breed}</span> says <span class="hljs-subst">{sound}</span>"</span>)<br><br><span class="hljs-comment"># 创建类的实例</span><br>my_dog = Dog()<br><br><span class="hljs-comment"># 调用成员方法，self不需要传</span><br>my_dog.bark(<span class="hljs-string">"Woof!"</span>)<br></code></pre></td></tr></table></figure><h3 id="构造方法"><a href="#构造方法" class="headerlink" title="构造方法"></a>构造方法</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 定义一个类</span><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">Dog</span>:<br> <span class="hljs-comment"># 构造方法</span><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, breed, age</span>):<br> <span class="hljs-comment"># 使用 self 初始化成员变量</span><br> <span class="hljs-variable language_">self</span>.breed = breed<br> <span class="hljs-variable language_">self</span>.age = age<br> <br> <span class="hljs-comment"># 成员方法</span><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">display_info</span>(<span class="hljs-params">self</span>):<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">f"This dog is a <span class="hljs-subst">{self.breed}</span> and is <span class="hljs-subst">{self.age}</span> years old."</span>)<br><br><span class="hljs-comment"># 创建类的实例，并传递参数给构造方法</span><br>my_dog = Dog(<span class="hljs-string">"Golden Retriever"</span>, <span class="hljs-number">3</span>)<br><br><span class="hljs-comment"># 调用成员方法</span><br>my_dog.display_info()<br></code></pre></td></tr></table></figure><h3 id="其他类内置方法（魔术方法）"><a href="#其他类内置方法（魔术方法）" class="headerlink" title="其他类内置方法（魔术方法）"></a>其他类内置方法（魔术方法）</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">class</span> <span class="hljs-title class_">Dog</span>:<br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, name, age</span>):<br> <span class="hljs-variable language_">self</span>.name = name<br> <span class="hljs-variable language_">self</span>.age = age<br><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__str__</span>(<span class="hljs-params">self</span>):<br> <span class="hljs-keyword">return</span> <span class="hljs-string">f"<span class="hljs-subst">{self.name}</span> is <span class="hljs-subst">{self.age}</span> years old"</span><br><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__lt__</span>(<span class="hljs-params">self, other</span>):<br> <span class="hljs-keyword">return</span> <span class="hljs-variable language_">self</span>.age < other.age<br><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__le__</span>(<span class="hljs-params">self, other</span>):<br> <span class="hljs-keyword">return</span> <span class="hljs-variable language_">self</span>.age <= other.age<br><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__eq__</span>(<span class="hljs-params">self, other</span>):<br> <span class="hljs-keyword">return</span> <span class="hljs-variable language_">self</span>.age == other.age<br><br><span class="hljs-comment"># 创建两个 Dog 对象</span><br>dog1 = Dog(<span class="hljs-string">"Buddy"</span>, <span class="hljs-number">5</span>)<br>dog2 = Dog(<span class="hljs-string">"Max"</span>, <span class="hljs-number">3</span>)<br><br><span class="hljs-comment"># 使用 __str__ 方法</span><br><span class="hljs-built_in">print</span>(dog1) <span class="hljs-comment"># 输出: Buddy is 5 years old</span><br><br><span class="hljs-comment"># 使用 __lt__ 方法</span><br><span class="hljs-built_in">print</span>(dog1 < dog2) <span class="hljs-comment"># 输出: False</span><br><br><span class="hljs-comment"># 使用 __le__ 方法</span><br><span class="hljs-built_in">print</span>(dog1 <= dog2) <span class="hljs-comment"># 输出: False</span><br><br><span class="hljs-comment"># 使用 __eq__ 方法</span><br><span class="hljs-built_in">print</span>(dog1 == dog2) <span class="hljs-comment"># 输出: False</span><br></code></pre></td></tr></table></figure><p><strong>当没有使用__eq__方法时若使用==，则对比的是内存地址。</strong></p><h3 id="私有成员"><a href="#私有成员" class="headerlink" title="私有成员"></a>私有成员</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">class</span> <span class="hljs-title class_">Dog</span>:<br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, name, age</span>):<br> <span class="hljs-variable language_">self</span>.name = name <span class="hljs-comment"># 公有成员变量</span><br> <span class="hljs-variable language_">self</span>.__age = age <span class="hljs-comment"># 私有成员变量</span><br><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">display_info</span>(<span class="hljs-params">self</span>):<br> <span class="hljs-comment"># 类内部可以访问私有成员变量</span><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">f"<span class="hljs-subst">{self.name}</span> is <span class="hljs-subst">{self.__age}</span> years old"</span>)<br><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__private_method</span>(<span class="hljs-params">self</span>):<br> <span class="hljs-comment"># 私有方法</span><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"This is a private method"</span>)<br><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">public_method</span>(<span class="hljs-params">self</span>):<br> <span class="hljs-comment"># 公有方法可以调用私有方法</span><br> <span class="hljs-variable language_">self</span>.__private_method()<br><br><span class="hljs-comment"># 创建 Dog 对象</span><br>my_dog = Dog(<span class="hljs-string">"Buddy"</span>, <span class="hljs-number">5</span>)<br><br><span class="hljs-comment"># 访问公有成员变量和方法</span><br><span class="hljs-built_in">print</span>(my_dog.name) <span class="hljs-comment"># 输出: Buddy</span><br>my_dog.display_info() <span class="hljs-comment"># 输出: Buddy is 5 years old</span><br><br><span class="hljs-comment"># 尝试访问私有成员变量（会报错）</span><br><span class="hljs-comment"># print(my_dog.__age) # AttributeError: 'Dog' object has no attribute '__age'</span><br><br><span class="hljs-comment"># 尝试调用私有方法（会报错）</span><br><span class="hljs-comment"># my_dog.__private_method() # AttributeError: 'Dog' object has no attribute '__private_method'</span><br><br><span class="hljs-comment"># 通过公有方法调用私有方法</span><br>my_dog.public_method() <span class="hljs-comment"># 输出: This is a private method</span><br></code></pre></td></tr></table></figure><p>私有方法可以调用公开变量，公开方法也可以调用私有变量。在类的内部，无论是私有方法还是公开方法，都可以访问类的<strong>所有成员变量</strong>。这是因为访问权限的限制主要是针对类的<strong>外部，而不是类的内部</strong>。</p><h3 id="继承"><a href="#继承" class="headerlink" title="继承"></a>继承</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 第一个父类</span><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">A</span>:<br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">method</span>(<span class="hljs-params">self</span>):<br> <span class="hljs-keyword">return</span> <span class="hljs-string">"Method from class A"</span><br><br><span class="hljs-comment"># 第二个父类</span><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">B</span>:<br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">method</span>(<span class="hljs-params">self</span>):<br> <span class="hljs-keyword">return</span> <span class="hljs-string">"Method from class B"</span><br><br><span class="hljs-comment"># 子类继承多个父类</span><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">C</span>(A, B):<br><span class="hljs-comment"># 通常用作占位符，用于在语法上需要语句但不需要执行任何操作的地方。</span><br> <span class="hljs-keyword">pass</span><br><br><span class="hljs-comment"># 创建子类对象</span><br>my_object = C()<br><span class="hljs-built_in">print</span>(my_object.method()) <span class="hljs-comment"># 输出: Method from class A</span><br></code></pre></td></tr></table></figure><p>当两个类中有相同的成员或方法名时，左边的优先级更高。</p><h3 id="复写"><a href="#复写" class="headerlink" title="复写"></a>复写</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 父类</span><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">Animal</span>:<br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, name</span>):<br> <span class="hljs-variable language_">self</span>.name = name<br><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">speak</span>(<span class="hljs-params">self</span>):<br> <span class="hljs-keyword">return</span> <span class="hljs-string">f"<span class="hljs-subst">{self.name}</span> makes a sound"</span><br><br><span class="hljs-comment"># 子类</span><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">Dog</span>(<span class="hljs-title class_ inherited__">Animal</span>):<br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, name, breed</span>):<br> <span class="hljs-comment"># 使用 super() 调用父类的构造方法</span><br> <span class="hljs-built_in">super</span>().__init__(name)<br> <span class="hljs-variable language_">self</span>.breed = breed<br><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">speak</span>(<span class="hljs-params">self</span>):<br> <span class="hljs-comment"># 复写父类的 speak 方法</span><br> <span class="hljs-comment"># 使用 super() 调用父类的 speak 方法</span><br> parent_speak = <span class="hljs-built_in">super</span>().speak()<br> <span class="hljs-comment"># 使用父类名调用父类的 speak 方法（效果与 super() 相同）</span><br> parent_speak_alt = Animal.speak(<span class="hljs-variable language_">self</span>)<br> <span class="hljs-keyword">return</span> <span class="hljs-string">f"<span class="hljs-subst">{parent_speak}</span> and <span class="hljs-subst">{self.name}</span> barks (breed: <span class="hljs-subst">{self.breed}</span>)"</span><br><br><span class="hljs-comment"># 创建子类对象</span><br>my_dog = Dog(<span class="hljs-string">"Buddy"</span>, <span class="hljs-string">"Golden Retriever"</span>)<br><br><span class="hljs-comment"># 调用复写后的方法</span><br><span class="hljs-built_in">print</span>(my_dog.speak())<br><span class="hljs-comment"># 输出: Buddy makes a sound and Buddy barks (breed: Golden Retriever)</span><br></code></pre></td></tr></table></figure><h3 id="类型注解"><a href="#类型注解" class="headerlink" title="类型注解"></a>类型注解</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 变量的类型注解</span><br>name: <span class="hljs-built_in">str</span> = <span class="hljs-string">"Alice"</span><br>age: <span class="hljs-built_in">int</span> = <span class="hljs-number">25</span><br><span class="hljs-comment"># 注释中的注解也可以被IDE识别，但如果标错了也不会报错</span><br>height = <span class="hljs-number">170</span> <span class="hljs-comment"># type: <span class="hljs-built_in">int</span></span><br><br><span class="hljs-comment"># 函数的类型注解</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">greet</span>(<span class="hljs-params">name: <span class="hljs-built_in">str</span></span>) -> <span class="hljs-built_in">str</span>:<br> <span class="hljs-keyword">return</span> <span class="hljs-string">f"Hello, <span class="hljs-subst">{name}</span>"</span><br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">add</span>(<span class="hljs-params">a: <span class="hljs-built_in">int</span>, b: <span class="hljs-built_in">int</span></span>) -> <span class="hljs-built_in">int</span>:<br> <span class="hljs-keyword">return</span> a + b<br><br><span class="hljs-comment"># 类型注解的灵活性</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">multiply</span>(<span class="hljs-params">a: <span class="hljs-built_in">int</span>, b: <span class="hljs-built_in">int</span></span>) -> <span class="hljs-built_in">int</span>:<br> <span class="hljs-keyword">return</span> a * b<br><br><span class="hljs-comment"># 调用函数</span><br><span class="hljs-built_in">print</span>(greet(<span class="hljs-string">"Bob"</span>)) <span class="hljs-comment"># 输出: Hello, Bob</span><br><span class="hljs-built_in">print</span>(add(<span class="hljs-number">3</span>, <span class="hljs-number">5</span>)) <span class="hljs-comment"># 输出: 8</span><br><br><span class="hljs-comment"># 即使传入的参数类型与注解不匹配，也不会报错</span><br>result = multiply(<span class="hljs-number">3.5</span>, <span class="hljs-number">2.5</span>) <span class="hljs-comment"># 输出: 8.75</span><br><span class="hljs-built_in">print</span>(result)<br></code></pre></td></tr></table></figure><h5 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h5><ul><li><strong>类型注解不会影响运行时的行为</strong>。</li><li><strong>静态类型检查工具</strong>：可以在开发阶段发现类型不匹配的问题，但不会影响代码运行。</li><li><strong>Python的动态特性</strong>：Python允许在运行时传入任何类型的参数，只要操作是合法的。<br>如果希望强制类型检查，可以使用静态类型检查工具（如 <code>mypy</code>），或者在运行时手动检查类型（例如使用 <code>isinstance</code>）。</li></ul><h4 id="联合类型注解"><a href="#联合类型注解" class="headerlink" title="联合类型注解"></a>联合类型注解</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> typing <span class="hljs-keyword">import</span> <span class="hljs-type">Union</span><br><br><span class="hljs-comment"># 变量的联合类型注解</span><br>value: <span class="hljs-type">Union</span>[<span class="hljs-built_in">int</span>, <span class="hljs-built_in">str</span>] = <span class="hljs-number">42</span><br>value = <span class="hljs-string">"hello"</span><br><br><span class="hljs-comment"># 函数的联合类型注解</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">process_value</span>(<span class="hljs-params">input_value: <span class="hljs-type">Union</span>[<span class="hljs-built_in">int</span>, <span class="hljs-built_in">str</span>]</span>) -> <span class="hljs-type">Union</span>[<span class="hljs-built_in">int</span>, <span class="hljs-built_in">str</span>]:<br> <span class="hljs-keyword">if</span> <span class="hljs-built_in">isinstance</span>(input_value, <span class="hljs-built_in">int</span>):<br> <span class="hljs-keyword">return</span> input_value * <span class="hljs-number">2</span><br> <span class="hljs-keyword">elif</span> <span class="hljs-built_in">isinstance</span>(input_value, <span class="hljs-built_in">str</span>):<br> <span class="hljs-keyword">return</span> input_value.upper()<br><br><span class="hljs-comment"># 使用示例</span><br><span class="hljs-built_in">print</span>(process_value(<span class="hljs-number">10</span>)) <span class="hljs-comment"># 输出: 20</span><br><span class="hljs-built_in">print</span>(process_value(<span class="hljs-string">"hello"</span>)) <span class="hljs-comment"># 输出: HELLO</span><br></code></pre></td></tr></table></figure><h3 id="多态和抽象类"><a href="#多态和抽象类" class="headerlink" title="多态和抽象类"></a>多态和抽象类</h3><h4 id="1-多态"><a href="#1-多态" class="headerlink" title="1. 多态"></a>1. <strong>多态</strong></h4><ul><li>同一行为，不同对象表现不同。</li></ul><h4 id="2-抽象类"><a href="#2-抽象类" class="headerlink" title="2. 抽象类"></a>2. <strong>抽象类</strong></h4><ul><li>包含未实现的方法（<code>pass</code>），用于定义标准，子类必须实现。</li></ul><h4 id="3-作用"><a href="#3-作用" class="headerlink" title="3. 作用"></a>3. <strong>作用</strong></h4><ul><li>多态：灵活调用。</li><li>抽象类：规范设计。</li></ul>]]></content>
<categories>
<category>python</category>
</categories>
</entry>
<entry>
<title>黑马python十一、十二章</title>
<link href="/2025/01/20/%E9%BB%91%E9%A9%ACpython%E5%8D%81%E4%B8%80%E3%80%81%E5%8D%81%E4%BA%8C%E7%AB%A0/"/>
<url>/2025/01/20/%E9%BB%91%E9%A9%ACpython%E5%8D%81%E4%B8%80%E3%80%81%E5%8D%81%E4%BA%8C%E7%AB%A0/</url>
<content type="html"><![CDATA[<h1 id="黑马python十一、十二章"><a href="#黑马python十一、十二章" class="headerlink" title="黑马python十一、十二章"></a>黑马python十一、十二章</h1><p>推荐网站：<a href="http://www.ab173.com/">www.ab173.com</a></p><p><strong><code>sort</code> 函数</strong>：</p><ul><li><p>是列表对象的方法，只能对列表进行操作。</p></li><li><p>会直接修改原始列表，而不会返回一个新的列表。</p></li><li><p>语法为 <code>list.sort(key=None, reverse=False)</code>，其中：</p></li></ul><p> - <code>key</code> 是一个函数，用于指定排序的依据，默认为 <code>None</code>。</p><p> - <code>reverse</code> 是一个布尔值，用于指定排序的顺序，默认为 <code>False</code>，表示升序排序，设置为 <code>True</code> 时为降序排序。</p><p><strong><code>sorted</code> 函数</strong>：</p><ul><li><p>是 Python 的内置函数，可以对任何可迭代对象进行操作，如列表、元组、集合等。</p></li><li><p>会返回一个新的已排序列表，不会修改原始的可迭代对象。</p></li><li><p>语法为 <code>sorted(iterable, key=None, reverse=False)</code>，其中：</p></li></ul><p> - <code>iterable</code> 是要排序的可迭代对象。</p><p> - <code>key</code> 和 <code>reverse</code> 参数的含义与 <code>sort</code> 方法中的相同。</p><p>以下是使用 <code>sort</code> 和 <code>sorted</code> 的示例代码：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><code class="hljs python"><br><span class="hljs-comment"># 使用 sorted 函数</span><br><br>students = [{<span class="hljs-string">'name'</span>: <span class="hljs-string">'Alice'</span>, <span class="hljs-string">'age'</span>: <span class="hljs-number">20</span>}, {<span class="hljs-string">'name'</span>: <span class="hljs-string">'Bob'</span>, <span class="hljs-string">'age'</span>: <span class="hljs-number">18</span>}, {<span class="hljs-string">'name'</span>: <span class="hljs-string">'Charlie'</span>, <span class="hljs-string">'age'</span>: <span class="hljs-number">22</span>}]<br><br>sorted_students = <span class="hljs-built_in">sorted</span>(students, key=<span class="hljs-keyword">lambda</span> x: x[<span class="hljs-string">'age'</span>])<br><br><span class="hljs-built_in">print</span>(sorted_students)<br><br><span class="hljs-built_in">print</span>(students) <span class="hljs-comment"># 原始列表 students 不变</span><br><br> <br> <br><br><span class="hljs-comment"># 使用 sort 方法</span><br><br>students = [{<span class="hljs-string">'name'</span>: <span class="hljs-string">'Alice'</span>, <span class="hljs-string">'age'</span>: <span class="hljs-number">20</span>}, {<span class="hljs-string">'name'</span>: <span class="hljs-string">'Bob'</span>, <span class="hljs-string">'age'</span>: <span class="hljs-number">18</span>}, {<span class="hljs-string">'name'</span>: <span class="hljs-string">'Charlie'</span>, <span class="hljs-string">'age'</span>: <span class="hljs-number">22</span>}]<br><br>students.sort(key=<span class="hljs-keyword">lambda</span> x: x[<span class="hljs-string">'age'</span>])<br><br><span class="hljs-built_in">print</span>(students) <span class="hljs-comment"># 原始列表 students 已被修改</span><br><br></code></pre></td></tr></table></figure><p>在使用时的注意事项：</p><ul><li><p>如果需要保留原始列表的顺序，使用 <code>sorted</code> 函数。</p></li><li><p>如果不需要保留原始列表，并且想要节省内存，使用 <code>sort</code> 方法。</p></li><li><p>对于其他可迭代对象（如元组、集合），只能使用 <code>sorted</code> 函数，因为它们没有 <code>sort</code> 方法。</p></li></ul>]]></content>
<categories>
<category>python</category>
</categories>
</entry>
<entry>
<title>黑马python九、十章</title>
<link href="/2025/01/19/%E9%BB%91%E9%A9%ACpython%E4%B9%9D%E3%80%81%E5%8D%81%E7%AB%A0/"/>
<url>/2025/01/19/%E9%BB%91%E9%A9%ACpython%E4%B9%9D%E3%80%81%E5%8D%81%E7%AB%A0/</url>
<content type="html"><![CDATA[<h1 id="黑马python九、十章"><a href="#黑马python九、十章" class="headerlink" title="黑马python九、十章"></a>黑马python九、十章</h1><h3 id="程序异常"><a href="#程序异常" class="headerlink" title="程序异常"></a>程序异常</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">try</span>:<br><br> <span class="hljs-comment"># 可能要发生异常的语句</span><br><br> result = <span class="hljs-number">10</span> / <span class="hljs-number">0</span> <br><br><span class="hljs-keyword">except</span> ZeroDivisionError <span class="hljs-keyword">as</span> e:<br><br> <span class="hljs-comment"># 出现异常的准备手段，这里捕获除零异常并打印错误信息</span><br><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">f"捕获到异常: <span class="hljs-subst">{e}</span>"</span>)<br><br><span class="hljs-keyword">else</span>:<br><br> <span class="hljs-comment"># 未出现异常时应做</span><br><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"没有异常发生，一切正常"</span>)<br><br><span class="hljs-keyword">finally</span>:<br><br> <span class="hljs-comment"># 不管出不出现异常都会做</span><br><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"这是finally块，总会执行"</span>)<br><br> <br><br><span class="hljs-comment"># 捕获所有异常的方式</span><br><br><span class="hljs-keyword">try</span>:<br><br> another_result = <span class="hljs-number">5</span> / <span class="hljs-number">0</span> <br><br><span class="hljs-keyword">except</span>:<br><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"捕获到了所有异常"</span>)<br></code></pre></td></tr></table></figure><p>其中，<code>except:</code>可以替换为<code>except Exception:</code>，也可以捕获所有异常。<br>异常具有传递性。</p><h3 id="导入代码块"><a href="#导入代码块" class="headerlink" title="导入代码块"></a>导入代码块</h3><p><strong>示例一：直接导入整个模块</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs python"><br><span class="hljs-keyword">import</span> math<br><br><span class="hljs-built_in">print</span>(math.sqrt(<span class="hljs-number">9</span>)) <br><br></code></pre></td></tr></table></figure><p>使用 <code>import math</code> 导入了Python的内置模块 <code>math</code>，然后可以通过 <code>模块名.函数名</code> 的方式（如 <code>math.sqrt</code>）来使用模块中的函数。</p><p><strong>示例二：从模块中导入特定函数或变量</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs python"><br><span class="hljs-keyword">from</span> math <span class="hljs-keyword">import</span> sqrt<br><br><span class="hljs-built_in">print</span>(sqrt(<span class="hljs-number">9</span>)) <br><br></code></pre></td></tr></table></figure><p>这里使用 <code>from math import sqrt</code> 从 <code>math</code> 模块中导入了 <code>sqrt</code> 函数，这样就可以直接使用 <code>sqrt</code> 函数而无需加上模块名前缀。</p><p><strong>示例三：导入模块并使用别名</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs python"><br><span class="hljs-keyword">import</span> math <span class="hljs-keyword">as</span> m<br><br><span class="hljs-built_in">print</span>(m.sqrt(<span class="hljs-number">9</span>)) <br><br></code></pre></td></tr></table></figure><p>通过 <code>import math as m</code> 给 <code>math</code> 模块起了一个别名 <code>m</code>，之后就可以用 <code>m</code> 来代替 <code>math</code> 使用模块中的内容。<br><strong>示例四：从模块中导入所有内容（不建议）</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs python"><br><span class="hljs-keyword">from</span> math <span class="hljs-keyword">import</span> *<br><br><span class="hljs-built_in">print</span>(sqrt(<span class="hljs-number">9</span>)) <br><br></code></pre></td></tr></table></figure><p><code>from math import *</code> 会将 <code>math</code> 模块中的所有内容导入到当前命名空间，这样可以直接使用模块中的函数和变量，但可能会导致命名冲突，所以一般不建议这样使用。<br><strong>当从不同的模块中导入同名的功能时，后导入的覆盖先导入的。</strong></p><h3 id="自定义模块导入"><a href="#自定义模块导入" class="headerlink" title="自定义模块导入"></a>自定义模块导入</h3><h4 id="main-的用法"><a href="#main-的用法" class="headerlink" title="__main__的用法"></a>__main__的用法</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">my_function</span>():<br><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"这是一个函数"</span>)<br><br> <br><br><span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:<br><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"程序直接执行，进入if内部"</span>)<br><br> my_function()<br><br><span class="hljs-keyword">else</span>:<br><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"程序被导入，不进入if内部"</span>)<br></code></pre></td></tr></table></figure><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs plaintext">程序直接执行，进入if内部<br>这是一个函数<br></code></pre></td></tr></table></figure><h4 id="all-的用法"><a href="#all-的用法" class="headerlink" title="__all__的用法"></a>__all__的用法</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 在module3.py中定义</span><br><br>__all__ = [<span class="hljs-string">'function1'</span>]<br><br> <br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">function1</span>():<br><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"这是function1"</span>)<br><br> <br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">function2</span>():<br><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"这是function2"</span>)<br><br> <br><br><span class="hljs-comment"># 在另一个文件中</span><br><br><span class="hljs-keyword">from</span> module3 <span class="hljs-keyword">import</span> *<br><br>function1() <br><br><span class="hljs-comment"># function2() # 由于module3中__all__只包含function1，所以这里如果取消注释会报错，但是若像下列代码导入，也可以正常使用，因为*对应的就是__all__，不阻止其他功能导入</span><br><br><span class="hljs-keyword">from</span> module3 <span class="hljs-keyword">import</span> function2<br><br>function2()<br></code></pre></td></tr></table></figure><ul><li><code>__init__.py</code>文件： <ul><li>包创建时自动生成，标识文件夹为Python包，可空或含初始化代码，导入包时执行。</li></ul></li><li><code>__all__</code>变量： <ul><li>在包的<code>__init__.py</code>中定义，控制<code>from 包名 import *</code>时导入的内容，使导入更清晰可控。</li></ul></li><li><code>package.module.function</code>的形式来导入。</li></ul><h3 id="JSON"><a href="#JSON" class="headerlink" title="JSON"></a>JSON</h3><ul><li><strong>概念</strong>：轻量数据交互格式，类似字符串，Python中可与字典或列表无缝转换。 </li><li><strong>转化</strong>： <ul><li><code>json.dumps(data)</code>：Python转JSON，含中文加<code>ensure_ascii=False</code>。 </li><li><code>json.loads(data)</code>：JSON转Python列表或字典。</li></ul></li></ul><p>pyecharts包可以处理图表，可以查看官方画廊。</p>]]></content>
<categories>
<category>python</category>
</categories>
</entry>
<entry>
<title>黑马python七、八章</title>
<link href="/2025/01/19/%E9%BB%91%E9%A9%ACpython%E4%B8%83%E3%80%81%E5%85%AB%E7%AB%A0/"/>
<url>/2025/01/19/%E9%BB%91%E9%A9%ACpython%E4%B8%83%E3%80%81%E5%85%AB%E7%AB%A0/</url>
<content type="html"><![CDATA[<h1 id="黑马python七、八章"><a href="#黑马python七、八章" class="headerlink" title="黑马python七、八章"></a>黑马python七、八章</h1><h3 id="函数的多种参数使用形式"><a href="#函数的多种参数使用形式" class="headerlink" title="函数的多种参数使用形式"></a>函数的多种参数使用形式</h3><ul><li>位置参数</li><li>关键字参数：通过“键=值”的形式传递参数。如果有位置参数，其必须在<strong>关键字参数前</strong>。</li><li>缺省参数：即默认参数。所有<strong>位置参数必须在默认参数前</strong>，包括定义和调用。</li><li>不定长参数：<ul><li>位置传递</li><li>关键字传递<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">print_info</span>(<span class="hljs-params">*args, **kwargs</span>):<br><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"不定长位置参数："</span>)<br><br> <span class="hljs-keyword">for</span> arg <span class="hljs-keyword">in</span> args:<br><br> <span class="hljs-built_in">print</span>(arg)<br><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"不定长关键字参数："</span>)<br><br> <span class="hljs-keyword">for</span> key, value <span class="hljs-keyword">in</span> kwargs.items():<br><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">f"<span class="hljs-subst">{key}</span>: <span class="hljs-subst">{value}</span>"</span>)<br><br> <br><br><span class="hljs-comment"># 调用函数，传递不定长位置参数和关键字参数</span><br><br>print_info(<span class="hljs-number">1</span>, <span class="hljs-number">2</span>, <span class="hljs-number">3</span>, name=<span class="hljs-string">"Alice"</span>, age=<span class="hljs-number">25</span>)<br></code></pre></td></tr></table></figure>在上述代码中：</li></ul></li><li><code>*args</code>用于接收不定长的位置参数，它会将传入的多个位置参数打包成一个元组。在<code>print_info</code>函数中，通过<code>for</code>循环遍历<code>args</code>元组来打印每个位置参数。</li><li><code>**kwargs</code>用于接收不定长的关键字参数，它会将传入的多个关键字参数打包成一个字典。在<code>print_info</code>函数中，通过<code>for</code>循环遍历<code>kwargs</code>字典的<code>items()</code>来打印每个关键字参数及其对应的值。</li></ul><h3 id="lambda匿名函数"><a href="#lambda匿名函数" class="headerlink" title="lambda匿名函数"></a>lambda匿名函数</h3><ol><li>作为简单函数的快捷方式</li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs python"><br><span class="hljs-comment"># 普通函数</span><br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">add</span>(<span class="hljs-params">x, y</span>):<br><br> <span class="hljs-keyword">return</span> x + y<br><br> <br><br><span class="hljs-comment"># lambda匿名函数</span><br><br>add_lambda = <span class="hljs-keyword">lambda</span> x, y: x + y<br><br> <br><br><span class="hljs-built_in">print</span>(add(<span class="hljs-number">3</span>, <span class="hljs-number">5</span>)) <br><br><span class="hljs-built_in">print</span>(add_lambda(<span class="hljs-number">3</span>, <span class="hljs-number">5</span>)) <br><br></code></pre></td></tr></table></figure><p>在这个例子中，<code>lambda x, y: x + y</code>创建了一个与<code>add</code>函数功能相同的匿名函数，并将其赋值给<code>add_lambda</code>变量。这样可以在需要简单函数的地方快速定义函数，而不需要使用<code>def</code>关键字定义一个完整的函数。</p><ol start="2"><li>与内置函数结合使用</li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs python"><br>numbers = [<span class="hljs-number">1</span>, <span class="hljs-number">2</span>, <span class="hljs-number">3</span>, <span class="hljs-number">4</span>, <span class="hljs-number">5</span>]<br><br><span class="hljs-comment"># 使用lambda函数和map函数将列表中的每个元素平方</span><br><br>squared_numbers = <span class="hljs-built_in">list</span>(<span class="hljs-built_in">map</span>(<span class="hljs-keyword">lambda</span> x: x ** <span class="hljs-number">2</span>, numbers))<br><br><span class="hljs-built_in">print</span>(squared_numbers) <br><br></code></pre></td></tr></table></figure><p>这里<code>map</code>函数需要一个函数作为参数，<code>lambda x: x ** 2</code>定义了一个匿名函数，用于计算每个元素的平方，然后<code>map</code>函数将这个匿名函数应用到<code>numbers</code>列表的每个元素上，最后将结果转换为列表并打印。</p><ol start="3"><li>作为排序的键函数</li></ol><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><code class="hljs python"><br>students = [<br><br> {<span class="hljs-string">'name'</span>: <span class="hljs-string">'Alice'</span>, <span class="hljs-string">'age'</span>: <span class="hljs-number">20</span>},<br><br> {<span class="hljs-string">'name'</span>: <span class="hljs-string">'Bob'</span>, <span class="hljs-string">'age'</span>: <span class="hljs-number">22</span>},<br><br> {<span class="hljs-string">'name'</span>: <span class="hljs-string">'Charlie'</span>, <span class="hljs-string">'age'</span>: <span class="hljs-number">19</span>}<br><br>]<br><br><span class="hljs-comment"># 按照学生的年龄排序</span><br><br>sorted_students = <span class="hljs-built_in">sorted</span>(students, key=<span class="hljs-keyword">lambda</span> x: x[<span class="hljs-string">'age'</span>])<br><br><span class="hljs-built_in">print</span>(sorted_students) <br><br></code></pre></td></tr></table></figure><p>在对<code>students</code>列表进行排序时，<code>sorted</code>函数的<code>key</code>参数接受一个函数，用于指定排序的依据。<code>lambda x: x['age']</code>定义了一个匿名函数，返回每个学生字典中的<code>'age'</code>值，<code>sorted</code>函数根据这个值对学生列表进行排序。</p><h3 id="mode三种基础访问模式"><a href="#mode三种基础访问模式" class="headerlink" title="mode三种基础访问模式"></a>mode三种基础访问模式</h3><ul><li>r：只读，指针放开头，默认模式。 </li><li>w：只写，文件存则删原内容从头编，不存在则建新。 </li><li>a：追加，文件存则新内容接后，不存在则建新。<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 打开文件获得文件对象</span><br><br>file_object = <span class="hljs-built_in">open</span>(<span class="hljs-string">'example.txt'</span>, <span class="hljs-string">'r'</span>, encoding=<span class="hljs-string">'utf-8'</span>) <br><br><span class="hljs-comment"># 读取指定长度字节（这里假设读取10个字节）</span><br><br>content = file_object.read(<span class="hljs-number">10</span>) <br><br><span class="hljs-built_in">print</span>(content) <br><br><span class="hljs-comment"># 读取一行</span><br><br>line = file_object.readline() <br><br><span class="hljs-built_in">print</span>(line) <br><br><span class="hljs-comment"># 读取全部行，得到列表</span><br><br>lines = file_object.readlines() <br><br><span class="hljs-built_in">print</span>(lines) <br><br><span class="hljs-comment"># for循环文件行，一次循环得到一行数据</span><br><br><span class="hljs-keyword">for</span> line <span class="hljs-keyword">in</span> file_object: <br><br> <span class="hljs-built_in">print</span>(line) <br><br><span class="hljs-comment"># 关闭文件对象</span><br><br>file_object.close() <br><br> <br><br><span class="hljs-comment"># 通过with open语法打开文件，可以自动关闭</span><br><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'example.txt'</span>, <span class="hljs-string">'r'</span>, encoding=<span class="hljs-string">'utf-8'</span>) <span class="hljs-keyword">as</span> f: <br><br> content = f.read() <br><br> <span class="hljs-built_in">print</span>(content)<br></code></pre></td></tr></table></figure></li><li>不管是read还是readline(s)都是使用的<strong>指针</strong>，因此如果其前面有read相关命令，后面会接着其读。</li><li>如果使用<code>read()</code>不加<code>num</code>参数则会<strong>读取全文。</strong></li><li>如果不关闭文件则程序会占用文件。</li></ul><h3 id="文件的写出、追加"><a href="#文件的写出、追加" class="headerlink" title="文件的写出、追加"></a>文件的写出、追加</h3><p><code>write()</code>写入到内存，<code>flush()</code>在打开文件的状态下从内存写入硬盘，如果不使用<code>flush()</code>，<code>close()</code>也会在文件关闭时写入硬盘。</p>]]></content>
<categories>
<category>python</category>
</categories>
</entry>
<entry>
<title>黑马python五、六章</title>
<link href="/2025/01/19/%E9%BB%91%E9%A9%ACpython%E4%BA%94%E3%80%81%E5%85%AD%E7%AB%A0/"/>
<url>/2025/01/19/%E9%BB%91%E9%A9%ACpython%E4%BA%94%E3%80%81%E5%85%AD%E7%AB%A0/</url>
<content type="html"><![CDATA[<h1 id="黑马python五、六章"><a href="#黑马python五、六章" class="headerlink" title="黑马python五、六章"></a>黑马python五、六章</h1><h3 id="函数"><a href="#函数" class="headerlink" title="函数"></a>函数</h3><ul><li><strong>定义</strong>：<code>def</code>函数名(参数1, 参数2)：<br> 函数体<br> <code>return</code>返回值（可省） </li><li><strong>使用</strong>：先定义，后调用 </li><li><strong>注意</strong>：参数、返回值皆可省<br>函数体在遇到return后就结束了，所以写在return后的代码不会执行。</li></ul><h3 id="None"><a href="#None" class="headerlink" title="None"></a>None</h3><ul><li><strong>定义</strong>：’NoneType’字面量，表空、无意义 </li><li><strong>函数返回</strong>：无return或return None </li><li><strong>场景</strong>：函数返回、if判断、变量定义</li></ul><h3 id="函数说明文档"><a href="#函数说明文档" class="headerlink" title="函数说明文档"></a>函数说明文档</h3><ul><li><strong>作用</strong>：对函数进行说明解释，帮助更好理解函数的功能。 </li><li><strong>定义语法</strong>： - 使用<code>def</code>定义函数，如<code>def func(x, y):</code>。 <ul><li>在函数名下方用三对双引号<code>"""</code>包裹函数说明。 </li><li>使用<code>:param</code>解释参数，如<code>:param x: 参数x的说明</code>，<code>:param y: 参数y的说明</code>。 </li><li>使用<code>:return</code>解释返回值，如<code>:return: 返回值的说明</code>。 </li><li>最后是函数体和<code>return</code>返回值。</li></ul></li></ul><h3 id="局部变量与全局变量"><a href="#局部变量与全局变量" class="headerlink" title="局部变量与全局变量"></a>局部变量与全局变量</h3><ul><li><strong>局部变量</strong>：作用范围在函数内部，函数外部无法使用。 </li><li><strong>全局变量</strong>：在函数内部和外部均可使用。 </li><li><strong>函数内变量声明为全局变量</strong>：使用<code>global</code>关键字，如<code>global 变量</code>。</li></ul><h3 id="列表"><a href="#列表" class="headerlink" title="列表"></a>列表</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 字面量定义列表 </span><br>my_list1 = [<span class="hljs-number">1</span>, <span class="hljs-string">'hello'</span>, <span class="hljs-number">3.14</span>, [<span class="hljs-number">4</span>, <span class="hljs-number">5</span>, <span class="hljs-number">6</span>]] <br><span class="hljs-comment"># 定义变量并赋值为列表 </span><br>my_list2 = [<span class="hljs-string">'apple'</span>, <span class="hljs-string">'banana'</span>, <span class="hljs-string">'orange'</span>] <br><span class="hljs-comment"># 定义空列表 </span><br>empty_list1 = [] <br>empty_list2 = <span class="hljs-built_in">list</span>() <br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs python">my_list = [<span class="hljs-string">'apple'</span>, <span class="hljs-string">'banana'</span>, <span class="hljs-string">'cherry'</span>, <span class="hljs-string">'date'</span>, <span class="hljs-string">'elderberry'</span>] <br>nested_list = [[<span class="hljs-string">'apple'</span>, <span class="hljs-string">'banana'</span>], [<span class="hljs-string">'cherry'</span>, <span class="hljs-string">'date'</span>], [<span class="hljs-string">'elderberry'</span>, <span class="hljs-string">'fig'</span>]] <br><span class="hljs-comment"># 从前向后通过下标索引取出元素 </span><br><span class="hljs-built_in">print</span>(my_list[<span class="hljs-number">0</span>]) <br><span class="hljs-built_in">print</span>(my_list[<span class="hljs-number">2</span>]) <br><span class="hljs-comment"># 从后向前通过下标索引取出元素 </span><br><span class="hljs-built_in">print</span>(my_list[-<span class="hljs-number">1</span>]) <br><span class="hljs-built_in">print</span>(my_list[-<span class="hljs-number">3</span>]) <br><br><span class="hljs-comment"># 注意下标索引的取值范围，超出范围会报错 </span><br><span class="hljs-comment"># 例如以下代码会报错，因为列表长度为5，最大下标为4 </span><br><span class="hljs-comment"># print(my_list[5]) </span><br><br><span class="hljs-comment"># 通过list[-1][0]访问嵌套列表中最后一个子列表的第一个元素 </span><br><span class="hljs-built_in">print</span>(nested_list[-<span class="hljs-number">1</span>][<span class="hljs-number">0</span>])<br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br></pre></td><td class="code"><pre><code class="hljs python">outer_list = [[<span class="hljs-number">1</span>, <span class="hljs-number">2</span>], [<span class="hljs-number">3</span>, <span class="hljs-number">4</span>]]<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"初始外层列表："</span>, outer_list)<br><br> <br><br><span class="hljs-comment"># append方法</span><br><br>outer_list.append([<span class="hljs-number">5</span>, <span class="hljs-number">6</span>])<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"外层append后："</span>, outer_list)<br><br> <br><br>outer_list[<span class="hljs-number">0</span>].append(<span class="hljs-number">7</span>)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"内层append后："</span>, outer_list)<br><br> <br><br>outer_list.extend([<span class="hljs-number">8</span>, <span class="hljs-number">9</span>])<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"外层extend后："</span>, outer_list)<br><br> <br><br>outer_list[<span class="hljs-number">1</span>].extend([<span class="hljs-number">10</span>, <span class="hljs-number">11</span>])<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"内层extend后："</span>, outer_list)<br><br> <br><br>outer_list.insert(-<span class="hljs-number">1</span>, [<span class="hljs-number">12</span>, <span class="hljs-number">13</span>])<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"外层insert后："</span>, outer_list)<br><br> <br><br>outer_list[<span class="hljs-number">2</span>].insert(<span class="hljs-number">0</span>, <span class="hljs-number">14</span>)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"内层insert后："</span>, outer_list)<br><br> <br><br><span class="hljs-keyword">del</span> outer_list[<span class="hljs-number">3</span>]<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"外层del后："</span>, outer_list)<br><br> <br><br><span class="hljs-keyword">del</span> outer_list[<span class="hljs-number">1</span>][<span class="hljs-number">0</span>]<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"内层del后："</span>, outer_list)<br><br> <br><br>popped_outer = outer_list.pop(<span class="hljs-number">2</span>)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"外层pop弹出的元素："</span>, popped_outer)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"外层pop后："</span>, outer_list)<br><br> <br><br>popped_inner = outer_list[<span class="hljs-number">0</span>].pop(<span class="hljs-number">1</span>)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"内层pop弹出的元素："</span>, popped_inner)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"内层pop后："</span>, outer_list)<br><br> <br><br><span class="hljs-comment"># 确保 [5, 6] 存在</span><br><br>outer_list.append([<span class="hljs-number">5</span>, <span class="hljs-number">6</span>])<br><br>outer_list.append([<span class="hljs-number">5</span>, <span class="hljs-number">6</span>])<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"外层列表中添加2个 [5, 6]："</span>, outer_list)<br><br> <br><br>outer_list.remove([<span class="hljs-number">5</span>, <span class="hljs-number">6</span>])<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"外层remove后："</span>, outer_list)<br><br> <br><br><span class="hljs-comment"># 确保 7 存在于内层列表</span><br><br>outer_list[<span class="hljs-number">0</span>].append(<span class="hljs-number">7</span>)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"内层列表中添加 7："</span>, outer_list)<br><br> <br><br>outer_list[<span class="hljs-number">0</span>].remove(<span class="hljs-number">7</span>)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"内层remove后："</span>, outer_list)<br><br> <br><br>outer_list.clear()<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"外层clear后："</span>, outer_list)<br><br> <br><br>outer_list = [[<span class="hljs-number">1</span>, <span class="hljs-number">2</span>], [<span class="hljs-number">3</span>, <span class="hljs-number">4</span>]]<br><br>outer_list[<span class="hljs-number">0</span>].clear()<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"内层clear后："</span>, outer_list)<br><br> <br><br>outer_list = [[<span class="hljs-number">1</span>, <span class="hljs-number">2</span>], [<span class="hljs-number">3</span>, <span class="hljs-number">4</span>], [<span class="hljs-number">1</span>, <span class="hljs-number">2</span>]]<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"外层count 2的次数："</span>, outer_list.count(<span class="hljs-number">2</span>))<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"外层count [1, 2]的次数："</span>, outer_list.count([<span class="hljs-number">1</span>, <span class="hljs-number">2</span>]))<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"内层count 1在[1, 2]中的次数："</span>, outer_list[<span class="hljs-number">0</span>].count(<span class="hljs-number">1</span>))<br><br> <br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"外层index [3, 4]的索引："</span>, outer_list.index([<span class="hljs-number">3</span>, <span class="hljs-number">4</span>]))<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"内层index 2在[1, 2]中的索引："</span>, outer_list[<span class="hljs-number">0</span>].index(<span class="hljs-number">2</span>))<br><br> <br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"外层len："</span>, <span class="hljs-built_in">len</span>(outer_list))<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"内层len [1, 2]："</span>, <span class="hljs-built_in">len</span>(outer_list[<span class="hljs-number">0</span>]))<br></code></pre></td></tr></table></figure><p><strong><code>insert</code>是在选中的元素前插入（替代输入索引的元素位置），不管是向前还是向后下标索引。</strong><br><strong><code>remove</code>只删除第一个匹配的元素。</strong></p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><code class="hljs plaintext">初始外层列表： [[1, 2], [3, 4]]<br>外层append后： [[1, 2], [3, 4], [5, 6]]<br>内层append后： [[1, 2, 7], [3, 4], [5, 6]]<br>外层extend后： [[1, 2, 7], [3, 4], [5, 6], 8, 9]<br>内层extend后： [[1, 2, 7], [3, 4, 10, 11], [5, 6], 8, 9]<br>外层insert后： [[1, 2, 7], [3, 4, 10, 11], [5, 6], 8, [12, 13], 9]<br>内层insert后： [[1, 2, 7], [3, 4, 10, 11], [14, 5, 6], 8, [12, 13], 9]<br>外层del后： [[1, 2, 7], [3, 4, 10, 11], [14, 5, 6], [12, 13], 9]<br>内层del后： [[1, 2, 7], [4, 10, 11], [14, 5, 6], [12, 13], 9]<br>外层pop弹出的元素： [14, 5, 6]<br>外层pop后： [[1, 2, 7], [4, 10, 11], [12, 13], 9]<br>内层pop弹出的元素： 2<br>内层pop后： [[1, 7], [4, 10, 11], [12, 13], 9]<br>外层列表中添加2个 [5, 6]： [[1, 7], [4, 10, 11], [12, 13], 9, [5, 6], [5, 6]]<br>外层remove后： [[1, 7], [4, 10, 11], [12, 13], 9, [5, 6]]<br>内层列表中添加 7： [[1, 7, 7], [4, 10, 11], [12, 13], 9, [5, 6]]<br>内层remove后： [[1, 7], [4, 10, 11], [12, 13], 9, [5, 6]]<br>外层clear后： []<br>内层clear后： [[], [3, 4]]<br>外层count 2的次数： 0<br>外层count [1, 2]的次数： 2<br>内层count 1在[1, 2]中的次数： 1<br>外层index [3, 4]的索引： 1<br>内层index 2在[1, 2]中的索引： 1<br>外层len： 3<br>内层len [1, 2]： 2<br></code></pre></td></tr></table></figure><p>列表具有以下特点： </p><ul><li>容量大：可以容纳多个元素，上限为2^63 - 1。 </li><li>元素类型多样：能够容纳不同类型的元素，可实现混装。 </li><li>有序存储：数据是有序存储的，每个元素都有下标序号。 </li><li>允许重复：允许重复数据存在。</li></ul><h3 id="元组"><a href="#元组" class="headerlink" title="元组"></a>元组</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 字面量定义元组</span><br>my_list1 = (<span class="hljs-number">1</span>, <span class="hljs-string">'hello'</span>, <span class="hljs-number">3.14</span>, [<span class="hljs-number">4</span>, <span class="hljs-number">5</span>, <span class="hljs-number">6</span>])<br><span class="hljs-comment"># 定义变量并赋值为元组 </span><br>my_list2 = ()<span class="hljs-string">'apple'</span>, <span class="hljs-string">'banana'</span>, <span class="hljs-string">'orange'</span>) <br><span class="hljs-comment"># 定义空元组</span><br>empty_list1 = ()<br>empty_list2 = <span class="hljs-built_in">tuple</span>() <br><span class="hljs-comment"># 定义1个元素的元组必须带有逗号，否则不是元组类型</span><br>t2 = (<span class="hljs-string">"hello"</span>, )<br></code></pre></td></tr></table></figure><p>也支持使用中括号下标索引。<br>相关操作仅支持index count len，元组中的列表可以修改。</p><h3 id="字符串"><a href="#字符串" class="headerlink" title="字符串"></a>字符串</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><code class="hljs python">text = <span class="hljs-string">" 1222hello world1111 "</span><br><br><span class="hljs-built_in">print</span>(text[<span class="hljs-number">0</span>])<br><br><span class="hljs-built_in">print</span>(text.index(<span class="hljs-string">"w"</span>))<br><br><span class="hljs-built_in">print</span>(text.index(<span class="hljs-string">"world"</span>))<br><br>text2 = text.replace(<span class="hljs-string">"world"</span>, <span class="hljs-string">"python"</span>)<br><br><span class="hljs-built_in">print</span>(text, text2)<br><br><span class="hljs-built_in">print</span>(text.split(<span class="hljs-string">" "</span>))<br><br><span class="hljs-built_in">print</span>(text.split())<br><br>text3 = text.strip()<br><br><span class="hljs-built_in">print</span>(text3)<br><br><span class="hljs-built_in">print</span>(text.strip(<span class="hljs-string">'12d'</span>))<br><br><span class="hljs-built_in">print</span>(text3.strip(<span class="hljs-string">'12d'</span>))<br><br><span class="hljs-built_in">print</span>(text.count(<span class="hljs-string">"l"</span>))<br><br><span class="hljs-built_in">print</span>(<span class="hljs-built_in">len</span>(text))<br></code></pre></td></tr></table></figure><p>字符串在Python中是<strong>不可变</strong>的。<strong>所有字符串操作方法</strong>(如strip()、replace()、upper()等)都会返回一个新字符串,而不是修改原字符串。如果要更新原变量,需要重新赋值。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs python">text = text.strip(<span class="hljs-string">'12'</span>) <span class="hljs-comment"># 重新赋值给text</span><br></code></pre></td></tr></table></figure><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs plaintext">14<br>14<br> 1222hello world1111 1222hello python1111<br>['', '', '1222hello', '', '', 'world1111', '', '']<br>['1222hello', 'world1111']<br>1222hello world1111<br> 1222hello world1111<br>hello worl<br>3<br>25<br></code></pre></td></tr></table></figure><ul><li><code>split(" ")</code>：严格按照空格分割，有几个空格就分几次，所以会产生多个空字符串</li><li><code>split()</code>：默认把连续空白字符当作一个分隔符处理</li><li><code>strip</code>同理，它的工作原理是：<ul><li>它把 ‘1’、’2’和’d’ 作为一个字符集</li><li>从两端开始，删除任何属于这个字符集的字符</li><li>只要遇到不在字符集中的字符就停止</li></ul></li></ul><h3 id="序列切片"><a href="#序列切片" class="headerlink" title="序列切片"></a>序列切片</h3><p>切片省略规则 ：</p><ul><li>列表、元组和字符串遵循起始位置、结束位置省略以及两者都省略的规则。</li><li>例如，对于<code>my_str = "01234567"</code>，<code>my_str[ :4]</code>的结果是<code>"0123"</code>，<code>my_str[1: ]</code>的结果是<code>"1234567"</code>，<code>my_str[ : ]</code>的结果是<code>"01234567"</code>。 步长省略代表步长为1。</li><li>此外，切片还可以指定步长。例如，<code>my_str[ : :2]</code>表示从字符串开头到结尾，步长为2进行切片，结果是<code>"0246"</code>。如果步长为负数，如<code>my_str[ : :-1]</code>，则表示从字符串结尾到开头进行切片，结果是<code>"76543210"</code>，注意此时起始下标和结束下标也要反向标。</li><li>反向标的意思是例如<code>print(my_str[3:1:-1])</code>和<code>print(my_str[-7:-5:-1])</code>都是可以的。<br><strong>上述针对容器的工具皆可以链式使用。</strong></li></ul><h3 id="集合"><a href="#集合" class="headerlink" title="集合"></a>集合</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 创建集合</span><br><br>my_set1 = {<span class="hljs-number">1</span>, <span class="hljs-number">2</span>, <span class="hljs-number">3</span>}<br><br>my_set2 = {<span class="hljs-number">3</span>, <span class="hljs-number">4</span>, <span class="hljs-number">5</span>}<br><br><span class="hljs-comment"># 创建空集合</span><br><span class="hljs-built_in">set</span>()<br><br><br><span class="hljs-comment"># add() 添加元素</span><br><br>my_set1.add(<span class="hljs-number">6</span>)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"添加元素后的my_set1:"</span>, my_set1) <br><br> <br><br><span class="hljs-comment"># remove() 移除元素</span><br><br>my_set1.remove(<span class="hljs-number">2</span>)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"移除元素后的my_set1:"</span>, my_set1) <br><br> <br><br><span class="hljs-comment"># pop() 随机取出一个元素</span><br><br>popped_element = my_set1.pop()<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"随机取出的元素:"</span>, popped_element) <br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"取出元素后的my_set1:"</span>, my_set1) <br><br> <br><br><span class="hljs-comment"># clear() 清空集合</span><br><br>my_set1.clear()<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"清空后的my_set1:"</span>, my_set1) <br><br> <br><br><span class="hljs-comment"># difference() 求差集</span><br><br>my_set1 = {<span class="hljs-number">1</span>, <span class="hljs-number">2</span>, <span class="hljs-number">3</span>}<br><br>my_set2 = {<span class="hljs-number">3</span>, <span class="hljs-number">4</span>, <span class="hljs-number">5</span>}<br><br>diff_set = my_set1.difference(my_set2)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"my_set1与my_set2的差集:"</span>, diff_set) <br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"my_set1:"</span>, my_set1) <br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"my_set2:"</span>, my_set2) <br><br> <br><br><span class="hljs-comment"># difference_update() 在集合1中删除集合2中存在的元素</span><br><br>my_set1 = {<span class="hljs-number">1</span>, <span class="hljs-number">2</span>, <span class="hljs-number">3</span>}<br><br>my_set2 = {<span class="hljs-number">3</span>, <span class="hljs-number">4</span>, <span class="hljs-number">5</span>}<br><br>my_set1.difference_update(my_set2)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"difference_update后的my_set1:"</span>, my_set1) <br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"my_set2:"</span>, my_set2) <br><br> <br><br><span class="hljs-comment"># union() 求并集</span><br><br>my_set1 = {<span class="hljs-number">1</span>, <span class="hljs-number">2</span>, <span class="hljs-number">3</span>}<br><br>my_set2 = {<span class="hljs-number">3</span>, <span class="hljs-number">4</span>, <span class="hljs-number">5</span>}<br><br>union_set = my_set1.union(my_set2)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"my_set1与my_set2的并集:"</span>, union_set) <br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"my_set1:"</span>, my_set1) <br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"my_set2:"</span>, my_set2) <br><br> <br><br><span class="hljs-comment"># len() 获取集合元素数量</span><br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"my_set1的元素数量:"</span>, <span class="hljs-built_in">len</span>(my_set1)) <br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"my_set2的元素数量:"</span>, <span class="hljs-built_in">len</span>(my_set2))<br></code></pre></td></tr></table></figure><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs plaintext">添加元素后的my_set1: {1, 2, 3, 6}<br>移除元素后的my_set1: {1, 3, 6}<br>随机取出的元素: 1<br>取出元素后的my_set1: {3, 6}<br>清空后的my_set1: set()<br>my_set1与my_set2的差集: {1, 2}<br>my_set1: {1, 2, 3}<br>my_set2: {3, 4, 5}<br>difference_update后的my_set1: {1, 2}<br>my_set2: {3, 4, 5}<br>my_set1与my_set2的并集: {1, 2, 3, 4, 5}<br>my_set1: {1, 2, 3}<br>my_set2: {3, 4, 5}<br>my_set1的元素数量: 3<br>my_set2的元素数量: 3<br></code></pre></td></tr></table></figure><ul><li>可以容纳不同类型的数据。</li><li>数据是无序存储的。</li><li>不允许重复数据存在。</li><li>支持for循环，不支持while循环。</li></ul><h3 id="字典"><a href="#字典" class="headerlink" title="字典"></a>字典</h3><p>字典或列表等数据容器中换行不影响。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br></pre></td><td class="code"><pre><code class="hljs python">{key: value, key: value, ......, key: value}<br><br><span class="hljs-comment">#定义空字典</span><br>{}<br><span class="hljs-built_in">dict</span>()<br><br><span class="hljs-comment"># 创建字典</span><br><br>my_dict = {<span class="hljs-string">"name"</span>: <span class="hljs-string">"Alice"</span>, <span class="hljs-string">"age"</span>: <span class="hljs-number">25</span>, <span class="hljs-string">"city"</span>: <span class="hljs-string">"New York"</span>}<br><br><span class="hljs-comment"># 获取指定Key对应的Value值</span><br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"name对应的Value:"</span>, my_dict[<span class="hljs-string">"name"</span>]) <br><br><span class="hljs-comment"># 添加或更新键值对（如果已经存在即修改）</span><br><br>my_dict[<span class="hljs-string">"gender"</span>] = <span class="hljs-string">"female"</span><br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"添加gender后的字典:"</span>, my_dict) <br><br><span class="hljs-comment"># 取出Key对应的Value并在字典内删除此Key的键值对</span><br><br>removed_value = my_dict.pop(<span class="hljs-string">"age"</span>)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"取出的age对应的值:"</span>, removed_value) <br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"删除age后的字典:"</span>, my_dict) <br><br><span class="hljs-comment"># 清空字典</span><br><br>my_dict.clear()<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"清空后的字典:"</span>, my_dict) <br><br><span class="hljs-comment"># 获取字典的全部Key，可用于for循环遍历字典</span><br><br>my_dict = {<span class="hljs-string">"name"</span>: <span class="hljs-string">"Alice"</span>, <span class="hljs-string">"age"</span>: <span class="hljs-number">25</span>, <span class="hljs-string">"city"</span>: <span class="hljs-string">"New York"</span>}<br><br>all_keys = my_dict.keys()<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"字典的全部Key:"</span>, <span class="hljs-built_in">list</span>(all_keys)) <br><br><span class="hljs-comment"># 计算字典内的元素数量</span><br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"字典的元素数量:"</span>, <span class="hljs-built_in">len</span>(my_dict))<br></code></pre></td></tr></table></figure><p>除了key不能为字典外，其余可以为一切内容；key不允许重复，重复添加覆盖原有数据。</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs plaintext">name对应的Value: Alice<br>添加gender后的字典: {'name': 'Alice', 'age': 25, 'city': 'New York', 'gender': 'female'}<br>取出的age对应的值: 25<br>删除age后的字典: {'name': 'Alice', 'city': 'New York', 'gender': 'female'}<br>清空后的字典: {}<br>字典的全部Key: ['name', 'age', 'city']<br>字典的元素数量: 3<br></code></pre></td></tr></table></figure><p>支持for循环，不支持while循环。</p><h3 id="字符串的通用操作"><a href="#字符串的通用操作" class="headerlink" title="字符串的通用操作"></a>字符串的通用操作</h3><p><img src="/img/learn/2025-01-18225311.jpg"><br><code>max()</code>最大元素，<code>min()</code>最小元素。<br>还可以通用类型转换：<code>list()</code>、<code>str()</code>、<code>tuple()</code>、<code>set()</code>。</p><ul><li>字符串转前两者会把每个字符提取出来，字典转前三两会抛弃value只保留key。</li><li>所有类型转字符串都不会丢失元素，只是在原来的形式下变成了字符串，读取时最外层会有引号。</li><li>字典转集合也是只保留key。<br><code>sorted(容器)</code>正向排序（从小到大），<code>sorted(容器, reverse=True)</code>反向排序（从大到小）。</li></ul>]]></content>
<categories>
<category>python</category>
</categories>
</entry>
<entry>
<title>黑马python三、四章</title>
<link href="/2025/01/16/%E9%BB%91%E9%A9%ACpython%E4%B8%89%E3%80%81%E5%9B%9B%E7%AB%A0/"/>
<url>/2025/01/16/%E9%BB%91%E9%A9%ACpython%E4%B8%89%E3%80%81%E5%9B%9B%E7%AB%A0/</url>
<content type="html"><![CDATA[<h1 id="黑马python三、四章"><a href="#黑马python三、四章" class="headerlink" title="黑马python三、四章"></a>黑马python三、四章</h1><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs python">num = <span class="hljs-number">10</span> <span class="hljs-comment"># 定义一个变量num，用于后续的条件判断 </span><br><span class="hljs-keyword">if</span> num > <span class="hljs-number">5</span>: <span class="hljs-comment"># 判断num是否大于5，这是要判断的条件，结果为布尔类型（True或False），这里num > 5为True </span><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"num大于5"</span>) <span class="hljs-comment"># 如果条件成立（num > 5为True），则执行打印操作，这是条件成立时要做的事情，注意这里的代码块相对于if语句缩进了4个空格</span><br></code></pre></td></tr></table></figure><p>if elif else要点： </p><ul><li>elif可多个 </li><li>判断互斥有序，前满足后不判 </li><li>条件判断可直写input省代码<br>print默认换行，加<code>end=''</code>可输出不换行，如<code>print("Hello", end='')</code>与<code>print("World", end='')</code>，结果<code>HelloWorld</code>同显一行，<code>end=''</code>乃传参用法。<br>字符串中<code>\t</code>同<code>tab</code>键，可对齐多行字符串。如<code>print("Hello World")</code>等用空格难对齐，<code>print("Hello\tWorld")</code>等用<code>\t</code>可对齐。<br><code>print()</code>空内容九四输出一个换行。</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs python">i = <span class="hljs-number">0</span> <br><span class="hljs-keyword">while</span> i < <span class="hljs-number">5</span>: <br><span class="hljs-built_in">print</span>(i) <br>i += <span class="hljs-number">1</span><br></code></pre></td></tr></table></figure><p><code>range(start, stop, step)</code>：生成从<code>start</code>开始，以<code>step</code>为步长，到<code>stop - 1</code>的整数序列（<code>step</code>可为正或负）。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">1</span>, <span class="hljs-number">10</span>, <span class="hljs-number">2</span>): <br><span class="hljs-built_in">print</span>(i)<br></code></pre></td></tr></table></figure><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs plaintext">1 <br>3<br>5 <br>7 <br>9<br></code></pre></td></tr></table></figure><p>continue临时中断，break永久中断</p>]]></content>
<categories>
<category>python</category>
</categories>
</entry>
<entry>
<title>黑马python一、二章</title>
<link href="/2025/01/15/%E9%BB%91%E9%A9%ACpython%E4%B8%80%E3%80%81%E4%BA%8C%E7%AB%A0/"/>
<url>/2025/01/15/%E9%BB%91%E9%A9%ACpython%E4%B8%80%E3%80%81%E4%BA%8C%E7%AB%A0/</url>
<content type="html"><![CDATA[<h1 id="黑马python一、二章"><a href="#黑马python一、二章" class="headerlink" title="黑马python一、二章"></a>黑马python一、二章</h1><p>刷数组-> 链表-> 哈希表->字符串->栈与队列->树->回溯->贪心->动态规划->图论->高级数据结构，再从简单刷起，做了几个类型题目之后，再慢慢做中等题目、困难题目。<br>写代码->翻译器->计算机，自然语言远比编程语言复杂<br>安装python就是安装python解释器，在windows上即是python.exe<br>字面量：<br><img src="/img/learn/20250114160606.png"><br># 注释<br>“”“ 注释 “””<br>print(变量1, 变量2)<br>type()不光能查看字面量数据的类型，还能查看变量内存储的数据类型。<strong>变量无类型，数据有类型。字符串变量表示变量存储了字符串而不是变量就是字符串。</strong><br>int() float() str() 任何类型都可以转换成字符串。<br>标识符（变量、类、方法的名字）命名规则：1、英文、数字（数字不可以用在开头）、下划线；2、大小写敏感。<br>//整除 %取余 **指数<br>定义字符串：单引号、双引号、三引号”””（内部支持换行操作，使用变量接收就是字符串，不接收就是注释）</p><h3 id="字符串的引号嵌套"><a href="#字符串的引号嵌套" class="headerlink" title="字符串的引号嵌套"></a>字符串的引号嵌套</h3><ul><li>单引号定义法，可内含双引号，如<code>'He said, "Hello!"'</code>。 </li><li>双引号定义法，可内含单引号，如<code>"I'm a student."</code>。 </li><li>可用转移字符（\）解除引号效用，如<code>'I don\'t know.'</code>。<br>print(a + b)<strong>字符串拼接</strong>（无法和非字符串类拼接）<br><strong>字符串占位</strong> “%占位符1 %占位符2” % (变量1, 变量2)<br>%s 字符串 %d 整数 %f 浮点数</li></ul><h3 id="字符串格式化"><a href="#字符串格式化" class="headerlink" title="字符串格式化"></a>字符串格式化</h3><ul><li>数字精度控制 - 用”m.n”控制数据宽精：m控宽（少用，宽小于数本身宽不生效），.n控小数点精度（四舍五入）。 </li><li>例：%5d将整数宽控5位，%5.2f宽控5且精度2（小数算宽，如11.345设%7.2f为[空格][空格]11.35）。</li></ul><ol><li>可通过f”{变量} {变量}”方式快速格式化。 </li><li>此方式：</li></ol><ul><li>不理会类型。 </li><li>不做精度控制。 </li><li>适合对精度无要求时快速使用。</li></ul><ol><li>表达式：有明确结果的代码语句，如1 + 1等，变量定义时等号右侧为表达式，结果赋给左侧变量。 </li><li>格式化：</li></ol><ul><li>f”{表达式}”。 </li><li>“%s%d%f” % (表达式、表达式、表达式)。</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs python">name = <span class="hljs-string">"示例公司"</span> <br>stock_price = <span class="hljs-number">10.5</span> <br>stock_code = <span class="hljs-string">"12345"</span> <br>stock_price_daily_growth_factor = <span class="hljs-number">1.2</span> <br>growth_days = <span class="hljs-number">5</span> <br>final_price = stock_price * (stock_price_daily_growth_factor ** growth_days) <span class="hljs-built_in">print</span>(<span class="hljs-string">"公司：{}\n初始股价：{:.2f}\n股票代码：{}\n经过{}天增长后，股价达到：{:.2f}"</span>.<span class="hljs-built_in">format</span>(name, stock_price, stock_code, growth_days, final_price))<br></code></pre></td></tr></table></figure><figure class="highlight cpp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs cpp"><span class="hljs-meta">#<span class="hljs-keyword">include</span> <span class="hljs-string"><iostream></span> </span><br><span class="hljs-meta">#<span class="hljs-keyword">include</span> <span class="hljs-string"><iomanip></span> </span><br><span class="hljs-function"><span class="hljs-type">int</span> <span class="hljs-title">main</span><span class="hljs-params">()</span> </span>{ <br>std::string name = <span class="hljs-string">"示例公司"</span>; <br><span class="hljs-type">double</span> stock_price = <span class="hljs-number">10.5</span>; <br>std::string stock_code = <span class="hljs-string">"12345"</span>; <br><span class="hljs-type">double</span> stock_price_daily_growth_factor = <span class="hljs-number">1.2</span>; <br><span class="hljs-type">int</span> growth_days = <span class="hljs-number">5</span>; <br><span class="hljs-type">double</span> final_price = stock_price * <span class="hljs-built_in">pow</span>(stock_price_daily_growth_factor, growth_days); <br>std::cout << <span class="hljs-string">"公司："</span> << name << std::endl; <br>std::cout << <span class="hljs-string">"初始股价："</span> << std::fixed << std::<span class="hljs-built_in">setprecision</span>(<span class="hljs-number">2</span>) << stock_price << std::endl; <br>std::cout << <span class="hljs-string">"股票代码："</span> << stock_code << std::endl; <br>std::cout << <span class="hljs-string">"经过"</span> << growth_days << <span class="hljs-string">"天增长后，股价达到："</span> << std::fixed << std::<span class="hljs-built_in">setprecision</span>(<span class="hljs-number">2</span>) << final_price << std::endl; <br><span class="hljs-keyword">return</span> <span class="hljs-number">0</span>; <br>}<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>python</category>
</categories>
</entry>
<entry>
<title>Deep Learning Based Gesture Recognition on ESP32-S3: From Training to Deployment</title>
<link href="/2024/11/30/Understanding%20Technical%20Documentation_2024-11-30T03-36-57/"/>
<url>/2024/11/30/Understanding%20Technical%20Documentation_2024-11-30T03-36-57/</url>
<content type="html"><![CDATA[<hr><h2 id="title-“Deep-Learning-Based-Gesture-Recognition-on-ESP32-S3-From-Training-to-Deployment”date-2024-11-30showAuthor-falseauthors-gao-jiaxuantags-ESP32-S3-Deep-Learning-Gesture-Recognition-ESP-DL-Model-Quantization"><a href="#title-“Deep-Learning-Based-Gesture-Recognition-on-ESP32-S3-From-Training-to-Deployment”date-2024-11-30showAuthor-falseauthors-gao-jiaxuantags-ESP32-S3-Deep-Learning-Gesture-Recognition-ESP-DL-Model-Quantization" class="headerlink" title="title: “Deep Learning Based Gesture Recognition on ESP32-S3: From Training to Deployment”date: 2024-11-30showAuthor: falseauthors: - gao-jiaxuantags: - ESP32-S3 - Deep Learning - Gesture Recognition - ESP-DL - Model Quantization"></a>title: “Deep Learning Based Gesture Recognition on ESP32-S3: From Training to Deployment”<br>date: 2024-11-30<br>showAuthor: false<br>authors:<br> - gao-jiaxuan<br>tags:<br> - ESP32-S3<br> - Deep Learning<br> - Gesture Recognition<br> - ESP-DL<br> - Model Quantization</h2><p>Integrating deep learning capabilities into embedded systems has become a crucial aspect of modern IoT applications. Although powerful deep learning models can achieve high recognition accuracy, deploying these models on resource-constrained devices poses considerable challenges. This article presents a gesture recognition system based on the ESP32-S3, detailing the entire workflow from model training to deployment on embedded hardware. The complete project implementation and code are available at <a href="https://github.com/BlakeHansen130/gesture-recognition-model">gesture-recognition-model</a>. By utilizing ESP-DL(master branch) and incorporating efficient quantization strategies with ESP-PPQ, this study demonstrates the feasibility of achieving real-time gesture recognition on resource-limited devices while maintaining satisfactory accuracy. Additionally, insights and methodologies were inspired by the work described in <a href="https://developer.espressif.com/blog/hand-gesture-recognition-on-esp32-s3-with-esp-deep-learning/">Espressif’s blog on hand gesture recognition</a>, which significantly influenced the approach taken in this article.</p><p><strong><em>This article provides an overview of the complete development process for a gesture recognition system, encompassing dataset preparation, model training, and deployment.</em></strong></p><p><em>The content is organized into five main sections. The first section, “System Design,” outlines the overall system architecture and development environment. The second section, “Model Development,” addresses the design of the network architecture and the training strategy. The third section discusses techniques for “Quantization Optimization.” The fourth section focuses on “Resource-Constrained Deployment,” and the final section presents a comprehensive analysis of experimental results.</em></p><h2 id="System-Design"><a href="#System-Design" class="headerlink" title="System Design"></a>System Design</h2><p>The gesture recognition system is built upon the ESP32-S3 platform, leveraging its computational capabilities and memory resources for deep learning inference. The system architecture encompasses both hardware and software components, carefully designed to achieve optimal performance within the constraints of embedded deployment.</p><h3 id="Development-Environment"><a href="#Development-Environment" class="headerlink" title="Development Environment"></a>Development Environment</h3><p>The development process requires two distinct Conda environments to handle different stages of the workflow. The primary training environment, designated as ‘dl_env’, manages dataset preprocessing, model training, and basic evaluation tasks. A separate quantization environment, ‘esp-dl’, is specifically configured for model quantization, accuracy assessment, and ESP-DL format conversion.</p><p>For the deployment phase, ESP-IDF version 5.x is used, with specific testing conducted on v5.3.1. The implementation relies on the master branch of ESP-DL, which can be obtained through:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs bash">git <span class="hljs-built_in">clone</span> -b v5.3.1 https://github.com/espressif/esp-idf.git<br>git <span class="hljs-built_in">clone</span> https://github.com/espressif/esp-dl.git<br></code></pre></td></tr></table></figure><p>It is important to note that using released versions of ESP-DL (such as idfv4.4 or v1.1) may not yield the same quantization performance as the latest master branch. The system configuration requires careful memory management, particularly in handling the ESP32-S3’s PSRAM. This is configured through the ESP-IDF menuconfig system with specific attention to flash size and SPIRAM settings.</p><h3 id="Memory-Management"><a href="#Memory-Management" class="headerlink" title="Memory Management"></a>Memory Management</h3><p>The system implements a sophisticated memory management strategy to optimize resource utilization. The CMake configuration is structured to properly integrate the ESP-DL library:</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs cmake"><span class="hljs-keyword">cmake_minimum_required</span>(VERSION <span class="hljs-number">3.5</span>)<br><br><span class="hljs-keyword">set</span>(EXTRA_COMPONENT_DIRS<br><span class="hljs-string">"$ENV{HOME}/esp/esp-dl/esp-dl"</span><br>)<br><br><span class="hljs-keyword">include</span>($ENV{IDF_PATH}/tools/cmake/<span class="hljs-keyword">project</span>.cmake)<br><span class="hljs-keyword">project</span>(gesture_recognition)<br></code></pre></td></tr></table></figure><p>__Note: Ensure that CMake can locate the esp-dl library cloned from GitHub by using relative paths to reference the esp-dl directory.</p><p>Component registration is handled through a dedicated CMakeLists.txt configuration:</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs cmake">idf_component_register(<br>SRCS<br><span class="hljs-string">"app_main.cpp"</span><br>INCLUDE_DIRS<br><span class="hljs-string">"."</span><br><span class="hljs-string">"model"</span><br>REQUIRES<br>esp-dl<br>)<br></code></pre></td></tr></table></figure><p>The memory architecture prioritizes efficient PSRAM utilization, with model weights and input/output tensors strategically allocated to optimize performance. Runtime monitoring of memory usage is implemented through built-in ESP-IDF functions:</p><figure class="highlight cpp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs cpp"><span class="hljs-type">size_t</span> free_mem = <span class="hljs-built_in">heap_caps_get_free_size</span>(MALLOC_CAP_SPIRAM);<br><span class="hljs-built_in">ESP_LOGI</span>(TAG, <span class="hljs-string">"Available PSRAM: %u bytes"</span>, free_mem);<br></code></pre></td></tr></table></figure><p>This comprehensive system design forms the foundation for subsequent model development and deployment stages, ensuring robust performance within the constraints of embedded hardware. The careful consideration of memory management and environment configuration is crucial for successful deep learning deployment on resource-constrained devices.</p><h2 id="Model-Development"><a href="#Model-Development" class="headerlink" title="Model Development"></a>Model Development</h2><p>The gesture recognition model employs a lightweight architecture optimized for embedded deployment while maintaining high accuracy. Based on MobileNetV2’s inverted residual blocks, the network architecture, termed LightGestureNet, is specifically designed for efficient execution on the ESP32-S3 platform.</p><p>The model architecture processes grayscale images of size 96x96 pixels and classifies them into eight distinct gesture classes. The network structure begins with an initial convolutional layer, followed by a series of inverted residual blocks, and concludes with a classifier layer. The implementation details are illustrated in the following code:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs python">first_layer = Conv2d(<span class="hljs-number">1</span>, <span class="hljs-number">16</span>, <span class="hljs-number">3</span>, stride=<span class="hljs-number">2</span>)<br>inverted_residual_blocks = [<br> (<span class="hljs-number">16</span>, <span class="hljs-number">24</span>, stride=<span class="hljs-number">2</span>, expand_ratio=<span class="hljs-number">6</span>),<br> (<span class="hljs-number">24</span>, <span class="hljs-number">24</span>, stride=<span class="hljs-number">1</span>, expand_ratio=<span class="hljs-number">6</span>),<br> (<span class="hljs-number">24</span>, <span class="hljs-number">32</span>, stride=<span class="hljs-number">2</span>, expand_ratio=<span class="hljs-number">6</span>),<br> (<span class="hljs-number">32</span>, <span class="hljs-number">32</span>, stride=<span class="hljs-number">1</span>, expand_ratio=<span class="hljs-number">6</span>)<br>]<br>classifier = Linear(<span class="hljs-number">32</span>, num_classes=<span class="hljs-number">8</span>)<br></code></pre></td></tr></table></figure><p>The initial convolutional layer processes single-channel grayscale input with 16 output channels and a stride of 2. The inverted residual blocks follow a systematic pattern of channel expansion and compression, with carefully chosen stride values to control spatial dimension reduction. The expand ratio of 6 in each block provides a balance between model capacity and computational efficiency.</p><p>The training process incorporates comprehensive data preprocessing and augmentation strategies. Input images undergo several transformations including resizing to 96x96 pixels, grayscale conversion, and normalization to the [0,1] range. Data augmentation techniques enhance model robustness through random rotation, scaling, and translation operations.</p><p>The training configuration utilizes the Adam optimizer with an initial learning rate of 0.001, implementing cosine annealing for learning rate decay. The following code demonstrates the model loading and inference process:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> torch<br><span class="hljs-keyword">from</span> model <span class="hljs-keyword">import</span> LightGestureNet<br><br>model = LightGestureNet()<br>model.load_state_dict(torch.load(<span class="hljs-string">'gesture_model.pth'</span>))<br>model.<span class="hljs-built_in">eval</span>()<br><br>input_tensor = torch.randn(<span class="hljs-number">1</span>, <span class="hljs-number">1</span>, <span class="hljs-number">96</span>, <span class="hljs-number">96</span>)<br>output = model(input_tensor)<br></code></pre></td></tr></table></figure><p>For deployment flexibility, the model supports multiple export formats. The ONNX format enables cross-platform inference capabilities, as demonstrated in the following implementation:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> onnxruntime<br><br>session = onnxruntime.InferenceSession(<span class="hljs-string">'gesture_model.onnx'</span>)<br><br>input_name = session.get_inputs()[<span class="hljs-number">0</span>].name<br>output = session.run(<span class="hljs-literal">None</span>, {input_name: input_array})<br></code></pre></td></tr></table></figure><p>The ONNX runtime session initialization creates an optimized execution environment for the model. The inference process requires proper input name extraction and tensor formatting to ensure correct model execution.</p><p>The gesture classification system encompasses eight distinct classes, mapped through a standardized dictionary:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs python">CLASS_NAMES = {<br> <span class="hljs-number">0</span>: <span class="hljs-string">'palm'</span>,<br> <span class="hljs-number">1</span>: <span class="hljs-string">'l'</span>,<br> <span class="hljs-number">2</span>: <span class="hljs-string">'fist'</span>,<br> <span class="hljs-number">3</span>: <span class="hljs-string">'thumb'</span>,<br> <span class="hljs-number">4</span>: <span class="hljs-string">'index'</span>,<br> <span class="hljs-number">5</span>: <span class="hljs-string">'ok'</span>,<br> <span class="hljs-number">6</span>: <span class="hljs-string">'c'</span>,<br> <span class="hljs-number">7</span>: <span class="hljs-string">'down'</span><br>}<br></code></pre></td></tr></table></figure><p>This class mapping ensures consistent interpretation of model outputs across different deployment scenarios.</p><p>The model development phase establishes a robust foundation for subsequent quantization and deployment stages, achieving an optimal balance between recognition accuracy and computational efficiency within the constraints of embedded systems.</p><h2 id="Quantization-Optimization"><a href="#Quantization-Optimization" class="headerlink" title="Quantization Optimization"></a>Quantization Optimization</h2><p>Model quantization serves as a critical bridge between high-precision deep learning models and resource-constrained embedded systems. For the ESP32-S3 platform, three distinct quantization strategies have been developed and implemented, each offering unique advantages for different deployment scenarios.</p><h3 id="INT8-Quantization-Implementation"><a href="#INT8-Quantization-Implementation" class="headerlink" title="INT8 Quantization Implementation"></a>INT8 Quantization Implementation</h3><p>The baseline quantization method implements uniform 8-bit quantization across all layers. This fundamental approach provides a solid starting point for model compression:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> onnxruntime<br><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np<br><span class="hljs-keyword">from</span> ppq <span class="hljs-keyword">import</span> *<br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">quantize_int8</span>(<span class="hljs-params">model</span>):<br><br> <span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'cal.pkl'</span>, <span class="hljs-string">'rb'</span>) <span class="hljs-keyword">as</span> f:<br> X_cal, _ = pickle.load(f)<br> <br> settings = QuantizationSettingFactory.default_setting()<br> settings.quantize_parameter_setting.bit_width = <span class="hljs-number">8</span><br> settings.quantize_parameter_setting.symmetrical = <span class="hljs-literal">True</span><br> settings.quantize_parameter_setting.per_channel = <span class="hljs-literal">True</span><br><br> quantum = quantize_torch_model(<br> model=model,<br> calib_dataloader=X_cal,<br> setting=settings,<br> platform=TargetPlatform.PPL_CUDA_INT8<br> )<br> <br> <span class="hljs-keyword">return</span> quantum<br></code></pre></td></tr></table></figure><p>In this implementation, the calibration data plays a crucial role in determining optimal quantization parameters. The <code>bit_width</code> parameter sets the quantization precision to 8 bits, while <code>symmetrical=True</code> ensures the quantization scheme maintains zero at the center of the range. The <code>per_channel=True</code> setting enables more fine-grained quantization by treating each channel independently, which helps preserve model accuracy.</p><h3 id="Mixed-Precision-Strategy"><a href="#Mixed-Precision-Strategy" class="headerlink" title="Mixed Precision Strategy"></a>Mixed Precision Strategy</h3><p>The mixed precision approach provides more flexibility by allowing different quantization precisions for different layers based on their sensitivity to quantization:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">mixed_precision_quantize</span>(<span class="hljs-params">model</span>):<br><br> precision_config = {<br> <span class="hljs-string">'conv1'</span>: {<span class="hljs-string">'w_bits'</span>: <span class="hljs-number">8</span>, <span class="hljs-string">'a_bits'</span>: <span class="hljs-number">8</span>},<br> <span class="hljs-string">'conv2'</span>: {<span class="hljs-string">'w_bits'</span>: <span class="hljs-number">16</span>, <span class="hljs-string">'a_bits'</span>: <span class="hljs-number">8</span>},<br> <span class="hljs-string">'fc'</span>: {<span class="hljs-string">'w_bits'</span>: <span class="hljs-number">16</span>, <span class="hljs-string">'a_bits'</span>: <span class="hljs-number">16</span>}<br> }<br> <br> settings = QuantizationSettingFactory.mixed_precision_setting()<br> settings.optimization_level = <span class="hljs-number">1</span><br> <br> <span class="hljs-keyword">for</span> layer, config <span class="hljs-keyword">in</span> precision_config.items():<br> settings.quantize_parameter_setting[layer] = LayerQuantSetting(<br> w_bits=config[<span class="hljs-string">'w_bits'</span>],<br> a_bits=config[<span class="hljs-string">'a_bits'</span>]<br> )<br> <br> quantum = quantize_torch_model(<br> model=model,<br> calib_dataloader=X_cal,<br> setting=settings,<br> platform=TargetPlatform.PPL_CUDA_MIX<br> )<br> <br> <span class="hljs-keyword">return</span> quantum<br></code></pre></td></tr></table></figure><p>This strategy allows precise control over the quantization of each layer. The <code>precision_config</code> dictionary defines both weight (<code>w_bits</code>) and activation (<code>a_bits</code>) precision for each layer. Early layers and critical feature extraction components often benefit from higher precision, while later layers can typically tolerate more aggressive quantization. The optimization level parameter controls how aggressively the quantization algorithm searches for optimal bit assignments.</p><h3 id="Equalization-Aware-Quantization"><a href="#Equalization-Aware-Quantization" class="headerlink" title="Equalization-Aware Quantization"></a>Equalization-Aware Quantization</h3><p>The equalization-aware approach introduces an advanced quantization method that considers the distribution of values across layers:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">equalization_quantize</span>(<span class="hljs-params">model</span>):<br> settings = QuantizationSettingFactory.default_setting()<br> settings.equalization = <span class="hljs-literal">True</span><br> settings.equalization_setting.iterations = <span class="hljs-number">4</span><br> settings.equalization_setting.value_threshold = <span class="hljs-number">0.4</span><br> settings.optimization_level = <span class="hljs-number">2</span><br> <br> model = convert_relu6_to_relu(model)<br><br> quantum = quantize_torch_model(<br> model=model,<br> calib_dataloader=X_cal,<br> setting=settings,<br> platform=TargetPlatform.PPL_CUDA_INT8<br> )<br> <br> <span class="hljs-keyword">return</span> quantum<br></code></pre></td></tr></table></figure><p>The equalization process iteratively adjusts scaling factors across layers to minimize quantization error. The <code>iterations</code> parameter controls how many times the equalization algorithm runs, while <code>value_threshold</code> determines when the algorithm considers values significant enough to influence scaling decisions. The conversion from ReLU6 to ReLU is necessary for compatibility with the equalization process, as the capped activation function can interfere with proper scale determination.</p><h3 id="Performance-Evaluation"><a href="#Performance-Evaluation" class="headerlink" title="Performance Evaluation"></a>Performance Evaluation</h3><p>To assess the effectiveness of each quantization strategy, a comprehensive evaluation framework has been implemented:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">evaluate_quantized_model</span>(<span class="hljs-params">model, test_data</span>):<br> metrics = {}<br><br> metrics[<span class="hljs-string">'size'</span>] = get_model_size(model)<br><br> predictions = model.predict(test_data)<br> metrics[<span class="hljs-string">'accuracy'</span>] = calculate_accuracy(predictions, test_labels)<br><br> metrics[<span class="hljs-string">'latency'</span>] = measure_inference_time(model, test_data[<span class="hljs-number">0</span>])<br> <br> <span class="hljs-keyword">return</span> metrics<br></code></pre></td></tr></table></figure><p>This evaluation framework examines three critical aspects of each quantized model: memory footprint, accuracy retention, and inference latency. The modular design allows for easy comparison between different quantization strategies and helps in selecting the most appropriate approach for specific deployment requirements.</p><p>Each quantization method presents its own set of trade-offs, and the choice between them depends on specific application requirements such as memory constraints, accuracy requirements, and inference speed needs. The systematic implementation and evaluation of these strategies ensure optimal deployment on the ESP32-S3 platform.</p><h2 id="Resource-Constrained-Deployment"><a href="#Resource-Constrained-Deployment" class="headerlink" title="Resource-Constrained Deployment"></a>Resource-Constrained Deployment</h2><p>The deployment phase of the gesture recognition system on ESP32-S3 requires careful consideration of hardware constraints and system configurations. The implementation focuses on optimal memory utilization and efficient inference execution while maintaining system stability.</p><h3 id="Memory-and-Partition-Configuration"><a href="#Memory-and-Partition-Configuration" class="headerlink" title="Memory and Partition Configuration"></a>Memory and Partition Configuration</h3><p>The ESP32-S3’s memory architecture necessitates specific configurations through the ESP-IDF menuconfig system. The following memory settings are crucial for optimal model deployment:</p><figure class="highlight arduino"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs arduino"><span class="hljs-built_in">Serial</span> Flasher Configuration<br>└── Flash Size: <span class="hljs-number">8</span>MB<br>Component Configuration<br>└── ESP PSRAM<br> ├── Support <span class="hljs-keyword">for</span> external, <span class="hljs-built_in">SPI</span>-connected RAM<br> ├── <span class="hljs-built_in">SPI</span> RAM config<br> │ ├── SPIRAM_MODE: Octal<br> │ ├── SPIRAM_SPEED: <span class="hljs-number">40</span> MHz<br> │ └── Enable <span class="hljs-built_in">SPI</span> RAM during startup<br> └── Allow .bss Segment Placement in PSRAM: Enabled<br></code></pre></td></tr></table></figure><p>These configurations enable efficient utilization of external RAM for model weight storage and inference operations. The partition table configuration requires customization to accommodate the model data:</p><figure class="highlight mathematica"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs mathematica"><span class="hljs-built_in">Partition</span> <span class="hljs-built_in">Table</span><br>└── <span class="hljs-built_in">Partition</span> <span class="hljs-built_in">Table</span><span class="hljs-operator">:</span> <span class="hljs-variable">Custom</span> <span class="hljs-built_in">Partition</span> <span class="hljs-built_in">Table</span> <span class="hljs-variable">CSV</span><br>└── <span class="hljs-variable">Custom</span> <span class="hljs-built_in">Partition</span> <span class="hljs-variable">CSV</span> <span class="hljs-built_in">File</span><span class="hljs-operator">:</span> <span class="hljs-variable">partitions</span><span class="hljs-operator">.</span><span class="hljs-variable">csv</span><br></code></pre></td></tr></table></figure><p>The custom partition layout ensures adequate space allocation for both the application and model data, facilitating efficient model loading during runtime.</p><h3 id="Build-System-Integration"><a href="#Build-System-Integration" class="headerlink" title="Build System Integration"></a>Build System Integration</h3><p>The project’s CMake configuration integrates the ESP-DL framework and establishes proper component relationships. The build system setup requires careful attention to include paths and dependencies:</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs cmake">idf_component_register(<br> SRCS<br> <span class="hljs-string">"app_main.cpp"</span><br> INCLUDE_DIRS<br> <span class="hljs-string">"."</span><br> <span class="hljs-string">"model"</span><br> REQUIRES<br> esp-dl<br>)<br></code></pre></td></tr></table></figure><p>This configuration ensures proper compilation of the application components and correct linking with the ESP-DL framework. The integration of model files and headers follows a systematic approach through the build system.</p><h3 id="Runtime-Memory-Management"><a href="#Runtime-Memory-Management" class="headerlink" title="Runtime Memory Management"></a>Runtime Memory Management</h3><p>The implementation employs strategic memory allocation to optimize resource utilization during model inference. Memory monitoring capabilities are integrated into the system to ensure stable operation:</p><figure class="highlight cpp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs cpp"><span class="hljs-type">size_t</span> free_mem = <span class="hljs-built_in">heap_caps_get_free_size</span>(MALLOC_CAP_SPIRAM);<br><span class="hljs-built_in">ESP_LOGI</span>(TAG, <span class="hljs-string">"Available PSRAM: %u bytes"</span>, free_mem);<br></code></pre></td></tr></table></figure><p>Model weights are stored in PSRAM while keeping critical runtime buffers in internal RAM for optimal performance. The system implements memory monitoring to prevent allocation failures and maintain stable operation during continuous inference.</p><h3 id="Serial-Communication-Configuration"><a href="#Serial-Communication-Configuration" class="headerlink" title="Serial Communication Configuration"></a>Serial Communication Configuration</h3><p>The deployment setup includes specific serial communication parameters for reliable device interaction:</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs routeros">Component<span class="hljs-built_in"> config</span><br><span class="hljs-built_in"></span>└── ESP<span class="hljs-built_in"> System Settings</span><br><span class="hljs-built_in"></span> └── Channel <span class="hljs-keyword">for</span><span class="hljs-built_in"> console </span>output<br> └── Default: UART0<br><br>Component<span class="hljs-built_in"> config</span><br><span class="hljs-built_in"></span>└── Bluetooth<br> └── NimBLE Options<br> └── Host-controller Transport<br> └── <span class="hljs-built_in">Enable</span> Uart Transport<br> └── Uart Hci Baud Rate: 115200<br></code></pre></td></tr></table></figure><p>These settings ensure stable communication during both development and deployment phases. The baud rate selection balances reliable communication with flash programming speed.</p><h3 id="Driver-Installation-and-System-Integration"><a href="#Driver-Installation-and-System-Integration" class="headerlink" title="Driver Installation and System Integration"></a>Driver Installation and System Integration</h3><p>The deployment process requires proper USB driver installation for device communication. The CH340/CH341 driver installation process follows system-specific procedures, ensuring reliable device connectivity.</p><h2 id="Experimental-Results"><a href="#Experimental-Results" class="headerlink" title="Experimental Results"></a>Experimental Results</h2><p>The gesture recognition system on ESP32-S3 showed strong performance across multiple metrics. Both quantitative and qualitative evaluations were conducted.</p><p>The baseline model (prior to quantization) established high accuracy across eight gesture classes, particularly distinguishing distinct gestures like ‘palm’ and ‘fist’. This floating-point model served as a benchmark for subsequent optimizations.</p><p>After quantization, the mixed-precision approach balanced resource use and recognition accuracy effectively. INT8 quantization minimized model size, while selectively using 16-bit precision in key layers maintained feature discrimination. Equalization-aware quantization helped keep consistent performance across all gestures.</p><p>On the ESP32-S3, the optimized model achieved real-time inference with moderate power consumption and efficient memory use, leveraging both internal RAM and PSRAM effectively. These results confirm that the optimization strategies are suitable for deploying advanced gesture recognition on resource-constrained platforms, enabling practical real-time applications.</p><h2 id="References"><a href="#References" class="headerlink" title="References"></a>References</h2><ol><li>ESP official resources</li></ol><ul><li><a href="https://github.com/espressif/esp-dl">ESP-DL main repository and documents</a></li><li><a href="https://github.com/espressif/esp-idf">ESP-IDF development framework</a></li><li><a href="https://www.espressif.com/sites/default/files/documentation/esp32-s3_technical_reference_manual_en.pdf">ESP32-S3 Technical Reference Manual</a></li><li><a href="https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/">ESP-IDF Programming Guide</a></li></ul><ol start="2"><li>ESP-PPQ toolchain</li></ol><ul><li><a href="https://github.com/espressif/esp-ppq/">ESP-PPQ project</a> Contains:<ul><li>Core API</li><li>Quantizer documentation</li><li>Executor implementation</li><li>Usage guide</li></ul></li></ul><ol start="3"><li>Tutorials and guides</li></ol><ul><li><a href="https://github.com/alibukharai/Blogs/tree/main/ESP-DL">ESP-DL Complete Workflow</a></li><li><a href="https://github.com/espressif/esp-dl/blob/master/tutorial/how_to_load_model_cn.md">Model Loading Tutorial</a></li><li><a href="https://github.com/espressif/esp-dl/blob/master/tutorial/how_to_deploy_mobilenet_v2_cn.md">MobileNet V2 Deployment Guide</a></li><li><a href="https://github.com/espressif/esp-dl/blob/master/tutorial/how_to_quantize_model_cn.md">Model Quantization Guide</a></li></ul><ol start="4"><li>Datasets</li></ol><ul><li><a href="https://www.kaggle.com/datasets/gti-upm/leapgestrecog">LeapGestRecog Dataset</a></li></ul>]]></content>
<categories>
<category>fyp</category>
</categories>
</entry>
<entry>
<title>手势识别项目环境配置文档</title>
<link href="/2024/11/29/development-environment-docs/"/>
<url>/2024/11/29/development-environment-docs/</url>
<content type="html"><![CDATA[<h1 id="手势识别项目环境配置文档"><a href="#手势识别项目环境配置文档" class="headerlink" title="手势识别项目环境配置文档"></a>手势识别项目环境配置文档</h1><h2 id="项目环境架构"><a href="#项目环境架构" class="headerlink" title="项目环境架构"></a>项目环境架构</h2><p>本项目采用分阶段环境配置策略，针对不同的开发阶段使用独立的环境，确保各阶段工具链的独立性和稳定性。</p><h3 id="数据处理与训练环境-dl-env"><a href="#数据处理与训练环境-dl-env" class="headerlink" title="数据处理与训练环境 (dl_env)"></a>数据处理与训练环境 (dl_env)</h3><p>训练环境主要用于数据集处理、模型开发和基础训练工作，配置如下：<br><a href="https://github.com/BlakeHansen130/gesture-recognition-model/blob/main/environment/dl_env_environment.yml">dl_env</a><br>该环境的核心功能：</p><ol><li>数据集预处理与增强</li><li>模型设计与训练</li><li>性能评估与可视化</li><li>模型导出为ONNX格式</li></ol><h3 id="量化优化环境-esp-dl"><a href="#量化优化环境-esp-dl" class="headerlink" title="量化优化环境 (esp-dl)"></a>量化优化环境 (esp-dl)</h3><p>量化环境专门用于模型优化和量化，配置如下：<br><a href="https://github.com/BlakeHansen130/gesture-recognition-model/blob/main/environment/esp-dl_environment.yml">esp-dl</a><br>该环境主要用于：</p><ol><li>ONNX模型优化</li><li>模型量化实验</li><li>量化模型评估</li><li>生成ESP32部署文件</li></ol><h3 id="ESP32开发环境"><a href="#ESP32开发环境" class="headerlink" title="ESP32开发环境"></a>ESP32开发环境</h3><p>ESP32开发环境需要独立于Conda环境，使用ESP-IDF原生工具链：</p><ol><li><p>环境准备：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 退出当前conda环境</span><br>conda deactivate<br><br><span class="hljs-comment"># 激活ESP-IDF环境</span><br>. <span class="hljs-variable">$HOME</span>/esp/esp-idf/export.sh<br></code></pre></td></tr></table></figure></li><li><p>项目编译与烧录：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 编译项目</span><br>idf.py build<br><br><span class="hljs-comment"># 烧录程序</span><br>idf.py -p /dev/ttyUSB0 flash<br><br><span class="hljs-comment"># 监视输出</span><br>idf.py monitor<br></code></pre></td></tr></table></figure></li></ol><h2 id="环境配置流程"><a href="#环境配置流程" class="headerlink" title="环境配置流程"></a>环境配置流程</h2><h3 id="训练环境配置"><a href="#训练环境配置" class="headerlink" title="训练环境配置"></a>训练环境配置</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 创建训练环境</span><br>conda <span class="hljs-built_in">env</span> create -f dl_env_environment.yml<br><br><span class="hljs-comment"># 激活环境</span><br>conda activate dl_env<br><br><span class="hljs-comment"># 验证PyTorch安装</span><br>python -c <span class="hljs-string">"import torch; print(torch.__version__)"</span><br></code></pre></td></tr></table></figure><h3 id="量化环境配置"><a href="#量化环境配置" class="headerlink" title="量化环境配置"></a>量化环境配置</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 创建量化环境</span><br>conda <span class="hljs-built_in">env</span> create -f esp-dl_environment.yml<br><br><span class="hljs-comment"># 激活环境</span><br>conda activate esp-dl<br><br><span class="hljs-comment"># 验证ONNX Runtime安装</span><br>python -c <span class="hljs-string">"import onnxruntime as ort; print(ort.__version__)"</span><br></code></pre></td></tr></table></figure><h3 id="ESP-IDF环境配置"><a href="#ESP-IDF环境配置" class="headerlink" title="ESP-IDF环境配置"></a>ESP-IDF环境配置</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 安装依赖包</span><br><span class="hljs-built_in">sudo</span> apt-get install git wget flex bison gperf python3-pip python3-setuptools cmake ninja-build ccache libffi-dev libssl-dev dfu-util libusb-1.0-0<br><br><span class="hljs-comment"># 获取ESP-IDF</span><br>git <span class="hljs-built_in">clone</span> --recursive https://github.com/espressif/esp-idf.git<br><br><span class="hljs-comment"># 安装工具链</span><br><span class="hljs-built_in">cd</span> esp-idf<br>./install.sh esp32s3<br><br><span class="hljs-comment"># 配置环境变量</span><br>. ./export.sh<br></code></pre></td></tr></table></figure><h2 id="开发工作流程"><a href="#开发工作流程" class="headerlink" title="开发工作流程"></a>开发工作流程</h2><ol><li><p>训练阶段</p><ul><li>激活dl_env环境</li><li>完成数据处理与模型训练</li><li>将模型导出为ONNX格式</li></ul></li><li><p>量化阶段</p><ul><li>切换到esp-dl环境</li><li>执行模型量化与优化</li><li>生成量化后的模型文件</li></ul></li><li><p>部署阶段</p><ul><li>退出Conda环境</li><li>激活ESP-IDF环境</li><li>使用idf.py命令进行编译和烧录</li></ul></li></ol><h2 id="环境维护建议"><a href="#环境维护建议" class="headerlink" title="环境维护建议"></a>环境维护建议</h2><ol><li><p>依赖管理</p><ul><li>使用environment.yml维护环境依赖</li><li>定期更新关键包的安全补丁</li><li>保持工具链版本的稳定性</li></ul></li><li><p>环境隔离</p><ul><li>严格遵循环境分离原则</li><li>避免在不同环境间混用工具</li><li>及时清理不需要的包和缓存</li></ul></li><li><p>版本控制</p><ul><li>记录各环境的具体版本信息</li><li>确保团队成员使用相同的环境配置</li><li>将环境文件纳入版本控制系统</li></ul></li></ol>]]></content>
<categories>
<category>fyp</category>
</categories>
</entry>
<entry>
<title>ESP32手势识别部署实现文档</title>
<link href="/2024/11/29/esp32-deployment-docs/"/>
<url>/2024/11/29/esp32-deployment-docs/</url>
<content type="html"><![CDATA[<h1 id="ESP32手势识别部署实现文档"><a href="#ESP32手势识别部署实现文档" class="headerlink" title="ESP32手势识别部署实现文档"></a>ESP32手势识别部署实现文档</h1><h2 id="项目结构设计"><a href="#项目结构设计" class="headerlink" title="项目结构设计"></a>项目结构设计</h2><p>本项目实现了在ESP32上部署量化后的手势识别模型。项目采用ESP-IDF框架进行开发，整体架构包括模型加载、数据预处理、推理执行和结果处理等核心模块。</p><h3 id="分区表配置"><a href="#分区表配置" class="headerlink" title="分区表配置"></a>分区表配置</h3><p>系统分区表（partitions.csv）的设计充分考虑了模型部署的需求：</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs csv"># Name, Type, SubType, Offset, Size, Flags<br>nvs, data, nvs, 0x9000, 24K,<br>phy_init, data, phy, 0xf000, 4K,<br>factory, app, factory, 0x10000, 4000K,<br>model, data, spiffs, , 256K,<br></code></pre></td></tr></table></figure><p>特别注意以下几点：</p><ul><li>factory分区分配4MB，保证应用程序有足够空间</li><li>专门划分model分区（256KB）用于存储量化后的模型</li><li>nvs分区用于存储系统配置信息</li><li>phy_init分区用于无线校准数据</li></ul><h3 id="构建系统配置"><a href="#构建系统配置" class="headerlink" title="构建系统配置"></a>构建系统配置</h3><p>项目的CMakeLists.txt配置体现了模块化的设计思想：</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs cmake"><span class="hljs-keyword">set</span>(srcs app_main.cpp)<br><span class="hljs-keyword">set</span>(requires esp-dl)<br><br>idf_component_register(SRCS <span class="hljs-variable">${srcs}</span><br> REQUIRES <span class="hljs-variable">${requires}</span><br> INCLUDE_DIRS <span class="hljs-string">"../model"</span><br> WHOLE_ARCHIVE)<br></code></pre></td></tr></table></figure><p>构建系统的关键设计：</p><ol><li>使用esp-dl组件提供深度学习支持</li><li>将模型目录添加到包含路径</li><li>使用WHOLE_ARCHIVE确保所有符号都被链接</li></ol><h3 id="模型打包流程"><a href="#模型打包流程" class="headerlink" title="模型打包流程"></a>模型打包流程</h3><p>构建系统中包含了自动化的模型打包流程：</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs cmake"><span class="hljs-keyword">set</span>(MVMODEL_EXE <span class="hljs-variable">${PROJECT_DIR}</span>/pack_model.py)<br>idf_build_get_property(build_dir BUILD_DIR)<br><span class="hljs-keyword">set</span>(image_file <span class="hljs-variable">${build_dir}</span>/espdl_models/models.espdl)<br><br><span class="hljs-keyword">add_custom_command</span>(<br> OUTPUT <span class="hljs-variable">${image_file}</span><br> <span class="hljs-keyword">COMMAND</span> python <span class="hljs-variable">${MVMODEL_EXE}</span> --model_path=<span class="hljs-variable">${MODEL_FILE_PATH}</span> <br> --out_file=<span class="hljs-variable">${image_file}</span><br> DEPENDS <span class="hljs-variable">${MODEL_FILE_PATH}</span><br> VERBATIM)<br></code></pre></td></tr></table></figure><p>这个过程可以自动完成：</p><ul><li>模型文件的打包</li><li>分区大小的验证</li><li>模型的烧录准备</li></ul><h2 id="图像预处理实现"><a href="#图像预处理实现" class="headerlink" title="图像预处理实现"></a>图像预处理实现</h2><h3 id="预处理工具开发"><a href="#预处理工具开发" class="headerlink" title="预处理工具开发"></a>预处理工具开发</h3><p>为了简化部署过程，开发了专门的图像预处理工具：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">process_image</span>(<span class="hljs-params">image_path</span>):<br> img = cv2.imread(image_path)<br> gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)<br> resized = cv2.resize(gray, (<span class="hljs-number">96</span>, <span class="hljs-number">96</span>))<br> normalized = resized.astype(<span class="hljs-string">'float32'</span>) / <span class="hljs-number">255.0</span><br> quantized = (normalized * <span class="hljs-number">128</span>).astype(np.int8)<br> <span class="hljs-keyword">return</span> quantized<br></code></pre></td></tr></table></figure><p>预处理步骤包括：</p><ol><li>图像加载和灰度转换</li><li>尺寸调整到96x96</li><li>归一化处理</li><li>量化到int8范围</li></ol><h3 id="头文件生成"><a href="#头文件生成" class="headerlink" title="头文件生成"></a>头文件生成</h3><p>预处理数据通过生成C++头文件的方式集成到项目中：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">generate_header</span>(<span class="hljs-params">data, output_path</span>):<br> <span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(output_path, <span class="hljs-string">'w'</span>) <span class="hljs-keyword">as</span> f:<br> f.write(<span class="hljs-string">"#pragma once\n\n"</span>)<br> f.write(<span class="hljs-string">"const int8_t test_image_data[9216] = {\n "</span>)<br> data_str = [<span class="hljs-built_in">str</span>(x) <span class="hljs-keyword">for</span> x <span class="hljs-keyword">in</span> data.flatten()]<br> chunk_size = <span class="hljs-number">12</span><br> <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">0</span>, <span class="hljs-built_in">len</span>(data_str), chunk_size):<br> chunk = data_str[i:i + chunk_size]<br> f.write(<span class="hljs-string">", "</span>.join(chunk))<br> <span class="hljs-keyword">if</span> i + chunk_size < <span class="hljs-built_in">len</span>(data_str):<br> f.write(<span class="hljs-string">",\n "</span>)<br> f.write(<span class="hljs-string">"};\n"</span>)<br></code></pre></td></tr></table></figure><p>这种设计的优势：</p><ul><li>预处理数据直接编译到固件中</li><li>避免运行时的预处理开销</li><li>确保数据格式的一致性</li></ul><h2 id="推理实现"><a href="#推理实现" class="headerlink" title="推理实现"></a>推理实现</h2><h3 id="核心推理模块"><a href="#核心推理模块" class="headerlink" title="核心推理模块"></a>核心推理模块</h3><p>推理实现的核心代码展示了整个工作流程：</p><figure class="highlight cpp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs cpp"><span class="hljs-function"><span class="hljs-keyword">class</span> TensorBase* <span class="hljs-title">prepare_input_tensor</span><span class="hljs-params">()</span> </span>{<br> TensorBase* input_tensor = <span class="hljs-keyword">new</span> <span class="hljs-built_in">TensorBase</span>(<br> {<span class="hljs-number">1</span>, <span class="hljs-number">1</span>, <span class="hljs-number">96</span>, <span class="hljs-number">96</span>}, <br> test_image_data, <br> <span class="hljs-number">-7</span>, <br> dl::DATA_TYPE_INT8, <br> <span class="hljs-literal">false</span>, <br> MALLOC_CAP_SPIRAM <br> );<br> <span class="hljs-keyword">return</span> input_tensor;<br>}<br></code></pre></td></tr></table></figure><p>推理过程的关键考虑：</p><ol><li>使用SPIRAM存储张量数据</li><li>采用int8量化数据类型</li><li>设置合适的量化参数（指数）</li></ol><h3 id="后处理实现"><a href="#后处理实现" class="headerlink" title="后处理实现"></a>后处理实现</h3><p>结果处理部分实现了softmax计算和置信度输出：</p><figure class="highlight cpp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><code class="hljs cpp"><span class="hljs-function"><span class="hljs-type">void</span> <span class="hljs-title">print_confidences</span><span class="hljs-params">(TensorBase* output_tensor)</span> </span>{<br> <span class="hljs-type">int8_t</span>* output_data = <span class="hljs-built_in">static_cast</span><<span class="hljs-type">int8_t</span>*>(<br> output_tensor-><span class="hljs-built_in">get_element_ptr</span>()<br> );<br> <span class="hljs-type">float</span> scale = std::<span class="hljs-built_in">pow</span>(<span class="hljs-number">2</span>, output_tensor->exponent);<br><br> <span class="hljs-comment">// Softmax计算</span><br> <span class="hljs-type">float</span> max_val = -INFINITY;<br> <span class="hljs-keyword">for</span> (<span class="hljs-type">int</span> i = <span class="hljs-number">0</span>; i < NUM_CLASSES; i++) {<br> <span class="hljs-type">float</span> val = output_data[i] * scale;<br> <span class="hljs-keyword">if</span> (val > max_val) max_val = val;<br> }<br><br> <span class="hljs-type">float</span> sum = <span class="hljs-number">0</span>;<br> <span class="hljs-function">std::vector<<span class="hljs-type">float</span>> <span class="hljs-title">confidences</span><span class="hljs-params">(NUM_CLASSES)</span></span>;<br> <span class="hljs-keyword">for</span> (<span class="hljs-type">int</span> i = <span class="hljs-number">0</span>; i < NUM_CLASSES; i++) {<br> <span class="hljs-type">float</span> val = std::<span class="hljs-built_in">exp</span>((output_data[i] * scale) - max_val);<br> confidences[i] = val;<br> sum += val;<br> }<br>}<br></code></pre></td></tr></table></figure><p>后处理的重点：</p><ul><li>正确的反量化处理</li><li>数值稳定性考虑</li><li>内存使用优化</li></ul><h2 id="性能优化"><a href="#性能优化" class="headerlink" title="性能优化"></a>性能优化</h2><h3 id="内存管理"><a href="#内存管理" class="headerlink" title="内存管理"></a>内存管理</h3><p>代码中实现了严格的内存管理策略：</p><figure class="highlight cpp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs cpp"><span class="hljs-type">size_t</span> free_mem_before = <span class="hljs-built_in">heap_caps_get_free_size</span>(MALLOC_CAP_SPIRAM);<br><span class="hljs-built_in">ESP_LOGI</span>(TAG, <span class="hljs-string">"Free memory before inference: %u bytes"</span>, free_mem_before);<br><br><span class="hljs-comment">// 运行推理...</span><br><br><span class="hljs-type">size_t</span> free_mem_after = <span class="hljs-built_in">heap_caps_get_free_size</span>(MALLOC_CAP_SPIRAM);<br><span class="hljs-built_in">ESP_LOGI</span>(TAG, <span class="hljs-string">"Free memory after inference: %u bytes"</span>, free_mem_after);<br></code></pre></td></tr></table></figure><p>内存优化措施：</p><ol><li>使用SPIRAM存储大型数据</li><li>及时释放不用的资源</li><li>监控内存使用情况</li></ol><h3 id="时间性能优化"><a href="#时间性能优化" class="headerlink" title="时间性能优化"></a>时间性能优化</h3><p>系统实现了详细的时间性能监控：</p><figure class="highlight cpp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs cpp"><span class="hljs-type">int64_t</span> start_time = <span class="hljs-built_in">esp_timer_get_time</span>();<br><span class="hljs-comment">// 模型加载...</span><br><span class="hljs-type">int64_t</span> load_time = <span class="hljs-built_in">esp_timer_get_time</span>();<br><span class="hljs-built_in">ESP_LOGI</span>(TAG, <span class="hljs-string">"Model loaded in %lld ms"</span>, (load_time - start_time) / <span class="hljs-number">1000</span>);<br><br><span class="hljs-type">int64_t</span> inference_start = <span class="hljs-built_in">esp_timer_get_time</span>();<br><span class="hljs-comment">// 执行推理...</span><br><span class="hljs-type">int64_t</span> inference_end = <span class="hljs-built_in">esp_timer_get_time</span>();<br><span class="hljs-built_in">ESP_LOGI</span>(TAG, <span class="hljs-string">"Inference time: %lld ms"</span>, <br> (inference_end - inference_start) / <span class="hljs-number">1000</span>);<br></code></pre></td></tr></table></figure><p>性能优化重点：</p><ol><li>模型加载时间优化</li><li>推理执行时间监控</li><li>系统响应时间分析</li></ol><h2 id="部署建议"><a href="#部署建议" class="headerlink" title="部署建议"></a>部署建议</h2><h3 id="内存配置"><a href="#内存配置" class="headerlink" title="内存配置"></a>内存配置</h3><ol><li><p>SPIRAM配置</p><ul><li>启用SPIRAM支持</li><li>配置适当的时钟频率</li><li>考虑内存对齐要求</li></ul></li><li><p>堆内存管理</p><ul><li>设置合理的堆大小</li><li>监控内存碎片</li><li>实现内存泄漏检测</li></ul></li></ol><h3 id="性能调优"><a href="#性能调优" class="headerlink" title="性能调优"></a>性能调优</h3><ol><li><p>时钟配置</p><ul><li>CPU频率选择</li><li>SPIRAM时钟配置</li><li>外设时钟优化</li></ul></li><li><p>中断处理</p><ul><li>最小化中断处理时间</li><li>合理的中断优先级设置</li><li>避免长时间禁用中断</li></ul></li></ol><h3 id="可靠性保障"><a href="#可靠性保障" class="headerlink" title="可靠性保障"></a>可靠性保障</h3><ol><li><p>看门狗配置</p><ul><li>设置合理的超时时间</li><li>实现任务监控</li><li>正确的复位处理</li></ul></li><li><p>错误处理</p><ul><li>完善的错误检查</li><li>日志记录机制</li><li>故障恢复策略</li></ul></li></ol>]]></content>
<categories>
<category>fyp</category>
</categories>
</entry>
<entry>
<title>模型量化优化技术文档</title>
<link href="/2024/11/29/model-quantization-docs/"/>
<url>/2024/11/29/model-quantization-docs/</url>
<content type="html"><![CDATA[<h1 id="模型量化优化技术文档"><a href="#模型量化优化技术文档" class="headerlink" title="模型量化优化技术文档"></a>模型量化优化技术文档</h1><h2 id="量化基础实现"><a href="#量化基础实现" class="headerlink" title="量化基础实现"></a>量化基础实现</h2><p>在边缘设备部署深度学习模型时，量化是一种重要的模型优化技术。本项目实现了三种不同的量化策略：基础8位量化、层间均衡量化和混合精度量化。</p><h3 id="数据准备"><a href="#数据准备" class="headerlink" title="数据准备"></a>数据准备</h3><p>所有量化策略都需要校准数据集来确定量化参数。以下是通用的数据加载实现：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">class</span> <span class="hljs-title class_">CalibrationDataset</span>(<span class="hljs-title class_ inherited__">Dataset</span>):<br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, X</span>):<br> <span class="hljs-variable language_">self</span>.X = torch.FloatTensor(X).unsqueeze(<span class="hljs-number">1</span>)<br> <br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__len__</span>(<span class="hljs-params">self</span>):<br> <span class="hljs-keyword">return</span> <span class="hljs-built_in">len</span>(<span class="hljs-variable language_">self</span>.X)<br> <br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__getitem__</span>(<span class="hljs-params">self, idx</span>):<br> <span class="hljs-keyword">return</span> <span class="hljs-variable language_">self</span>.X[idx]<br><br><span class="hljs-comment"># 加载校准数据</span><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'cal.pkl'</span>, <span class="hljs-string">'rb'</span>) <span class="hljs-keyword">as</span> f:<br> X_cal, _ = pickle.load(f)<br><br>cal_dataset = CalibrationDataset(X_cal)<br>cal_loader = DataLoader(cal_dataset, batch_size=<span class="hljs-number">32</span>, shuffle=<span class="hljs-literal">False</span>)<br></code></pre></td></tr></table></figure><p>这个实现的关键点在于：</p><ol><li>使用校准集而不是训练集进行量化参数确定</li><li>保持数据格式与训练时一致（添加通道维度）</li><li>使用固定的batch size以确保量化结果的稳定性</li></ol><h2 id="量化策略实现"><a href="#量化策略实现" class="headerlink" title="量化策略实现"></a>量化策略实现</h2><h3 id="基础8位量化"><a href="#基础8位量化" class="headerlink" title="基础8位量化"></a>基础8位量化</h3><p>最简单的量化策略，将模型参数和激活值量化为8位定点数：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs python">graph = espdl_quantize_onnx(<br> onnx_import_file=ONNX_MODEL_PATH,<br> espdl_export_file=EXPORT_PATH,<br> calib_dataloader=cal_loader,<br> calib_steps=<span class="hljs-number">50</span>,<br> input_shape=[input_shape],<br> target=<span class="hljs-string">"esp32s3"</span>,<br> num_of_bits=<span class="hljs-number">8</span>,<br> device=<span class="hljs-string">"cpu"</span>,<br> error_report=<span class="hljs-literal">True</span>,<br> verbose=<span class="hljs-number">1</span><br>)<br></code></pre></td></tr></table></figure><p>这种方法的特点：</p><ol><li>实现简单，整体压缩率固定在4倍左右</li><li>所有层使用相同的量化位宽</li><li>可能在某些关键层造成较大精度损失</li></ol><h3 id="层间均衡量化"><a href="#层间均衡量化" class="headerlink" title="层间均衡量化"></a>层间均衡量化</h3><p>通过调整相邻层的数值范围来减小量化误差：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 创建量化设置</span><br>quant_setting = QuantizationSettingFactory.espdl_setting()<br><br><span class="hljs-comment"># 启用层间均衡</span><br>quant_setting.equalization = <span class="hljs-literal">True</span><br>quant_setting.equalization_setting.iterations = <span class="hljs-number">4</span><br>quant_setting.equalization_setting.value_threshold = <span class="hljs-number">0.4</span><br>quant_setting.equalization_setting.opt_level = <span class="hljs-number">2</span><br><br><span class="hljs-comment"># 由于均衡优化，需要将ReLU6转换为ReLU</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">convert_relu6_to_relu</span>(<span class="hljs-params">model</span>):<br> <span class="hljs-keyword">for</span> node <span class="hljs-keyword">in</span> model.graph.node:<br> <span class="hljs-keyword">if</span> node.op_type == <span class="hljs-string">'Relu6'</span>:<br> node.op_type = <span class="hljs-string">'Relu'</span><br> <span class="hljs-keyword">return</span> model<br><br>model = convert_relu6_to_relu(model)<br></code></pre></td></tr></table></figure><p>这种方法的优势：</p><ol><li>通过层间权重重平衡减小量化误差</li><li>不增加模型大小的情况下提升精度</li><li>特别适合相邻层数值范围差异大的网络</li></ol><h3 id="混合精度量化"><a href="#混合精度量化" class="headerlink" title="混合精度量化"></a>混合精度量化</h3><p>对不同层使用不同的量化位宽，平衡精度和效率：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 设置混合精度量化配置</span><br>setting = QuantizationSettingFactory.espdl_setting()<br><br><span class="hljs-comment"># 对高误差层使用16位量化</span><br>high_error_layers = [<br> <span class="hljs-string">"/layers/layers.3/conv/conv.3/Conv"</span>,<br> <span class="hljs-string">"/layers/layers.3/conv/conv.6/Conv"</span>,<br> <span class="hljs-string">"/layers/layers.3/conv/conv.0/Conv"</span>,<br> <span class="hljs-string">"/layers/layers.3/conv/conv.3/Conv"</span><br>]<br><br><span class="hljs-keyword">for</span> layer <span class="hljs-keyword">in</span> high_error_layers:<br> setting.dispatching_table.append(<br> layer, <br> get_target_platform(<span class="hljs-string">"esp32s3"</span>, <span class="hljs-number">16</span>)<br> )<br></code></pre></td></tr></table></figure><p>这种方法的特点：</p><ol><li>精度敏感层保持高精度</li><li>其他层进行8位量化压缩</li><li>在模型大小和精度间取得更好的平衡</li></ol><h2 id="量化模型评估"><a href="#量化模型评估" class="headerlink" title="量化模型评估"></a>量化模型评估</h2><p>所有量化策略使用相同的评估方法：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">evaluate_quantized_model</span>(<span class="hljs-params">graph, test_loader, y_test</span>):<br> executor = TorchExecutor(graph=graph, device=<span class="hljs-string">'cpu'</span>)<br> total_time = <span class="hljs-number">0</span><br> correct = <span class="hljs-number">0</span><br> total = <span class="hljs-number">0</span><br> <br> <span class="hljs-keyword">with</span> torch.no_grad():<br> <span class="hljs-keyword">for</span> batch <span class="hljs-keyword">in</span> tqdm(test_loader):<br> start = time.time()<br> outputs = executor.forward(inputs=batch)<br> total_time += (time.time() - start)<br> <br> _, predicted = torch.<span class="hljs-built_in">max</span>(outputs[<span class="hljs-number">0</span>], <span class="hljs-number">1</span>)<br> total += batch.size(<span class="hljs-number">0</span>)<br> correct += (predicted == y_test[total-batch.size(<span class="hljs-number">0</span>):total]).<span class="hljs-built_in">sum</span>().item()<br><br> avg_time = (total_time / <span class="hljs-built_in">len</span>(test_loader)) * <span class="hljs-number">1000</span><br> accuracy = (correct / total) * <span class="hljs-number">100</span><br> <span class="hljs-keyword">return</span> avg_time, accuracy<br></code></pre></td></tr></table></figure><p>评估指标包含两个关键维度：</p><ol><li>推理时间：反映量化后模型的实际执行效率</li><li>模型精度：验证量化对模型性能的影响</li></ol><h2 id="实际部署验证"><a href="#实际部署验证" class="headerlink" title="实际部署验证"></a>实际部署验证</h2><p>为验证量化效果，实现了单张图片的推理测试：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">predict_image</span>(<span class="hljs-params">graph, image_tensor</span>):<br> executor = TorchExecutor(graph=graph, device=<span class="hljs-string">'cpu'</span>)<br> <span class="hljs-keyword">with</span> torch.no_grad():<br> outputs = executor.forward(inputs=[image_tensor])<br> confidences = torch.softmax(outputs[<span class="hljs-number">0</span>], dim=<span class="hljs-number">1</span>).numpy()<br> <span class="hljs-keyword">return</span> confidences<br></code></pre></td></tr></table></figure><p>图像预处理保持与训练时一致：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">preprocess_image</span>(<span class="hljs-params">image_path</span>):<br> img = cv2.imread(image_path)<br> gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)<br> resized = cv2.resize(gray, TARGET_SIZE)<br> normalized = resized.astype(<span class="hljs-string">'float32'</span>) / <span class="hljs-number">255.0</span><br> image_tensor = torch.FloatTensor(normalized).unsqueeze(<span class="hljs-number">0</span>).unsqueeze(<span class="hljs-number">0</span>)<br> <span class="hljs-keyword">return</span> image_tensor, img<br></code></pre></td></tr></table></figure><h2 id="量化策略对比"><a href="#量化策略对比" class="headerlink" title="量化策略对比"></a>量化策略对比</h2><h3 id="基础8位量化-1"><a href="#基础8位量化-1" class="headerlink" title="基础8位量化"></a>基础8位量化</h3><ul><li>优点：实现简单，压缩率固定</li><li>缺点：某些层可能精度损失较大</li><li>适用场景：模型结构简单，精度要求不高的场景</li></ul><h3 id="层间均衡量化-1"><a href="#层间均衡量化-1" class="headerlink" title="层间均衡量化"></a>层间均衡量化</h3><ul><li>优点：不增加模型大小的情况下提升精度</li><li>缺点：需要修改激活函数，可能影响模型表达能力</li><li>适用场景：相邻层数值分布差异大的网络</li></ul><h3 id="混合精度量化-1"><a href="#混合精度量化-1" class="headerlink" title="混合精度量化"></a>混合精度量化</h3><ul><li>优点：精度和压缩率的最佳平衡</li><li>缺点：需要针对网络结构进行层的量化位宽选择</li><li>适用场景：对某些层的精度有特殊要求的场景</li></ul><h2 id="实施建议"><a href="#实施建议" class="headerlink" title="实施建议"></a>实施建议</h2><ol><li><p>量化策略选择</p><ul><li>首先尝试基础8位量化</li><li>如果精度下降明显，考虑层间均衡</li><li>如果仍不满足要求，使用混合精度量化</li></ul></li><li><p>校准数据选择</p><ul><li>使用能代表实际应用场景的数据</li><li>数据量不需要太大，但要覆盖各种情况</li><li>标注数据需要准确</li></ul></li><li><p>评估验证</p><ul><li>在实际部署硬件上进行测试</li><li>关注模型大小和推理延迟</li><li>验证不同输入下的模型表现</li></ul></li><li><p>优化改进</p><ul><li>记录精度损失较大的层</li><li>针对性调整量化策略</li><li>在精度和效率间寻找平衡点</li></ul></li></ol>]]></content>
<categories>
<category>fyp</category>
</categories>
</entry>
<entry>
<title>手势识别数据集处理和模型训练文档</title>
<link href="/2024/11/29/gesture-recognition-code-docs/"/>
<url>/2024/11/29/gesture-recognition-code-docs/</url>
<content type="html"><![CDATA[<h1 id="手势识别数据集处理和模型训练文档"><a href="#手势识别数据集处理和模型训练文档" class="headerlink" title="手势识别数据集处理和模型训练文档"></a>手势识别数据集处理和模型训练文档</h1><h2 id="数据预处理实现"><a href="#数据预处理实现" class="headerlink" title="数据预处理实现"></a>数据预处理实现</h2><h3 id="图像预处理核心代码"><a href="#图像预处理核心代码" class="headerlink" title="图像预处理核心代码"></a>图像预处理核心代码</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">preprocess_image</span>(<span class="hljs-params">image_path</span>):<br> <span class="hljs-string">"""</span><br><span class="hljs-string"> 图片预处理函数:</span><br><span class="hljs-string"> 1. 读取图片并转灰度</span><br><span class="hljs-string"> 2. resize到96x96</span><br><span class="hljs-string"> 3. 归一化到[0,1]</span><br><span class="hljs-string"> """</span><br> img = cv2.imread(image_path)<br> gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)<br> resized = cv2.resize(gray, TARGET_SIZE)<br> normalized = resized.astype(<span class="hljs-string">'float32'</span>) / <span class="hljs-number">255.0</span><br> <span class="hljs-keyword">return</span> normalized<br></code></pre></td></tr></table></figure><p>这个预处理函数实现了多个关键功能：</p><ol><li><p>使用cv2.imread读取图片时返回BGR格式,需要转换为灰度图。选择灰度图而不是彩色图有两个原因：首先，手势识别主要依赖形状特征而不是颜色信息；其次，单通道输入可以显著减少模型参数量和计算量。</p></li><li><p>resize到96x96是在效果和性能间的平衡。过大的尺寸会增加计算量但边缘细节过多可能引入噪声，过小的尺寸会丢失重要特征。经过实验，96x96是一个比较好的平衡点。</p></li><li><p>归一化使用了float32而不是float64，这是深度学习中的最佳实践：</p><ul><li>float32提供足够的精度</li><li>相比float64可以节省一半的内存和带宽</li><li>现代GPU对float32的计算做了优化</li></ul></li></ol><h3 id="数据集构建与加载"><a href="#数据集构建与加载" class="headerlink" title="数据集构建与加载"></a>数据集构建与加载</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">class</span> <span class="hljs-title class_">GestureDataset</span>(<span class="hljs-title class_ inherited__">Dataset</span>):<br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, X, y, transform=<span class="hljs-literal">None</span></span>):<br> <span class="hljs-comment"># 确保数据格式正确,添加通道维度</span><br> <span class="hljs-variable language_">self</span>.X = torch.FloatTensor(X).unsqueeze(<span class="hljs-number">1</span>) <br> <span class="hljs-variable language_">self</span>.y = torch.LongTensor(y)<br> <span class="hljs-variable language_">self</span>.transform = transform<br> <br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__getitem__</span>(<span class="hljs-params">self, idx</span>):<br> image = <span class="hljs-variable language_">self</span>.X[idx]<br> label = <span class="hljs-variable language_">self</span>.y[idx]<br> <span class="hljs-keyword">if</span> <span class="hljs-variable language_">self</span>.transform:<br> image = <span class="hljs-variable language_">self</span>.transform(image)<br> <span class="hljs-keyword">return</span> image, label<br></code></pre></td></tr></table></figure><p>这个Dataset类的设计体现了几个重要的技术考量：</p><ol><li><p>继承PyTorch的Dataset类，这样可以利用PyTorch的DataLoader进行高效的数据加载：</p><ul><li>支持多进程加载，提高数据读取效率</li><li>自动处理batch划分</li><li>支持打乱数据顺序</li><li>支持内存固定(pin_memory)，提高GPU传输效率</li></ul></li><li><p>unsqueeze(1)操作在实现中非常关键：</p><ul><li>原始图片数据形状为(H,W)</li><li>CNN需要的输入形状是(C,H,W)</li><li>unsqueeze(1)在第1维度(通道维度)增加一个维度</li><li>这样就从(H,W)变成了(1,H,W)，符合CNN的输入要求</li></ul></li><li><p>transform的设计采用了可选参数模式：</p><ul><li>训练集需要数据增强，会传入transform</li><li>验证集和测试集不需要数据增强，transform为None</li><li>这种设计让一个Dataset类可以同时用于训练和评估</li></ul></li></ol><h2 id="模型架构实现"><a href="#模型架构实现" class="headerlink" title="模型架构实现"></a>模型架构实现</h2><h3 id="倒残差块-Inverted-Residual-Block"><a href="#倒残差块-Inverted-Residual-Block" class="headerlink" title="倒残差块(Inverted Residual Block)"></a>倒残差块(Inverted Residual Block)</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">class</span> <span class="hljs-title class_">InvertedResidual</span>(nn.Module):<br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, in_c, out_c, stride, expand_ratio</span>):<br> <span class="hljs-built_in">super</span>().__init__()<br> hidden_dim = in_c * expand_ratio<br> <span class="hljs-variable language_">self</span>.use_res = stride == <span class="hljs-number">1</span> <span class="hljs-keyword">and</span> in_c == out_c<br> <br> layers = []<br> <span class="hljs-keyword">if</span> expand_ratio != <span class="hljs-number">1</span>:<br> <span class="hljs-comment"># 升维卷积</span><br> layers.extend([<br> nn.Conv2d(in_c, hidden_dim, <span class="hljs-number">1</span>, bias=<span class="hljs-literal">False</span>),<br> nn.BatchNorm2d(hidden_dim),<br> nn.ReLU6(inplace=<span class="hljs-literal">True</span>)<br> ])<br> <br> layers.extend([<br> <span class="hljs-comment"># 深度可分离卷积</span><br> nn.Conv2d(hidden_dim, hidden_dim, <span class="hljs-number">3</span>, stride, <span class="hljs-number">1</span>, <br> groups=hidden_dim, bias=<span class="hljs-literal">False</span>),<br> nn.BatchNorm2d(hidden_dim),<br> nn.ReLU6(inplace=<span class="hljs-literal">True</span>),<br> <span class="hljs-comment"># 降维卷积</span><br> nn.Conv2d(hidden_dim, out_c, <span class="hljs-number">1</span>, bias=<span class="hljs-literal">False</span>),<br> nn.BatchNorm2d(out_c)<br> ])<br> <span class="hljs-variable language_">self</span>.conv = nn.Sequential(*layers)<br></code></pre></td></tr></table></figure><p>倒残差块是MobileNetV2中提出的创新结构，其设计包含多个精妙之处：</p><ol><li><p>expand_ratio控制特征通道的扩展：</p><ul><li>常规残差块是先降维再升维</li><li>倒残差块反其道而行之，先升维再降维</li><li>升维可以提供更大的特征空间，有利于特征提取</li><li>主要计算量在中间层，通过分组卷积降低计算复杂度</li></ul></li><li><p>深度可分离卷积的实现：</p><ul><li>groups=hidden_dim使每个通道独立卷积</li><li>相比常规3x3卷积可以降低9倍计算量</li><li>在精度损失很小的情况下大幅提升效率</li></ul></li><li><p>bias=False的使用：</p><ul><li>卷积层后接BatchNorm时可以省略偏置项</li><li>BatchNorm会学习偏置的作用</li><li>减少参数量并提供更好的正则化效果</li></ul></li><li><p>ReLU6的选择：</p><ul><li>相比普通ReLU，ReLU6在6处截断</li><li>适合低精度量化部署</li><li>有更好的数值稳定性</li></ul></li></ol><h3 id="完整网络结构"><a href="#完整网络结构" class="headerlink" title="完整网络结构"></a>完整网络结构</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">class</span> <span class="hljs-title class_">LightGestureNet</span>(nn.Module):<br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, num_classes=<span class="hljs-number">8</span></span>):<br> <span class="hljs-built_in">super</span>().__init__()<br> <br> <span class="hljs-comment"># 第一层卷积</span><br> <span class="hljs-variable language_">self</span>.first = nn.Sequential(<br> nn.Conv2d(<span class="hljs-number">1</span>, <span class="hljs-number">16</span>, <span class="hljs-number">3</span>, <span class="hljs-number">2</span>, <span class="hljs-number">1</span>, bias=<span class="hljs-literal">False</span>),<br> nn.BatchNorm2d(<span class="hljs-number">16</span>),<br> nn.ReLU6(inplace=<span class="hljs-literal">True</span>)<br> )<br> <br> <span class="hljs-comment"># 主干网络:4个倒残差块</span><br> <span class="hljs-variable language_">self</span>.layers = nn.Sequential(<br> InvertedResidual(<span class="hljs-number">16</span>, <span class="hljs-number">24</span>, <span class="hljs-number">2</span>, <span class="hljs-number">6</span>), <span class="hljs-comment"># stride=2</span><br> InvertedResidual(<span class="hljs-number">24</span>, <span class="hljs-number">24</span>, <span class="hljs-number">1</span>, <span class="hljs-number">6</span>), <span class="hljs-comment"># stride=1</span><br> InvertedResidual(<span class="hljs-number">24</span>, <span class="hljs-number">32</span>, <span class="hljs-number">2</span>, <span class="hljs-number">6</span>), <span class="hljs-comment"># stride=2</span><br> InvertedResidual(<span class="hljs-number">32</span>, <span class="hljs-number">32</span>, <span class="hljs-number">1</span>, <span class="hljs-number">6</span>) <span class="hljs-comment"># stride=1</span><br> )<br> <br> <span class="hljs-comment"># 分类头</span><br> <span class="hljs-variable language_">self</span>.classifier = nn.Sequential(<br> nn.AdaptiveAvgPool2d(<span class="hljs-number">1</span>),<br> nn.Flatten(),<br> nn.Linear(<span class="hljs-number">32</span>, num_classes)<br> )<br></code></pre></td></tr></table></figure><p>网络整体架构经过精心设计，每个部分都有其特定的作用：</p><ol><li><p>第一层卷积设计：</p><ul><li>使用stride=2直接下采样，减少后续计算量</li><li>16个输出通道提供基础特征表达</li><li>3x3卷积核在第一层提取基础纹理特征</li><li>padding=1保持特征图尺寸关系简单</li></ul></li><li><p>主干网络的设计考量：</p><ul><li>采用4个倒残差块，深度适中</li><li>通道数从16->24->32逐步上升，符合特征提取的规律</li><li>stride=2的块实现特征图尺寸降低</li><li>stride=1的块在相同尺度上精炼特征</li><li>expand_ratio=6提供了充足的特征变换空间</li></ul></li><li><p>分类头的特殊设计：</p><ul><li>使用AdaptiveAvgPool2d自适应池化到1x1</li><li>这种设计可以适应不同输入尺寸</li><li>全局平均池化相比全连接层参数更少</li><li>最后一层才使用全连接，减少过拟合风险</li></ul></li></ol><h2 id="训练实现"><a href="#训练实现" class="headerlink" title="训练实现"></a>训练实现</h2><h3 id="优化器与学习率设置"><a href="#优化器与学习率设置" class="headerlink" title="优化器与学习率设置"></a>优化器与学习率设置</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 优化器配置</span><br>optimizer = optim.Adam(model.parameters(), lr=<span class="hljs-number">0.001</span>)<br>scheduler = optim.lr_scheduler.CosineAnnealingLR(<br> optimizer, <br> T_max=num_epochs<br>)<br><br><span class="hljs-comment"># 自定义权重初始化</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">weight_init</span>(<span class="hljs-params">m</span>):<br> <span class="hljs-keyword">if</span> <span class="hljs-built_in">isinstance</span>(m, nn.Conv2d):<br> init.kaiming_normal_(m.weight, mode=<span class="hljs-string">'fan_out'</span>)<br> <span class="hljs-keyword">elif</span> <span class="hljs-built_in">isinstance</span>(m, nn.BatchNorm2d):<br> init.constant_(m.weight, <span class="hljs-number">1</span>)<br> init.constant_(m.bias, <span class="hljs-number">0</span>)<br> <span class="hljs-keyword">elif</span> <span class="hljs-built_in">isinstance</span>(m, nn.Linear):<br> init.xavier_normal_(m.weight)<br> <br>model.apply(weight_init)<br></code></pre></td></tr></table></figure><p>训练配置的每个选择都经过了仔细考虑：</p><ol><li><p>优化器选择Adam的原因：</p><ul><li>自适应学习率，减少手动调整</li><li>动量项帮助跨过局部最优</li><li>对学习率不太敏感，便于调试</li><li>RMSprop和动量的结合提供了良好的收敛性</li></ul></li><li><p>余弦退火学习率调度的优势：</p><ul><li>前期快速下降探索空间</li><li>中期缓慢下降精细搜索</li><li>后期小学习率精调模型</li><li>周期性调整可能跳出局部最优</li></ul></li><li><p>权重初始化的策略：</p><ul><li>卷积层用He初始化，适合ReLU激活函数</li><li>BatchNorm层的gamma初始化为1，beta初始化为0</li><li>全连接层采用Xavier初始化，在该层更适合</li><li>mode=’fan_out’考虑了输出神经元数量</li></ul></li></ol><h3 id="训练循环"><a href="#训练循环" class="headerlink" title="训练循环"></a>训练循环</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">for</span> epoch <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(num_epochs):<br> model.train()<br> train_loss = <span class="hljs-number">0</span><br> train_correct = <span class="hljs-number">0</span><br> <br> <span class="hljs-keyword">for</span> images, labels <span class="hljs-keyword">in</span> train_loader:<br> images, labels = images.to(device), labels.to(device)<br> <br> optimizer.zero_grad()<br> outputs = model(images)<br> loss = criterion(outputs, labels)<br> loss.backward()<br> optimizer.step()<br> <br> train_loss += loss.item()<br> _, predicted = outputs.<span class="hljs-built_in">max</span>(<span class="hljs-number">1</span>)<br> train_correct += predicted.eq(labels).<span class="hljs-built_in">sum</span>().item()<br> <br> <span class="hljs-comment"># 更新学习率</span><br> scheduler.step()<br> <br> <span class="hljs-comment"># 验证</span><br> model.<span class="hljs-built_in">eval</span>()<br> val_loss = <span class="hljs-number">0</span><br> val_correct = <span class="hljs-number">0</span><br> <br> <span class="hljs-keyword">with</span> torch.no_grad():<br> <span class="hljs-keyword">for</span> images, labels <span class="hljs-keyword">in</span> val_loader:<br> images, labels = images.to(device), labels.to(device)<br> outputs = model(images)<br> loss = criterion(outputs, labels)<br> val_loss += loss.item()<br> _, predicted = outputs.<span class="hljs-built_in">max</span>(<span class="hljs-number">1</span>)<br> val_correct += predicted.eq(labels).<span class="hljs-built_in">sum</span>().item()<br></code></pre></td></tr></table></figure><p>训练循环实现了多个重要的训练技巧：</p><ol><li><p>训练和验证的模式切换：</p><ul><li>model.train()启用BatchNorm和Dropout</li><li>model.eval()关闭这些随机行为</li><li>这对于获得稳定的验证结果很重要</li></ul></li><li><p>梯度清零和更新：</p><ul><li>optimizer.zero_grad()避免梯度累积</li><li>loss.backward()计算梯度</li><li>optimizer.step()更新参数</li><li>这个顺序保证了正确的参数更新</li></ul></li><li><p>验证阶段的优化：</p><ul><li>with torch.no_grad()避免存储计算图</li><li>减少了显存占用</li><li>提高了验证速度</li></ul></li><li><p>指标统计方法：</p><ul><li>loss.item()只获取标量值</li><li>predicted.eq(labels)计算正确预测</li><li>sum().item()统计正确总数</li><li>这些操作都很轻量，不影响训练速度</li></ul></li></ol><h2 id="模型导出"><a href="#模型导出" class="headerlink" title="模型导出"></a>模型导出</h2><h3 id="ONNX导出"><a href="#ONNX导出" class="headerlink" title="ONNX导出"></a>ONNX导出</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs python">dummy_input = torch.randn(<span class="hljs-number">1</span>, <span class="hljs-number">1</span>, <span class="hljs-number">96</span>, <span class="hljs-number">96</span>)<br>torch.onnx.export(model, <br> dummy_input, <br> <span class="hljs-string">'gesture_model.onnx'</span>,<br> input_names=[<span class="hljs-string">'input'</span>],<br> output_names=[<span class="hljs-string">'output'</span>],<br> dynamic_axes={<br> <span class="hljs-string">'input'</span>: {<span class="hljs-number">0</span>: <span class="hljs-string">'batch_size'</span>},<br> <span class="hljs-string">'output'</span>: {<span class="hljs-number">0</span>: <span class="hljs-string">'batch_size'</span>}<br> }<br>)<br></code></pre></td></tr></table></figure><p>ONNX导出过程包含几个关键技术点：</p><ol><li><p>dummy_input的设置：</p><ul><li>形状必须匹配模型输入要求</li><li>使用randn生成随机值</li><li>第一维是batch维度</li><li>单通道输入所以第二维是1</li></ul></li><li><p>dynamic_axes的作用：</p><ul><li>允许改变batch_size大小</li><li>适应不同的推理场景</li><li>其他维度保持固定</li><li>有利于部署时的灵活性</li></ul></li><li><p>命名规范：</p><ul><li>input_names和output_names要有意义</li><li>便于在不同框架中识别</li><li>方便推理时的张量绑定</li></ul></li></ol><h3 id="TFLite转换"><a href="#TFLite转换" class="headerlink" title="TFLite转换"></a>TFLite转换</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 转换为TFLite</span><br>converter = tf.lite.TFLiteConverter.from_saved_model(<span class="hljs-string">'tf_gesture_model'</span>)<br>converter.optimizations = [tf.lite.Optimize.DEFAULT]<br>converter.target_spec.supported_types = [tf.float32]<br>tflite_model = converter.convert()<br><br><span class="hljs-comment"># 保存模型</span><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'gesture_model.tflite'</span>, <span class="hljs-string">'wb'</span>) <span class="hljs-keyword">as</span> f:<br> f.write(tflite_model)<br></code></pre></td></tr></table></figure><p>TFLite转换涉及一些重要的优化设置：</p><ol><li><p>优化级别选择：</p><ul><li>DEFAULT优化是基础优化集合</li><li>包括常量折叠、算子融合等</li><li>在精度和性能间取得平衡</li><li>适合大多数移动端场景</li></ul></li><li><p>数据类型设置：</p><ul><li>使用float32保持精度</li><li>避免量化带来的精度损失</li><li>手势识别任务对精度敏感</li><li>模型足够小，不需要量化压缩</li></ul></li><li><p>部署考虑：</p><ul><li>TFLite模型可直接在移动端使用</li><li>支持CPU、GPU和NPU加速</li><li>兼容Android和iOS平台</li><li>便于集成到移动应用中</li></ul></li></ol><p>通过这样的导出配置，我们得到了一个既保持了原始精度，又适合移动端部署的模型。整个导出过程充分考虑了实际应用场景的需求，为后续的部署工作打下了良好的基础。</p>]]></content>
<categories>
<category>fyp</category>
</categories>
</entry>
<entry>
<title>ESP32-S3环境配置完整记录</title>
<link href="/2024/11/20/esp32-notes-nodup/"/>
<url>/2024/11/20/esp32-notes-nodup/</url>
<content type="html"><![CDATA[<h1 id="ESP32-S3环境配置完整记录"><a href="#ESP32-S3环境配置完整记录" class="headerlink" title="ESP32-S3环境配置完整记录"></a>ESP32-S3环境配置完整记录</h1><h2 id="1-ESP-IDF环境搭建"><a href="#1-ESP-IDF环境搭建" class="headerlink" title="1. ESP-IDF环境搭建"></a>1. ESP-IDF环境搭建</h2><h3 id="1-1-基础安装"><a href="#1-1-基础安装" class="headerlink" title="1.1 基础安装"></a>1.1 基础安装</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 克隆ESP-IDF仓库v4.4版本</span><br>git <span class="hljs-built_in">clone</span> -b release/v4.4 --recursive https://github.com/espressif/esp-idf.git<br><br><span class="hljs-comment"># 安装工具链</span><br><span class="hljs-built_in">cd</span> esp-idf<br>./install.sh<br><br><span class="hljs-comment"># 配置环境变量</span><br>. ./export.sh<br></code></pre></td></tr></table></figure><h3 id="1-2-自启动脚本配置"><a href="#1-2-自启动脚本配置" class="headerlink" title="1.2 自启动脚本配置"></a>1.2 自启动脚本配置</h3><p>最初尝试创建类似Windows下的快捷方式，但发现在Ubuntu中直接修改当前shell更合适。</p><p>最终验证可用的脚本(hello_world.sh):</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-meta">#!/bin/bash</span><br><br><span class="hljs-comment"># 加载 conda 命令</span><br><span class="hljs-built_in">source</span> ~/anaconda3/etc/profile.d/conda.sh<br><br><span class="hljs-comment"># 激活环境</span><br>conda activate dl_env<br><br><span class="hljs-comment"># 配置 ESP-IDF 环境</span><br><span class="hljs-built_in">source</span> ~/esp/esp-idf/export.sh<br><br><span class="hljs-comment"># 设置开发板和端口</span><br>idf.py set-target esp32s3<br><span class="hljs-built_in">export</span> ESPPORT=/dev/ttyCH341USB0<br><br><span class="hljs-comment"># 切换目录</span><br><span class="hljs-built_in">cd</span> ~/esp/esp-idf/examples/get-started/hello_world<br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"Conda 环境已激活，ESP-IDF 已配置为 esp32s3，串口设备为 <span class="hljs-variable">${ESPPORT}</span>"</span><br></code></pre></td></tr></table></figure><p><strong>重要说明</strong>: 必须使用 <code>source hello_world.sh</code> 运行脚本，而不是直接执行 <code>./hello_world.sh</code>，否则环境变量无法在当前shell中生效。</p><h3 id="1-3-Python环境问题处理"><a href="#1-3-Python环境问题处理" class="headerlink" title="1.3 Python环境问题处理"></a>1.3 Python环境问题处理</h3><p>运行脚本时遇到的Python依赖错误:</p><figure class="highlight subunit"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs subunit"><span class="hljs-keyword">Error: </span>The following Python requirements are not satisfied:<br>click>=7.0<br>future>=0.15.2<br>pyserial>=3.3<br>pyparsing>=2.0.3,<2.4.0<br>pyelftools>=0.22<br>...<br>construct 2.10.68 requires python<3.11<br></code></pre></td></tr></table></figure><p>解决步骤:</p><ol><li><p>确保Python版本正确（必须 <= 3.10）：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 创建新的conda环境</span><br>conda create -n esp-idf-env python=3.10<br>conda activate esp-idf-env<br></code></pre></td></tr></table></figure></li><li><p>安装所需依赖:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash">pip install -r ~/esp/esp-idf/requirements.txt<br></code></pre></td></tr></table></figure></li></ol><h2 id="2-CH341串口驱动安装配置"><a href="#2-CH341串口驱动安装配置" class="headerlink" title="2. CH341串口驱动安装配置"></a>2. CH341串口驱动安装配置</h2><h3 id="2-1-驱动安装过程"><a href="#2-1-驱动安装过程" class="headerlink" title="2.1 驱动安装过程"></a>2.1 驱动安装过程</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 1. 卸载旧版本驱动</span><br><span class="hljs-built_in">rm</span> -rf /lib/modlues/$(<span class="hljs-built_in">uname</span> -r)/kernel/drivers/usb/serial/ch341.ko<br><br><span class="hljs-comment"># 2. 解压驱动</span><br>unzip driver.zip<br><span class="hljs-built_in">cd</span> CH341SER_LINUX/driver<br><br><span class="hljs-comment"># 3. 编译驱动</span><br>make<br><br><span class="hljs-comment"># 4. 首次加载尝试</span><br><span class="hljs-built_in">sudo</span> insmod ch341.ko<br>或<br><span class="hljs-built_in">sudo</span> make load<br><span class="hljs-comment"># 错误: ERROR: could not insert module ch341.ko: Key was rejected by service</span><br><br><span class="hljs-comment"># 5. 强制加载驱动（如果首次加载失败其实强制加载也会失败，最终通过关闭bios中的secure boot解决，成功安装）</span><br><span class="hljs-built_in">sudo</span> modprobe -f ch341<br><br><span class="hljs-comment"># 6. 开机自动加载驱动</span><br><span class="hljs-built_in">sudo</span> <span class="hljs-built_in">cp</span> ch341.ko /lib/modules/$(<span class="hljs-built_in">uname</span> -r)/kernel/drivers/usb/serial/<br><span class="hljs-built_in">sudo</span> depmod<br></code></pre></td></tr></table></figure><h3 id="2-2-串口权限和验证"><a href="#2-2-串口权限和验证" class="headerlink" title="2.2 串口权限和验证"></a>2.2 串口权限和验证</h3><ol><li><p>添加用户到dialout组（需要注销重新登录生效）：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-built_in">sudo</span> usermod -a -G dialout <span class="hljs-variable">$USER</span><br><span class="hljs-built_in">groups</span> <span class="hljs-comment"># 验证用户组</span><br></code></pre></td></tr></table></figure></li><li><p>如果权限问题仍然存在：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 检查设备权限</span><br><span class="hljs-built_in">ls</span> -l /dev/ttyUSB0<br><span class="hljs-comment"># 输出: crw-rw---- 1 root dialout 188, 0 日期 时间 /dev/ttyUSB0</span><br><br><span class="hljs-comment"># 临时修改权限</span><br><span class="hljs-built_in">sudo</span> <span class="hljs-built_in">chmod</span> 666 /dev/ttyCH341USB0<br></code></pre></td></tr></table></figure></li><li><p>验证驱动加载和设备识别：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 检查驱动状态</span><br>lsmod | grep ch341<br><br><span class="hljs-comment"># 检查设备文件</span><br><span class="hljs-built_in">ls</span> /dev/ttyCH341USB*<br><span class="hljs-comment"># 应显示: /dev/ttyCH341USB0</span><br><br><span class="hljs-comment"># 查看系统日志</span><br>dmesg | grep <span class="hljs-built_in">tty</span><br><span class="hljs-comment"># 应包含:</span><br><span class="hljs-comment"># ch341-uart converter now attached to ttyUSB0</span><br><span class="hljs-comment"># ttyCH341USB0: ch341 USB device</span><br></code></pre></td></tr></table></figure></li></ol><h2 id="3-ESP32开发操作"><a href="#3-ESP32开发操作" class="headerlink" title="3. ESP32开发操作"></a>3. ESP32开发操作</h2><h3 id="3-1-编译和烧录"><a href="#3-1-编译和烧录" class="headerlink" title="3.1 编译和烧录"></a>3.1 编译和烧录</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 不指定串口时会报错</span><br>idf.py flash<br><span class="hljs-comment"># 错误: No serial ports found. Connect a device, or use '-p PORT' option to set a specific port.</span><br><br><span class="hljs-comment"># 正确的串口指定方式：</span><br><span class="hljs-comment"># 方式1: 直接指定串口</span><br>idf.py -p /dev/ttyCH341USB0 flash<br><br><span class="hljs-comment"># 方式2: 设置环境变量(推荐，已包含在启动脚本中)</span><br><span class="hljs-built_in">export</span> ESPPORT=/dev/ttyCH341USB0<br>idf.py flash<br></code></pre></td></tr></table></figure><h3 id="3-2-串口监控操作"><a href="#3-2-串口监控操作" class="headerlink" title="3.2 串口监控操作"></a>3.2 串口监控操作</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 启动监控</span><br>idf.py monitor <span class="hljs-comment"># 会使用ESPPORT环境变量</span><br><span class="hljs-comment"># 或</span><br>idf.py -p /dev/ttyCH341USB0 monitor<br><br><span class="hljs-comment"># 退出监控: Ctrl + ]</span><br></code></pre></td></tr></table></figure><h3 id="3-3-设备连接管理"><a href="#3-3-设备连接管理" class="headerlink" title="3.3 设备连接管理"></a>3.3 设备连接管理</h3><p>断开设备前的注意事项：</p><ol><li>使用 <code>Ctrl + ]</code> 退出监控状态</li><li>确认没有正在进行的数据传输或固件烧录</li><li>确认完成后可以直接拔出USB连接线</li></ol><p>重新连接后：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 检查设备是否被正确识别</span><br><span class="hljs-built_in">ls</span> /dev/ttyCH341USB*<br></code></pre></td></tr></table></figure><h2 id="4-常见问题和解决方案"><a href="#4-常见问题和解决方案" class="headerlink" title="4. 常见问题和解决方案"></a>4. 常见问题和解决方案</h2><h3 id="4-1-设备识别问题"><a href="#4-1-设备识别问题" class="headerlink" title="4.1 设备识别问题"></a>4.1 设备识别问题</h3><p>当设备无法被识别时：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-built_in">ls</span> /dev/ttyUSB*<br><span class="hljs-comment"># 报错: ls: cannot access '/dev/ttyUSB*': No such file or directory</span><br></code></pre></td></tr></table></figure><p>解决步骤：</p><ol><li>重新插拔设备</li><li>尝试其他USB端口</li><li>使用<code>ls /dev/ttyCH341USB*</code></li><li>按之前的步骤重新检查驱动状态</li></ol><h3 id="4-2-编译和烧录失败"><a href="#4-2-编译和烧录失败" class="headerlink" title="4.2 编译和烧录失败"></a>4.2 编译和烧录失败</h3><p>可能的原因和解决方案：</p><ol><li><p>目标板设置错误：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash">idf.py set-target esp32s3<br></code></pre></td></tr></table></figure></li><li><p>烧录失败：</p></li></ol><ul><li>检查ESPPORT环境变量是否正确</li><li>验证串口权限</li><li>确认USB连接稳定</li></ul><h3 id="4-3-监控问题处理"><a href="#4-3-监控问题处理" class="headerlink" title="4.3 监控问题处理"></a>4.3 监控问题处理</h3><ol><li>如果无法启动监控：</li></ol><ul><li>确认没有其他程序占用串口</li><li>检查串口权限和设备状态</li></ul><ol start="2"><li>监控退出：</li></ol><ul><li>始终使用 <code>Ctrl + ]</code> 正常退出</li><li>避免直接关闭终端窗口</li></ul>]]></content>
<categories>
<category>ai chat</category>
</categories>
</entry>
<entry>
<title>Ubuntu 系统配置指南</title>
<link href="/2024/11/20/ubuntu-config-guide-final/"/>
<url>/2024/11/20/ubuntu-config-guide-final/</url>
<content type="html"><![CDATA[<h1 id="Ubuntu-系统配置指南"><a href="#Ubuntu-系统配置指南" class="headerlink" title="Ubuntu 系统配置指南"></a>Ubuntu 系统配置指南</h1><h2 id="一、系统工具简介"><a href="#一、系统工具简介" class="headerlink" title="一、系统工具简介"></a>一、系统工具简介</h2><h3 id="1-监控工具"><a href="#1-监控工具" class="headerlink" title="1. 监控工具"></a>1. 监控工具</h3><ul><li><strong>Conky</strong>: 高度自定义，配置复杂，支持全面监控</li><li><strong>Indicator-SysMonitor</strong>: 顶部面板显示，轻量简洁</li><li><strong>Psensor</strong>: 专注温度监控，图形界面</li></ul><h3 id="2-截图工具"><a href="#2-截图工具" class="headerlink" title="2. 截图工具"></a>2. 截图工具</h3><ul><li><strong>Flameshot</strong>: GUI工具，功能全面，支持标注</li><li><strong>Scrot</strong>: 命令行工具，适合自动化脚本</li></ul><h2 id="二、Conky-配置"><a href="#二、Conky-配置" class="headerlink" title="二、Conky 配置"></a>二、Conky 配置</h2><h3 id="1-基础命令"><a href="#1-基础命令" class="headerlink" title="1. 基础命令"></a>1. 基础命令</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 安装</span><br><span class="hljs-built_in">sudo</span> apt update<br><span class="hljs-built_in">sudo</span> apt install conky-all<br><br><span class="hljs-comment"># 启动和调试</span><br>conky -c ~/.conkyrc <span class="hljs-comment"># 指定配置启动</span><br>conky -D <span class="hljs-comment"># 调试模式</span><br>killall conky <span class="hljs-comment"># 关闭所有实例</span><br></code></pre></td></tr></table></figure><h3 id="2-网络配置"><a href="#2-网络配置" class="headerlink" title="2. 网络配置"></a>2. 网络配置</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 检查网络接口</span><br>ip addr<br>ip route<br><br><span class="hljs-comment"># 获取活动接口</span><br>ip route get 8.8.8.8 | awk <span class="hljs-string">'/dev/ {print $5}'</span><br></code></pre></td></tr></table></figure><h3 id="3-完整配置文件"><a href="#3-完整配置文件" class="headerlink" title="3. 完整配置文件"></a>3. 完整配置文件</h3><p>保存为 <code>~/.conkyrc</code>:</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br></pre></td><td class="code"><pre><code class="hljs lua">conky.<span class="hljs-built_in">config</span> = {<br> background = <span class="hljs-literal">false</span>,<br> update_interval = <span class="hljs-number">1</span>,<br> double_buffer = <span class="hljs-literal">true</span>,<br> no_buffers = <span class="hljs-literal">true</span>,<br> imlib_cache_size = <span class="hljs-number">10</span>,<br><br> <span class="hljs-comment">-- Window specifications</span><br> gap_x = <span class="hljs-number">10</span>,<br> gap_y = <span class="hljs-number">50</span>,<br> minimum_width = <span class="hljs-number">400</span>,<br> maximum_width = <span class="hljs-number">400</span>,<br> alignment = <span class="hljs-string">'top_right'</span>,<br> own_window = <span class="hljs-literal">true</span>,<br> own_window_type = <span class="hljs-string">'normal'</span>,<br> own_window_transparent = <span class="hljs-literal">true</span>,<br> own_window_argb_visual = <span class="hljs-literal">true</span>,<br> own_window_argb_value = <span class="hljs-number">200</span>,<br> own_window_hints = <span class="hljs-string">'undecorated,sticky,skip_taskbar,skip_pager,below'</span>,<br><br> <span class="hljs-comment">-- Graphics settings</span><br> draw_shades = <span class="hljs-literal">false</span>,<br> draw_outline = <span class="hljs-literal">false</span>,<br> draw_borders = <span class="hljs-literal">false</span>,<br> draw_graph_borders = <span class="hljs-literal">false</span>,<br><br> <span class="hljs-comment">-- Text settings</span><br> use_xft = <span class="hljs-literal">true</span>,<br> xftalpha = <span class="hljs-number">0</span>,<br> font = <span class="hljs-string">'DejaVu Sans Mono:size=10'</span>,<br> text_buffer_size = <span class="hljs-number">2048</span>,<br><br> <span class="hljs-comment">-- Color scheme</span><br> default_color = <span class="hljs-string">'FFFFFF'</span>,<br> color1 = <span class="hljs-string">'FFA500'</span>, <span class="hljs-comment">-- 标题的橙色</span><br> color2 = <span class="hljs-string">'87CEEB'</span>, <span class="hljs-comment">-- 数据的天蓝色</span><br> color3 = <span class="hljs-string">'00FF00'</span>, <span class="hljs-comment">-- 正常状态的绿色</span><br> color4 = <span class="hljs-string">'FFFFFF'</span>, <span class="hljs-comment">-- 普通文本的白色</span><br> color5 = <span class="hljs-string">'FF4500'</span>, <span class="hljs-comment">-- 警告状态的橙红色</span><br> color6 = <span class="hljs-string">'DCDCDC'</span>, <span class="hljs-comment">-- 次要信息的灰色</span><br>}<br><br>conky.text = <span class="hljs-string">[[</span><br><span class="hljs-string">${color1}SYSTEM INFORMATION${color}</span><br><span class="hljs-string">Hostname: ${nodename}</span><br><span class="hljs-string"></span><br><span class="hljs-string">${color1}CPU INFORMATION${color}</span><br><span class="hljs-string">CPU Usage: ${cpu}% ${cpubar}</span><br><span class="hljs-string">${color6}Frequency:${color}</span><br><span class="hljs-string">${goto 20}Core 1: ${freq 1}MHz ${goto 200}Core 2: ${freq 2}MHz</span><br><span class="hljs-string">${goto 20}Core 3: ${freq 3}MHz ${goto 200}Core 4: ${freq 4}MHz</span><br><span class="hljs-string">${goto 20}Core 5: ${freq 5}MHz ${goto 200}Core 6: ${freq 6}MHz</span><br><span class="hljs-string">${goto 20}Core 7: ${freq 7}MHz ${goto 200}Core 8: ${freq 8}MHz</span><br><span class="hljs-string">${color6}Usage:${color}</span><br><span class="hljs-string">${goto 20}Core 1: ${cpu cpu1}% ${goto 150}${cpubar cpu1}</span><br><span class="hljs-string">${goto 20}Core 2: ${cpu cpu2}% ${goto 150}${cpubar cpu2}</span><br><span class="hljs-string">${goto 20}Core 3: ${cpu cpu3}% ${goto 150}${cpubar cpu3}</span><br><span class="hljs-string">${goto 20}Core 4: ${cpu cpu4}% ${goto 150}${cpubar cpu4}</span><br><span class="hljs-string">${goto 20}Core 5: ${cpu cpu5}% ${goto 150}${cpubar cpu5}</span><br><span class="hljs-string">${goto 20}Core 6: ${cpu cpu6}% ${goto 150}${cpubar cpu6}</span><br><span class="hljs-string">${goto 20}Core 7: ${cpu cpu7}% ${goto 150}${cpubar cpu7}</span><br><span class="hljs-string">${goto 20}Core 8: ${cpu cpu8}% ${goto 150}${cpubar cpu8}</span><br><span class="hljs-string">${color6}Temperature:${color}</span><br><span class="hljs-string">${goto 20}CPU Package: ${if_match ${execi 1 sensors | grep 'Package id 0:' | cut -c17-20} > 80}${color5}${endif}${execi 1 sensors | grep 'Package id 0:' | cut -c17-20}°C${color}</span><br><span class="hljs-string">${goto 20}Core 1: ${if_match ${execi 1 sensors | grep 'Core 0:' | cut -c17-20} > 80}${color5}${endif}${execi 1 sensors | grep 'Core 0:' | cut -c17-20}°C${color} ${goto 200}Core 2: ${if_match ${execi 1 sensors | grep 'Core 1:' | cut -c17-20} > 80}${color5}${endif}${execi 1 sensors | grep 'Core 1:' | cut -c17-20}°C${color}</span><br><span class="hljs-string">${color6}Fan Speed:${color}</span><br><span class="hljs-string">${goto 20}Fan 1: ${cat /sys/class/hwmon/hwmon4/fan1_input} RPM</span><br><span class="hljs-string">${goto 20}Fan 2: ${cat /sys/class/hwmon/hwmon4/fan2_input} RPM</span><br><span class="hljs-string">${color6}Power:${color}</span><br><span class="hljs-string">${goto 20}Package: ${execi 1 sudo turbostat --quiet --show PkgWatt --interval 1 --num_iterations 1 | tail -n 1 | awk '{print $1}'}W</span><br><span class="hljs-string"></span><br><span class="hljs-string">${color1}INTEL GPU INFORMATION${color}</span><br><span class="hljs-string">${color6}Frequency:${color} ${execi 1 timeout 1s sudo intel_gpu_top -l | tail -n 1 | awk '{print $1"/"$2" MHz"}'}</span><br><span class="hljs-string">${color6}Power:${color} ${execi 1 timeout 1s sudo intel_gpu_top -l | tail -n 1 | awk '{print $5"/"$6" W"}'}</span><br><span class="hljs-string">${color6}RC6:${color} ${execi 1 timeout 1s sudo intel_gpu_top -l | tail -n 1 | awk '{print $4}'}%</span><br><span class="hljs-string">${color6}Engine Usage:${color}</span><br><span class="hljs-string">${goto 20}Render: ${execi 1 timeout 1s sudo intel_gpu_top -l | head -n 3 | tail -n 1 | awk '{if ($8 != "") printf "%.1f%%", $8; else print "N/A"}'}</span><br><span class="hljs-string">${goto 20}Video: ${execi 1 timeout 1s sudo intel_gpu_top -l | head -n 3 | tail -n 1 | awk '{if ($14 != "") printf "%.1f%%", $14; else print "N/A"}'}</span><br><span class="hljs-string">${goto 20}Blitter: ${execi 1 timeout 1s sudo intel_gpu_top -l | head -n 3 | tail -n 1 | awk '{if ($11 != "") printf "%.1f%%", $11; else print "N/A"}'}</span><br><span class="hljs-string">${goto 20}Video Enhanced: ${execi 1 timeout 1s sudo intel_gpu_top -l | head -n 3 | tail -n 1 | awk '{if ($17 != "") printf "%.1f%%", $17; else print "N/A"}'}</span><br><span class="hljs-string"></span><br><span class="hljs-string">${color1}NVIDIA GPU INFORMATION${color}</span><br><span class="hljs-string">${color6}Utilization:${color}</span><br><span class="hljs-string">${goto 20}GPU: ${execi 1 nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits}%</span><br><span class="hljs-string">${goto 20}Memory: ${execi 1 nvidia-smi --query-gpu=utilization.memory --format=csv,noheader,nounits}%</span><br><span class="hljs-string">${color6}Memory Usage:${color} ${execi 1 nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits}/${execi 1 nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits} MB</span><br><span class="hljs-string">${color6}Frequency:${color}</span><br><span class="hljs-string">${goto 20}Core: ${execi 1 nvidia-smi --query-gpu=clocks.gr --format=csv,noheader,nounits} MHz</span><br><span class="hljs-string">${goto 20}Memory: ${execi 1 nvidia-smi --query-gpu=clocks.mem --format=csv,noheader,nounits} MHz</span><br><span class="hljs-string">${color6}Temperature:${color} ${if_match ${execi 1 nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader,nounits} > 80}${color5}${endif}${execi 1 nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader,nounits}°C${color}</span><br><span class="hljs-string">${color6}Power:${color} ${execi 1 nvidia-smi --query-gpu=power.draw --format=csv,noheader,nounits}W</span><br><span class="hljs-string">${color6}Real-Time Monitoring:${color}</span><br><span class="hljs-string">${goto 20}SM Usage: ${execi 1 nvidia-smi dmon -s u | head -n 3 | tail -n 1 | awk '{print $2}'} %</span><br><span class="hljs-string">${goto 20}Encoder (NVENC): ${execi 1 nvidia-smi dmon -s u | head -n 3 | tail -n 1 | awk '{print $4}'} %</span><br><span class="hljs-string">${goto 20}Decoder (NVDEC): ${execi 1 nvidia-smi dmon -s u | head -n 3 | tail -n 1 | awk '{print $5}'} %</span><br><span class="hljs-string">${goto 20}JPG Engine: ${execi 1 nvidia-smi dmon -s u | head -n 3 | tail -n 1 | awk '{print $6}'} %</span><br><span class="hljs-string">${goto 20}OFA Usage: ${execi 1 nvidia-smi dmon -s u | head -n 3 | tail -n 1 | awk '{print $7}'} %</span><br><span class="hljs-string"></span><br><span class="hljs-string">${color1}MEMORY USAGE${color}</span><br><span class="hljs-string">RAM: $mem/$memmax (${memperc}%) ${membar}</span><br><span class="hljs-string">Swap: $swap/$swapmax (${swapperc}%) ${swapbar}</span><br><span class="hljs-string">${color6}Frequency:${color}</span><br><span class="hljs-string">${goto 20}${execi 300 sudo lshw -C memory | grep 'clock' | awk 'NR==1 {print $2, $3}'}</span><br><span class="hljs-string">${goto 20}${execi 300 sudo lshw -C memory | grep 'clock' | awk 'NR==2 {print $2, $3}'}</span><br><span class="hljs-string">${goto 20}${execi 300 sudo lshw -C memory | grep 'clock' | awk 'NR==3 {print $2, $3}'}</span><br><span class="hljs-string"></span><br><span class="hljs-string">${color1}NETWORK INFORMATION${color}</span><br><span class="hljs-string">Download: ${downspeed enx7cc2c62733e5} ${goto 150}${downspeedgraph enx7cc2c62733e5 30,200 FFFFFF FFFFFF}</span><br><span class="hljs-string">Upload: ${upspeed enx7cc2c62733e5} ${goto 150}${upspeedgraph enx7cc2c62733e5 30,200 FFFFFF FFFFFF}</span><br><span class="hljs-string"></span><br><span class="hljs-string">${color1}DISK I/O${color}</span><br><span class="hljs-string">${goto 20}Read: ${diskio_read /dev/sda}</span><br><span class="hljs-string">${goto 20}Write: ${diskio_write /dev/sda}</span><br><span class="hljs-string">]]</span><br></code></pre></td></tr></table></figure><h4 id="Conky-配置详细说明"><a href="#Conky-配置详细说明" class="headerlink" title="Conky 配置详细说明"></a>Conky 配置详细说明</h4><h5 id="系统信息"><a href="#系统信息" class="headerlink" title="系统信息"></a>系统信息</h5><ul><li><strong>主机名</strong>: 显示系统主机名。</li></ul><h5 id="CPU-信息"><a href="#CPU-信息" class="headerlink" title="CPU 信息"></a>CPU 信息</h5><ul><li><strong>CPU 使用率</strong>: 显示整体 CPU 使用率百分比，包含条形图用于可视化。</li><li><strong>核心频率</strong>: 显示每个核心的频率，表示每个 CPU 核心的时钟速度（MHz）。</li><li><strong>核心使用率</strong>: 显示每个核心的使用率百分比，并用条形图表示，帮助理解各核心的负载分布。</li><li><strong>CPU 温度</strong>: 显示整个 CPU 的封装温度和各核心温度，超过 80°C 会触发警告颜色来提示过热。</li><li><strong>风扇速度</strong>: 显示 CPU 风扇的转速（RPM），表示散热系统的性能。</li><li><strong>功耗</strong>: 显示 CPU 封装的功耗（瓦特），反映当前的功率消耗。</li></ul><h5 id="Intel-GPU-信息"><a href="#Intel-GPU-信息" class="headerlink" title="Intel GPU 信息"></a>Intel GPU 信息</h5><ul><li><strong>频率</strong>: 显示 Intel GPU 当前的频率（MHz），指示其性能水平。</li><li><strong>功耗</strong>: 显示 Intel GPU 的功耗（瓦特），反映其活动时的功率消耗。</li><li><strong>RC6 节能状态</strong>: 显示 GPU 处于低功耗状态的时间百分比，评估其功耗管理效率。</li><li><strong>引擎使用率</strong>:<ul><li><strong>渲染引擎</strong>: 显示渲染引擎的使用率，负责处理 3D 图形。</li><li><strong>视频引擎</strong>: 显示视频引擎的使用率，处理视频任务。</li><li><strong>Blitter 引擎</strong>: 显示 Blitter 引擎的使用率，负责数据传输和 2D 操作。</li><li><strong>视频增强</strong>: 显示增强视频引擎的使用率，进一步优化视频处理。</li></ul></li></ul><h5 id="NVIDIA-GPU-信息"><a href="#NVIDIA-GPU-信息" class="headerlink" title="NVIDIA GPU 信息"></a>NVIDIA GPU 信息</h5><ul><li><strong>GPU 核心使用率</strong>: 显示 GPU 核心的使用率，表示计算能力的使用情况。</li><li><strong>内存使用率</strong>: 显示 GPU 内存的使用百分比，反映显存的负载。</li><li><strong>内存使用情况</strong>: 显示 GPU 使用的内存和总内存（MB），提供更清晰的内存负载视图。</li><li><strong>核心频率</strong>: 显示 GPU 核心的频率（MHz），表示时钟速度。</li><li><strong>内存频率</strong>: 显示 GPU 内存的频率（MHz），反映 GPU 与其内存之间的数据传输速率。</li><li><strong>温度</strong>: 显示 GPU 的当前温度，超过 80°C 时使用警告颜色突出显示。</li><li><strong>功耗</strong>: 显示 GPU 的功耗（瓦特），反映其功率消耗。</li><li><strong>实时监控</strong>:<ul><li><strong>流处理器（SM）使用率</strong>: 显示流处理器单元的使用率，负责并行计算任务。</li><li><strong>编码器（NVENC）使用率</strong>: 显示 NVENC 编码器的使用率，用于硬件视频编码。</li><li><strong>解码器（NVDEC）使用率</strong>: 显示 NVDEC 解码器的使用率，用于硬件视频解码。</li><li><strong>JPEG 引擎使用率</strong>: 显示 JPEG 引擎的使用情况，负责 JPEG 压缩和解压缩。</li><li><strong>光流加速器（OFA）使用率</strong>: 显示光流加速器的使用率，用于视频中的运动估计等任务。</li></ul></li></ul><h5 id="内存使用情况"><a href="#内存使用情况" class="headerlink" title="内存使用情况"></a>内存使用情况</h5><ul><li><strong>RAM 使用情况</strong>: 显示系统 RAM 的总量、已用量和使用百分比，并用条形图表示。</li><li><strong>交换分区使用情况</strong>: 显示交换分区的使用情况，表示虚拟内存的使用量。</li><li><strong>内存频率</strong>: 显示不同内存模块的时钟频率，影响数据传输速度。</li></ul><h5 id="网络信息"><a href="#网络信息" class="headerlink" title="网络信息"></a>网络信息</h5><ul><li><strong>下载速度</strong>: 显示当前下载速度，并提供图表用于可视化。</li><li><strong>上传速度</strong>: 显示当前上传速度，并提供图表用于可视化。</li></ul><h5 id="磁盘-I-O"><a href="#磁盘-I-O" class="headerlink" title="磁盘 I/O"></a>磁盘 I/O</h5><ul><li><strong>读取速度</strong>: 显示磁盘的读取速度（MB/s），表示数据从磁盘读取的速度。</li><li><strong>写入速度</strong>: 显示磁盘的写入速度（MB/s），表示数据写入磁盘的速度。</li></ul><h3 id="4-自启动配置"><a href="#4-自启动配置" class="headerlink" title="4. 自启动配置"></a>4. 自启动配置</h3><p>创建文件 <code>~/.config/autostart/conky.desktop</code>:</p><figure class="highlight ini"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs ini"><span class="hljs-section">[Desktop Entry]</span><br><span class="hljs-attr">Type</span>=Application<br><span class="hljs-attr">Exec</span>=bash -c <span class="hljs-string">"sleep 10 && conky -c ~/.conkyrc"</span><br><span class="hljs-attr">Hidden</span>=<span class="hljs-literal">false</span><br><span class="hljs-attr">NoDisplay</span>=<span class="hljs-literal">false</span><br><span class="hljs-attr">X-GNOME-Autostart-enabled</span>=<span class="hljs-literal">true</span><br><span class="hljs-attr">Name</span>=Conky<br><span class="hljs-attr">Comment</span>=System Monitor<br></code></pre></td></tr></table></figure><h2 id="三、Flameshot-配置"><a href="#三、Flameshot-配置" class="headerlink" title="三、Flameshot 配置"></a>三、Flameshot 配置</h2><h3 id="1-基础命令-1"><a href="#1-基础命令-1" class="headerlink" title="1. 基础命令"></a>1. 基础命令</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># Installation</span><br><span class="hljs-built_in">sudo</span> apt install flameshot<br><br><span class="hljs-comment"># Common commands</span><br>flameshot gui <span class="hljs-comment"># 启动GUI</span><br>flameshot full <span class="hljs-comment"># 全屏截图</span><br>flameshot screen <span class="hljs-comment"># 截取当前屏幕</span><br>flameshot launcher <span class="hljs-comment"># 打开启动器</span><br>flameshot config <span class="hljs-comment"># 打开配置窗口</span><br><br><span class="hljs-comment"># Command line options</span><br>-h, --<span class="hljs-built_in">help</span> <span class="hljs-comment"># 显示帮助信息</span><br>-v, --version <span class="hljs-comment"># 显示版本信息</span><br>-c, --clipboard <span class="hljs-comment"># 截图复制到剪贴板</span><br>-p, --path <span class="hljs-comment"># 指定保存路径</span><br>-d, --delay <span class="hljs-comment"># 设置延迟（毫秒）</span><br>-r, --raw <span class="hljs-comment"># 输出原始数据</span><br>-g, --gui <span class="hljs-comment"># 显示界面</span><br>-s, --selection <span class="hljs-comment"># 选择区域截图</span><br></code></pre></td></tr></table></figure><h3 id="2-快捷键列表"><a href="#2-快捷键列表" class="headerlink" title="2. 快捷键列表"></a>2. 快捷键列表</h3><h4 id="2-1-截图界面快捷键"><a href="#2-1-截图界面快捷键" class="headerlink" title="2.1 截图界面快捷键"></a>2.1 截图界面快捷键</h4><ul><li><code>Space</code>: 切换左侧工具栏</li><li><code>Ctrl + C</code>: 复制截图到剪贴板</li><li><code>Ctrl + S</code>: 保存截图</li><li><code>Ctrl + Z</code>: 撤销上一步操作</li><li><code>Right Click</code>: 显示上下文菜单</li><li><code>Mouse Wheel</code>: 更改工具笔画宽度</li><li><code>Shift</code>: 在绘制时保持正方形/圆形</li><li><code>Esc</code> / <code>Delete</code>: 退出截图</li><li><code>Return</code>: 确认截图</li></ul><h4 id="2-2-工具快捷键"><a href="#2-2-工具快捷键" class="headerlink" title="2.2 工具快捷键"></a>2.2 工具快捷键</h4><ul><li><code>1</code>: 矩形工具</li><li><code>2</code>: 圆形工具</li><li><code>3</code>: 箭头工具</li><li><code>4</code>: 画笔工具</li><li><code>5</code>: 马克笔工具</li><li><code>6</code>: 文本工具</li><li><code>7</code>: 选择工具</li><li><code>8</code>: 贴纸工具</li><li><code>9</code>: 数字工具</li></ul><h3 id="3-自启动配置"><a href="#3-自启动配置" class="headerlink" title="3. 自启动配置"></a>3. 自启动配置</h3><p>创建文件 <code>~/.config/autostart/flameshot.desktop</code>:</p><figure class="highlight ini"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs ini"><span class="hljs-section">[Desktop Entry]</span><br><span class="hljs-attr">Type</span>=Application<br><span class="hljs-attr">Exec</span>=flameshot<br><span class="hljs-attr">Hidden</span>=<span class="hljs-literal">false</span><br><span class="hljs-attr">NoDisplay</span>=<span class="hljs-literal">false</span><br><span class="hljs-attr">X-GNOME-Autostart-enabled</span>=<span class="hljs-literal">true</span><br><span class="hljs-attr">Name</span>=Flameshot<br><span class="hljs-attr">Comment</span>=Start Flameshot in system tray<br></code></pre></td></tr></table></figure><h2 id="四、问题排查"><a href="#四、问题排查" class="headerlink" title="四、问题排查"></a>四、问题排查</h2><h3 id="1-Conky-常见问题"><a href="#1-Conky-常见问题" class="headerlink" title="1. Conky 常见问题"></a>1. Conky 常见问题</h3><h4 id="1-1-网速显示为-0"><a href="#1-1-网速显示为-0" class="headerlink" title="1.1 网速显示为 0"></a>1.1 网速显示为 0</h4><ul><li><strong>现象</strong>：网络速度一直显示为 0，即使有实际流量</li><li><strong>原因</strong>：<ul><li>网络接口名称配置错误</li><li>动态接口检测命令解析问题</li></ul></li><li><strong>解决方案</strong>：<ol><li>检查实际网络接口名称，并确认活动接口（见上文网络配置部分）</li><li>更新配置文件中的接口名称：<figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs lua">${downspeed 实际接口名称}<br>${upspeed 实际接口名称}<br></code></pre></td></tr></table></figure></li></ol></li></ul><h4 id="1-2-页面显示重叠"><a href="#1-2-页面显示重叠" class="headerlink" title="1.2 页面显示重叠"></a>1.2 页面显示重叠</h4><ul><li><strong>现象</strong>：文本和数据显示重叠，界面混乱</li><li><strong>原因</strong>：<ul><li>配置文件语法错误</li><li>Lua 语法版本不兼容</li><li>窗口大小设置不当</li></ul></li><li><strong>解决方案</strong>：<ol><li>确保使用正确的配置文件结构：<figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs lua">conky.<span class="hljs-built_in">config</span> = {<br> <span class="hljs-comment">-- 配置项</span><br>}<br>conky.text = <span class="hljs-string">[[</span><br><span class="hljs-string"> -- 显示内容</span><br><span class="hljs-string">]]</span><br></code></pre></td></tr></table></figure></li><li>调整窗口大小：<figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs lua">minimum_width = <span class="hljs-number">400</span>,<br>maximum_width = <span class="hljs-number">400</span>,<br></code></pre></td></tr></table></figure></li><li>使用对齐和间距：<figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs lua">gap_x = <span class="hljs-number">10</span>,<br>gap_y = <span class="hljs-number">50</span>,<br>alignment = <span class="hljs-string">'top_right'</span>,<br></code></pre></td></tr></table></figure></li></ol></li></ul><h4 id="1-3-显示格式问题"><a href="#1-3-显示格式问题" class="headerlink" title="1.3 显示格式问题"></a>1.3 显示格式问题</h4><ul><li><strong>现象</strong>：<ul><li>速度单位混合显示B、 KiB 和 MiB</li><li>磁盘 I/O 显示异常</li></ul></li><li><strong>解决方案</strong>：<ol><li>移除配置中手动添加的单位</li><li>使用内置变量：<figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs lua"># 正确写法<br>${diskio_read}<br>${diskio_write}<br></code></pre></td></tr></table></figure></li><li>网络速度显示：<figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs lua">${downspeed enx7cc2c62733e5}<br>${upspeed enx7cc2c62733e5}<br></code></pre></td></tr></table></figure></li></ol></li></ul><h3 id="2-Flameshot-自启动问题"><a href="#2-Flameshot-自启动问题" class="headerlink" title="2. Flameshot 自启动问题"></a>2. Flameshot 自启动问题</h3><ul><li><strong>检查步骤</strong>：<ol><li>确认配置文件权限：<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-built_in">chmod</span> +x ~/.config/autostart/flameshot.desktop<br></code></pre></td></tr></table></figure></li><li>检查系统托盘支持：<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-built_in">sudo</span> apt install gnome-shell-extension-appindicator<br></code></pre></td></tr></table></figure></li><li>验证启动命令：<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash">flameshot<br></code></pre></td></tr></table></figure></li></ol></li></ul><h2 id="五、其他注意事项"><a href="#五、其他注意事项" class="headerlink" title="五、其他注意事项"></a>五、其他注意事项</h2><h3 id="1-文件权限"><a href="#1-文件权限" class="headerlink" title="1. 文件权限"></a>1. 文件权限</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># Set permissions</span><br><span class="hljs-built_in">chmod</span> +x ~/.config/autostart/*.desktop<br><span class="hljs-built_in">chmod</span> 644 ~/.conkyrc<br><br><span class="hljs-comment"># Check permissions</span><br><span class="hljs-built_in">ls</span> -l ~/.config/autostart/<br><span class="hljs-built_in">ls</span> -l ~/.conkyrc<br></code></pre></td></tr></table></figure><h3 id="2-日志查看"><a href="#2-日志查看" class="headerlink" title="2. 日志查看"></a>2. 日志查看</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># Check processes</span><br>ps aux | grep conky<br>ps aux | grep flameshot<br><br><span class="hljs-comment"># View logs</span><br>journalctl -xe | grep conky<br>journalctl -xe | grep flameshot<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>ai chat</category>
</categories>
</entry>
<entry>
<title>Python虚拟环境与深度学习环境配置详细笔记</title>
<link href="/2024/11/19/comprehensive-learning-notes/"/>
<url>/2024/11/19/comprehensive-learning-notes/</url>
<content type="html"><![CDATA[<h1 id="Python虚拟环境与深度学习环境配置详细笔记"><a href="#Python虚拟环境与深度学习环境配置详细笔记" class="headerlink" title="Python虚拟环境与深度学习环境配置详细笔记"></a>Python虚拟环境与深度学习环境配置详细笔记</h1><h2 id="1-Python虚拟环境的选择与使用"><a href="#1-Python虚拟环境的选择与使用" class="headerlink" title="1. Python虚拟环境的选择与使用"></a>1. Python虚拟环境的选择与使用</h2><h3 id="1-1-主流虚拟环境方案详细对比"><a href="#1-1-主流虚拟环境方案详细对比" class="headerlink" title="1.1 主流虚拟环境方案详细对比"></a>1.1 主流虚拟环境方案详细对比</h3><ol><li><p><strong>venv</strong>（Python标准库自带）</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 创建环境</span><br>python -m venv myenv<br><br><span class="hljs-comment"># 激活环境</span><br><span class="hljs-comment"># Windows</span><br>myenv\Scripts\activate<br><span class="hljs-comment"># Linux/macOS</span><br><span class="hljs-built_in">source</span> myenv/bin/activate<br></code></pre></td></tr></table></figure><ul><li>优点：<ul><li>轻量级，不占用额外空间</li><li>Python标准库自带，无需额外安装</li><li>使用pip管理包，过程简单直观</li></ul></li><li>缺点：<ul><li>功能相对简单</li><li>科学计算包可能需要额外编译</li></ul></li><li>适用场景：<ul><li>Web开发项目</li><li>API开发</li><li>轻量级Python应用</li><li>需要部署到资源受限环境的项目</li></ul></li></ul></li><li><p><strong>conda</strong>（Anaconda自带）</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 创建指定Python版本的环境</span><br>conda create -n myenv python=3.9<br><br><span class="hljs-comment"># 激活环境</span><br>conda activate myenv<br><br><span class="hljs-comment"># 安装包示例</span><br>conda install numpy pandas scikit-learn<br></code></pre></td></tr></table></figure><ul><li>优点：<ul><li>提供完整的科学计算环境</li><li>包含Jupyter Notebook、Spyder等IDE</li><li>预编译的科学计算库，安装快速</li><li>强大的包管理和环境管理功能</li><li>支持Python之外的其他编程语言</li><li>GPU相关配置更简单</li></ul></li><li>缺点：<ul><li>安装包体积大（几GB）</li><li>占用磁盘空间较多</li><li>学习曲线相对较陡</li></ul></li><li>适用场景：<ul><li>数据科学项目</li><li>机器学习研究</li><li>需要大量科学计算的项目</li><li>跨平台开发需求</li></ul></li></ul></li><li><p><strong>virtualenv</strong>（第三方工具）</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 安装virtualenv</span><br>pip install virtualenv<br><br><span class="hljs-comment"># 创建环境</span><br>virtualenv myenv<br></code></pre></td></tr></table></figure><ul><li>优点：<ul><li>比venv功能更丰富</li><li>可以指定Python版本</li><li>更灵活的配置选项</li></ul></li><li>缺点：<ul><li>需要单独安装</li><li>配置相对复杂</li></ul></li><li>适用场景：<ul><li>需要特定Python版本的项目</li><li>需要更多控制选项的场景</li></ul></li></ul></li><li><p><strong>pipenv</strong>（集成工具）</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 安装pipenv</span><br>pip install pipenv<br><br><span class="hljs-comment"># 在项目目录下创建环境</span><br>pipenv install<br></code></pre></td></tr></table></figure><ul><li>优点：<ul><li>自动管理项目依赖</li><li>集成了包管理和虚拟环境功能</li><li>提供锁文件确保环境一致性</li></ul></li><li>缺点：<ul><li>使用方式与传统虚拟环境不同</li><li>某些情况下速度较慢</li></ul></li><li>适用场景：<ul><li>注重依赖管理的项目</li><li>团队协作项目</li></ul></li></ul></li></ol><h3 id="1-2-深度学习项目的环境选择"><a href="#1-2-深度学习项目的环境选择" class="headerlink" title="1.2 深度学习项目的环境选择"></a>1.2 深度学习项目的环境选择</h3><ol><li><p><strong>为什么选择Anaconda</strong></p><ul><li>预装了大量机器学习相关包</li><li>CUDA和cuDNN的配置更简单</li><li>环境管理更直观，便于切换不同框架版本</li><li>预编译的科学计算库减少了安装错误</li><li>跨平台支持好，便于在不同系统间迁移项目</li></ul></li><li><p><strong>实际使用中的重要特性</strong></p><ul><li><p>在conda环境中使用pip：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 激活环境后使用pip安装</span><br>conda activate myenv<br>pip install tensorflow<br></code></pre></td></tr></table></figure><p>包会被正确安装到<strong>当前conda环境</strong></p></li><li><p>环境复制和分享：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 导出环境配置</span><br>conda <span class="hljs-built_in">env</span> <span class="hljs-built_in">export</span> > environment.yml<br><br><span class="hljs-comment"># 从配置文件创建环境</span><br>conda <span class="hljs-built_in">env</span> create -f environment.yml<br></code></pre></td></tr></table></figure></li></ul></li></ol><h2 id="2-深度学习环境配置实战"><a href="#2-深度学习环境配置实战" class="headerlink" title="2. 深度学习环境配置实战"></a>2. 深度学习环境配置实战</h2><h3 id="2-1-版本选择与兼容性分析"><a href="#2-1-版本选择与兼容性分析" class="headerlink" title="2.1 版本选择与兼容性分析"></a>2.1 版本选择与兼容性分析</h3><ol><li><p><strong>深度学习框架与CUDA版本对应关系</strong></p><p>TensorFlow支持的CUDA版本：</p><ul><li>TensorFlow 2.17.0：支持CUDA 12.3</li><li>TensorFlow 2.18.0：支持CUDA 12.3</li></ul><p>PyTorch支持的CUDA版本：</p><ul><li>PyTorch 2.4.0：支持CUDA 12.1</li><li>PyTorch 2.5.1：支持CUDA 12.1和12.4</li></ul></li><li><p><strong>Python版本选择考虑</strong></p><ul><li>Python 3.9：部分旧版本框架的选择</li><li>Python 3.10：PyTorch 2.4.0的最低要求</li><li>Python 3.11：最新版本的推荐选择</li></ul></li><li><p><strong>最终挑选版本组合</strong></p><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">Python</span>: <span class="hljs-number">3</span>.<span class="hljs-number">11</span><br><span class="hljs-attribute">CUDA</span>: <span class="hljs-number">12</span>.<span class="hljs-number">4</span><br><span class="hljs-attribute">cuDNN</span>: <span class="hljs-number">9</span>.<span class="hljs-number">3</span><br><span class="hljs-attribute">TensorFlow</span>: <span class="hljs-number">2</span>.<span class="hljs-number">18</span>.<span class="hljs-number">0</span><br><span class="hljs-attribute">PyTorch</span>: <span class="hljs-number">2</span>.<span class="hljs-number">5</span>.<span class="hljs-number">1</span><br></code></pre></td></tr></table></figure></li></ol><h3 id="2-2-NVIDIA驱动与CUDA安装详解"><a href="#2-2-NVIDIA驱动与CUDA安装详解" class="headerlink" title="2.2 NVIDIA驱动与CUDA安装详解"></a>2.2 NVIDIA驱动与CUDA安装详解</h3><ol><li><p><strong>系统环境检查</strong></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 检查NVIDIA驱动版本和当前CUDA支持</span><br>nvidia-smi<br></code></pre></td></tr></table></figure><p>示例输出分析：</p><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">NVIDIA</span>-SMI <span class="hljs-number">565</span>.<span class="hljs-number">57</span>.<span class="hljs-number">01</span> Driver Version: <span class="hljs-number">565</span>.<span class="hljs-number">57</span>.<span class="hljs-number">01</span> CUDA Version: <span class="hljs-number">12</span>.<span class="hljs-number">7</span><br></code></pre></td></tr></table></figure><ul><li>Driver Version表示当前安装的NVIDIA驱动版本</li><li>CUDA Version表示驱动<strong>支持</strong>的最高CUDA版本，注意这里不是<strong>安装</strong>的版本</li></ul></li><li><p><strong>CUDA安装方法选择</strong></p><p>a. 包管理器安装（不推荐）：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs bash">wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb<br><span class="hljs-built_in">sudo</span> dpkg -i cuda-keyring_1.1-1_all.deb<br><span class="hljs-built_in">sudo</span> apt-get update<br><span class="hljs-built_in">sudo</span> apt-get install cuda-toolkit-12-4<br></code></pre></td></tr></table></figure><p>遇到的问题：</p><ul><li>Ubuntu 24.04兼容性问题，即使成功添加还是无法安装</li><li>依赖关系复杂</li></ul><p>b. runfile安装（推荐）：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 下载runfile（从NVIDIA官网）</span><br>wget https://developer.download.nvidia.com/compute/cuda/12.4.0/local_installers/cuda_12.4.0_550.54.14_linux.run<br><br><span class="hljs-comment"># 使用推荐的参数安装</span><br><span class="hljs-built_in">sudo</span> sh cuda_12.4.XX_linux.run --no-drm --silent --toolkit --override --override-driver-check<br></code></pre></td></tr></table></figure><p>参数说明：</p><ul><li><code>--no-drm</code>：不安装NVIDIA驱动模块</li><li><code>--silent</code>：静默安装</li><li><code>--toolkit</code>：只安装CUDA工具包</li><li><code>--override</code>：忽略兼容性检查</li><li><code>--override-driver-check</code>：忽略驱动检查<br>这里不得不忽略各种检查，否则因为我安装了新版本的显卡驱动，一直无法安装12.4版本的toolkit。</li></ul></li><li><p><strong>处理已有CUDA版本</strong></p><p>检查现有CUDA安装：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-built_in">ls</span> -l /usr/local/cuda*<br></code></pre></td></tr></table></figure><p>如果发现多个版本：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 删除不需要的版本</span><br><span class="hljs-built_in">sudo</span> <span class="hljs-built_in">rm</span> -rf /usr/local/cuda-12.6<br><br><span class="hljs-comment"># 确认软链接指向</span><br><span class="hljs-built_in">ls</span> -l /usr/local/cuda<br><br><span class="hljs-comment"># 如需要，重新创建软链接</span><br><span class="hljs-built_in">sudo</span> <span class="hljs-built_in">rm</span> /usr/local/cuda<br><span class="hljs-built_in">sudo</span> <span class="hljs-built_in">ln</span> -s /usr/local/cuda-12.4 /usr/local/cuda<br></code></pre></td></tr></table></figure></li><li><p><strong>安装过程中的常见问题</strong></p><p>a. 权限相关：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 下载文件时的权限错误</span><br><span class="hljs-built_in">sudo</span> wget [URL] <span class="hljs-comment"># 使用sudo解决</span><br></code></pre></td></tr></table></figure><p>b. 驱动冲突（比较麻烦，不建议首选）：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 检查现有驱动</span><br>nvidia-smi<br><br><span class="hljs-comment"># 如果需要重新安装驱动</span><br><span class="hljs-built_in">sudo</span> apt-get --purge remove <span class="hljs-string">'*nvidia*'</span><br></code></pre></td></tr></table></figure><p>c. 安装失败排查：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 查看安装日志</span><br>less /var/log/cuda-installer.log<br><br><span class="hljs-comment"># 查看驱动安装日志</span><br>less /var/log/nvidia-installer.log<br></code></pre></td></tr></table></figure></li></ol><h3 id="2-3-环境变量配置与验证"><a href="#2-3-环境变量配置与验证" class="headerlink" title="2.3 环境变量配置与验证"></a>2.3 环境变量配置与验证</h3><ol><li><p><strong>CUDA环境变量设置</strong></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 编辑~/.bashrc文件</span><br>vim ~/.bashrc<br><br><span class="hljs-comment"># 添加以下内容</span><br><span class="hljs-built_in">export</span> PATH=/usr/local/cuda-12.4/bin<span class="hljs-variable">${PATH:+:<span class="hljs-variable">${PATH}</span>}</span><br><span class="hljs-built_in">export</span> LD_LIBRARY_PATH=/usr/local/cuda-12.4/lib64<span class="hljs-variable">${LD_LIBRARY_PATH:+:<span class="hljs-variable">${LD_LIBRARY_PATH}</span>}</span><br><br><span class="hljs-comment"># 使设置生效</span><br><span class="hljs-built_in">source</span> ~/.bashrc<br></code></pre></td></tr></table></figure></li><li><p><strong>CUDA版本验证与分析</strong></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 检查CUDA编译器版本</span><br>nvcc -V<br><br><span class="hljs-comment"># 检查NVIDIA驱动和CUDA支持</span><br>nvidia-smi<br></code></pre></td></tr></table></figure><p>版本不一致分析：</p><ul><li><code>nvcc -V</code>显示实际安装的CUDA工具链版本</li><li><code>nvidia-smi</code>显示驱动支持的最高CUDA版本</li><li>开发时以<code>nvcc -V</code>显示的版本为准</li></ul></li></ol><h3 id="2-4-深度学习框架安装与GPU验证"><a href="#2-4-深度学习框架安装与GPU验证" class="headerlink" title="2.4 深度学习框架安装与GPU验证"></a>2.4 深度学习框架安装与GPU验证</h3><ol><li><p><strong>创建环境并安装框架</strong></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 创建conda环境</span><br>conda create -n dl_env python=3.11<br>conda activate dl_env<br><br><span class="hljs-comment"># 安装TensorFlow</span><br>pip install tensorflow[and-cuda]<br><br><span class="hljs-comment"># 安装PyTorch（CUDA 12.4）</span><br>pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124<br></code></pre></td></tr></table></figure></li><li><p><strong>TensorFlow GPU验证</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> tensorflow <span class="hljs-keyword">as</span> tf<br><br><span class="hljs-comment"># 检查GPU设备</span><br><span class="hljs-built_in">print</span>(tf.config.list_physical_devices(<span class="hljs-string">'GPU'</span>))<br><br><span class="hljs-comment"># GPU计算测试</span><br>A = tf.random.normal([<span class="hljs-number">10</span>, <span class="hljs-number">10</span>])<br>B = tf.random.normal([<span class="hljs-number">10</span>, <span class="hljs-number">10</span>])<br>C = tf.matmul(A, B)<br><span class="hljs-built_in">print</span>(tf.reduce_sum(C))<br></code></pre></td></tr></table></figure><p>输出分析：</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs routeros">[PhysicalDevice(<span class="hljs-attribute">name</span>=<span class="hljs-string">'/physical_device:GPU:0'</span>, <span class="hljs-attribute">device_type</span>=<span class="hljs-string">'GPU'</span>)]<br></code></pre></td></tr></table></figure><p>表示成功检测到GPU设备</p><p>常见警告说明：</p><ul><li>cuFFT/cuDNN/cuBLAS插件重复注册：不影响功能</li><li>CPU指令集优化警告：仅影响CPU计算性能</li></ul></li><li><p><strong>PyTorch GPU验证</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> torch<br><br><span class="hljs-comment"># 检查CUDA是否可用</span><br><span class="hljs-built_in">print</span>(torch.cuda.is_available())<br><br><span class="hljs-comment"># 查看可用的GPU数量</span><br><span class="hljs-built_in">print</span>(torch.cuda.device_count())<br><br><span class="hljs-comment"># 查看当前GPU设备名称</span><br><span class="hljs-built_in">print</span>(torch.cuda.get_device_name(<span class="hljs-number">0</span>))<br></code></pre></td></tr></table></figure></li></ol><h2 id="3-参考资源"><a href="#3-参考资源" class="headerlink" title="3. 参考资源"></a>3. 参考资源</h2><ol><li><p><strong>官方文档</strong></p><ul><li><a href="https://www.tensorflow.org/install">TensorFlow安装指南</a></li><li><a href="https://pytorch.org/get-started/locally/">PyTorch安装指南</a></li><li><a href="https://developer.nvidia.com/cuda-toolkit">NVIDIA CUDA Toolkit文档</a></li><li><strong>注意文档语言调成英文，因为中英文的官方文档更新并不同步</strong></li></ul></li><li><p><strong>软件下载</strong></p><ul><li>CUDA Toolkit：<a href="https://developer.nvidia.com/cuda-downloads">https://developer.nvidia.com/cuda-downloads</a></li><li>cuDNN：<a href="https://developer.nvidia.com/cudnn">https://developer.nvidia.com/cudnn</a></li><li>NVIDIA驱动：<a href="https://www.nvidia.com/Download/index.aspx">https://www.nvidia.com/Download/index.aspx</a></li></ul></li></ol>]]></content>
<categories>
<category>ai chat</category>
</categories>
</entry>
<entry>
<title>PC端配置ONNX和TensorFlow GPU环境学习笔记 - 实践版</title>
<link href="/2024/11/19/complete-revised-notes/"/>
<url>/2024/11/19/complete-revised-notes/</url>
<content type="html"><![CDATA[<h1 id="PC端配置ONNX和TensorFlow-GPU环境学习笔记-实践版"><a href="#PC端配置ONNX和TensorFlow-GPU环境学习笔记-实践版" class="headerlink" title="PC端配置ONNX和TensorFlow GPU环境学习笔记 - 实践版"></a>PC端配置ONNX和TensorFlow GPU环境学习笔记 - 实践版</h1><h2 id="1-深度学习模型格式与部署实践"><a href="#1-深度学习模型格式与部署实践" class="headerlink" title="1. 深度学习模型格式与部署实践"></a>1. 深度学习模型格式与部署实践</h2><h3 id="1-1-在PC端运行ONNX模型的主要方法"><a href="#1-1-在PC端运行ONNX模型的主要方法" class="headerlink" title="1.1 在PC端运行ONNX模型的主要方法"></a>1.1 在PC端运行ONNX模型的主要方法</h3><ol><li><p><strong>ONNX Runtime</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> onnxruntime <span class="hljs-keyword">as</span> ort<br><br><span class="hljs-comment"># 加载模型</span><br>session = ort.InferenceSession(<span class="hljs-string">"model.onnx"</span>)<br><br><span class="hljs-comment"># 获取输入名称</span><br>input_name = session.get_inputs()[<span class="hljs-number">0</span>].name<br><br><span class="hljs-comment"># 准备输入数据并推理</span><br>inputs = {input_name: input_data}<br>results = session.run(<span class="hljs-literal">None</span>, inputs)<br></code></pre></td></tr></table></figure><ul><li>优势：开源、高性能、使用简单</li><li>支持多平台：Windows、Linux、macOS</li><li>适合快速部署和测试场景</li></ul></li><li><p><strong>OpenVINO（Intel平台）</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> openvino.runtime <span class="hljs-keyword">import</span> Core<br><br><span class="hljs-comment"># 加载并编译模型</span><br>core = Core()<br>model = core.read_model(<span class="hljs-string">"model.onnx"</span>)<br>compiled_model = core.compile_model(model, <span class="hljs-string">"CPU"</span>)<br><br><span class="hljs-comment"># 进行推理</span><br>result = compiled_model.infer_new_request({<span class="hljs-string">"input_name"</span>: input_data})<br></code></pre></td></tr></table></figure><ul><li>适合Intel硬件平台</li><li>对CPU进行了优化</li><li>支持多种设备（CPU、GPU、FPGA等）</li></ul></li><li><p><strong>TensorRT（NVIDIA平台）</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> tensorrt <span class="hljs-keyword">as</span> trt<br><span class="hljs-keyword">import</span> pycuda.driver <span class="hljs-keyword">as</span> cuda<br><span class="hljs-keyword">import</span> pycuda.autoinit<br><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np<br><br><span class="hljs-comment"># 创建TensorRT引擎</span><br>TRT_LOGGER = trt.Logger(trt.Logger.WARNING)<br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">"model.engine"</span>, <span class="hljs-string">"rb"</span>) <span class="hljs-keyword">as</span> f, trt.Runtime(TRT_LOGGER) <span class="hljs-keyword">as</span> runtime:<br> engine = runtime.deserialize_cuda_engine(f.read())<br><br><span class="hljs-comment"># 创建执行上下文</span><br>context = engine.create_execution_context()<br><br><span class="hljs-comment"># 分配GPU内存</span><br>d_input = cuda.mem_alloc(input_data.nbytes)<br>d_output = cuda.mem_alloc(engine.get_binding_shape(<span class="hljs-number">1</span>).size * input_data.dtype.itemsize)<br>bindings = [<span class="hljs-built_in">int</span>(d_input), <span class="hljs-built_in">int</span>(d_output)]<br><br><span class="hljs-comment"># 拷贝数据到GPU并执行推理</span><br>cuda.memcpy_htod(d_input, input_data)<br>context.execute_v2(bindings)<br><br><span class="hljs-comment"># 获取结果</span><br>output = np.empty(engine.get_binding_shape(<span class="hljs-number">1</span>), dtype=np.float32)<br>cuda.memcpy_dtoh(output, d_output)<br></code></pre></td></tr></table></figure><ul><li>针对NVIDIA GPU优化</li><li>支持模型量化和优化</li><li>性能最优但配置较复杂</li></ul></li></ol><h3 id="1-2-模型格式对比（基于实践经验）"><a href="#1-2-模型格式对比（基于实践经验）" class="headerlink" title="1.2 模型格式对比（基于实践经验）"></a>1.2 模型格式对比（基于实践经验）</h3><ol><li><p><strong>H5格式（Keras原生格式）</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 加载H5模型</span><br><span class="hljs-keyword">import</span> tensorflow <span class="hljs-keyword">as</span> tf<br>model = tf.keras.models.load_model(<span class="hljs-string">'model.h5'</span>)<br><br><span class="hljs-comment"># 进行预测</span><br>predictions = model.predict(input_data)<br></code></pre></td></tr></table></figure><ul><li>保持完整的模型信息和训练状态</li><li>适合继续训练和模型调优</li><li>支持TensorFlow的所有功能</li><li>在<strong>Windows</strong>上可能需要配置GPU环境</li></ul></li><li><p><strong>ONNX格式</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 使用ONNX Runtime推理</span><br><span class="hljs-keyword">import</span> onnxruntime <span class="hljs-keyword">as</span> ort<br><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np<br><br><span class="hljs-comment"># 创建推理会话</span><br>session = ort.InferenceSession(<span class="hljs-string">"model.onnx"</span>, <br> providers=[<span class="hljs-string">'CUDAExecutionProvider'</span>, <br> <span class="hljs-string">'CPUExecutionProvider'</span>])<br><br><span class="hljs-comment"># 获取输入输出名称</span><br>input_name = session.get_inputs()[<span class="hljs-number">0</span>].name<br>output_name = session.get_outputs()[<span class="hljs-number">0</span>].name<br><br><span class="hljs-comment"># 准备输入数据</span><br>input_data = np.array([your_input_data], dtype=np.float32)<br><br><span class="hljs-comment"># 运行推理</span><br>results = session.run([output_name], {input_name: input_data})<br></code></pre></td></tr></table></figure><ul><li>跨平台兼容性好</li><li>可以使用多种推理引擎</li><li>部署更加灵活</li><li>可能会有轻微的<strong>精度损失</strong></li></ul></li></ol><h3 id="1-3-模型部署到嵌入式设备的实践要点"><a href="#1-3-模型部署到嵌入式设备的实践要点" class="headerlink" title="1.3 模型部署到嵌入式设备的实践要点"></a>1.3 模型部署到嵌入式设备的实践要点</h3><ol><li><p><strong>模型转换注意事项</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 从H5转换到ONNX</span><br><span class="hljs-keyword">import</span> keras2onnx<br><span class="hljs-keyword">import</span> tf2onnx<br><br><span class="hljs-comment"># 方法1：使用keras2onnx</span><br>onnx_model = keras2onnx.convert_keras(model, model.name)<br>keras2onnx.save_model(onnx_model, <span class="hljs-string">'model.onnx'</span>)<br><br><span class="hljs-comment"># 方法2：使用tf2onnx</span><br><span class="hljs-keyword">import</span> tensorflow <span class="hljs-keyword">as</span> tf<br><br><span class="hljs-comment"># 保存为SavedModel格式</span><br>tf.saved_model.save(model, <span class="hljs-string">"saved_model_path"</span>)<br><br><span class="hljs-comment"># 转换为ONNX</span><br><span class="hljs-comment"># 使用命令行：</span><br><span class="hljs-comment"># python -m tf2onnx.convert --saved-model saved_model_path --output model.onnx</span><br></code></pre></td></tr></table></figure></li><li><p><strong>模型优化和验证</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># ONNX模型验证</span><br><span class="hljs-keyword">import</span> onnx<br><br><span class="hljs-comment"># 加载并检查模型</span><br>model = onnx.load(<span class="hljs-string">"model.onnx"</span>)<br>onnx.checker.check_model(model)<br><br><span class="hljs-comment"># 查看模型输入输出信息</span><br><span class="hljs-keyword">for</span> <span class="hljs-built_in">input</span> <span class="hljs-keyword">in</span> model.graph.<span class="hljs-built_in">input</span>:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">f"Input: <span class="hljs-subst">{<span class="hljs-built_in">input</span>.name}</span>, Shape: <span class="hljs-subst">{<span class="hljs-built_in">input</span>.<span class="hljs-built_in">type</span>.tensor_type.shape}</span>"</span>)<br><span class="hljs-keyword">for</span> output <span class="hljs-keyword">in</span> model.graph.output:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">f"Output: <span class="hljs-subst">{output.name}</span>, Shape: <span class="hljs-subst">{output.<span class="hljs-built_in">type</span>.tensor_type.shape}</span>"</span>)<br></code></pre></td></tr></table></figure></li></ol><h2 id="2-TensorFlow-GPU环境配置实践"><a href="#2-TensorFlow-GPU环境配置实践" class="headerlink" title="2. TensorFlow GPU环境配置实践"></a>2. TensorFlow GPU环境配置实践</h2><h3 id="2-1-Windows环境配置及问题解决"><a href="#2-1-Windows环境配置及问题解决" class="headerlink" title="2.1 Windows环境配置及问题解决"></a>2.1 Windows环境配置及问题解决</h3><h4 id="重要版本说明"><a href="#重要版本说明" class="headerlink" title="重要版本说明"></a>重要版本说明</h4><ul><li>从TensorFlow 2.11.0开始，Windows平台不再支持原生的CUDA构建</li><li>TensorFlow 2.10.0是最后一个在Windows上原生支持GPU的版本<ul><li>必须搭配CUDA 11.2</li><li>必须搭配cuDNN 8.1</li><li>不要使用更高版本的CUDA（如11.8），因为不兼容</li></ul></li></ul><h4 id="Windows下使用GPU的选择"><a href="#Windows下使用GPU的选择" class="headerlink" title="Windows下使用GPU的选择"></a>Windows下使用GPU的选择</h4><ol><li><p>使用TensorFlow 2.10.0（推荐方案，如需原生Windows支持）</p></li><li><p>使用WSL2运行更新版本的TensorFlow</p></li><li><p>使用tensorflow-cpu配合TensorFlow-DirectML-Plugin</p></li><li><p><strong>环境准备</strong></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 创建新的conda环境</span><br>conda create -n tf_gpu python=3.8<br>conda activate tf_gpu<br><br><span class="hljs-comment"># 安装TensorFlow</span><br>pip install tensorflow==2.10.0<br><br><span class="hljs-comment"># 安装匹配版本的依赖</span><br>pip install keras==2.10.0<br>pip install tensorboard==2.10.0<br>pip install tensorflow-estimator==2.10.0<br>pip install tensorflow-intel==2.10.0<br></code></pre></td></tr></table></figure></li><li><p><strong>版本冲突解决</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 检查已安装包的版本</span><br>pip <span class="hljs-built_in">list</span> | findstr <span class="hljs-string">"tensorflow keras tensorboard tensorflow-estimator"</span><br><br><span class="hljs-comment"># 降级冲突包</span><br>pip install package_name==specific_version<br><br><span class="hljs-comment"># 例如：降级 werkzeug</span><br>pip install werkzeug==<span class="hljs-number">0.16</span><span class="hljs-number">.1</span><br></code></pre></td></tr></table></figure></li><li><p><strong>GPU检测与验证</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> tensorflow <span class="hljs-keyword">as</span> tf<br><br><span class="hljs-comment"># 检查TensorFlow版本</span><br><span class="hljs-built_in">print</span>(tf.__version__)<br><br><span class="hljs-comment"># 列出可用的GPU设备</span><br><span class="hljs-built_in">print</span>(tf.config.list_physical_devices(<span class="hljs-string">'GPU'</span>))<br><br><span class="hljs-comment"># 检查CUDA是否可用</span><br><span class="hljs-built_in">print</span>(tf.test.is_built_with_cuda())<br><span class="hljs-built_in">print</span>(tf.test.is_gpu_available())<br><br><span class="hljs-comment"># 查看设备信息</span><br><span class="hljs-keyword">from</span> tensorflow.python.client <span class="hljs-keyword">import</span> device_lib<br><span class="hljs-built_in">print</span>(device_lib.list_local_devices())<br></code></pre></td></tr></table></figure></li></ol><h3 id="2-2-Ubuntu环境配置（推荐）"><a href="#2-2-Ubuntu环境配置（推荐）" class="headerlink" title="2.2 Ubuntu环境配置（推荐）"></a>2.2 Ubuntu环境配置（推荐）</h3><p>详情见<a href="https://blakehansen130.github.io/2024/11/19/comprehensive-learning-notes/">《Python虚拟环境与深度学习环境配置详细笔记》</a>，最终因为windows使用tensorflow太麻烦因此还是选择了ubuntu。</p><h3 id="2-3-故障排除和性能优化"><a href="#2-3-故障排除和性能优化" class="headerlink" title="2.3 故障排除和性能优化"></a>2.3 故障排除和性能优化</h3><ol><li><p><strong>常见问题解决方案</strong></p><ul><li>内存不足：启用显存动态增长</li><li>驱动程序问题：更新到最新的NVIDIA驱动</li><li>环境变量问题：检查CUDA和cuDNN路径配置</li></ul></li><li><p><strong>性能监控</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># GPU使用情况监控</span><br><span class="hljs-keyword">import</span> GPUtil<br><span class="hljs-keyword">import</span> time<br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">monitor_gpu</span>():<br> <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>:<br> GPUtil.showUtilization()<br> time.sleep(<span class="hljs-number">1</span>)<br><br><span class="hljs-comment"># 在另一个线程中运行监控</span><br><span class="hljs-keyword">from</span> threading <span class="hljs-keyword">import</span> Thread<br>Thread(target=monitor_gpu).start()<br></code></pre></td></tr></table></figure></li></ol><h2 id="3-实践经验总结"><a href="#3-实践经验总结" class="headerlink" title="3. 实践经验总结"></a>3. 实践经验总结</h2><ol><li><p><strong>Windows环境问题</strong></p><ul><li>版本选择很关键：<ul><li>TensorFlow 2.10.0 + CUDA 11.2 + cuDNN 8.1是Windows原生GPU支持的最后组合</li><li>2.11.0及以上版本需要使用WSL2或DirectML方案</li></ul></li><li>环境变量配置复杂</li><li>依赖冲突频繁</li><li>调试和错误信息不清晰</li></ul></li><li><p><strong>Ubuntu优势</strong></p><ul><li>CUDA工具链集成度高</li><li>环境配置更简单</li></ul></li><li><p><strong>开发建议</strong></p><ul><li>记录详细的环境搭建步骤</li><li>定期备份工作环境</li><li>优先选择Ubuntu进行深度学习开发</li></ul></li></ol><hr><p><em>注：本笔记基于NVIDIA显卡在Windows 11和Ubuntu系统上的实际配置经验，包含了常见问题的解决方案和实用代码示例。最近更新：补充了TensorFlow版本与CUDA兼容性的重要说明。</em></p>]]></content>
<categories>
<category>ai chat</category>
</categories>
</entry>
<entry>
<title>v2rayN使用笔记</title>
<link href="/2024/11/19/v2rayn-notes/"/>
<url>/2024/11/19/v2rayn-notes/</url>
<content type="html"><![CDATA[<h1 id="v2rayN使用笔记"><a href="#v2rayN使用笔记" class="headerlink" title="v2rayN使用笔记"></a>v2rayN使用笔记</h1><h2 id="1-三种代理模式详解"><a href="#1-三种代理模式详解" class="headerlink" title="1. 三种代理模式详解"></a>1. 三种代理模式详解</h2><h3 id="1-1-V3-绕过大陆-Whitelist"><a href="#1-1-V3-绕过大陆-Whitelist" class="headerlink" title="1.1 V3-绕过大陆 (Whitelist)"></a>1.1 V3-绕过大陆 (Whitelist)</h3><ul><li><strong>适用场景</strong>：主要访问国内网站，偶尔需要访问被屏蔽的网站</li><li><strong>工作原理</strong>：只有特定目标（如被墙网站）走代理，其它走本地网络</li><li><strong>优点</strong>：国内网站访问速度快，按需代理节省资源</li></ul><h3 id="1-2-V3-黑名单-Blacklist"><a href="#1-2-V3-黑名单-Blacklist" class="headerlink" title="1.2 V3-黑名单 (Blacklist)"></a>1.2 V3-黑名单 (Blacklist)</h3><ul><li><strong>适用场景</strong>：需要代理访问大部分网站</li><li><strong>工作原理</strong>：默认全部代理，只有明确指定的走直连</li><li><strong>实际体验</strong>：可能导致某些网站无法访问<del>，如18comic.vip</del></li></ul><h3 id="1-3-V3-全局-Global"><a href="#1-3-V3-全局-Global" class="headerlink" title="1.3 V3-全局 (Global)"></a>1.3 V3-全局 (Global)</h3><ul><li><strong>适用场景</strong>：需要全程加密或不想复杂配置</li><li><strong>工作原理</strong>：所有流量都走代理服务器</li><li><strong>注意事项</strong>：可能影响访问速度，消耗更多代理流量</li></ul><h2 id="2-规则配置详解"><a href="#2-规则配置详解" class="headerlink" title="2. 规则配置详解"></a>2. 规则配置详解</h2><h3 id="2-1-基础规则设置步骤"><a href="#2-1-基础规则设置步骤" class="headerlink" title="2.1 基础规则设置步骤"></a>2.1 基础规则设置步骤</h3><ol><li>打开v2rayN主界面</li><li>点击”规则”按钮</li><li>点击”添加规则”</li></ol><h3 id="2-2-规则配置关键字段"><a href="#2-2-规则配置关键字段" class="headerlink" title="2.2 规则配置关键字段"></a>2.2 规则配置关键字段</h3><ul><li><strong>outboundTag</strong>: <ul><li><code>direct</code>: 直连</li><li><code>proxy</code>: 代理</li></ul></li><li><strong>Domain</strong>: 域名规则格式<ul><li>单个域名：<code>domain:example.com</code></li><li>多个域名：用逗号分隔</li></ul></li></ul><h3 id="2-3-常用直连规则示例"><a href="#2-3-常用直连规则示例" class="headerlink" title="2.3 常用直连规则示例"></a>2.3 常用直连规则示例</h3><figure class="highlight avrasm"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs avrasm"><span class="hljs-meta"># AI相关域名</span><br><span class="hljs-symbol">domain:</span>dawuai.buzz<br><span class="hljs-symbol">domain:</span>dwai.world<br><span class="hljs-symbol">domain:</span>claudesvip.top<br><span class="hljs-symbol">domain:</span>fuclaude.com<br><span class="hljs-symbol">domain:</span>files.closeai.biz<br><br><span class="hljs-meta"># 常用CDN</span><br><span class="hljs-symbol">domain:</span>cdnjs.cloudflare.com<br><span class="hljs-symbol">domain:</span>code.jquery.com<br><span class="hljs-symbol">domain:</span>s-cdn.anthropic.com<br><br><span class="hljs-meta"># dwai相关</span><br><span class="hljs-symbol">domain:</span>france.dwai.work<br><span class="hljs-symbol">domain:</span>transfer.dwai.work<br></code></pre></td></tr></table></figure><h3 id="2-4-规则优先级"><a href="#2-4-规则优先级" class="headerlink" title="2.4 规则优先级"></a>2.4 规则优先级</h3><ul><li>特定域名规则要放在”最终代理”规则之前</li><li>规则从上往下匹配，第一个匹配的规则生效</li></ul><h2 id="3-代理状态检查方法"><a href="#3-代理状态检查方法" class="headerlink" title="3. 代理状态检查方法"></a>3. 代理状态检查方法</h2><h3 id="3-1-通过v2rayN日志查看"><a href="#3-1-通过v2rayN日志查看" class="headerlink" title="3.1 通过v2rayN日志查看"></a>3.1 通过v2rayN日志查看</h3><ol><li><p><strong>查看日志步骤</strong>：</p><ul><li>打开v2rayN主界面</li><li>点击”设置”（齿轮图标）</li><li>确保日志级别选择info或warning</li><li>点击主界面的”查看日志”按钮</li></ul></li><li><p><strong>日志解读</strong>：</p><ul><li><code>[http -> direct]</code> = 直连访问</li><li><code>[http -> proxy]</code> = 通过代理</li></ul></li></ol><h3 id="3-2-通过IP查询网站验证"><a href="#3-2-通过IP查询网站验证" class="headerlink" title="3.2 通过IP查询网站验证"></a>3.2 通过IP查询网站验证</h3><p><strong>常用IP查询网站</strong>：</p><ul><li><a href="https://ipinfo.io/">https://ipinfo.io/</a></li><li><a href="https://whatismyipaddress.com/">https://whatismyipaddress.com/</a></li><li><a href="https://whoer.net/">https://whoer.net/</a></li></ul><h3 id="3-3-使用浏览器开发者工具"><a href="#3-3-使用浏览器开发者工具" class="headerlink" title="3.3 使用浏览器开发者工具"></a>3.3 使用浏览器开发者工具</h3><ol><li>打开方式：F12或右键检查</li><li>切换到Network标签</li><li>刷新页面</li><li>查看请求的Headers信息<ul><li>可能包含代理相关信息如X-Forwarded-For</li><li>响应头中可能显示服务器位置</li></ul></li></ol><h2 id="4-常见问题与解决"><a href="#4-常见问题与解决" class="headerlink" title="4. 常见问题与解决"></a>4. 常见问题与解决</h2><h3 id="4-1-网站访问问题"><a href="#4-1-网站访问问题" class="headerlink" title="4.1 网站访问问题"></a>4.1 网站访问问题</h3><p><strong>问题</strong>：某些网站在黑名单模式下无法访问</p><p><strong>解决方案</strong>：</p><ol><li>检查并调整路由规则</li><li>确认DNS设置是否正确</li><li>尝试切换到其他代理模式</li><li>检查域名是否被错误归类（如广告类）</li></ol><h3 id="4-2-直连设置注意事项"><a href="#4-2-直连设置注意事项" class="headerlink" title="4.2 直连设置注意事项"></a>4.2 直连设置注意事项</h3><ol><li><p><strong>保存配置</strong>：</p><ul><li>添加规则后必须点击”确定”保存</li><li>确保在主界面选择正确的模式</li></ul></li><li><p><strong>DNS设置</strong>：</p><ul><li>可以尝试使用不同的DNS服务</li><li>常用DNS：<ul><li>Google: 8.8.8.8</li><li>Cloudflare: 1.1.1.1</li><li>阿里: 223.5.5.5</li></ul></li></ul></li><li><p><strong>配置验证</strong>：</p><ul><li>添加规则后通过日志验证效果</li><li>必要时清除浏览器DNS缓存</li></ul></li></ol><h2 id="5-实用建议"><a href="#5-实用建议" class="headerlink" title="5. 实用建议"></a>5. 实用建议</h2><h3 id="5-1-模式选择"><a href="#5-1-模式选择" class="headerlink" title="5.1 模式选择"></a>5.1 模式选择</h3><ul><li>推荐使用V3-绕过大陸模式</li><li>按需配置直连规则</li><li>避免使用全局模式以节省资源</li></ul><h3 id="5-2-故障排查"><a href="#5-2-故障排查" class="headerlink" title="5.2 故障排查"></a>5.2 故障排查</h3><ol><li>先检查日志确认流量走向</li><li>测试不同模式下的表现</li><li>必要时可临时切换到全局模式测试</li></ol><h2 id="6-补充说明"><a href="#6-补充说明" class="headerlink" title="6. 补充说明"></a>6. 补充说明</h2><ul><li>这些规则和设置可能需要根据实际使用情况和网络环境进行调整</li><li>建议定期检查规则是否仍然有效</li><li>在遇到问题时可以参考此文档快速排查</li></ul>]]></content>
<categories>
<category>ai chat</category>
</categories>
</entry>
<entry>
<title>Ubuntu下深度学习环境配置历程和经验总结</title>
<link href="/2024/11/19/Summarizing%20AI%20Assistant%20Conversation%20for%20Notes%20and%20Troubleshooting_2024-11-19T08-38-12/"/>
<url>/2024/11/19/Summarizing%20AI%20Assistant%20Conversation%20for%20Notes%20and%20Troubleshooting_2024-11-19T08-38-12/</url>
<content type="html"><![CDATA[<h1 id="Ubuntu下深度学习环境配置历程和经验总结"><a href="#Ubuntu下深度学习环境配置历程和经验总结" class="headerlink" title="Ubuntu下深度学习环境配置历程和经验总结"></a>Ubuntu下深度学习环境配置历程和经验总结</h1><h4 id="一、CUDA安装和配置"><a href="#一、CUDA安装和配置" class="headerlink" title="一、CUDA安装和配置"></a>一、CUDA安装和配置</h4><h5 id="1-1-检查现有CUDA版本"><a href="#1-1-检查现有CUDA版本" class="headerlink" title="1.1 检查现有CUDA版本"></a>1.1 检查现有CUDA版本</h5><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash">nvcc -V<br></code></pre></td></tr></table></figure><p>在我的系统中显示是CUDA 12.4，而最新版本是12.6，因此想要更新。</p><h5 id="1-2-CUDA安装选项"><a href="#1-2-CUDA安装选项" class="headerlink" title="1.2 CUDA安装选项"></a>1.2 CUDA安装选项</h5><p>安装CUDA 12.6时遇到两个选项：</p><ol><li><p>Open Kernel Module (<code>nvidia-open</code>)：</p><ul><li>NVIDIA最近推出的开源驱动模块</li><li>适合较新的显卡(Ampere及以后架构)</li><li>对开发人员友好，与新Linux内核兼容性好</li></ul></li><li><p>Legacy Kernel Module (<code>cuda-drivers</code>)：</p><ul><li>NVIDIA传统的闭源驱动</li><li>更加稳定和成熟</li><li>与深度学习框架兼容性更好</li></ul></li></ol><p><strong>选择建议</strong>：对于深度学习应用，建议选择Legacy Kernel Module，因为兼容性更好。</p><h5 id="1-3-CUDA路径问题"><a href="#1-3-CUDA路径问题" class="headerlink" title="1.3 CUDA路径问题"></a>1.3 CUDA路径问题</h5><p>安装完成后发现系统中存在多个CUDA相关目录：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs bash">/usr/local/cuda -> /etc/alternatives/cuda<br>/usr/local/cuda-12 -> /etc/alternatives/cuda-12<br>/usr/local/cuda-12.6<br></code></pre></td></tr></table></figure><p>每个目录占用约5.3G空间，通过检查发现这些是符号链接，都指向了alternatives系统。虽然链接正确指向了CUDA 12.6，但<code>nvcc -V</code>仍显示12.4版本，这是因为cuda的默认启动被设置成了conda的相关目录，需要移除并像下文中重新添加系统路径。</p><h5 id="1-4-环境变量配置"><a href="#1-4-环境变量配置" class="headerlink" title="1.4 环境变量配置"></a>1.4 环境变量配置</h5><p>在<code>~/.bashrc</code>中添加CUDA路径：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-built_in">export</span> PATH=/usr/local/cuda-12.6/bin:<span class="hljs-variable">$PATH</span><br><span class="hljs-built_in">export</span> LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64:<span class="hljs-variable">$LD_LIBRARY_PATH</span><br></code></pre></td></tr></table></figure><h4 id="二、PyTorch配置问题"><a href="#二、PyTorch配置问题" class="headerlink" title="二、PyTorch配置问题"></a>二、PyTorch配置问题</h4><h5 id="2-1-初始问题"><a href="#2-1-初始问题" class="headerlink" title="2.1 初始问题"></a>2.1 初始问题</h5><p>安装PyTorch后，GPU检测失败：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs python">torch.cuda.is_available() <span class="hljs-comment"># 返回False</span><br></code></pre></td></tr></table></figure><p>错误信息：</p><figure class="highlight subunit"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs subunit">UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount()<br><span class="hljs-keyword">Error </span>804: forward compatibility was attempted on non supported HW<br></code></pre></td></tr></table></figure><p>这表明CUDA和GPU驱动存在兼容性问题。</p><h5 id="2-2-验证PyTorch-CUDA支持"><a href="#2-2-验证PyTorch-CUDA支持" class="headerlink" title="2.2 验证PyTorch CUDA支持"></a>2.2 验证PyTorch CUDA支持</h5><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> torch<br><span class="hljs-built_in">print</span>(torch.cuda.is_available())<br><span class="hljs-built_in">print</span>(torch.version.cuda)<br></code></pre></td></tr></table></figure><h4 id="三、TensorFlow配置问题"><a href="#三、TensorFlow配置问题" class="headerlink" title="三、TensorFlow配置问题"></a>三、TensorFlow配置问题</h4><h5 id="3-1-NumPy版本冲突"><a href="#3-1-NumPy版本冲突" class="headerlink" title="3.1 NumPy版本冲突"></a>3.1 NumPy版本冲突</h5><p>安装PyTorch成功后，安装TensorFlow后遇到NumPy版本冲突：</p><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">ImportError</span>: A module that was compiled using NumPy <span class="hljs-number">1</span>.x cannot be run in NumPy <span class="hljs-number">2</span>.<span class="hljs-number">0</span><br></code></pre></td></tr></table></figure><p>尝试降级NumPy时又遇到新问题：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs bash">pip install numpy==1.22.4 <span class="hljs-comment"># 失败</span><br><span class="hljs-comment">### 错误：AttributeError: module pkgutil has no attribute 'ImpImporter'</span><br></code></pre></td></tr></table></figure><h5 id="3-2-验证TensorFlow-GPU支持"><a href="#3-2-验证TensorFlow-GPU支持" class="headerlink" title="3.2 验证TensorFlow GPU支持"></a>3.2 验证TensorFlow GPU支持</h5><p>两种验证方法：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment">### 方法1：检查GPU设备</span><br><span class="hljs-keyword">import</span> tensorflow <span class="hljs-keyword">as</span> tf<br><span class="hljs-built_in">print</span>(tf.config.list_physical_devices(<span class="hljs-string">'GPU'</span>))<br><br><span class="hljs-comment">### 方法2：实际运行GPU计算（更可靠）</span><br><span class="hljs-keyword">import</span> tensorflow <span class="hljs-keyword">as</span> tf<br><span class="hljs-keyword">with</span> tf.device(<span class="hljs-string">'/GPU:0'</span>):<br> a = tf.constant([[<span class="hljs-number">1.0</span>, <span class="hljs-number">2.0</span>, <span class="hljs-number">3.0</span>], [<span class="hljs-number">4.0</span>, <span class="hljs-number">5.0</span>, <span class="hljs-number">6.0</span>]])<br> b = tf.constant([[<span class="hljs-number">1.0</span>, <span class="hljs-number">2.0</span>], [<span class="hljs-number">3.0</span>, <span class="hljs-number">4.0</span>], [<span class="hljs-number">5.0</span>, <span class="hljs-number">6.0</span>]])<br> c = tf.matmul(a, b)<br> <span class="hljs-built_in">print</span>(c)<br></code></pre></td></tr></table></figure><h4 id="四、环境清理"><a href="#四、环境清理" class="headerlink" title="四、环境清理"></a>四、环境清理</h4><h5 id="4-1-遇到的问题"><a href="#4-1-遇到的问题" class="headerlink" title="4.1 遇到的问题"></a>4.1 遇到的问题</h5><ol><li>base环境中的包依赖关系复杂</li><li>Python 3.12版本过新，很多包不支持</li><li>多个conda环境占用大量空间</li></ol><h5 id="4-2-清理步骤"><a href="#4-2-清理步骤" class="headerlink" title="4.2 清理步骤"></a>4.2 清理步骤</h5><ol><li><p>删除conda环境：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash">conda remove --name env_name --all<br></code></pre></td></tr></table></figure></li><li><p>清理conda缓存：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash">conda clean --all<br></code></pre></td></tr></table></figure></li><li><p>对于无法识别的环境，需要手动删除：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-built_in">rm</span> -rf /path/to/env<br></code></pre></td></tr></table></figure></li></ol><h4 id="五、经验教训"><a href="#五、经验教训" class="headerlink" title="五、经验教训"></a>五、经验教训</h4><h5 id="5-1-版本选择"><a href="#5-1-版本选择" class="headerlink" title="5.1 版本选择"></a>5.1 版本选择</h5><ol><li><p>Python版本：</p><ul><li>深度学习环境建议使用Python 3.10-3.11</li><li>避免使用最新的Python版本（如3.12）</li></ul></li><li><p>CUDA版本：</p><ul><li>确保CUDA版本与GPU驱动兼容</li><li>注意检查深度学习框架对CUDA版本的要求</li></ul></li></ol><h5 id="5-2-最佳实践"><a href="#5-2-最佳实践" class="headerlink" title="5.2 最佳实践"></a>5.2 最佳实践</h5><ol><li><p>使用独立的conda环境：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment">### 创建新环境</span><br>conda create -n dl_env python=3.10<br><br><span class="hljs-comment">### 激活环境</span><br>conda activate dl_env<br><br><span class="hljs-comment">### 安装框架</span><br>pip install torch torchvision torchaudio<br>pip install tensorflow<br></code></pre></td></tr></table></figure></li><li><p>环境检查流程：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment">### 1. 检查CUDA版本</span><br>nvcc -V<br><br><span class="hljs-comment">### 2. 检查GPU驱动</span><br>nvidia-smi<br><br><span class="hljs-comment">### 3. 检查深度学习框架</span><br><span class="hljs-comment">#### PyTorch</span><br>import torch<br><span class="hljs-built_in">print</span>(torch.cuda.is_available())<br><br><span class="hljs-comment">#### TensorFlow</span><br>import tensorflow as tf<br>with tf.device(<span class="hljs-string">'/GPU:0'</span>):<br><span class="hljs-comment">## # 运行测试计算</span><br> ...<br></code></pre></td></tr></table></figure></li></ol><h5 id="5-3-关键教训"><a href="#5-3-关键教训" class="headerlink" title="5.3 关键教训"></a>5.3 关键教训</h5><ol><li>不要在base环境中安装深度学习框架</li><li>安装前先检查各组件的版本兼容性</li><li>遇到环境问题时，创建新环境比修复旧环境更高效</li><li>保持良好的环境管理习惯，定期清理不用的环境和缓存</li></ol><h4 id="六、有用的命令汇总"><a href="#六、有用的命令汇总" class="headerlink" title="六、有用的命令汇总"></a>六、有用的命令汇总</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment">### CUDA和驱动相关</span><br>nvidia-smi <span class="hljs-comment"># 检查GPU状态和驱动版本</span><br>nvcc -V <span class="hljs-comment"># 检查CUDA版本</span><br><span class="hljs-built_in">ls</span> -l /usr/local/cuda <span class="hljs-comment"># 检查CUDA符号链接</span><br><br><span class="hljs-comment">### Conda环境管理</span><br>conda <span class="hljs-built_in">env</span> list <span class="hljs-comment"># 列出所有环境</span><br>conda create -n env_name python=3.10 <span class="hljs-comment"># 创建新环境</span><br>conda activate env_name <span class="hljs-comment"># 激活环境</span><br>conda remove --name env_name --all <span class="hljs-comment"># 删除环境</span><br>conda clean --all <span class="hljs-comment"># 清理缓存</span><br><br><span class="hljs-comment">### 系统检查</span><br><span class="hljs-built_in">du</span> -sh ~/anaconda3/ <span class="hljs-comment"># 检查Anaconda占用空间</span><br><span class="hljs-built_in">which</span> nvcc <span class="hljs-comment"># 查找nvcc位置</span><br></code></pre></td></tr></table></figure><h4 id="七、特殊情况：ESP-IDF环境清理"><a href="#七、特殊情况：ESP-IDF环境清理" class="headerlink" title="七、特殊情况：ESP-IDF环境清理"></a>七、特殊情况：ESP-IDF环境清理</h4><h5 id="8-1-问题发现和清理尝试"><a href="#8-1-问题发现和清理尝试" class="headerlink" title="8.1 问题发现和清理尝试"></a>8.1 问题发现和清理尝试</h5><p>检查发现遗留的ESP-IDF环境：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment">### 发现两个位置</span><br>/home/dahao/anaconda3/envs/esp-idf-env <span class="hljs-comment"># conda环境</span><br>/home/dahao/esp/esp-idf <span class="hljs-comment"># ESP-IDF主程序</span><br></code></pre></td></tr></table></figure><p>尝试使用conda删除但失败：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs bash">conda remove --name esp-idf-env --all<br><span class="hljs-comment">### 错误：EnvironmentLocationNotFound: Not a conda environment: /home/dahao/anaconda3/envs/esp-idf-env</span><br></code></pre></td></tr></table></figure><h5 id="8-2-解决方案"><a href="#8-2-解决方案" class="headerlink" title="8.2 解决方案"></a>8.2 解决方案</h5><p>直接手动删除相关目录：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment">### 删除conda环境</span><br><span class="hljs-built_in">rm</span> -rf /home/dahao/anaconda3/envs/esp-idf-env<br><br><span class="hljs-comment">### 删除ESP-IDF主程序</span><br><span class="hljs-built_in">rm</span> -rf /home/dahao/esp/esp-idf<br><br><span class="hljs-comment">### 删除工具链目录（如果存在）</span><br><span class="hljs-built_in">rm</span> -rf ~/.espressif<br></code></pre></td></tr></table></figure><h5 id="8-3-额外清理"><a href="#8-3-额外清理" class="headerlink" title="8.3 额外清理"></a>8.3 额外清理</h5><p>检查并清理<code>.bashrc</code>中的ESP-IDF相关配置：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs bash">vim ~/.bashrc<br><br><span class="hljs-comment">### 找到并删除类似的配置行</span><br><span class="hljs-comment">### export IDF_PATH="/home/dahao/esp/esp-idf"</span><br><span class="hljs-comment">### export PATH="$IDF_PATH/tools:$PATH"</span><br></code></pre></td></tr></table></figure><h4 id="结论"><a href="#结论" class="headerlink" title="结论"></a>结论</h4><p>配置深度学习环境是一个复杂的过程，需要注意多个组件之间的版本兼容性。通过合理的环境管理和版本选择，可以避免很多不必要的问题。建议在开始配置之前先做好规划，使用独立的conda环境，并保持良好的环境管理习惯。</p>]]></content>
<categories>
<category>ai chat</category>
</categories>
</entry>
<entry>
<title>human_face_detect-1.1</title>
<link href="/2024/11/11/human_face_detect-1.1/"/>
<url>/2024/11/11/human_face_detect-1.1/</url>
<content type="html"><![CDATA[<h1 id="human-face-detect-1-1"><a href="#human-face-detect-1-1" class="headerlink" title="human_face_detect-1.1"></a>human_face_detect-1.1</h1><p>这篇博客写了复现esp-dl官方给的算例human_face_detect的过程和一些解决思路。<br>原文直接参考文档：<br><a href="https://github.com/espressif/esp-dl/blob/release/v1.1/examples/human_face_detect/README_cn.md">https://github.com/espressif/esp-dl/blob/release/v1.1/examples/human_face_detect/README_cn.md</a><br><a href="https://docs.espressif.com/projects/esp-dl/zh_CN/release-v1.1/esp32s3/introduction.html#id4">https://docs.espressif.com/projects/esp-dl/zh_CN/release-v1.1/esp32s3/introduction.html#id4</a></p><h2 id="需要补充的一点点shell知识"><a href="#需要补充的一点点shell知识" class="headerlink" title="需要补充的一点点shell知识"></a>需要补充的一点点shell知识</h2><p>在windows中的powershell中，和ubuntu中的bash语法稍有不同。例如：<br>使用 <code>Remove-Item</code> 删除本地 <code>esp-dl</code> 文件夹：<br>`Remove-Item -Recurse -Force .\esp-d</p><h2 id="驱动问题"><a href="#驱动问题" class="headerlink" title="驱动问题"></a>驱动问题</h2><p>当设备管理器中看到一个usb serial设备显示黄色感叹号时，证明电脑能读取到一个esp32s3设备但是没法正常通信，因此需要重新打驱动CH340（CH341是和340通用的）<br>windows：<br><a href="https://www.wch.cn/downloads/file/65.html?time=2024-11-11%2016:23:41&code=NokOcQndbAiFrUztVEKcf1eZsx3x82vmIITMErTp">https://www.wch.cn/downloads/file/65.html?time=2024-11-11%2016:23:41&code=NokOcQndbAiFrUztVEKcf1eZsx3x82vmIITMErTp</a><br>linux：<br><a href="https://www.wch.cn/downloads/file/177.html?time=2024-11-11%2016:25:36&code=Dj3svYfYENNLKkggzMqECTTlDbDCGU8NUgONgvuE">https://www.wch.cn/downloads/file/177.html?time=2024-11-11%2016:25:36&code=Dj3svYfYENNLKkggzMqECTTlDbDCGU8NUgONgvuE</a><br>安装即可识别。</p><h2 id="对espressif的esp-dl官方教程的解释"><a href="#对espressif的esp-dl官方教程的解释" class="headerlink" title="对espressif的esp-dl官方教程的解释"></a>对espressif的esp-dl官方教程的解释</h2><p>我从官方在线pdf版的教程进入，不管选择esp32s3的pre-release的master分支，还是release的release/v1.1分支（在线pdf版本中只有这两个分支的教程），只要从尝试模型库中的模型的人脸检测部分跳转，挑战后的内容实际都是根据release/v1.1写的。而本人也本着稳定使用的原则采用release/v1.1的版本，这两个库中的内容有很大区别，如下图对比。<br><img src="/img/learn/20241111163039.png"><br><img src="/img/learn/20241111163149.png"></p><h3 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h3><h4 id="1-项目结构的模块化和标准化"><a href="#1-项目结构的模块化和标准化" class="headerlink" title="1. 项目结构的模块化和标准化"></a><strong>1. 项目结构的模块化和标准化</strong></h4><ul><li><strong>release/v1.1</strong>：<ul><li>项目结构偏向传统的嵌入式开发方式，文件分布比较直接，比如有硬编码的头文件（<code>image.hpp</code>）用于存储图像数据。</li><li>头文件、库、代码文件都需要显式地包含和链接。这使得项目更加紧密集成。</li></ul></li><li><strong>master</strong>：<ul><li>使用 <code>idf_component.yml</code> 来管理项目中的依赖项，使得项目各个部分的开发和复用更加方便。这种方式更符合 <strong>ESP-IDF</strong> 的现代化组件标准，让项目更适合长期维护和扩展。</li></ul></li></ul><h4 id="2-文件组织与依赖的简化"><a href="#2-文件组织与依赖的简化" class="headerlink" title="2. 文件组织与依赖的简化"></a><strong>2. 文件组织与依赖的简化</strong></h4><ul><li><strong>代码和头文件的变化</strong>：<ul><li>在 <strong>release/v1.1</strong> 中，使用了 <code>image.hpp</code> 直接嵌入图像数据，并不灵活。</li><li>在 <strong>master</strong> 中，删除了这种硬编码的做法，改为通过组件的方式（比如依赖 <code>esp_jpeg</code>）来处理图像，提供了更高的灵活性和解耦能力。</li></ul></li><li><strong>库链接管理</strong>：<ul><li>在 <strong>release/v1.1</strong> 中，<code>CMakeLists.txt</code> 中显式地对不同芯片类型进行静态库的链接管理，这使得项目对芯片的支持有了更多的控制，但也增加了代码的复杂性。</li><li><strong>master</strong> 版本不再显式链接多个库，而是通过组件管理自动处理。</li></ul></li></ul><h4 id="3-文件角色的变化"><a href="#3-文件角色的变化" class="headerlink" title="3. 文件角色的变化"></a><strong>3. 文件角色的变化</strong></h4><ul><li><strong>数据存储方式的变化</strong>：（这一点尤其重要，相信动手自己试过的朋友肯定能感受出来很大差异）<ul><li>**<code>image.hpp</code> vs <code>human_face.jpg</code>**：<ul><li><strong>release/v1.1</strong> 使用 <code>image.hpp</code> 将图像数据直接嵌入代码，这种方式可以让开发者快速测试模型，但不适合大规模使用。</li><li>在 <strong>master</strong> 中，示例图像文件（<code>Mona_Lisa.jpg</code>）被直接作为文件嵌入项目。这表明新的结构倾向于从文件系统加载数据，而不是将数据硬编码。这种方式更通用，并且减少了对代码体积的影响。</li></ul></li></ul></li></ul><h4 id="总结：从紧耦合到模块化的转变"><a href="#总结：从紧耦合到模块化的转变" class="headerlink" title="总结：从紧耦合到模块化的转变"></a><strong>总结：从紧耦合到模块化的转变</strong></h4><ul><li><strong>release/v1.1</strong> 的设计更关注直接有效的嵌入式实现，适合于快速开发和验证。它的代码和数据紧耦合，所有内容都集中管理，但这也意味着不太灵活，且难以扩展和维护。</li><li><strong>master</strong> 的改进方向明显朝着现代嵌入式开发的最佳实践靠拢，重点在于模块化、复用性和自动化管理。通过使用组件化管理工具（如 <code>idf_component.yml</code>）和外部资源加载，它减少了手动配置的复杂性，使得项目更适合长期维护和功能扩展。</li></ul><h3 id="esp32s3配置文件替代性分析"><a href="#esp32s3配置文件替代性分析" class="headerlink" title="*.esp32s3配置文件替代性分析"></a>*.esp32s3配置文件替代性分析</h3><ol><li><p><strong>基础配置</strong>：如果项目不需要使用特殊的 SPI RAM 模式（如 Octal 模式）或自定义分区表等高级配置，<strong>release/v1.1</strong> 的文件已经包含了基础的 CPU 频率和数据缓存设置，可以直接使用。</p></li><li><p><strong>性能优化和兼容性</strong>：如果项目需要更高的性能（如特定的 SPI RAM 模式、闪存设置）或者兼容更高版本的 ESP-IDF（例如 5.4.0），那么可以参考 <strong>master</strong> 的配置文件。</p></li><li><p><code>sdkconfig.defaults</code> 文件只在编译过程中被读取，设置相应的编译选项。它们不需要读取文件夹或其他文件，也不会有相互依赖。</p></li></ol><h3 id="partitions文件变化"><a href="#partitions文件变化" class="headerlink" title="partitions文件变化"></a>partitions文件变化</h3><p><strong>存储空间的变化</strong>：</p><ul><li><strong>master</strong> 分支将 <code>factory</code> 分区的大小从 <strong>2MB</strong> 增加到 <strong>8MB</strong>，这可能是为了给更大的应用程序提供足够的存储空间。</li></ul><h3 id="位于项目-main-目录中的CMakeLists-txt文件执行差异"><a href="#位于项目-main-目录中的CMakeLists-txt文件执行差异" class="headerlink" title="位于项目 main 目录中的CMakeLists.txt文件执行差异"></a>位于项目 <code>main</code> 目录中的<code>CMakeLists.txt</code>文件执行差异</h3><h4 id="Release-v1-1-执行过程："><a href="#Release-v1-1-执行过程：" class="headerlink" title="Release/v1.1 执行过程："></a><strong>Release/v1.1 执行过程</strong>：</h4><ol><li><strong>组件注册</strong>：<ul><li><strong><code>idf_component_register</code></strong> 会把所有的源文件（如 <code>app_main.cpp</code>）和包含目录加入到编译过程中。</li><li>因为有很多 <code>include</code> 目录，编译器需要处理和查找多个模块的头文件。</li></ul></li><li><strong>静态库链接</strong>：<ul><li>根据目标芯片（如 <code>esp32s3</code>），选择对应的库文件路径（<code>libhuman_face_detect.a</code>、<code>libdl.a</code>），并将这些库文件链接到项目中。</li><li>这些库文件提供了特定的功能，例如人脸检测算法。</li></ul></li><li><strong>总结</strong>：<ul><li>这种方式适合需要多个模块和特定功能的复杂项目，因为它手动管理和链接了多个资源。</li></ul></li></ol><h4 id="Master-执行过程："><a href="#Master-执行过程：" class="headerlink" title="Master 执行过程："></a><strong>Master 执行过程</strong>：</h4><ol><li><strong>组件注册</strong>：<ul><li>只包含当前目录的源文件和头文件，非常简洁。</li><li>依赖的组件（如 <code>esp_jpeg</code>）是通过 <strong><code>idf_component_register</code></strong> 自动引用的，ESP-IDF 会在构建时处理这些依赖，减少手动配置的步骤。</li></ul></li><li><strong>嵌入文件</strong>：<ul><li>有个嵌入的文件（<code>human_face.jpg</code>），这意味着这个文件会被直接打包到程序中，便于在程序中直接访问。</li></ul></li><li><strong>总结</strong>：<ul><li>这种方式更模块化和简单，适合对依赖要求较少的项目，编译过程也会更快一些，因为引用的头文件和库更少。</li></ul></li></ol><h3 id="yml文件"><a href="#yml文件" class="headerlink" title="yml文件"></a>yml文件</h3><p><strong><code>idf_component.yml</code> 文件是用于模块化管理项目的组件和依赖</strong>，目的是为了使开发更加灵活和自动化。</p><h2 id="执行过程"><a href="#执行过程" class="headerlink" title="执行过程"></a>执行过程</h2><h3 id="部署部分代码"><a href="#部署部分代码" class="headerlink" title="部署部分代码"></a>部署部分代码</h3><p>1、修改代码后先删除原来的build文件夹<br>2、运行esp-idf的cmd或powershell快捷方式<br>本人桌面上的powershell快捷方式不知为何打开变成了cmd命令行，因此可以先执行下列代码完成环境配置：<br><code>& "D:\language\Espressif\Initialize-Idf.ps1" -IdfId esp-idf-c1b3982edf7b5833f785c865a3465866</code><br>若读者也出现类似问题可以尝试找到自己esp-idf安装位置的ps1脚本进行同样的跳转<br>3、<code>idf.py build</code>创建build文件夹，也可同时选中esp32s3型号：<code>idf.py -D IDF_TARGET=esp32s3 build</code><br>4、<code>idf.py flash</code>进行烧录<br>5、<code>idf.py monitor</code>监视esp32s3的输出；按<code>ctrl+]</code>退出监视终端<br>本人因为为了实现后续批注图片的编写性，因此这里执行<code>idf.py monitor | .\parse_face_detect.ps1</code>，后文会介绍功能</p><h3 id="master部分"><a href="#master部分" class="headerlink" title="master部分"></a>master部分</h3><p>master部分可以直接执行成功，但因为我还没有太搞懂在不引用include文件夹中的各种hpp头文件的过程是怎么运行的，并且这是一个pre-release版本后续可能还会有官方改动，因此我没有尝试采用其他的图片。烧录成功后即进行下一部分的release/v1.1部分。</p><h3 id="release-v1-1部分"><a href="#release-v1-1部分" class="headerlink" title="release/v1.1部分"></a>release/v1.1部分</h3><h4 id="hpp头文件问题："><a href="#hpp头文件问题：" class="headerlink" title="hpp头文件问题："></a>hpp头文件问题：</h4><p>执行遇到的第一个问题是如下图所示，可以看到是”\esp-dl\include\tool\dl_tool.hpp”出了问题：<br><img src="/img/learn/20241111171022.png"><br>因此将对应的hpp头文件中的对应行进行如下修改：<br>增加包含头文件：<code>#include <inttypes.h></code><br>第399行改为：<code>printf("latency: %" PRIu32 " us\n", this->get_average_period());</code><br>第413行改为：<code>printf("%s: %" PRIu32 " us\n", message, this->get_average_period());</code><br>第428行改为：<code>printf("%s:%s: %" PRIu32 " us\n", prefix, key, this->get_average_period());</code><br>虽然397、411、426行也有对应的语法，但是并未报错因此不做修改，否则会报错。</p><h4 id="PSRAM初始化失败"><a href="#PSRAM初始化失败" class="headerlink" title="PSRAM初始化失败"></a>PSRAM初始化失败</h4><p>错误信息：</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs pgsql">E (<span class="hljs-number">351</span>) spiram: SPI RAM enabled but initialization failed. Bailing <span class="hljs-keyword">out</span>.<br>E (<span class="hljs-number">351</span>) cpu_start: Failed <span class="hljs-keyword">to</span> init <span class="hljs-keyword">external</span> RAM!<br><span class="hljs-keyword">abort</span>() was <span class="hljs-keyword">called</span> at PC <span class="hljs-number">0x40375451</span> <span class="hljs-keyword">on</span> core <span class="hljs-number">0</span><br></code></pre></td></tr></table></figure><p>禁用PSRAM步骤：<br>执行<code>idf.py menuconfig</code></p><ul><li><strong>进入 <code>ESP PSRAM</code> 菜单</strong>：<ul><li>选择 <code>ESP PSRAM ---></code> 选项。</li></ul></li><li><strong>禁用 PSRAM 支持</strong>：<ul><li>有一个选项 <code>Enable PSRAM support</code> 或 <code>Support for external, SPI-connected RAM (PSRAM)</code>。</li><li>将这个选项<strong>取消勾选</strong>，确保它是禁用状态。</li></ul></li></ul><h4 id="第一种因为大小报错的情况：Flash-或分区表问题导致build过程报错"><a href="#第一种因为大小报错的情况：Flash-或分区表问题导致build过程报错" class="headerlink" title="第一种因为大小报错的情况：Flash 或分区表问题导致build过程报错"></a>第一种因为大小报错的情况：Flash 或分区表问题导致build过程报错</h4><p>报错图如下：<br><img src="/img/learn/20241111185548.png"><br>解决方法：</p><h5 id="方法1：修改-Flash-大小配置"><a href="#方法1：修改-Flash-大小配置" class="headerlink" title="方法1：修改 Flash 大小配置"></a>方法1：修改 Flash 大小配置</h5><p><code>idf.py menuconfig</code><br><code>Serial Flasher Config -> Flash size</code>，将其改为更大的值，esp32s3根据型号等不同有两种最大闪存16MB或32MB。<br>按<code>Esc</code>退出，按<code>y</code>保存更改。</p><h5 id="方法-2：调整分区表大小"><a href="#方法-2：调整分区表大小" class="headerlink" title="方法 2：调整分区表大小"></a>方法 2：调整分区表大小</h5><p>如果按照方法1中，使用的硬件的 Flash 已经调整至<strong>最大值</strong>，则需要调整分区表以适配 Flash 容量。<br>将<code>partitions.csv</code>中的<code>factory</code>行（应用分区）最后一个值（其总大小）调整至至少比main文件夹大（因为占flash最大的部分是图片hpp文件）。</p><h4 id="第二种因为大小报错的情况：能成功烧录但monitor循环报错"><a href="#第二种因为大小报错的情况：能成功烧录但monitor循环报错" class="headerlink" title="第二种因为大小报错的情况：能成功烧录但monitor循环报错"></a>第二种因为大小报错的情况：能成功烧录但monitor循环报错</h4><p>这种情况报错如下图所示，其前半部分绿色的参数部分输出是正常的，在该输出坐标的白色位置开始报错，结尾有大段关于build文件夹中路径的黄色报错，并且三部分反复循环，<code>ctrl+]</code>也无法关闭monitor界面，只能直接关闭终端。<br><img src="/img/learn/xunhuanbaocuo.jpg"><br>这种情况是因为本人一开始本人弄错了图片<strong>清晰度</strong>和<strong>尺寸</strong>的关系，一开始我以为只要将网上下载的图片处理到和示例蒙娜丽莎的<strong>大小</strong>40KB左右即可（<em>这个处理实际上是降低清晰度的过程</em>），但总是出现这种情况的报错，后来在观察从<code>image.jpg</code>生成的<code>image.hpp</code>文件大小时，发现需要将图片处理成<strong>宽度和高度都在约600px以下</strong>的图片才能完成正常识别。<br>图片处理网站：<a href="https://www.iloveimg.com/zh-cn/resize-image#resize-options,pixels">https://www.iloveimg.com/zh-cn/resize-image#resize-options,pixels</a><br>后续会自制离线能运行的cpp代码，方便编入esp32s3。</p><h4 id="后续操作步骤简化"><a href="#后续操作步骤简化" class="headerlink" title="后续操作步骤简化"></a>后续操作步骤简化</h4><h5 id="提取python命令"><a href="#提取python命令" class="headerlink" title="提取python命令"></a>提取python命令</h5><p>原文说打印检测结果的分数值和坐标值为如下时：</p><figure class="highlight smali"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs smali">idf.py flash<span class="hljs-built_in"> monitor</span><br><span class="hljs-built_in"></span><br><span class="hljs-keyword">.</span>.. ...<br><br>[0] score: 0.987580, box: [137, 75, 246, 215]<br> left eye: (157, 131), right eye: (199, 133)<br> nose: (170, 163)<br> mouth left: (158, 177), mouth right: (193, 180)<br></code></pre></td></tr></table></figure><p>运行如下标注命令：</p><figure class="highlight stylus"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs stylus">python display_image<span class="hljs-selector-class">.py</span> -<span class="hljs-selector-tag">i</span> ../human_face_detect/<span class="hljs-selector-tag">image</span><span class="hljs-selector-class">.jpg</span> -<span class="hljs-selector-tag">b</span> <span class="hljs-string">"(137, 75, 246, 215)"</span> -k <span class="hljs-string">"(157, 131, 199, 133, 170, 163, 158, 177, 193, 180)"</span><br></code></pre></td></tr></table></figure><p>但这还需要繁琐的编辑，因此在这里编辑一个powershell脚本，可以实现python命令的自动输出，脚本如下：</p><figure class="highlight powershell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br></pre></td><td class="code"><pre><code class="hljs powershell"><span class="hljs-comment"># 简化版人脸检测结果解析脚本</span><br><br><span class="hljs-variable">$input</span> | <span class="hljs-built_in">ForEach-Object</span> {<br><br> <span class="hljs-variable">$line</span> = <span class="hljs-variable">$_</span><br><br> <span class="hljs-comment"># 匹配人脸检测结果行，修改正则表达式以更准确匹配输出格式</span><br><br> <span class="hljs-keyword">if</span> (<span class="hljs-variable">$line</span> <span class="hljs-operator">-match</span> <span class="hljs-string">'^\[(\d+)\] score: ([\d.]+), box: \[([\d, ]+)\]\s*$'</span>) {<br><br> <span class="hljs-comment"># 保存box信息</span><br><br> <span class="hljs-variable">$faceIndex</span> = <span class="hljs-variable">$matches</span>[<span class="hljs-number">1</span>]<br><br> <span class="hljs-variable">$box</span> = <span class="hljs-variable">$matches</span>[<span class="hljs-number">3</span>]<br><br> }<br><br> <span class="hljs-comment"># 匹配关键点信息</span><br><br> <span class="hljs-keyword">elseif</span> (<span class="hljs-variable">$line</span> <span class="hljs-operator">-match</span> <span class="hljs-string">'left eye: \(([\d, ]+)\), right eye: \(([\d, ]+)\)'</span>) {<br><br> <span class="hljs-variable">$leftEye</span> = <span class="hljs-variable">$matches</span>[<span class="hljs-number">1</span>]<br><br> <span class="hljs-variable">$rightEye</span> = <span class="hljs-variable">$matches</span>[<span class="hljs-number">2</span>]<br><br> }<br><br> <span class="hljs-keyword">elseif</span> (<span class="hljs-variable">$line</span> <span class="hljs-operator">-match</span> <span class="hljs-string">'nose: \(([\d, ]+)\)'</span>) {<br><br> <span class="hljs-variable">$nose</span> = <span class="hljs-variable">$matches</span>[<span class="hljs-number">1</span>]<br><br> }<br><br> <span class="hljs-keyword">elseif</span> (<span class="hljs-variable">$line</span> <span class="hljs-operator">-match</span> <span class="hljs-string">'mouth left: \(([\d, ]+)\), mouth right: \(([\d, ]+)\)'</span>) {<br><br> <span class="hljs-variable">$mouthLeft</span> = <span class="hljs-variable">$matches</span>[<span class="hljs-number">1</span>]<br><br> <span class="hljs-variable">$mouthRight</span> = <span class="hljs-variable">$matches</span>[<span class="hljs-number">2</span>]<br><br> <span class="hljs-comment"># 当收集到所有信息后，输出命令</span><br><br> <span class="hljs-keyword">if</span> (<span class="hljs-variable">$box</span> <span class="hljs-operator">-and</span> <span class="hljs-variable">$leftEye</span> <span class="hljs-operator">-and</span> <span class="hljs-variable">$rightEye</span> <span class="hljs-operator">-and</span> <span class="hljs-variable">$nose</span> <span class="hljs-operator">-and</span> <span class="hljs-variable">$mouthLeft</span> <span class="hljs-operator">-and</span> <span class="hljs-variable">$mouthRight</span>) {<br><br> <span class="hljs-variable">$keypoints</span> = <span class="hljs-string">"<span class="hljs-variable">$leftEye</span>, <span class="hljs-variable">$rightEye</span>, <span class="hljs-variable">$nose</span>, <span class="hljs-variable">$mouthLeft</span>, <span class="hljs-variable">$mouthRight</span>"</span><br><br> <span class="hljs-variable">$command</span> = <span class="hljs-string">"python display_image.py -i ../human_face_detect/image.jpg -b `"(<span class="hljs-variable">$box</span>)`" -k `"(<span class="hljs-variable">$keypoints</span>)`""</span><br><br> <span class="hljs-built_in">Write-Host</span> <span class="hljs-variable">$command</span><br><br> <span class="hljs-comment"># 清除变量，为下一个人脸做准备</span><br><br> <span class="hljs-variable">$box</span> = <span class="hljs-variable">$leftEye</span> = <span class="hljs-variable">$rightEye</span> = <span class="hljs-variable">$nose</span> = <span class="hljs-variable">$mouthLeft</span> = <span class="hljs-variable">$mouthRight</span> = <span class="hljs-variable">$null</span><br><br> }<br><br> }<br><br>}<br></code></pre></td></tr></table></figure><p>该脚本可以实现将任意数量的人脸输出结果中需要的参数提取，并输出下一步需要执行的python命令。如果是多张人脸则逐行输入各自独立的python代码，然后复制粘贴执行即可。</p><h5 id="python命令改写"><a href="#python命令改写" class="headerlink" title="python命令改写"></a>python命令改写</h5><p>原库中的display图片的python脚本在多张人脸时，生成的第二张图中不会保留第一张图的批注痕迹，因此改为如下代码：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> ast<br><br><span class="hljs-keyword">import</span> cv2<br><br><span class="hljs-keyword">import</span> os<br><br> <br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">draw_boxes</span>(<span class="hljs-params">image_path, boxes, keypoints=<span class="hljs-literal">None</span>, output_path=<span class="hljs-literal">None</span></span>):<br><br> <span class="hljs-comment"># Load the image from the given path</span><br><br> image = cv2.imread(image_path)<br><br> <span class="hljs-comment"># Ensure boxes is a list of tuples, even if only a single tuple is provided</span><br><br> <span class="hljs-keyword">if</span> <span class="hljs-built_in">isinstance</span>(boxes, <span class="hljs-built_in">tuple</span>):<br><br> boxes = [boxes]<br><br> <span class="hljs-comment"># Loop through each box coordinate and draw on the image</span><br><br> <span class="hljs-keyword">for</span> box <span class="hljs-keyword">in</span> boxes:<br><br> x1, y1, x2, y2 = box<br><br> cv2.rectangle(image, (x1, y1), (x2, y2), (<span class="hljs-number">0</span>, <span class="hljs-number">255</span>, <span class="hljs-number">0</span>), <span class="hljs-number">2</span>) <span class="hljs-comment"># Draw green rectangles</span><br><br> <span class="hljs-comment"># If keypoints are provided, draw them on the image</span><br><br> <span class="hljs-keyword">if</span> keypoints:<br><br> <span class="hljs-keyword">if</span> <span class="hljs-built_in">isinstance</span>(keypoints[<span class="hljs-number">0</span>], <span class="hljs-built_in">int</span>):<br><br> keypoints = [keypoints]<br><br> <span class="hljs-keyword">for</span> kp <span class="hljs-keyword">in</span> keypoints:<br><br> <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">0</span>, <span class="hljs-built_in">len</span>(kp), <span class="hljs-number">2</span>):<br><br> x, y = kp[i], kp[i + <span class="hljs-number">1</span>]<br><br> cv2.circle(image, (x, y), <span class="hljs-number">3</span>, (<span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">255</span>), -<span class="hljs-number">1</span>) <span class="hljs-comment"># Draw red circles for keypoints</span><br><br> <span class="hljs-comment"># Save the modified image if output_path is provided</span><br><br> <span class="hljs-keyword">if</span> output_path:<br><br> cv2.imwrite(output_path, image)<br><br> <span class="hljs-comment"># Display the image with all boxes</span><br><br> cv2.imshow(<span class="hljs-string">'Image with boxes'</span>, image)<br><br> cv2.waitKey(<span class="hljs-number">0</span>) <span class="hljs-comment"># Wait for any key to be pressed</span><br><br> cv2.destroyAllWindows() <span class="hljs-comment"># Close the image window</span><br><br> <br><br><span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:<br><br> <span class="hljs-keyword">import</span> argparse<br><br> <span class="hljs-comment"># Set up argument parser</span><br><br> parser = argparse.ArgumentParser()<br><br> parser.add_argument(<span class="hljs-string">"-b"</span>, <span class="hljs-string">"--boxes"</span>, required=<span class="hljs-literal">True</span>, <span class="hljs-built_in">help</span>=<span class="hljs-string">"Bounding boxes for faces in the format of a list of tuples"</span>)<br><br> parser.add_argument(<span class="hljs-string">"-i"</span>, <span class="hljs-string">"--image"</span>, required=<span class="hljs-literal">True</span>, <span class="hljs-built_in">help</span>=<span class="hljs-string">"Path to the image file"</span>)<br><br> parser.add_argument(<span class="hljs-string">"-k"</span>, <span class="hljs-string">"--keypoints"</span>, required=<span class="hljs-literal">False</span>, <span class="hljs-built_in">help</span>=<span class="hljs-string">"Keypoints for faces in the format of a list of tuples"</span>)<br><br> parser.add_argument(<span class="hljs-string">"-o"</span>, <span class="hljs-string">"--output"</span>, required=<span class="hljs-literal">False</span>, <span class="hljs-built_in">help</span>=<span class="hljs-string">"Path to save the updated image file"</span>)<br><br> args = parser.parse_args()<br><br> <br><br> <span class="hljs-comment"># Determine the path to save intermediate results if not specified</span><br><br> output_path = args.output <span class="hljs-keyword">if</span> args.output <span class="hljs-keyword">else</span> args.image<br><br> <br><br> <span class="hljs-comment"># Convert the input string of boxes to a list of tuples</span><br><br> boxes = ast.literal_eval(args.boxes) <span class="hljs-comment"># Use ast.literal_eval to safely evaluate the string as Python literal</span><br><br> <span class="hljs-comment"># Convert the input string of keypoints to a list of tuples if provided</span><br><br> keypoints = ast.literal_eval(args.keypoints) <span class="hljs-keyword">if</span> args.keypoints <span class="hljs-keyword">else</span> <span class="hljs-literal">None</span><br><br> <span class="hljs-comment"># Call the function to draw the boxes and keypoints on the image</span><br><br> draw_boxes(output_path, boxes, keypoints, output_path)<br></code></pre></td></tr></table></figure><p>该代码可以实现批注痕迹覆盖。</p><h2 id="最终结果"><a href="#最终结果" class="headerlink" title="最终结果"></a>最终结果</h2><p>最终结果如图所示，可以实现多张人脸的识别：<br><img src="/img/learn/20241111193523.png"></p>]]></content>
<categories>
<category>fyp</category>
</categories>
</entry>
<entry>
<title>building_with_espdl-3-failure</title>
<link href="/2024/11/07/building_with_espdl-3-failure/"/>
<url>/2024/11/07/building_with_espdl-3-failure/</url>
<content type="html"><![CDATA[<h1 id="building-with-espdl-3-failure"><a href="#building-with-espdl-3-failure" class="headerlink" title="building_with_espdl-3-failure"></a>building_with_espdl-3-failure</h1><p>这部分在实践的过程中知识有点杂，因此笔记稍显混乱。</p><h2 id="windows中的powershell"><a href="#windows中的powershell" class="headerlink" title="windows中的powershell"></a>windows中的powershell</h2><h3 id="类似ls的shell语言"><a href="#类似ls的shell语言" class="headerlink" title="类似ls的shell语言"></a>类似ls的shell语言</h3><h4 id="Get-ChildItem"><a href="#Get-ChildItem" class="headerlink" title="Get-ChildItem"></a>Get-ChildItem</h4><p>列出当前目录下的所有文件和文件夹：<code>Get-ChildItem</code><br>递归列出指定目录下的所有文件和文件夹：<code>Get-ChildItem -Recurse C:\MyDocuments</code><br>筛选特定类型的文件：<code>Get-ChildItem -Filter *.txt</code><br><strong>常用参数：</strong></p><ul><li><code>-Path</code>: 指定要获取子项的路径。</li><li><code>-Recurse</code>: 递归搜索子目录。</li><li><code>-Filter</code>: 根据文件名模式筛选文件。</li><li><code>-Include</code>: 指定要包含的文件名模式。</li><li><code>-Exclude</code>: 指定要排除的文件名模式。</li><li><code>-Directory</code>: 只返回目录。</li><li><code>-File</code>: 只返回文件。</li></ul><h4 id="dir"><a href="#dir" class="headerlink" title="dir"></a>dir</h4><p>列出当前目录下的所有文件和文件夹：<code>dir</code><br>以简略格式列出：<code>dir /B</code><br>只列出目录：<code>dir /AD</code><br><strong>常用参数：</strong></p><ul><li><code>/B</code>: 以简略格式列出。</li><li><code>/AD</code>: 只列出目录。</li><li><code>/S</code>: 递归搜索子目录。</li></ul><h3 id="其他杂项"><a href="#其他杂项" class="headerlink" title="其他杂项"></a>其他杂项</h3><p>打开jupyter notebook中ipynb文件自动关闭jupyter终端：<br><a href="https://blog.csdn.net/qq_45404853/article/details/121310483?fromshare=blogdetail&sharetype=blogdetail&sharerId=121310483&sharerefer=PC&sharesource=m0_60571820&sharefrom=from_link">https://blog.csdn.net/qq_45404853/article/details/121310483?fromshare=blogdetail&sharetype=blogdetail&sharerId=121310483&sharerefer=PC&sharesource=m0_60571820&sharefrom=from_link</a><br>windows的conda安装：<br><a href="https://blog.csdn.net/ciagrate/article/details/140129703?fromshare=blogdetail&sharetype=blogdetail&sharerId=140129703&sharerefer=PC&sharesource=m0_60571820&sharefrom=from_link">https://blog.csdn.net/ciagrate/article/details/140129703?fromshare=blogdetail&sharetype=blogdetail&sharerId=140129703&sharerefer=PC&sharesource=m0_60571820&sharefrom=from_link</a><br><strong>在powershell中识别不了conda环境是因为也需要像ubuntu中修改.bashrc一样修改powershell的配置文件。</strong></p><h2 id="原文第三部分的复现（失败）"><a href="#原文第三部分的复现（失败）" class="headerlink" title="原文第三部分的复现（失败）"></a>原文第三部分的复现（失败）</h2><p>本人在windows11上重新尝试了全文，参数是esp-idf的release/v4.4的branch，esp-dl的idfv4.4的branch，python=3.7，和原作者完全相同。但在执行到第二部分优化和量化模型，在前面笔记提到的三行代码处仍然失败（如下图），证明esp-dl确实有问题。<br><img src="/img/learn/20241107004007.png"><br>因此本人改变思路，既然作者自己都无法在24年复现22年的优化和量化过程，那我直接使用22年作者机缘巧合成功生成的cpp文件，直接往esp32s3里面烧录，复现第三部分模型部署，但是仍然遇到问题：</p><h3 id="cmake文件路径未标明"><a href="#cmake文件路径未标明" class="headerlink" title="cmake文件路径未标明"></a>cmake文件路径未标明</h3><p>外层 <code>CMakeLists.txt</code> 文件已经包含以下行：`set(EXTRA_COMPONENT_DIRS ./components/esp-dl)·<br>为了确保路径一致性，建议简化路径设置：</p><figure class="highlight stylus"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs stylus"><span class="hljs-function"><span class="hljs-title">set</span><span class="hljs-params">(EXTRA_COMPONENT_DIRS ${CMAKE_SOURCE_DIR}/components/esp-dl)</span></span><br><span class="hljs-function"><span class="hljs-title">include_directories</span><span class="hljs-params">(${CMAKE_SOURCE_DIR}/components/esp-dl)</span></span><br></code></pre></td></tr></table></figure><p>在 <code>main</code> 文件夹中的 <code>CMakeLists.txt</code> 文件，可以做以下调整：<br>1、**更新 <code>include_dirs</code>**：</p><ul><li>将 <code>include_dirs</code> 添加 <code>components/esp-dl</code> 路径，以确保 <code>dl_tool.hpp</code> 能被找到。<br><code>set(include_dirs ../model ${CMAKE_SOURCE_DIR}/components/esp-dl)</code><br>2、<strong>检查 <code>idf_component_register</code> 设置</strong>：</li><li>更新 <code>idf_component_register</code> 部分，将 <code>INCLUDE_DIRS</code> 变量更新为刚刚修改的 <code>include_dirs</code>，确保 <code>esp-dl</code> 的头文件路径被正确添加：<br><code>idf_component_register(SRCS ${srcs} INCLUDE_DIRS ${include_dirs} REQUIRES ${requires})</code></li></ul><h3 id="头文件未放置在-components-esp-dl-文件夹中"><a href="#头文件未放置在-components-esp-dl-文件夹中" class="headerlink" title="头文件未放置在.components/esp-dl/文件夹中"></a>头文件未放置在.components/esp-dl/文件夹中</h3><p><code>.hpp</code> 文件是一种 C++ 代码文件的扩展名，用来定义 C++ 的头文件（header file），类似于 <code>.h</code> 文件。本文需要将下图的esp-dl官方的头文件放到此文件夹中。<br><img src="/img/learn/2024-11-07005756.png"></p><h3 id="printf-函数的格式化字符串上（未解决）"><a href="#printf-函数的格式化字符串上（未解决）" class="headerlink" title="printf 函数的格式化字符串上（未解决）"></a><code>printf</code> 函数的格式化字符串上（未解决）</h3><p><img src="/img/learn/2024-11-07005117.png"><br>当前的代码中，以下几行使用了 <code>printf</code>，并且 <code>%u</code> 与 <code>uint32_t</code> 类型不匹配：</p><figure class="highlight perl"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs perl"><span class="hljs-keyword">printf</span>(<span class="hljs-string">"Latency: <span class="hljs-variable">%u</span>"</span>, this->get_average_period());<br><span class="hljs-keyword">printf</span>(<span class="hljs-string">"<span class="hljs-variable">%s</span>: <span class="hljs-variable">%u</span>"</span>, prefix, key, this->get_average_period());<br></code></pre></td></tr></table></figure><p>要修正这些格式化问题，需要引入 <code><inttypes.h></code> 头文件，然后使用 <code>PRIu32</code> 来替代 <code>%u</code>。这是因为 <code>PRIu32</code> 是一个与 <code>uint32_t</code> 类型兼容的格式宏。以下是修改后的代码示例：</p><figure class="highlight cpp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs cpp"><span class="hljs-meta">#<span class="hljs-keyword">include</span> <span class="hljs-string"><inttypes.h></span> <span class="hljs-comment">// 添加这个头文件</span></span><br><br><span class="hljs-type">void</span> dl::tool::Latency::<span class="hljs-built_in">print</span>() {<br> <span class="hljs-built_in">printf</span>(<span class="hljs-string">"Latency: %"</span> PRIu32 <span class="hljs-string">"\n"</span>, <span class="hljs-keyword">this</span>-><span class="hljs-built_in">get_average_period</span>());<br>}<br><br><span class="hljs-type">void</span> dl::tool::Latency::<span class="hljs-built_in">print</span>(<span class="hljs-type">const</span> <span class="hljs-type">char</span>* prefix, <span class="hljs-type">uint32_t</span> key) {<br> <span class="hljs-built_in">printf</span>(<span class="hljs-string">"%s: %"</span> PRIu32 <span class="hljs-string">"\n"</span>, prefix, key, <span class="hljs-keyword">this</span>-><span class="hljs-built_in">get_average_period</span>());<br>}<br></code></pre></td></tr></table></figure><h2 id="后续可能解决方案"><a href="#后续可能解决方案" class="headerlink" title="后续可能解决方案"></a>后续可能解决方案</h2><p>只有放弃esp-dl，使用其他模型转化工具了。</p>]]></content>
<categories>
<category>fyp</category>
</categories>
</entry>
<entry>
<title>building_with_espdl-2-failure</title>
<link href="/2024/11/05/model_quantization_and_optimization/"/>
<url>/2024/11/05/model_quantization_and_optimization/</url>
<content type="html"><![CDATA[<h1 id="building-with-espdl-2-failure"><a href="#building-with-espdl-2-failure" class="headerlink" title="building_with_espdl-2-failure"></a>building_with_espdl-2-failure</h1><h2 id="本人代码"><a href="#本人代码" class="headerlink" title="本人代码"></a>本人代码</h2><h3 id="正常部分"><a href="#正常部分" class="headerlink" title="正常部分"></a>正常部分</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 列出当前工作目录中的文件</span><br>%ls<br><br></code></pre></td></tr></table></figure><p>这部分要注意windows的python库是_<strong>.pyd</strong>形式，而linux是esp-dl的git库中同名的_<strong>.so</strong>文件。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 导入必要的库</span><br><span class="hljs-keyword">from</span> optimizer <span class="hljs-keyword">import</span> *<br><span class="hljs-keyword">from</span> calibrator <span class="hljs-keyword">import</span> *<br><span class="hljs-keyword">from</span> evaluator <span class="hljs-keyword">import</span> *<br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python">onnx_model = onnx.load(<span class="hljs-string">"handrecognition_model.onnx"</span>)<br><br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python">optimized_model_path = optimize_fp_model(<span class="hljs-string">"handrecognition_model.onnx"</span>)<br><br></code></pre></td></tr></table></figure><h4 id="打印模型的计算图信息"><a href="#打印模型的计算图信息" class="headerlink" title="打印模型的计算图信息"></a>打印模型的计算图信息</h4><h5 id="代码"><a href="#代码" class="headerlink" title="代码"></a>代码</h5><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs python">optimized_model = onnx.load(optimized_model_path)<br><span class="hljs-built_in">print</span>(<span class="hljs-string">"Optimized ONNX model loaded successfully."</span>)<br><span class="hljs-built_in">print</span>(onnx.helper.printable_graph(optimized_model.graph))<br><br></code></pre></td></tr></table></figure><h5 id="输出"><a href="#输出" class="headerlink" title="输出"></a>输出</h5><pre><code class="hljs">Optimized ONNX model loaded successfully.graph tf2onnx ( %conv2d_input[FLOAT, unk__33x96x96x1]) initializers ( %new_shape__31[INT64, 4] %const_fold_opt__32[INT64, 2] %StatefulPartitionedCall/sequential/dense_1/MatMul/ReadVariableOp:0[FLOAT, 128x6] %StatefulPartitionedCall/sequential/dense_1/BiasAdd/ReadVariableOp:0[FLOAT, 6] %StatefulPartitionedCall/sequential/dense/MatMul/ReadVariableOp:0[FLOAT, 6400x128] %StatefulPartitionedCall/sequential/dense/BiasAdd/ReadVariableOp:0[FLOAT, 128] %StatefulPartitionedCall/sequential/conv2d_2/Conv2D/ReadVariableOp:0[FLOAT, 64x64x3x3] %StatefulPartitionedCall/sequential/conv2d_2/BiasAdd/ReadVariableOp:0[FLOAT, 64] %StatefulPartitionedCall/sequential/conv2d_1/Conv2D/ReadVariableOp:0[FLOAT, 64x32x3x3] %StatefulPartitionedCall/sequential/conv2d_1/BiasAdd/ReadVariableOp:0[FLOAT, 64] %StatefulPartitionedCall/sequential/conv2d/Conv2D/ReadVariableOp:0[FLOAT, 32x1x5x5] %StatefulPartitionedCall/sequential/conv2d/BiasAdd/ReadVariableOp:0[FLOAT, 32]) { %StatefulPartitionedCall/sequential/conv2d/BiasAdd__6:0 = Reshape(%conv2d_input, %new_shape__31) %StatefulPartitionedCall/sequential/conv2d/BiasAdd:0 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], strides = [1, 1]](%StatefulPartitionedCall/sequential/conv2d/BiasAdd__6:0, %StatefulPartitionedCall/sequential/conv2d/Conv2D/ReadVariableOp:0, %StatefulPartitionedCall/sequential/conv2d/BiasAdd/ReadVariableOp:0) %StatefulPartitionedCall/sequential/conv2d/Relu:0 = Relu(%StatefulPartitionedCall/sequential/conv2d/BiasAdd:0) %StatefulPartitionedCall/sequential/max_pooling2d/MaxPool:0 = MaxPool[kernel_shape = [2, 2], strides = [2, 2]](%StatefulPartitionedCall/sequential/conv2d/Relu:0) %StatefulPartitionedCall/sequential/conv2d_1/BiasAdd:0 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], strides = [1, 1]](%StatefulPartitionedCall/sequential/max_pooling2d/MaxPool:0, %StatefulPartitionedCall/sequential/conv2d_1/Conv2D/ReadVariableOp:0, %StatefulPartitionedCall/sequential/conv2d_1/BiasAdd/ReadVariableOp:0) %StatefulPartitionedCall/sequential/conv2d_1/Relu:0 = Relu(%StatefulPartitionedCall/sequential/conv2d_1/BiasAdd:0) %StatefulPartitionedCall/sequential/max_pooling2d_1/MaxPool:0 = MaxPool[kernel_shape = [2, 2], strides = [2, 2]](%StatefulPartitionedCall/sequential/conv2d_1/Relu:0) %StatefulPartitionedCall/sequential/conv2d_2/BiasAdd:0 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], strides = [1, 1]](%StatefulPartitionedCall/sequential/max_pooling2d_1/MaxPool:0, %StatefulPartitionedCall/sequential/conv2d_2/Conv2D/ReadVariableOp:0, %StatefulPartitionedCall/sequential/conv2d_2/BiasAdd/ReadVariableOp:0) %StatefulPartitionedCall/sequential/conv2d_2/Relu:0 = Relu(%StatefulPartitionedCall/sequential/conv2d_2/BiasAdd:0) %StatefulPartitionedCall/sequential/max_pooling2d_2/MaxPool:0 = MaxPool[kernel_shape = [2, 2], strides = [2, 2]](%StatefulPartitionedCall/sequential/conv2d_2/Relu:0) %StatefulPartitionedCall/sequential/max_pooling2d_2/MaxPool__28:0 = Transpose[perm = [0, 2, 3, 1]](%StatefulPartitionedCall/sequential/max_pooling2d_2/MaxPool:0) %StatefulPartitionedCall/sequential/flatten/Reshape:0 = Reshape(%StatefulPartitionedCall/sequential/max_pooling2d_2/MaxPool__28:0, %const_fold_opt__32) %StatefulPartitionedCall/sequential/dense/MatMul:0 = MatMul(%StatefulPartitionedCall/sequential/flatten/Reshape:0, %StatefulPartitionedCall/sequential/dense/MatMul/ReadVariableOp:0) %StatefulPartitionedCall/sequential/dense/BiasAdd:0 = Add(%StatefulPartitionedCall/sequential/dense/MatMul:0, %StatefulPartitionedCall/sequential/dense/BiasAdd/ReadVariableOp:0) %StatefulPartitionedCall/sequential/dense/Relu:0 = Relu(%StatefulPartitionedCall/sequential/dense/BiasAdd:0) %StatefulPartitionedCall/sequential/dense_1/MatMul:0 = MatMul(%StatefulPartitionedCall/sequential/dense/Relu:0, %StatefulPartitionedCall/sequential/dense_1/MatMul/ReadVariableOp:0) %StatefulPartitionedCall/sequential/dense_1/BiasAdd:0 = Add(%StatefulPartitionedCall/sequential/dense_1/MatMul:0, %StatefulPartitionedCall/sequential/dense_1/BiasAdd/ReadVariableOp:0) %dense_1 = Softmax(%StatefulPartitionedCall/sequential/dense_1/BiasAdd:0) return %dense_1}</code></pre><h4 id="加载pickle数据集"><a href="#加载pickle数据集" class="headerlink" title="加载pickle数据集"></a>加载pickle数据集</h4><h5 id="代码-1"><a href="#代码-1" class="headerlink" title="代码"></a>代码</h5><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> pickle<br><br><span class="hljs-comment"># 加载校准数据集</span><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'X_cal.pkl'</span>, <span class="hljs-string">'rb'</span>) <span class="hljs-keyword">as</span> f:<br> test_images = pickle.load(f)<br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'y_cal.pkl'</span>, <span class="hljs-string">'rb'</span>) <span class="hljs-keyword">as</span> f:<br> test_labels = pickle.load(f)<br><br><span class="hljs-comment"># 提取校准数据子集</span><br>calib_dataset = test_images[<span class="hljs-number">0</span>:<span class="hljs-number">1800</span>:<span class="hljs-number">20</span>]<br>pickle_file_path = <span class="hljs-string">'handrecognition_calib.pickle'</span><br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"Calibration dataset loaded and subset selected."</span>)<br><br></code></pre></td></tr></table></figure><h5 id="输出-1"><a href="#输出-1" class="headerlink" title="输出"></a>输出</h5><pre><code class="hljs">Calibration dataset loaded and subset selected.</code></pre><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 加载优化后的模型</span><br>model_proto = onnx.load(optimized_model_path)<br><span class="hljs-built_in">print</span>(<span class="hljs-string">'Generating the quantization table:'</span>)<br><br><span class="hljs-comment"># 初始化校准器</span><br>calib = Calibrator(<span class="hljs-string">'int16'</span>, <span class="hljs-string">'per-tensor'</span>, <span class="hljs-string">'minmax'</span>)<br><span class="hljs-comment"># 如果需要 int8 量化，可以使用以下替代代码：</span><br><span class="hljs-comment"># calib = Calibrator('int8', 'per-channel', 'minmax')</span><br><br><span class="hljs-comment"># 设置执行设备</span><br>calib.set_providers([<span class="hljs-string">'CPUExecutionProvider'</span>])<br></code></pre></td></tr></table></figure><h3 id="报错部分"><a href="#报错部分" class="headerlink" title="报错部分"></a>报错部分</h3><h4 id="代码-2"><a href="#代码-2" class="headerlink" title="代码"></a>代码</h4><p>就是这部分一直报错，不管是使用自己的1通道、10种代码，还是使用自己的3通道、6种代码，甚至直接运行原作者的model_development文件夹的model_development.ipynb和esp_dl_formate_conversionw文件夹的esp_dl_formate_conversion.py，也是失败，显示<code>ValueError: current model is not supported by esp-dl</code>。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 生成量化参数</span><br>calib.generate_quantization_table(model_proto, calib_dataset, pickle_file_path)<br><br><span class="hljs-comment"># 导出系数文件</span><br>calib.export_coefficient_to_cpp(model_proto, pickle_file_path, <span class="hljs-string">'esp32s3'</span>, <span class="hljs-string">'.'</span>, <span class="hljs-string">'handrecognition_coefficient'</span>, <span class="hljs-literal">True</span>)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"Calibration and quantization completed successfully."</span>)<br><br></code></pre></td></tr></table></figure><h4 id="报错输出"><a href="#报错输出" class="headerlink" title="报错输出"></a>报错输出</h4><pre><code class="hljs">Generating the quantization table:MatMul is not supported on esp-dl yet---------------------------------------------------------------------------ValueError Traceback (most recent call last)/tmp/ipykernel_107082/766837025.py in <module> 12 13 # 生成量化参数---> 14 calib.generate_quantization_table(model_proto, calib_dataset, pickle_file_path) 15 16 # 导出系数文件~/code/gesture recognition/calibrator.so in calibrator.Calibrator.generate_quantization_table()~/code/gesture recognition/calibrator.so in calibrator.Calibrator.generate_output_model()~/code/gesture recognition/calibrator.so in calibrator.Calibrator.check_model()（执行到calibrator.Calibrator.check_model的部分有问题）ValueError: current model is not supported by esp-dl</code></pre><h3 id="尝试过的检验方向（均已失败）"><a href="#尝试过的检验方向（均已失败）" class="headerlink" title="尝试过的检验方向（均已失败）"></a>尝试过的检验方向（均已失败）</h3><h4 id="模型有效性"><a href="#模型有效性" class="headerlink" title="模型有效性"></a>模型有效性</h4><h5 id="代码-3"><a href="#代码-3" class="headerlink" title="代码"></a>代码</h5><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> onnx<br>model_proto = onnx.load(optimized_model_path)<br>onnx.checker.check_model(model_proto) <span class="hljs-comment"># 验证 ONNX 模型是否有效</span><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"Model loaded and verified successfully."</span>)<br><br></code></pre></td></tr></table></figure><h5 id="输出-2"><a href="#输出-2" class="headerlink" title="输出"></a>输出</h5><pre><code class="hljs">Model loaded and verified successfully.</code></pre><h4 id="数据形状是否符合"><a href="#数据形状是否符合" class="headerlink" title="数据形状是否符合"></a>数据形状是否符合</h4><h5 id="代码-4"><a href="#代码-4" class="headerlink" title="代码"></a>代码</h5><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-built_in">print</span>(<span class="hljs-built_in">type</span>(calib_dataset))<br><span class="hljs-built_in">print</span>(calib_dataset.shape) <span class="hljs-comment"># 确认它的形状</span><br><span class="hljs-built_in">print</span>(calib_dataset[:<span class="hljs-number">5</span>]) <span class="hljs-comment"># 查看部分数据是否符合预期</span><br><br></code></pre></td></tr></table></figure><h5 id="输出-3"><a href="#输出-3" class="headerlink" title="输出"></a>输出</h5><pre><code class="hljs"><class 'numpy.ndarray'>(90, 96, 96, 1)[[[[0.01176471] [0.01568627] [0.01568627] ... [0.02352941] [0.01960784] [0.01960784]] [[0.01568627] [0.01960784] [0.01568627] ... [0.01960784] [0.01568627] [0.01176471]] [[0.01960784] [0.01960784] [0.01568627] ... [0.01568627] [0.01568627] [0.01176471]] ... [[0.04313725] [0.07058824] [0.08627451] ... [0.01568627] [0.01568627] [0.01568627]] [[0.04705882] [0.07058824] [0.07843137] ... [0.01960784] [0.01960784] [0.01568627]] [[0.04705882] [0.07058824] [0.07843137] ... [0.01960784] [0.01568627] [0.01568627]]] [[[0.01960784] [0.02352941] [0.01960784] ... [0.01960784] [0.01568627] [0.01568627]] [[0.01568627] [0.01568627] [0.01960784] ... [0.01568627] [0.01960784] [0.01176471]] [[0.01960784] [0.01960784] [0.02352941] ... [0.01568627] [0.01568627] [0.01568627]] ... [[0.01960784] [0.01960784] [0.01960784] ... [0.01960784] [0.01960784] [0.01960784]] [[0.01568627] [0.01960784] [0.01960784] ... [0.01176471] [0.01568627] [0.01568627]] [[0.01960784] [0.01960784] [0.01960784] ... [0.01960784] [0.01960784] [0.01568627]]] [[[0.02352941] [0.01960784] [0.02352941] ... [0.02352941] [0.02352941] [0.02352941]] [[0.02352941] [0.02352941] [0.02745098] ... [0.02352941] [0.02352941] [0.01568627]] [[0.02352941] [0.02352941] [0.01960784] ... [0.01960784] [0.01960784] [0.02352941]] ... [[0.01960784] [0.01960784] [0.02352941] ... [0.02352941] [0.01960784] [0.01960784]] [[0.01568627] [0.01960784] [0.01960784] ... [0.02352941] [0.02352941] [0.02745098]] [[0.02352941] [0.02352941] [0.02352941] ... [0.02352941] [0.02352941] [0.03137255]]] [[[0.01568627] [0.01960784] [0.01960784] ... [0.01568627] [0.01960784] [0.01568627]] [[0.01176471] [0.01960784] [0.01568627] ... [0.01568627] [0.01568627] [0.01176471]] [[0.01960784] [0.01960784] [0.01960784] ... [0.01176471] [0.01568627] [0.01568627]] ... [[0.06666667] [0.05882353] [0.06666667] ... [0.01960784] [0.01568627] [0.01568627]] [[0.05882353] [0.05490196] [0.05882353] ... [0.01568627] [0.01568627] [0.01568627]] [[0.05098039] [0.05098039] [0.0627451 ] ... [0.01960784] [0.01568627] [0.02352941]]] [[[0.02745098] [0.01960784] [0.02352941] ... [0.01568627] [0.01568627] [0.01568627]] [[0.01960784] [0.01960784] [0.02352941] ... [0.01960784] [0.01568627] [0.01960784]] [[0.01568627] [0.01960784] [0.01960784] ... [0.01568627] [0.01568627] [0.01568627]] ... [[0.01568627] [0.01568627] [0.01960784] ... [0.01568627] [0.01960784] [0.01568627]] [[0.02352941] [0.01960784] [0.01960784] ... [0.01568627] [0.01960784] [0.01568627]] [[0.01960784] [0.01568627] [0.01568627] ... [0.01960784] [0.01568627] [0.02352941]]]]</code></pre><h4 id="pickel路径有效性"><a href="#pickel路径有效性" class="headerlink" title="pickel路径有效性"></a>pickel路径有效性</h4><h5 id="代码-5"><a href="#代码-5" class="headerlink" title="代码"></a>代码</h5><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> os<br><br><span class="hljs-comment"># 设置文件路径</span><br>pickle_file_path = <span class="hljs-string">'output/handrecognition_calib_params.pickle'</span><br><br><span class="hljs-comment"># 检查路径是否存在，不存在则创建</span><br>output_dir = os.path.dirname(pickle_file_path)<br><span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> os.path.exists(output_dir):<br> os.makedirs(output_dir)<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">f"Created directory: <span class="hljs-subst">{output_dir}</span>"</span>)<br><br><span class="hljs-comment"># 检查是否可写</span><br>is_writeable = os.access(output_dir, os.W_OK)<br><span class="hljs-built_in">print</span>(<span class="hljs-string">"Writable:"</span>, is_writeable)<br><br></code></pre></td></tr></table></figure><h5 id="输出-4"><a href="#输出-4" class="headerlink" title="输出"></a>输出</h5><pre><code class="hljs">Writable: True</code></pre><p>在第一次显示不可写并且创建了文件夹以后，仍然无法被esp-dl支持，因此也不是读写性的问题。</p><h2 id="原作者代码运行证明失败"><a href="#原作者代码运行证明失败" class="headerlink" title="原作者代码运行证明失败"></a>原作者代码运行证明失败</h2><p>这部分我把原作者esp_dl_formate_conversion.py拆开到jupyter notebook中创建ipynb（当然前面模型训练部分也是直接使用model_development.ipynb），一步一步执行，最终同样卡在了同样的位置。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment">#import libraries </span><br><span class="hljs-keyword">from</span> optimizer <span class="hljs-keyword">import</span> *<br><span class="hljs-keyword">from</span> calibrator <span class="hljs-keyword">import</span> *<br><span class="hljs-keyword">from</span> evaluator <span class="hljs-keyword">import</span> *<br><span class="hljs-keyword">import</span> pickle<br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment">#load ONNX model </span><br>onnx_model = onnx.load(<span class="hljs-string">"handrecognition_model.onnx"</span>)<br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment">#optimize ONNX model </span><br>optimized_model_path = optimize_fp_model(<span class="hljs-string">"handrecognition_model.onnx"</span>)<br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment">#load calibration dataset</span><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'X_cal.pkl'</span>, <span class="hljs-string">'rb'</span>) <span class="hljs-keyword">as</span> f:<br> (test_images) = pickle.load(f)<br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'y_cal.pkl'</span>, <span class="hljs-string">'rb'</span>) <span class="hljs-keyword">as</span> f:<br> (test_labels) = pickle.load(f)<br><br>calib_dataset = test_images[<span class="hljs-number">0</span>:<span class="hljs-number">1800</span>:<span class="hljs-number">20</span>]<br>pickle_file_path = <span class="hljs-string">'handrecognition_calib.pickle'</span><br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment">#calibration </span><br>model_proto = onnx.load(optimized_model_path)<br><span class="hljs-built_in">print</span>(<span class="hljs-string">'Generating the quantization table:'</span>)<br>calib = Calibrator(<span class="hljs-string">'int16'</span>, <span class="hljs-string">'per-tensor'</span>, <span class="hljs-string">'minmax'</span>)<br></code></pre></td></tr></table></figure><pre><code class="hljs">Generating the quantization table:</code></pre><h3 id="出错位置"><a href="#出错位置" class="headerlink" title="出错位置"></a>出错位置</h3><h4 id="代码-6"><a href="#代码-6" class="headerlink" title="代码"></a>代码</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment">###for in8 conversion </span><br><span class="hljs-comment">#calib = Calibrator('int8', 'per-channel', 'minmax') </span><br><br>calib.set_providers([<span class="hljs-string">'CPUExecutionProvider'</span>])<br><span class="hljs-comment"># Obtain the quantization parameter</span><br>calib.generate_quantization_table(model_proto,calib_dataset, pickle_file_path)<br><span class="hljs-comment"># Generate the coefficient files for esp32s3</span><br>calib.export_coefficient_to_cpp(model_proto, pickle_file_path, <span class="hljs-string">'esp32s3'</span>, <span class="hljs-string">'.'</span>, <span class="hljs-string">'handrecognition_coefficient'</span>, <span class="hljs-literal">True</span>)<br><br></code></pre></td></tr></table></figure><h4 id="输出-5"><a href="#输出-5" class="headerlink" title="输出"></a>输出</h4><pre><code class="hljs">MatMul is not supported on esp-dl yet---------------------------------------------------------------------------ValueError Traceback (most recent call last)/tmp/ipykernel_102222/2534009994.py in <module> 5 calib.set_providers(['CPUExecutionProvider']) 6 # Obtain the quantization parameter----> 7 calib.generate_quantization_table(model_proto,calib_dataset, pickle_file_path) 8 # Generate the coefficient files for esp32s3 9 calib.export_coefficient_to_cpp(model_proto, pickle_file_path, 'esp32s3', '.', 'handrecognition_coefficient', True)~/code/Blogs/ESP-DL/esp_dl_formate_conversion/calibrator.so in calibrator.Calibrator.generate_quantization_table()~/code/Blogs/ESP-DL/esp_dl_formate_conversion/calibrator.so in calibrator.Calibrator.generate_output_model()~/code/Blogs/ESP-DL/esp_dl_formate_conversion/calibrator.so in calibrator.Calibrator.check_model()ValueError: current model is not supported by esp-dl</code></pre><p>最终，感谢在薛兄的提醒下，发现有人给作者提过这个问题，但作者至今也并未解决：<a href="https://github.com/alibukharai/Blogs/issues/9">https://github.com/alibukharai/Blogs/issues/9</a><br><img src="/img/learn/2024-11-05172636.png"></p><h2 id="考虑可能成功的新方向"><a href="#考虑可能成功的新方向" class="headerlink" title="考虑可能成功的新方向"></a>考虑可能成功的新方向</h2><ul><li>在windows中搭conda尝试，因为之前只在ubuntu 24.04中尝试过，有可能esp-dl官方库的.pyd能成功，只是.so文件有问题。</li><li>更换esp-idf和esp-dl的branch和tag，因为目前使用的是v5.3的idf和release/v1.1的dl，原作者采用的是v4.4的idf和idf4.4的idf。</li><li>使用其他的优化和量化模型的工具，esp-dl确实不是很完善。</li></ul>]]></content>
<categories>
<category>fyp</category>
</categories>
</entry>
<entry>
<title>building_with_espdl-1-supplement</title>
<link href="/2024/11/05/generate_calibration_data/"/>
<url>/2024/11/05/generate_calibration_data/</url>
<content type="html"><![CDATA[<h1 id="building-with-espdl-1-supplement"><a href="#building-with-espdl-1-supplement" class="headerlink" title="building_with_espdl-1-supplement"></a>building_with_espdl-1-supplement</h1><p>因为模型优化和量化部分需要X_cal.pkl和y_cal.pkl文件，而在第一部分训练模型时未生成，因此新建一个项目生成pkl。</p><h2 id="初版"><a href="#初版" class="headerlink" title="初版"></a>初版</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> tensorflow <span class="hljs-keyword">as</span> tf<br><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np<br><span class="hljs-keyword">import</span> pickle<br><br><span class="hljs-comment"># 设置图片的尺寸和批量大小</span><br>img_height, img_width = <span class="hljs-number">96</span>, <span class="hljs-number">96</span><br>batch_size = <span class="hljs-number">16</span><br><br><span class="hljs-comment"># 数据集根目录</span><br>data_dir = <span class="hljs-string">"./leapGestRecog"</span> <span class="hljs-comment"># 请替换为你的实际路径</span><br><br><span class="hljs-comment"># 加载和预处理数据集</span><br>train_ds = tf.keras.preprocessing.image_dataset_from_directory(<br> data_dir,<br> validation_split=<span class="hljs-number">0.2</span>,<br> subset=<span class="hljs-string">"training"</span>,<br> seed=<span class="hljs-number">123</span>,<br> image_size=(img_height, img_width),<br> batch_size=batch_size<br>)<br><br>val_ds = tf.keras.preprocessing.image_dataset_from_directory(<br> data_dir,<br> validation_split=<span class="hljs-number">0.2</span>,<br> subset=<span class="hljs-string">"validation"</span>,<br> seed=<span class="hljs-number">123</span>,<br> image_size=(img_height, img_width),<br> batch_size=batch_size<br>)<br><br><span class="hljs-comment"># 创建用于保存数据的列表</span><br>X_cal = []<br>y_cal = []<br><br><span class="hljs-comment"># 遍历训练集，将图像和标签存储到列表中</span><br><span class="hljs-keyword">for</span> images, labels <span class="hljs-keyword">in</span> train_ds:<br> X_cal.append(images.numpy())<br> y_cal.append(labels.numpy())<br><br><span class="hljs-comment"># 遍历验证集，将图像和标签存储到列表中</span><br><span class="hljs-keyword">for</span> images, labels <span class="hljs-keyword">in</span> val_ds:<br> X_cal.append(images.numpy())<br> y_cal.append(labels.numpy())<br><br><span class="hljs-comment"># 将列表转换为 NumPy 数组</span><br>X_cal = np.concatenate(X_cal)<br>y_cal = np.concatenate(y_cal)<br><br><span class="hljs-comment"># 保存为 pickle 文件</span><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'X_cal.pkl'</span>, <span class="hljs-string">'wb'</span>) <span class="hljs-keyword">as</span> f:<br> pickle.dump(X_cal, f)<br><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'y_cal.pkl'</span>, <span class="hljs-string">'wb'</span>) <span class="hljs-keyword">as</span> f:<br> pickle.dump(y_cal, f)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"X_cal.pkl 和 y_cal.pkl 已成功创建。"</span>)<br><br></code></pre></td></tr></table></figure><h2 id="次版"><a href="#次版" class="headerlink" title="次版"></a>次版</h2><h3 id="1"><a href="#1" class="headerlink" title="1"></a>1</h3><p>原因同模型训练部分，因此修改3通道改为1通道。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> tensorflow <span class="hljs-keyword">as</span> tf<br><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np<br><span class="hljs-keyword">import</span> pickle<br><br><span class="hljs-comment"># 设置图片的尺寸和批量大小</span><br>img_height, img_width = <span class="hljs-number">96</span>, <span class="hljs-number">96</span><br>batch_size = <span class="hljs-number">16</span><br><br><span class="hljs-comment"># 数据集根目录</span><br>data_dir = <span class="hljs-string">"./leapGestRecog"</span> <span class="hljs-comment"># 请替换为你的实际路径</span><br><br><span class="hljs-comment"># 加载和预处理数据集</span><br>train_ds = tf.keras.preprocessing.image_dataset_from_directory(<br> data_dir,<br> validation_split=<span class="hljs-number">0.2</span>,<br> subset=<span class="hljs-string">"training"</span>,<br> seed=<span class="hljs-number">123</span>,<br> image_size=(img_height, img_width),<br> batch_size=batch_size<br>)<br><br>val_ds = tf.keras.preprocessing.image_dataset_from_directory(<br> data_dir,<br> validation_split=<span class="hljs-number">0.2</span>,<br> subset=<span class="hljs-string">"validation"</span>,<br> seed=<span class="hljs-number">123</span>,<br> image_size=(img_height, img_width),<br> batch_size=batch_size<br>)<br><br><span class="hljs-comment"># 转换为灰度图并归一化</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">normalize_img</span>(<span class="hljs-params">image, label</span>):<br> image = tf.image.rgb_to_grayscale(image) <span class="hljs-comment"># 转换为灰度</span><br> image = tf.cast(image, tf.float32) / <span class="hljs-number">255.0</span> <span class="hljs-comment"># 归一化到 [0, 1]</span><br> <span class="hljs-keyword">return</span> image, label<br><br><span class="hljs-comment"># 应用转换函数到数据集</span><br>train_ds = train_ds.<span class="hljs-built_in">map</span>(normalize_img)<br>val_ds = val_ds.<span class="hljs-built_in">map</span>(normalize_img)<br><br><span class="hljs-comment"># 创建用于保存数据的列表</span><br>X_cal = []<br>y_cal = []<br><br><span class="hljs-comment"># 遍历训练集，将图像和标签存储到列表中</span><br><span class="hljs-keyword">for</span> images, labels <span class="hljs-keyword">in</span> train_ds:<br> X_cal.append(images.numpy())<br> y_cal.append(labels.numpy())<br><br><span class="hljs-comment"># 遍历验证集，将图像和标签存储到列表中</span><br><span class="hljs-keyword">for</span> images, labels <span class="hljs-keyword">in</span> val_ds:<br> X_cal.append(images.numpy())<br> y_cal.append(labels.numpy())<br><br><span class="hljs-comment"># 将列表转换为 NumPy 数组</span><br>X_cal = np.concatenate(X_cal)<br>y_cal = np.concatenate(y_cal)<br><br><span class="hljs-comment"># 保存为 pickle 文件</span><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'X_cal.pkl'</span>, <span class="hljs-string">'wb'</span>) <span class="hljs-keyword">as</span> f:<br> pickle.dump(X_cal, f)<br><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'y_cal.pkl'</span>, <span class="hljs-string">'wb'</span>) <span class="hljs-keyword">as</span> f:<br> pickle.dump(y_cal, f)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"X_cal.pkl 和 y_cal.pkl 已成功创建。"</span>)<br><br></code></pre></td></tr></table></figure><h3 id="2"><a href="#2" class="headerlink" title="2"></a>2</h3><h4 id="代码"><a href="#代码" class="headerlink" title="代码"></a>代码</h4><p>手动遍历文件系统以加载数据，仅选择前 6 个手势类别。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> os<br><span class="hljs-keyword">import</span> cv2<br><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np<br><span class="hljs-keyword">import</span> pickle<br><br><span class="hljs-comment"># 设置数据集目录和参数</span><br>data_dir = <span class="hljs-string">"./leapGestRecog"</span> <span class="hljs-comment"># 请替换为你的实际路径</span><br>img_height, img_width = <span class="hljs-number">96</span>, <span class="hljs-number">96</span><br><br><span class="hljs-comment"># 定义需要加载的手势类别（仅加载前6类）</span><br>selected_gestures = [<span class="hljs-string">"01_palm"</span>, <span class="hljs-string">"02_l"</span>, <span class="hljs-string">"03_fist"</span>, <span class="hljs-string">"04_fist_moved"</span>, <span class="hljs-string">"05_thumb"</span>, <span class="hljs-string">"06_index"</span>]<br><br><span class="hljs-comment"># 初始化数据和标签列表</span><br>X_cal = []<br>y_cal = []<br><br><span class="hljs-comment"># 遍历每个顶层文件夹（例如 00, 01...）</span><br><span class="hljs-keyword">for</span> subdir <span class="hljs-keyword">in</span> os.listdir(data_dir):<br> subdir_path = os.path.join(data_dir, subdir)<br> <br> <span class="hljs-comment"># 确保顶层文件夹是目录</span><br> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> os.path.isdir(subdir_path):<br> <span class="hljs-keyword">continue</span><br> <br> <span class="hljs-comment"># 仅遍历前六个手势类别文件夹</span><br> <span class="hljs-keyword">for</span> idx, gesture <span class="hljs-keyword">in</span> <span class="hljs-built_in">enumerate</span>(selected_gestures):<br> gesture_path = os.path.join(subdir_path, gesture)<br> <br> <span class="hljs-comment"># 确保手势文件夹存在</span><br> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> os.path.isdir(gesture_path):<br> <span class="hljs-keyword">continue</span><br> <br> <span class="hljs-comment"># 遍历每张图像文件</span><br> <span class="hljs-keyword">for</span> img_file <span class="hljs-keyword">in</span> os.listdir(gesture_path):<br> img_path = os.path.join(gesture_path, img_file)<br> <br> <span class="hljs-comment"># 读取图像，转换为灰度，并调整大小</span><br> img = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)<br> img_resized = cv2.resize(img, (img_width, img_height))<br> <br> <span class="hljs-comment"># 归一化图像数据</span><br> img_normalized = img_resized / <span class="hljs-number">255.0</span><br> X_cal.append(img_normalized)<br> y_cal.append(idx) <span class="hljs-comment"># 使用手势的索引作为标签</span><br><br><span class="hljs-comment"># 转换为 NumPy 数组，并调整维度</span><br>X_cal = np.array(X_cal).reshape(-<span class="hljs-number">1</span>, img_height, img_width, <span class="hljs-number">1</span>)<br>y_cal = np.array(y_cal)<br><br><span class="hljs-comment"># 保存为 pickle 文件</span><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'X_cal.pkl'</span>, <span class="hljs-string">'wb'</span>) <span class="hljs-keyword">as</span> f:<br> pickle.dump(X_cal, f)<br><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'y_cal.pkl'</span>, <span class="hljs-string">'wb'</span>) <span class="hljs-keyword">as</span> f:<br> pickle.dump(y_cal, f)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"X_cal.pkl 和 y_cal.pkl 已成功创建，包含前6个手势类别的数据。"</span>)<br><br></code></pre></td></tr></table></figure><h4 id="输出"><a href="#输出" class="headerlink" title="输出"></a>输出</h4><pre><code class="hljs">X_cal.pkl 和 y_cal.pkl 已成功创建，包含前6个手势类别的数据。</code></pre>]]></content>
<categories>
<category>fyp</category>
</categories>
</entry>
<entry>
<title>building_with_espdl-1-success</title>
<link href="/2024/11/02/chatgpt_version1/"/>
<url>/2024/11/02/chatgpt_version1/</url>
<content type="html"><![CDATA[<h1 id="building-with-espdl-1-success"><a href="#building-with-espdl-1-success" class="headerlink" title="building_with_espdl-1-success"></a>building_with_espdl-1-success</h1><p>基本完成原文第1部分。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># Import necessary libraries</span><br><span class="hljs-keyword">import</span> tensorflow <span class="hljs-keyword">as</span> tf<br><span class="hljs-keyword">from</span> keras.models <span class="hljs-keyword">import</span> Sequential<br><span class="hljs-keyword">from</span> keras.layers <span class="hljs-keyword">import</span> Conv2D, MaxPooling2D, Flatten, Dense<br><span class="hljs-keyword">from</span> keras.utils <span class="hljs-keyword">import</span> to_categorical<br><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np<br><span class="hljs-keyword">import</span> pandas <span class="hljs-keyword">as</span> pd<br><br><span class="hljs-comment"># Load the dataset (you may need to download it manually from Kaggle and upload to Colab)</span><br><span class="hljs-comment"># Assuming you have a folder 'gestures' with images sorted in subdirectories</span><br>data_dir = <span class="hljs-string">"./leapgestrecog/leapGestRecog"</span> <span class="hljs-comment"># 修改此处，指定手势数据集的路径</span><br><br></code></pre></td></tr></table></figure><h2 id="处理数据集部分"><a href="#处理数据集部分" class="headerlink" title="处理数据集部分"></a>处理数据集部分</h2><h3 id="无用1"><a href="#无用1" class="headerlink" title="无用1"></a>无用1</h3><p>这一段代码可无需理会，因为在和gpt交流的过程中，明白了pkl也只是原作者切割并且整理好的数据集，我这里可以完全不采用，并且原作中导入数据集的方法较为繁琐，此处也有更简单的方法</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> os<br><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np<br><span class="hljs-keyword">from</span> sklearn.model_selection <span class="hljs-keyword">import</span> train_test_split<br><span class="hljs-keyword">from</span> PIL <span class="hljs-keyword">import</span> Image<br><span class="hljs-keyword">from</span> keras.preprocessing.image <span class="hljs-keyword">import</span> img_to_array<br><br><span class="hljs-comment"># 数据集根目录</span><br>data_dir = <span class="hljs-string">"./leapgestrecog/leapGestRecog"</span> <span class="hljs-comment"># 请替换为你的实际路径</span><br><br><span class="hljs-comment"># 定义类别标签映射（根据你的数据集的第二层目录进行映射）</span><br>class_labels = {<br> <span class="hljs-string">'01_palm'</span>: <span class="hljs-number">0</span>,<br> <span class="hljs-string">'02_l'</span>: <span class="hljs-number">1</span>,<br> <span class="hljs-string">'03_fist'</span>: <span class="hljs-number">2</span>,<br> <span class="hljs-string">'04_fist_moved'</span>: <span class="hljs-number">3</span>,<br> <span class="hljs-string">'05_thumb'</span>: <span class="hljs-number">4</span>,<br> <span class="hljs-string">'06_index'</span>: <span class="hljs-number">5</span>,<br> <span class="hljs-string">'07_ok'</span>: <span class="hljs-number">6</span>,<br> <span class="hljs-string">'08_palm_moved'</span>: <span class="hljs-number">7</span>,<br> <span class="hljs-string">'09_c'</span>: <span class="hljs-number">8</span>,<br> <span class="hljs-string">'10_down'</span>: <span class="hljs-number">9</span><br>}<br><br><span class="hljs-comment"># 初始化图像路径和标签的列表</span><br>image_paths = []<br>labels = []<br><br><span class="hljs-comment"># 遍历第一层文件夹</span><br><span class="hljs-keyword">for</span> folder <span class="hljs-keyword">in</span> os.listdir(data_dir):<br> folder_path = os.path.join(data_dir, folder)<br> <span class="hljs-keyword">if</span> os.path.isdir(folder_path): <span class="hljs-comment"># 确保是文件夹</span><br> <span class="hljs-comment"># 遍历第二层文件夹</span><br> <span class="hljs-keyword">for</span> class_name, label <span class="hljs-keyword">in</span> class_labels.items():<br> class_folder_path = os.path.join(folder_path, class_name)<br> <span class="hljs-keyword">if</span> os.path.isdir(class_folder_path):<br> <span class="hljs-comment"># 遍历图像文件</span><br> <span class="hljs-keyword">for</span> img_file <span class="hljs-keyword">in</span> os.listdir(class_folder_path):<br> img_path = os.path.join(class_folder_path, img_file)<br> image_paths.append(img_path)<br> labels.append(label)<br><br><span class="hljs-comment"># 设置图像尺寸</span><br>img_height, img_width = <span class="hljs-number">96</span>, <span class="hljs-number">96</span><br><br><span class="hljs-comment"># 加载图像并转换为数组</span><br>images = []<br><span class="hljs-keyword">for</span> path <span class="hljs-keyword">in</span> image_paths:<br> <span class="hljs-keyword">try</span>:<br> img = Image.<span class="hljs-built_in">open</span>(path).convert(<span class="hljs-string">'RGB'</span>)<br> img = img.resize((img_width, img_height))<br> img_array = img_to_array(img) / <span class="hljs-number">255.0</span> <span class="hljs-comment"># 归一化</span><br> images.append(img_array)<br> <span class="hljs-keyword">except</span> Exception <span class="hljs-keyword">as</span> e:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">f"无法处理图像 <span class="hljs-subst">{path}</span>: <span class="hljs-subst">{e}</span>"</span>)<br><br><span class="hljs-comment"># 转换为 NumPy 数组</span><br>X = np.array(images)<br>y = np.array(labels)<br><br><span class="hljs-comment"># 划分训练集、验证集和测试集</span><br>ts = <span class="hljs-number">0.3</span> <span class="hljs-comment"># 30% 的数据作为测试集</span><br>X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=ts, random_state=<span class="hljs-number">42</span>)<br>X_test, X_cal, y_test, y_cal = train_test_split(X_temp, y_temp, test_size=<span class="hljs-number">0.5</span>, random_state=<span class="hljs-number">42</span>)<br><br><span class="hljs-comment"># 检查数据集形状</span><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"训练集样本数:"</span>, X_train.shape[<span class="hljs-number">0</span>])<br><span class="hljs-built_in">print</span>(<span class="hljs-string">"测试集样本数:"</span>, X_test.shape[<span class="hljs-number">0</span>])<br><span class="hljs-built_in">print</span>(<span class="hljs-string">"校准集样本数:"</span>, X_cal.shape[<span class="hljs-number">0</span>])<br><br></code></pre></td></tr></table></figure><h3 id="简化的切割、标注、导入数据集的方法"><a href="#简化的切割、标注、导入数据集的方法" class="headerlink" title="简化的切割、标注、导入数据集的方法"></a>简化的切割、标注、导入数据集的方法</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 设置图片的尺寸和批量大小</span><br>img_height, img_width = <span class="hljs-number">96</span>, <span class="hljs-number">96</span> <span class="hljs-comment"># 按照文档中的建议将图像调整为 (96, 96)</span><br>batch_size = <span class="hljs-number">16</span><br><br><span class="hljs-comment"># 加载和预处理数据集</span><br>train_ds = tf.keras.preprocessing.image_dataset_from_directory(<br> data_dir,<br> validation_split=<span class="hljs-number">0.2</span>, <span class="hljs-comment"># 将20%数据作为验证集</span><br> subset=<span class="hljs-string">"training"</span>,<br> seed=<span class="hljs-number">123</span>, <span class="hljs-comment"># 保证数据划分的可重复性</span><br> image_size=(img_height, img_width),<br> batch_size=batch_size<br>)<br><br>val_ds = tf.keras.preprocessing.image_dataset_from_directory(<br> data_dir,<br> validation_split=<span class="hljs-number">0.2</span>,<br> subset=<span class="hljs-string">"validation"</span>,<br> seed=<span class="hljs-number">123</span>,<br> image_size=(img_height, img_width),<br> batch_size=batch_size<br>)<br><br><span class="hljs-comment"># 缓存和优化数据加载，以提高训练速度</span><br>AUTOTUNE = tf.data.AUTOTUNE<br>train_ds = train_ds.cache().shuffle(<span class="hljs-number">1000</span>).prefetch(buffer_size=AUTOTUNE)<br>val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)<br><br></code></pre></td></tr></table></figure><h3 id="像素值必须归一化，否则无法收敛"><a href="#像素值必须归一化，否则无法收敛" class="headerlink" title="像素值必须归一化，否则无法收敛"></a>像素值必须归一化，否则无法收敛</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 插入归一化代码</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">normalize_img</span>(<span class="hljs-params">image, label</span>):<br> image = tf.cast(image, tf.float32) / <span class="hljs-number">255.0</span> <span class="hljs-comment"># 将像素值归一化到 [0, 1]</span><br> <span class="hljs-keyword">return</span> image, label<br><br><span class="hljs-comment"># 应用归一化到数据集</span><br>train_ds = train_ds.<span class="hljs-built_in">map</span>(normalize_img)<br>val_ds = val_ds.<span class="hljs-built_in">map</span>(normalize_img)<br></code></pre></td></tr></table></figure><h3 id="无用2，一开始是为了检查gpt声称简化的切割方法和原作者的方法得到的样本量差距大不大，若不大则可以使用简化方法"><a href="#无用2，一开始是为了检查gpt声称简化的切割方法和原作者的方法得到的样本量差距大不大，若不大则可以使用简化方法" class="headerlink" title="无用2，一开始是为了检查gpt声称简化的切割方法和原作者的方法得到的样本量差距大不大，若不大则可以使用简化方法"></a>无用2，一开始是为了检查gpt声称简化的切割方法和原作者的方法得到的样本量差距大不大，若不大则可以使用简化方法</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 检查样本数是否一致</span><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"使用 image_dataset_from_directory 分割的数据集："</span>)<br>train_count = tf.data.experimental.cardinality(train_ds).numpy() * batch_size<br>val_count = tf.data.experimental.cardinality(val_ds).numpy() * batch_size<br><span class="hljs-built_in">print</span>(<span class="hljs-string">f"训练集样本数: <span class="hljs-subst">{train_count}</span>"</span>)<br><span class="hljs-built_in">print</span>(<span class="hljs-string">f"验证集样本数: <span class="hljs-subst">{val_count}</span>"</span>)<br><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"使用 train_test_split 分割的数据集："</span>)<br><span class="hljs-built_in">print</span>(<span class="hljs-string">f"训练集样本数: <span class="hljs-subst">{<span class="hljs-built_in">len</span>(X_train)}</span>"</span>)<br><span class="hljs-built_in">print</span>(<span class="hljs-string">f"测试集样本数: <span class="hljs-subst">{<span class="hljs-built_in">len</span>(X_test)}</span>"</span>)<br><span class="hljs-built_in">print</span>(<span class="hljs-string">f"校准集样本数: <span class="hljs-subst">{<span class="hljs-built_in">len</span>(X_cal)}</span>"</span>)<br><br></code></pre></td></tr></table></figure><h2 id="cnn算法部分"><a href="#cnn算法部分" class="headerlink" title="cnn算法部分"></a>cnn算法部分</h2><p>*必须注意这里原作者只抽取了kaggle数据集中的6种，但我用了全部10种，一开始也直接复制代码写的<code>Dense(6</code>，但实际上应该是<code>Dense(10</code>。</p><h3 id="这一段是由gpt给出的版本，和原作者类似-属于链式调用"><a href="#这一段是由gpt给出的版本，和原作者类似-属于链式调用" class="headerlink" title="这一段是由gpt给出的版本，和原作者类似,属于链式调用"></a>这一段是由gpt给出的版本，和原作者类似,属于链式调用</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> keras.models <span class="hljs-keyword">import</span> Sequential<br><span class="hljs-keyword">from</span> keras.layers <span class="hljs-keyword">import</span> Conv2D, MaxPooling2D, Flatten, Dense, Dropout<br><br>model = Sequential([<br> Conv2D(<span class="hljs-number">32</span>, (<span class="hljs-number">3</span>, <span class="hljs-number">3</span>), activation=<span class="hljs-string">'relu'</span>, input_shape=(<span class="hljs-number">96</span>, <span class="hljs-number">96</span>, <span class="hljs-number">3</span>)),<br> MaxPooling2D(pool_size=(<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)),<br> Dropout(<span class="hljs-number">0.2</span>),<br><br> Conv2D(<span class="hljs-number">64</span>, (<span class="hljs-number">3</span>, <span class="hljs-number">3</span>), activation=<span class="hljs-string">'relu'</span>),<br> MaxPooling2D(pool_size=(<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)),<br> Dropout(<span class="hljs-number">0.2</span>),<br><br> Conv2D(<span class="hljs-number">128</span>, (<span class="hljs-number">3</span>, <span class="hljs-number">3</span>), activation=<span class="hljs-string">'relu'</span>),<br> MaxPooling2D(pool_size=(<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)),<br> Dropout(<span class="hljs-number">0.2</span>),<br><br> Flatten(),<br> Dense(<span class="hljs-number">128</span>, activation=<span class="hljs-string">'relu'</span>),<br> Dropout(<span class="hljs-number">0.2</span>),<br> Dense(<span class="hljs-number">10</span>, activation=<span class="hljs-string">'softmax'</span>)<br>])<br><br></code></pre></td></tr></table></figure><h3 id="无用3，这是原作者部分，属于逐层添加"><a href="#无用3，这是原作者部分，属于逐层添加" class="headerlink" title="无用3，这是原作者部分，属于逐层添加"></a>无用3，这是原作者部分，属于逐层添加</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> keras.models <span class="hljs-keyword">import</span> Sequential<br><span class="hljs-keyword">from</span> keras.layers <span class="hljs-keyword">import</span> Conv2D, MaxPooling2D, Flatten, Dense, Dropout<br><br>model = Sequential()<br>model.add(Conv2D(<span class="hljs-number">32</span>, (<span class="hljs-number">3</span>, <span class="hljs-number">3</span>), activation=<span class="hljs-string">'relu'</span>, input_shape=(<span class="hljs-number">96</span>, <span class="hljs-number">96</span>, <span class="hljs-number">3</span>)))<br>model.add(MaxPooling2D(pool_size=(<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)))<br>model.add(Dropout(<span class="hljs-number">0.2</span>))<br><br>model.add(Conv2D(<span class="hljs-number">64</span>, (<span class="hljs-number">3</span>, <span class="hljs-number">3</span>), activation=<span class="hljs-string">'relu'</span>))<br>model.add(MaxPooling2D(pool_size=(<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)))<br>model.add(Dropout(<span class="hljs-number">0.2</span>))<br><br>model.add(Conv2D(<span class="hljs-number">128</span>, (<span class="hljs-number">3</span>, <span class="hljs-number">3</span>), activation=<span class="hljs-string">'relu'</span>))<br>model.add(MaxPooling2D(pool_size=(<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)))<br>model.add(Dropout(<span class="hljs-number">0.2</span>))<br><br>model.add(Flatten())<br>model.add(Dense(<span class="hljs-number">128</span>, activation=<span class="hljs-string">'relu'</span>))<br>model.add(Dropout(<span class="hljs-number">0.2</span>))<br>model.add(Dense(<span class="hljs-number">10</span>, activation=<span class="hljs-string">'softmax'</span>))<br><br></code></pre></td></tr></table></figure><h3 id="这部分可以去掉，但输出结构摘要可以与原作者的过程相对比"><a href="#这部分可以去掉，但输出结构摘要可以与原作者的过程相对比" class="headerlink" title="这部分可以去掉，但输出结构摘要可以与原作者的过程相对比"></a>这部分可以去掉，但输出结构摘要可以与原作者的过程相对比</h3><p><strong>这里要强调的是原作者用的是96,96,1的灰度图（最后一个数字表示通道数），而本文采用的是96,96,3的三通道的彩色图，会使参数多一些，但影响不大。</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 输出模型结构摘要</span><br>model.summary()<br><br></code></pre></td></tr></table></figure><h2 id="模型的编译部分"><a href="#模型的编译部分" class="headerlink" title="模型的编译部分"></a>模型的编译部分</h2><h3 id="这一段是我采用的模型编译部分"><a href="#这一段是我采用的模型编译部分" class="headerlink" title="这一段是我采用的模型编译部分"></a>这一段是我采用的模型编译部分</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 编译模型</span><br>model.<span class="hljs-built_in">compile</span>(optimizer=<span class="hljs-string">'adam'</span>, loss=<span class="hljs-string">'sparse_categorical_crossentropy'</span>, metrics=[<span class="hljs-string">'accuracy'</span>])<br><br><span class="hljs-comment"># 模型训练</span><br>epochs = <span class="hljs-number">5</span> <span class="hljs-comment"># 设置训练轮数</span><br><br><span class="hljs-comment"># 使用 train_ds 和 val_ds 进行训练</span><br>history = model.fit(<br> train_ds, <span class="hljs-comment"># 使用训练集数据集对象</span><br> epochs=epochs,<br> validation_data=val_ds, <span class="hljs-comment"># 使用验证集数据集对象</span><br> verbose=<span class="hljs-number">1</span><br>)<br><br></code></pre></td></tr></table></figure><h3 id="无用4，这一段是修改原作者的"><a href="#无用4，这一段是修改原作者的" class="headerlink" title="无用4，这一段是修改原作者的"></a>无用4，这一段是修改原作者的</h3><p>因为从导入数据集那里就产生了区别，我在导入的过程中并没有产生X_train、y_train这类变量，因此也不使用这段代码。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 编译模型</span><br>model.<span class="hljs-built_in">compile</span>(optimizer=<span class="hljs-string">'adam'</span>, loss=<span class="hljs-string">'sparse_categorical_crossentropy'</span>, metrics=[<span class="hljs-string">'accuracy'</span>])<br><br><span class="hljs-comment"># 训练参数设置</span><br>epochs = <span class="hljs-number">10</span> <span class="hljs-comment"># 设置训练轮数，可以根据需求调整</span><br>batch_size = <span class="hljs-number">32</span> <span class="hljs-comment"># 每批次的样本数量</span><br><br><span class="hljs-comment"># 使用训练集和验证集进行训练</span><br>history = model.fit(<br> X_train, y_train, <span class="hljs-comment"># 训练集数据</span><br> epochs=epochs,<br> batch_size=batch_size,<br> validation_data=(X_test, y_test), <span class="hljs-comment"># 验证集数据</span><br> verbose=<span class="hljs-number">1</span><br>)<br><br></code></pre></td></tr></table></figure><h2 id="拍错部分，分别从不同可能导致精确度提升不上去的角度进行排查"><a href="#拍错部分，分别从不同可能导致精确度提升不上去的角度进行排查" class="headerlink" title="拍错部分，分别从不同可能导致精确度提升不上去的角度进行排查"></a>拍错部分，分别从不同可能导致精确度提升不上去的角度进行排查</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 查看训练集和验证集标签的示例数据类型</span><br><span class="hljs-keyword">for</span> images, labels <span class="hljs-keyword">in</span> train_ds.take(<span class="hljs-number">1</span>):<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"训练集标签数据类型:"</span>, labels.dtype)<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"训练集标签示例:"</span>, labels.numpy()[:<span class="hljs-number">10</span>])<br><br><span class="hljs-keyword">for</span> images, labels <span class="hljs-keyword">in</span> val_ds.take(<span class="hljs-number">1</span>):<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"验证集标签数据类型:"</span>, labels.dtype)<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"验证集标签示例:"</span>, labels.numpy()[:<span class="hljs-number">10</span>])<br><br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 查看训练集和验证集图像的示例数据范围</span><br><span class="hljs-keyword">for</span> images, labels <span class="hljs-keyword">in</span> train_ds.take(<span class="hljs-number">1</span>):<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"训练集图像数据最小值:"</span>, images.numpy().<span class="hljs-built_in">min</span>())<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"训练集图像数据最大值:"</span>, images.numpy().<span class="hljs-built_in">max</span>())<br><br><span class="hljs-keyword">for</span> images, labels <span class="hljs-keyword">in</span> val_ds.take(<span class="hljs-number">1</span>):<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"验证集图像数据最小值:"</span>, images.numpy().<span class="hljs-built_in">min</span>())<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"验证集图像数据最大值:"</span>, images.numpy().<span class="hljs-built_in">max</span>())<br><br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 查看训练集和验证集的结构</span><br><span class="hljs-built_in">print</span>(<span class="hljs-string">"训练集数据集对象:"</span>, train_ds)<br><span class="hljs-built_in">print</span>(<span class="hljs-string">"验证集数据集对象:"</span>, val_ds)<br><br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 在归一化之后，检查是否有 NaN 或 Inf 值</span><br><span class="hljs-keyword">for</span> images, labels <span class="hljs-keyword">in</span> train_ds.take(<span class="hljs-number">1</span>):<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"训练集图像中是否存在 NaN:"</span>, np.isnan(images.numpy()).<span class="hljs-built_in">any</span>())<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"训练集图像中是否存在 Inf:"</span>, np.isinf(images.numpy()).<span class="hljs-built_in">any</span>())<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"训练集标签中是否存在 NaN:"</span>, np.isnan(labels.numpy()).<span class="hljs-built_in">any</span>())<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"训练集标签中是否存在 Inf:"</span>, np.isinf(labels.numpy()).<span class="hljs-built_in">any</span>())<br><br></code></pre></td></tr></table></figure><h2 id="查看训练过程的历史记录，但我只在失败后用过，成功后还没有运行"><a href="#查看训练过程的历史记录，但我只在失败后用过，成功后还没有运行" class="headerlink" title="查看训练过程的历史记录，但我只在失败后用过，成功后还没有运行"></a>查看训练过程的历史记录，但我只在失败后用过，成功后还没有运行</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> matplotlib.pyplot <span class="hljs-keyword">as</span> plt<br><br><span class="hljs-comment"># 检查是否存在历史记录</span><br><span class="hljs-keyword">if</span> history.history:<br> <span class="hljs-comment"># 绘制训练和验证准确率</span><br> plt.plot(history.history[<span class="hljs-string">'accuracy'</span>], label=<span class="hljs-string">'训练准确率'</span>)<br> plt.plot(history.history[<span class="hljs-string">'val_accuracy'</span>], label=<span class="hljs-string">'验证准确率'</span>)<br> plt.xlabel(<span class="hljs-string">'训练轮次'</span>)<br> plt.ylabel(<span class="hljs-string">'准确率'</span>)<br> plt.legend()<br> plt.show()<br><br> <span class="hljs-comment"># 绘制训练和验证损失</span><br> plt.plot(history.history[<span class="hljs-string">'loss'</span>], label=<span class="hljs-string">'训练损失'</span>)<br> plt.plot(history.history[<span class="hljs-string">'val_loss'</span>], label=<span class="hljs-string">'验证损失'</span>)<br> plt.xlabel(<span class="hljs-string">'训练轮次'</span>)<br> plt.ylabel(<span class="hljs-string">'损失'</span>)<br> plt.legend()<br> plt.show()<br><span class="hljs-keyword">else</span>:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"没有历史记录，可能训练过程中出现错误。"</span>)<br><br></code></pre></td></tr></table></figure><h2 id="自我完整代码"><a href="#自我完整代码" class="headerlink" title="自我完整代码"></a>自我完整代码</h2><h3 id="初版"><a href="#初版" class="headerlink" title="初版"></a>初版</h3><p>唯一做的微调是batch_size和学习率<del>学习率目前在我看来对我影响不大</del>，一开始accuracy上不去的愿意一个是Dense设置不对，还有一个是没有归一化。batch_size的影响在下一节提到。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> tensorflow <span class="hljs-keyword">as</span> tf<br><span class="hljs-keyword">from</span> keras.models <span class="hljs-keyword">import</span> Sequential<br><span class="hljs-keyword">from</span> keras.layers <span class="hljs-keyword">import</span> Conv2D, MaxPooling2D, Flatten, Dense, Dropout<br><br><span class="hljs-comment"># 设置图片的尺寸和批量大小</span><br>img_height, img_width = <span class="hljs-number">96</span>, <span class="hljs-number">96</span><br>batch_size = <span class="hljs-number">16</span> <span class="hljs-comment"># 为了适应4GB显存，将批量大小设置为16</span><br><br><span class="hljs-comment"># 数据集根目录</span><br>data_dir = <span class="hljs-string">"./leapgestrecog/leapGestRecog"</span> <span class="hljs-comment"># 请替换为你的实际路径</span><br><br><span class="hljs-comment"># 加载和预处理数据集</span><br>train_ds = tf.keras.preprocessing.image_dataset_from_directory(<br> data_dir,<br> validation_split=<span class="hljs-number">0.2</span>, <span class="hljs-comment"># 将20%数据作为验证集</span><br> subset=<span class="hljs-string">"training"</span>,<br> seed=<span class="hljs-number">123</span>, <span class="hljs-comment"># 保证数据划分的可重复性</span><br> image_size=(img_height, img_width),<br> batch_size=batch_size<br>)<br><br>val_ds = tf.keras.preprocessing.image_dataset_from_directory(<br> data_dir,<br> validation_split=<span class="hljs-number">0.2</span>,<br> subset=<span class="hljs-string">"validation"</span>,<br> seed=<span class="hljs-number">123</span>,<br> image_size=(img_height, img_width),<br> batch_size=batch_size<br>)<br><br><span class="hljs-comment"># 缓存和优化数据加载，以提高训练速度</span><br>AUTOTUNE = tf.data.AUTOTUNE<br>train_ds = train_ds.cache().shuffle(<span class="hljs-number">1000</span>).prefetch(buffer_size=AUTOTUNE)<br>val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)<br><br><span class="hljs-comment"># 应用归一化到数据集</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">normalize_img</span>(<span class="hljs-params">image, label</span>):<br> image = tf.cast(image, tf.float32) / <span class="hljs-number">255.0</span> <span class="hljs-comment"># 将像素值归一化到 [0, 1]</span><br> <span class="hljs-keyword">return</span> image, label<br><br>train_ds = train_ds.<span class="hljs-built_in">map</span>(normalize_img)<br>val_ds = val_ds.<span class="hljs-built_in">map</span>(normalize_img)<br><br><span class="hljs-comment"># 定义模型结构</span><br>model = Sequential([<br> Conv2D(<span class="hljs-number">32</span>, (<span class="hljs-number">3</span>, <span class="hljs-number">3</span>), activation=<span class="hljs-string">'relu'</span>, input_shape=(<span class="hljs-number">96</span>, <span class="hljs-number">96</span>, <span class="hljs-number">3</span>)),<br> MaxPooling2D(pool_size=(<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)),<br> Dropout(<span class="hljs-number">0.2</span>),<br><br> Conv2D(<span class="hljs-number">64</span>, (<span class="hljs-number">3</span>, <span class="hljs-number">3</span>), activation=<span class="hljs-string">'relu'</span>),<br> MaxPooling2D(pool_size=(<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)),<br> Dropout(<span class="hljs-number">0.2</span>),<br><br> Conv2D(<span class="hljs-number">128</span>, (<span class="hljs-number">3</span>, <span class="hljs-number">3</span>), activation=<span class="hljs-string">'relu'</span>),<br> MaxPooling2D(pool_size=(<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)),<br> Dropout(<span class="hljs-number">0.2</span>),<br><br> Flatten(),<br> Dense(<span class="hljs-number">128</span>, activation=<span class="hljs-string">'relu'</span>),<br> Dropout(<span class="hljs-number">0.2</span>),<br> Dense(<span class="hljs-number">10</span>, activation=<span class="hljs-string">'softmax'</span>)<br>])<br><br><span class="hljs-comment"># 编译模型，设置较小的学习率以提高数值稳定性</span><br>model.<span class="hljs-built_in">compile</span>(optimizer=tf.keras.optimizers.Adam(learning_rate=<span class="hljs-number">1e-4</span>), <br> loss=<span class="hljs-string">'sparse_categorical_crossentropy'</span>, <br> metrics=[<span class="hljs-string">'accuracy'</span>])<br><br><span class="hljs-comment"># 模型训练</span><br>epochs = <span class="hljs-number">10</span><br>history = model.fit(<br> train_ds,<br> epochs=epochs,<br> validation_data=val_ds,<br> verbose=<span class="hljs-number">1</span><br>)<br></code></pre></td></tr></table></figure><h4 id="batch-size影响，据我观察很小"><a href="#batch-size影响，据我观察很小" class="headerlink" title="batch_size影响，据我观察很小"></a>batch_size影响，据我观察很小</h4><p>为8时：<br><img src="/img/learn/2024-11-0221-17-47.png"><br>为16时：<br><img src="/img/learn/2024-11-0221-19-49.png"><br>为32时：<br><img src="/img/learn/2024-11-0221-22-25.png"><br><del>有一说一至少我看区别不大……</del></p><h3 id="次版"><a href="#次版" class="headerlink" title="次版"></a>次版</h3><p>后来esp-dl无法完成优化和量化部分，怀疑原因是</p><ul><li>输入数据集的形状不一样，是3通道的彩色图导致，在次版中改为1通道的灰度图</li><li>原文只采用了kaggle中10种手势的6种，而我初版使用了全部种类，这里也削减为前6种<br><strong>最后发现都不是</strong><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> os<br><span class="hljs-keyword">import</span> cv2<br><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np<br><span class="hljs-keyword">import</span> pickle<br><span class="hljs-keyword">from</span> sklearn.model_selection <span class="hljs-keyword">import</span> train_test_split<br><span class="hljs-keyword">import</span> tensorflow <span class="hljs-keyword">as</span> tf<br><span class="hljs-keyword">from</span> tensorflow <span class="hljs-keyword">import</span> keras<br><span class="hljs-keyword">from</span> keras.models <span class="hljs-keyword">import</span> Sequential<br><span class="hljs-keyword">from</span> keras.layers <span class="hljs-keyword">import</span> Conv2D, MaxPooling2D, Flatten, Dense, Dropout<br><span class="hljs-keyword">import</span> tf2onnx<br><br><span class="hljs-comment"># 数据集路径</span><br>data_dir = <span class="hljs-string">"./leapGestRecog"</span> <span class="hljs-comment"># 新的根路径</span><br><br><span class="hljs-comment"># 设置图片的尺寸和批量大小</span><br>img_height, img_width = <span class="hljs-number">96</span>, <span class="hljs-number">96</span><br>batch_size = <span class="hljs-number">16</span><br><br><span class="hljs-comment"># 定义只处理前6个手势文件夹</span><br>allowed_folders = [<span class="hljs-string">'01_palm'</span>, <span class="hljs-string">'02_l'</span>, <span class="hljs-string">'03_fist'</span>, <span class="hljs-string">'04_fist_moved'</span>, <span class="hljs-string">'05_thumb'</span>, <span class="hljs-string">'06_index'</span>]<br><br><span class="hljs-comment"># 读取图像数据和标签</span><br>X = []<br>y = []<br>label_map = {folder: idx <span class="hljs-keyword">for</span> idx, folder <span class="hljs-keyword">in</span> <span class="hljs-built_in">enumerate</span>(allowed_folders)} <span class="hljs-comment"># 创建标签映射</span><br><br><span class="hljs-keyword">for</span> main_folder <span class="hljs-keyword">in</span> <span class="hljs-built_in">sorted</span>(os.listdir(data_dir)):<br> main_folder_path = os.path.join(data_dir, main_folder)<br> <span class="hljs-keyword">if</span> os.path.isdir(main_folder_path):<br> <span class="hljs-keyword">for</span> folder_name <span class="hljs-keyword">in</span> allowed_folders: <span class="hljs-comment"># 仅处理前6个手势文件夹</span><br> folder_path = os.path.join(main_folder_path, folder_name)<br> <span class="hljs-keyword">if</span> os.path.isdir(folder_path):<br> label = label_map[folder_name] <span class="hljs-comment"># 使用映射中的标签</span><br> <span class="hljs-keyword">for</span> img_name <span class="hljs-keyword">in</span> os.listdir(folder_path):<br> img_path = os.path.join(folder_path, img_name)<br> img = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)<br> <span class="hljs-keyword">if</span> img <span class="hljs-keyword">is</span> <span class="hljs-keyword">not</span> <span class="hljs-literal">None</span>:<br> img = cv2.resize(img, (img_width, img_height))<br> X.append(img)<br> y.append(label)<br><br><span class="hljs-comment"># 将数据转换为 NumPy 数组并规范化</span><br>X = np.array(X).reshape(-<span class="hljs-number">1</span>, img_height, img_width, <span class="hljs-number">1</span>) / <span class="hljs-number">255.0</span> <span class="hljs-comment"># 归一化</span><br>y = np.array(y)<br><br><span class="hljs-comment"># 划分数据集</span><br>ts = <span class="hljs-number">0.3</span> <span class="hljs-comment"># 测试集比例</span><br>X_train, X_test1, y_train, y_test1 = train_test_split(X, y, test_size=ts, random_state=<span class="hljs-number">42</span>)<br>X_test, X_cal, y_test, y_cal = train_test_split(X_test1, y_test1, test_size=ts, random_state=<span class="hljs-number">42</span>)<br><br><span class="hljs-comment"># 保存校准数据集为 pickle 文件</span><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'X_test.pkl'</span>, <span class="hljs-string">'wb'</span>) <span class="hljs-keyword">as</span> file:<br> pickle.dump(X_test, file)<br><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'y_test.pkl'</span>, <span class="hljs-string">'wb'</span>) <span class="hljs-keyword">as</span> file:<br> pickle.dump(y_test, file)<br><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'X_train.pkl'</span>, <span class="hljs-string">'wb'</span>) <span class="hljs-keyword">as</span> file:<br> pickle.dump(X_train, file)<br><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'y_train.pkl'</span>, <span class="hljs-string">'wb'</span>) <span class="hljs-keyword">as</span> file: <br> pickle.dump(y_train, file)<br><br><span class="hljs-comment"># 打印 TensorFlow 版本</span><br><span class="hljs-built_in">print</span>(tf.__version__)<br><br><span class="hljs-comment"># 定义模型结构</span><br>model = Sequential()<br>model.add(Conv2D(<span class="hljs-number">32</span>, (<span class="hljs-number">5</span>, <span class="hljs-number">5</span>), activation=<span class="hljs-string">'relu'</span>, input_shape=(<span class="hljs-number">96</span>, <span class="hljs-number">96</span>, <span class="hljs-number">1</span>))) <br>model.add(MaxPooling2D((<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)))<br>model.add(Dropout(<span class="hljs-number">0.2</span>))<br>model.add(Conv2D(<span class="hljs-number">64</span>, (<span class="hljs-number">3</span>, <span class="hljs-number">3</span>), activation=<span class="hljs-string">'relu'</span>)) <br>model.add(MaxPooling2D((<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)))<br>model.add(Dropout(<span class="hljs-number">0.2</span>))<br>model.add(Conv2D(<span class="hljs-number">64</span>, (<span class="hljs-number">3</span>, <span class="hljs-number">3</span>), activation=<span class="hljs-string">'relu'</span>))<br>model.add(MaxPooling2D((<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)))<br>model.add(Flatten())<br>model.add(Dense(<span class="hljs-number">128</span>, activation=<span class="hljs-string">'relu'</span>))<br>model.add(Dense(<span class="hljs-number">6</span>, activation=<span class="hljs-string">'softmax'</span>)) <span class="hljs-comment"># 修改为6个分类</span><br><br><span class="hljs-comment"># 编译模型</span><br>model.<span class="hljs-built_in">compile</span>(optimizer=<span class="hljs-string">'adam'</span>, loss=<span class="hljs-string">'sparse_categorical_crossentropy'</span>, metrics=[<span class="hljs-string">'accuracy'</span>])<br><br><span class="hljs-comment"># 训练模型</span><br>history = model.fit(X_train, y_train, epochs=<span class="hljs-number">5</span>, batch_size=<span class="hljs-number">64</span>, verbose=<span class="hljs-number">1</span>, validation_data=(X_test, y_test))<br><br><span class="hljs-comment"># 保存模型为 .h5 格式</span><br>model.save(<span class="hljs-string">'handrecognition_model.h5'</span>)<br><br><span class="hljs-comment"># 加载保存的模型</span><br>model = tf.keras.models.load_model(<span class="hljs-string">"handrecognition_model.h5"</span>)<br><br><span class="hljs-comment"># 将模型转换为 SavedModel 格式，准备转换为 ONNX</span><br>tf.saved_model.save(model, <span class="hljs-string">"tmp_model"</span>)<br><br><span class="hljs-comment"># 使用 tf2onnx 将 SavedModel 格式模型转换为 ONNX 格式</span><br>!python -m tf2onnx.convert --saved-model <span class="hljs-string">"tmp_model"</span> --output <span class="hljs-string">"handrecognition_model.onnx"</span><br></code></pre></td></tr></table></figure></li></ul><h2 id="后续保存和转化模型"><a href="#后续保存和转化模型" class="headerlink" title="后续保存和转化模型"></a>后续保存和转化模型</h2><h3 id="保存模型"><a href="#保存模型" class="headerlink" title="保存模型"></a>保存模型</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs python">model.save(<span class="hljs-string">'handrecognition_model.h5'</span>)<br></code></pre></td></tr></table></figure><h3 id="转化模型"><a href="#转化模型" class="headerlink" title="转化模型"></a>转化模型</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs python">model = tf.keras.models.load_model(<span class="hljs-string">"handrecognition_model.h5"</span>)<br>tf.saved_model.save(model, <span class="hljs-string">"tmp_model"</span>)<br><span class="hljs-keyword">import</span> tf2onnx<br>!python -m tf2onnx.convert --saved-model <span class="hljs-string">"tmp_model"</span> --output <span class="hljs-string">"handrecognition_model.onnx"</span><br></code></pre></td></tr></table></figure><h3 id="和google-colab结合（我还目前用不到，所以未尝试）"><a href="#和google-colab结合（我还目前用不到，所以未尝试）" class="headerlink" title="和google.colab结合（我还目前用不到，所以未尝试）"></a>和google.colab结合（我还目前用不到，所以未尝试）</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># Step 3: 压缩文件（可选）</span><br>!<span class="hljs-built_in">zip</span> -r handrecognition_model.<span class="hljs-built_in">zip</span> tmp_model<br></code></pre></td></tr></table></figure><h4 id="这是压缩成功的意思"><a href="#这是压缩成功的意思" class="headerlink" title="这是压缩成功的意思"></a>这是压缩成功的意思</h4><pre><code class="hljs"> adding: tmp_model/ (stored 0%) adding: tmp_model/variables/ (stored 0%) adding: tmp_model/variables/variables.data-00000-of-00001 (deflated 10%) adding: tmp_model/variables/variables.index (deflated 64%) adding: tmp_model/fingerprint.pb (stored 0%) adding: tmp_model/assets/ (stored 0%) adding: tmp_model/saved_model.pb (deflated 89%)</code></pre><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># Step 4: 下载模型文件（适用于 Google Colab）</span><br><span class="hljs-keyword">from</span> google.colab <span class="hljs-keyword">import</span> files<br>files.download(<span class="hljs-string">"handrecognition_model.h5"</span>)<br>files.download(<span class="hljs-string">"handrecognition_model.onnx"</span>)<br>files.download(<span class="hljs-string">"handrecognition_model.zip"</span>)<br></code></pre></td></tr></table></figure><h2 id="netron可视化"><a href="#netron可视化" class="headerlink" title="netron可视化"></a>netron可视化</h2><p>初版模型可视化：<br><img src="/img/learn/2024-11-0517-51-13.png"><br>次版模型可视化：<br><img src="/img/learn/2024-11-0519-06-18.png"><br>原版模型可视化：<br><img src="/img/learn/2024-11-0517-55-04.png"></p>]]></content>
<categories>
<category>fyp</category>
</categories>
</entry>
<entry>
<title>building_with_espdl-1-failure</title>
<link href="/2024/11/01/Ali/"/>
<url>/2024/11/01/Ali/</url>
<content type="html"><![CDATA[<h1 id="building-with-espdl-1-failure"><a href="#building-with-espdl-1-failure" class="headerlink" title="building_with_espdl-1-failure"></a>building_with_espdl-1-failure</h1><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> pickle<br><span class="hljs-keyword">import</span> os<br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> tensorflow <span class="hljs-keyword">as</span> tf<br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> keras.models <span class="hljs-keyword">import</span> Sequential<br><span class="hljs-keyword">from</span> keras.layers <span class="hljs-keyword">import</span> Conv2D, MaxPooling2D, Dense, Flatten, Dropout<br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> tensorflow <span class="hljs-keyword">as</span> tf<br><span class="hljs-built_in">print</span>(tf.__version__) <span class="hljs-comment"># 打印 TensorFlow 版本</span><br><span class="hljs-built_in">print</span>(tf.keras.__file__) <span class="hljs-comment"># 打印 Keras 的实际路径</span><br></code></pre></td></tr></table></figure><h1 id="这两行是上面打印版本和实际路径的输出结果"><a href="#这两行是上面打印版本和实际路径的输出结果" class="headerlink" title="这两行是上面打印版本和实际路径的输出结果"></a>这两行是上面打印版本和实际路径的输出结果</h1><pre><code class="hljs">2.17.0/home/dahao/anaconda3/envs/lowgensim/lib/python3.10/site-packages/keras/api/_tf_keras/keras/__init__.py</code></pre><h1 id="这一步pkl的作用尚且未知，不懂pkl的意义是什么"><a href="#这一步pkl的作用尚且未知，不懂pkl的意义是什么" class="headerlink" title="这一步pkl的作用尚且未知，不懂pkl的意义是什么"></a>这一步pkl的作用尚且未知，不懂pkl的意义是什么</h1><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'/home/dahao/code/Blogs/ESP-DL/model_development/X_test.pkl'</span>, <span class="hljs-string">'rb'</span>) <span class="hljs-keyword">as</span> file:<br> X_test = pickle.load(file)<br><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'/home/dahao/code/Blogs/ESP-DL/model_development/y_test.pkl'</span>, <span class="hljs-string">'rb'</span>) <span class="hljs-keyword">as</span> file:<br> y_test = pickle.load(file)<br><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'/home/dahao/code/Blogs/ESP-DL/model_development/X_train.pkl'</span>, <span class="hljs-string">'rb'</span>) <span class="hljs-keyword">as</span> file:<br> X_train = pickle.load(file)<br><br><span class="hljs-keyword">with</span> <span class="hljs-built_in">open</span>(<span class="hljs-string">'/home/dahao/code/Blogs/ESP-DL/model_development/y_train.pkl'</span>, <span class="hljs-string">'rb'</span>) <span class="hljs-keyword">as</span> file:<br> y_train = pickle.load(file)<br></code></pre></td></tr></table></figure><h1 id="从数据集中选了其中一个文件夹中的其中一部分数据"><a href="#从数据集中选了其中一个文件夹中的其中一部分数据" class="headerlink" title="从数据集中选了其中一个文件夹中的其中一部分数据"></a>从数据集中选了其中一个文件夹中的其中一部分数据</h1><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 设置数据集路径</span><br>data_dir = <span class="hljs-string">"./leapgestrecog/leapGestRecog"</span><br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python">X = []<br>y = []<br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> PIL <span class="hljs-keyword">import</span> Image<br><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np<br><span class="hljs-comment"># 读取每个手势类别文件夹中的图像文件</span><br><span class="hljs-keyword">for</span> gesture_folder <span class="hljs-keyword">in</span> os.listdir(data_dir):<br> gesture_folder_path = os.path.join(data_dir, gesture_folder)<br> <span class="hljs-keyword">if</span> os.path.isdir(gesture_folder_path): <span class="hljs-comment"># 检查是否是文件夹</span><br> <span class="hljs-keyword">for</span> gesture_instance <span class="hljs-keyword">in</span> os.listdir(gesture_folder_path):<br> instance_dir = os.path.join(gesture_folder_path, gesture_instance)<br> <span class="hljs-keyword">if</span> os.path.isdir(instance_dir): <span class="hljs-comment"># 检查是否是文件夹</span><br> <span class="hljs-keyword">for</span> image_file <span class="hljs-keyword">in</span> os.listdir(instance_dir):<br> image_path = os.path.join(instance_dir, image_file)<br> <span class="hljs-comment"># 读取图像并进行预处理</span><br> image = Image.<span class="hljs-built_in">open</span>(image_path).convert(<span class="hljs-string">"L"</span>) <span class="hljs-comment"># 转为灰度图像</span><br> image = image.resize((<span class="hljs-number">96</span>, <span class="hljs-number">96</span>)) <span class="hljs-comment"># 调整图像大小</span><br> X.append(np.array(image)) <span class="hljs-comment"># 将图像添加到列表</span><br> y.append(gesture_instance) <span class="hljs-comment"># 使用手势实例作为标签</span><br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 将列表转为 NumPy 数组</span><br>X = np.array(X)<br>y = np.array(y)<br><br></code></pre></td></tr></table></figure><h1 id="检验数据集形状和标签形状"><a href="#检验数据集形状和标签形状" class="headerlink" title="检验数据集形状和标签形状"></a>检验数据集形状和标签形状</h1><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-built_in">print</span>(<span class="hljs-string">f"数据集形状: <span class="hljs-subst">{X.shape}</span>, 标签形状: <span class="hljs-subst">{y.shape}</span>"</span>)<br></code></pre></td></tr></table></figure><pre><code class="hljs">数据集形状: (20000, 96, 96), 标签形状: (20000,)</code></pre><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> sklearn.model_selection <span class="hljs-keyword">import</span> train_test_split<br><br>ts = <span class="hljs-number">0.3</span> <span class="hljs-comment"># Percentage of images that we want to use for testing. </span><br>X_train, X_test1, y_train, y_test1 = train_test_split(X, y, test_size=ts, random_state=<span class="hljs-number">42</span>)<br>X_test, X_cal, y_test, y_cal = train_test_split(X_test1, y_test1, test_size=ts, random_state=<span class="hljs-number">42</span>)<br><br></code></pre></td></tr></table></figure><h1 id="cnn算法部分"><a href="#cnn算法部分" class="headerlink" title="cnn算法部分"></a>cnn算法部分</h1><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-built_in">print</span>(tf.__version__)<br><br>model = Sequential()<br>model.add(Conv2D(<span class="hljs-number">32</span>, (<span class="hljs-number">5</span>, <span class="hljs-number">5</span>), activation=<span class="hljs-string">'relu'</span>, input_shape=(<span class="hljs-number">96</span>, <span class="hljs-number">96</span>, <span class="hljs-number">1</span>))) <br>model.add(MaxPooling2D((<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)))<br>model.add(Dropout(<span class="hljs-number">0.2</span>))<br>model.add(Conv2D(<span class="hljs-number">64</span>, (<span class="hljs-number">3</span>, <span class="hljs-number">3</span>), activation=<span class="hljs-string">'relu'</span>)) <br>model.add(MaxPooling2D((<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)))<br>model.add(Dropout(<span class="hljs-number">0.2</span>))<br>model.add(Conv2D(<span class="hljs-number">64</span>, (<span class="hljs-number">3</span>, <span class="hljs-number">3</span>), activation=<span class="hljs-string">'relu'</span>))<br>model.add(MaxPooling2D((<span class="hljs-number">2</span>, <span class="hljs-number">2</span>)))<br>model.add(Flatten())<br>model.add(Dense(<span class="hljs-number">128</span>, activation=<span class="hljs-string">'relu'</span>))<br>model.add(Dense(<span class="hljs-number">6</span>, activation=<span class="hljs-string">'softmax'</span>))<br><br>model.<span class="hljs-built_in">compile</span>(optimizer=<span class="hljs-string">'adam'</span>,loss=<span class="hljs-string">'sparse_categorical_crossentropy'</span>,metrics=[<span class="hljs-string">'accuracy'</span>])<br><br>model.summary()<br></code></pre></td></tr></table></figure><h1 id="这部分是cnn算法的输出结果，和原作者的相吻合"><a href="#这部分是cnn算法的输出结果，和原作者的相吻合" class="headerlink" title="这部分是cnn算法的输出结果，和原作者的相吻合"></a>这部分是cnn算法的输出结果，和原作者的相吻合</h1><pre><code class="hljs">2.17.0</code></pre><pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="font-weight: bold">Model: "sequential_8"</span></pre><pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓┃<span style="font-weight: bold"> Layer (type) </span>┃<span style="font-weight: bold"> Output Shape </span>┃<span style="font-weight: bold"> Param # </span>┃┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩│ conv2d_24 (<span style="color: #0087ff; text-decoration-color: #0087ff">Conv2D</span>) │ (<span style="color: #00d7ff; text-decoration-color: #00d7ff">None</span>, <span style="color: #00af00; text-decoration-color: #00af00">92</span>, <span style="color: #00af00; text-decoration-color: #00af00">92</span>, <span style="color: #00af00; text-decoration-color: #00af00">32</span>) │ <span style="color: #00af00; text-decoration-color: #00af00">832</span> │├─────────────────────────────────┼────────────────────────┼───────────────┤│ max_pooling2d_24 (<span style="color: #0087ff; text-decoration-color: #0087ff">MaxPooling2D</span>) │ (<span style="color: #00d7ff; text-decoration-color: #00d7ff">None</span>, <span style="color: #00af00; text-decoration-color: #00af00">46</span>, <span style="color: #00af00; text-decoration-color: #00af00">46</span>, <span style="color: #00af00; text-decoration-color: #00af00">32</span>) │ <span style="color: #00af00; text-decoration-color: #00af00">0</span> │├─────────────────────────────────┼────────────────────────┼───────────────┤│ dropout_16 (<span style="color: #0087ff; text-decoration-color: #0087ff">Dropout</span>) │ (<span style="color: #00d7ff; text-decoration-color: #00d7ff">None</span>, <span style="color: #00af00; text-decoration-color: #00af00">46</span>, <span style="color: #00af00; text-decoration-color: #00af00">46</span>, <span style="color: #00af00; text-decoration-color: #00af00">32</span>) │ <span style="color: #00af00; text-decoration-color: #00af00">0</span> │├─────────────────────────────────┼────────────────────────┼───────────────┤│ conv2d_25 (<span style="color: #0087ff; text-decoration-color: #0087ff">Conv2D</span>) │ (<span style="color: #00d7ff; text-decoration-color: #00d7ff">None</span>, <span style="color: #00af00; text-decoration-color: #00af00">44</span>, <span style="color: #00af00; text-decoration-color: #00af00">44</span>, <span style="color: #00af00; text-decoration-color: #00af00">64</span>) │ <span style="color: #00af00; text-decoration-color: #00af00">18,496</span> │├─────────────────────────────────┼────────────────────────┼───────────────┤│ max_pooling2d_25 (<span style="color: #0087ff; text-decoration-color: #0087ff">MaxPooling2D</span>) │ (<span style="color: #00d7ff; text-decoration-color: #00d7ff">None</span>, <span style="color: #00af00; text-decoration-color: #00af00">22</span>, <span style="color: #00af00; text-decoration-color: #00af00">22</span>, <span style="color: #00af00; text-decoration-color: #00af00">64</span>) │ <span style="color: #00af00; text-decoration-color: #00af00">0</span> │├─────────────────────────────────┼────────────────────────┼───────────────┤│ dropout_17 (<span style="color: #0087ff; text-decoration-color: #0087ff">Dropout</span>) │ (<span style="color: #00d7ff; text-decoration-color: #00d7ff">None</span>, <span style="color: #00af00; text-decoration-color: #00af00">22</span>, <span style="color: #00af00; text-decoration-color: #00af00">22</span>, <span style="color: #00af00; text-decoration-color: #00af00">64</span>) │ <span style="color: #00af00; text-decoration-color: #00af00">0</span> │├─────────────────────────────────┼────────────────────────┼───────────────┤│ conv2d_26 (<span style="color: #0087ff; text-decoration-color: #0087ff">Conv2D</span>) │ (<span style="color: #00d7ff; text-decoration-color: #00d7ff">None</span>, <span style="color: #00af00; text-decoration-color: #00af00">20</span>, <span style="color: #00af00; text-decoration-color: #00af00">20</span>, <span style="color: #00af00; text-decoration-color: #00af00">64</span>) │ <span style="color: #00af00; text-decoration-color: #00af00">36,928</span> │├─────────────────────────────────┼────────────────────────┼───────────────┤│ max_pooling2d_26 (<span style="color: #0087ff; text-decoration-color: #0087ff">MaxPooling2D</span>) │ (<span style="color: #00d7ff; text-decoration-color: #00d7ff">None</span>, <span style="color: #00af00; text-decoration-color: #00af00">10</span>, <span style="color: #00af00; text-decoration-color: #00af00">10</span>, <span style="color: #00af00; text-decoration-color: #00af00">64</span>) │ <span style="color: #00af00; text-decoration-color: #00af00">0</span> │├─────────────────────────────────┼────────────────────────┼───────────────┤│ flatten_8 (<span style="color: #0087ff; text-decoration-color: #0087ff">Flatten</span>) │ (<span style="color: #00d7ff; text-decoration-color: #00d7ff">None</span>, <span style="color: #00af00; text-decoration-color: #00af00">6400</span>) │ <span style="color: #00af00; text-decoration-color: #00af00">0</span> │├─────────────────────────────────┼────────────────────────┼───────────────┤│ dense_16 (<span style="color: #0087ff; text-decoration-color: #0087ff">Dense</span>) │ (<span style="color: #00d7ff; text-decoration-color: #00d7ff">None</span>, <span style="color: #00af00; text-decoration-color: #00af00">128</span>) │ <span style="color: #00af00; text-decoration-color: #00af00">819,328</span> │├─────────────────────────────────┼────────────────────────┼───────────────┤│ dense_17 (<span style="color: #0087ff; text-decoration-color: #0087ff">Dense</span>) │ (<span style="color: #00d7ff; text-decoration-color: #00d7ff">None</span>, <span style="color: #00af00; text-decoration-color: #00af00">6</span>) │ <span style="color: #00af00; text-decoration-color: #00af00">774</span> │└─────────────────────────────────┴────────────────────────┴───────────────┘</pre><pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="font-weight: bold"> Total params: </span><span style="color: #00af00; text-decoration-color: #00af00">876,358</span> (3.34 MB)</pre><pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="font-weight: bold"> Trainable params: </span><span style="color: #00af00; text-decoration-color: #00af00">876,358</span> (3.34 MB)</pre><pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="font-weight: bold"> Non-trainable params: </span><span style="color: #00af00; text-decoration-color: #00af00">0</span> (0.00 B)</pre><h1 id="尝试解决部分，未成功"><a href="#尝试解决部分，未成功" class="headerlink" title="尝试解决部分，未成功"></a>尝试解决部分，未成功</h1><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 添加通道维度</span><br>X_train = X_train.reshape(X_train.shape[<span class="hljs-number">0</span>], <span class="hljs-number">96</span>, <span class="hljs-number">96</span>, <span class="hljs-number">1</span>)<br>X_test = X_test.reshape(X_test.shape[<span class="hljs-number">0</span>], <span class="hljs-number">96</span>, <span class="hljs-number">96</span>, <span class="hljs-number">1</span>)<br><br><span class="hljs-comment"># 确保数据类型为float32</span><br>X_train = X_train.astype(<span class="hljs-string">'float32'</span>) / <span class="hljs-number">255.0</span><br>X_test = X_test.astype(<span class="hljs-string">'float32'</span>) / <span class="hljs-number">255.0</span><br><br><span class="hljs-comment"># 对标签进行编码</span><br><span class="hljs-keyword">from</span> keras.utils <span class="hljs-keyword">import</span> to_categorical<br>y_train = to_categorical(y_train)<br>y_test = to_categorical(y_test)<br><br><span class="hljs-comment"># 训练模型</span><br>history = model.fit(X_train, y_train, epochs=<span class="hljs-number">5</span>, batch_size=<span class="hljs-number">64</span>, verbose=<span class="hljs-number">1</span>, validation_data=(X_test, y_test))<br></code></pre></td></tr></table></figure><h2 id="一直显示这部分报错结果"><a href="#一直显示这部分报错结果" class="headerlink" title="一直显示这部分报错结果"></a>一直显示这部分报错结果</h2><pre><code class="hljs">---------------------------------------------------------------------------ValueError Traceback (most recent call last)Cell In[272], line 11 9 # 对标签进行编码 10 from keras.utils import to_categorical---> 11 y_train = to_categorical(y_train) 12 y_test = to_categorical(y_test) 14 # 训练模型File ~/anaconda3/envs/lowgensim/lib/python3.10/site-packages/keras/src/utils/numerical_utils.py:86, in to_categorical(x, num_classes) 84 x = backend.numpy.reshape(x, newshape) 85 return backend.nn.one_hot(x, num_classes)---> 86 x = np.array(x, dtype="int64") 87 input_shape = x.shape 89 # Shrink the last dimension if the shape is (..., 1).ValueError: invalid literal for int() with base 10: '01_palm'</code></pre><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-built_in">type</span>(y_train)<br></code></pre></td></tr></table></figure><pre><code class="hljs">numpy.ndarray</code></pre><h1 id="今日有效学习链接："><a href="#今日有效学习链接：" class="headerlink" title="今日有效学习链接："></a>今日有效学习链接：</h1><p>1、本文基于并且想尝试复现的github文档：<br><a href="https://github.com/alibukharai/Blogs/blob/main/ESP-DL/building_with_espdl.md">https://github.com/alibukharai/Blogs/blob/main/ESP-DL/building_with_espdl.md</a><br>2、No module named ‘tensorflow.keras‘报错信息的解决方法：<br><a href="https://blog.csdn.net/Zinnir/article/details/125999939?fromshare=blogdetail&sharetype=blogdetail&sharerId=125999939&sharerefer=PC&sharesource=m0_60571820&sharefrom=from_link">https://blog.csdn.net/Zinnir/article/details/125999939?fromshare=blogdetail&sharetype=blogdetail&sharerId=125999939&sharerefer=PC&sharesource=m0_60571820&sharefrom=from_link</a><br>我没有用python，我用的是shell，原理相同<br>3、kaggle怎么下载数据集？<br><a href="https://blog.csdn.net/weixin_73577120/article/details/143170847?fromshare=blogdetail&sharetype=blogdetail&sharerId=143170847&sharerefer=PC&sharesource=m0_60571820&sharefrom=from_link">https://blog.csdn.net/weixin_73577120/article/details/143170847?fromshare=blogdetail&sharetype=blogdetail&sharerId=143170847&sharerefer=PC&sharesource=m0_60571820&sharefrom=from_link</a></p><p>原来失恋是这种感觉，也有可能我本来就从未得到过吧……一直是我在自我幻想罢了，无所谓，短时间我不会再相信爱情了……<br>人，总是要向前看的，找不到答案的话就找自己吧！</p>]]></content>
<categories>
<category>fyp</category>
</categories>
</entry>
<entry>
<title>Working with Your Own Data and Documents in Python</title>
<link href="/2024/10/24/AI%20Python%20for%20Beginners%EF%BC%9AWorking%20with%20Your%20Own%20Data%20and%20Documents%20in%20Python/"/>
<url>/2024/10/24/AI%20Python%20for%20Beginners%EF%BC%9AWorking%20with%20Your%20Own%20Data%20and%20Documents%20in%20Python/</url>
<content type="html"><![CDATA[<h1 id="Working-with-Your-Own-Data-and-Documents-in-Python"><a href="#Working-with-Your-Own-Data-and-Documents-in-Python" class="headerlink" title="Working with Your Own Data and Documents in Python"></a>Working with Your Own Data and Documents in Python</h1><h2 id="Using-files-in-Python"><a href="#Using-files-in-Python" class="headerlink" title="Using files in Python"></a>Using files in Python</h2><p><strong>open()</strong> 函数是 Python 内置的用于打开文件并返回一个文件对象的函数。<br><code>file_object = open(file_name, mode)</code></p><h3 id="mode-打开文件的模式："><a href="#mode-打开文件的模式：" class="headerlink" title="mode: 打开文件的模式："></a>mode: 打开文件的模式：</h3><ul><li>**’r’**：读取模式（默认）。</li><li>**’w’**：写入模式，会覆盖原有文件内容。</li><li>**’a’**：追加模式，在文件末尾追加内容。</li><li>**’x’**：创建新文件，如果文件已存在则报错。</li><li>**’b’**：以二进制模式打开。</li><li>**’+’**：可读写模式。</li></ul><h3 id="文件对象的方法"><a href="#文件对象的方法" class="headerlink" title="文件对象的方法"></a>文件对象的方法</h3><ul><li><strong>read()</strong>: 读取文件内容。</li><li><strong>readline()</strong>: 读取一行内容。</li><li><strong>readlines()</strong>: 读取所有行并返回一个列表。</li><li><strong>write()</strong>: 写入内容。</li><li><strong>close()</strong>: 关闭文件。</li><li><strong>seek()</strong>: 移动文件指针。</li><li><strong>tell()</strong>: 返回文件指针的当前位置。-</li></ul><h2 id="Reading-journals-from-food-critics"><a href="#Reading-journals-from-food-critics" class="headerlink" title="Reading journals from food critics"></a>Reading journals from food critics</h2><h3 id="split"><a href="#split" class="headerlink" title=".split()"></a>.split()</h3><p> 在 Python 中，<code>.split()</code> 是一个字符串方法，它的作用是将一个字符串按照指定的分隔符拆分成一个列表。<br> <code>字符串.split(分隔符, maxsplit)</code></p><ul><li><strong>字符串:</strong> 你要进行分割的字符串。</li><li><strong>分隔符:</strong> 用来分割字符串的字符或子字符串。如果不指定分隔符，则默认以空格为分隔符。</li><li><strong>maxsplit:</strong> 可选参数，指定最多分割的次数。如果不指定，则默认分割所有可能的子字符串。<br>例如：<figure class="highlight dart"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs dart"># 按特定字符分割 filename = <span class="hljs-string">"data.txt"</span> <span class="hljs-keyword">extension</span> = filename.split(<span class="hljs-string">"."</span>)[<span class="hljs-number">1</span>] <span class="hljs-built_in">print</span>(<span class="hljs-keyword">extension</span>) # 输出：txt<br></code></pre></td></tr></table></figure>可以实现切割扩展名的作用。</li></ul><h2 id="Vacation-planning-using-CSV-files"><a href="#Vacation-planning-using-CSV-files" class="headerlink" title="Vacation planning using CSV files"></a>Vacation planning using CSV files</h2><p><strong><code>csv.DictReader</code></strong>: 这是一个 Python 的 CSV 模块中的类，用于读取 CSV 文件，并将其中的每一行数据作为字典返回。字典的键就是 CSV 文件的第一行（即表头），值就是对应行的值。</p><h2 id="Turning-code-blocks-into-reusable-functions"><a href="#Turning-code-blocks-into-reusable-functions" class="headerlink" title="Turning code blocks into reusable functions"></a>Turning code blocks into reusable functions</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">函数名</span>(<span class="hljs-params">参数<span class="hljs-number">1</span>, 参数<span class="hljs-number">2</span>, ...</span>):<br> <span class="hljs-string">"""函数的文档字符串（可选）"""</span><br> <span class="hljs-comment"># 函数体</span><br> <span class="hljs-keyword">return</span> 返回值 <span class="hljs-comment"># 可选，如果没有 return 语句，则默认返回 None</span><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>python</category>
</categories>
</entry>
<entry>
<title>Extending Python with Packages and APIs</title>
<link href="/2024/10/24/AI%20Python%20for%20Beginners%EF%BC%9AExtending%20Python%20with%20Packages%20and%20APIs/"/>
<url>/2024/10/24/AI%20Python%20for%20Beginners%EF%BC%9AExtending%20Python%20with%20Packages%20and%20APIs/</url>
<content type="html"><![CDATA[<h1 id="Extending-Python-with-Packages-and-APIs"><a href="#Extending-Python-with-Packages-and-APIs" class="headerlink" title="Extending Python with Packages and APIs"></a>Extending Python with Packages and APIs</h1><h2 id="Using-functions-from-a-local-file"><a href="#Using-functions-from-a-local-file" class="headerlink" title="Using functions from a local file"></a>Using functions from a local file</h2><p>导入函数的方式：<br>1、将整个模块导入，然后通过模块名加点号的方式来访问其中的函数。也可以在第一行末尾加as，用别名。</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs lua">import <span class="hljs-built_in">math</span><br><br>result = <span class="hljs-built_in">math</span>.<span class="hljs-built_in">sqrt</span>(<span class="hljs-number">16</span>) # 调用<span class="hljs-built_in">math</span>模块中的<span class="hljs-built_in">sqrt</span>函数<br><span class="hljs-built_in">print</span>(result) # 输出 <span class="hljs-number">4.0</span><br></code></pre></td></tr></table></figure><p>2、只导入需要的特定函数。</p><figure class="highlight isbl"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs isbl"><span class="hljs-variable">from</span> <span class="hljs-variable">math</span> <span class="hljs-variable">import</span> <span class="hljs-variable">sqrt</span><br><br><span class="hljs-variable"><span class="hljs-class">result</span></span> = <span class="hljs-function"><span class="hljs-title">sqrt</span>(<span class="hljs-number">16</span>)</span><br><span class="hljs-function"><span class="hljs-title">print</span>(<span class="hljs-variable"><span class="hljs-class">result</span></span>)</span><br></code></pre></td></tr></table></figure><p>3、导入模块中的所有函数和变量。</p><figure class="highlight isbl"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs isbl"><span class="hljs-variable">from</span> <span class="hljs-variable">math</span> <span class="hljs-variable">import</span> *<br><br><span class="hljs-variable"><span class="hljs-class">result</span></span> = <span class="hljs-function"><span class="hljs-title">sqrt</span>(<span class="hljs-number">16</span>)</span><br><span class="hljs-function"><span class="hljs-title">print</span>(<span class="hljs-variable"><span class="hljs-class">result</span></span>)</span><br></code></pre></td></tr></table></figure><h2 id="Built-in-packages"><a href="#Built-in-packages" class="headerlink" title="Built-in packages"></a>Built-in packages</h2><ul><li><strong>module模块</strong> 是最小的代码单元。</li><li><strong>package包</strong> 是多个模块的集合，用于组织代码。</li><li><strong>library库</strong> 是一个更广泛的概念，可以指单个模块、多个模块或一个包，通常提供特定的功能。</li></ul><h3 id="math-库"><a href="#math-库" class="headerlink" title="math 库"></a>math 库</h3><ul><li><strong>主要功能：</strong> 提供了大量的数学函数，用于进行基本的数学运算。</li><li><strong>常用函数：</strong><ul><li><code>sqrt(x)</code>：计算 x 的平方根。</li><li><code>pow(x, y)</code>：计算 x 的 y 次方。</li><li><code>sin(x)</code>、<code>cos(x)</code>、<code>tan(x)</code>：三角函数。</li><li><code>exp(x)</code>：计算 e 的 x 次方。</li><li><code>log(x)</code>、<code>log10(x)</code>：自然对数和以 10 为底的对数。</li><li><code>ceil(x)</code>：向上取整。</li><li><code>floor(x)</code>：向下取整。</li><li><code>radians(degrees)</code>：将角度转换为弧度。</li><li><code>degrees(radians)</code>：将弧度转换为角度。</li></ul></li></ul><h3 id="statistics-库"><a href="#statistics-库" class="headerlink" title="statistics 库"></a>statistics 库</h3><ul><li><strong>主要功能：</strong> 提供了计算各种统计数据的函数。</li><li><strong>常用函数：</strong><ul><li><code>mean(data)</code>：计算数据的平均值。</li><li><code>median(data)</code>：计算数据的中位数。</li><li><code>mode(data)</code>：计算数据的众数。</li><li><code>variance(data)</code>：计算数据的方差。</li><li><code>stdev(data)</code>：计算数据的标准差。</li><li><code>pvariance(data)</code>：计算总体方差。</li><li><code>pstdev(data)</code>：计算总体标准差。</li></ul></li></ul><h3 id="random-库"><a href="#random-库" class="headerlink" title="random 库"></a>random 库</h3><ul><li><strong>主要功能：</strong> 用于生成各种随机数。</li><li><strong>常用函数：</strong><ul><li><code>random()</code>：生成 [0.0, 1.0) 之间的随机浮点数。</li><li><code>randint(a, b)</code>：生成 [a, b] 之间的随机整数。</li><li><code>uniform(a, b)</code>：生成 [a, b) 之间的随机浮点数。</li><li><code>choice(seq)</code>：从序列 seq 中随机选择一个元素。</li><li><code>shuffle(x)</code>：将序列 x 中的元素随机打乱。</li><li><code>sample(population, k)</code>：从 population 序列中随机抽取 k 个元素，组成一个新列表。</li></ul></li></ul><h2 id="Using-third-party-packages"><a href="#Using-third-party-packages" class="headerlink" title="Using third-party packages"></a>Using third-party packages</h2><h3 id="pandas"><a href="#pandas" class="headerlink" title="pandas"></a>pandas</h3><ul><li><strong>主要功能：</strong> pandas 是 Python 中强大的数据分析和操作工具，特别擅长处理表格型数据。</li><li><strong>常用功能：</strong><ul><li><strong>Series:</strong> 一维带标签数组，类似于 Excel 中的一列。</li><li><strong>DataFrame:</strong> 二维带标签表格，类似于 Excel 工作表。</li><li><strong>读取数据：</strong> 从各种数据源（如 CSV、Excel、SQL 数据库）读取数据。</li><li><strong>数据清洗：</strong> 处理缺失值、重复值、异常值。</li><li><strong>数据转换：</strong> 数据类型转换、重塑、合并、分组。</li><li><strong>数据分析：</strong> 计算统计量、排序、筛选、分组聚合。</li></ul></li><li><strong>常用函数：</strong><ul><li><code>read_csv()</code>, <code>read_excel()</code>: 读取 CSV 和 Excel 文件。</li><li><code>head()</code>, <code>tail()</code>: 查看数据的前几行或后几行。</li><li><code>describe()</code>: 计算数据的基本统计量。</li><li><code>groupby()</code>: 按一列或多列分组。</li><li><code>merge()</code>: 合并 DataFrame。</li><li><code>pivot_table()</code>: 创建透视表。</li></ul></li></ul><h3 id="NumPy"><a href="#NumPy" class="headerlink" title="NumPy"></a>NumPy</h3><ul><li><strong>主要功能：</strong> 提供了高性能的多维数组对象和对这些数组进行操作的工具。</li><li><strong>常用功能：</strong><ul><li><strong>ndarray：</strong> 多维数组对象，是 NumPy 的核心数据结构。</li><li><strong>数组创建：</strong> 创建各种类型的数组，如随机数组、全零数组、全一数组等。</li><li><strong>数组运算：</strong> 支持向量化运算，大大提高计算效率。</li><li><strong>线性代数：</strong> 提供线性代数运算，如矩阵乘法、求逆等。</li><li><strong>随机数生成：</strong> 生成各种随机数。</li></ul></li><li><strong>常用函数：</strong><ul><li><code>array()</code>: 创建数组。</li><li><code>arange()</code>: 创建等差数组。</li><li><code>linspace()</code>: 创建等间隔数组。</li><li><code>reshape()</code>: 改变数组形状。</li><li><code>dot()</code>: 计算矩阵乘积。</li><li><code>random.rand()</code>, <code>random.randn()</code>: 生成随机数。</li></ul></li></ul><h3 id="Matplotlib"><a href="#Matplotlib" class="headerlink" title="Matplotlib"></a>Matplotlib</h3><ul><li><strong>主要功能：</strong> Matplotlib 是 Python 的绘图库，用于创建高质量的静态、动态和交互式图表。</li><li><strong>常用功能：</strong><ul><li><strong>线图：</strong> 绘制折线图。</li><li><strong>散点图：</strong> 绘制散点图。</li><li><strong>直方图：</strong> 绘制直方图。</li><li><strong>饼图：</strong> 绘制饼图。</li><li><strong>箱线图：</strong> 绘制箱线图。</li><li><strong>子图：</strong> 在一个画布上绘制多个子图。</li><li><strong>自定义：</strong> 对图表进行自定义设置，如标题、标签、颜色等。</li></ul></li><li><strong>常用函数：</strong><ul><li><code>plot()</code>: 绘制线图。</li><li><code>scatter()</code>: 绘制散点图。</li><li><code>hist()</code>: 绘制直方图。</li><li><code>pie()</code>: 绘制饼图。</li><li><code>boxplot()</code>: 绘制箱线图。</li><li><code>subplot()</code>: 创建子图。</li></ul></li></ul><h3 id="Exercise-1"><a href="#Exercise-1" class="headerlink" title="Exercise 1"></a>Exercise 1</h3><p>答：</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">import</span> pandas <span class="hljs-keyword">as</span> pd<br><br># Assuming <span class="hljs-string">'df'</span> <span class="hljs-keyword">is</span> your DataFrame containing the data<br><br># <span class="hljs-keyword">Filter</span> <span class="hljs-keyword">rows</span> <span class="hljs-keyword">where</span> <span class="hljs-string">'Model'</span> contains <span class="hljs-string">'Accord'</span><br>filtered_df = df[df[<span class="hljs-string">'Model'</span>].str.contains(<span class="hljs-string">'Accord'</span>, <span class="hljs-keyword">case</span>=<span class="hljs-keyword">False</span>)]<br><br># Display the filtered <span class="hljs-keyword">table</span><br>print(filtered_df)<br><br></code></pre></td></tr></table></figure><p><code>df[]</code> 不是函数，它是 pandas DataFrame 的索引运算符（indexing operator），是一种特殊的语法。</p><ul><li><code>df['Model']</code> 选择数据表中的’Model’列</li><li><code>.str.contains('Accord', case=False)</code><ul><li>检查每一行的Model列是否包含’Accord’这个词</li><li><code>case=False</code> 表示不区分大小写，比如’accord’、’ACCORD’都会被匹配到</li><li>这会返回一个布尔值序列（True/False）</li></ul></li><li><code>df[...]</code> 使用这个布尔序列来筛选原始数据表，只保留结果为True的行。</li></ul><h2 id="APIs-to-get-data-from-the-web"><a href="#APIs-to-get-data-from-the-web" class="headerlink" title="APIs to get data from the web"></a>APIs to get data from the web</h2><figure class="highlight prolog"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><code class="hljs prolog"># 假设data是这样的<span class="hljs-symbol">JSON</span>结构：<br>data = {<br> <span class="hljs-string">'list'</span>: [ # list是一个列表<br> { # [<span class="hljs-number">0</span>]表示取第一个元素（第一个字典）<br> <span class="hljs-string">'main'</span>: { <br> <span class="hljs-string">'temp'</span>: <span class="hljs-number">20.5</span> # 温度数据<br> },<br> <span class="hljs-string">'weather'</span>: [ # weather也是一个列表<br> { # [<span class="hljs-number">0</span>]表示取第一个天气描述<br> <span class="hljs-string">'description'</span>: <span class="hljs-string">'晴天'</span><br> }<br> ],<br> <span class="hljs-string">'wind'</span>: {<br> <span class="hljs-string">'speed'</span>: <span class="hljs-number">5.2</span> # 风速数据<br> }<br> },<br> { # 这是列表的第二个元素[<span class="hljs-number">1</span>]<br> # 更多天气数据...<br> }<br> ]<br>}<br><br># 所以代码是这样工作的：<br>temperature = data[<span class="hljs-string">'list'</span>][<span class="hljs-number">0</span>][<span class="hljs-string">'main'</span>][<span class="hljs-string">'temp'</span>] # 获取<span class="hljs-number">20.5</span><br>description = data[<span class="hljs-string">'list'</span>][<span class="hljs-number">0</span>][<span class="hljs-string">'weather'</span>][<span class="hljs-number">0</span>][<span class="hljs-string">'description'</span>] # 获取<span class="hljs-string">'晴天'</span><br>wind_speed = data[<span class="hljs-string">'list'</span>][<span class="hljs-number">0</span>][<span class="hljs-string">'wind'</span>][<span class="hljs-string">'speed'</span>] # 获取<span class="hljs-number">5.2</span><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>python</category>
</categories>
</entry>
<entry>
<title>Basics of AI Python Coding</title>
<link href="/2024/10/23/AI%20Python%20for%20Beginners%EF%BC%9ABasics%20of%20AI%20Python%20Coding/"/>
<url>/2024/10/23/AI%20Python%20for%20Beginners%EF%BC%9ABasics%20of%20AI%20Python%20Coding/</url>
<content type="html"><![CDATA[<h1 id="Basics-of-AI-Python-Coding"><a href="#Basics-of-AI-Python-Coding" class="headerlink" title="Basics of AI Python Coding"></a>Basics of AI Python Coding</h1><h2 id="Navigating-the-learning-platform"><a href="#Navigating-the-learning-platform" class="headerlink" title="Navigating the learning platform"></a>Navigating the learning platform</h2><p>jupyter L2_student通过shift+enter运行代码</p><h2 id="Running-your-first-program"><a href="#Running-your-first-program" class="headerlink" title="Running your first program"></a>Running your first program</h2><p>#+空格表示注释</p><h2 id="Data-in-Python"><a href="#Data-in-Python" class="headerlink" title="Data in Python"></a>Data in Python</h2><h3 id="print"><a href="#print" class="headerlink" title="print()"></a>print()</h3><p>图像和声音都是以文本或数字形式处理的<br>“字符串”<br>“””<br>多行字符串<br>“””</p><h3 id="type"><a href="#type" class="headerlink" title="type()"></a>type()</h3><p>int 整数 float 浮点数<br>a ** b a的b次方</p><h2 id="Combining-text-and-calculations"><a href="#Combining-text-and-calculations" class="headerlink" title="Combining text and calculations"></a>Combining text and calculations</h2><h3 id="f-strings-格式化字符串"><a href="#f-strings-格式化字符串" class="headerlink" title="f-strings 格式化字符串"></a>f-strings 格式化字符串</h3><p>print(f”{:.0f}”)<br>f-strings 也可以作为被赋的值</p><h2 id="Variables"><a href="#Variables" class="headerlink" title="Variables"></a>Variables</h2><p>变量名不能包含空格<br><strong>Python 变量命名规则总结：</strong></p><ul><li>变量名只能包含字母、数字和下划线（_）。</li><li>变量名必须以字母或下划线开头。</li><li>变量名区分大小写（例如，age 和 Age 是不同的变量）。</li></ul><h2 id="Functions-Actions-on-Data"><a href="#Functions-Actions-on-Data" class="headerlink" title="Functions: Actions on Data"></a>Functions: Actions on Data</h2><p>向函数提供参数<br>len() 计算字符串长度或列表中元素个数（长度）<br>round(number) 四舍五入数</p>]]></content>
<categories>
<category>python</category>
</categories>
</entry>
<entry>
<title>Automating Tasks with Python</title>
<link href="/2024/10/23/AI%20Python%20for%20Beginners%EF%BC%9AAutomating%20Tasks%20with%20Python/"/>
<url>/2024/10/23/AI%20Python%20for%20Beginners%EF%BC%9AAutomating%20Tasks%20with%20Python/</url>
<content type="html"><![CDATA[<h1 id="Automating-Tasks-with-Python"><a href="#Automating-Tasks-with-Python" class="headerlink" title="Automating Tasks with Python"></a>Automating Tasks with Python</h1><h2 id="Completing-a-task-list-with-AI"><a href="#Completing-a-task-list-with-AI" class="headerlink" title="Completing a task list with AI"></a>Completing a task list with AI</h2><p>列表变量list_name = [a, b, c]，从0开始计数，通过list_name[n]来访问对应的单个元素。<br>list.append() 增加元素到末尾 list.remove()删除元素。<br>初始化list_name = []</p><h2 id="Repeating-tasks-with-for-loops"><a href="#Repeating-tasks-with-for-loops" class="headerlink" title="Repeating tasks with for loops"></a>Repeating tasks with for loops</h2><p>Python依靠缩进来表示代码块的层级关系，而不像其他语言使用 <code>{}</code> 来包裹代码块。因此，在Python中<strong>缩进</strong>是语法的一部分，不能随意省略。通常使用4个空格作为标准缩进（而不是Tab键），这是为了代码风格的一致性。<br>在Python中，每一层嵌套代码块都需要<strong>缩进4个空格</strong>。<br>for 变量 in 可迭代对象:<br> 循环体代码</p><ul><li>**<code>变量</code>**：循环中的迭代变量，每次循环会自动被赋予当前元素的值。</li><li>**<code>可迭代对象</code>**：可以是列表、元组、字符串、字典、集合，甚至使用 <code>range()</code> 函数生成的数列。</li><li><strong>循环体代码</strong>：每次循环时要执行的代码，必须缩进（通常 4 个空格）。</li></ul><h2 id="Prioritizing-tasks-with-dictionaries-and-AI"><a href="#Prioritizing-tasks-with-dictionaries-and-AI" class="headerlink" title="Prioritizing tasks with dictionaries and AI"></a>Prioritizing tasks with dictionaries and AI</h2><h3 id="dictionary"><a href="#dictionary" class="headerlink" title="dictionary"></a>dictionary</h3><p>my_dict = {‘name’: ‘Alice’, ‘age’: 30, ‘city’: ‘Beijing’}定义时用大括号，访问时用方括号，访问key时不加双引号。<br>字典中的键必须是唯一的，且通常是不可变数据类型（如字符串、数字或元组），而值可以是任意数据类型。<br><code>.keys()</code>可以查看键，<code>.values()</code>可以查看值，都不需要参数。</p><h2 id="Customizing-recipes-with-lists-dictionaries-and-AI"><a href="#Customizing-recipes-with-lists-dictionaries-and-AI" class="headerlink" title="Customizing recipes with lists, dictionaries and AI"></a>Customizing recipes with lists, dictionaries and AI</h2><p><strong>bool</strong> 在 Python 中是一种表示真假值的<strong>布尔型</strong>数据类型。它只有两个值：</p><ul><li><strong>True</strong>：表示真</li><li><strong>False</strong>：表示假</li></ul><h2 id="Comparing-data-in-Python"><a href="#Comparing-data-in-Python" class="headerlink" title="Comparing data in Python"></a>Comparing data in Python</h2><p>python中两个数比大小输出的值是一个bool<br>逻辑运算符 and or</p><h2 id="Helping-AI-make-decisions"><a href="#Helping-AI-make-decisions" class="headerlink" title="Helping AI make decisions"></a>Helping AI make decisions</h2><p>if 条件表达式:<br> # 条件为真时执行的代码块<br>else:<br> # 条件为假时执行的代码块</p><ul><li><strong>代码块：</strong> 条件为真时执行的代码，需要缩进。</li><li><strong>else：</strong> 可选的，当条件为假时执行的代码块，也需要缩进。<br>在 <strong>Python 的 f-string</strong> 中，<code>{}</code> 内的内容会被解析为 <strong>Python 表达式</strong>。由于 <strong>f-string</strong> 直接解释表达式，所以不允许在 <code>{}</code> 中使用反斜杠 <code>\</code> 来转义字符。方案就是内层用单引号，外层用双引号或者反之，总之不能重复。</li></ul>]]></content>
<categories>
<category>python</category>
</categories>
</entry>
<entry>
<title>Command-line Environment</title>
<link href="/2024/10/22/Command-line%20Environment/"/>
<url>/2024/10/22/Command-line%20Environment/</url>
<content type="html"><![CDATA[<h1 id="Command-line-Environment"><a href="#Command-line-Environment" class="headerlink" title="Command-line Environment"></a>Command-line Environment</h1><h2 id="常用信号"><a href="#常用信号" class="headerlink" title="常用信号"></a>常用信号</h2><ul><li><strong>SIGTERM:</strong> 终止信号，通常用于请求进程正常退出。</li><li><strong>SIGINT:</strong> 中断信号，通常由用户按下Ctrl+C产生，用于中断正在运行的进程。</li><li><strong>SIGKILL:</strong> 杀死信号，强制终止进程，<strong>无法被进程捕获或忽略。</strong></li><li><strong>SIGSTOP:</strong> 停止信号，暂停进程的执行。</li><li><strong>SIGCONT:</strong> 继续信号，继续被暂停的进程。</li><li><strong>SIGALRM:</strong> 闹钟信号，用于实现定时器功能。</li><li><strong>SIGCHLD:</strong> 子进程状态改变信号，当子进程终止或停止时，父进程会收到该信号。</li><li><strong>SIGUSR1，SIGUSR2:</strong> 用户自定义信号，可以用来实现进程之间的通信。</li><li><strong>Ctrl+C (SIGINT):</strong> 这是最常用的信号，当你在终端中运行一个程序时，按下Ctrl+C，就会向该程序发送SIGINT信号，通常会终止程序的运行。</li><li><strong>Ctrl+Z (SIGTSTP):</strong> 这个信号会将当前的前台进程挂起，你可以使用命令<code>fg</code>将挂起的进程恢复到前台，或者使用<code>bg</code>将挂起的进程放到后台运行。</li><li><strong>Ctrl+\ (SIGQUIT):</strong> 这个信号除了终止程序外，还会生成一个core dump文件，这个文件可以用来调试程序。</li></ul><h2 id="nohup"><a href="#nohup" class="headerlink" title="nohup"></a>nohup</h2><p>用于在终端窗口关闭后继续运行命令。<br><code>nohup command &</code></p><ul><li><code>nohup</code>: 告诉系统在终端窗口关闭后继续运行命令。</li><li><code>command</code>: 要执行的命令。</li><li><code>&</code>: 将命令放到后台运行。</li></ul><h2 id="jobs"><a href="#jobs" class="headerlink" title="jobs"></a>jobs</h2><ul><li><strong>-l 或 –list：</strong> 以长格式显示作业信息，包括作业编号、进程 ID、命令行等。</li><li><strong>-p 或 –pids：</strong> 只显示作业的进程 ID。</li><li><strong>-r 或 –running：</strong> 只显示正在运行的作业。</li><li><strong>-s 或 –stopped：</strong> 只显示已停止的作业。<br>可以使用百分号 + 任务编号（<code>jobs</code> 会打印任务编号）来选取该任务。如果要选择最近的一个任务，可以使用 <code>$!</code> 这一特殊参数。</li></ul><h3 id="与-jobs-命令配合使用的命令"><a href="#与-jobs-命令配合使用的命令" class="headerlink" title="与 jobs 命令配合使用的命令"></a>与 jobs 命令配合使用的命令</h3><ul><li><strong>fg：</strong> 将后台作业调回前台继续运行。</li><li><strong>bg：</strong> 将一个停止的作业放到后台继续运行。</li><li><code>kill [signal] jobspec</code>：其中signal如果不指定，默认发送 <code>SIGTERM</code>。<code>-l</code>可以查看可发送的信号列表。</li></ul><h2 id="pgrep"><a href="#pgrep" class="headerlink" title="pgrep"></a>pgrep</h2><ul><li><strong>-a:</strong> 显示进程的命令行。</li><li><strong>-f:</strong> 根据完整的命令行进行匹配。</li><li><strong>-l:</strong> 显示进程名。</li><li><strong>-u:</strong> 根据用户 ID 匹配。</li><li><strong>-g:</strong> 根据进程组 ID 匹配。</li><li><strong>-P:</strong> 根据父进程 ID 匹配。</li><li><strong>-t:</strong> 根据终端名匹配。</li><li><strong>-x:</strong> 精确匹配进程名。</li><li><strong>-i:</strong> 忽略大小写。</li><li><strong>-v:</strong> 反向匹配，即查找不匹配的进程。</li></ul><h2 id="disown"><a href="#disown" class="headerlink" title="disown"></a>disown</h2><p>用于将后台运行的作业从当前 shell 的作业列表中删除。这意味着，即使你关闭了终端窗口，这些作业仍然会继续运行。<br><code>disown [-r] jobspec</code></p><ul><li><strong>jobspec</strong>：指定要删除的作业。可以使用作业号（如 <code>%1</code>）、作业名（如 <code>myjob</code>）或进程 ID（PID）。</li><li><strong>-r</strong>：删除所有后台作业。</li></ul><h2 id="tmux"><a href="#tmux" class="headerlink" title="tmux"></a>tmux</h2><h3 id="tmux-的核心概念"><a href="#tmux-的核心概念" class="headerlink" title="tmux 的核心概念"></a>tmux 的核心概念</h3><ul><li><strong>会话 (session)：</strong> 一个独立的工作环境，可以包含多个窗口。</li><li><strong>窗口 (window)：</strong> 会话中的一个视图，可以包含多个窗格。</li><li><strong>窗格 (pane)：</strong> 窗口中的一个子区域，用于显示不同的命令行。</li></ul><h3 id="常用操作"><a href="#常用操作" class="headerlink" title="常用操作"></a>常用操作</h3><ul><li><p><strong>会话</strong> - 每个会话都是一个独立的工作区，其中包含一个或多个窗口</p><ul><li><code>tmux</code> 开始一个新的会话</li><li><code>tmux new -s NAME</code> 以指定名称开始一个新的会话</li><li><code>tmux ls</code> 列出当前所有会话</li><li>在 <code>tmux</code> 中输入 <code><C-b> d</code> ，将当前会话分离</li><li><code>tmux a</code> 重新连接最后一个会话。您也可以通过 <code>-t</code> 来指定具体的会话</li></ul></li><li><p><strong>窗口</strong> - 相当于编辑器或是浏览器中的标签页，从视觉上将一个会话分割为多个部分</p><ul><li><code><C-b> c</code> 创建一个新的窗口，使用 <code><C-d></code> 关闭</li><li><code><C-b> N</code> 跳转到第 <em>N</em> 个窗口，注意每个窗口都是有编号的</li><li><code><C-b> p</code> 切换到前一个窗口</li><li><code><C-b> n</code> 切换到下一个窗口</li><li><code><C-b> ,</code> 重命名当前窗口</li><li><code><C-b> w</code> 列出当前所有窗口</li></ul></li><li><p><strong>面板</strong> - 像 vim 中的分屏一样，面板使我们可以在一个屏幕里显示多个 shell</p><ul><li><code><C-b> "</code> 水平分割</li><li><code><C-b> %</code> 垂直分割</li><li><code><C-b> <方向></code> 切换到指定方向的面板，<方向> 指的是键盘上的方向键</li><li><code><C-b> z</code> 切换当前面板的缩放</li><li><code><C-b> [</code> 开始往回卷动屏幕。您可以按下空格键来开始选择，回车键复制选中的部分</li><li><code><C-b> <空格></code> 在不同的面板排布间切换</li></ul></li></ul><h2 id="alias"><a href="#alias" class="headerlink" title="alias"></a>alias</h2><p>用于创建命令别名。<br>例如：<code>alias ll='ls -la'</code><strong>注意等号左右都没有空格。</strong><br><code>unalias 别名</code>删除别名 <code>alias</code>查看所有别名<br>为了使自定义的别名在每次打开终端时都生效，可以将它们写入 shell 的配置文件中。</p><ul><li><strong>Bash:</strong> <code>~/.bashrc</code></li><li><strong>Zsh:</strong> <code>~/.zshrc</code></li></ul><h2 id="配置文件dotfiles"><a href="#配置文件dotfiles" class="headerlink" title="配置文件dotfiles"></a>配置文件dotfiles</h2><p>这些文件通常<strong>隐藏</strong>在你的用户目录下，用来存储各种应用程序和操作系统的个性化设置。<br>实际上，很多程序都要求您在 shell 的配置文件中包含一行类似 <code>export PATH="$PATH:/path/to/program/bin"</code> 的命令，这样才能确保这些程序能够被 shell 找到。</p><h2 id="ssh"><a href="#ssh" class="headerlink" title="ssh"></a>ssh</h2><h3 id="ssh-keygen"><a href="#ssh-keygen" class="headerlink" title="ssh-keygen"></a>ssh-keygen</h3><p><code>ssh-keygen -t rsa -b 4096 -C "[email protected]"</code></p><ul><li><code>-t rsa</code>：指定使用 RSA 算法。</li><li><code>-b 4096</code>：密钥长度为 4096 位，保证更高的安全性。</li><li><code>-C "[email protected]"</code>：注释，用于标识这个密钥，一般填写你的邮箱地址。<br><code>Enter file in which to save the key (/home/your_user/.ssh/id_rsa):</code><br>这里保持默认按enter。<figure class="highlight mathematica"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs mathematica"><span class="hljs-built_in">Enter</span> <span class="hljs-variable">passphrase</span> <span class="hljs-punctuation">(</span><span class="hljs-variable">empty</span> <span class="hljs-variable">for</span> <span class="hljs-variable">no</span> <span class="hljs-variable">passphrase</span><span class="hljs-punctuation">)</span><span class="hljs-operator">:</span> <br><span class="hljs-built_in">Enter</span> <span class="hljs-variable">same</span> <span class="hljs-variable">passphrase</span> <span class="hljs-variable">again</span><span class="hljs-operator">:</span><br></code></pre></td></tr></table></figure><code>ls -al ~/.ssh</code>查看密钥文件<br><code>cat ~/.ssh/id_rsa.pub</code><br>将该公钥粘贴到你想要连接的远程服务器或服务上。通常是粘贴到 <code>~/.ssh/authorized_keys</code> 文件或在服务中配置 SSH 公钥。</li></ul><h3 id="ssh-copy-id"><a href="#ssh-copy-id" class="headerlink" title="ssh-copy-id"></a>ssh-copy-id</h3><p>用于将本地主机的公钥复制到远程主机的 <code>authorized_keys</code> 文件中的命令行工具。允许使用 SSH 密钥进行无密码登录。<br><code>ssh-copy-id [options] user@host</code><br><strong>选项：</strong></p><ul><li><code>-i identity_file</code>: 指定要使用的私钥文件（默认为 <code>~/.ssh/id_rsa</code>）。</li><li><code>-f authorized_keys_file</code>: 指定远程主机的 <code>authorized_keys</code> 文件（默认为 <code>~/.ssh/authorized_keys</code>）。</li><li><code>-K</code>: 将公钥添加到远程主机的 <code>known_hosts</code> 文件中。</li><li><code>-R</code>: 从远程主机的 <code>authorized_keys</code> 文件中删除公钥。</li></ul><h3 id="scp"><a href="#scp" class="headerlink" title="scp"></a>scp</h3><ul><li><strong>-r</strong>：递归复制整个目录。</li><li><strong>-p</strong>：保留文件属性（权限、时间戳等）。</li><li><strong>-v</strong>：显示详细的传输过程。</li><li><strong>-q</strong>：静默模式，不显示输出。</li><li><strong>-i identity_file</strong>：指定使用的私钥文件。</li></ul><h3 id="rsync"><a href="#rsync" class="headerlink" title="rsync"></a>rsync</h3><p><code>rsync [options] source destination</code><br><strong>常用选项</strong></p><ul><li><strong>-a</strong>：归档模式，保留文件属性（权限、时间戳等）。</li><li><strong>-v</strong>：详细模式，显示传输过程。</li><li><strong>-z</strong>：压缩传输数据，提高效率。</li><li><strong>-r</strong>：递归同步整个目录。</li><li><strong>-u</strong>：更新模式，仅传输更新的文件。</li><li><strong>-t</strong>：保留文件修改时间。</li><li><strong>-p</strong>：保留文件权限。</li><li><strong>-P</strong>：保留文件所有者和组。</li><li><strong>–progress</strong>：显示传输进度。</li><li><strong>–delete</strong>：删除目标中不存在的源文件。</li></ul><h2 id="课后练习"><a href="#课后练习" class="headerlink" title="课后练习"></a>课后练习</h2><p>1、答：</p><figure class="highlight powershell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs powershell"><span class="hljs-built_in">sleep</span> <span class="hljs-number">10000</span> &<br>pgrep <span class="hljs-literal">-af</span> <span class="hljs-built_in">sleep</span><br>pkill <span class="hljs-operator">-f</span> <span class="hljs-built_in">sleep</span><br></code></pre></td></tr></table></figure><p>2、<code>kill</code> 命令成功退出时其状态码为 0 ，其他状态则是非 0。<code>kill -0</code> 则不会发送信号，但是会在进程不存在时返回一个不为 0 的状态码。<br>答：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-meta">#!/bin/bash</span><br><br> <br><br><span class="hljs-comment"># 定义 pidwait 函数</span><br><br><span class="hljs-function"><span class="hljs-title">pidwait</span></span>() {<br><br><span class="hljs-built_in">local</span> pid=<span class="hljs-variable">$1</span> <span class="hljs-comment"># 获取进程的 PID</span><br><span class="hljs-comment"># 将标准暑促和标准错误输出丢弃，不输出在终端</span><br><br><span class="hljs-keyword">while</span> ps -p <span class="hljs-string">"<span class="hljs-variable">$pid</span>"</span> > /dev/null 2>&1; <span class="hljs-keyword">do</span> <span class="hljs-comment"># 使用 ps 检查进程是否存在，只要 `ps` 命令的返回值为 0（表示进程存在），就一直循环。</span><br><br><span class="hljs-built_in">sleep</span> 1 <span class="hljs-comment"># 每秒轮询一次，减少 CPU 占用</span><br><br><span class="hljs-keyword">done</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"Process <span class="hljs-variable">$pid</span> has finished."</span><br><br>}<br><br> <br><br><span class="hljs-comment"># 将函数导出到当前 shell 会话中</span><br><br><span class="hljs-built_in">export</span> -f pidwait<br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"pidwait function loaded into the current session."</span><br></code></pre></td></tr></table></figure><p>然后在当前目录下运行该脚本，并确保它在当前shell会话中定义了pidwait函数<code>source filename.sh</code>。然后再执行pidwait <pid>即可。</p>]]></content>
<categories>
<category>shell</category>
</categories>
</entry>
<entry>
<title>Git</title>
<link href="/2024/10/22/Git/"/>
<url>/2024/10/22/Git/</url>
<content type="html"><![CDATA[<h1 id="Git"><a href="#Git" class="headerlink" title="Git"></a>Git</h1><p>本文基于<a href="https://liaoxuefeng.com/books/git/what-is-git/svn-vs-git/index.html">https://liaoxuefeng.com/books/git/what-is-git/svn-vs-git/index.html</a><br>Git Cheat Sheet<a href="https://liaoxuefeng.com/books/git/conclusion/git-cheat-sheet.pdf">https://liaoxuefeng.com/books/git/conclusion/git-cheat-sheet.pdf</a></p><h2 id="创建版本库"><a href="#创建版本库" class="headerlink" title="创建版本库"></a>创建版本库</h2><p><code>git init</code>把这个目录变成Git可以管理的仓库<br><code>git add file</code>添加文件到库<br><code>git commit</code>命令，<code>-m</code>后面输入的是本次提交的说明，可以输入任意内容。</p><h2 id="时光机穿梭"><a href="#时光机穿梭" class="headerlink" title="时光机穿梭"></a>时光机穿梭</h2><p><code>git status</code>命令可以让我们时刻掌握仓库当前的状态<br><code>git diff</code>顾名思义就是查看difference，显示的格式正是Unix通用的diff格式<br>提交新文件或者修改仍然使用<code>git add file</code></p><h3 id="版本回退"><a href="#版本回退" class="headerlink" title="版本回退"></a>版本回退</h3><p><code>git log</code>命令显示从最近到最远的提交日志，加上<code>--pretty=oneline</code>参数可以使输出更简化。<br>在Git中，用<code>HEAD</code>表示当前版本，上一个版本就是<code>HEAD^</code>，上上一个版本就是<code>HEAD^^</code>，当然往上100个版本写成<code>HEAD~100</code>。<br><code>git reset</code>：<code>--hard</code>会回退到上个版本的已提交状态，而<code>--soft</code>会回退到上个版本的未提交状态，<code>--mixed</code>会回退到上个版本已添加但未提交的状态。<br>例如：<code>git reset --hard HEAD^</code><br>如果要退回现在的版本，只要上面的命令行窗口还没有被关掉，找到HEAD版本的找到那个<code>append GPL</code>的<code>commit id</code>是什么，使用<code>git reset --hard <commit id></code>即可，版本号没必要写全，前几位就可以了。<br>如果已经关闭了命令行窗口，则要通过<code>git reflog</code>（用来记录你的每一次命令）来查看当时的commit id。</p><h3 id="工作区和暂存区"><a href="#工作区和暂存区" class="headerlink" title="工作区和暂存区"></a>工作区和暂存区</h3><p><code>git add</code>命令实际上就是把要提交的所有修改放到暂存区（Stage），然后，执行<code>git commit</code>就可以一次性把暂存区的所有修改提交到分支。</p><h3 id="管理修改"><a href="#管理修改" class="headerlink" title="管理修改"></a>管理修改</h3><p>提交后，用<code>git diff HEAD -- readme.txt</code>命令可以查看工作区和版本库里面最新版本的区别。</p><h3 id="撤销修改"><a href="#撤销修改" class="headerlink" title="撤销修改"></a>撤销修改</h3><p><code>git checkout -- file</code>可以丢弃工作区的修改（<code>git restore file</code>也可以）<br>这里有两种情况：<br>一种是<code>readme.txt</code>自修改后还没有被放到暂存区，现在，撤销修改就回到和版本库一模一样的状态；<br>一种是<code>readme.txt</code>已经添加到暂存区后，又作了修改，现在，撤销修改就回到添加到暂存区后的状态。<br><code>it reset HEAD <file></code>可以把暂存区的修改撤销掉（unstage），重新放回工作区</p><h3 id="删除文件"><a href="#删除文件" class="headerlink" title="删除文件"></a>删除文件</h3><p><code>git rm</code>删掉，并且<code>git commit</code>：从版本库中删除该文件<br>先手动删除文件，然后使用<code>git rm <file></code>和<code>git add <file></code>效果是一样的。<br>把误删的文件恢复到最新版本：<code>git checkout -- file</code></p><h2 id="远程仓库"><a href="#远程仓库" class="headerlink" title="远程仓库"></a>远程仓库</h2><h2 id="添加远程库"><a href="#添加远程库" class="headerlink" title="添加远程库"></a>添加远程库</h2><p>远程库的名字就是<code>origin</code><br>以后只要本地作了提交，就可以通过命令<code>git push origin main</code>把本地main分支的最新修改推送至github。</p><h3 id="从远程库克隆"><a href="#从远程库克隆" class="headerlink" title="从远程库克隆"></a>从远程库克隆</h3><p><code>git clone [email protected]:BlakeHansen130/repositoy-name.git</code><br>Git支持多种协议，包括<code>https</code>，但<code>ssh</code>协议速度最快。</p><h2 id="分支管理"><a href="#分支管理" class="headerlink" title="分支管理"></a>分支管理</h2><h3 id="创建与合并分支"><a href="#创建与合并分支" class="headerlink" title="创建与合并分支"></a>创建与合并分支</h3><h4 id="checkout"><a href="#checkout" class="headerlink" title="checkout"></a>checkout</h4><p><code>git checkout -b dev</code>：<code>-b</code>表示创建并切换，相当于以下两条命令：</p><figure class="highlight q"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs q">git branch <span class="hljs-built_in">dev</span><br>git checkout <span class="hljs-built_in">dev</span> # 切换到<span class="hljs-built_in">dev</span>分支<br></code></pre></td></tr></table></figure><p>用<code>git branch</code>命令查看当前分支：当前分支前面会标一个<code>*</code>号。<br>把dev分支的工作成果合并到main上：<code>git merge dev</code>用于合并指定分支到当前分支。<br><code>git branch -d dev</code>删除dev分支。</p><h4 id="switch"><a href="#switch" class="headerlink" title="switch"></a>switch</h4><p><code>git switch -c dev</code>创建并切换到新的分支<br><code>git switch main</code>切换到已有的分支</p><h3 id="解决冲突"><a href="#解决冲突" class="headerlink" title="解决冲突"></a>解决冲突</h3><p>Git用<code><<<<<<<</code>，<code>=======</code>，<code>>>>>>>></code>标记出不同分支的内容<br><code>git log --graph --pretty=oneline --abbrev-commit</code>可以看到分支的合并情况。<br>当Git无法自动合并分支时，就必须首先解决冲突。解决冲突后，再提交，合并完成。<br>解决冲突就是把Git合并失败的文件手动编辑为我们希望的内容，再提交。</p><h3 id="分支管理策略"><a href="#分支管理策略" class="headerlink" title="分支管理策略"></a>分支管理策略</h3><p>首先，<code>master</code>分支应该是非常稳定的，也就是仅用来发布新版本，平时不能在上面干活；<br>那在哪干活呢？干活都在<code>dev</code>分支上，也就是说，<code>dev</code>分支是不稳定的，到某个时候，比如1.0版本发布时，再把<code>dev</code>分支合并到<code>master</code>上，在<code>master</code>分支发布1.0版本；<br>你和你的小伙伴们每个人都在<code>dev</code>分支上干活，每个人都有自己的分支，时不时地往<code>dev</code>分支上合并就可以了。<br>合并分支时，加上<code>--no-ff</code>参数就可以用普通模式合并，合并后的历史有分支，能看出来曾经做过合并，而<code>fast forward</code>合并就看不出来曾经做过合并。</p><h3 id="Bug分支"><a href="#Bug分支" class="headerlink" title="Bug分支"></a>Bug分支</h3><p><code>git stash</code>：把当前工作现场“储藏”起来，等以后恢复现场后继续工作。<br><code>git stash list</code>查看工作现场：<br>一是用<code>git stash apply</code>恢复，但是恢复后，<code>stash</code>内容并不删除，你需要用<code>git stash drop</code>来删除；<br>另一种方式是用<code>git stash pop</code>，恢复的同时把<code>stash</code>内容也删了。<br>你可以多次<code>stash</code>，恢复的时候，先用<code>git stash list</code>查看，然后恢复指定的<code>stash</code>，用命令：<code>git stash apply stash@{0}</code></p><h4 id="如何在另一分支上修改同样的bug？"><a href="#如何在另一分支上修改同样的bug？" class="headerlink" title="如何在另一分支上修改同样的bug？"></a>如何在另一分支上修改同样的bug？</h4><p><code>git cherry-pick <bug-fixed commit id></code>能复制一个特定的commit到当前分支</p><h3 id="Feature分支"><a href="#Feature分支" class="headerlink" title="Feature分支"></a>Feature分支</h3><p>每添加一个新功能，最好新建一个feature分支，在上面开发，完成后，合并，最后，删除该feature分支。<br><code>git branch -D <name></code>强行丢弃没有合并过的分支。</p><h3 id="多人协作"><a href="#多人协作" class="headerlink" title="多人协作"></a>多人协作</h3><p><code>git remote</code>查看远程库信息<br><code>git remote -v</code>显示更详细的信息，上面显示了可以抓取和推送的<code>origin</code>的地址。如果没有推送权限，就看不到push的地址。<br>从远程库clone时，默认情况下，只能看到本地的<code>master</code>分支。<br>要在<code>dev</code>分支上开发，就必须创建远程<code>origin</code>的<code>dev</code>分支到本地，用这个命令创建本地<code>dev</code>分支：<code>git checkout -b dev origin/dev</code><br>有冲突时，先用<code>git pull</code>把最新的提交从<code>origin/dev</code>抓下来，然后，在本地合并，解决冲突，再推送。<br>如果<code>git pull</code>提示<code>no tracking information</code>，则说明本地分支和远程分支的链接关系没有创建，即没有关联，用命令<code>git branch --set-upstream-to <branch-name> origin/<branch-name></code>。</p><h3 id="rebase"><a href="#rebase" class="headerlink" title="rebase"></a>rebase</h3><ul><li>rebase操作可以把本地未push的分叉提交历史整理成直线；</li><li>rebase的目的是使得我们在查看历史提交的变化时更容易，因为分叉的提交需要三方对比。<br><code>git rebase</code></li></ul><h2 id="标签管理"><a href="#标签管理" class="headerlink" title="标签管理"></a>标签管理</h2><h3 id="创建标签"><a href="#创建标签" class="headerlink" title="创建标签"></a>创建标签</h3><p>先切换到需要打标签的分支上，再<code>git tag <name></code>即可对最新的commit打标签。如果要对历史commit打标签，则<code>git tag <name> <commit id></code>。<br><code>git tag</code>可以查看所有标签，标签不是按时间顺序列出，而是按字母排序的。<br><code>git show <name></code>可以查看标签信息。<br>还可以创建带有说明的标签，用<code>-a</code>指定标签名，<code>-m</code>指定说明文字：<br>例如：<code>git tag -a v0.1 -m "version 0.1 released" 1094adb</code><br><strong>标签总是和某个commit挂钩。如果这个commit既出现在master分支，又出现在dev分支，那么在这两个分支上都可以看到这个标签。</strong></p><h3 id="操作标签"><a href="#操作标签" class="headerlink" title="操作标签"></a>操作标签</h3><p><code>git tag -d <name></code>删除标签。创建的标签都只存储在本地，不会自动推送到远程。<br><code>git push origin <name></code>推送标签。一次性推送全部：<code>git push origin --tags</code><br>如果标签已经推送到远程，先从本地删除<code>git tag -d <name></code>，再从远程删除<code>git push origin :refs/tags/<name></code>。</p><h2 id="使用GitHub"><a href="#使用GitHub" class="headerlink" title="使用GitHub"></a>使用GitHub</h2><p>点“Fork”就在自己的账号下克隆了一个仓库，然后，从自己的账号下clone,这样才能推送修改。如果你希望官方库能接受你的修改，你就可以在GitHub上发起一个pull request。</p><h2 id="自定义Git"><a href="#自定义Git" class="headerlink" title="自定义Git"></a>自定义Git</h2><h3 id="忽略特殊文件"><a href="#忽略特殊文件" class="headerlink" title="忽略特殊文件"></a>忽略特殊文件</h3><p>在Git工作区的<strong>根目录</strong>下创建一个特殊的<code>.gitignore</code>文件，然后把要忽略的文件名填进去，Git就会自动忽略这些文件。<br>检验<code>.gitignore</code>的标准是<code>git status</code>命令是不是说<code>working directory clean</code>。<br>如果想强制添加.ignore文件中的文件到Git，那么<code>git add -f file</code>；也可以用<code>git check-ignore</code>来检查哪行规则忽略了该文件。<br>把指定文件排除在<code>.gitignore</code>规则外的写法就是<code>!</code>+文件名，所以，只需把例外文件添加进去即可。<br>在线生成<code>.gitignore</code>文件并直接下载：<a href="https://michaelliao.github.io/gitignore-online-generator/">https://michaelliao.github.io/gitignore-online-generator/</a><br>一个Git仓库也可以有多个<code>.gitignore</code>文件，<code>.gitignore</code>文件放在哪个目录下，就对哪个目录（包括子目录）起作用。</p><h3 id="配置别名"><a href="#配置别名" class="headerlink" title="配置别名"></a>配置别名</h3><p>配置Git的时候，加上<code>--global</code>是针对当前用户起作用的，如果不加，那只针对当前的仓库起作用。每个仓库的Git配置文件都放在<code>.git/config</code>文件中，当前用户的Git配置文件放在用户主目录下的一个隐藏文件<code>.gitconfig</code>中。</p>]]></content>
<categories>
<category>git</category>
</categories>
</entry>
<entry>
<title>Data Wrangling</title>
<link href="/2024/10/22/Data%20Wrangling/"/>
<url>/2024/10/22/Data%20Wrangling/</url>
<content type="html"><![CDATA[<h1 id="Data-Wrangling"><a href="#Data-Wrangling" class="headerlink" title="Data Wrangling"></a>Data Wrangling</h1><h2 id="sed"><a href="#sed" class="headerlink" title="sed"></a>sed</h2><p><strong>sed</strong> 是 <strong>stream editor</strong> 的缩写，中文译为 <strong>流编辑器</strong>。它是一种强大的文本处理工具，可以在命令行下编辑文本文件或管道中的数据。</p><h3 id="sed的常用命令格式"><a href="#sed的常用命令格式" class="headerlink" title="sed的常用命令格式"></a>sed的常用命令格式</h3><p>Bash</p><figure class="highlight nginx"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs nginx"><span class="hljs-attribute">sed</span> [选项] <span class="hljs-string">'命令'</span> 输入文本<br></code></pre></td></tr></table></figure><ul><li><p><strong>选项</strong>：</p><ul><li><code>-n</code>: 静默模式，只有经过sed命令处理的行才会被输出。</li><li><code>-e</code>: 直接在命令行上进行sed的编辑。</li><li><code>-f</code>: 从指定文件中读取sed命令。</li><li><code>-r</code>: 支持扩展正则表达式。</li></ul></li><li><p><strong>命令</strong>：</p><ul><li><code>s/pattern/replacement/</code>: 搜索并替换，其中<code>pattern</code>是正则表达式，<code>replacement</code>是替换字符串。</li><li><code>d</code>: 删除行。</li><li><code>p</code>: 打印行。</li><li><code>a\text</code>: 在行后追加文本。</li><li><code>i\text</code>: 在行前插入文本。</li><li><code>c\text</code>: 用新文本替换行。</li><li><code>n</code>: 读取下一行，并输出pattern space中的行。</li><li><code>g：</code> global 的缩写，表示用于在<strong>同一行</strong>中对所有符合条件的子字符串进行全局替换。</li><li><code>y</code>：替换命令，两个字符集中的字符必须一一对应。</li></ul></li></ul><p>例如：<code>sed 's/old/new/g' file.txt</code>表示在每行行首添加“prefix_”。<br><img src="/img/learn/20241021142246.png"><br>第一段意思是去掉含有a或b的内容；第三段是去掉含有“ab”的内容；第二段失败是因为没有对小括号进行转义，-E是表明采用扩展正则表达式（可以理解为整句转义），也可以采用\(\)的方法。</p><h3 id="正则表达式"><a href="#正则表达式" class="headerlink" title="正则表达式"></a>正则表达式</h3><p>假设有一个字符串 “Hello, World!”，那么：</p><ul><li><p><code>.*</code> 可以匹配整个字符串 “Hello, World!”</p></li><li><p><code>.*World</code> 可以匹配 “Hello, World”</p></li><li><p><code>^.*World$</code> 可以精确匹配整个字符串 “Hello, World!”</p></li><li><p><strong>贪婪匹配：</strong> 默认情况下，<code>. *</code> 是贪婪匹配的，也就是会尽可能匹配更多的字符。</p></li><li><p><strong>非贪婪匹配：</strong> 如果想让它匹配尽可能少的字符，可以在 <code>*</code> 后加 <code>?</code>，变成 <code>.*?</code>。</p></li><li><p><strong>匹配顺序：</strong> 正则表达式引擎是从左向右进行匹配的。一旦找到一个匹配 <code>.*World</code> 的子串，它就会认为匹配成功，而不会继续向后查找。</p></li></ul><h3 id="正则表达式的常见模式："><a href="#正则表达式的常见模式：" class="headerlink" title="正则表达式的常见模式："></a>正则表达式的常见模式：</h3><ul><li><code>.</code> 除换行符之外的 “任意单个字符”，即通配符。而.用专门的\.来表示。问号<code>?</code>同理。</li><li><code>*</code> 匹配前面字符零次或多次</li><li><code>+</code> 匹配前面字符一次或多次</li><li><code>[abc]</code> 匹配 <code>a</code>, <code>b</code> 和 <code>c</code> 中的任意一个。</li><li><code>(RX1|RX2)</code> 任何能够匹配 <code>RX1</code> 或 <code>RX2</code> 的结果</li><li><code>^</code> 行首</li><li><code>$</code> 行尾</li><li>字符\d可用于代替0 到 9 之间的任何数字。</li><li>在正则表达式中使用的最常见空格形式是空格( ␣ <em>)<em>、_制表符</em>( \t )、_换行符</em>( \n ) 和回车符 ( \r )（在 Windows 环境中很有用），这些特殊字符可匹配其各自的空格。此外，空格特殊字符\s可匹配上述<strong>任何</strong>特定空格，</li><li>\b，它匹配单词和非单词字符之间的边界。它在捕获整个单词时非常有用（例如，通过使用模式\w+\b）。<br><img src="/img/learn/20241021224424.png"></li></ul><h2 id="less"><a href="#less" class="headerlink" title="less"></a>less</h2><p>主要用于分页显示文件内容。</p><h3 id="常用命令："><a href="#常用命令：" class="headerlink" title="常用命令："></a>常用命令：</h3><ul><li><strong>基本操作：</strong><ul><li><code>less filename</code>: 打开文件</li><li>空格键：向下翻一页</li><li>b：向上翻一页</li><li>G：跳到文件末尾</li><li>g：跳到文件开头</li><li>/pattern：向下搜索 pattern</li><li>?pattern：向上搜索 pattern</li><li>n：重复上一次搜索</li><li>N：反向重复上一次搜索</li></ul></li><li><strong>退出：</strong><ul><li>q：退出 less</li></ul></li></ul><h2 id="代码详解部分"><a href="#代码详解部分" class="headerlink" title="代码详解部分"></a>代码详解部分</h2><h3 id="示例1"><a href="#示例1" class="headerlink" title="示例1"></a>示例1</h3><p><code>cat ssh.log | sed -E 's/^.*Disconnected from (invalid |authenticating )?user (.*) [0-9.]+ port [0-9]+( \[preauth\])?$/\2/' | head -n100</code><br><strong>正则表达式部分：</strong><br>这是一个匹配 SSH 日志中特定行的正则表达式：</p><ul><li>**<code>^</code>**：表示行的开头。</li><li>**<code>.*</code>**：匹配任意数量的字符。</li><li>**<code>Disconnected from</code>**：匹配具体的 “Disconnected from” 这部分文本。</li><li>**<code>(invalid |authenticating )?</code>**：可选地匹配 “invalid “ 或 “authenticating “（有空格）。<code>?</code> 表示这部分是可选的。这是第一捕获组，如果这部分存在就作为<code>\1</code></li><li>**<code>user (.*)</code>**：匹配 “user “ 后的用户名，并捕获用户名为 **<code>\2</code>**。</li><li>**<code>[0-9.]</code>**：匹配 IP 地址中的数字和点号。</li><li>**<code>port [0-9]</code>**：匹配端口信息。</li><li><code>+</code>：表示前面的表达式 <strong>必须至少出现一次，且可以多次出现</strong>。所以，**<code>[0-9]+</code>** 匹配的是<strong>由一位或多位数字组成的数字序列</strong>，即 1 位、2 位、3 位……或更多位的数字。如果只写成 <strong><code>[0-9]</code></strong> 而没有 <strong><code>+</code><strong>，那么只能匹配</strong>一位数字</strong>的端口号，如 <code>0</code>、<code>1</code>、<code>9</code> 等。</li><li><strong><code>( \[preauth\])?</code><strong>：可选地匹配 “[preauth]”。这是</strong>第三捕获组</strong>，匹配可选的 <code>"[preauth]"</code> 部分。由于这部分不影响用户的提取，因此在替换时没有用到它。</li><li><strong><code>\2</code> 替换：</strong> 提取匹配的用户名，并作为最终输出。<br>**<code>| head -n100</code>**：</li><li>将处理后的结果传递给 <code>head</code>，并只显示前 100 行。</li></ul><h3 id="示例2"><a href="#示例2" class="headerlink" title="示例2"></a>示例2</h3><p>正则表达式测试工具<a href="https://regex101.com/">https://regex101.com/</a> <strong>强烈推荐！！！</strong><br><code>/.* Disconnected from (invalid |authenticating )?user (.*)[^ ]+ port [0-9]+( \[preauth\])?$/gm</code><br><img src="/img/learn/20241021181659.png"><br>其中[^ ]表示除了空格以外的其他字符。这里不表示行首的原因是：<br>^在方括号（也称字符类）外表示行首，即匹配字符串的开头；在方括号内部表示取反，即匹配不属于该字符类的字符。例如：<code>[^a-z]</code>表示匹配非小写字母的字符。<br>我们可以发现此时用户名是“Disconnected from invalid user wp-user 89.134.42.194 port 38866 [preauth]”，但是因为**<code>.*</code>：** 这里的 <code>.</code> 是贪婪的，它会匹配尽可能多的字符。在匹配 “Disconnected from” 之后，它会一直匹配到字符串的结尾，除非后面的模式无法匹配。因此它把用户名截成了“wp-user 89.134.42.194 port 38866 [preauth]”。我们要修改它只需要添加<code>?</code>。<br><img src="/img/learn/20241021182437.png"><br><strong><code>.*?</code>：</strong> 这里的 <code>?</code> 使 <code>.</code> 变得懒惰（lazy），这意味着它会匹配尽可能少的字符。在匹配 “Disconnected from” 之后，它会尽量匹配最少的字符，直到遇到下一个模式。因此它完整捕获了用户名。<br>但注意sed<strong>不支持该后缀</strong>，因此可以使用perl，例如<code>perl -pe 's/.*?Disconnected from //'</code><br>其中</p><ul><li><strong>-p:</strong> 命令行选项，表示对输入的每一行都执行一次表达式，并将结果输出。</li><li><strong>-e:</strong> 命令行选项，用于在命令行中直接执行Perl表达式。</li></ul><h2 id="wc"><a href="#wc" class="headerlink" title="wc"></a>wc</h2><p>用于统计文本文件中的行数、字数和字节数。</p><h3 id="基本用法"><a href="#基本用法" class="headerlink" title="基本用法"></a>基本用法</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-built_in">wc</span> [选项] 文件名<br></code></pre></td></tr></table></figure><ul><li><strong>文件名:</strong> 可以是单个文件，也可以是多个文件，甚至可以是通配符。</li><li><strong>选项:</strong><ul><li><strong>-l:</strong> 统计行数</li><li><strong>-w:</strong> 统计字数（以空格分隔）</li><li><strong>-c:</strong> 统计字节数</li><li><strong>-m:</strong> 统计字符数（与 -c 相似，但处理多字节字符的方式不同）</li><li><strong>-L:</strong> 统计最长行的长度</li></ul></li></ul><h2 id="sort"><a href="#sort" class="headerlink" title="sort"></a>sort</h2><p>用于对文本文件进行排序的常用工具。</p><h3 id="基本用法-1"><a href="#基本用法-1" class="headerlink" title="基本用法"></a>基本用法</h3><figure class="highlight arcade"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs arcade"><span class="hljs-built_in">sort</span> [选项] 文件名<br></code></pre></td></tr></table></figure><ul><li><strong>文件名:</strong> 可以是单个文件，也可以是多个文件，甚至可以是通配符。</li><li><strong>选项:</strong><ul><li><strong>-r:</strong> 反向排序（降序）</li><li><strong>-n:</strong> 按数值排序</li><li><strong>-f:</strong> 忽略大小写</li><li><strong>-u:</strong> 去除重复行</li><li><strong>-t 分隔符:</strong> 指定字段分隔符，默认为空白字符</li><li><strong>-k POS1[,POS2]:</strong> 其中<strong>POS1:</strong> 排序开始的列或字符位置。<strong>POS2:</strong> 排序结束的列或字符位置（可选）。<strong>列号:</strong> 如果数据以特定的分隔符分隔（如逗号、空格），可以使用列号来指定排序范围。例如，<code>-k 2</code> 表示按第二列排序。<strong>字符位置:</strong> 如果数据没有明确的分隔符，可以使用字符位置来指定排序范围。例如，<code>-k 5,10</code> 表示从第5个字符到第10个字符之间的文本作为排序键。例如：<code>-k 3,5: 命令会提取每行数据的第3个到第5个字符作为排序的“键”。</code></li><li><strong>-o 输出文件:</strong> 将排序结果输出到指定文件</li></ul></li></ul><h2 id="uniq"><a href="#uniq" class="headerlink" title="uniq"></a>uniq</h2><p>用于去除文本文件中重复行的工具。### 基本用法</p><figure class="highlight perl"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs perl">uni<span class="hljs-string">q [选项]</span> 文件名<br></code></pre></td></tr></table></figure><ul><li><strong>文件名:</strong> 可以是单个文件，也可以是多个文件，甚至可以是通配符。</li><li><strong>选项:</strong><ul><li><strong>-c:</strong> 在每行前面显示重复的行数</li><li><strong>-d:</strong> 只显示重复的行</li><li><strong>-u:</strong> 只显示唯一的行</li><li><strong>-i:</strong> 忽略大小写</li><li><strong>-f N:</strong> 忽略前 N 个字段</li><li><strong>-s N:</strong> 忽略前 N 个字符</li><li>-<strong>l</strong>:显示总行数。</li></ul></li></ul><h2 id="awk"><a href="#awk" class="headerlink" title="awk"></a>awk</h2><p>可以读取文本文件，逐行处理，并对每一行进行各种操作，包括提取、修改、过滤、统计等。适合对列操作。<code>awk</code> 的域分隔符（默认是空格，可以通过 <code>-F</code> 来修改）。</p><h3 id="基本用法-2"><a href="#基本用法-2" class="headerlink" title="基本用法"></a>基本用法</h3><figure class="highlight delphi"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs delphi">awk [选项] <span class="hljs-string">'pattern { action }'</span> <span class="hljs-keyword">file</span><br></code></pre></td></tr></table></figure><ul><li><strong>选项:</strong><ul><li><strong>-F 分隔符:</strong> 指定字段分隔符，默认为空白字符</li><li><strong>-v 变量=值:</strong> 设置变量的值</li></ul></li><li><strong>pattern:</strong> 匹配模式，可以是正则表达式或条件表达式</li><li><strong>action:</strong> 要执行的命令或代码块<br>例如：<figure class="highlight nginx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs nginx"><span class="hljs-comment"># 打印文件 data.txt 的第二列</span><br><span class="hljs-attribute">awk</span> <span class="hljs-string">'{ print <span class="hljs-variable">$2</span> }'</span> data.txt<br><br><span class="hljs-comment"># 打印文件 data.txt 中包含 "apple" 的行</span><br>awk <span class="hljs-string">'/apple/ { print }'</span> data.txt<br><br><span class="hljs-comment"># 计算文件 data.txt 的第三列的总和</span><br>awk <span class="hljs-string">'{ sum += <span class="hljs-variable">$3</span> } END { print sum }'</span> data.txt<br><br><span class="hljs-comment"># 将文件 data.txt 中的第二列和第三列互换</span><br>awk <span class="hljs-string">'{ temp = <span class="hljs-variable">$2</span>; <span class="hljs-variable">$2</span> = <span class="hljs-variable">$3</span>; <span class="hljs-variable">$3</span> = temp; print }'</span> data.txt > new_data.txt<br></code></pre></td></tr></table></figure>例如：<code>awk '$1 == 1 && $2 ~ /^c.*e$/ {print $0}'</code>中：<br><code>~</code> 表示正则表达式匹配。<strong>^:</strong> 表示字符串的开头。**.*:** 匹配任意数量的任意字符（包括0个）。**$:** 表示字符串的结尾。<code>$0</code> 表示整个当前行。<br><code>awk 'BEGIN { rows = 0 } $1 == 1 && $2 ~ /^c.*e$/ {rows += 1} END {print rows}'</code><br><code>BEGIN { rows = 0 }</code>: 在处理输入数据之前，初始化一个变量 <code>rows</code>，用于计数。<br><code>{rows += 1}</code>：如果两个条件都满足，则将计数器 <code>rows</code> 加1。</li></ul><h3 id="paste"><a href="#paste" class="headerlink" title="paste"></a>paste</h3><p>将多个文件的内容按照列的形式拼接在一起，形成一个新的文件。</p><h3 id="基本用法-3"><a href="#基本用法-3" class="headerlink" title="基本用法"></a>基本用法</h3><figure class="highlight prolog"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs prolog">paste [选项] 文件<span class="hljs-number">1</span> 文件<span class="hljs-number">2</span> ...<br></code></pre></td></tr></table></figure><ul><li><strong>选项:</strong><ul><li><strong>-d <间隔字符>:</strong> 指定列与列之间的分隔符，默认为制表符。</li><li><strong>-s:</strong> 将每个输入文件的行合并为一行输出。</li><li><strong>-d’字符’：</strong> 指定多个分隔符，如-d’, ‘表示用逗号和空格作为分隔符。</li></ul></li></ul><h2 id="bc"><a href="#bc" class="headerlink" title="bc"></a>bc</h2><p>支持任意精度计算的语言。</p><ul><li><strong>-i:</strong> 强制进入交互式模式，方便输入多条命令。</li><li><strong>-l:</strong> 定义使用的标准数学库，提供了更多的数学函数。</li></ul><h2 id="xargs"><a href="#xargs" class="headerlink" title="xargs"></a>xargs</h2><p>能将一个命令的输出作为另一个命令的参数。</p><h3 id="xargs-的常用选项"><a href="#xargs-的常用选项" class="headerlink" title="xargs 的常用选项"></a>xargs 的常用选项</h3><ul><li><strong>-n num:</strong> 指定每次传递给 command2 的参数个数。</li><li><strong>-p:</strong> 在执行命令前询问用户是否执行。</li><li><strong>-t:</strong> 在执行命令前打印命令行。</li><li><strong>-i{} 或 -I{}:</strong> 将 xargs 的每项名称替换为 {}。</li><li><strong>-d delimiter:</strong> 指定分隔符，默认是空格或换行符。</li><li><strong>-L num:</strong> 从标准输入一次读取 num 行送给 command 命令。<br>例如：<figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs routeros"><span class="hljs-comment"># 查找当前目录下所有 .txt 文件并删除</span><br><span class="hljs-built_in">find</span> . -name <span class="hljs-string">"*.txt"</span> | xargs rm<br><br><span class="hljs-comment"># 将 find 命令的输出每一行作为参数传递给 echo 命令</span><br><span class="hljs-built_in">find</span> . -name <span class="hljs-string">"*.txt"</span> | xargs echo<br></code></pre></td></tr></table></figure><code>find . -name "*.txt" | xargs -I {} mv {} {}.bak</code><br><code>-I {}</code>: 告诉 <code>xargs</code> 用 <code>{}</code> 来替换每个输入项。<br><code>mv {} {}.bak</code>中：</li><li><strong>第一个 <code>{}</code>：</strong> 代表要被重命名的原始文件名。它会根据 <code>find</code> 命令的输出，逐一替换为找到的每个 .txt 文件名。</li><li><strong>第二个 <code>{}</code>：</strong> 同样代表原始文件名，但在这里，它会被加上 <code>.bak</code> 后缀，形成新的文件名。</li><li>例如<code>mv file1.txt file1.txt.bak</code>，最后的文件名是原来整体的复制后加.bak。</li></ul><h2 id="ffmpeg"><a href="#ffmpeg" class="headerlink" title="ffmpeg"></a>ffmpeg</h2><p><code>ffmpeg -loglevel panic -i /dev/video0 -frames 1 -f image2 - | convert - -colorspace gray - | gzip | ssh tsp 'gzip -d | tee copy.png' | feh -</code></p><ul><li><code>-loglevel panic</code>: 只显示致命错误信息</li><li><code>-i /dev/video0</code>: 从摄像头设备读取输入</li><li><code>-frames 1</code>: 只捕获一帧</li><li><code>-f image2</code>: 输出为图片格式</li><li>最后的 <code>-</code> 表示输出到标准输出而不是文件</li><li>gray表示将图像转换为灰度图</li><li>convert中第一个 <code>-</code> 表示从标准输入读取</li><li>convert中第二个 <code>-</code> 表示输出到标准输出</li><li><code>gzip</code>表示压缩数据流，<code>gzip -d</code>: 解压缩数据</li><li>远程主机的名字叫做tsp，在tsp上执行<code>gzip -d | tee copy.png</code></li><li><code>tee copy.png</code>: 保存一份数据到copy.png文件，同时继续输出到标准输出</li><li><code>feh -</code> 表示从标准输入读取图像并显示<br>这个命令链实现了从摄像头捕获一帧图像，转换为灰度图，在保存副本的同时显示图像的功能。<br>ssh甚至可以将<code>cat /dev/video0</code>的视频流实时传输到另一个服务器。</li></ul><h2 id="scp"><a href="#scp" class="headerlink" title="scp"></a>scp</h2><p>用于在不同主机之间安全地复制文件和目录。基于SSH协议。</p><h3 id="常用选项"><a href="#常用选项" class="headerlink" title="常用选项"></a>常用选项</h3><ul><li><strong>-r:</strong> 递归复制整个目录。</li><li><strong>-p:</strong> 保留源文件的权限、时间戳等属性。</li><li><strong>-P port:</strong> 指定SSH连接使用的端口号。</li><li><strong>-i identity_file:</strong> 指定私钥文件。</li><li><strong>-c cipher:</strong> 指定加密算法。<br>例如：假设你想将本地目录<code>/home/user/documents</code>中的所有文件复制到远程服务器<code>192.168.1.100</code>的<code>/home/user/backup</code>目录下，并保留所有属性，则：<br><code>scp -r -p /home/user/documents [email protected]:/home/user/backup/</code></li></ul><h2 id="tr"><a href="#tr" class="headerlink" title="tr"></a>tr</h2><h3 id="常用选项-1"><a href="#常用选项-1" class="headerlink" title="常用选项"></a>常用选项</h3><ul><li><strong>-d:</strong> 删除字符集1中的所有字符。</li><li><strong>-s:</strong> 压缩重复字符。</li><li><strong>-c:</strong> 补集，即转换或删除字符集1中字符的补集。</li><li><strong>-t:</strong> 截断字符集1，使其与字符集2长度相同。<br>例如：<code>echo "hello world" | tr 'a-z' 'A-Z'</code>将小写字母转换为大写字母<br><code>echo " hello world " | tr -d ' '</code>删除所有空格</li></ul><h2 id="课后练习"><a href="#课后练习" class="headerlink" title="课后练习"></a>课后练习</h2><p>2、统计 words 文件 (<code>/usr/share/dict/words</code>) 中包含至少三个 <code>a</code> 且不以 <code>'s</code> 结尾的单词个数。这些单词中，出现频率前三的末尾两个字母是什么？ <code>sed</code> 的 <code>y</code> 命令，或者 <code>tr</code> 程序也许可以帮你解决大小写的问题。共存在多少种词尾两字母组合？还有一个很 有挑战性的问题：哪个组合从未出现过？<br>答：<br>chatgpt:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># 1. 将所有单词转换为小写，并过滤至少包含三个 'a' 且不以 's' 结尾的单词</span><br><span class="hljs-comment"># 这里第一行的\无实际作用；()除了有捕获的作用还有分组的租用，这里是吧前面三个看为一组。</span><br><span class="hljs-comment"># `'s$`: 这是我们使用的模式，它会匹配字符串末尾的撇号(')后跟着s。这正是我们要找的所有格形式。</span><br><span class="hljs-comment"># `s$`: 这个模式只会匹配末尾的s，会匹配到太多单词，包括：</span><br> - 普通的复数形式（books）<br> - 第三人称单数（walks）<br> - 恰好以s结尾的单词（gas）<br><span class="hljs-comment"># `'s$'`: 这种写法在 grep 中是错误的，因为外层的单引号会被 shell 解释为字符串定界符，使得模式变成了 s$，而不是要匹配的实际字符。</span><br><br><span class="hljs-built_in">tr</span> <span class="hljs-string">'[:upper:]'</span> <span class="hljs-string">'[:lower:]'</span> < /usr/share/dict/words | \<br><br>grep -E <span class="hljs-string">'(.*a.*){3,}'</span> | grep -v <span class="hljs-string">"'s$"</span> | <span class="hljs-built_in">sort</span> > filtered_words.txt<br><br> <br><br><span class="hljs-comment"># 2. 提取最后两个字母并统计频率，其中substr是截取后缀的意思</span><br><br>awk <span class="hljs-string">'{</span><br><span class="hljs-string"></span><br><span class="hljs-string">if (length($0) >= 2) # 确保单词长度大于等于2</span><br><span class="hljs-string"></span><br><span class="hljs-string">print substr($0, length($0)-1)</span><br><span class="hljs-string"></span><br><span class="hljs-string">}'</span> filtered_words.txt | <span class="hljs-built_in">sort</span> | <span class="hljs-built_in">uniq</span> -c | <span class="hljs-built_in">sort</span> -nr > suffix_frequency.txt<br><br> <br><br><span class="hljs-comment"># 3. 显示频率最高的三个词尾组合</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"频率最高的三个词尾组合："</span><br><br><span class="hljs-built_in">head</span> -n 3 suffix_frequency.txt<br><br> <br><br><span class="hljs-comment"># 4. 统计不同的词尾组合数量</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"不同的词尾两字母组合数量："</span><br><br>awk <span class="hljs-string">'{print $2}'</span> suffix_frequency.txt | <span class="hljs-built_in">sort</span> | <span class="hljs-built_in">uniq</span> | <span class="hljs-built_in">wc</span> -l<br><br> <br><br><span class="hljs-comment"># 5. 生成所有可能的两字母组合，并按字典顺序排序</span><br><br><span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> {a..z}; <span class="hljs-keyword">do</span><br><br><span class="hljs-keyword">for</span> j <span class="hljs-keyword">in</span> {a..z}; <span class="hljs-keyword">do</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"$i<span class="hljs-variable">$j</span>"</span><br><br><span class="hljs-keyword">done</span><br><br><span class="hljs-keyword">done</span> | <span class="hljs-built_in">sort</span> > all_combinations.txt<br><br> <br><br><span class="hljs-comment"># 6. 提取已出现的组合并排序</span><br><br>awk <span class="hljs-string">'{print $2}'</span> suffix_frequency.txt | <span class="hljs-built_in">sort</span> > existing_combinations.txt<br><br> <br><br><span class="hljs-comment"># 7. 找出未出现的组合</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"未出现的词尾两字母组合："</span><br><br><span class="hljs-built_in">comm</span> -23 all_combinations.txt existing_combinations.txt<br></code></pre></td></tr></table></figure><h3 id="substr"><a href="#substr" class="headerlink" title="substr"></a>substr</h3><p><code>substr(string, start, length)</code> 这个函数在 awk 中有三个参数：</p><ul><li><strong>string:</strong> 要进行截取的字符串。</li><li><strong>start:</strong> 开始截取的位置，从1开始计数。</li><li><strong>length:</strong> 截取的字符数。如果省略，则截取到字符串末尾。</li></ul><h3 id="comm"><a href="#comm" class="headerlink" title="comm"></a>comm</h3><p>用于比较两个已排序的文件，前面的是文件1，后面的是文件2。</p><figure class="highlight asciidoc"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs asciidoc"><span class="hljs-bullet">- </span>-1: 不显示只在第一个文件中出现过的行。<br><span class="hljs-bullet">- </span>-2: 不显示只在第二个文件中出现过的行。<br><span class="hljs-bullet">- </span>-3: 不显示在两个文件中都出现过的行。<br></code></pre></td></tr></table></figure><p>claude:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-meta">#!/bin/bash</span><br><br> <br><br><span class="hljs-comment"># 1. 首先统计包含至少三个a且不以's结尾的单词</span><br><br><span class="hljs-comment"># 使用tr将所有字母转为小写，然后用grep筛选</span><br><br><span class="hljs-built_in">cat</span> /usr/share/dict/words | <span class="hljs-built_in">tr</span> <span class="hljs-string">'[:upper:]'</span> <span class="hljs-string">'[:lower:]'</span> | \<br><br>grep -E <span class="hljs-string">'a.*a.*a'</span> | \<br><br>grep -v <span class="hljs-string">"'s$"</span> > filtered_words.txt<br><br> <br><br><span class="hljs-comment"># 打印符合条件的单词总数</span><br><br>total_words=$(<span class="hljs-built_in">wc</span> -l < filtered_words.txt)<br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"符合条件的单词总数: <span class="hljs-variable">$total_words</span>"</span><br><br> <br><br><span class="hljs-comment"># 2. 提取并统计末尾两个字母</span><br><br><span class="hljs-comment"># 使用awk提取最后两个字母并排序计数</span><br><br><span class="hljs-built_in">echo</span> -e <span class="hljs-string">"\n末尾两字母出现频率前三："</span><br><br><span class="hljs-built_in">cat</span> filtered_words.txt | \<br><br>awk <span class="hljs-string">'{print substr($0, length($0)-1)}'</span> | \<br><br><span class="hljs-built_in">sort</span> | <span class="hljs-built_in">uniq</span> -c | <span class="hljs-built_in">sort</span> -nr | <span class="hljs-built_in">head</span> -3<br><br> <br><br><span class="hljs-comment"># 3. 统计所有出现过的词尾两字母组合数量</span><br><br>total_combinations=$(<span class="hljs-built_in">cat</span> filtered_words.txt | \<br><br>awk <span class="hljs-string">'{print substr($0, length($0)-1)}'</span> | \<br><br><span class="hljs-built_in">sort</span> | <span class="hljs-built_in">uniq</span> | <span class="hljs-built_in">wc</span> -l)<br><br><span class="hljs-built_in">echo</span> -e <span class="hljs-string">"\n不同的词尾两字母组合总数: <span class="hljs-variable">$total_combinations</span>"</span><br><br> <br><br><span class="hljs-comment"># 4. 找出未出现过的两字母组合</span><br><br><span class="hljs-built_in">echo</span> -e <span class="hljs-string">"\n生成所有可能的两字母组合并对比："</span><br><br><span class="hljs-comment"># 生成所有可能的两字母组合</span><br><br><span class="hljs-keyword">for</span> c1 <span class="hljs-keyword">in</span> {a..z}; <span class="hljs-keyword">do</span><br><br><span class="hljs-keyword">for</span> c2 <span class="hljs-keyword">in</span> {a..z}; <span class="hljs-keyword">do</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"$c1<span class="hljs-variable">$c2</span>"</span><br><br><span class="hljs-keyword">done</span><br><br><span class="hljs-keyword">done</span> > all_combinations.txt<br><br> <br><br><span class="hljs-comment"># 获取实际出现的组合</span><br><br><span class="hljs-built_in">cat</span> filtered_words.txt | \<br><br>awk <span class="hljs-string">'{print substr($0, length($0)-1)}'</span> | \<br><br><span class="hljs-built_in">sort</span> | <span class="hljs-built_in">uniq</span> > actual_combinations.txt<br><br> <br><br><span class="hljs-comment"># 找出未出现过的组合</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"部分未出现过的两字母组合示例："</span><br><br><span class="hljs-built_in">comm</span> -23 all_combinations.txt actual_combinations.txt | <span class="hljs-built_in">head</span> -5<br><br> <br><br><span class="hljs-comment"># 清理临时文件</span><br><br><span class="hljs-built_in">rm</span> filtered_words.txt all_combinations.txt actual_combinations.txt<br></code></pre></td></tr></table></figure><p>原理类似。<br>3、sed 原地替换的原理：<br>当使用 <code>sed -i 's/REGEX/SUBSTITUTION/ input.txt'</code> 这样的命令时，sed 实际上并不是直接修改原始文件。它的工作流程大致如下：</p><ol><li><strong>读取文件：</strong> sed 将 <code>input.txt</code> 的内容读入内存。</li><li><strong>执行替换：</strong> 根据提供的正则表达式和替换字符串，对内存中的内容进行修改。</li><li><strong>写入临时文件：</strong> 将修改后的内容写入一个临时文件。</li><li><strong>覆盖原文件：</strong> 删除原始的 <code>input.txt</code>，并将临时文件重命名为 <code>input.txt</code>。<br>风险：<br><strong>意外覆盖：</strong> 如果替换命令写错，或者正则表达式匹配不正确，可能会导致大量数据被意外删除或修改，无法恢复。<br>如何安全地进行文本替换：例如：<code>sed 's/REGEX/SUBSTITUTION/g' input.txt > output.txt</code><br>4、找出您最近十次开机的开机时间平均数、中位数和最长时间。在 Linux 上需要用到 <code>journalctl</code> ，而在 macOS 上使用 <code>log show</code>。找到每次起到开始和结束时的时间戳。<br>答：<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-meta">#!/bin/bash</span><br><br> <br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"分析最近10次启动时间..."</span><br><br> <br><br><span class="hljs-comment"># 使用单独的 --list-boots 来获取可用的启动记录</span><br><br>boot_count=$(journalctl --list-boots | <span class="hljs-built_in">wc</span> -l)<br><br><span class="hljs-keyword">if</span> [ <span class="hljs-variable">$boot_count</span> -eq 0 ]; <span class="hljs-keyword">then</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"未找到启动记录"</span><br><br><span class="hljs-built_in">exit</span> 1<br><br><span class="hljs-keyword">fi</span><br><br> <br><br><span class="hljs-comment"># 获取实际的启动时间数据，每次单独查询</span><br><br><span class="hljs-built_in">times</span>=()<br><span class="hljs-comment"># 初始化数组，用于存储每次启动的时间。</span><br><br><span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> $(<span class="hljs-built_in">seq</span> 0 -1 -$((boot_count-<span class="hljs-number">1</span>))); <span class="hljs-keyword">do</span><br><br><span class="hljs-comment"># 使用 --boot 参数查询每次启动</span><br><br>boot_time=$(journalctl --boot=<span class="hljs-variable">$i</span> | grep <span class="hljs-string">"Startup finished in"</span> | grep -oP <span class="hljs-string">'[0-9.]+(?=s)'</span> | <span class="hljs-built_in">tail</span> -n1)<br><span class="hljs-comment"># seq 0 -1 -$((boot_count-1))：生成一个从 0 开始，步长为 -1，结束值为-boot_count+1的数字序列。这个序列的目的是为了从最新的启动记录开始逆序遍历，以便获取最近的boot_count次启动的时间。</span><br><br><span class="hljs-keyword">if</span> [ ! -z <span class="hljs-string">"<span class="hljs-variable">$boot_time</span>"</span> ]; <span class="hljs-keyword">then</span><br><span class="hljs-comment"># 判断是否找到启动时间：如果boot_time不为空，则表示成功提取到了启动时间。</span><br><br><span class="hljs-built_in">times</span>+=(<span class="hljs-variable">$boot_time</span>)<br><span class="hljs-comment"># 将启动时间添加到数组： 将提取到的启动时间添加到times数组的末尾。</span><br><span class="hljs-comment"># 如果已经收集了10条记录就退出</span><br><br><span class="hljs-keyword">if</span> [ <span class="hljs-variable">${#times[@]}</span> -eq 10 ]; <span class="hljs-keyword">then</span><br><br><span class="hljs-built_in">break</span><br><br><span class="hljs-keyword">fi</span><br><span class="hljs-comment"># 限制记录数量：如果times数组中的元素个数达到了 10 个，说明已经收集了足够的数据，就跳出循环。</span><br><br><span class="hljs-keyword">fi</span><br><br><span class="hljs-keyword">done</span><br><br> <br><br><span class="hljs-comment"># 检查是否获取到数据</span><br><br><span class="hljs-keyword">if</span> [ <span class="hljs-variable">${#times[@]}</span> -eq 0 ]; <span class="hljs-keyword">then</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"未能获取到任何启动时间数据"</span><br><br><span class="hljs-built_in">exit</span> 1<br><br><span class="hljs-keyword">fi</span><br><br> <br><br><span class="hljs-comment"># 计算总和用于平均值</span><br><br><span class="hljs-built_in">sum</span>=0<br><br><span class="hljs-keyword">for</span> t <span class="hljs-keyword">in</span> <span class="hljs-string">"<span class="hljs-variable">${times[@]}</span>"</span>; <span class="hljs-keyword">do</span><br><br><span class="hljs-built_in">sum</span>=$(<span class="hljs-built_in">echo</span> <span class="hljs-string">"<span class="hljs-variable">$sum</span> + <span class="hljs-variable">$t</span>"</span> | bc -l)<br><br><span class="hljs-keyword">done</span><br><br> <br><br><span class="hljs-comment"># 计算平均值</span><br><br>avg=$(<span class="hljs-built_in">echo</span> <span class="hljs-string">"scale=2; <span class="hljs-variable">$sum</span> / <span class="hljs-variable">${#times[@]}</span>"</span> | bc -l)<br><br> <br><br><span class="hljs-comment"># 排序数组用于计算中位数</span><br><br>IFS=$<span class="hljs-string">'\n'</span> sorted=($(<span class="hljs-built_in">sort</span> -n <<<<span class="hljs-string">"<span class="hljs-variable">${times[*]}</span>"</span>))<br><br><span class="hljs-built_in">unset</span> IFS<br><br> <br><br><span class="hljs-comment"># 计算中位数</span><br><br>mid=$((<span class="hljs-variable">${#sorted[@]}</span> / <span class="hljs-number">2</span>))<br><br><span class="hljs-keyword">if</span> [ $((<span class="hljs-variable">${#sorted[@]}</span> % <span class="hljs-number">2</span>)) -eq 0 ]; <span class="hljs-keyword">then</span><br><br>median=$(<span class="hljs-built_in">echo</span> <span class="hljs-string">"scale=2; (<span class="hljs-variable">${sorted[$mid-1]}</span> + <span class="hljs-variable">${sorted[$mid]}</span>) / 2"</span> | bc -l)<br><br><span class="hljs-keyword">else</span><br><br>median=<span class="hljs-variable">${sorted[$mid]}</span><br><br><span class="hljs-keyword">fi</span><br><br> <br><br><span class="hljs-comment"># 找出最长时间</span><br><br>max=<span class="hljs-variable">${sorted[-1]}</span><br><br> <br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"分析结果："</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"平均启动时间: <span class="hljs-variable">${avg}</span>秒"</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"中位数启动时间: <span class="hljs-variable">${median}</span>秒"</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"最长启动时间: <span class="hljs-variable">${max}</span>秒"</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"总计分析了 <span class="hljs-variable">${#times[@]}</span> 次启动记录"</span><br></code></pre></td></tr></table></figure></li></ol><h3 id="先行和后行断言"><a href="#先行和后行断言" class="headerlink" title="先行和后行断言"></a>先行和后行断言</h3><ul><li><strong>正向先行断言：</strong> <code>(?=pattern)</code> 表示当前位置的后面必须匹配 pattern。</li><li><strong>负向先行断言：</strong> <code>(?!pattern)</code> 表示当前位置的后面不能匹配 pattern。</li><li><strong>正向后行断言：</strong> <code>(?<=pattern)</code> 表示当前位置的前面必须匹配 pattern。</li><li><strong>负向后行断言：</strong> <code>(?<!pattern)</code> 表示当前位置的前面不能匹配 pattern。<br>例如：假设我们有一个字符串 “This is a test string.”，我们想匹配所有的 “is”，但是要求 “is” 后面必须跟着一个空格。则<code>is(?= )</code></li></ul><h3 id="shell中的数组仍需后续继续学习！！！"><a href="#shell中的数组仍需后续继续学习！！！" class="headerlink" title="shell中的数组仍需后续继续学习！！！"></a>shell中的数组仍需后续继续学习！！！</h3><p>5、1. 查看之前三次重启启动信息中不同的部分(参见 <code>journalctl</code> 的 <code>-b</code> 选项)。将这一任务分为几个步骤，首先获取之前三次启动的启动日志，也许获取启动日志的命令就有合适的选项可以帮助您提取前三次启动的日志，亦或者您可以使用 <code>sed '0,/STRING/d'</code> 来删除 <code>STRING</code> 匹配到的字符串前面的全部内容。然后，过滤掉每次都不相同的部分，例如时间戳。下一步，重复记录输入行并对其计数(可以使用 <code>uniq</code> )。最后，删除所有出现过 3 次的内容（因为这些内容是三次启动日志中的重复部分）。<br>答：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-meta">#!/bin/bash</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"=== 错误信息类型统计 ==="</span><br>(journalctl -b 0; journalctl -b -1; journalctl -b -2) | \<br>grep -i <span class="hljs-string">"error\|warning\|failed\|can't"</span> | \<br>sed -E <span class="hljs-string">'s/^[A-Za-z]+ [0-9]+ [0-9:.]+ [^ ]+ [^:]+: //'</span> | \<br><span class="hljs-comment"># 提取错误信息的主要类型（取每行前40个字符作为关键特征）</span><br>sed -E <span class="hljs-string">'s/^(.{40}).*/\1.../'</span> | \<br><span class="hljs-built_in">sort</span> | \<br><span class="hljs-built_in">uniq</span> -c | \<br><span class="hljs-built_in">sort</span> -rn | \<br><span class="hljs-built_in">head</span> -n 5<br><br><span class="hljs-built_in">echo</span> -e <span class="hljs-string">"\n=== 问题最多的应用 TOP 5 ==="</span><br>(journalctl -b 0; journalctl -b -1; journalctl -b -2) | \<br>grep -i <span class="hljs-string">"error\|warning\|failed\|can't"</span> | \<br>sed -E <span class="hljs-string">'s/^[A-Za-z]+ [0-9]+ [0-9:.]+ [^ ]+ ([^:]+).*/\1/'</span> | \<br><span class="hljs-built_in">sort</span> | \<br><span class="hljs-built_in">uniq</span> -c | \<br><span class="hljs-built_in">sort</span> -rn | \<br><span class="hljs-built_in">head</span> -n 5<br><br><span class="hljs-built_in">echo</span> -e <span class="hljs-string">"\n=== 错误等级分布 ==="</span><br>(journalctl -b 0; journalctl -b -1; journalctl -b -2) | \<br>grep -i <span class="hljs-string">"error\|warning\|failed\|can't"</span> | \<br>awk <span class="hljs-string">'{</span><br><span class="hljs-string"> if (tolower($0) ~ /error/) errors++;</span><br><span class="hljs-string"> if (tolower($0) ~ /warning/) warnings++;</span><br><span class="hljs-string"> if (tolower($0) ~ /failed/) failures++;</span><br><span class="hljs-string"> if (tolower($0) ~ /can'</span>\'<span class="hljs-string">'t/) cants++;</span><br><span class="hljs-string">}</span><br><span class="hljs-string">END {</span><br><span class="hljs-string"> print "错误(Error)数量: " errors;</span><br><span class="hljs-string"> print "警告(Warning)数量: " warnings;</span><br><span class="hljs-string"> print "失败(Failed)数量: " failures;</span><br><span class="hljs-string"> print "无法执行(Can'</span>\'<span class="hljs-string">'t)数量: " cants;</span><br><span class="hljs-string">}'</span><br></code></pre></td></tr></table></figure><p>6、</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-meta">#!/bin/bash</span><br><br> <br><br><span class="hljs-comment"># 下载页面并提取数据</span><br><br><span class="hljs-function"><span class="hljs-title">fetch_and_extract</span></span>() {<br><br><span class="hljs-comment"># 使用curl获取页面内容，使用pup提取表格数据，-s表示静默模式不输出进度信息</span><br><br><span class="hljs-comment"># 安装pup: brew install pup (MacOS) 或 去github下载二进制文件</span><br><br>curl -s <span class="hljs-string">"https://stats.wikimedia.org/EN/TablesWikipediaZH.htm"</span> | \<br><span class="hljs-comment"># 从 HTML 中提取所有<table>标签内的<tr>（表格行）元素，并将其转换为 JSON 格式。</span><br><br>pup <span class="hljs-string">'table tr json{}'</span> | \<br><br><span class="hljs-comment"># 使用jq提取文本内容，-r表示原始输出不包含额外的引号</span><br><span class="hljs-comment"># []: 遍历 JSON 数组中的每个元素（即每个表格行）。</span><br><span class="hljs-comment"># [.children[].text]: 提取每个表格行中所有子元素（即表格单元格）的文本内容，并将其作为一个数组。</span><br><span class="hljs-comment"># @tsv: 将数组中的元素以 Tab 分隔符的形式转换为 TSV 格式的字符串。</span><br><br>jq -r <span class="hljs-string">'.[] | [.children[].text] | @tsv'</span> | \<br><br><span class="hljs-comment"># 过滤掉空行</span><br><br>grep -v <span class="hljs-string">'^$'</span> > wiki_stats.txt<br><br>}<br><br> <br><br><span class="hljs-comment"># 分析指定列的数据</span><br><br><span class="hljs-function"><span class="hljs-title">analyze_column</span></span>() {<br><br><span class="hljs-built_in">local</span> col=<span class="hljs-variable">$1</span><br><br><span class="hljs-built_in">local</span> title=<span class="hljs-variable">$2</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"=== 分析 <span class="hljs-variable">$title</span> (第<span class="hljs-variable">${col}</span>列) ==="</span><br><br><span class="hljs-comment"># 提取指定列的数字数据，移除k后缀并转换为数字</span><br><br><span class="hljs-built_in">cat</span> wiki_stats.txt | \<br><span class="hljs-comment"># 将shell变量$col的值赋给awk变量col</span><br><br>awk -v col=<span class="hljs-string">"<span class="hljs-variable">$col</span>"</span> <span class="hljs-string">'{</span><br><span class="hljs-string"></span><br><span class="hljs-string">val = $col;</span><br><span class="hljs-string"></span><br><span class="hljs-string"># 如果值以k结尾，将其转换为数字</span><br><span class="hljs-string"></span><br><span class="hljs-string">if (val ~ /k$/) {</span><br><span class="hljs-string"></span><br><span class="hljs-string">val = substr(val, 1, length(val)-1) * 1000;</span><br><span class="hljs-string"></span><br><span class="hljs-string">}</span><br><span class="hljs-string"></span><br><span class="hljs-string"># 如果是数字则输出</span><br><span class="hljs-string"></span><br><span class="hljs-string">if (val ~ /^[0-9]+\.?[0-9]*$/) {</span><br><span class="hljs-string"></span><br><span class="hljs-string">print val;</span><br><span class="hljs-string"></span><br><span class="hljs-string">}</span><br><span class="hljs-string"></span><br><span class="hljs-string">}'</span> | \<br><br><span class="hljs-built_in">sort</span> -n | \<br><br>awk <span class="hljs-string">'</span><br><span class="hljs-string"></span><br><span class="hljs-string">BEGIN {min=999999999; max=-999999999; sum=0; count=0}</span><br><span class="hljs-string"></span><br><span class="hljs-string">{</span><br><span class="hljs-string"></span><br><span class="hljs-string">if ($1 < min) min = $1;</span><br><span class="hljs-string"></span><br><span class="hljs-string">if ($1 > max) max = $1;</span><br><span class="hljs-string"></span><br><span class="hljs-string">sum += $1;</span><br><span class="hljs-string"></span><br><span class="hljs-string">count++;</span><br><span class="hljs-string"></span><br><span class="hljs-string">}</span><br><span class="hljs-string"></span><br><span class="hljs-string">END {</span><br><span class="hljs-string"></span><br><span class="hljs-string">printf "最小值: %.2f\n", min;</span><br><span class="hljs-string"></span><br><span class="hljs-string">printf "最大值: %.2f\n", max;</span><br><span class="hljs-string"></span><br><span class="hljs-string">printf "平均值: %.2f\n", sum/count;</span><br><span class="hljs-string"></span><br><span class="hljs-string">printf "总和: %.2f\n", sum;</span><br><span class="hljs-string"></span><br><span class="hljs-string">}'</span><br><br>}<br><br> <br><br><span class="hljs-comment"># 计算两列之差的总和</span><br><br><span class="hljs-function"><span class="hljs-title">calculate_difference</span></span>() {<br><br><span class="hljs-built_in">local</span> col1=<span class="hljs-variable">$1</span><br><br><span class="hljs-built_in">local</span> col2=<span class="hljs-variable">$2</span><br><br><span class="hljs-built_in">local</span> title1=<span class="hljs-variable">$3</span><br><br><span class="hljs-built_in">local</span> title2=<span class="hljs-variable">$4</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"=== 计算 <span class="hljs-variable">$title1</span> 和 <span class="hljs-variable">$title2</span> 的差值总和 ==="</span><br><br><span class="hljs-built_in">cat</span> wiki_stats.txt | \<br><br>awk -v col1=<span class="hljs-string">"<span class="hljs-variable">$col1</span>"</span> -v col2=<span class="hljs-string">"<span class="hljs-variable">$col2</span>"</span> <span class="hljs-string">'</span><br><span class="hljs-string"></span><br><span class="hljs-string">function convert(val) {</span><br><span class="hljs-string"></span><br><span class="hljs-string">if (val ~ /k$/) {</span><br><span class="hljs-string"></span><br><span class="hljs-string">return substr(val, 1, length(val)-1) * 1000;</span><br><span class="hljs-string"></span><br><span class="hljs-string">}</span><br><span class="hljs-string"></span><br><span class="hljs-string">return val;</span><br><span class="hljs-string"></span><br><span class="hljs-string">}</span><br><span class="hljs-string"></span><br><span class="hljs-string">{</span><br><span class="hljs-string"></span><br><span class="hljs-string">val1 = convert($col1);</span><br><span class="hljs-string"></span><br><span class="hljs-string">val2 = convert($col2);</span><br><span class="hljs-string"></span><br><span class="hljs-string">if (val1 ~ /^[0-9]+\.?[0-9]*$/ && val2 ~ /^[0-9]+\.?[0-9]*$/) {</span><br><span class="hljs-string"></span><br><span class="hljs-string">diff += (val1 - val2);</span><br><span class="hljs-string"></span><br><span class="hljs-string">}</span><br><span class="hljs-string"></span><br><span class="hljs-string">}</span><br><span class="hljs-string"></span><br><span class="hljs-string">END {</span><br><span class="hljs-string"></span><br><span class="hljs-string">printf "差值总和: %.2f\n", diff;</span><br><span class="hljs-string"></span><br><span class="hljs-string">}'</span><br><br>}<br><br> <br><br><span class="hljs-comment"># 主程序</span><br><br><span class="hljs-function"><span class="hljs-title">main</span></span>() {<br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"开始获取维基百科统计数据..."</span><br><br>fetch_and_extract<br><br><span class="hljs-comment"># 分析第2列(total)数据</span><br><br>analyze_column 2 <span class="hljs-string">"总用户数"</span><br><br><span class="hljs-comment"># 分析第3列(new)数据</span><br><br>analyze_column 3 <span class="hljs-string">"新用户数"</span><br><br><span class="hljs-comment"># 计算第2列和第3列的差值总和</span><br><br>calculate_difference 2 3 <span class="hljs-string">"总用户数"</span> <span class="hljs-string">"新用户数"</span><br><br>}<br><br> <br><br><span class="hljs-comment"># 执行主程序</span><br><br>main<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>shell</category>
</categories>
</entry>
<entry>
<title>vim</title>
<link href="/2024/10/20/vim/"/>
<url>/2024/10/20/vim/</url>
<content type="html"><![CDATA[<h1 id="vim"><a href="#vim" class="headerlink" title="vim"></a>vim</h1><p>Vim 是一个 <em>多模态</em> 编辑 器：它对于插入文字和操纵文字有不同的模式。Vim 是可编程的（可以使用 Vimscript 或者像 Python 一样的其他程序语言），Vim 的接口本身也是一个程序语言：键入操作（以及其助记名） 是命令，这些命令也是可组合的。<br>normal模式下按i进入insert模式；按r进入replace模式；按v进入visual模式；按<S-V>ji进入visual-line模式；按^V进入visual-block模式，，再按esc返回normal。<br>按：进入command-line模式，输入q<enter>退出当前选项卡，qa<enter>退出所有。</p><h2 id="命令行模式"><a href="#命令行模式" class="headerlink" title="命令行模式"></a>命令行模式</h2><p>control+v的三种表示方法：^V Ctrl-V <C-V></p><h3 id="保存"><a href="#保存" class="headerlink" title="保存"></a>保存</h3><p>命令行模式下，:help 后输入想查询的内容可看到功能，注意要问带冒号的命令则也要带冒号，例如:help :w。<code>:w</code>表示保存更改。</p><h3 id="打开文件和缓存"><a href="#打开文件和缓存" class="headerlink" title="打开文件和缓存"></a>打开文件和缓存</h3><ul><li><code>:e {文件名}</code> 打开要编辑的文件</li><li><code>:ls</code> 显示打开的缓存</li></ul><h3 id="窗口管理"><a href="#窗口管理" class="headerlink" title="窗口管理"></a>窗口管理</h3><ul><li><strong>分割窗口</strong>：<ul><li><code>:split</code> 或 <code>Ctrl-w s</code>: 水平分割窗口</li><li><code>:vsplit</code> 或 <code>Ctrl-w v</code>: 垂直分割窗口</li></ul></li><li><strong>关闭窗口</strong>：<ul><li><code>:q</code> 或 <code>:close</code>: 关闭当前窗口</li><li><code>:qa</code> 或 <code>:qall</code>: 关闭所有窗口</li></ul></li><li><strong>移动光标到其他窗口</strong>：<ul><li><code>Ctrl-w w</code>: 在窗口之间循环切换</li><li><code>Ctrl-w h/j/k/l</code>: 移动光标到左/下/上/右的窗口</li></ul></li></ul><h3 id="缓冲区管理"><a href="#缓冲区管理" class="headerlink" title="缓冲区管理"></a>缓冲区管理</h3><ul><li><strong>查看缓冲区列表</strong>：<ul><li><code>:buffers</code> 或 <code>:ls</code></li></ul></li><li><strong>切换缓冲区</strong>：<ul><li><code>:bn</code> 或 <code>:bp</code>: 切换到下一个/上一个缓冲区</li><li><code>:b {number}</code>: 切换到指定编号的缓冲区</li></ul></li><li><strong>删除缓冲区</strong>：<ul><li><code>:bd {number}</code>: 删除指定编号的缓冲区</li></ul></li></ul><h3 id="选项卡管理"><a href="#选项卡管理" class="headerlink" title="选项卡管理"></a>选项卡管理</h3><ul><li><strong>创建新选项卡</strong>：<ul><li><code>:tabnew</code> 或 <code>gt</code></li></ul></li><li><strong>关闭选项卡</strong>：<ul><li><code>:tabclose</code> 或 <code>:tabc</code></li></ul></li><li><strong>切换选项卡</strong>：<ul><li><code>gt</code>: 切换到下一个选项卡</li><li><code>gT</code>: 切换到上一个选项卡</li><li><code>:tabn {number}</code>: 切换到指定编号的选项卡</li></ul></li></ul><h2 id="normal模式下的移动"><a href="#normal模式下的移动" class="headerlink" title="normal模式下的移动"></a>normal模式下的移动</h2><ul><li><strong>h:</strong> 向左移动一个字符</li><li><strong>j:</strong> 向下移动一行。在命令前加上数字，表示重复执行该命令的次数。例如，<code>5j</code> 表示向下移动5行。</li><li><strong>k:</strong> 向上移动一行</li><li><strong>l:</strong> 向右移动一个字符</li><li><strong>w:</strong> 移动到下一个单词的开头</li><li><strong>b:</strong> 移动到上一个单词的开头</li><li><strong>e:</strong> 移动到下一个单词的末尾</li><li><strong>^:</strong> 移动到行首</li><li><strong>$:</strong> 移动到行尾</li><li><strong>gg:</strong> 移动到文件开头</li><li><strong>G:</strong> 移动到文件末尾</li><li><strong>Ctrl+u:</strong> 向上滚动半屏</li><li><strong>Ctrl+d:</strong> 向下滚动半屏</li><li><strong>Ctrl+f:</strong> 向下滚动一屏</li><li><strong>Ctrl+b:</strong> 向上滚动一屏</li><li><strong>跳转：</strong> 使用<code>f</code>或<code>t</code>可以跳转到当前行中下一个或上一个出现的指定字符。例如，<code>fx</code> 表示跳转到下一个出现的字母x。</li><li>行数： <code>:{行数}\<CR></code> 或者 <code>{行数}G</code> ({行数}为行数)</li><li>杂项： <code>%</code> （找到配对，比如括号或者 /* */ 之类的注释对）</li></ul><h3 id="f命令"><a href="#f命令" class="headerlink" title="f命令"></a>f命令</h3><ul><li><strong>功能：</strong> 将光标移动到当前行中下一个出现的指定字符处。</li><li><strong>用法：</strong> <code>fx</code>，其中x表示你要跳转到的字符。例如，<code>fw</code>表示跳转到下一个单词的开头。</li></ul><h3 id="t命令"><a href="#t命令" class="headerlink" title="t命令"></a>t命令</h3><ul><li><strong>功能：</strong> 将光标移动到当前行中下一个出现的指定字符<strong>之前</strong>。</li><li><strong>用法：</strong> <code>tx</code>，其中x表示你要跳转到的字符。例如，<code>tw</code>表示跳转到下一个单词的开头<strong>之前</strong>。</li></ul><h3 id="扩展用法"><a href="#扩展用法" class="headerlink" title="扩展用法"></a>扩展用法</h3><ul><li><strong>重复跳转：</strong> 使用分号<code>;</code>可以重复上一次的f或t命令。例如，如果你输入<code>fa</code>，然后输入<code>;</code>，光标会继续跳转到下一个出现的”a”。</li><li><strong>反向跳转：</strong> 使用逗号<code>,</code>可以反向重复上一次的f或t命令。例如，如果你输入<code>fa</code>，然后输入<code>,</code>，光标会跳转到上一个出现的”a”。</li><li><strong>结合数字：</strong> 在f或t命令前加上数字，可以指定跳转的次数。例如，<code>2fw</code>表示跳转到下一个单词的开头，然后再跳转到下一个单词的开头。</li><li><strong>大写F和T：</strong> F和T命令与f和t类似，但它们是向左搜索。例如，<code>Fx</code>会向左搜索第一个出现的x。</li><li>屏幕： <code>H</code> （屏幕首行）， <code>M</code> （屏幕中间）， <code>L</code> （屏幕底部）</li></ul><h2 id="insert模式"><a href="#insert模式" class="headerlink" title="insert模式"></a>insert模式</h2><p><strong>Vim 的编辑命令也被称为 “动词”， 因为动词可以施动于名词。</strong></p><h3 id="o命令"><a href="#o命令" class="headerlink" title="o命令"></a>o命令</h3><ul><li><strong>功能：</strong> 在当前行的<strong>下方</strong>插入一个新行，并进入插入模式。</li><li><strong>用法：</strong> 在正常模式下，将光标移动到想要插入新行的位置，然后输入<code>o</code>。</li></ul><h3 id="O命令"><a href="#O命令" class="headerlink" title="O命令"></a>O命令</h3><ul><li><strong>功能：</strong> 在当前行的<strong>上方</strong>插入一个新行，并进入插入模式。</li><li><strong>用法：</strong> 在正常模式下，将光标移动到想要插入新行的位置，然后输入<code>O</code>。</li></ul><h3 id="基本用法"><a href="#基本用法" class="headerlink" title="基本用法"></a>基本用法</h3><ul><li><strong>删除字符：</strong><ul><li><code>x</code>：删除光标下的字符</li><li><code>X</code>：删除光标前的字符</li></ul></li><li><strong>删除整行：</strong><ul><li><code>dd</code>：删除当前行</li></ul></li><li><strong>删除到行尾：</strong><ul><li><code>d$</code>：删除从光标处到行尾的所有字符</li></ul></li><li><strong>删除到行首：</strong><ul><li><code>d0</code>：删除从光标处到行首的所有字符</li></ul></li><li><code>s</code> 替换字符（等同于 <code>xi</code>）</li></ul><h3 id="与移动命令结合"><a href="#与移动命令结合" class="headerlink" title="与移动命令结合"></a>与移动命令结合</h3><ul><li><code>dw</code>：删除到下一个单词的开头</li><li><code>de</code>：删除到下一个单词的结尾</li><li><code>db</code>：删除到上一个单词的开头</li><li><code>dd</code>：删除当前行</li><li><code>2dd</code>：删除当前行和下一行</li><li><code>daw</code>：删除光标所在单词</li></ul><h3 id="撤销操作："><a href="#撤销操作：" class="headerlink" title="撤销操作："></a>撤销操作：</h3><ul><li><strong>u：</strong> 撤销上一步操作。连续按多次u可以撤销多个步骤。并且如果从insert模式返回到normal模式以后按u，会撤销在insert模式的所有操作。</li></ul><h3 id="重做操作："><a href="#重做操作：" class="headerlink" title="重做操作："></a>重做操作：</h3><ul><li><strong>Ctrl+r：</strong> 重做上一步撤销的操作。</li></ul><h3 id="其他撤销相关命令："><a href="#其他撤销相关命令：" class="headerlink" title="其他撤销相关命令："></a>其他撤销相关命令：</h3><ul><li><strong>U：</strong> 撤销对当前行的所有修改。</li><li><strong>:undo</strong>：显示撤销列表，并允许你选择要撤销到哪个状态。</li></ul><h3 id="c命令：修改文本"><a href="#c命令：修改文本" class="headerlink" title="c命令：修改文本"></a>c命令：修改文本</h3><ul><li><strong>cw</strong>: 修改光标后的一个单词</li><li><strong>ce</strong>: 修改光标后的一个单词，直到单词末尾</li><li><strong>cb</strong>: 修改光标前的一个单词</li><li><strong>cc</strong>: 修改整行</li><li><strong>c$</strong>: 修改从光标到行尾的所有字符</li><li><strong>c0</strong>: 修改从光标到行首的所有字符<br>c和d唯一的区别就是在删除内容以后将你设置为insert状态</li></ul><h3 id="结合数字"><a href="#结合数字" class="headerlink" title="结合数字"></a>结合数字</h3><ul><li><strong>c2w</strong>: 修改接下来的两个单词</li><li><strong>c3l</strong>: 修改接下来的三个字符</li><li><strong>c.$</strong>: 修改到句号。</li><li><strong>c)</strong>: 修改到下一个”)”。</li><li><strong>c{</strong>: 修改到下一个”{“。</li></ul><h3 id="r命令"><a href="#r命令" class="headerlink" title="r命令"></a>r命令</h3><p><strong>rx</strong>: 将光标下的字符替换为x。例如，<code>ra</code> 将把光标下的字符替换为”a”。</p><ul><li><strong>字符选取：</strong><ul><li><code>v</code>：进入可视模式，然后使用光标键选择字符。</li></ul></li><li><strong>单词选取：</strong><ul><li><code>vw</code>：选中光标所在单词。</li><li><code>ve</code>：选中光标所在单词到单词末尾。</li><li><code>vb</code>：选中光标所在单词到单词开始。</li></ul></li><li><strong>行选取：</strong><ul><li><code>V</code>：进入行可视模式，选中整行。甚至可以按方块形状进行选择。</li><li><code>ggVG</code>：选中整个文件。</li></ul></li></ul><h3 id="复制"><a href="#复制" class="headerlink" title="复制"></a>复制</h3><p>选中文本后，使用<code>y</code>命令进行复制。例如：</p><ul><li><code>vwyy</code>：选中当前单词并复制。</li><li><code>Vyy</code>：选中当前行并复制。</li></ul><h3 id="粘贴"><a href="#粘贴" class="headerlink" title="粘贴"></a>粘贴</h3><p>复制后的文本可以使用<code>p</code>命令进行粘贴。</p><ul><li><code>p</code>：在光标下方粘贴。</li><li><code>P</code>：在光标上方粘贴。</li></ul><h3 id="大小写命令"><a href="#大小写命令" class="headerlink" title="大小写命令"></a>大小写命令</h3><h3 id="单个字符"><a href="#单个字符" class="headerlink" title="#单个字符"></a>#单个字符</h3><ul><li><strong><code>~</code> 命令：</strong> 将光标下的字符大小写反转。</li></ul><h4 id="单词"><a href="#单词" class="headerlink" title="单词"></a>单词</h4><ul><li><strong><code>g~iw</code>：</strong> 将光标下的单词大小写反转。<ul><li>例子：将单词”hello”转换为”HELLO”。</li></ul></li></ul><h4 id="行"><a href="#行" class="headerlink" title="行"></a>行</h4><ul><li><strong><code>guu</code>：</strong> 将当前行所有字母转换为小写。</li><li><strong><code>gUU</code>：</strong> 将当前行所有字母转换为大写。</li><li><strong><code>gu0</code>：</strong> 将光标所在位置到行首的字母转换为小写。</li><li><strong><code>gU0</code>：</strong> 将光标所在位置到行首的字母转换为大写。</li><li><strong><code>gu$</code>：</strong> 将光标所在位置到行尾的字母转换为小写。</li><li><strong><code>gU$</code>：</strong> 将光标所在位置到行尾的字母转换为大写。</li></ul><h4 id="可视模式"><a href="#可视模式" class="headerlink" title="可视模式"></a>可视模式</h4><ul><li><strong><code>gu</code> 或 <code>gU</code>：</strong> 在可视模式下，选中部分文本后，使用<code>gu</code>将其转换为小写，使用<code>gU</code>将其转换为大写。</li></ul><h2 id="修饰语"><a href="#修饰语" class="headerlink" title="修饰语"></a>修饰语</h2><p>修饰语有 <code>i</code>，表示“内部”或者“在内”，和 <code>a</code>， 表示“周围”。</p><ul><li><code>ci(</code> 改变当前括号内的内容</li><li><code>ci[</code> 改变当前方括号内的内容</li><li><code>da'</code> 删除一个单引号字符串， 包括周围的单引号</li></ul>]]></content>
<categories>
<category>shell</category>
</categories>
</entry>
<entry>
<title>Shell Tools and Scripting</title>
<link href="/2024/10/20/Shell%20Tools%20and%20Scripting/"/>
<url>/2024/10/20/Shell%20Tools%20and%20Scripting/</url>
<content type="html"><![CDATA[<h1 id="Shell-Tools-and-Scripting"><a href="#Shell-Tools-and-Scripting" class="headerlink" title="Shell Tools and Scripting"></a>Shell Tools and Scripting</h1><h2 id="变量-函数"><a href="#变量-函数" class="headerlink" title="变量 函数"></a>变量 函数</h2><p>在shell中要注意空格的使用，尤其是在进行与名字字符串中带有空格的文件有关的操作时。<br><code>"$ variable name"</code>可以使用变量。<br><code>source function</code>可以在shell中执行函数，然后<code>function folder/file</code>就可以检验了。<br><code>$0</code>是脚本名称，<code>$?</code>获取上一个命令的错误代码，<code>$_</code>获取上一个命令参数，例如文件或文件夹等，<code>$#</code>给出的参数数量，<code>$$</code>获取pid(process id)，<code>$@</code>扩展到所有参数。<br>常见的退出状态码示例：</p><ul><li><strong>1：</strong> 一般表示命令执行时出现了错误。</li><li><strong>2：</strong> 可能表示命令使用不正确，例如参数错误。</li><li><strong>127：</strong> 表示命令未找到。</li><li><strong>126：</strong> 表示命令找到，但无法执行（例如没有执行权限）。<br><img src="/img/learn/20241006222138.png"><br><code>!!</code>表示获取上一段命令，例如上一段命令执行结果是没有管理员权限，那么可以<code>sudo !!</code>来解决。</li></ul><h2 id="运算符"><a href="#运算符" class="headerlink" title="运算符"></a>运算符</h2><p><code>||</code>或，<code>&&</code>与，<code>;</code>分隔多个命令，顺序执行。<br><code>$(command)</code>也可以代表相应的输出内容，<code>cat <(command)</code>也可以实现输出内容。</p><h3 id="dev-null"><a href="#dev-null" class="headerlink" title="/dev/null"></a>/dev/null</h3><p>在类Unix系统中是一个特殊的设备文件，常被称为“空设备”或“比特桶”。你可以把它想象成一个无底洞，任何写入到它的数据都会被丢弃，而从它读取数据永远只会得到一个空的回应。<br>/dev/null 的作用</p><ul><li><strong>丢弃输出：</strong> 当你执行一个命令，但不想看到它的输出结果时，可以将输出重定向到 /dev/null。这在自动化脚本或批量处理任务中非常常见。</li><li><strong>抑制错误信息：</strong> 某些命令执行时可能会产生错误信息。如果这些错误信息对你来说无关紧要，可以将错误信息重定向到 /dev/null。</li><li><strong>创建空文件：</strong> 可以通过将特定内容写入 /dev/null 来创建一个空文件。<br>例如：<code>find / -name "*.tmp" -delete > /dev/null 2>&1</code>这条命令会查找系统中所有以 <code>.tmp</code> 结尾的文件并删除，同时将命令的标准输出和标准错误输出都重定向到 /dev/null，避免在终端显示任何信息。<code>2>&1</code>这是一个在 Shell 脚本中经常用到的重定向符号，用于 <strong>将标准错误输出重定向到标准输出</strong>。**<code>2>&1</code> 的意思就是：** 把标准错误输出（2）重定向到标准输出（1）。也就是说，原本应该显示在终端上的错误信息，现在也会和正常的输出信息一起显示。</li><li><strong>标准输出（stdout）：</strong> 通常是显示在终端上的输出信息，用数字 1 表示。</li><li><strong>标准错误输出（stderr）：</strong> 通常是显示在终端上的错误信息，用数字 2 表示。</li></ul><h2 id="查看、修改文件"><a href="#查看、修改文件" class="headerlink" title="查看、修改文件"></a>查看、修改文件</h2><p><code>*.sh</code>是大多数sehll文件扩展名。<br><code>ls filename?</code>可以查看相应问号位数名字的文件。<br><code>convert filename.{png.jpg}</code>可以把图片格式从png转化成jpg，也可以加名字序号。<br><code>touch {folder1,folder2}/{file1..filen}</code>可以遍历这两个文件夹中的所有名字中含有某个字符的文件。<br><strong>touch</strong> 命令是一个常用的 Linux/Unix 命令，主要用于 <strong>修改文件的时间戳</strong> 或 <strong>创建新文件</strong>。<br><strong>mv -i</strong> 命令在移动或重命名文件时会 <strong>先询问用户</strong>，以避免误操作导致数据丢失。<br>convert可用于转换图片，ffmep可用于处理视频。</p><h2 id="shebang"><a href="#shebang" class="headerlink" title="shebang"></a>shebang</h2><p><strong>/usr/bin/env</strong>： 这部分告诉系统使用 <code>env</code> 命令来查找指定的可执行程序。<code>env</code> 命令会搜索当前用户的环境变量 <code>PATH</code>，找到第一个匹配的程序。<br>例如：</p><figure class="highlight d"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs d"><span class="hljs-meta">#!/usr/bin/env python3</span><br>print(<span class="hljs-string">"Hello, world!"</span>)<br></code></pre></td></tr></table></figure><p>用系统环境中的 Python3 解释器来执行这个脚本。</p><h2 id="调试-查找"><a href="#调试-查找" class="headerlink" title="调试 查找"></a>调试 查找</h2><p><code>shellcheck *.sh</code>检查警告和语法错误等。<br><strong><code>tldr command</code></strong> 查询命令用法，是一个为命令行工具提供简明易懂的使用说明的工具。它从冗长的 <code>man</code> 手册中提取出最常用的用法和示例，让用户能快速上手各种命令。<br><code>find</code>或<code>fd</code>用于查找文件名字，<code>locate</code>用于查找并列出所有，<code>grep</code>用于查找文件或文件夹中的具体内容，<code>rg</code>是ripgrep的缩写，是比grep更快的文本搜索工具，<code>fzf</code>模糊查找。<br>例如：<code>rg -U --files-without-match "^#\!" -t sh</code>表示<strong>在当前目录及其子目录中，查找所有的 shell 脚本文件（.sh），并找出那些</strong>第一行不是以 <code>#</code> 和 <code>!</code> 开头的的文件。其中<code>^</code>: 表示行首。<code>#</code>: 匹配字符#。<code>\!</code>: 匹配字符 !。<br><code>Ctrl+R</code>可以查找历史中出现要查找的字符的代码，每按一次就出现一条。<br><code>zsh</code>可以提供历史推荐代码。<br><code>tree</code>和<code>broot</code>都是很好的文件搜索和管理工具。</p><h2 id="课后练习"><a href="#课后练习" class="headerlink" title="课后练习"></a>课后练习</h2><p>1、要求我们使用 <code>ls</code> 命令，并通过添加不同的参数来实现以下功能：</p><ul><li><strong>显示所有文件（包括隐藏文件）</strong></li><li><strong>以人类可读的格式显示文件大小</strong></li><li><strong>按照文件访问时间排序</strong></li><li><strong>使用彩色输出</strong><br>答：ls -lah –color=auto<br>-l 以长格式显示；-a 显示所有文件，包括隐藏文件；-h 以人类可读的格式显示文件大小。</li></ul><p>2、编写两个 bash 函数 <code>marco</code> 和 <code>polo</code> 执行下面的操作。 每当你执行 <code>marco</code> 时，当前的工作目录应当以某种形式保存，当执行 <code>polo</code> 时，无论现在处在什么目录下，都应当 <code>cd</code> 回到当时执行 <code>marco</code> 的目录。 为了方便 debug，你可以把代码写在单独的文件 <code>marco.sh</code> 中，并通过 <code>source marco.sh</code> 命令，（重新）加载函数。<br>答：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-comment"># marco.sh</span><br><span class="hljs-comment"># marco 函数：保存当前目录路径</span><br><span class="hljs-function"><span class="hljs-title">marco</span></span>() {<br><span class="hljs-built_in">export</span> SAVED_DIR=<span class="hljs-string">"<span class="hljs-subst">$(pwd)</span>"</span> <span class="hljs-comment"># 使用 export 将变量导出，使得它可以被其他 shell 访问</span><br>}<br><span class="hljs-comment"># polo 函数：跳转回保存的目录</span><br><span class="hljs-function"><span class="hljs-title">polo</span></span>() {<br><span class="hljs-keyword">if</span> [ -z <span class="hljs-string">"<span class="hljs-variable">$SAVED_DIR</span>"</span> ]; <span class="hljs-keyword">then</span> <span class="hljs-comment"># 检查 SAVED_DIR 是否为空，其中-z STRING：如果字符串STRING的长度为零（即为空字符串），则返回真（true），否则返回假（false）。</span><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"Error: No directory saved. Please run 'marco' first."</span><br><span class="hljs-keyword">else</span><br><span class="hljs-built_in">cd</span> <span class="hljs-string">"<span class="hljs-variable">$SAVED_DIR</span>"</span> || <span class="hljs-built_in">echo</span> <span class="hljs-string">"Error: Could not change to directory <span class="hljs-variable">$SAVED_DIR</span>"</span><br><span class="hljs-keyword">fi</span><br>}<br></code></pre></td></tr></table></figure><p>3、假设您有一个命令，它很少出错。因此为了在出错时能够对其进行调试，需要花费大量的时间重现错误并捕获输出。 编写一段 bash 脚本，运行如下的脚本直到它出错，将它的标准输出和标准错误流记录到文件，并在最后输出所有内容。<br>答：修改后的代码：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-meta">#!/usr/bin/env bash</span><br><br> <br><br><span class="hljs-comment"># 初始化计数器</span><br><br>count=0<br><br>log_file=<span class="hljs-string">"command_log.txt"</span> <span class="hljs-comment"># 定义一个变量 log_file ，用来存储日志文件路径</span><br><br> <br><br><span class="hljs-comment"># 清空或创建日志文件</span><br><br>> <span class="hljs-string">"<span class="hljs-variable">$log_file</span>"</span> <span class="hljs-comment"># 将日志文件清空，如果文件不存在则创建。</span><br><br> <br><br><span class="hljs-keyword">while</span> <span class="hljs-literal">true</span>; <span class="hljs-keyword">do</span><br><br><span class="hljs-comment"># 运行给定的命令，同时捕获标准输出和标准错误</span><br><br>n=$((RANDOM % <span class="hljs-number">100</span>)) <span class="hljs-comment"># 生成一个 0 到 99 之间的随机数，并赋值给变量 n。</span><br><br><span class="hljs-keyword">if</span> [[ n -eq 42 ]]; <span class="hljs-keyword">then</span> <span class="hljs-comment"># 判断 n 是否等于 42，如果等于，则执行 then 后面的语句。</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"Something went wrong"</span> <span class="hljs-comment"># 标准输出。在bash中如果不指定输出重定向，则echo默认标准输出，因此不需要显式指定>&1。</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"The error was using magic numbers"</span> >&2 <span class="hljs-comment"># 标准错误输出</span><br><br><span class="hljs-comment"># 记录错误信息到日志文件</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"执行失败，总共执行了 <span class="hljs-variable">$count</span> 次"</span> >> <span class="hljs-string">"<span class="hljs-variable">$log_file</span>"</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"最后一次执行的输出："</span> >> <span class="hljs-string">"<span class="hljs-variable">$log_file</span>"</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"标准输出：Something went wrong"</span> >> <span class="hljs-string">"<span class="hljs-variable">$log_file</span>"</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"标准错误：The error was using magic numbers"</span> >> <span class="hljs-string">"<span class="hljs-variable">$log_file</span>"</span><br><br><span class="hljs-comment"># 显示日志文件内容</span><br><br><span class="hljs-built_in">cat</span> <span class="hljs-string">"<span class="hljs-variable">$log_file</span>"</span><br><br><span class="hljs-built_in">exit</span> 1 <span class="hljs-comment"># 退出脚本，并返回一个非零的退出状态，表示程序执行失败。从这里可以跳出while和do的循环。</span><br><br><span class="hljs-keyword">fi</span><br><br><span class="hljs-built_in">echo</span> <span class="hljs-string">"Everything went according to plan"</span><br><br>((count++))<br><br><span class="hljs-keyword">done</span><br></code></pre></td></tr></table></figure><p>对于标准输出和标准错误输出，有如下例子：</p><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs lua"># 将正确的信息输出到 <span class="hljs-built_in">output</span>.txt，错误信息输出到 <span class="hljs-built_in">error</span>.<span class="hljs-built_in">log</span><br>command > <span class="hljs-built_in">output</span>.txt <span class="hljs-number">2</span>> <span class="hljs-built_in">error</span>.<span class="hljs-built_in">log</span><br></code></pre></td></tr></table></figure><p>4、您的任务是编写一个命令，它可以递归地查找文件夹中所有的 HTML 文件，并将它们压缩成 zip 文件。注意，即使文件名中包含空格，您的命令也应该能够正确执行。<br>答：<code>find . -type f -name "*.html" -print0 | xargs -0 zip html_files.zip **使用 </code>-print0<code>和</code>xargs -0`：**</p><ul><li><code>-print0</code> 会让 <code>find</code> 输出的文件名之间用<strong>null 字符</strong> (<code>\0</code>) 进行分隔，而不是空格或换行符。</li><li><code>xargs -0</code> 会识别这些 null 字符，并将整个文件名（包括空格在内）作为一个单独的参数传递。</li></ul><p>5、编写一个命令或脚本递归的查找文件夹中最近使用的文件。更通用的做法，你可以按照最近的使用时间列出文件吗？<br>答：`find /path -type f -printf “%T+ %p\n” | sort -r<br>其中</p><ul><li><strong><code>-printf "%T+ %p\n"</code>：</strong> 以 “+%Y-%m-%d %H:%M:%S” 格式输出文件的访问时间和路径。</li><li><strong><code>sort -r</code>：</strong> 按时间戳逆序排序，最近访问的文件排在前面。</li></ul>]]></content>
<categories>
<category>shell</category>
</categories>
</entry>
<entry>
<title>Deep Learning by 3Blue1Brown</title>
<link href="/2024/10/15/3Blue1Brown/"/>
<url>/2024/10/15/3Blue1Brown/</url>
<content type="html"><![CDATA[<h1 id="Deep-Learning-by-3Blue1Brown"><a href="#Deep-Learning-by-3Blue1Brown" class="headerlink" title="Deep Learning by 3Blue1Brown"></a>Deep Learning by 3Blue1Brown</h1><h3 id="MLP的结构"><a href="#MLP的结构" class="headerlink" title="MLP的结构"></a>MLP的结构</h3><ul><li><strong>输入层：</strong> 接收原始数据，每个神经元对应一个输入特征。</li><li><strong>隐藏层：</strong> 位于输入层和输出层之间，包含多个神经元。隐藏层的神经元通过非线性激活函数对输入数据进行变换，从而提取出数据的特征。</li><li><strong>输出层：</strong> 输出网络的预测结果，神经元的个数取决于任务类型（分类、回归等）。<br><strong>MLP（Multi-Layer Perceptron）</strong>，即<strong>多层感知机</strong>，是深度学习中最基础的神经网络模型之一。它可以看作是感知机的扩展，通过引入多个隐藏层，使得网络能够学习到更加复杂的非线性映射关系。</li></ul><h3 id="Sigmoid函数"><a href="#Sigmoid函数" class="headerlink" title="Sigmoid函数"></a>Sigmoid函数</h3><p>也称为<strong>S型函数</strong>或<strong>逻辑斯蒂函数</strong>，是一种在生物学、信息科学、神经网络等多个领域广泛应用的数学函数。<br>表达式：<code>f(x) = 1 / (1 + e^(-x))</code></p><ul><li><strong>取值范围：</strong> 函数的值域为(0, 1)，这使得它可以将任意实数映射到0到1之间的概率值，常用于表示神经元的激活程度。</li><li><strong>连续性：</strong> Sigmoid函数是连续可导的，这使得它可以用于梯度下降等优化算法。</li><li><strong>非线性：</strong> Sigmoid函数是非线性的，这使得神经网络能够拟合复杂的非线性关系。<br>Sigmoid函数在神经网络中的作用</li><li><strong>作为激活函数：</strong> Sigmoid函数是早期神经网络中常用的激活函数。它可以将线性输入转化为非线性输出，从而使神经网络具有学习复杂模式的能力。</li><li><strong>将输出限制在0-1之间：</strong> 在二分类问题中，Sigmoid函数可以将输出解释为属于正类的概率。<br><strong>机器学习即是要找到正确的权重的偏置，使代价函数最小化。<br>神经网络本身就是一个大型函数。</strong><br><img src="/img/learn/20241014220021.png"></li></ul><h3 id="反向传播算法（Backpropagation，BP）"><a href="#反向传播算法（Backpropagation，BP）" class="headerlink" title="反向传播算法（Backpropagation，BP）"></a>反向传播算法（Backpropagation，BP）</h3><p>工作原理：</p><ol><li><strong>前向传播：</strong> 输入数据从输入层开始，经过隐藏层，最后到达输出层。每一层的神经元计算加权和，并通过激活函数得到输出。</li><li><strong>计算损失：</strong> 将网络的输出与实际的标签进行比较，计算出损失函数的值。常见的损失函数有均方误差、交叉熵损失等。</li><li><strong>反向传播：</strong> 从输出层开始，计算损失函数对输出层每个神经元的偏导数。然后利用链式法则，逐层向前计算损失函数对隐藏层和输入层神经元的偏导数。</li><li><strong>更新参数：</strong> 根据计算得到的梯度，利用梯度下降法更新网络中的权重和偏置。<br>四个计算方程：</li></ol><ul><li><p><strong>误差项 δ 的定义：</strong></p> <figure class="highlight cos"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cos">δ<span class="hljs-symbol">^L</span> = ∂C/∂z<span class="hljs-symbol">^L</span><br></code></pre></td></tr></table></figure><p> 其中：</p><ul><li>δ^L：第 L 层第 j 个神经元的误差项。</li><li>C：损失函数。</li><li>z^L：第 L 层第 j 个神经元的输入。</li></ul></li><li><p><strong>误差项 δ 的计算：</strong></p> <figure class="highlight cos"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cos">δ<span class="hljs-symbol">^l</span> = (<span class="hljs-keyword">w</span>^(<span class="hljs-keyword">l</span>+<span class="hljs-number">1</span>))<span class="hljs-symbol">^T</span> δ^(<span class="hljs-keyword">l</span>+<span class="hljs-number">1</span>) * σ'(z<span class="hljs-symbol">^l</span>)<br></code></pre></td></tr></table></figure><p> 其中：</p><ul><li>w^(l+1)：第 l 层到第 l+1 层之间的权重矩阵。</li><li>σ’(z^l)：第 l 层第 j 个神经元的激活函数的导数。</li></ul></li><li><p><strong>偏置的梯度：</strong></p> <figure class="highlight cos"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cos">∂C/∂b<span class="hljs-symbol">^l_j</span> = δ<span class="hljs-symbol">^l_j</span><br></code></pre></td></tr></table></figure></li><li><p><strong>权重的梯度：</strong></p> <figure class="highlight cos"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cos">∂C/∂<span class="hljs-keyword">w</span><span class="hljs-symbol">^l_jk</span> = a^(<span class="hljs-keyword">l</span>-<span class="hljs-number">1</span>)_k δ<span class="hljs-symbol">^l_j</span><br></code></pre></td></tr></table></figure><p> 其中：</p><ul><li>a^(l-1)_k：第 l-1 层第 k 个神经元的输出。</li></ul></li></ul><h3 id="随机梯度下降（Stochastic-Gradient-Descent，SGD）"><a href="#随机梯度下降（Stochastic-Gradient-Descent，SGD）" class="headerlink" title="随机梯度下降（Stochastic Gradient Descent，SGD）"></a>随机梯度下降（Stochastic Gradient Descent，SGD）</h3><p>SGD 的主要特点在于每次迭代只随机选取一个样本（或一小批样本）来计算梯度，并根据这个梯度来更新参数。</p><h3 id="gpt"><a href="#gpt" class="headerlink" title="gpt"></a>gpt</h3><p>GPT，全称Generative Pre-trained Transformer，中文翻译为生成式预训练变换器。<br><strong>Transformer架构:</strong> GPT模型的核心是Transformer架构。Transformer是一种神经网络架构，特别适合处理序列数据，如文本。它利用自注意力机制，让模型能够关注输入序列的不同部分，从而更好地理解文本的上下文。<br>word embedding过后方向可以代表语义。</p><h3 id="softmax函数"><a href="#softmax函数" class="headerlink" title="softmax函数"></a>softmax函数</h3><p>它的作用是将一个任意实数的K维向量“压缩”到另一个K维实向量中，使得每一个元素的范围都在(0,1)之间，并且所有元素的和为1。换句话说，它将输入的数值转化为一个概率分布。其输入称为logits，输出称为probabilities。<br>Softmax函数引入温度参数后的形式：</p><figure class="highlight excel"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs excel">y_i = <span class="hljs-built_in">exp</span>(z_i / <span class="hljs-built_in">T</span>) / Σ(<span class="hljs-built_in">exp</span>(z_j / <span class="hljs-built_in">T</span>))<br></code></pre></td></tr></table></figure><ul><li><code>y_i</code>: 输出向量y的第i个元素，表示输入数据属于第i类的概率。</li><li><code>z_i</code>: 输入向量z的第i个元素。</li><li><code>T</code>: 温度参数。<br><strong>高温度 (T较大):</strong> 概率分布会变得更加平滑，各类别之间的概率差异减小，模型会倾向于给所有类别分配较高的概率。这在一些场景下可以增加模型的探索性，例如在生成模型中生成多样化的样本。 <strong>低温度 (T较小):</strong> 概率分布会变得更加尖锐，模型会更加自信地将概率分配给最有可能的类别。这在需要模型做出明确决策的场景下非常有用，例如分类任务。</li></ul><h3 id="注意力机制"><a href="#注意力机制" class="headerlink" title="注意力机制"></a>注意力机制</h3><p>它允许模型在处理大量输入数据时，有选择性地关注其中最相关的部分，从而提升模型的性能。<br><strong>数学表达:</strong></p><figure class="highlight mathematica"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs mathematica"><span class="hljs-variable">Attention</span><span class="hljs-punctuation">(</span><span class="hljs-built_in">Query</span><span class="hljs-operator">,</span> <span class="hljs-built_in">Key</span><span class="hljs-operator">,</span> <span class="hljs-variable">Value</span><span class="hljs-punctuation">)</span> <span class="hljs-operator">=</span> <span class="hljs-variable">softmax</span><span class="hljs-punctuation">(</span><span class="hljs-built_in">Query</span> <span class="hljs-operator">*</span> <span class="hljs-built_in">Key</span><span class="hljs-operator">^</span><span class="hljs-variable">T</span> <span class="hljs-operator">/</span> <span class="hljs-variable">sqrt</span><span class="hljs-punctuation">(</span><span class="hljs-type">d_k</span><span class="hljs-punctuation">)</span><span class="hljs-punctuation">)</span> <span class="hljs-operator">*</span> <span class="hljs-variable">Value</span><br></code></pre></td></tr></table></figure><ul><li><strong>Query:</strong> 表示当前需要关注的元素。</li><li><strong>Key:</strong> 表示所有输入元素的键。</li><li><strong>Value:</strong> 表示所有输入元素的值。</li><li><strong>softmax:</strong> 将相似度转化为概率分布。</li><li><strong>d_k:</strong> 是Key向量的维度。</li></ul><h3 id="掩码"><a href="#掩码" class="headerlink" title="掩码"></a>掩码</h3><p>通常，掩码是一个与输入序列长度相同的向量，每个元素的值为0或1。值为0的位置会被屏蔽，即在计算注意力权重时，这些位置的权重会被设置为一个非常小的负数（例如-inf），从而使得softmax函数将这些位置的概率归为0。<br>掩码的应用场景</p><ul><li><strong>机器翻译:</strong> 防止模型在翻译过程中“偷看”目标序列。</li><li><strong>文本生成:</strong> 确保生成的文本是连贯的，符合语法规则。</li><li><strong>问答系统:</strong> 控制模型只关注与问题相关的部分。</li></ul>]]></content>
<categories>
<category>deep learning</category>
</categories>
</entry>
<entry>
<title>fyp</title>
<link href="/2024/10/07/fyp/"/>
<url>/2024/10/07/fyp/</url>
<content type="html"><![CDATA[<h1 id="fyp"><a href="#fyp" class="headerlink" title="fyp"></a>fyp</h1><p><strong>使用的mcu：</strong><br>esp32<br><strong>历史设备：</strong><br>Kinect for Windows<br><a href="https://learn.microsoft.com/en-us/windows/apps/design/devices/kinect-for-windows">https://learn.microsoft.com/en-us/windows/apps/design/devices/kinect-for-windows</a><br>Nintendo Wii<br><a href="https://www.nintendo.com/en-gb/Wii/Wii-94559.html?srsltid=AfmBOor8Xo1tet1IBXyGMjGHbShVLkBfER8bSEsyGtisKmdRG3AfMNrp">https://www.nintendo.com/en-gb/Wii/Wii-94559.html?srsltid=AfmBOor8Xo1tet1IBXyGMjGHbShVLkBfER8bSEsyGtisKmdRG3AfMNrp</a><br><strong>相关仓库：</strong></p><ul><li><strong>MediaPipe Solutions（重点参考应用案例）:</strong> Google 开发的跨平台框架，包含多种机器学习解决方案，其中手势识别解决方案非常强大，提供了实时、高精度的手势跟踪和分类功能。<ul><li><strong>链接:</strong> <a href="https://github.com/google/mediapipe">https://github.com/google/mediapipe</a></li></ul></li><li><strong>OpenPose:</strong> 一个实时多人二维姿态估计库，可以用于手势识别。它可以准确地检测人体关键点，为手势识别提供基础数据。<ul><li><strong>链接:</strong> <a href="https://github.com/CMU-Perceptual-Computing-Lab/openpose">https://github.com/CMU-Perceptual-Computing-Lab/openpose</a></li></ul></li><li><strong>TensorFlow Examples:</strong> TensorFlow 官方提供的示例代码，包含多个手势识别相关的示例，如使用 TensorFlow 和 Keras 实现手势分类。<ul><li><strong>链接:</strong> <a href="https://github.com/tensorflow/examples">https://github.com/tensorflow/examples</a></li></ul></li><li><strong>RealGes:</strong> 一个基于视觉的实时动态手势识别系统，包含数据采集、模型训练、实时预测等完整流程。<ul><li><strong>链接:</strong> <a href="https://github.com/wzh99/RealGes">https://github.com/wzh99/RealGes</a><br><strong>编程和平台：</strong><br>arduino（功能较弱）<br><strong>Arduino IDE：</strong> Arduino IDE 是一个开源的集成开发环境，通过安装 ESP32 的核心库，可以方便地进行开发。<br>vscode上可以装插件用C++编程。<br><strong>学习：</strong><br>不要看forest和svm，要从cnn、rnn、gan等开始看起。从类似项目中找使用cnn、rnn的方法，从中入手学习。每个人都要交单独的report。中期报告的时候要有设计方案和初步产品。汇报进度时准备PPT。<br><strong>理想流程：</strong><br>先做图像处理，再把数据放进learing model中处理，观察得到的结果。<br><strong>老师推荐的学习链接：</strong><br><a href="https://docs.espressif.com/projects/esp-dl/zh_CN/latest/esp32/index.html">https://docs.espressif.com/projects/esp-dl/zh_CN/latest/esp32/index.html</a><br><a href="https://developer.espressif.com/blog/hand-gesture-recognition-on-esp32-s3-with-esp-deep-learning/">https://developer.espressif.com/blog/hand-gesture-recognition-on-esp32-s3-with-esp-deep-learning/</a><br><a href="https://github.com/margaretmz/awesome-tensorflow-lite">https://github.com/margaretmz/awesome-tensorflow-lite</a><br><a href="https://github.com/espressif/esp-tflite-micro">https://github.com/espressif/esp-tflite-micro</a></li></ul></li></ul>]]></content>
<categories>
<category>fyp</category>
</categories>
</entry>
<entry>
<title>The Shell</title>
<link href="/2024/10/04/The%20Shell/"/>
<url>/2024/10/04/The%20Shell/</url>
<content type="html"><![CDATA[<h1 id="The-Shell"><a href="#The-Shell" class="headerlink" title="The Shell"></a>The Shell</h1><p>刚启动shell terminal的时候出现的是shell prompt<br>shell是一门编程语言，依靠invariable environment</p><h2 id="date-echo-which"><a href="#date-echo-which" class="headerlink" title="date echo which"></a>date echo which</h2><p><code>date</code>显示日期<br><code>echo "string"</code>回复字符串内容<br><code>echo $PATH</code>查看路径<br><code>which echo</code>查看echo程序位置</p><h2 id="path"><a href="#path" class="headerlink" title="path"></a>path</h2><p>linux和macOS中用/(forward slash)分隔path，windows中用(back slash)分隔path<br><code>pwd</code>print working directory<br><code>cd</code>change directory<br><code>.</code>current directory <code>..</code>parent directory <code>~</code>表示用户目录，例如在linux中表示/home/dahao，在windows中表示C:\Users\dahao <code>-</code>表示上一个目录</p><h2 id="ls-man"><a href="#ls-man" class="headerlink" title="ls man"></a>ls man</h2><p><code>ls</code>留空表示显示当前目录，后面也可以加例如..来显示其父目录<br><code>ls --help</code>help <code>ls -l</code>long list help<br><img src="/img/learn/20241004144604.png"><br>每三个字母为一组，第一组为user owner的权限，第二组为group owner的权限，第三组为everyone else，每行的第一个字母表示文件的类型或文件夹</p><ul><li><strong>d:</strong> 代表这是一个目录 (directory)。</li><li><strong>-:</strong> 代表这是一个普通文件 (regular file)。</li><li><strong>l:</strong> 代表这是一个符号链接 (symbolic link)。</li><li><strong>b:</strong> 代表这是一个块设备文件 (block special file)。</li><li><strong>c:</strong> 代表这是一个字符设备文件 (character special file)。</li><li><strong>p:</strong> 代表这是一个管道文件 (FIFO pipe)。</li><li><strong>s:</strong> 代表这是一个套接字文件 (socket)。<br>其中write的权限，对于文件夹来说表示可以在其中rename\create\remove files。例如对文件夹中的某个文件具有write权限，但对文件夹没有write权限，那么只能把文件内容清空而不能把这个文件从文件夹中删掉。<br>execute权限表示想访问一个文件（夹）就需要对它和它父目录有execute权限。<br><code>man ls</code>manual page，可能需要q退出<br><code>Ctrl+L</code>clean terminal<br>在macOS中，<code>xdg-open file</code>用对应软件打开文件</li></ul><h2 id="mv-cp-rm"><a href="#mv-cp-rm" class="headerlink" title="mv cp rm"></a>mv cp rm</h2><p><code>mv</code>move and rename<br><code>cp</code>copy and rename<br><code>rm</code>remove，linux中不能rm文件夹，因为不是递归删除，需要删除加<code>-r</code><br><code>rmdir</code>only if the directory is empty<br><code>mkdir</code>创建新文件夹</p><h2 id="input-output"><a href="#input-output" class="headerlink" title="input output"></a>input output</h2><p><code><</code>input <code>></code>output <code>>></code>append<br><code>cat < file</code>表示把file的内容当做input内容，默认output到terminal<br><code>cat < file > file2</code>表示output到file2中<br><code>tail -n+number</code>例如<code>tail -n2</code>表示print最后2行<br><code>A | B</code>把A代码段的输出作为B代码段的输入，A和B彼此不知<br><strong>重定向的优先级更高</strong></p><h2 id="root"><a href="#root" class="headerlink" title="root"></a>root</h2><p>linux和macOS的root用户id是0，是super user(su)，类似于windows的administrator<br><code>ls /sys</code>kernel parameters<br><code>$</code>表示你不是在以root方式运行 <code>#</code>相反<br><code>sudo su</code>获得root权限 <code>exit</code>退出<br>例如<code>echo 1060 | sudo tee brightness</code>表示先把1060作为输入内容，再获得管理员权限并同时输出到brightness文件和terminal screen</p><h3 id="为什么sudo-echo-500-brightness不行而echo-500-sudo-tee-brightness可以"><a href="#为什么sudo-echo-500-brightness不行而echo-500-sudo-tee-brightness可以" class="headerlink" title="为什么sudo echo 500 > brightness不行而echo 500 | sudo tee brightness可以"></a>为什么<code>sudo echo 500 > brightness</code>不行而<code>echo 500 | sudo tee brightness</code>可以</h3><p>原因：<br><code>sudo echo 500 > brightness</code> 为什么不行？</p><ul><li><strong>重定向优先级高于 sudo：</strong> 在这个命令中，Shell首先会执行重定向操作 <code>> brightness</code>，试图将 <code>echo 500</code> 的输出重定向到 <code>brightness</code> 文件。</li><li><strong>权限不足：</strong> 在执行重定向时，Shell发现你没有写入 <code>brightness</code> 文件的权限，因此报错“不允许”。</li><li><strong>sudo 权限范围：</strong> 即使你使用了 <code>sudo</code>，它也只是提升了 <code>echo</code> 命令的权限，并没有提升 Shell 执行重定向操作的权限。<br><code>echo 500 | sudo tee brightness</code> 为什么可以？</li><li><strong>管道操作：</strong> <code>|</code> 符号表示管道，将 <code>echo 500</code> 的输出作为 <code>sudo tee brightness</code> 的输入。</li><li><strong>sudo tee 权限：</strong> <code>sudo tee brightness</code> 这部分命令由 <code>sudo</code> 提升了权限，因此可以将数据写入 <code>brightness</code> 文件。</li><li><strong>tee 命令的作用：</strong> <code>tee</code> 命令可以同时将标准输入的内容输出到标准输出和指定的文件。<br>总结：</li><li><strong>重定向操作发生在 sudo 之前：</strong> 在 <code>sudo echo 500 > brightness</code> 中，重定向操作先于 <code>sudo</code> 执行，导致权限不足。</li><li><strong>管道操作将输出交给 sudo 处理：</strong> 在 <code>echo 500 | sudo tee brightness</code> 中，重定向操作通过管道交给 <code>sudo tee</code> 处理，从而绕过了权限问题。</li></ul><h2 id="exercise"><a href="#exercise" class="headerlink" title="exercise"></a>exercise</h2><p>使用<code>echo $SHELL</code>命令可以查看您的shell是否满足要求。如果打印结果为<code>/bin/bash</code>或<code>/usr/bin/zsh</code>则证明可以使用类Unix shell的。<br>如何使用shell，将以下内容一行一行地写入 semester 文件：<br> <code>#!/bin/sh curl --head --silent https://missing.csail.mit.edu</code><br> 答案：<code>echo '#!/bin/sh' > semester</code><br><code>echo 'curl --head --silent https://missing.csail.mit.edu' >> semester</code><br>为什么使用单引号而不是双引号？</p><ul><li>单引号： 单引号内的所有字符都会被原样输出，包括特殊字符（如$、`、\等）。也就是说，单引号内的内容不会被shell解释。</li><li>双引号： 双引号内的部分特殊字符会被shell解释，比如变量会被替换，命令替换也会执行。<br>为什么 # 和 ! 不需要转义？</li><li>#号： 在shell脚本中，#号通常表示注释。当shell遇到#号时，会忽略该行#号后面的所有内容。但是，当#号被包含在单引号或双引号中时，它就失去了注释的作用，而被视为普通字符。</li><li>!号：!号在shell中也有多种用途，但在这里，它只是字符串的一部分，不需要转义。</li><li>如果要在单引号中包含单引号本身，需要使用转义字符'。</li><li>如果需要在单引号中嵌入变量，可以先将变量的值赋值给另一个变量，然后使用双引号包裹这个变量。</li></ul><h2 id="chmod"><a href="#chmod" class="headerlink" title="chmod"></a>chmod</h2><p>可以通过修改 chmod 命令中的字母来赋予文件不同的权限。</p><h3 id="权限字母详解"><a href="#权限字母详解" class="headerlink" title="权限字母详解"></a>权限字母详解</h3><ul><li>u: 表示文件所有者 (user)</li><li>g: 表示与文件所有者同组的用户 (group)</li><li>o: 表示其他用户 (others)</li><li>a: 表示所有用户 (all users，即 u、g、o 的总和)</li><li>+: 增加权限</li><li>-: 删除权限</li><li>=: 设定权限（仅设置指定的权限，其他权限取消）</li><li>r: 读权限</li><li>w: 写权限</li><li>x: 执行权限<br>举例<br>chmod g-w semester: 取消同组用户的写权限<br>chmod o=r semester: 仅给其他用户赋予读权限，其他权限取消<br>chmod a+x semester: 给所有用户增加执行权限</li></ul><h3 id="数字表示法"><a href="#数字表示法" class="headerlink" title="数字表示法"></a>数字表示法</h3><p>除了使用字母表示，还可以使用数字表示权限，每个数字代表一组权限：</p><ul><li>rwx: 7</li><li>rw-: 6</li><li>r-x: 5</li><li>r–: 4</li><li>-wx: 3</li><li>-w-: 2</li><li>–x: 1</li><li>—: 0<br>例如：<br>chmod 755 semester: 表示文件所有者有读、写、执行权限，同组用户和其他人有读和执行权限。<br><strong>Shebang 行（#!）的作用：</strong><br>解释器指定: Shebang 行（#!）位于脚本的第一行，其后紧跟着解释器的路径。这行告诉系统，用指定的解释器来执行这个脚本。<br>示例: #!/bin/sh 表示使用 /bin/sh 这个 shell 来解释执行脚本。<br>作用: 当系统执行一个脚本时，会首先读取 Shebang 行，然后找到指定的解释器，并把脚本的内容传递给解释器执行。<br><strong>获取修改时间：</strong></li><li>方法一：<code>stat -c %y semester | cut -d ' ' -f 1 > ./last-modified.txt</code><br>stat -c %y semester：获取semester文件的修改时间，并以年-月-日的格式输出。<br>cut -d ‘ ‘ -f 1：从输出结果中提取第一个字段（即日期），也就是修改时间。<blockquote><p>~/last-modified.txt：将提取到的修改时间重定向到当前用户主目录下的last-modified.txt文件中。</p></blockquote></li><li>方法二：<code>ls -l semester | awk '{print $6" "$7}' | cut -d ' ' -f 1 > ./last-modified.txt</code><br>ls -l semester：以长格式列出semester文件的信息，包括修改时间。<br>awk ‘{print $6” “$7}’：提取输出中的第6和第7个字段，即月日和时间。<br>cut -d ‘ ‘ -f 1：从提取到的结果中提取第一个字段（即日期），也就是修改时间。<blockquote><p>~/last-modified.txt：将提取到的修改时间重定向到当前用户主目录下的last-modified.txt文件中。</p></blockquote></li></ul>]]></content>
<categories>
<category>shell</category>
</categories>
</entry>
<entry>
<title>Hello World</title>
<link href="/2024/10/02/hello-world/"/>
<url>/2024/10/02/hello-world/</url>
<content type="html"><![CDATA[<p>Welcome to <a href="https://hexo.io/">Hexo</a>! This is your very first post. Check <a href="https://hexo.io/docs/">documentation</a> for more info. If you get any problems when using Hexo, you can find the answer in <a href="https://hexo.io/docs/troubleshooting.html">troubleshooting</a> or you can ask me on <a href="https://github.com/hexojs/hexo/issues">GitHub</a>.</p><h2 id="Quick-Start"><a href="#Quick-Start" class="headerlink" title="Quick Start"></a>Quick Start</h2><h3 id="Create-a-new-post"><a href="#Create-a-new-post" class="headerlink" title="Create a new post"></a>Create a new post</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash">$ hexo new <span class="hljs-string">"My New Post"</span><br></code></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/writing.html">Writing</a></p><h3 id="Run-server"><a href="#Run-server" class="headerlink" title="Run server"></a>Run server</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash">$ hexo server<br></code></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/server.html">Server</a></p><h3 id="Generate-static-files"><a href="#Generate-static-files" class="headerlink" title="Generate static files"></a>Generate static files</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash">$ hexo generate<br></code></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/generating.html">Generating</a></p><h3 id="Deploy-to-remote-sites"><a href="#Deploy-to-remote-sites" class="headerlink" title="Deploy to remote sites"></a>Deploy to remote sites</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash">$ hexo deploy<br></code></pre></td></tr></table></figure><p>More info: <a href="https://hexo.io/docs/one-command-deployment.html">Deployment</a></p>]]></content>
</entry>
</search>