-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathindex.html
1082 lines (668 loc) · 75.2 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>perlcperl - a perl5 with classes, types, compilable, company friendly</title>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<link rev="made" href="mailto:[email protected]" />
</head>
<body>
<ul id="index">
<li><a href="#NAME">NAME</a></li>
<li><a href="#SYNOPSIS">SYNOPSIS</a></li>
<li><a href="#DEVELOPMENT-SETUP">DEVELOPMENT SETUP</a></li>
<li><a href="#Incompatible-changes">Incompatible changes</a></li>
<li><a href="#Fix-breakage-and-bad-design">Fix breakage and bad design</a>
<ul>
<li><a href="#signatures">signatures</a></li>
<li><a href="#Changed-calls-to-signatures">Changed calls to signatures</a></li>
<li><a href="#Fix-my-_-handling">Fix my $_ handling</a></li>
<li><a href="#Undo-B-bootstrap-breakage">Undo B bootstrap breakage</a></li>
<li><a href="#Undo-constant-folding-i_opt-de-optimization">Undo constant folding i_opt de-optimization</a></li>
<li><a href="#Undo-support-for-binary-names">Undo support for binary names</a></li>
<li><a href="#Warn-on-0shellcode-attempts-for-names">Warn on \0shellcode attempts for names</a></li>
<li><a href="#strict-names">strict names</a></li>
<li><a href="#TR39-confusable-names">TR39 confusable names</a></li>
<li><a href="#Undo-the-double-readonly-system">Undo the double readonly system</a></li>
<li><a href="#Fix-the-hashes">Fix the hashes</a></li>
<li><a href="#Compile-time-attribute-hooks">Compile-time attribute hooks</a></li>
<li><a href="#Run-time-attribute-variables">Run-time attribute variables</a></li>
<li><a href="#coretypes:-Int-UInt-Num-Str">coretypes: Int, UInt, Num, Str</a></li>
<li><a href="#native-unboxed-types:-int-uint-num-str-NY">native unboxed types: int, uint, num, str (NY)</a></li>
<li><a href="#types">types</a></li>
<li><a href="#const-for-all">:const for all</a></li>
<li><a href="#Compile-time-optimizations">Compile-time optimizations</a>
<ul>
<li><a href="#Static-method-calls">Static method calls</a></li>
</ul>
</li>
<li><a href="#Proper-object-system">Proper object system</a></li>
<li><a href="#Typed-and-shaped-arrays">Typed and shaped arrays</a></li>
<li><a href="#Typed-and-shaped-hashes-NY">Typed and shaped hashes (NY)</a></li>
<li><a href="#Inlined-functions-NY">Inlined functions (NY)</a></li>
<li><a href="#Rewrote-critical-core-modules-in-C-as-builtins">Rewrote critical core modules in C as builtins</a>
<ul>
<li><a href="#strict-1.11c-as-builtin">strict 1.11c as builtin</a></li>
<li><a href="#attributes-0.26_01c-as-builtin">attributes 0.26_01c as builtin</a></li>
<li><a href="#DynaLoader-2.00c-and-XSLoader-as-builtins">DynaLoader 2.00c and XSLoader as builtins</a></li>
<li><a href="#Config-as-XS-extension">Config as XS extension</a></li>
<li><a href="#warnings-as-XS-extension-NY">warnings as XS extension (NY)</a></li>
</ul>
</li>
<li><a href="#Added-the-compiler-back-to-core">Added the compiler back to core</a></li>
<li><a href="#Backport-core-testsuite-fixes-for-the-compiler">Backport core testsuite fixes for the compiler</a></li>
<li><a href="#libffi-in-core">libffi in core</a></li>
<li><a href="#unicode-folding-tables-as-XS-extension">unicode folding tables as XS extension</a></li>
</ul>
</li>
<li><a href="#Plans-for-further-core-features">Plans for further core features</a>
<ul>
<li><a href="#Lexical-methods">Lexical methods</a>
<ul>
<li><a href="#Carp-2.00c-as-builtin">Carp 2.00c as builtin</a></li>
</ul>
</li>
<li><a href="#Remove-Attribute::Handler-from-core">Remove Attribute::Handler from core</a></li>
</ul>
</li>
<li><a href="#Longer-term-goals">Longer term goals</a>
<ul>
<li><a href="#Faster-functions-and-method-calls">Faster functions and method calls</a></li>
<li><a href="#tailcall-elimination">tailcall elimination</a></li>
<li><a href="#Macros">Macros</a></li>
<li><a href="#Editor-integration">Editor integration</a></li>
<li><a href="#Jit">Jit</a></li>
<li><a href="#Optimize-the-vm-the-runloop">Optimize the vm, the runloop</a></li>
<li><a href="#Better-symbol-table">Better symbol table</a></li>
<li><a href="#Coro-support">Coro support</a></li>
<li><a href="#Untangle-the-IO-layer-mess">Untangle the IO layer mess</a></li>
<li><a href="#Bring-back-proper-match-and-given-when">Bring back proper `match` and `given/when`</a></li>
<li><a href="#clp">clp</a></li>
</ul>
</li>
<li><a href="#Development-policies">Development policies</a></li>
<li><a href="#With-classes-types-compilable-company-friendly">With classes, types, compilable, company friendly</a></li>
<li><a href="#How-to-detect-cperl">How to detect cperl?</a></li>
<li><a href="#Type-links">Type links</a></li>
</ul>
<h1 id="NAME">NAME</h1>
<p>perlcperl - a perl5 with classes, types, compilable, company friendly</p>
<h1 id="SYNOPSIS">SYNOPSIS</h1>
<p>Description of changes and enhancements of the cperl variant of perl5.</p>
<h1 id="DEVELOPMENT-SETUP">DEVELOPMENT SETUP</h1>
<p>init from a perl5 git dir:</p>
<pre><code> git remote add cp ssh://[email protected]/perl11/cperl.git
git fetch cp
git checkout -t cp master</code></pre>
<p>for rerere:</p>
<pre><code> git config --add rerere enable
git config --add rerere.autoupdate true
git submodule update --init
ln -s ../.git-rr-cache .git/rr-cache
git branch -r | grep cp/</code></pre>
<p>We need a shared rerere cache to be able to continously merge and rebase with perl5 upstream and our branch progress. See <a href="https://medium.com/@porteneuve/fix-conflicts-only-once-with-git-rerere-7d116b2cec67">https://medium.com/@porteneuve/fix-conflicts-only-once-with-git-rerere-7d116b2cec67</a> The commits with <i>git-rr-cache</i> and <i>cp-rb</i> in the Subject can be safely ignored upstream.</p>
<p>All branches are frequently rebased. Use the provided helpers <i>cp-rb</i>, <i>cp-rbi</i>, <i>cp-lb</i> and <i>cp-rh</i>.</p>
<p>The fastest performance unstable branches are currently <i>feature/gh87-types-proto</i>, <i>feature/gh23-inline-subs</i> and <i>feature/gh14-native-types</i>. The branch with the biggest memory savings is <i>feature/gh9-warnings-xs</i>.</p>
<p><b>INSTALL:</b></p>
<p>Simple:</p>
<pre><code> ./Configure -sder -Dusedevel -Dusecperl
make -s -j4
make -s -j4 test
sudo make install</code></pre>
<p>Experts:</p>
<pre><code> ./Configure -sder -Dusedevel -Dusecperl \
-Accflags='-msse4.2 -DPERL_FAKE_SIGNATURE' --optimize='-O3 -g' \
-Dinstallman1dir=none -Dinstallman3dir=none -Dinstallsiteman1dir=none \
-Dinstallsiteman3dir=none
make -s -j4 ECHO=true
make -s -j4 ECHO=true test
sudo make install</code></pre>
<p>Debugging with private archlibs and exename:</p>
<pre><code> git=`git rev-parse @|cut -c-7`
archname="`uname -s`-debug@$git"
exesuff=d-nt
./Configure -sder -Dusedevel -Dusecperl -DDEBUGGING \
-Accflags='-msse4.2 -DDEBUG_LEAKING_SCALARS -DPERL_FAKE_SIGNATURE' --optimize='-g3' \
-Dinstallman1dir=none -Dinstallman3dir=none -Dinstallsiteman1dir=none -Dinstallsiteman3dir=none \
-Darchname="$archname" -Darchlib="/usr/local/lib/cperl/5.22.1/$archname" \
-Dsitearch="/usr/local/lib/cperl/site_cperl/5.22.1/$archname" \
-Dperlpath="/usr/local/bin/cperl5.22.1$exesuff" -Dstartperl="#!/usr/local/bin/cperl5.22.1$exesuff" \
make -s -j4 ECHO=true
make -s -j4 ECHO=true test
sudo make install</code></pre>
<h1 id="Incompatible-changes">Incompatible changes</h1>
<p>cperl tries to follow the old perl5 spirit and principles, unlike recent perl5 changes, which wildly deviate from it.</p>
<p>cperl can parse and run 99.9% all of perl5 code. But there are a few incompatibilities, which arise from late perl5 signatures design changes and 2002 constant folding changes which we do not follow. perl5 signatures are marked as experimental after all.</p>
<p>CPAN works. Some modules currently need patches in our distroprefs repo, where the maintainers refuse to support it. See <a href="https://github.com/rurban/distroprefs">https://github.com/rurban/distroprefs</a>.</p>
<p>Some internal modules are explicitly modernized, which is denoted by the <b>c</b> suffix in the version number. Those modules use proper signatures and hereby do safer type-checking at compile-time, and are also 2x faster because of using signatures. Some toolchain modules also switched over to support 10x faster and more secure XS variants of JSON and YAML, and support the cperl improvements (i.e. builtin strict, DynaLoader, XSLoader) and add more security (i.e. DynaLoader, Storable, YAML). These modules usually get a +1 major version bump, so you cannot easily override them accidently with worse cpan updates. Such as EUMM, bignum or Test2.</p>
<dl>
<dt id="Illegal-prototypes-die-are-not-stored"><b>Illegal prototypes die, are not stored</b></dt>
<dd>
<p>In perl illegal prototypes warn with <code>'illegalproto'</code> and are stored as such.</p>
<p>In cperl illegal prototypes in signatures used without <code>:prototype()</code> immediately die, they cannot be suppressed with <code>no warnings 'illegalproto';</code> and they are not stored.</p>
<p><b>Rationale:</b> Illegal prototypes are parsed as signatures. Illegal signatures throw a parser error. The <code>'illegalproto'</code> warning is only thrown within explicit extra <code>:prototype()</code> declarations.</p>
<p>Technically this is not a incompatible change, as signatures are marked as experimental.</p>
</dd>
<dt id="is-empty-in-functions-with-signatures"><b><code>@_</code> is empty in functions with signatures</b></dt>
<dd>
<p>We only copy or reference arguments to signature variables, but not to <code>@_</code> also. <code>@_</code> is empty when signatures are declared.</p>
<p><b>Rationale:</b> With signatures copying all values to <code>@_</code> also leads to double copying which is 2x slower. <code>@_</code> is not needed anymore. Either use no signatures to use <code>@_</code> variables or use signatures.</p>
<p>Technically this is not a incompatible change, as signatures are marked as experimental.</p>
</dd>
<dt id="Empty-signature-variables-die"><b>Empty signature variables <code>$</code> die</b></dt>
<dd>
<p>Using a bare <code>$</code> sigil signature variable is illegal in cperl, but legal in perl.</p>
<p><b>Rationale:</b> This clashes with prototypes. A bare <code>$</code> is a prototype declaration, not a signature. Use a name in the signature and don't use this name in the body to get back to the same behaviour.</p>
<p>Technically this is not a incompatible change, as signatures are marked as experimental.</p>
</dd>
<dt id="negative-integer-modulo"><b>negative integer modulo</b></dt>
<dd>
<p>With typed integer variables or integer constants the modulo operator in cperl works like with use integer, it uses the libc functionality for <code>"%"</code>; using the <code>i_modulo</code> operator which is different than the generic perl5 <code>modulo</code> operator.</p>
<p>Perl5 without use integer uses a different modulo definition: <i>"If $n is negative, then $m % $n is $m minus the smallest multiple of $n that is not less than $m (that is, the result will be less than or equal to zero)"</i> from <a href="/cperl/perlop.html#Multiplicative-Operators">"Multiplicative Operators" in perlop</a>.</p>
<p>cperl with typed or constant integers behaves like perl5 with use integer or before 2002, when this was changed (<code>e7311069</code>). The compiler simply promotes <code>modulo</code> to <code>i_modulo</code> if both arguments are integers at compile-time. To fall back to the old behavior use untyped variables and no integer.</p>
<pre><code> perl5:
-13 % 4 => 3
13 % -4 => -3
{
use integer;
-13 % 4 => -1
13 % -4 => 1
}
cperl:
-13 % 4 => -1
13 % -4 => 1
# old perl5 behavior:
{
no integer;
($a,$b) = (-13,4);
$a % $b => 3
}</code></pre>
<p><b>Background:</b> In <a href="http://www.nntp.perl.org/group/perl.perl5.porters/2001/10/msg45590.html">http://www.nntp.perl.org/group/perl.perl5.porters/2001/10/msg45590.html</a> p5p was arguing that it is using the mathematical "correct" semantics, vs this definition from the C standards committee: <a href="http://www.open-std.org/JTC1/SC22/wg14/www/docs/n1124.pdf">http://www.open-std.org/JTC1/SC22/wg14/www/docs/n1124.pdf</a></p>
<p><i>"When integers are divided, the result of the / operator is the algebraic quotient with any fractional part discarded. If the quotient <code>a/b</code> is representable, the expression <code>(a/b)*b + a%b</code> shall equal a."</i></p>
<p>Perl does not obey this, only with use integer.</p>
<pre><code> use integer; my ($a,$b)=(13,-4); print int(($a/$b)*$b) + $a % $b => 13
my ($a,$b)=(13,-4); print int(($a/$b)*$b) + $a % $b => 10</code></pre>
<p>We found only once place in a module using constant negative integers with <code>%</code>. It is a core module, and is therefore fixed. Testing many other CPAN modules found no further problems, but watch out our distroprefs repo for future patches.</p>
<p><b>Rationale:</b> This is a bad perl5 design change and a side-effect from using type-promotion before constant folding. cperl uses proper type dispatch and with both integer arguments it promotes the <code>modulo</code> op to <code>i_modulo</code>, which behaves differently (or "normally"). The perl5 constant folder should do the same, but does not currently. Using extra logic to prevent from using the libc implementation also makes perl5 <code>"%"</code> operator slower.</p>
</dd>
<dt id="compile-time-constant-folding-overflows-with-integer-literals"><b>compile-time constant folding overflows with integer literals</b></dt>
<dd>
<p>Operations with only literal integers will not overflow to numbers, similar to perl5 with use integer.</p>
<p>Only <code>"+"</code> and <code>"*"</code> will try to use UV (an unsigned value) as result, but not NV (an inexact number). All other ops besides divide and shift return IV (signed integers). Note that even comparison operators are integerized.</p>
<pre><code> my $iv_min = -(~0 >> 1) - 1; # ok
my $iv_min_1 = -(~0 >> 1) - 2; # BUT this overflows
=>
my $iv_min_1 = -(~0 >> 1) - 2.0; # avoid overflow
my int $z = 4 / 5; # => number 0.8, violating the type checker
# but!
$z == 0; # => TRUE, because lexical int == const int uses i_eq,
# the integer variant, which integerizes $z to 0</code></pre>
<p>All coretests pass, no tests had to be changed. In practice only compile-time constants in the negative IV_MIN to UV_MIN range will need to be fixed.</p>
<p><b>Rationale:</b> This was the original issue 2002 which disabled automatic integerization of all arithmetic ops with constants. <code>"+"</code> and <code>"*"</code> are a good exceptions to allow unsigned results (with the new <code>u_add</code> and <code>u_multiply</code> ops) to create large constants, but all other integer operations need to be preciser when inexact numbers are demanded, and return signed integers.</p>
<p><b>Exceptions</b>, keeping the old behaviour: Division for two integer constants is an internal exception and is not promoted to use integer division. <code>2/5 => 0.4</code> Compile-time shift operations return unsigned integers.</p>
</dd>
</dl>
<h1 id="Fix-breakage-and-bad-design">Fix breakage and bad design</h1>
<h2 id="signatures">signatures</h2>
<p>The p5p signature implementions is still lacking many important features and is twice as slow as doing without, and twice as slow as the cperl implementation which uses the stack variables directly without copying them to <code>@_</code>. There is no point in using it. It is the biggest cause for backwards incompatible changes, but it is marked as experimental, so the perl5 implementation can eventually be improved, and our changes are technically not incompatible.</p>
<dl>
<dt id="Add-optional-types-in-signatures"><b>Add optional types in signatures</b></dt>
<dd>
<pre><code> sub (int $i) # as in Perl6, or
sub ($i: int) # as in the other gradual typed languages</code></pre>
<p>We use the samed syntax as provided for lexical variable declarations. In both variants, in leading position as with <code>my int $a;</code> and as attribute, as with <code>($i :int :const)</code></p>
<p>We need to seperate coretypes (int, uint, str, num) and user-defined types (existing class names), and the 2 core attributes <code>:const</code> and <code>:unsigned</code>. For more perl6 like traits see below.</p>
<p>Follow the same rules as in lexical declarations. The type must exist already as package, otherwise a syntax error is thrown.</p>
<pre><code> $ cperl -e'sub x(x $x){}'
No such class x at -e line 1, near "sub x(x"
$ cperl -e'sub x(str $s){}' # coretypes implicitly loaded
$ cperl -e'%MyStr::; sub x(MyStr $s){}' # user-defined type MyStr</code></pre>
</dd>
<dt id="Add-subroutine-return-types"><b>Add subroutine return types</b></dt>
<dd>
<p>For easier implementation we support subattributes, <code>:<type></code> only, not the other possible syntax variants <code>=> type</code> or <code>returns type</code>.</p>
<p>There are just a few semantic conflicts. Note that we can use the builtin attributes <code>:lvalue</code>, <code>:method</code>, <code>:const</code> also here.</p>
<p><code>:const</code> does not mean constant result, it rather means constant subroutine. Having this constant means that the compiler is able to inline it without run-time checks if it has changed.</p>
<p><code>:unsigned</code> as coretype or <code>:uint</code>? As sub attribute it could mean strictly typed to a return result of <code>UV</code>, like <code>:int :unsigned</code>, where the <code>:int</code> is optional. <code>:-unsigned</code> would mean <code>:int</code> then, return a signed int. <code>:unsigned</code> could also be no coretype, just a hint for <code>:int</code>, and without <code>:int</code> it will just be an attribute, not a strictly checked coretype. This decision is still open. For now we use <code>:uint</code>.</p>
</dd>
<dt id="Add-call-by-ref-via-arg"><b>Add call by-ref via \$arg</b></dt>
<dd>
<p>Support scalar lvalue references - sub (\$var)</p>
<p>With perl5 upstream all arguments are copied only, as with <code>my $arg1 = shift;</code> but alternate syntax for fast <code>$_[0]</code> access is not provided. So they have to keep the otherwise unneeded <code>@_</code> array around.</p>
<p>cperl uses <code>\$name</code> to denote references to scalar lvalues, which change the calling variable.</p>
<pre><code> sub myfunc(int \$i) : int { $i++; }
my $i;
print myfunc($i); => 1
print $i; => 1</code></pre>
<p>For now scalar lvalue references only, <code>\@a</code> or <code>\%h</code> would be nice with type checks for arrayref or hashref. maybe <code>\[$]</code> also.</p>
</dd>
<dt id="Improve-_-handling"><b>Improve <code>@_</code> handling</b></dt>
<dd>
<p>Remove <code>@_</code> when not needed. Use the mark stack as in the ops.</p>
<p>With cperl <code>@_</code> will only hold the <code>&rest</code> args, the undeclared rest if no other slurpy args are declared. i.e. <code>@_</code> will be empty when signatures are declared with a slurpy ending <code>@</code> or <code>%</code> arg, and <code>@_</code> is not referenced in the immediate function body, visible to the compiler, i.e. not hidden by a string eval.</p>
<p>We want to use the mark stack for signature calls, same as with OPs and XS calls. We don't need to copy to <code>@_</code>.</p>
<p>Notes on improving the current old zefram purple signatures:</p>
<p>Internally the elements of <code>@_</code> are currently accessed via aelem, not aelemfast. But this is moot with the introduction of the new OP_SIGNATURE op, which is even faster.</p>
<p>With the old 5.18-5.22 implementation the perl5 arity check is overly big and needs two checks and big strings with fixed arity subs.</p>
<pre><code> perl -MO=Deparse -e'sub x($a){$a++}'
sub x {
die sprintf("Too many arguments for subroutine at %s line %d.\n",
(caller)[1, 2]) unless @_ <= 1;
die sprintf("Too few arguments for subroutine at %s line %d.\n",
(caller)[1, 2]) unless @_ >= 1;
my $a = $_[0];
();
$a++;
}</code></pre>
<p>which violated the existing error message: "Not enough arguments for "</p>
</dd>
<dt id="Support-default-self-invocant-with-methods"><b>Support default $self invocant with methods</b></dt>
<dd>
<p>If a method is declared via <code>method</code> the <code>$self</code> argument is used as default invocant argument name, which can be overridden via the <code>($class: $args,...)</code> colon syntax.</p>
<pre><code> method adder ($a) { $self->add + $a; }</code></pre>
<p>With the method attribute you have to provide it:</p>
<pre><code> sub adder ($self, $a) :method { $self->add + $a; }</code></pre>
</dd>
<dt id="Add-:pure-attribute-for-subroutines"><b>Add :pure attribute for subroutines</b></dt>
<dd>
<p>You can flag a function as purely functional, without any side-effects, to allow further compiler optimizations. Note that pure functions may throw, but may not access globals or do IO.</p>
<p>A pure function only reacts to its input arguments and will always return the same value with the same arguments, thus can be safely memoized or constant-folded or inlined without having to embed them into ENTER/LEAVE blocks.</p>
</dd>
<dt id="Improved-error-reporting"><b>Improved error reporting</b></dt>
<dd>
<p>In violations do not only print the position, also the declaration which is violated. e.g.</p>
<pre><code> @a=(); sub x(\@b) {$b->[0]++} print x(\$a)</code></pre>
<p><code>Type of arg 1 to main::x must be arrayref (not a scalar ref) at -e line 1, near "\$a)"</code></p>
<p><code>=></code></p>
<p><code>Type of arg 1 \@b to x must be arrayref (not a scalar ref) at -e line 1, near "\$a)"</code></p>
<p>Proper signature types are not only a great help for catching errors early or improve documentation. They are performance critical, see coffescript, dart, microsoft typescript, google soundscript, facebook hack, mypy, ruby 3.0 and partially even perl6. The type inferencer will not be able to infer many types without explicit types. But with typed signatures, besides the obvious solution of private methods or closed classes we can inline most small methods, ignore run-time magic and improve most loops and array accesses. It is also critical to implement multi methods (compile-time optimized generics) and implement an advanced object system.</p>
<p>Note that the reported main subroutines are listed without the <code>main::</code> prefix.</p>
</dd>
<dt id="Support-...-for-efficient-varargs-passing-NY"><b>Support <code>...</code> for efficient varargs passing (NY)</b></dt>
<dd>
<p><code>...</code> as empty function body already has a special meaning as <b>yadayada</b> operator, just croaking, but interestingly not the usual meaning of varargs.</p>
<p>cperl uses <code>...</code> in the natural way to denote unnamed and uncopied rest args, and passes the varargs through to the next call.</p>
<p><code>...</code> denotes a slurpy unnamed signature, and <code>...</code> in a otherwise non-empy function body denotes passing those arguments efficiently to the next function. Internally <code>...</code> does not need to copy the values into a temporay array, we just need to pass the argument stack position down to the next calls using <code>...</code>. By using <code>...</code> instead of <code>@_</code> we can avoid copying the values to <code>@_</code>, we only need the stack index, not all the values.</p>
<pre><code> sub foometh($self, ...) { func(...) }</code></pre>
<p>In an extern sub declaration, the <code>...</code> denotes varargs as in C.</p>
</dd>
<dt id="strict-prototype-and-signature-syntax-no-pragmas-required"><b>strict prototype and signature syntax, no pragmas required.</b></dt>
<dd>
<p>Illegal prototype and signature syntax does not just warn, it dies with an syntax error as it should be.</p>
<pre><code> $ cperl -e'no warnings "illegalproto"; sub x(x){}'
No such class x at -e line 1, near "sub x(x"</code></pre>
<p><code>no warnings "illegalproto";</code> is a noop.</p>
</dd>
<dt id="no-feature-signatures-or-lexsubs-or-lexical_topical-pragmas-required"><b>no feature signatures or lexsubs or lexical_topical pragmas required</b></dt>
<dd>
<p><code>use feature "signatures"</code> or <code>use feature "lexsubs"</code> is not required and is ignored. All prototypes and signatures are parsed either as prototypes or signature, regardless of the scope of a <code>use feature "signatures"</code> pragma. <code>my sub</code> is parsed without previous activation.</p>
<p><code>my $_</code> does not require <code>use experimental::lexical_topic</code>. <code>use cperl</code> can be used, if incompatible features are used, but it is optional.</p>
</dd>
</dl>
<h2 id="Changed-calls-to-signatures">Changed calls to signatures</h2>
<p><code>goto</code> to a signature is now a true tail-call. The stack and pad frames (lexical variables) are not duplicated as in a recursive call, they are re-used. The call-stack behaves as before and in python, so a tail-call (i.e. a goto to a signature) is visible in the call-stack.</p>
<p>But since the args are passed directly on the stack, any old <code>&$sub;</code> call without <code>()</code> to a signature will not work, you need to use <code>goto \&$sub;</code> instead. Only tailcalls via goto can translate from the old pure-perl stack to the new re-used stack via signatures.</p>
<p>The error message is <code>"Not enough arguments for subroutine \w+. Want: \d, but got: 0."</code>.</p>
<p><b>How to detect if a subroutine has a signature?</b></p>
<p>prototypes with a signature are always embedded into parentheses, <code>"()"</code>.</p>
<pre><code> sub _hassig {
my $sub = shift;
substr(prototype $sub,0,1) eq '(';
}
return _hassig($sub) ? goto \&$sub : &$sub;</code></pre>
<p>See e.g. <i>perl5db.pl/_hassig()</i>.</p>
<p>But you can also change all calls via <code>&$sub;</code> to <code>goto \&$sub;</code> to avoid the <code>_hassig</code> check.</p>
<h2 id="Fix-my-_-handling">Fix my $_ handling</h2>
<p>Perl5 had long-standing problems with lexical <code>$_</code> since the introduction of the SASSIGN optimization via OA_TARGLEX and OPpTARGET_MY in 2002.</p>
<p>cperl fixed this (it was using a wrong bit-testing, using AND as OR) and work is ongoing to harmonize further internal code exceptions and code-smell, e.g. SASSIGN, given/when and the match functions, everything with nested blocks.</p>
<h2 id="Undo-B-bootstrap-breakage">Undo B bootstrap breakage</h2>
<p><a href="/cperl/lib/B.html">B</a> was changed to <code>use strict</code>, which broke the <a href="/cperl/lib/B/Bytecode.html">B::Bytecode</a> compiler performance advantage, adding all compiler internal constants to the emitted bytecode. Reverting this breakage was denied because this developer did not understand the code. It is too bothersome in the long run to maintain our reversion of this breakage over years. It easier to check for <i>usecperl</i> in the compiler to be able to compile to bytecode properly again. Note that they broke it again with 5.22, and are again refusing to fix it.</p>
<h2 id="Undo-constant-folding-i_opt-de-optimization">Undo constant folding i_opt de-optimization</h2>
<p>Automatic integer optimizations for constants were removed from constant folding against community consent in the early p5p times with commit <code>e7311069</code>.</p>
<p>We re-add this optimization to treat constant integers and typed lexicals as such (as via an implicit use integer) in constant foldable expressions. I.e. integer overflow in constant expressions is only checked at compile-time, not run-time.</p>
<p>We re-instate the two special cases for I_DIVIDE and I_MODULO, which deviate from the untyped generic variants. div int / int returns a float even with constants, and mod with negative integers uses the standard C variant only if typed or within use integer. The untyped variant uses the perl5 definition of modulo as in group theory, which violates the definition of a remainder.</p>
<pre><code> my int $i = -3; $i % 2 => -1, not 1.
-3 % 2 => -1, not 1.
1/2 => 0.5 (unchanged)</code></pre>
<p>We are also now able to do constant-folding on subroutine bodies, to either inline the body into the caller or replace the body with a constant in the general case. (i.e. without the <code>()</code> prototype)</p>
<h2 id="Undo-support-for-binary-names">Undo support for binary names</h2>
<p>By announcing unicode support for names with 5.16 p5p silently allowed <code>\0</code> inside names, which they called an advantage by supporting now binary safe names and "harmonization". In reality unicode names were already supported since 5.8.4 (with negative HEK key lengths) and the whole 5.16 unicode name theatre was only about binary names. They didn't support binary names in all other code parts which had to deal with names, and thus enabled a huge attack vector to hide arbitrary user strings behind <code>\0</code> names, which was silently stripped before and when used in syscalls. They moved resonsibility to the user, as previously for input strings only, now also for input names, e.g. for package names which search the filesystem directly without sanitation or proper checks. Support for those binary names is still not complete with 5.24 in core, even if p5p pushed so strong for it against my protest.</p>
<p>We have to keep the new GV API - accepting the string length - but even without <code>strict names</code> we strip everything behind the <code>\0</code> as before 5.16.</p>
<p>For a more efficient dynamic namespace implementation we might be switching from chained hash tables to a single ternary trie, radix tree or DAFSA, without support for <code>\0</code> and maybe even without optional support for unicode names. Only with <code>use utf8</code> we might need to fallback to the old slow method then.</p>
<h2 id="Warn-on-0shellcode-attempts-for-names">Warn on \0shellcode attempts for names</h2>
<p>Make our <code>use warnings "syscalls"</code> the default.</p>
<p>Any attempt to attack package names with shellcode behing <code>\0</code> is being warned per default, and not only optionally with <code>use warnings "syscalls"</code>. There is no uninnocent or wrong usage of such names, only malicious intent, and this must appear in logfiles. This is worse than syntax errors and syntax errors are warned by default. p5p was vehemently against this change.</p>
<p>cperl has a new 'security' warnings category, which bypasses STDERR capture and also tries to log the remote user IP.</p>
<h2 id="strict-names">strict names</h2>
<p>With <code>use strict "names"</code> we do not accept unparsable symbols created from strings. This is a new run-time error for use <a href="/cperl/lib/strict.html">strict</a>.</p>
<h2 id="TR39-confusable-names">TR39 confusable names</h2>
<p>cperl rejects since 5.26 confusables as described in TR39, rejects most mixed scripts and normalizes unicode identifiers, similar to python 3. cperl is actually one of the very rare dynamic languages with full unicode identifer support which actually does follow the unicode consortium security recommendations and profile. cperl implements the <a href="http://www.unicode.org/reports/tr39/#General_Security_Profile">Moderately Restrictive level</a> for unicode identifiers. See <a href="/cperl/perldata.html#Identifier-parsing">"Identifier parsing" in perldata</a></p>
<p>The compiler had to add this warning since 5.16 until 5.26:</p>
<p><i>Perl handling of new unicode identifiers - package and symbol names -</i> <i>without proper TR39 handling is considered a security risc and is not fully supported.</i> <i>See</i> <a href="http://websec.github.io/unicode-security-guide/">http://websec.github.io/unicode-security-guide/</a>.</p>
<p><i>Check your code for syntax spoofs, confusables, strip \0 from package names</i> <i>and enable use warnings 'syscalls'</i>.</p>
<h2 id="Undo-the-double-readonly-system">Undo the double readonly system</h2>
<p>In order to support Hash unlock code, i.e. undoing readonly setting of hash values, p5p added a second readonly bit <code>SVf_PROTECT</code> for special values which are not allowed to be writable, even if the better solution to check for this special values at Hash unlock would have been trivial. It does not need to take away the last free SV bit which we used to implement coretype checks on pad values, and unlock really only needs to unlock the previously locked values, not make all values unconditionally writable, thus making previously readonly values writable. <code>SVf_PROTECT</code> does not help with that.</p>
<p><code>SVf_PROTECT</code> is now the same as <code>SVf_READONLY</code>, and special checks were added for the two usecases when the readonly bit is unset. We need the <code>SVf_PROTECT</code> bit to mark native SV's in pad's, and even reserved it in 2012.</p>
<h2 id="Fix-the-hashes">Fix the hashes</h2>
<p>Provide proper hash table abstractions. We don't need 5 times the same bad code copied along for all different kind of HEK (hash key) types.</p>
<p>Do not check the hash key for collisions with 4 different comparisons in the hot hash loop, use one instead.</p>
<p>For the old hash tables use the new default strategy PERL_PERTURB_KEYS_TOP to move each found bucket to the top of the chain. This is how you usually implement a slow hash table with linked lists.</p>
<p>Use fast hash functions, not secure slow ones. We get security by fixing the algorithmic problem in the collision handling, not by using extremely slow hash functions and obscuring the users and fellow developers. We properly analyzed many hash functions and hash tables, for security and speed. The fastest, FNV1a, was the one the fellow developer schmorp choose <i>by his guts</i> in his stableperl fork.</p>
<p>Provide a Configure argument to define the hash function: <code>-Dhash_func=FNV1A</code></p>
<p>Make the load factor definable, and change the default from 100% to 90%, which was tested as superior. Use <code>-Accflags='-DHV_FILL_RATE=100'</code> for the old behavior.</p>
<p><b>Further plans:</b></p>
<p>Hash functions need to be implemented as macros, not functions, undo that. (<i>maybe</i>)</p>
<p>Use cache-friendly open addressing, not simple, slow and DOS'able (i.e. insecure) linked lists.</p>
<p>Seperate the keys from the values to fit the search into a cache line.</p>
<p>Added support for restricted hashes, esp. stashes which we needed for our class implementation, to detect wrong fields at compile-time already. restricted hashes are not properly supported in perl5.</p>
<p>Later we will provide a special <code>:const</code> hash table type to enable optimizations to perfect hashes. With <code>study %hash</code> you can do the similar costly optimizations on non-const hashes at run-time to allow faster key access.</p>
<h2 id="Compile-time-attribute-hooks">Compile-time attribute hooks</h2>
<p>Added a <code>CHECK_SCALAR_ATTRIBUTES</code> callback. Add native <code>:const</code>, <code>:int</code>, <code>:num</code>, <code>:str</code>, ... attributes for all new core types. This is basically a read-only <code>MODIFY_*_ATTRIBUTES</code> hook at compile-time with a better name to disassociate from the run-time check of <code>FETCH_*_ATTRIBUTES</code> with <code>my</code> lexicals. See also <a href="#Rewrote-critical-core-modules-in-C-as-builtins">"Rewrote critical core modules in C as builtins"</a> below.</p>
<h2 id="Run-time-attribute-variables">Run-time attribute variables</h2>
<p>cperl defers the <code>attribute->import</code> call from compile-time to run-time with non-constant attribute arguments. See <a href="http://perl11.github.io/blog/attributes-args.html">http://perl11.github.io/blog/attributes-args.html</a>.</p>
<h2 id="coretypes:-Int-UInt-Num-Str">coretypes: Int, UInt, Num, Str</h2>
<p>coretypes can only be built-in, there's no way to implement it as extension, similar to an object system. cperl provides the 4 basic ones, and the combinations with <code>Undef</code>, and the lowercase native types. Same as in perl6. Type combination are done via <code>@ISA</code>, i.e.</p>
<pre><code> class ?Int :const { our @ISA :const = qw(Int Undef); }</code></pre>
<p>Our coretypes classes and its members are readonly. Provide fast ops variants for these types to omit type checks and magic calls at run-time. Scalars declared as <code>Int</code>, <code>Uint</code>, <code>Num</code>, <code>Str</code> cannot hold magic associations, such as <code>tie</code>.</p>
<p>We also enabled the <code>:const</code> attribute for all data: scalar, arrays, hash, functions, packages+classes.</p>
<p>At compile-time most UNOP's and BINOP's are promoted from the generic ops to more specific typed ops, similar to <code>use integer</code>. But use integer does not know the types of the variables at compile-time, many ops are only dispatched at run-time. See e.g. <a href="#negative-integer-modulo">"negative integer modulo"</a>.</p>
<p>See <a href="/cperl/perltypes.html#coretypes">"coretypes" in perltypes</a> and <a href="/cperl/perldata.html#Constant-and-typed-lexical-variables">"Constant and typed lexical variables" in perldata</a>.</p>
<h2 id="native-unboxed-types:-int-uint-num-str-NY">native unboxed types: int, uint, num, str (NY)</h2>
<p>Internally the types for all scalar SV's always start with uppercase classnames, same as with most user-defined classes. The four lower-case variants <code>int</code>, <code>uint</code>, <code>num</code>, <code>str</code> denote possible optimizations to direct unboxed values on the stack, which are not reference counted, and cannot yet be used across function calls. They are only safe to use within certain op sequences, and those optimizations are done automatically.</p>
<p>Unlike all other types native types are only hints, not promises. The compiler promotes data and code to native types only if its sees fit.</p>
<p>With PERL_NATIVE_TYPES enabled, most literal constants are stored as native types, native type declarations are a promise not a hint, and the optimizations involves up- and downgrading of data and ops in possible native chains. This leads to much tighter native expressions, with performance and memory gains. (3x less memory, ~3x faster).</p>
<p>For the builtin FFI we provide also FFI-specific native types, like int32, int64, uint32, uint64, ptr and more.</p>
<p>See <a href="/cperl/perltypes.html#native-types">"native types" in perltypes</a>.</p>
<h2 id="types">types</h2>
<p>Provide a compile-time type inferencer, type checker and type optimizer. The inferencer runs automatically and can currently only infer <code>int</code> on array indices, ranges and <code>str</code> on hash keys, but has to give up on magic, dualvars, and <code>no strict 'refs'</code>. With the help of declarations and type checks, as e.g. in <code>smartmatch</code> or <code>given</code>/<code>when</code> with type support it can infer much more.</p>
<pre><code> if (type $a eq "int") { => $a is an int in this scope }
$str =~ /(\d+)/; => $1 is a typed Int</code></pre>
<p><i>Compile-time type checks need to be enabled with <code>use types;</code> though.</i></p>
<p>Typed signatures are backwards incompatible to perl5, as the trivial 4 line changes are still not yet supported upstream. The performance win is ~2-10x faster, you get compile-time type warnings, a business friendly coding environment and the possibility to display and put infered types automatically in your code, with a cooperating editor. e.g.</p>
<pre><code> my $n=1000;
for (my $i=0; $i<$n; $i++) { }
=>
my int $n :const = 1000;
for (my int $i=0; $i<$n; $i++) { }</code></pre>
<p><b>Note</b>: When in doubt leave out types. If the inferer cannot find it, it might not be worth the trouble. But for hot code and to be precise always use types, as compile-time types prevent from costly run-time checks for types and magic hooks.</p>
<p><b>Builtin types</b> are the coretypes <code>Int, Num, Str, UInt, ?Int, ?Num, ?Str</code> and for builtin op-dispatch: <code>Void int uint num str Int UInt Num Str Bool Numeric Scalar Ref Sub Array Hash List Any</code>, with a <code>?</code> prefix denoting <code>| Undef</code> <i>"or undef"</i>, a <code>?</code> suffix is optional, and for aggregate types using <code>()</code> brackets, like <code>:Array(:Int)</code>.</p>
<p><b>Status</b>: User code in pure perl or XS is currently not typed-checked nor inferred, only internal ops. But wrong type declarations do lead to compile-time type violation errors.</p>
<h2 id="const-for-all">:const for all</h2>
<p>Our <code>:const</code> attribute applies to all data types: scalar, arrays, hash, functions, packages + classes. <code>:const</code> hashes should of course be perfect, i.e. optimized to constant-time lookup, eliminating hash collisions.</p>
<p>The following declarations are all compile-time assigned, and allow subsequent constant folding on all usages. I.e. const assignments with all constant rhs values.</p>
<pre><code> my $i :const = 1;
our $i :const = 1;
my @a :const = (1); # also sets the array shaped as int @a[1]
my @a :const = (1,2); # shaped as int @a[2]
my @a :const = (0..2); # shaped as int @a[3]</code></pre>
<p>With non-constant right-hand side values the assignment is done at run-time and as thus only writes are caught at compile-time, but constant folding is not available. Hashes are also not yet compile-time assigned.</p>
<p>:const arrays with values of unique types will be optimized to native shaped arrays. See <a href="#Typed-and-shaped-arrays">"Typed and shaped arrays"</a>.</p>
<p>See <a href="/cperl/perltypes.html#const">":const" in perltypes</a>.</p>
<h2 id="Compile-time-optimizations">Compile-time optimizations</h2>
<p>cperl adds many more traditional compile-time optimizations: more and earlier constant folding, type promotions, shaped arrays, usage of literal and typed constants, loop unrolling, omit unnecessary array bounds checks, function inlining and conversion of static method calls to functions.</p>
<p>Perl 5 only inlines constant function bodies with an explicit empty <code>()</code> prototype.</p>
<pre><code> sub x() {1+2} # inlined in perl5
sub x {1+2} # inlined in cperl only</code></pre>
<p>cperl inlines constant function bodies even without empty prototype declaration, has type declarations for most internal ops, and optimizes these ops depending on the argument types; currently for all arithmetic unops and binops, and the data-accessing ops padsv, svop, and sassign. <i>opnames.h</i> stores <code>PL_op_type_variants</code>, all possible type promotions for each op. <i>opcode.h</i> stores <code>PL_op_type</code> with the type declarations of all ops.</p>
<p> </p>
<p>Small non-constant bodies may be inlined automatically, with the args replaced in the body. constants args and literals are replaced as is, also args as simple rvalues. lvalue args are checked for call-by-ref or call-by-value semantics, and use the arg either directly or as copy.</p>
<p>With shaped arrays and it's new unchecked <code>aelem_u</code> variants, loop bodies are optimized when the upper loop bound is declared via <code>$#</code>, the <i>arylen</i>.</p>
<pre><code> for (0..$#array) { .. $array[$_] .. }</code></pre>
<p>Here the <code>$array[$_]</code> call does no bounds check of the index, since <code>$#array</code> is the last valid index, and we do not shrink the array in the loop, even if the array is not explicitly declared as shaped array.</p>
<p>Work is ongoing in loop unrolling, function inlining and speculative method inlining, which should speed up run-time performance dramatically and enable new optimizations which were previously stopped on each function call border. Planned is also polymorphic inline caching with a usage counter, not just naive monomorphic inline cache for method calls. This is needed for the jit optimizer.</p>
<p>Check the new DEBUGGING option <code>-Dk</code> which lists all optimizations and checks at compile-time, optionally verbose together with <code>-Dkv</code>.</p>
<h3 id="Static-method-calls">Static method calls</h3>
<pre><code> strict->import(...)
=> strict::import("strict", ...)</code></pre>
<p>When the method is defined directly in the package, it is not possible to inject another package at run-time into the method search, thus the method call is optimized from a dynamic method dispatch to a normal static function call.</p>
<h2 id="Proper-object-system">Proper object system</h2>
<p>No, not Moose, rather perl6 without the antics. Closer to Mouse though, which is currently the recommended OO. Mouse with a perl6-like syntax. Rather an optimizable perl6-like object system in core, similar to an extended base/fields OO with pseudohashes, roles, mop and multi-dispatch. See <a href="/cperl/perlclass.html">perlclass</a>.</p>
<p>Provide a simple MOP for reflection purposes, but not yet for overrides, i.e. metaobjects for classes, proper class and method syntax, anonymous classes by pointer not name, proper multi dispatch with types, roles, class and method lookup by pointer, not by name. Create native optimized shapes via mixins as in perl6 or p2 (an enhanced bless). Lexical methods are private, optimize dispatch for single inheritance. i.e the convenient class syntax extends on a single class only, classes are finalizable by the calling application.</p>
<p>Support multi-dispatch in perl6 syntax and optimize for early-bound method calls with typed arguments and closed classes.</p>
<p>Class dispatch is via C3, not the old depth-first left-to-right mro implementation. Many old core modules have broken inheritance, so C3 could not yet be enabled as default.</p>
<p>For type inference the special Mu methods <code>new</code> and <code>CREATE</code> are detected and cause the left-hand side of a lexical assignment to progate its type into the lexical.</p>
<h2 id="Typed-and-shaped-arrays">Typed and shaped arrays</h2>
<p>Enable faster array access, uniformly typed array values, use less memory (cache friendly), help the type system. See <a href="/cperl/perltypes.html">perltypes</a>.</p>
<p>Type arrays specify the uniform type of the values. Typed arrays with native types use much less memory and provide faster direct access. Natively typed arrays allow access and passing as arguments to external C functions.</p>
<p>Shaped arrays define a compile-time constant "shape", the size, which cannot be changed. All values are pre-initialized.</p>
<pre><code> my int @a[20]; # or
my @a[20] :int;</code></pre>
<p>The number of elements can be compile-time computed from the elements of a rhs list of all constant expressions, similar to the initialization of :const arrays. See <a href="#const-for-all">":const for all"</a>.</p>
<pre><code> my @a[] = (0..2);</code></pre>
<p>With known indices the compiler can omit bounds checks on array accesses.</p>
<pre><code> my int @a[5];
for (0..$#a) { $a[$_] ... }</code></pre>
<p>Here <code>$a[$_]</code> uses the unchecked aelem_u operator, because the index <code>$_</code> cannot be out of bounds.</p>
<p>Shaped arrays are pre-initialized according to its type and cannot change the size. Attempts are caught at compile-time and run-time. With constant or type-checked indices in range the access op is optimized to omit the bounds check, via unchecked aelem_u variants. Negative constant indices are converted to positive at compile-time.</p>
<h2 id="Typed-and-shaped-hashes-NY">Typed and shaped hashes (NY)</h2>
<pre><code> my int %h; # or
my str %a; # hash with str values only
my int %a{20}; # hash with int values only. fixed hash size,
# no grow on insertion.
my str %a{20} :const # fixed hash size, no grow on insertion,
# perfect hash (keys may not change, values do).
= (...);</code></pre>
<p>No sparse arrays (i.e. hash with int keys) yet. but this would need a different declaration syntax if to be supported natively.</p>
<p>e.g. <code>my int %a{int};</code> - sparse array with int keys and int values. or <code>my %a : hash(int);</code></p>
<p>variant 2: <code>my IntSparseHash %a</code>, which can go with a user class and methods, but this will be slow, without native ops.</p>
<h2 id="Inlined-functions-NY">Inlined functions (NY)</h2>
<p>If a function body is inlinable, i.e. simple, with no control ops like return, goto caller, warn, die, reset, runcv, padrange, adds no seperate lexicals and has less than 10 ops, they are inlined. Static methods are converted to functions before and then possibly inlined.</p>
<p>This is the most important optimization, even more important then a jit.</p>
<h2 id="Rewrote-critical-core-modules-in-C-as-builtins">Rewrote critical core modules in C as builtins</h2>
<p>Builtins: <code>strict</code>, <code>attributes</code>, <code>DynaLoader</code>, <code>XSLoader</code>. NY: <code>Carp</code>, <code>Exporter</code>.</p>
<p>As shared lib: <code>Config</code> and later <code>warnings</code> and <i>unicode folding tables</i> to save memory, startup time and reduce bloat.</p>
<p>Big constant hashes and tables need to be in a shared memory segment, not recompiled for every fork or thread, similar to the <a href="/cperl/lib/Encode.html">Encode</a> tables which are implemented properly. The risc to introduce even more performance regressions by keeping some critical core modules as <i>.pm</i> is too high and broke the compiler too often. Most developers have no idea of the impact of innocently looking additions.</p>
<p>We need to reduce memory, and want to reduce the size of compiled code by 30%, but in some cases it will be 200%. As builtin or shared library we go to zero startup-time overhead for those modules. With compiled Config alone the memory savings are down to 5.6 levels.</p>
<h3 id="strict-1.11c-as-builtin">strict 1.11c as builtin</h3>
<p>Starting with cperl (based on Perl 5.22) <a href="/cperl/lib/strict.html">strict</a> is now a builtin module, implemented as XS functions which are always available.</p>
<p><b>Changes:</b></p>
<p><i>strict.pm</i> is only provided for documentation, <code>$INC{'strict.pm'} = 'xsutils.c'</code> With a list of wrong tags only the wrong tags are reportyed one-by-one, and not together. All other functionality stays the same.</p>
<p>strict hash two new modes: strict hashpairs and names.</p>
<h3 id="attributes-0.26_01c-as-builtin">attributes 0.26_01c as builtin</h3>
<p>Starting with cperl (based on Perl 5.22) <a href="/cperl/lib/attributes.html">attributes</a> is now again a builtin XS module. There's no need to dynaload it at parse time.</p>
<p><i>attributes.pm</i> moved back to <i>lib/attributes.pm</i> and is only provided for documentation and import.</p>
<p><code>CHECK_<i>type</i>_ATTRIBUTES</code> is a new compile-time hook, like a readonly variant of <code>MODIFY_<i>type</i>_ATTRIBUTES</code>, or the compile-time variant of <code>FETCH_<i>type</i>_ATTRIBUTES</code>.</p>
<p>Attribute arguments can now be variables, deferred from compile-time to run-time.</p>
<p>There are several new builtin attributes:</p>
<dl>
<dt id="const-for-all-types"><b>:const for all types</b></dt>
<dd>
</dd>
<dt id="unsigned-for-all-integer-types-sets-SvIsUV_on-off"><b>unsigned for all integer types, sets SvIsUV_on|off</b></dt>
<dd>
</dd>
<dt id="existing-classes-as-types-are-recognized"><b>existing classes as types are recognized</b></dt>
<dd>
<p>and stored for lexical types and subroutine return types.</p>
</dd>
<dt id="some-new-FFI-attributes"><b>some new FFI attributes</b></dt>
<dd>
</dd>
</dl>
<h3 id="DynaLoader-2.00c-and-XSLoader-as-builtins">DynaLoader 2.00c and XSLoader as builtins</h3>
<p>Starting with cperl (based on Perl 5.22) <a href="/cperl/lib/DynaLoader.html">DynaLoader</a> and XSLoader have no perl code anymore, <i>DynaLoader.pm</i> was rewritten as <i>dlboot.c</i>.</p>
<p><b>Changes</b>:</p>
<p><code>@dl_library_path</code> eliminates now all duplicate paths and resolves symlinks of <code>$Config{libpth}</code> at build time.</p>
<p>Only the <code>$ENV{PERL_BUILD_EXPAND_CONFIG_VARS}</code> settings are implemented. All Config settings are compiled in at build time, run-time changes are not honored. Config is now also a compiled module, <i>Config_heavy.pl</i> is gone, and its hash was always readonly, so there's no way to change Config values at run-time without recompiling it.</p>
<p>Not sure yet about keeping support for <i>.bs</i> hooks and <code>@dl_resolve_using</code>.</p>
<p>The <code>XSLoader::load_file($module, $modlibname, ...)</code> function is new, <a href="/cperl/lib/XSLoader.html">XSLoader</a> is a builtin also. <code>...</code> is passed to the loaded XS function as with <code>XSLoader::load($module, ...)</code>.</p>
<h3 id="Config-as-XS-extension">Config as XS extension</h3>
<p>It is compiled as shared library, with all keys as readonly perfect hash. Some internal variables are not accessible anymore, the API is via the documented functions. See our <a>Mock::Config</a> module if you need to change a Config value for tests.</p>
<p>Also usable for perl5, via <code>cpan XSConfig</code>.</p>
<h3 id="warnings-as-XS-extension-NY">warnings as XS extension (NY)</h3>
<p>It is compiled as shared library, the builtin categories are implemented as perfect hash, and extended with a normal perl hash. Some internal variables are not accessible anymore, the API is via the documented functions.</p>
<p><b>Status:</b> 1 scope bug with Carp. Not yet enabled.</p>
<h2 id="Added-the-compiler-back-to-core">Added the compiler back to core</h2>
<p>The <a href="http://search.cpan.org/dist/B-C/">B::C</a> testsuite runs too long for cpan users, and it needs to be developed in sync to avoid typical 6 months wait-time after a core change. The compiler is smoked on all smoker platforms (linux 64bit, linux -m32, darwin, windows 32+64bit).</p>
<p>Maybe provide python-like precompiled <code>ByteCache</code> <i>.pmc</i> as default. You could pre-compile then modules with higher optimization levels, esp. the type inferencer. <i>.pmc</i> handling has been extended for reflection inside a .pmc, needed for the jit cache.</p>
<p>Maybe include a <code>Data::Compile</code> module to dump only data without all the code to a shared library, and possibly <a href="https://github.com/rurban/Perfect-Hash">Perfect::Hash</a> and a new <i>ph.c</i> to create and optimize readonly hashes, which is needed for the shared XS hashes of Config, warnings and unicode tables.</p>
<h2 id="Backport-core-testsuite-fixes-for-the-compiler">Backport core testsuite fixes for the compiler</h2>
<p>Honor differences between compile-time und run-time, when run compiled. Other than a few wrong testcases, the compilers does pass the core testsuite.</p>
<p>See <a href="https://github.com/perl11/p5-coretests">https://github.com/perl11/p5-coretests</a> for a generally improved testsuite for perl5, cperl and B::C.</p>
<h2 id="libffi-in-core">libffi in core</h2>
<p>Declare extern functions and libraries and call them. There's no need for XS and seperate compilation for most bindings. Not everybody has a compiler, the very same compiler perl was compiled with. libffi is the slowest ffi library, but has the best platform support, as it is integrated with gcc for java jni support.</p>
<pre><code> extern sub atoi(str $s) :int;
extern sub itoa(int $s) :str;
extern sub printf(str $s, ...);</code></pre>
<p>Note that this deviates from perl6 synatx with <code>is native</code>, but perl6 uses the <code>is</code> traits for all function attributes, which we do not. And we prefer the more natural C-like syntax over the more obscure perl6 syntax.</p>
<p>The alternative syntax to <code>extern sub</code> is via the <code>:native</code> sub attribute, which does allow options, as in <code>sub atoi(str $s) :int :native($libc);</code>.</p>
<p>Later with a jit in core we would not need the external <i>libffi</i> dependency.</p>
<h2 id="unicode-folding-tables-as-XS-extension">unicode folding tables as XS extension</h2>
<p>They should be compiled as shared library, with all keys as readonly perfect hash or as trie. Some internal variables are not accessible anymore, the API is via the documented functions. The trie lookup should not be implemented vi pure-perl (80x slower) and use esp. the memory in a readonly shared library for huge space savings (2M).</p>
<p>Maybe seperate them into a light folding table and heavy names and properties.</p>
<p>This was done by p5p with 5.30 eventually.</p>
<h1 id="Plans-for-further-core-features">Plans for further core features</h1>
<h2 id="Lexical-methods">Lexical methods</h2>
<p>Lexical methods are of course <b>private</b> to its enclosing class, i.e. not visable from outside the class. And they are also <b>closed</b> (or also called <i>sealed</i> or <i>final</i>) as planned by Damian Conway, i.e. they cannot be changed later. This enables the compiler to inline them automatically when its worthwhile. e.g. when they are small enough. They are either defined in the new style: <code>class .. { my method .. }</code> or old style: <code>package .. { my sub .. }</code></p>
<h3 id="Carp-2.00c-as-builtin">Carp 2.00c as builtin</h3>
<p><a href="/cperl/lib/Carp.html">Carp</a> might be a implemented as builtin XS module. There's no need to require it. <i>Carp.pm</i> is only provided for documentation, <code>$INC{'Carp.pm'} = 'xsutils.c'</code> Many carp function are added to the perl5 API and available to core and extensions.</p>
<p><b>Changes</b>:</p>
<p>Currently shortmess is only simplified, the step to skip packages which trust each other (via <code>@CARP_NOT</code> or <code>@ISA</code>) is not yet implemented, neither is the <code>CARP_TRACE</code> formatting hook. The deprecated <code>$Carp::CarpLevel</code> variable is now ignored.</p>
<p>Other not yet implemented variables: <code>$Carp::MaxEvalLen</code>, <code>$Carp::MaxArgLen</code>, <code>$Carp::MaxArgNums</code>, <code>$Carp::RefArgFormatter</code></p>
<p><b>Note</b>:</p>
<p>This branch is currently not included. It's too instable to get the caller depth right from pure perl vs XS, and most extended hooks (format, CARP_TRACE, CARP_NOT) are not yet implemented. Note that XS calls usually do not get counted in caller, unless you use the SCOPE keyword.</p>
<h2 id="Remove-Attribute::Handler-from-core">Remove Attribute::Handler from core</h2>
<p>This evals all attributes at compile-time. Discourage its usage.</p>
<h1 id="Longer-term-goals">Longer term goals</h1>
<h2 id="Faster-functions-and-method-calls">Faster functions and method calls</h2>
<p>Optionally omit caller, @_, freetmps, exception handler when possible, i.e. no string eval's are present in the body, the compiler can detect it or the right compiler hint is given.</p>
<h2 id="tailcall-elimination">tailcall elimination</h2>
<p>Detect tail positions and replace the call with a fast goto. This needs to be in core, not as external module. Without tailcall elimination handling longer lists leads to stack exhaustion. python did the same mistake as p5p by enforcing a new context for every goto. goto to signatures are now real fast tailcalls.</p>
<h2 id="Macros">Macros</h2>
<p>Use the existing perl6 S06 synopsis and testcases. In the absence of a signature to the contrary, a macro is called as if it were a method on the current match object returned from the grammar rule being reduced; that is, all the current parse information is available by treating <code>self</code> as if it were a <code>$/</code> object. Macros may return either a string to be reparsed, or a syntax tree that needs no further parsing.</p>
<p>Change the syntax a little bit to harmonize with perl5:</p>
<pre><code> {{{ }}} => `` for unquote splicing.</code></pre>
<p>Expand the ast inside the backticks at compile-time as a subprocess would be expanded with its result. Use <code>qx()</code> for subprocesses instead.</p>
<pre><code> macro infix:<name> () => macro 'name' :infix ()</code></pre>
<p>Use perl5 attributes as perl6 operator overloaders to tell macros if they have non-standard position: infix, prefix, postfix, circumfix, postcircumfix.</p>
<p>Don't support the quasi attributes <code>:ast/:lang/:unquote</code>, other than <code>:COMPILING</code>. Support the <code>COMPILING::</code> pseudo-package.</p>
<p>See the <i>feature/gh261-macros</i> branch.</p>
<h2 id="Editor-integration">Editor integration</h2>
<p>With type inference we can provide a much nicer development environment which also supports the debugger. <b>plsense</b> for emacs should give type feedback. I'm thinking of a port of <b>ZeroBrane Studio</b> for perl. This means provide core support for the needed serializers and introspection facilities.</p>
<h2 id="Jit">Jit</h2>
<p>Add a jit for the easiest platforms, and provide dynamic javascript-based optimizations when stacks need to be replaced. Jit's are a bit overrated. a fast bytecode loop can easily beat a jit and optimizing compiler, see luajit2, <a href="http://www.emulators.com/docs/nx25_nostradamus.htm">http://www.emulators.com/docs/nx25_nostradamus.htm</a> or <a href="http://nominolo.blogspot.de/2012/07/implementing-fast-interpreters.html">http://nominolo.blogspot.de/2012/07/implementing-fast-interpreters.html</a></p>
<p>A tracing jit, and not a method jit sounds best, together with the inliner, native op sequences and loop optimizations. Currently we are trying a simple llvm based method jit, which kicks in after counting methods and loops (similar to unladen-swallow and pyston, just smaller and faster), and inlines some <code>pp</code> runtime functions and hereby eliminations to function intro, parameter and return overhead, and then can apply the usual optimizations applied to the bodies without the call overhead.</p>
<h2 id="Optimize-the-vm-the-runloop">Optimize the vm, the runloop</h2>
<p>We carry around way too much bloat in the ops and the data, which is not needed at run-time. e.g. the compiler throws away the nested symbol table stashes if not needed, which frees 20% memory. All the op pointers are not needed at run-time. But think of a lua/p2-like redesign of tagged values and slimmer ops, and eventually put the stack onto the CPU stack.</p>
<p>Note that p5p argues the opposite way. They want to add even more run-time branches to the ops, without any justification.</p>
<p>Optimize special arithmetic op sequences to use unboxed integers and strings on the stack. We experiment in allowing unboxed values on the stack, because the stack is not garbage collected and not refcounted. We just need to be sure to box them before entering a non-collaborating sub, leaving a block with possible exceptions and stack cleanup. Those unboxed values are internally typed as <code>:int</code> and <code>:str</code>. Note that the coretypes <code>:int</code> and <code>:str</code> are not guaranteed to be unboxed, only if the compiler sees fit. In most cases those values are boxed but without a class pointer and magic attached. (Done in the native branch)</p>
<p>Maybe rewrite to a better register-based compiler with fixed-length 2 operands as in p2, but this might be too tricky for XS, mapping the global stack to the local stack. Probably no SSA (three arguments), just a cpu friendly two argument form as in p2/lua 5.1.</p>
<p>Cuurently there's a branch with linearized ops, called OPL without any op pointers, just indices into a oparray per sub. Similar to python, just with common-sized ops, not 1-3 word ops as in python. op_next is always the next op in the array, so cached. op_other are skips.</p>
<p>Allow faster XS calls, user-provided function calls and method calls. Provide support for named arguments in the vm, fast not via hashes. Many of the current io+sys ops are better implemented as library methods. With ~50ops instead of >300 the runloop might fit into the L1 cache again. Seperate calling of fixed arity methods from varargs. detect and use tailcalls automatically. Do not step into a seperate runloop for every single function call, only for coros, which do need to record the the stack information.</p>
<p>Run-time optimize the data, no 2x indirection to access hash and array structs. Provide forwarding pointers to single tuples to hold all. This could provide also the possibility for a GC if a second sweep for timely destruction is doable.</p>
<p>Optimize <b>HEK</b> for faster hash comparison, and use it for native str types, which might benefit from its immutability and the pre-calculated hash, len and utf8 fields.</p>
<p>Check readonly support for <code>PL_strtab</code> for the compiler. Builtin ro ph + dynamic as in warnings XS.</p>
<h2 id="Better-symbol-table">Better symbol table</h2>
<p>Check converting the GV stash tree of hashes into a single global data structure, not a nested hash of hashes: Hash, AVL tree, Trie (TST or R² TST), Patricia trie or DAFSA (Deterministic acyclic finite state automaton) for faster dynamic variable and function name lookup. No binary names, all as UTF8. Maybe restrict to ASCII or valid identifiers to limit the trie memory (array of 26 vs 256). Stashes point then to trie nodes and need a HV check. Optionally provide partial read-only support for the compiler, as for <code>PL_strtab</code>. See the branch <i>feature/gh127-gvflat</i>.</p>
<h2 id="Coro-support">Coro support</h2>
<p>Keep native threads asis (this is not fixable, better remove it), but actively help coroutine and async IO support.</p>
<h2 id="Untangle-the-IO-layer-mess">Untangle the IO layer mess</h2>
<p>A stack is a stack.</p>
<h2 id="Bring-back-proper-match-and-given-when">Bring back proper `match` and `given/when`</h2>
<p>With type support it would be even efficient and helps the inferencer. <code>match</code> needs to be structural, p5p <code>smartmatch</code> can stay dumb as it is now.</p>
<h2 id="clp">clp</h2>
<p>Possibly add a <b>clp</b> library, a constraint logic solver, with possible bindings to external sat solvers, like minisat, which can be included due to its small size and license. It is solved by checking for lvalue function calls in assignments, when the function is not declared as <code>:lvalue</code>.</p>
<pre><code> use clp;
sub fact(int $i=0) :int { assert $i>=0; return $i ? fact($i-1) : 1 }
say fact(7); # => 5040
fact($_) = 5040; # solve it!
say $_; # => 7</code></pre>
<p>The type optimizer with advanced types might eventually benefit from its native performance.</p>
<h1 id="Development-policies">Development policies</h1>
<p>We favor community-friendly democratic development policies as e.g. in perl6 over the usual old-style dictatorial model. That means the powerful (those with management and commit roles) are not allowed to abuse their powers, while the powerless users are allowed and need to have the abilities to criticise them and their code.</p>
<p>In the old trust-based dictatorial model as e.g. in linux or perl5 the powerful call the not powerful abusive names ("asshole" or "jerk" is very common, or "trolls"), and are allowed to avoid discussions of features or problems by directly committing to master, rejecting tickets or selectively abuse their powers. This is forbidden in cperl.</p>
<p>We track stable upstream releases for our releases. Of each major perl5 release we merge eventually most of the p5p commits into cperl, but we make publicly clear beforehand in the merge window which commits will be rejected, and why. Everything is done in public branches. So you can raise your objections or improvements beforehand. Merge and reject follows the same development process as cperl bugfixes, features and security.</p>
<p>Features are developed in sprint-like 1-2 week cycles. If it doesn't work out in this period, switch to the next branches or sometimes continue on it. Several feature branches are already done, but not yet merged, because we wait for other decisions to be made.</p>
<h1 id="With-classes-types-compilable-company-friendly">With classes, types, compilable, company friendly</h1>
<dl>
<dt id="classes"><b>classes</b></dt>
<dd>
<p>A true perl6-like and efficient object system is in work, and a lot of support had been added already: Core and User-types, handling of restricted stashes, parsing the new syntax, efficient fields and compile-time role composition.</p>
<p>One cannot just add classes and objects to core without a type system first.</p>
<p>It supports the perl6-like syntax for class, method, has, roles (mixins, compile-time composable classes), is, does, finalized classes (compile-time optimizations). Later method modifiers (:before, :after, :around), multiple type dispatch (multi, no need to overload methods), and easier perl6-like syntax, which closely resembles early perl6 designs, Damian Conway's perl5i and <code>Moose</code>, without its massive overhead.</p>
<p>See <a href="/cperl/perlclass.html">perlclass</a></p>
</dd>
<dt id="types1"><b>types</b></dt>
<dd>
<p>Types are optional, and make code safer, faster and better documented. cperl includes builtin coretypes, native ffi types, user-defined types and type dispatch, checks, subtype relationships, inference and optimizations on user-defined functions and methods, for XS and PP. Later native aggregate types for arrays, hashes and native classes (structs).</p>
<p>See <a href="/cperl/perltypes.html">perltypes</a></p>
</dd>
<dt id="compilable"><b>compilable</b></dt>
<dd>
<p>perl5 proper started being compiler unfriendly with 5.16, with changed security and then COW handling with 5.18. It can be compiled but it is not recommended, as it features only about 5% memory savings. B::C can still strip the nested stashes (namespaces), but needs to keep all COW strings dynamic. There are possible workarounds such as storing all simple strings as hash keys, or making them immutable, but this is not a viable solution.</p>
<p>With cperl we can again compile to more efficient code, with >35% memory savings.</p>
<p>See <a href="http://search.cpan.org/dist/B-C/">B::C</a></p>
</dd>
<dt id="company-friendly"><b>company friendly</b></dt>
<dd>
<p>The optional types lead to better documentation, earlier compile-time type violations, less needed tests and more performance. Managers love this.</p>
<p>cperl uses a professional development process which is different to the old established dictatorial process of power-abuse, the right to commit to master of without discussion and developer burnout.</p>
<p>It is also better compilable.</p>
</dd>
</dl>
<h1 id="How-to-detect-cperl">How to detect cperl?</h1>
<dl>
<dt id="V-c"><code>"$^V" =~ /c$/</code></dt>
<dd>
</dd>
<dt id="config.h-defines-USE_CPERL"><i>config.h</i> defines <USE_CPERL></dt>
<dd>
</dd>
<dt id="Config.pm-defines-usecperl"><i>Config.pm</i> defines <code>usecperl</code></dt>
<dd>