Ariane Cache

[jctullos]

Hello,

I saw on the website cache isn't implemented yet for Ariane. Is that still the case? Is there any branches that I could test out that uses cache?

Thank you!

[paulmnt]

Thank you for your interest @jctullos.
A preliminary interface of Ariane to the ESP cache hierarchy is available on the branch dev-ariane-smp.

This branch is still not stable, but it supports already Linux booting with one Ariane core and the cache hierarchy present. In addition, all accelerators coherence models (non-coherent DMA, LLC-coherent DMA, coherent DMA and fully-coherent mode) are working properly.

In order to enable multi-core, however, we must first change the Ariane L1 and bus interfaces to enable L1 cache invalidation. Once this edit is complete and merged to the upstream repository in Ariane, we will be able to merge the feature in ESP as well.

[jctullos]

@paulmnt Oh that's perfect, thank you for the response! For my research, I only need 1 core instantiated. I will check out the branch and see how it does!

One last question, does ESP incorporate the Ariane changes to add PMP from around 14 July?

[paulmnt]

Not yet, but I plan to bump Ariane before changing its code.
You may try upgrading it right away, if you need that specific commit:

cd esp/third-party/ariane
git checkout master
git pull
git submodule update --recursive

After you pull, most likely you will have to apply minor patches to the following files:

• esp/utils/ariane_verilog.txt is the list of RTL files to compile inside the Ariane repository.
• esp/rtl/src/ariane/ariane/ariane_wrap.sv is the SystemVerilog wrapper for Ariane, based on the code from Ariane's repo in esp/third-party/ariane/fpga/src/ariane_xilinx.sv
• esp/socs/common/ariane_soc_pkg.sv is the SystemVerilog package that defines a few constants for the interface of Ariane.

[jctullos]

Thank you for the information. I was able to build and boot on the VCU118 before doing an Ariane bump. But Linux had some problems. It gets through the bootup until mounting a files system and I get this error:

It happens each time I try to boot. I'm assuming it's having trouble with the initramfs. Have you seen this before?

[davide-giri]

Hi,

It seems you don't have the full file system, that happens when you don't run the utils/scripts/build_riscv_toolchain.sh script in your ESP clone. To know if this is the problem, after compiling Linux with make linux, check the content of the sysroot folder in your VCU118 design folder.

If you run ls in the sysroot folder you should get this:

applications  bin  dev  etc  init  lib  lib64  linuxrc  media  mnt  opt  proc  root  run  sbin  sys  tmp  usr  var

Instead, if you didn't run the script correctly you will get this:

etc  usr

The instructions for running the script are here: https://www.esp.cs.columbia.edu/docs/setup/setup-guide/#software-toolchain.

[jctullos]

Thanks @davide-giri ! I was in the process of running the riscv scripts as you posted that. I already had the tools built for another build, but figured there's added things that ESP includes.

After deleting the sysroot folder and running make-linux again, all the folders show up now. Once it loads, I'll let you know if it works!

[jctullos]

It worked! Was able to reach login. Thank you both!

[davide-giri]

Perfect!

If you have already installed the toolchain once on a machine, you just have to repeat the third step of the utils/scripts/build_riscv_toolchain.sh for each new ESP clone, you can skip the first two. Specifically this is the step you need to execute:

*** Populating root file system w/ buildroot ... ***

[jctullos]

Am I able to change the ram size easily? The build only has 500mb, but the vcu118 has extra I'd like to use.

[paulmnt]

The current setup builds with 1GB of addressable DRAM, of which 512MB are cached in Ariane's L1 and used by the OS (`0x80000000 - 0xA0000000).
The remaining 512MB are cached by the ESP L2 and LLC, but not Ariane's L1 and are used for devices (e.g. Ethernet) and accelerators (0xA0000000 - 0xC0000000).

You may easily expand the OS memory up to almost 2GB (0x80000000 - 0xF8000000).
Going beyond that requires a bit of work to expand the address width for some of the interconnect adapters and proxy components. This is already in our TODO list.

For the time being, if 2GB can do, edit these lines

DRAMLength       => X"0000_0000_7800_0000",
DRAMCachedLength => X"0000_0000_7800_0000")

This setup will not leave any area for the ESP allocator, so ESP accelerators will not work correctly, not even in fully-coherent mode, because Ariane is still missing the invalidate interface for the L1. If you wish to preserve some area for accelerators, reduce the DRAMCachedLength.

Note that after changing these parameters, you need to update the device tree ariane.dts, and, if using accelerators, the init script that loads ESP drivers in Linux sysroot/etc/init.d/S64esp.
Both files are generated, but I would change them manually until you have a stable version. Then you may edit the generator utils/socmap/socmap_gen.py.

Optionally, you can update the size of the available DRAM for the bootloader as well.

[jctullos]

Thank you Paul for the detailed instructions.

So I'm assuming that when Linux boots, and shows the various allocations for accelerators, it's only listing them.

They're not activated correct? The cache I'm good with, but I don't want accelerators like the fft and adder turned on for benchmarks.

[paulmnt]

We've added an lsmod in the init script just to show the drivers that have been loaded. However, since you don't have accelerators in the system instance, no device is registered and drivers won't affect your experiments.

If you don't want those to be loaded at all, you can remove sysroot/etc/init.d/S64esp and sysroot/etc/init.d/S65drivers from sysroot in your design folder. Just recompile Linux and at the next boot accelerators' drivers won't be loaded.

[jctullos]

Ah thank you!

And sorry for so many questions! I didn't see any documentation for changing the CPU frequency. If I change the freq in top.vhd to 100MHz instead of 78, are there any additional changes I would make that would affect either the UART or Ethernet clocks?

I synthesized the build and at first glance, everything was generated correctly. No issues with timing on the design. I'm remaking Linux right now to test it.

[paulmnt]

We are happy to answer questions, no worries!

We have set frequencies on all boards to make sure that most designs will meet timing, regardless of the number of accelerator tiles that you add and considering the performance achievable by the third-party IPs (such as Ariane).
Please note that Ariane will not close faster than 78MHz on VCU118 very easily.
I believe that 100 MHz with the default ESP configuration (1 CPU tile + 1 memory tile + I/O tile) should work, although you may notice small setup violations. Typically, the critical paths are inside the FPU.

Having said that, changing the CPU_FREQ constant in top.vhd will not actually change the frequency of the synthesized design, but it will change the values of the scaler computed for Ethernet and UART, so they will not work correctly.

If you want to change the frequency of the design, then you must also edit the constraint files. For VCU118, for example, all constraints and pin allocations are stored in the folder constraints/xilinx-vcu118-xcvu9p.
The file you need to change is mig.tcl:

set_property -dict [list \
                        CONFIG.C0_CLOCK_BOARD_INTERFACE {default_250mhz_clk1} \
                        CONFIG.C0.DDR4_TimePeriod {1600} \
                        CONFIG.C0.DDR4_InputClockPeriod {4000} \
                        CONFIG.C0.DDR4_CLKOUT0_DIVIDE {8} \
                        CONFIG.C0.DDR4_MemoryPart {MT40A256M16GE-083E} \
                        CONFIG.C0.DDR4_DataWidth {64} \
                        CONFIG.C0.DDR4_DataMask {DM_NO_DBI} \
                        CONFIG.C0.DDR4_Ordering {Strict} \
                        CONFIG.C0.DDR4_CasLatency {9} \
                        CONFIG.C0.DDR4_CasWriteLatency {9} \
                        CONFIG.C0.DDR4_Mem_Add_Map {BANK_ROW_COLUMN} \
                        CONFIG.ADDN_UI_CLKOUT1_FREQ_HZ {78} \ # EDIT THIS LINE to change Ariane's clock
                        CONFIG.C0.BANK_GROUP_WIDTH {1}\
                       ] [get_ips mig]

After applying this change, make sure to run make vivado-distclean before repeating synthesis, or the memory interface generator will not pick the update.

[jctullos]

That's great! I'll try it out later.

FYI, bumping the Ariane submodule so that I can include PMP works! The files you mentioned earlier were exactly the ones needing to be patched. Nothing needed to be changed in ariane_soc_pkg.sv file.

[paulmnt]

Great and thank you for letting me know!

[Kendidi]

@jctullos

Hi, did you went through the whole process, from building bootroom, bbl to vmlinux?
Is it possible to just build the bbl and vmlinux and use Ariane bit file from CVA6?
Thanks.

[jctullos]

Hey Kendidi,

I used the whole process. You can build bbl and Linux without generating the entire bitstream. But it has configurations and patches specific to ESP, so it probably wouldn't work on other builds.

[Kendidi]

Hi Jctullos,

I see. I do not have the Xilinx license to build bitfiles. Would be nice if they have pre-built bit files for Ariane on Xilinx VCU128.

[davide-giri]

Hi @Kendidi,

I have uploaded on the ESP website the pre-built files for the Xilinx VCU128 board for an SoC with 1 Ariane core. You can find it here https://www.esp.cs.columbia.edu/prebuilt/singlecore/ with the name ESP_singlecore_ariane_nocaches_vcu128_20200925_GitRev278f1a2.tar.gz.

The guide on how to design and use a single core ESP SoC is here: https://www.esp.cs.columbia.edu/docs/singlecore/singlecore-guide. At the bottom, the guide includes the instructions on how to use the prebuilt material.

Please, let us know if you have any further question. Thank you!

[paulmnt]

Hi, did you went through the whole process, from building bootroom, bbl to vmlinux?

Is it possible to just build the bbl and vmlinux and use Ariane bit file from CVA6?

Yes, the generated bbl file can be used for Ariane's SD card boot. The patches in Linux and BBL are simply adding support for peripherals from GRLIB, which we use in ESP. Other than that, both Linux and BBL are not modified. Please note that ESP drivers are not necessary with the CVA6 design, so you can remove their init scripts from the sysroot/etc/init.d to avoid error messages after boot completes. The scripts are S64 and S65. After removing them, you just need to recompile with 'make linux'.

[Kendidi]

Thank you @davide-giri !! I will check it out. Appreciate it.

Hi @paulmnt, the Xilinx VCU128 board I have has no SD card slot. It has jTAG support though. Can I use OpenOCD & GDB to load the bbl (with vmlinux embedded) to certain memory address and run instead?

[paulmnt]

Hi @Kendidi

When I mentioned the SD card, I was referring to the FPGA bit built with the CVA6 repo. Their default flow to run on FPGA relies on SD card and, for that flow, you may use the bbl image built within ESP, instead of the image built with Ariane SDK.

Instead, if you use the ESP prebuilt image that @davide-giri created, you only need to connect the VCU128 to the same local Ethernet network of your computer and use the included runme.sh script.

We are not relying on GDB to load programs. Instead, ESP has an Ethernet link based on the GRLIB EDCL IP, which allows for faster boot. The ESP guide to configure the debug link describes a couple of options to get the connection with the FPGA working.

I recommend using a router, if available, but the direct link does work as well. If you use a router, your computer can connect to it via Wi-Fi, while the FPGA must use a cable.

The prebuilt image for VCU128 is configured as follows:

  # Assume local network DHCP IPs are of the type 192.168.1.x and 192.168.1.12 is available
  ESPLINK_IP  : 192.168.1.12
  ESPLINK_PORT  :  46392

Whether you're using a router or a direct link with your computer NIC, it is possible to configure DHCP such that 192.168.1.12 is a valid IP.

When you run the runme.sh script in the prebuilt folder two things will happen:

1. Vivado will try to program the VCU128 FPGA assuming the JTAG port is connected to your machine (localhost, port 3121).
2. the esplink application will load the bootloader on a BRAM and the Linux image on the VCU128 DDR memory. Then esplink sends a reset to the Ariane processor to boot the system. The application can only work if your computer and the FPGA are connected to the same Ethernet subnetwork and 192.168.1.12 is a valid address on such network.

Please let us know if you have troubles booting the system. Thank you!

[Kendidi]

Thank you @paulmnt for the detailed information. I will try it out.

[Kendidi]

@jctullos

I am curious which FPGA board have you been using?

[jctullos]

I've been using the VCU118. But having some trouble with Linux again.

It's hanging now between BBL and Linux. I was having the same proble with the openpiton build. I wonder if it's because of PMP, there might be an issue with it.

[davide-giri]

Just a clarification, the pre-built material for the single-core SoC tutorial does not contain the runme.sh script. The instructions on how to use the pre-built material are at the end of the tutorial.

[Kendidi]

Ooh, I see. Thank you @davide-giri !!

[paulmnt]

I am looking into the PMP implementation in Ariane. The objective is to track which new information is exposed (if any) to the bus to "tag" the address accesses. Or, at least, to see if there is any of the recent change is likely trigger it.

[Kendidi]

Thank you @paulmnt very much for your help!! It works. FPGA was programmed, binary files were loaded and Kernel was running all the way to the "esp login" prompt.

Are the pre-built files using the latest source code? I wonder if I build files from scratch, will it be able to boot again or will I encounter the PMP issue. Thanks.

[davide-giri]

That's great news @Kendidi ! The prebuilt for VCU128 uses the latest commit on the master branch of ESP. You won't encounter the PMP issue, because at the moment ESP doesn't point to the latest commit of the Ariane (aka cva6) repository, so the PMP is not included.

[Kendidi]

Cool! So which Ariane repository ESP is pointing to? Thanks.

[paulmnt]

The repo is the same, but it's pointing to an older commit: 465bb209a "wt_cache_subsystem: Fix spelling mistakes (#453)"

[Kendidi]

Thanks @paulmnt !

As an experiment, I tried to build from existing ESP source code on my system and see if the generated files work or not.
I issued "make linux". I see prom.bin and linux.bin were updated.
I then run "runme.sh" and I got the following panic situation. Any idea what may be an issue? Thanks.

bbl loader                                                                      
[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000           
[    0.000000] Linux version 5.1.0-g77bea430730a-dirty (aie@aie-Machine) (gcc v0
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')                      
[    0.000000] printk: bootconsole [sbi0] enabled                               
[    0.000000] initrd not found or empty - disabling initrd                     
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a0000000, B
[    0.000000] OF: reserved mem: initialized node buffer@A0000000, compatible il
[    0.000000] Zone ranges:                                                     
[    0.000000]   DMA32    [mem 0x0000000080200000-0x000000009fffffff]           
[    0.000000]   Normal   empty                                                 
[    0.000000] Movable zone start for each node                                 
[    0.000000] Early memory node ranges                                         
[    0.000000]   node   0: [mem 0x0000000080200000-0x000000009fffffff]          
[    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x000000009fffffff] 
[    0.000000] software IO TLB: mapped [mem 0x9b8fd000-0x9f8fd000] (64MB)       
[    0.000000] elf_hwcap is 0x112d                                              
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 128775    
[    0.000000] Kernel command line: earlyprintk console=hvc earlycon=sbi        
[    0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)  
[    0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)   
[    0.000000] Sorting __ex_table...                                            
[    0.000000] Memory: 441732K/522240K available (3904K kernel code, 233K rwdat)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1       
[    0.000000] rcu: Preemptible hierarchical RCU implementation.                
[    0.000000] rcu:     RCU event tracing is enabled.                           
[    0.000000]  Tasks RCU enabled.                                              
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 ji.
[    0.000000] NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0                     
[    0.000000] plic: mapped 16 interrupts with 1 handlers for 2 contexts.       
[    0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycs
[    0.000186] sched_clock: 64 bits at 37MHz, resolution 26ns, wraps every 2199s
[    0.026225] Calibrating delay loop (skipped), value calculated using timer f)
[    0.064851] pid_max: default: 32768 minimum: 301                             
[    0.087278] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)      
[    0.107545] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes) 
[    0.145088] *** VALIDATE proc ***                                            
[    0.173814] rcu: Hierarchical SRCU implementation.                           
[    0.206259] devtmpfs: initialized                                            
[    0.241034] random: get_random_bytes called from setup_net+0x32/0x150 with c0
[    0.271356] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, ms
[    0.301697] futex hash table entries: 256 (order: 0, 6144 bytes)             
[    0.330616] NET: Registered protocol family 16                               
[    0.535651] usbcore: registered new interface driver usbfs                   
[    0.553781] usbcore: registered new interface driver hub                     
[    0.571738] usbcore: registered new device driver usb                        
[    0.604563] clocksource: Switched to clocksource riscv_clocksource           
[    0.655155] NET: Registered protocol family 2                                
[    0.690728] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096)
[    0.715424] TCP established hash table entries: 4096 (order: 3, 32768 bytes) 
[    0.739335] TCP bind hash table entries: 4096 (order: 3, 32768 bytes)        
[    0.761385] TCP: Hash tables configured (established 4096 bind 4096)         
[    0.783127] UDP hash table entries: 256 (order: 1, 8192 bytes)               
[    0.801400] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)          
[    0.825383] NET: Registered protocol family 1                                
[    0.850128] RPC: Registered named UNIX socket transport module.              
[    0.868804] RPC: Registered udp transport module.                            
[    0.883043] RPC: Registered tcp transport module.                            
[    0.897252] RPC: Registered tcp NFSv4.1 backchannel transport module.        
[    2.242773] workingset: timestamp_bits=62 max_order=17 bucket_order=0        
[    2.540842] NFS: Registering the id_resolver key type                        
[    2.556731] Key type id_resolver registered                                  
[    2.569418] Key type id_legacy registered                                    
[    2.581776] nfs4filelayout_init: NFSv4 File Layout Driver Registering...     
[    2.793278] io scheduler mq-deadline registered                              
[    2.808436] io scheduler kyber registered                                    
[    4.145684] Serial: GRLIB APBUART driver                                     
[    4.167553] 60000100.uart: ttyS0 at MMIO 0x60000100 (irq = 3, base_baud = 46T
[    4.199767] grlib-apbuart at 0x60000100, irq 3                               
[    4.225054] libphy: Fixed MDIO Bus: probed                                   
[    4.257928] libphy: greth-mdio: probed                                       
[    5.919794] grlib-greth 60080000.greth: assigned reserved memory node buffer0
[    5.956958] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver       
[    5.987725] usbcore: registered new interface driver usbhid                  
[    6.004809] usbhid: USB HID core driver                                      
[    6.040784] NET: Registered protocol family 10                               
[    6.081677] Segment Routing with IPv6                                        
[    6.096058] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver              
[    6.130435] NET: Registered protocol family 17                               
[    6.148563] Key type dns_resolver registered                                 
[    6.178768] Warning: unable to open an initial console.                      
[    6.197641] Failed to create /dev/root: -2                                   
[    6.211951] VFS: Cannot open root device "(null)" or unknown-block(0,0): err2
[    6.235532] Please append a correct "root=" boot option; here are the availa:
[    6.261619] Kernel panic - not syncing: VFS: Unable to mount root fs on unkn)
[    6.286336] CPU: 0 PID: 1 Comm: swapper Not tainted 5.1.0-g77bea430730a-dirt5
[    6.308415] Call Trace:                                                      
[    6.316041] [<ffffffe00007c06a>] walk_stackframe+0x0/0xa0                    
[    6.332081] [<ffffffe00007c266>] show_stack+0x2a/0x34                        
[    6.347203] [<ffffffe0004323e8>] dump_stack+0x20/0x28                        
[    6.362398] [<ffffffe00007fff2>] panic+0xe2/0x218                            
[    6.376502] [<ffffffe000000e96>] mount_block_root+0x1e2/0x27a                
[    6.393712] [<ffffffe000000fbc>] mount_root+0x8e/0x98                        
[    6.408886] [<ffffffe0000010e0>] prepare_namespace+0x11a/0x164               
[    6.426395] [<ffffffe000000b10>] kernel_init_freeable+0x184/0x1a0            
[    6.444738] [<ffffffe000445dea>] kernel_init+0x12/0xf0                       
[    6.460120] [<ffffffe00007aff0>] ret_from_exception+0x0/0xc                  
[    6.476894] ---[ end Kernel panic - not syncing: VFS: Unable to mount root f-

[jctullos]

@Kendidi

This happened to me earlier, it's due to not running the script build_riscv_toolchain, make sure to run that in the utils/scripts directory. It builds RISCV and buildroot, which builds the initramfs system.

[Kendidi]

Ohh. I see. I will run it. Thanks @jctullos !

[davide-giri]

Correct, that step is part of the setup guide: https://www.esp.cs.columbia.edu/docs/setup/setup-guide/#software-toolchain.

[Kendidi]

I followed the steps from the beginning but have trouble installing the toolchain. Folder "/opt/riscv/bin" was not installed. There is only a "sysroot" folder under "/opt/riscv/". Please advise.

......
......
/usr/bin/install -m 0644 support/misc/target-dir-warning.txt /home/aie/esp/20200929/esp/_riscv_build/buildroot/output/target/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM
>>> skeleton-init-sysv  Extracting
>>> skeleton-init-sysv  Patching
>>> skeleton-init-sysv  Configuring
>>> skeleton-init-sysv  Building
>>> skeleton-init-sysv  Installing to target
rsync -a --ignore-times --exclude .svn --exclude .git --exclude .hg --exclude .bzr --exclude CVS --chmod=u=rwX,go=rX --exclude .empty --exclude '*~' package/skeleton-init-sysv//skeleton/ /home/aie/esp/20200929/esp/_riscv_build/buildroot/output/target/
>>> skeleton  Extracting
>>> skeleton  Patching
>>> skeleton  Configuring
>>> skeleton  Building
>>> skeleton  Installing to target
>>> toolchain-external-custom  Extracting
>>> toolchain-external-custom  Patching
>>> toolchain-external-custom  Configuring
Cannot execute cross-compiler '/opt/riscv/bin/riscv64-unknown-linux-gnu-gcc'
package/pkg-generic.mk:219: recipe for target '/home/aie/esp/20200929/esp/_riscv_build/buildroot/output/build/toolchain-external-custom/.stamp_configured' failed
make: *** [/home/aie/esp/20200929/esp/_riscv_build/buildroot/output/build/toolchain-external-custom/.stamp_configured] Error 1


=== Use the following to load RISCV environment ===
  export PATH=/opt/riscv/bin:$PATH
  export RISCV=/opt/riscv

*** Successfully installed RISCV toolchain to /opt/riscv ***

[davide-giri]

The error you are reporting is the consequence of a previous error, what's the first error you get?

[Kendidi]

OK. Backtracking the toolchian installation process showed that package "bison" was missing. Now it appears something is being installing because the system responses very very slowly and the disk drive LED keeps flashing. Thanks!

[Kendidi]

It went a lot further and then encountered a lot of "g++: internal compiler error: Killed (program cc1plus)".

......
......
../.././riscv-gcc/gcc/c/c-typeck.c: In function ‘void error_init(location_t, const char*)’:
../.././riscv-gcc/gcc/c/c-typeck.c:6182:24: warning: format not a string literal and no format arguments [-Wformat-security]
   error_at (loc, gmsgid);
						^
../.././riscv-gcc/gcc/c/c-typeck.c: In function ‘void warning_init(location_t, int, const char*)’:
../.././riscv-gcc/gcc/c/c-typeck.c:6228:43: warning: format not a string literal and no format arguments [-Wformat-security]
   warned = warning_at (exploc, opt, gmsgid);
										   ^
/bin/bash: line 3: 20195 Killed                  makeinfo --split-size=5000000 --no-split -I . -I ../.././riscv-gcc/gcc/doc -I ../.././riscv-gcc/gcc/doc/include -o doc/gcc.info ../.././riscv-gcc/gcc/doc/gcc.texi
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
../.././riscv-gcc/gcc/collect-utils.c: In function ‘pex_obj* collect_execute(const char*, char**, const char*, const char*, int, bool)’:
../.././riscv-gcc/gcc/collect-utils.c:195:37: warning: format not a string literal and no format arguments [-Wformat-security]
  fatal_error (input_location, errmsg);
									 ^
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
g++: internal compiler error: Killed (program cc1plus)
g++: internal compiler error: Killed (program cc1plus)
......
......

[davide-giri]

It's possible your system ran out of memory. If you run dmesg right after the errors there may be an "out of memory" message to confirm this hypothesis. What did you specify when the script asked how many threads to use? If my guess is right, running with less threads (like 1 or 2) should solve the issue.

[Kendidi]

Yup, Out of Memory.

I did not specify. I just pressed "ENTER".
Any recommended number?

dmesg:

......
......
[47401.477906] [  21846]     0 21846    31645    16687   266240      896             0 cc1plus
[47401.477907] [  21847]     0 21847    27215    10339   233472     1295             0 cc1plus
[47401.477909] [  21849]     0 21849    30955    13893   266240      858             0 cc1plus
[47401.477910] [  21850]     0 21850    28725    13446   241664     1290             0 cc1plus
[47401.477911] [  21855]     0 21855    33902    16320   294912     1265             0 cc1plus
[47401.477912] [  21859]     0 21859    32858    16364   282624      810             0 cc1plus
[47401.477913] [  21862]     0 21862    32170    14521   278528     1886             0 cc1plus
[47401.477915] [  21866]     0 21866    34404    16985   299008     1123             0 cc1plus
[47401.477916] [  21871]     0 21871    30132    13097   266240     1851             0 cc1plus
[47401.477917] [  21872]     0 21872    26377     9385   229376     1105             0 cc1plus
[47401.477918] [  21873]     0 21873    31013    13099   270336     1692             0 cc1plus
[47401.477920] [  21886]  1000 21886    62056      218   417792        2             0 gnome-shell
[47401.477921] [  21941]     0 21941     7613     1333    94208        0             0 as
[47401.477923] [  21951]     0 21951     4190      170    69632        0             0 as
[47401.477924] [  21952]     0 21952     4190      254    73728        0             0 as
[47401.477925] [  21953]     0 21953      311        3    28672        0             0 as
[47401.477926] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=cc1plus,pid=21782,uid=0
[47401.477932] Out of memory: Killed process 21782 (cc1plus) total-vm:145996kB, anon-rss:78640kB, file-rss:8592kB, shmem-rss:0kB, UID:0 pgtables:300kB oom_score_adj:0
[47401.481110] oom_reaper: reaped process 21782 (cc1plus), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[47404.302775] systemd-journald[359]: /dev/kmsg buffer overrun, some messages lost.

[davide-giri]

So my guess was right, there is an "out of memory".

Pressing "ENTER" executes the default option, which is always listed between brackets. Specify 1 this time, that will be slow, but hopefully it will avoid the "out of memory" issue.

[Kendidi]

@davide-giri

Yup. You are right. I tried '4' and everything seems built OK.

In my case (for a freshly installed Ubuntu 18.04 platform), I needed to run the following as well.

sudo apt-get install gawk
sudo apt-get install bison

Thanks a lot for your help!!

[Kendidi]

I run the bbl+Kernel built with ESP today against a bit file built from recent Ariane code and encountered the following.

bbl loader
[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[    0.000000] Linux version 5.1.0-g77bea430730a-dirty (aie@aie-ROG-STRIX-Z390-E-GAMING) (gcc vers0
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] initrd not found or empty - disabling initrd   
[    0.000000] Zone ranges:                                   
[    0.000000]   DMA32    [mem 0x0000000080200000-0x00000000bfffffff]
[    0.000000]   Normal   empty                               
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080200000-0x00000000bfffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x00000000bfffffff]
[    0.000000] software IO TLB: mapped [mem 0xbb1fd000-0xbf1fd000] (64MB)
[    0.000000] elf_hwcap is 0x112d
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 258055
[    0.000000] Kernel command line: earlyprintk console=hvc earlycon=sbi
[    0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.000000] Sorting __ex_table...
[    0.000000] BUG: Bad page state in process swapper  pfn:80c01
[    0.000000] page:ffffffe03f02a038 count:1 mapcount:0 mapping:0000000000000000 index:0x0
[    0.000000] flags: 0x0()
[    0.000000] raw: 0000000000000000 0000000000000100 0000000000000200 0000000000000000
[    0.000000] raw: 0000000000000000 0000000000000000 00000001ffffffff
[    0.000000] page dumped because: nonzero _refcount
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.1.0-g77bea430730a-dirty #7
[    0.000000] Call Trace:
[    0.000000] [<ffffffe000409066>] walk_stackframe+0x0/0xa0
[    0.000000] [<ffffffe000409262>] show_stack+0x2a/0x34
[    0.000000] [<ffffffe0007bf522>] dump_stack+0x20/0x28
[    0.000000] [<ffffffe00045f6dc>] bad_page+0xe0/0xfe
[    0.000000] [<ffffffe00045f732>] free_pages_check_bad+0x38/0x7a
[    0.000000] [<ffffffe00045fd48>] free_pcppages_bulk+0x106/0x38e
[    0.000000] [<ffffffe0004603a4>] free_unref_page_commit.isra.27+0x86/0x8e
[    0.000000] [<ffffffe000460daa>] free_unref_page+0x40/0x54
[    0.000000] [<ffffffe000460dcc>] __free_pages.part.4+0xe/0x22
[    0.000000] [<ffffffe000460ff4>] __free_pages_core+0x94/0xa0
[    0.000000] [<ffffffe000005db6>] memblock_free_pages+0x12/0x1a
[    0.000000] [<ffffffe000008b4e>] memblock_free_all+0x190/0x1f4
[    0.000000] [<ffffffe0000023ee>] mem_init+0x2a/0x38
[    0.000000] [<ffffffe000000848>] start_kernel+0x1cc/0x360
[    0.000000] [<ffffffe000000076>] clear_bss_done+0x3a/0x3e
[    0.000000] Disabling lock debugging due to kernel taint
[    0.000000] BUG: Bad page state in process swapper  pfn:80c02
[    0.000000] page:ffffffe03f02a070 count:1 mapcount:0 mapping:0000000000000000 index:0x0
......
......

The bbl+kernel built yesterday (w/o initramfs system built) did not encounter this. Could this be PMP related or something else? Thanks.

[Kendidi]

The following is resulted from pre-built top.bit, prom.bit and systest.bin downloaded today, plus the linux.bin that I built today with "make linux". Please advise if I have missed any steps that may caused the PANIC? Thanks a lot in advance!

bbl loader
[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[    0.000000] Linux version 5.1.0-g77bea430730a-dirty (aie@aie-ROG-STRIX-Z390-E-GAMING) (gcc version 8.3.0 (GCC)) #4 PREEMPT Wed Sep 30 17:35:52 PDT 0
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] initrd not found or empty - disabling initrd
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a0000000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node buffer@A0000000, compatible id shared-dma-pool
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000080200000-0x000000009fffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080200000-0x000000009fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x000000009fffffff]
[    0.000000] software IO TLB: mapped [mem 0x9b8fd000-0x9f8fd000] (64MB)
[    0.000000] elf_hwcap is 0x112d
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 128775
[    0.000000] Kernel command line: earlyprintk console=hvc earlycon=sbi
[    0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.000000] Sorting __ex_table...
[    0.000000] Memory: 438092K/522240K available (3904K kernel code, 236K rwdata, 1161K rodata, 4124K init, 795K bss, 84148K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000]  Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0
[    0.000000] plic: mapped 16 interrupts with 1 handlers for 2 contexts.
[    0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x8a60dd6a9, max_idle_ns: 440795204056 ns
[    0.000178] sched_clock: 64 bits at 37MHz, resolution 26ns, wraps every 2199023255540ns
[    0.026255] Calibrating delay loop (skipped), value calculated using timer frequency.. 75.00 BogoMIPS (lpj=375000)
[    0.064848] pid_max: default: 32768 minimum: 301
[    0.087359] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)
[    0.107624] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes)
[    0.145007] *** VALIDATE proc ***
[    0.173853] rcu: Hierarchical SRCU implementation.
[    0.203860] devtmpfs: initialized
[    0.239443] random: get_random_bytes called from setup_net+0x32/0x150 with crng_init=0
[    0.269934] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.300275] futex hash table entries: 256 (order: 0, 6144 bytes)
[    0.329137] NET: Registered protocol family 16
[    0.537003] usbcore: registered new interface driver usbfs
[    0.555743] usbcore: registered new interface driver hub
[    0.573741] usbcore: registered new device driver usb
[    0.606923] clocksource: Switched to clocksource riscv_clocksource
[    0.658108] NET: Registered protocol family 2
[    0.693631] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes)
[    0.718143] TCP established hash table entries: 4096 (order: 3, 32768 bytes)
[    0.742071] TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
[    0.764134] TCP: Hash tables configured (established 4096 bind 4096)
[    0.785864] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    0.804178] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.828071] NET: Registered protocol family 1
[    0.852802] RPC: Registered named UNIX socket transport module.
[    0.871502] RPC: Registered udp transport module.
[    0.885732] RPC: Registered tcp transport module.
[    0.899967] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    8.989749] workingset: timestamp_bits=62 max_order=17 bucket_order=0
[    9.295323] NFS: Registering the id_resolver key type
[    9.311314] Key type id_resolver registered
[    9.323990] Key type id_legacy registered
[    9.336344] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    9.549426] io scheduler mq-deadline registered
[    9.564460] io scheduler kyber registered
[   10.924237] Serial: GRLIB APBUART driver
[   10.946223] 60000100.uart: ttyS0 at MMIO 0x60000100 (irq = 3, base_baud = 4687500) is a GRLIB/APBUART
[   10.979224] grlib-apbuart at 0x60000100, irq 3
[   11.004214] libphy: Fixed MDIO Bus: probed
[   11.038504] libphy: greth-mdio: probed
[   12.718269] grlib-greth 60080000.greth: assigned reserved memory node buffer@A0000000
[   12.755499] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[   12.786623] usbcore: registered new interface driver usbhid
[   12.803701] usbhid: USB HID core driver
[   12.840331] NET: Registered protocol family 10
[   12.881028] Segment Routing with IPv6
[   12.894957] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[   12.929866] NET: Registered protocol family 17
[   12.948112] Key type dns_resolver registered
[   12.979023] Warning: unable to open an initial console.
[   13.055678] Freeing unused kernel memory: 4124K
[   13.069504] This architecture does not have kernel memory protection.
[   13.088952] Run /init as init process
[   13.290321] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[   13.313279] CPU: 0 PID: 1 Comm: init Not tainted 5.1.0-g77bea430730a-dirty #4
[   13.334544] Call Trace:
[   13.342158] [<ffffffe000409066>] walk_stackframe+0x0/0xa0
[   13.358206] [<ffffffe000409262>] show_stack+0x2a/0x34
[   13.373324] [<ffffffe0007bf522>] dump_stack+0x20/0x28
[   13.388511] [<ffffffe00040cff4>] panic+0xe2/0x218
[   13.402586] [<ffffffe00040e4e4>] do_exit+0x75a/0x778
[   13.417484] [<ffffffe00040ef2c>] do_group_exit+0x28/0x92
[   13.433437] [<ffffffe00040efae>] __wake_up_parent+0x0/0x22
[   13.449906] [<ffffffe000407fde>] ret_from_syscall+0x0/0xe
[   13.466171] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 ]---

[Kendidi]

The following is resulted from pre-built top.bit, prom.bit and systest.bin downloaded today, plus the linux.bin that I built today with "make linux". Please advise if I have missed any steps that may caused the PANIC? Thanks a lot in advance!

This issue appears to be related to the changes I made to config file ariane_defconfig. After I reverted the changes and rebuilt linux.bin, it can boot successfully again. Thanks..

[Kendidi]

It appears riscv64-unknown-elf-gdb is not included in the toolchain build. Is it possible to enable it? Thanks.

[jctullos]

@Kendidi Yes, you would just enable it in the build toolchain script, if you go through it you'll see where it takes out GDB build so that it builds faster.

[jctullos]

@paulmnt I'm going to close this so it doesn't continue to build in messages. Sorry for so many different issues to jump into one github issue!

[Kendidi]

@jctullos

Got it. Thank you!

[Kendidi]

@paulmnt I'm going to close this so it doesn't continue to build in messages. Sorry for so many different issues to jump into one github issue!

My bad. Sorry! But appreciate all your help!!

[paulmnt]

Apologies for not managing this earlier. I've created two separate issues for further discussion: #65 and #66