RAM Memory access from RVCop64 - dependency on C64 memory bank configuration?

[pottendo]

hi,
long time since I've played with the Orangecart and in particular with the RVCop64.
I've written a few gfx-coproc routines for the OC (on my Zephyr port *) to support the C64 with fast line- or circle drawing directly into the main RAM memory.
In my test-programs this works pretty well (e.g. large filled circles draw in <15 milliseconds).
My use-case would be to replace 'Elite's' drawing routines - as this superior C64 game would benefit from such an accelerator
(at least I assume).
Elite has been reverse engineered and build/runs flawlessly on my real C64 - see Elite Harmless

I've already found the code to replace and here my question:
Elite plays a lot with mem-cfgs ($01 register) as it's pretty memory hungry. So basically all the time all ROMs and I/Os are disabled and only occasionally ROMS or I/O is switched on, when needed.
What I see now is:
before Elite starts and initializes, my busy wait routine on the OC sees the following, reading the shared memory handover register (ctr_reg):

ctr_reg(0xf0004d0): 01/00/fa/fb/a4/a7/a9/9b/48

as soon Elite initializes, the same address shows:

ctr_reg(0xf0004d0): ff/ff/ff/ff/ff/ff/ff/ff/ff

The address 0x04d0 isn't used by elite - I've double checked with another address in 0xc000 area - here the same change to ff/ff/...
As debugging Elite on a real C64 without some expansion cartridge is not easy, I'm stuck.

So, can there be an easy explanation that if $01 disables the ROMs that, as a consequence, the RVCop64 won't see the respective RAM?
The RAM accessed is neither below any ROM or I/O space (0x04d0), so I thought, I'd be OK with this approach.
The code uses the cache disabling instruction you've told me some months ago.

*) I needed to map the C64 memory in my RVCop64 incarnation to 0x0f00000 - not 0x00000000 as in the original, in order to support booting Linux alternatively.

thanks for any feedback, pottendo

PS: I'll share the code soon - need to fix the build properly for recent Zephyr release,

[zeldin]

No, the only thing which should prevent you from seeing RAM below $8000 is if you enter Ultimax mode (#GAME = 0, #EXROM = 1). And even then anything below $1000 should be RAM. (It would leave $C000-$CFFF as floating though.)

A DMA cycle from the cartridge is seeing the same memory map as the 6510 does (with the exception of the internal registers mapped to $0000 and $0001), because they happen during Phi2 cycles when BA is high, meaning that AEC is also high just like when the CPU does memory accesses. So the address decoder PLA will interpret A12-A15, #LORAM, #HIRAM, #CHAREN, #GAME, #EXROM and R/#W in the same way as when the 6510 does an access.

Here's a nice document about exactly how the address decoding is performed in the C64, if you are interrested: http://www.zimmers.net/anonftp/pub/cbm/firmware/computers/c64/C64_PLA_Dissected.pdf

So, my random guess would rather be that the RiscV core is aborting the memory access because it's taking too long or something. The cartridge will only start a DMA cycle when it considers it safe. This means either after a bad line, or after the 6510 writes to the bus. So if Elite blanks the screen and only does reads for a while, I guess something could timeout?

[zeldin]

Expanding a bit on the timeout idea:
If you look in hw/rtl/basesoc.py, you will see a call to SoCCore.__init__ around line 80. You can amend this call to include a keyword argument bus_timeout=1e8. If this keyword argument is not stated, the default is 1e6, meaning bus cycles are aborted after one million clock cycles if they haven't completed by then. If you increase this by a factor 100, then there should at least be a noticeable delay if a bus abort happens, giving you a bit of a hint...

[pottendo]

Hi,
thanks for the clear explanation and the reference - I'll do some experiments later playing with the bus_timeout parameter.
I now try to isolate the behavior into a tiny testprogram now. Elite is a bit bulky.
Stay tuned, pottendo

[pottendo]

Hi,
Trying to isolate the behavior I confirm that membanking isn't an issue. Memory is accessible regardless of $01 configuration, as you said.
Now I can't reproduce the specific issue with elite anymore...
...even better: it starts working! Not yet perfect and not noticeably faster, but the OC draws lines and theoretically circles, on behalf of elites algorithms.
I did change the coproc register address. Probably the first attempt wasn't free of side effects for the main program and it just crashed.

It seems that the line drawing isn't the main bottleneck determining elites framedrops - many lines drawn are.short, so the benefit isn't substantial. I have to get the filled circle (sun in elite) going. But here elite seems to be very clever and therefore I haven't found the right entry yet.

Anyway, it's fun building kind of a GPU for the old C64.

Happy hacking, pottendo

[pottendo]

Hi,

here the code so far:
Zephyr for RVCop64: https://github.com/pottendo/zephyr-pottendo (branch wip2)
and here the co-proc: https://github.com/pottendo/zephyr-pottendo/tree/wip2/samples/oc-coproc
You need my RVCop64: https://github.com/pottendo/RVCop64-pottendo
and finally the pimped Elite code: https://github.com/pottendo/elite-harmless-coproc
...the version 'elite-harmless-fastlines' is changed, the others not.

Still things like circles / sun aren't properly drawn as they are not properly deleted - but already 'nicer' as the original as a real circle is drawn (instead of a approximation with 64 lines).

Not (yet) noticably 'faster' lines... maybe I find more algos in the code to be replaced by a co-proc version.
Circles I think are faster (and nicer).

have fun, pottendo

[zeldin]

Nice!
BTW, you may want to check out the new "mailbox" function I just added. I think it could be useful for you because it lets the RISC-V poll a shared area as often as it likes without causing a DMA cycle, so no need for "usleep" if you use it. 😸
zeldin/RVCop64@2d33985e

[pottendo]

Hi, This looks great! Indeed that will improve the coproc.So far I need to peek into the i/o area on the c64 side to enable faster RAM access from the RiscV side. If I do busy waiting on the shared RAM address without some.dummy reading i/o, it's damn slow writing into the c64 mem.In the particular case of Elite this means switching on/off I/O via $01 on every busy wait cycle. Just the mapping into c64 addresses needs to be changed in my use case to avoid banking. Ideally I'd like to use zero page addresses e.g. $e0 onwards.But that's a detail, probably solvable and specific to elite (where $e0.. is unused)I wonder if my Zephyr port automatically supports the interrupts you've implemented. Stay tuned...Which interrupt vector is to be used on the c64 side to implement an ISR?Good to see that you're still improving the OC! Much appreciated!I may open another discussion to share my ideas (simple things I think).Happy hacking,  pottendo Am 10.09.2023 17:36 schrieb Marcus Comstedt ***@***.***>: Nice! BTW, you may want to check out the new "mailbox" function I just added. I think it could be useful for you because it lets the RISC-V poll a shared area as often as it likes without causing a DMA cycle, so no need for "usleep" if you use it. 😸 ***@***.*** —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[zeldin]

Which interrupt vector is to be used on the c64 side to implement an ISR?

On the C64 side the $FFFE vector is used. (That's the only one there is unless you count the NMI vector, but this is not an NMI. 😄 ) If you have other interrupts enabled as well, you'll need to read the doorbell location to see if there is a message for you (using whatever protocol you devise) and if there is modify the byte so that you don't process the same message a second time.

[pottendo]

...yes, thought so. Otoh, why should one want to mask the OC?! 😉
Need check, how to fiddle that into Elite!
Bye, pottendo

[zeldin]

Well, if you do bank switching you might rely on SEI/CLI to not take interrupts while in the "wrong" bank configuration. In general NMIs are more tricky to handle. But yes, I did skip having a separate mask for the interrupt source similar to $D01A, because it didn't seem necessary.

[pottendo]

Hi,
a simple test works here! thumbs up under Zephyr. Fired interrupts in both directions and handled them in respective ISRs!

However, now I'm again at the point where my version of the bitstream (with FPU) won't fire interrupts in Zephyr. I found that already some months ago, when trying to get the UARTs working interrupt driven. I stopped back then, as I was stuck. UARTS are working but not interrupt driven. Zephyr seems to support that in its litex drivers.
For whatever reason the vexriscv CPU behaves 'better' (or at least differently) wrt. interrupts than the vexriscv_smp CPU.
For the specific case of Elite accelerator this isn't the big issue, as both line and circle functions are using integer algorithms.

Howevr, this is not good in general - I should find the cause for the non-working interrupts in the vexriscv_smp CPU.
Any idea about the root-cause would be much appreciated!
bye, pottendo

[zeldin]

AFAICT the difference is that the vexriscv_smp CPU includes a PLIC, whereas the vexriscv one does not. So probably a PLIC driver is needed in Zephyr to enable and route the interrupts (in a multicore setup this includes deciding on which core handles which interrupts).
https://github.com/riscv/riscv-plic-spec/blob/master/riscv-plic.adoc describes the function of the PLIC.

[zeldin]

So I think you'll need to make sure that your device tree has a section like this:
https://github.com/sifive/riscv-zephyr/blob/87462df3ef18bbf018a4ed66e5105fe13b9da40a/dts/riscv32/microsemi-miv.dtsi#L45
(but with the correct memory addresses and so on for the vexriscv_smp core).

[pottendo]

I do use the sifive PLIC driver since long... and a device tree connects UARTs to the PLIC. The mailbox is (still) hardwired in the code.
And now after several hours, I found the missing piece - my arch-dep code wasn't calling the plic specific IRQ enabler... argh, silly me!
...need more testing, but at least the mailbox ISR is called when C64 writes to the specific location - and also the C64 runs its ISR when the co-proc triggers! This looks good.
If this turns out to be solving the UART interrupts as well, I may revisit the SLIP attempts I did (over UART2) - but that's something for another evening.
bye, pottendo

[pottendo]

Here the device tree section: https://github.com/pottendo/zephyr-pottendo/blob/wip2/dts/riscv/riscv32-FPU-RVCop64.dtsi
and here the main test-program: https://github.com/pottendo/zephyr-pottendo/blob/wip2/samples/oc-mailbox/src/main.cpp
+c64 side: https://github.com/pottendo/c64-userport-driver/blob/master/oc-mailbox.asm

Note that besides the different C64 RAM mapping to 0x0f00000, I have a slight different IRQ mapping: mailbox is 5, uart2=4 - this makes the bitstream Linux compatible.

[pottendo]

Hi,

...and I thought I understand what I'm doing:
Sorry to bother again with my vexriscv_smp CPU: there's a strange behavior with the mailbox shared memory.

I have a simple testprog, which writes the 32 bytes starting at $dec0 on the c64 side and then triggers an interrupt by writing 4 bytes starting at $defc.
With the original RVCop64 (vexriscv) I get this on 0xe0000000 once the ISR is triggered - as one would expect:
0xe0000000: 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f |................|
0xe0000010: 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f |................|

however, on the vexriscv_smp CPU i get this:
0xe0000000: 00 01 02 03 00 01 02 03 08 09 0a 0b 08 09 0a 0b |................|
0xe0000010: 10 11 12 13 10 11 12 13 18 19 1a 1b 18 19 1a 1b |................|

what the hell is going on? Any idea?
thanks, pottendo

[zeldin]

Hm, looks like the lowest bit of the 32-bit address is getting lost - you are reading the lowest 32-bit word correct but then you are getting the same word again, and after that you are back on the right place again. Maybe the SMP core does 64-bit burst reads (i.e. a burst of two 32-bit reads) and that messes up my wishbone timing somehow? Can you check if 0xe0000000 is in a cached or uncached region on vexriscv_smp? (A byte read should not cause 64 bits to be read unless the core is trying to fill up a data cache line...)

[pottendo]

Hmm, where shall Iook?https://github.com/pottendo/RVCop64-pottendo/tree/wip-mailbox/hwAm 13.09.2023 21:23 schrieb Marcus Comstedt ***@***.***>: Hm, looks like the lowest bit of the 32-bit address is getting lost - you are reading the lowest 32-bit word correct but then you are getting the same word again, and after that you are back on the right place again. Maybe the SMP core does 64-bit burst reads (i.e. a burst of two 32-bit reads) and that messes up my wishbone timing somehow? Can you check if 0xe0000000 is in a cached or uncached region on vexriscv_smp? (A byte read should not cause 64 bits to be read unless the core is trying to fill up a data cache line...) —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[zeldin]

Well, if there is no documentation to look at, the source code would be the best place I guess.
In the case of the regular vexriscv core, here is the code that sets up address translation:
https://github.com/litex-hub/pythondata-cpu-vexriscv/blob/master/pythondata_cpu_vexriscv/verilog/src/main/scala/vexriscv/GenCoreDefault.scala#L160
Depending on whether or not it's a Linux version of the core, it either makes addresses starting with 0xb, 0xe and 0xf uncacheable (that is what ioRange signifies), or any address which has the msb set (so everything between 0x80000000 and 0xffffffff).
Presumably the code to generate the vexriscv_smp cores has some similar code; look for the StaticMemoryTranslatorPlugin call.

While it's probably possible to fix the mailbox so that it works with burst reads, there is a point in itself to have it in an uncached region since a cache that hides your new mail is rather contraproductive. 😄

[pottendo]

Found a few spots with StaticMemoryTranslatorPlugin - but not exactly similar with what you've cited.
Tried to move the 0xe0000000 to something else, eg. 0xfe000000 - no difference:
0xfe000000: 00 01 02 03 00 01 02 03 08 09 0a 0b 08 09 0a 0b |................|
0xfe000010: 10 11 12 13 10 11 12 13 18 19 1a 1b 18 19 1a 1b |................|
Anyway thanks for the hints - I'll dig further - but not today.
br, pottendo

ps: why 0xF?..., because I found things like this:

$ grep -1 StaticMemoryTranslatorPlugin $x 2>/dev/null

./pythondata_cpu_vexriscv_smp/verilog/ext/VexRiscv/src/test/scala/vexriscv/TestIndividualFeatures.scala- override def applyOn(config: VexRiscvConfig): Unit = {
./pythondata_cpu_vexriscv_smp/verilog/ext/VexRiscv/src/test/scala/vexriscv/TestIndividualFeatures.scala: config.plugins += new StaticMemoryTranslatorPlugin(
./pythondata_cpu_vexriscv_smp/verilog/ext/VexRiscv/src/test/scala/vexriscv/TestIndividualFeatures.scala- ioRange = _ (31 downto 28) === 0xF

[pottendo]

hi,
I found the following:
around here: https://github.com/zeldin/litex/blob/4959c6c8d8db0f58758f78737a6de80c86bfbdeb/litex/soc/cores/cpu/vexriscv_smp/core.py#L104C1-L105C1

[...]
if(args.with_fpu):
VexRiscvSMP.with_fpu = True
VexRiscvSMP.icache_width = 64
VexRiscvSMP.dcache_width = 64 # Required for F64
[...]

so your assumption - again - was correct and indeed removing --with-fpu from the build makes the vexriscv_smp CPU working with the mailbox. I can't change the dcache to just 32 here, as this breaks the bitstream build-process somehow and I assume the person who has written the comment had a reason for it! So, even if the build would run, the result would be probably broken (wrt. FPU at least).
Also for multicore the code sets 64bit dcache width: https://github.com/zeldin/litex/blob/4959c6c8d8db0f58758f78737a6de80c86bfbdeb/litex/soc/cores/cpu/vexriscv_smp/core.py#L84

Pls, let me know what you think! Can you imagine a workaround/fix on the mailbox side? I wonder, why other RAM or I/O access isn't affected? I/Os (like uarts, etc.) seem to work properly

In the meantime I'll try to fix the build (some 'sbt' issue in my setup).

thanks, pottendo

PS:
the test now shows results as expected:

0xe0000000: 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f |................|
0xe0000010: 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f |................|
0xe0000020: 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f | !"#$%&'()*+,-./|
0xe0000030: 30 31 32 33 34 35 36 37 38 39                   |0123456789      |

PPS:
as a side effect of this change also the read speed / s is at the level of the stock RVCop64 using the vexriscv CPU:

--=============== SoC ==================--
CPU:		VexRiscv SMP-LINUX @ 80MHz
BUS:		WISHBONE 32-bit @ 4GiB
CSR:		32-bit data
ROM:		48KiB
SRAM:		16KiB
MAIN-RAM:	16384KiB 

--========== Initialization ============--
Memtest at 0x40000000 (2.0MiB)...
  Write: 0x40000000-0x40200000 2.0MiB
   Read: 0x40000000-0x40200000 2.0MiB
Memtest OK
Memspeed at 0x40000000 (Sequential, 2.0MiB)...
  Write speed: 21.8MiB/s
   Read speed: 55.4MiB/s

[zeldin]

Right. That does confirm that the behaviour is triggered by the dcache. However, changing the cache line size is not the correct solution, even if it has been allowed by the FPU. Basically there are two ways to go:

• Ideally you'd want the accesses to bypass the cache entirely, since you want to read fresh data from the C64. So that would involve finding an uncached region (i.e. where ioRange evaluates to true) to place the mailbox in, as I mentioned previously. I would assume that the memory mapped CSRs (UART etc) reside in such an uncached region, so maybe try sneaking the mailbox in next to those?
• If the mailbox is left in a cached area, it could still be made to work if I sort out the issue with burst reads. It should be possible to do, but I don't have time to look at it right now. And even if I did this you would need to flush the dcache before reading the mailbox, which is suboptimal.

[pottendo]

thanks,

I've moved the mailbox around e.g. like this:

IO Regions: (1)
io0                 : Origin: 0x80000000, Size: 0x80000000, Mode: RW, Cached: False Linker: False
Bus Regions: (9)
rom                 : Origin: 0x00000000, Size: 0x0000c000, Mode: RW, Cached: True Linker: False
c64                 : Origin: 0x0f000000, Size: 0x00010000, Mode: RW, Cached: True Linker: False
sram                : Origin: 0x10000000, Size: 0x00004000, Mode: RW, Cached: True Linker: False
main_ram            : Origin: 0x40000000, Size: 0x01000000, Mode: RW, Cached: True Linker: False
opensbi             : Origin: 0x40f00000, Size: 0x00080000, Mode: RW, Cached: True Linker: True
mailbox             : Origin: 0x80000000, Size: 0x00000040, Mode: RW, Cached: False Linker: False
csr                 : Origin: 0xf0000000, Size: 0x00010000, Mode: RW, Cached: False Linker: False
clint               : Origin: 0xf0010000, Size: 0x00010000, Mode: RW, Cached: False Linker: False
plic                : Origin: 0xf0c00000, Size: 0x00400000, Mode: RW, Cached: False Linker: False

as you can see everything above 0x80000000 is IO Region.
still get:

0x80000000: 00 01 02 03 00 01 02 03 08 09 0a 0b 08 09 0a 0b |................|
0x80000010: 10 11 12 13 10 11 12 13 18 19 1a 1b 18 19 1a 1b |................|
0x80000020: 20 21 22 23 20 21 22 23 28 29 2a 2b 28 29 2a 2b | !"# !"#()*+()*+|
0x80000030: 30 31 32 33 30 31 32 33 38 39                   |0123012389      |

I've also tried 0xf0020000 - next to the clint, but same result.

So the cache property seems not to be the primary issue, as the mailbox is listed as 'Cached: False'.

[pottendo]

Hi @zeldin,

for now I gave up on the issue with the 64bit dcache for vexriscv_smp and use the stock RVCop64 *) to avoid unnessary differences.
*) still self built, as I need to adjust the phase-shift parameter to make the mem-transfers stable (currently using 17).

Here some observations on the mailbox feature:

• to acknowledge the interrupt on the C64 it needs also write access (e.g. by inc $defb). My ISR looks like this:

.label oc_interrupt = $defb

oc_req:
    lda oc_interrupt    // the oc driver triggers by writing $ff to trigger
    bpl !+              
    inc oc_interrupt    // why needed to ack the interrupt? read should be sufficient
    inc crsync          // this can be polled to syncronize with finish of a co-routine
    inc VIC.BgC         // show something by setting background color
    restore_regs()      // restore registers as needed for proper ISRs
    rti
!:
    jmp STD.IRQ         // process other interrupt, not an orangecart provoked one

according to your doc, it should be sufficien to read $defb, right? without the inc oc_interrupt above, the ISR is called permanently.

As you see above, I've also introduced a busy-wait variable crsync, other code can poll to synchronize with some result delivered. This variable is only modified by the C64 and relies somewhere in RAM (not I/O space).
See below, how this sync looks in the code.

• with this approach I see principle functioning as specified: ISRs (on both sides) are called and data can be exchanged via the shared memory at the specified addresses.
  However, doing this in loops (see code below), even with some artificial delays between triggers, the C64 side crashes very quickly randomly. The symptom looks like that there's an interrupt, e.g. NMI, not properly handled and puts the machine into some reset state. If a bitmap mode was active, the crash puts the video chip back into text mode. Sometimes part of the bootscreen appears (see attached pic) and the machine is halted. I've tried to catch the NMI by providing an ISR at $0318/19 but it's not called.
  The orangecart side works as expected. It receives some interrupts and processes them and triggers the ISR on the C64 to signal that it has finished the routine.
  Even after the crash my driver continues and handles new requests (interrupts) after resetting and relaunch of the testprogram on the C64 side.
  So I wonder what I'm doing wrong here. Do I have to further protect the ISR on the C64 side somehow?
  Any clue?

thanks, pottendo

Code repo:
C64-code
Zephy-driver
Further code:

.label oc_shm       = $dec0
.label oc_interrupt = $defb
.label oc_triggeroc = $deff

.macro wait4cr()
{
!:
    inc VIC.BoC         // give room by accessing I/O and see activity
    inc VIC.BoC
    inc VIC.BoC
    lda crsync          // poll the sync variable
    beq !-
}

.macro trigger_oc()
{
    poke8_(crsync, 0)
    poke8_(oc_triggeroc, $ff)   // trigger OCs ISR
    wait4cr()   // if coproc is still working, we block here, busy waiting
}

.macro poke8_(addr, val) {
    lda #val
    sta addr
}

[...]

do_test:
    poke8_(oc_shm + 1, $fe) // $fe at oc_shm + 1 requests the test coroutine
    poke8_(P.zpp1 , 0)      // P.zzp1 is a zeropage tmp address
    ldy #$20                // loop 32x

!next:
    ldx #$3a                // fill the entire mailbox shm
!:  
    txa
    clc 
    adc P.zpp1              // add something to alter the shm over the iterations
    sta oc_shm, x           // put in the oc_shm
    dex
    bne !-
    stx oc_shm              // also fill byte 0 (always with #0)
    poke8_(oc_shm + 1, $fe) // request cr-test, as oc_shm + 1 was overritten
    trigger_oc()            // trigger oc by issuing an interrupt and busy wait for finish

    inc P.zpp1              // change for the next iteration
    dey
    bne !next-

    rts

I use WIC64 kernal and a WIC64 - this allows very quick compile/assemble/test cycles.

[zeldin]

The mailbox interrupt is cleared by just reading $DEFB. However, if you write your interrupt service routine just like that, and leave out the modification of $DEFB, then what is going to happen is that when the next other interrupt (timer, raster, whatever) happens then you will still read the same value from $DEFB, and exit the routine through RTI without proceeding to STD.IRQ. Thus this other interrupt is not processed and cleared, and so you will get immediately back into the ISR. Therefore it is recommended that you change the value so that you yourself know that the messages has been processed.

I'm working on adding an interrupt controller function to the C64 side, which will allow you to explicitly check whether or not the mailbox IRQ is active, and also to select whether to route the mailbox interrupt function to IRQ or NMI, or disable it completely.

[zeldin]

By the way, if you are just going to set a variable crsync and then poll on that, then why use the interrupt function at all? Just have the RISC-V side write to a non-interrupting mailbox location and poll that instead... A typical setup would be to have the RISC-V side clear $DEFF (i.e. 0x4000003f) when it finished with the job started by a write to that location, and poll $DEFF from the C64 side.

[pottendo]

Hi,
Thanks for the quick response.
The polling ist just an intermediate step to ensure that one request ist finished before anything on the main program happens.
Later I'd like to only check the flag when really necessary. So things can go really in parallel.
But this can be.done also just looking at the address, as you propose. No need for the ISR.

Anyway, I assume with your changes coming, the handling in the ISR needs to change anyway. I'll redo my ISR tests then. Let's see if my crashes are then reproducible...
...do you have an idea, why my code currently is broken?

Looking forward to your commits,
pottendo

Ps: I've made the mailbox configurable through the device tree spec in Zephyr. Cool, isn't it? 😉

[zeldin]

Well, yes, with the new changes you need to explicitly enable the interrupt if you want to use it (i.e. a single write of 1 to $DE41 or $DE42, depending on whether you want IRQ or NMI, on startup). Other than that you can keep everything as before if you like. Or you can use the new $DE40 register to verify and/or clear the mailbox interrupt, if you want to (clearing the interrupt by reading $DEFB will still work).

[pottendo]

hi,
after a few days I continued with further experiments.

When I remove the access to the mailbox and just paint into the memory from both sides, all runs stable - even into the same memory region. So the principle RAM access from the RVCop64 works without making the C64 unhappy. Even if the access is likely to collide on the very same memory address.

I've changed the 'protocoll' as suggested to poll $deff on the C64 side and from the co-proc side now I set to \0 in case an interrupt has to be confirmed.
With this I still get the frequent C64 crashes as described above.
As soon as I trigger interrupts from the C64 by writing to $deff things get unstable - only on the C64 side.

It further seems that if the RVCop side does a lengthy operation, involving heavy memory access (e.g. drawing a filled circle) and an interrupt asynchronously comes in, this slows down this lengthy operation substantially.
See the video here: https://photos.app.goo.gl/XZCdhjH6WuSDPwYQ7
3 times the circles are drawn and at the 4'th time (at ~0;07s) I generate a single interrupt by pressing a key. Once the circles have finished drawing, the other function triggered by the interrupt executes on the RVCop and draws some pattern. Then the circles continue.
I can't rule fully out that the interrupt handling has some weird behavior on Zephyr - However I can confirm that only one interrupt arrives at the ISR.

Anyway - I'll have a look once you've finished your work on the mailbox.

But maybe you have an idea, what could cause the problem? It seems to be a race condition where memory access and interrupt handling on the OC is involved.

bye, pottendo

[pottendo]

hi,
one more observation:
I've reduced the example to the minimum:

• RVCop64 side: just a handler for the interrupt, acknowledging by reading 32 bit from 0xe000003c
• C64 side: just trigger by writing 0xff to $deff
  triggers the c64 crash already in my setup very quickly. See video here: Testrun
  The program sets some Hires GFX (at $4000) mode and fills it with some patterns including the color rams ($7c00 and $d800) and the triggers the mailbox interrupt by writing $ff to $deff on hitting of a key. The first time it immediately crashes the second time (after loading) it crashed at the second hit.

Find attached the testprogram for the C64 - maybe you have some raw-testprogram on the RVCop64, handling the interrupt somehow.
My RVCop64 SW is a Zephy program.
Here the C64 side.

bye, pottendo
Testprog C64 side:
oc-mailbox-test.zip

[pottendo]

arghh - the longpress works!
BUT: as one keeps pressing it enters the 'bitstream download mode'!
not exactly what you want! ;-)

[zeldin]

Yes, the InterruptController stuff should be unrelated as it's for the interrupts on the C64 side.
As for the longpress, entering download mode was exactly what I wanted. Maybe it was not what you wanted... 😉
If you just want to restart the RiscV processor, you can simply do RVSYS$70000000. If that does not work, another option is to enter the download mode with longpress and then send the USB command to start the main bitstream. You can use the shell command

dfu-util -a 0 -e

to do this. This will work even if the button is still being pressed (but you'll need to release it within one second to avoid going back to the loader again of course).

[pottendo]

OK,
reproduced a working bitstream - both on vexriscv and vexriscv_smp (+FPU and some more RiscV fancy stuff).
I found, that it shows my problem when configuring for 80Mhz!
Both CPUs fail @80Mhz and work @64mhz.
For completeness: the 64bit dcache-width (vexriscv_smp cpu) still shows the problem that every second 4 bytes are somehow messed up.

[zeldin]

You will probably see a warning from nextpnr saying that it failed to close timing at 80 MHz. It's normal to get one warning due to the DDR I/Os used for HyperRAM, but if you get 2 or more you should check what it is.

[pottendo]

yes,
vexriscv: @80Mhz
Warning: Use of IDDRXN and ODDRXN primitives on the same pin is unofficial and unsupported!
Warning: Max frequency for clock '$glbnet$platform_ecp5pll0_clkout0': 70.25 MHz (FAIL at 80.00 MHz)
2 warnings, 0 errors

vexriscv: @64mhz
Warning: Use of IDDRXN and ODDRXN primitives on the same pin is unofficial and unsupported!
1 warning, 0 errors

vexriscv_smp: @64mhz
Warning: Use of IDDRXN and ODDRXN primitives on the same pin is unofficial and unsupported!
Warning: Max frequency for clock '$glbnet$platform_ecp5pll0_clkout0': 55.13 MHz (FAIL at 64.00 MHz)
2 warnings, 0 errors

Besides the mailbox, I don't see any problem with 80Mhz.

[zeldin]

Regarding your issues with the mailbox on vexriscv_smp, I can't really test that since vexriscv_smp seems to requires sbt 1.x, and I fail to build sbt 1.7 (I do have 0.13.18 installed) on ppc64 because of some JNA nonsense...

[pottendo]

argh - this sbt-hell gave me headaches as well.
Note that I needed to patch some build.sbt - look here.

I could try to send you the generated class files, built via sbt. Not sure if this works, though.
AFAIK, the sbt build run generates class files here:
hw/deps/pythondata-cpu-vexriscv_smp/pythondata_cpu_vexriscv_smp/verilog
and looks like this:

drwxrwxr-x 3 pottendo pottendo    4096 Oct  2 19:16 .
drwxrwxr-x 4 pottendo pottendo    4096 Sep 29 17:25 ..
drwxrwxr-x 4 pottendo pottendo    4096 Sep 29 17:18 ext
-rw-rw-r-- 1 pottendo pottendo    1356 Sep 29 17:18 Ram_1w_1rs_Efinix.v
-rw-rw-r-- 1 pottendo pottendo    1445 Sep 29 17:18 Ram_1w_1rs_Generic.v
-rw-rw-r-- 1 pottendo pottendo    1399 Sep 29 17:18 Ram_1w_1rs_Intel.v
-rw-rw-r-- 1 pottendo pottendo    2825 Sep 29 17:18 README.md
-rw-rw-r-- 1 pottendo pottendo 1343861 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw32Is4096Iy1_Dw32Ds4096Dy1_ITs4DTs4_Ldw128_Cdma_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1058283 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw32Is4096Iy1_Dw32Ds4096Dy1_ITs4DTs4_Ldw128_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1338157 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw32Is4096Iy1_Dw32Ds4096Dy1_ITs4DTs4_Ldw16_Cdma_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1052737 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw32Is4096Iy1_Dw32Ds4096Dy1_ITs4DTs4_Ldw16_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1310914 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw32Is4096Iy1_Dw32Ds4096Dy1_ITs4DTs4_Ldw32_Cdma_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1026643 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw32Is4096Iy1_Dw32Ds4096Dy1_ITs4DTs4_Ldw32_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1340109 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw32Is4096Iy1_Dw32Ds4096Dy1_ITs4DTs4_Ldw64_Cdma_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1054531 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw32Is4096Iy1_Dw32Ds4096Dy1_ITs4DTs4_Ldw64_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1349562 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw32Is8192Iy2_Dw32Ds8192Dy2_ITs4DTs4_Ldw16_Cdma_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1063299 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw32Is8192Iy2_Dw32Ds8192Dy2_ITs4DTs4_Ldw16_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1322319 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw32Is8192Iy2_Dw32Ds8192Dy2_ITs4DTs4_Ldw32_Cdma_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1037205 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw32Is8192Iy2_Dw32Ds8192Dy2_ITs4DTs4_Ldw32_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1584973 Sep 30 11:30 VexRiscvLitexSmpCluster_Cc1_Iw64Is4096Iy1_Dw64Ds4096Dy1_ITs4DTs4_Ood_Wm_Fpu4_Rvc.v
-rw-rw-r-- 1 pottendo pottendo  919505 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw64Is4096Iy1_Dw64Ds4096Dy1_ITs4DTs4_Ood_Wm.v
-rw-rw-r-- 1 pottendo pottendo 1589611 Sep 30 16:31 VexRiscvLitexSmpCluster_Cc1_Iw64Is4096Iy1_Dw64Ds8192Dy2_ITs4DTs4_Ood_Wm_Fpu4_Rvc.v
-rw-rw-r-- 1 pottendo pottendo 1410635 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw64Is8192Iy2_Dw64Ds8192Dy2_ITs4DTs4_Ldw128_Cdma_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1103210 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw64Is8192Iy2_Dw64Ds8192Dy2_ITs4DTs4_Ldw128_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1381365 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw64Is8192Iy2_Dw64Ds8192Dy2_ITs4DTs4_Ldw64_Cdma_Ood.v
-rw-rw-r-- 1 pottendo pottendo 1075247 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc1_Iw64Is8192Iy2_Dw64Ds8192Dy2_ITs4DTs4_Ldw64_Ood.v
-rw-rw-r-- 1 pottendo pottendo  962281 Oct  2 19:16 VexRiscvLitexSmpCluster_Cc1_Iw64Is8192Iy2_Dw64Ds8192Dy2_ITs4DTs4_Ood_Wm_Rvc.v
-rw-rw-r-- 1 pottendo pottendo 2150313 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc2_Iw64Is8192Iy2_Dw64Ds8192Dy2_ITs4DTs4_Ldw128_Cdma_Ood.v
-rw-rw-r-- 1 pottendo pottendo 3587078 Sep 29 17:18 VexRiscvLitexSmpCluster_Cc4_Iw64Is8192Iy2_Dw64Ds8192Dy2_ITs4DTs4_Ldw128_Cdma_Ood.v

as you can see, there just one file from today, and just one was generated by
python3 bitstream.py --platform=orangecart --uart=usb_acm --cpu=vexriscv_smp --sys-clk-freq=64e6 --cpu-count=1 --dcache-size=8192 --dcache-ways=2 --dcache-width=64 --icache-width=64 --icache-size=8192 --icache-ways=2 --with-rvc 2>&1 |tee make.log

so maybe it's even sufficient to put VexRiscvLitexSmpCluster_Cc1_Iw64Is8192Iy2_Dw64Ds8192Dy2_ITs4DTs4_Ood_Wm_Rvc.v into the directory and run with the commandline above.
(make sure you've cloned my tree (branch wip-mailbox).

but, again, this might be not working just to place the one file.

Thanks for trying and bothering with my stuff.
I'll play with the mailbox interrupt controller tonight, so I'll let you know how it works - still wanting to get 'elite @steroids' running! ;-)
bye, pottendo

[pottendo]

oops forgot to push back the merge - so now the master branch contains all the recent updates, merged with your tree and changes to support vexriscv_smp.

[pottendo]

I'm working on adding an interrupt controller function to the C64 side, which will allow you to explicitly check whether or not the mailbox IRQ is active, and also to select whether to route the mailbox interrupt function to IRQ or NMI, or disable it completely.

Hi,
mailbox interrupt controller works as specified - RiscV -> C64.
So far I write to 0xe000003b and trigger both an IRQ and NMI - I've connected ISRs respectively; however if I attached both ISRs, the NMI seems to 'win' (which is intuitively correct) and the IRQ is lost.
Is there a way to distinguish on the RiscV side to trigger NMI or IRQs seperately?
Simplest thing would be to use 0xe000003b for IRQ and some other address for NMI?
I think this would be handy...
pottendo

[zeldin]

Well, presumably the IRQ is "lost" because you acknowledge the interrupt source by reading $DEFB before returning from the NMI with RTI (and thereby clearing the I flag in the status register). 😄 Otherwise it would have triggered.
There is only one "event" that the RiscV side can trigger. How to handle that event (IRQ, NMI, polling $DE40) is up to the C64 side. Of course the RiscV side could access $DE41, $DE42 to mess with that, but that doesn't seem inherently useful...