Major update Live AI Assistant

[fbgallet]

Note that source repo has change from 'https://github.com/fbgallet/roam-extension-speech-to-roam.git' to 'https://github.com/fbgallet/roam-extension-live-ai-assistant.git'

This is a big update, with a lot of new features and refactoring (see beginning of the readme for main new features)

All functions that directly use the Roam API are in principle gathered here: https://github.com/fbgallet/roam-extension-live-ai-assistant/blob/main/src/utils/roamAPI.js

There are two "sensitive" features:

• the Smart Search Agent uses a fairly complex set of queries via .q API. In my tests it works well, but I do not have a very large graph. Currently, it is not optimized, and I fear it could be slow or even freeze for certain queries in large graphs. I have clearly indicated in the documentation that this is an experimental feature.
• the Outliner Agent plans and executes modifications to a set of blocks using the API. To ensure it does not accidentally alter data, I included a test of the blocks it can act on, which are strictly limited to a tree structure that the user must have chosen themselves (here in the code). I need to add a clear warning in the documentation, knowing that they might lose some data located there (but there is a button to go back if necessary).

I've made a lot of tests and will continue to test it. It's in principle ready for production but need some small changes in the readme.md.

[github-actions]

Here’s your link to the diff:

Changed fbgallet/roam-extension-speech-to-roam 8d47567 → 6cd5e16 (PR-shorthand: fbgallet+roam-extension-speech-to-roam+992)

[github-actions]

Here’s your link to the diff:

Changed fbgallet/roam-extension-speech-to-roam 8d47567 → 6cd5e16 (PR-shorthand: fbgallet+roam-extension-speech-to-roam+992)

[github-actions]

Here’s your link to the diff:

Changed fbgallet/roam-extension-speech-to-roam 8d47567 → 61a61dc (PR-shorthand: fbgallet+roam-extension-speech-to-roam+992)

[github-actions]

Here’s your link to the diff:

Changed fbgallet/roam-extension-speech-to-roam 8d47567 → 61a61dc (PR-shorthand: fbgallet+roam-extension-speech-to-roam+992)

[github-actions]

Here’s your link to the diff:

Changed fbgallet/roam-extension-speech-to-roam 8d47567 → 00b14d8 (PR-shorthand: fbgallet+roam-extension-speech-to-roam+992)

[panterarocks49]

fyi I just fixed query so this works now {{query: {search:test}}} (as well as the single search with OR/AND you reported on slack)

[panterarocks49]

Clicking on the lightning icon for me causes this error, reloading fixed it. might be because I didn't have a key when I first loaded

[panterarocks49]

This is very cool!

The natural language roam query works very well for me, I actually built this awhile ago and never shipped it but yours works much better than mine did.

The Smart Search Agent does crash the page with OOM for me sometimes (not all the time) on a large graph (the graph takes up about 1GB of RAM normally and spikes to 4GB when running smart search). It's also pretty slow.

Ask my graph is also slow but haven't noticed it crash my graph yet.

The extension is a little too large for me to fully understand it but I did notice in some places you use descendant rules These can be really really slow compared to doing a recursive pull. If smart search uses this you could try replacing it with pulls if possible. You can also use :block/parents to search children / parents in datalog queries

I also wonder if you could replace some of the queries you are doing for a Roam Query, I could expose a frontend API endpoint to execute Roam Queries, and I have been thinking about adding regex search to this as well.

[github-actions]

Here’s your link to the diff:

Changed fbgallet/roam-extension-speech-to-roam 8d47567 → 00b14d8 (PR-shorthand: fbgallet+roam-extension-speech-to-roam+992)

[github-actions]

Here’s your link to the diff:

Changed fbgallet/roam-extension-speech-to-roam 8d47567 → 204bee5 (PR-shorthand: fbgallet+roam-extension-speech-to-roam+992)

[fbgallet]

I've just fixed the small issue when no API key.

Yes, Smart Search Agent (and Ask to my graph... since, it's using the same search engine) rely on recursive descendant rules (if search is not limited to 2 children levels or less).
I don't have much experience with pull, but I'll try to use it to optimize queries. While I find a working solution, I suggest limiting the search to a maximum of two levels. If this makes it faster, we can publish a first version with this limitation, and I'll work on an update in the coming days.

[fbgallet]

Frontend API for Roam queries would also be a great solution

[panterarocks49]

I think it's fine to publish even if it crashes someone's graph, just making suggestions for the future. In that respect I'd actually be more worried that the datomic one would generate a query which crashes the graph because reloading wouldn't fix that, but we will figure something out on our end for that

If you think roam queries could replace it then hold off on optimizing for pull, I can work on that next week

[panterarocks49]

Is does o3-mini require special API access?

[fbgallet]

Yes, i think that o3-mini require to be minimum tier 3 currently. I should add something in readme about that.

[panterarocks49]

Is there anyway I can use other openAI models like o1 preview or mini? I think I'm on usage tier 2 which would allow for that

[fbgallet]

It should be possible for o1-mini and o1-preview since it's tier 1+ . I can't test because i'm tier 3.
But o1 and o3 should also be available for all users through OpenRouter

[panterarocks49]

There must be some caching issue, I added more money to my openai account and now I'm on tier 3 but it's still not working

Should this add it to the list of default models?

I tried adding there but can't figure out how to use it

[github-actions]

Here’s your link to the diff:

Changed fbgallet/roam-extension-speech-to-roam 8d47567 → 8358dda (PR-shorthand: fbgallet+roam-extension-speech-to-roam+992)

[fbgallet]

I've pushed a version limited to 2 children levels, without recursive rule, and also removed sibling search temporarily. I would prefer to first test a version that seems more stable.
Regarding the API for queries, after reflection, even if it would be useful, it cannot replace certain queries made with Smart Search Agent (particularly concerning the directionality of the hierarchical relationship or the number of child levels).

o3-mini is working for me, i suppose that there is some delay after accessing to a new tier.

[panterarocks49]

retested smart search and none of the queries which crashed my graph crash it anymore

[fbgallet]

On my side, it's ready for publication, I've still some additionnal doc to write, but not in Readme.md

[panterarocks49]

alright this seems good to me to merge now