- Fixed the repo state. Remove a lot of old testing garbage files, fi…

…xed the compile-all.sh runstream. Now if you compile on your own it will closer reflect the release, artifacts-wise, and not be a dev-mode disaster (solo pushing to a private repo for a long period of time creates a lot of garbo).

- Dockerfile created - first Docker image release w v0.1.1, not perfect but seems stable and fully functional
- Fixed issue where DuckDB files did not get proper metadata for UI browsing
    - Cache DB is now Duck by default (configurable in the future)
- Materialize to Cache DB now has options for persitent or memory (cache.db or cache.db.memory) - both Duck ATM
- Compressed the shipping assets/openai-images examples to JPGs (rather than 3MB PNGs, kek)
- In addition to defs/config.edn you can have a map in defs/secrets.edn for API keys and such - can be accessed in any part of the system as a reactive parameter with the :settings prefix. So :settings/openai-api-key will resolve to that key on execution. etc.
    - changed the example flow "openai-image-gen-ladder" to use :settings/openai-api-key if it exists.
- audio transcription also uses OpenAI at the moment, so the icon will only appear if a valid key is detected (key gets checked on boot)
- added a config.edn option to disable the sub/unsub reactor messages. They are useful for debugging pub/sub issues, but can get annoying. :hide-sub-and-unsub-alerts?
    - reminder that you can disable ALL pop-up notifications via a toggle on the lower right hand corner of the canvas
- added rotating-theme-blank screen, which will cycle randomly through all the images found in assets/openai-images and change the theme colors to match. Not for a prod dashboard, obvs, but for random Data eng boards where im just monitoring data, it's nice to have some vibes flowing.
- Fixed a double-query bug
- Added an optional pop-up notification about SQL query time execution. Off by defaut, can be changed in the config.edn
    - :show-query-times? true
- Added more OAI images to show off dynamic themes better. Not your cup of tea? Cool beans. Replace them with Smurfs or Lumberjacks or Kittens - that's the beauty of a fully customizable and hackable platform! Will have a tutorial video in a few days on how to automate image gen with a flow. Express yourself!

README.md

@@ -265,6 +265,7 @@ Besides the built-in runners of SQL queries and Views (UI) - the combination of
 ## Caveats
 
 - Use Engelbart's beautiful invention please 🐁
+- Reccommend JDK 17+
 - Smattering of various UI jank; being worked on
 - Limited SQL DB support ATM
 	Tested with SQLite, Postgres, MySQL, SQL Server, Vertica, Clickhouse, DuckDB

backend/defs/config.edn

@@ -8,8 +8,10 @@
 
 {:debug-level 1 ;; for console output. somewhat deprecated.
  :hide-sub-and-unsub-alerts? false
+ :show-query-times? false 
  :default "./screens/frost-terminal2.edn"
  ;; ^^ if no screen given in URL, load this one. the "start page" essentially.
+ ;;    if this screen can't be read or is nil, you will get a basic blank screen with a default theme
  :harvest-on-boot? true ;;false
  ;; ^^ do a full database sniff for each connection on boot? (generally not needed)
  :modes {:pre       [:pre
@@ -132,7 +134,39 @@
                                     Since this system is EDN based, all code has to be wrapped in a (do ..) form in order to be evaluated as a single expression. If a user-provided do block already exists, just use it, do not double wrap it - a single parent (do ...) form is all that is needed.
                                     Being saved in EDN also means that it cannot use Clojure artifacts that are invalid EDN - so, do not use the shorthand for anon functions (#(this %)), or shorthand atom derefs (@atom), or shortcode sets (#{}) always use the full for version for these things - instead w/o the special characters (deref, set, (fn[x]...))."
                      :icon "images/clojure-alt.svg"
-                     :kits {:export-to-csv {:when-fn (fn [client-panels client-data] ;; return a vector of [:block-id :runner-type :view-id] tuple for things that are "eligible"
+                     :kits {;; unfinished: might work better as a flow block... WIP
+                            ;; :openai-data-clean {:when-fn (fn [client-panels client-data] ;; return a vector of [:block-id :runner-type :view-id] tuple for things that are "eligible"
+                            ;;                                (vec (apply concat (for [[k v] client-panels ;; ^^ runs on the local repl, not the host
+                            ;;                                                         :when (get v :queries)] ;; only things eligble are query results
+                            ;;                                                     (for  [kk (keys (get v :queries))]
+                            ;;                                                       [k :queries kk])))))
+                            ;;                     :output :kit-map
+                            ;;                     :icon-location :query
+                            ;;                     :tooltip "Experimental! Use OpenAI's API to clean data. Use as little data as possible to save tokens."
+                            ;;                     :icon "ri-openai-fill"
+                            ;;                     :kit-expr (do
+                            ;;                                 (require '[cognitect.transit :as transit]
+                            ;;                                          '[clojure.data.csv :as csv]
+                            ;;                                          '[rvbbit-backend.external :as ext]
+                            ;;                                          '[rvbbit-backend.assistant :as ass]
+                            ;;                                          '[rvbbit-backend.util :as ut]
+                            ;;                                          '[clojure.string :as cstr]
+                            ;;                                          '[clojure.java.io :as io])
+                            ;;                                 (ass/oai-call {:url "https://api.openai.com/v1/chat/completions"
+                            ;;                                                :method :post
+                            ;;                                                :body {:model "gpt-4o"
+                            ;;                                                       :messages [{:role "system"
+                            ;;                                                                   :content (str "You are a data cleaner. You are given a dataset and instructions, 
+                            ;;                                                                                      you will not respond to anything, you will just clean the data and return it as valid Clojure EDN. 
+                            ;;                                                                                           Do NOT send backticks or language identifiers, JUST the EDN structure please. 
+                            ;;                                                                                      The instructions are: 'Fix the YEAR col - make a guess if you have to with the information given.
+                            ;;                                                                                      If a range is given, pick something in the middle - just make a reasonable assumption as to what
+                            ;;                                                                                      the user intended. If you CANNOT decide or there is not enough information, 
+                            ;;                                                                                      use -1 as the answer for that column.'"
+                            ;;                                                                                 "here is the data: " (pr-str test-data))}]}}
+                            ;;                                               :settings/openai-api-key
+                            ;;                                               :settings/openai-org-id))}
+                            :export-to-csv {:when-fn (fn [client-panels client-data] ;; return a vector of [:block-id :runner-type :view-id] tuple for things that are "eligible"
                                                        (vec (apply concat (for [[k v] client-panels ;; ^^ runs on the local repl, not the host
                                                                                 :when (get v :queries)] ;; only things eligble are query results
                                                                             (for  [kk (keys (get v :queries))]

backend/defs/solvers.edn

@@ -4,6 +4,16 @@
                         :data   {:flowmap "node-color-thief-js-webp"
                                  :opts    {:overrides {:image-path :solver/wallpaper-of-the-moment}}
                                  :return  nil}}
+ :materialize-offenses-drag-4 {:signal    :signal/daily-at-7am
+                               :type      :sql
+                               :snapshot? false
+                               :data      {:select        [:DAY_OF_WEEK :DISTRICT :HOUR :INCIDENT_NUMBER :Lat :Location
+                                                           :Long :MONTH :OCCURRED_ON_DATE :OFFENSE_CODE
+                                                           :OFFENSE_CODE_GROUP :OFFENSE_DESCRIPTION :REPORTING_AREA
+                                                           :SHOOTING :STREET :UCR_PART :YEAR]
+                                           :from          [[:offenses :dd96]]
+                                           :page          -4
+                                           :connection-id "boston-crime"}}
  :all-large-webp-files {:signal false
                         :type   :clojure
                         :cache? true
@@ -32,6 +42,31 @@
                                                                   default-field-attributes
                                                                   default-derived-fields
                                                                   default-viz-shapes)))}
+ :materialize-cached-offenses-drag-43-1
+   {:signal    :signal/daily-at-7am
+    :type      :sql
+    :snapshot? false
+    :data      {:select        [:DAY_OF_WEEK :DISTRICT :HOUR :INCIDENT_NUMBER :Lat :Location :Long :MONTH
+                                :OCCURRED_ON_DATE :OFFENSE_CODE :OFFENSE_CODE_GROUP :OFFENSE_DESCRIPTION :REPORTING_AREA
+                                :SHOOTING :STREET :UCR_PART :YEAR]
+                :from          [[:offenses_drag_43 :zz175]]
+                :page          -4
+                :connection-id "cache.db.memory"}}
+ :materialize-DISTRICT-drag-18
+   {:signal :signal/daily-at-7am
+    :type :sql
+    :snapshot? false
+    :data
+      {:select [:DISTRICT [[:count 1] :rowcnt] :OFFENSE_CODE :STREET]
+       :from
+         [[{:select [:DAY_OF_WEEK :DISTRICT :HOUR :INCIDENT_NUMBER :Lat :Location :Long :MONTH :OCCURRED_ON_DATE
+                     :OFFENSE_CODE :OFFENSE_CODE_GROUP :OFFENSE_DESCRIPTION :REPORTING_AREA :SHOOTING :STREET :UCR_PART
+                     :YEAR]
+            :from   [[:offenses :ww131]]} :kk71]]
+       :group-by [:DISTRICT :OFFENSE_CODE :STREET]
+       :order-by [[:rowcnt :desc]]
+       :page -4
+       :connection-id "boston-crime"}}
  :random-mp3
    {:signal :signal/changed-wallpaper?
     :type :clojure
@@ -93,6 +128,26 @@
        :connection-id   "system-db"
        :select          [[[[:sum :mem_used]] :mem_used] [[:sum :recent_messages_per_second] :messages_per_second] :ts]
        :post-process-fn (fn [x] (vec (reverse (for [row x] (assoc row :mb (int (/ (get row :mem_used 0) 1048576)))))))}}
+ :materialize-offenses-drag-43 {:signal    :signal/daily-at-7am
+                                :type      :sql
+                                :snapshot? false
+                                :data      {:select        [:DAY_OF_WEEK :DISTRICT :HOUR :INCIDENT_NUMBER :Lat :Location
+                                                            :Long :MONTH :OCCURRED_ON_DATE :OFFENSE_CODE
+                                                            :OFFENSE_CODE_GROUP :OFFENSE_DESCRIPTION :REPORTING_AREA
+                                                            :SHOOTING :STREET :UCR_PART :YEAR]
+                                            :from          [[:offenses :aa855]]
+                                            :page          -4
+                                            :connection-id "boston-crime"}}
+ :materialize-offenses-drag-20 {:signal    :signal/daily-at-7am
+                                :type      :sql
+                                :snapshot? false
+                                :data      {:select        [:DAY_OF_WEEK :DISTRICT :HOUR :INCIDENT_NUMBER :Lat :Location
+                                                            :Long :MONTH :OCCURRED_ON_DATE :OFFENSE_CODE
+                                                            :OFFENSE_CODE_GROUP :OFFENSE_DESCRIPTION :REPORTING_AREA
+                                                            :SHOOTING :STREET :UCR_PART :YEAR]
+                                            :from          [[:offenses :zz384]]
+                                            :page          -4
+                                            :connection-id "boston-crime"}}
  :static-data {:signal false :type :clojure :data [(apply + [1 2 3])]}
  :refresh-boston-crime {:signal :signal/daily-at-9am
                         :type   :clojure
@@ -113,6 +168,16 @@
                   :type   :clojure
                   :cache? true
                   :data   (true? (clojure.string/ends-with? :solver/wallpaper-of-the-moment ".png"))}
+ :materialize-offenses-drag-24 {:signal    :signal/daily-at-7am
+                                :type      :sql
+                                :snapshot? false
+                                :data      {:select        [:DAY_OF_WEEK :DISTRICT :HOUR :INCIDENT_NUMBER :Lat :Location
+                                                            :Long :MONTH :OCCURRED_ON_DATE :OFFENSE_CODE
+                                                            :OFFENSE_CODE_GROUP :OFFENSE_DESCRIPTION :REPORTING_AREA
+                                                            :SHOOTING :STREET :UCR_PART :YEAR]
+                                            :from          [[:offenses :ww131]]
+                                            :page          -4
+                                            :connection-id "boston-crime"}}
  :wallpaper-of-the-moment
    {:signal  :signal/every-5-minutes
     :type    :clojure
@@ -141,4 +206,18 @@
                                                                    default-field-attributes
                                                                    default-derived-fields
                                                                    default-viz-shapes)))}
+ :materialize-Lat-drag-22
+   {:signal :signal/daily-at-7am
+    :type :sql
+    :snapshot? false
+    :data
+      {:select [[[:sum :Lat] :Lat_sum] :OFFENSE_CODE_GROUP]
+       :from
+         [[{:select [:DAY_OF_WEEK :DISTRICT :HOUR :INCIDENT_NUMBER :Lat :Location :Long :MONTH :OCCURRED_ON_DATE
+                     :OFFENSE_CODE :OFFENSE_CODE_GROUP :OFFENSE_DESCRIPTION :REPORTING_AREA :SHOOTING :STREET :UCR_PART
+                     :YEAR]
+            :from   [[:offenses :ww131]]} :ee900]]
+       :group-by [:OFFENSE_CODE_GROUP]
+       :page -4
+       :connection-id "boston-crime"}}
  :dazzling-gnu-solver {:signal :signal/hourly? :type :clojure :cache? true :data (+ 1 2 3 90 (rand-int 124345))}}