Log linearized curve overlays #490
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Computes the maxmin values for each underlying plot's in-view range as well as the max up/down swing (in percentage terms) from the plot with most dispersion and returns a all these values plus a `dict` of plots to their ranges as part of output.
It's kind of hard to understand with the C++ fan-out to multiple views (imo a cluster-f#$*&) and seems honestly just plain faster to loop (in python) through all the linked view handlers XD Core adjustments: - make the panning and wheel-scroll handlers just call `.maybe_downsample_graphics()` directly; drop all signal emissions. - make `.maybe_downsample_graphics()` loop through all vizs per subchart and use the new pipeline-style call sequence of: - `Viz.update_graphics() -> <read_slc>: tuple` - `Viz.maxmin(i_read_range=<read_slc>) -> yrange: tuple` - `Viz.plot.vb._set_yrange(yrange=yrange)` which inlines all the necessary calls in the most efficient way whilst leveraging `.maxmin()` caching and ymxmn-from-m4-during-render to boot. - drop registering `._set_yrange()` for handling `.sigRangeChangedManually`.
We were hacking this before using the whole `ChartView._maxmin()` setting stuff since in some cases you might want similarly ranged paths on the same view, but of course you need to max/min them together.. This adds that group sorting by using a table of `dict[PlotItem, tuple[float, float]` and taking the abs highest/lowest value for each plot in the viz interaction update loop. Also removes the now commented signal registry calls and thus `._yranger`, drops the `set_range: bool` from `._set_yrange` and adds and extra `.maybe_downsample_graphics()` to the mouse wheel handler to avoid a weird slow debounce where ds-ing is delayed until a further interaction.
On overlaid ohlc vizs we compute the largest max/min spread and apply that maxmimum "up and down swing" proportion to each `Viz`'s viewbox in the group. We obviously still need to clip to the shortest x-range so that it doesn't look exactly the same as before XD
In the dispersion swing calcs, use the series median from the in-view data to determine swing proportions to apply on each "minor curve" (series with lesser dispersion the one with the greatest). Track the major `Viz` as before by max dispersion. Apply the dispersion swing proportions to each minor curve-series in a third loop/pass of all overlay groups: this ensures all overlays are dispersion normalized in their ranges but, minor curves are currently (vertically) centered (vs. the major) via their medians. There is a ton of commented code from attempts to try and vertically align minor curves to the major via the "first datum" in-view/available. This still needs work and we may want to offer it as optional. Also adds logic to allow skipping margin adjustments in `._set_yrange()` if you pass `range_margin=None`.
In very close manner to the original (gut instinct) attempt, this properly (y-axis-vertically) aligns and scales overlaid curves according to what we are calling a "log-linearized y-range multi-plot" B) The basic idea is that a simple returns measure (eg. `R = (p1 - p0) / p0`) applied to all curves gives a constant output `R` no matter the price co-domain in use and thus gives a constant returns over all assets in view styled scaling; a intuitive visual of returns correlation. The reference point is for now the left-most point in view (or highest common index available to all curves), though we can make this a parameter based on user needs. A slew of debug `print()`s are left in for now until we iron out the remaining edge cases to do with re-scaling a major (dispersion) curve based on a minor now requiring a larger log-linear y-range from that previous major' range.
When there are `N`-curves we need to consider the smallest x-data-support subset when figuring out for each major-minor pair such that the "shorter" series is always returns aligned to the longer one. This makes the var naming more explicit with `major/minor_i_start` as well as clarifies more stringently a bunch of other variables and explicitly uses the `minor_y_intersect` y value in the scaling transform calcs. Also fixes some debug prints.
We ended up doing groups maxmin sorting at the interaction layer (new the view box) and thus this method is no longer needed, though it was the reference for the code now in `ChartView.interact_graphics_cycle()`. Further this adds a `remove_axes: bool` arg to `.insert_plotitem()` which can be used to drop axis entries from the inserted pi (though it doesn't seem like we really ever need that?) and does the removal in a separate loop to avoid removing axes before they are registered in `ComposedGridLayout._pi2axes`.
Turns out this is a limitation of the `ViewBox.setYRange()` api: you can't call it more then once and expect anything but the first call to be applied without letting a render cycle run. As such, we wait until the end of the log-linear scaling loop to finally apply the major curves y-mx/mn after all minor curves have been evaluated. This also drops all the debug prints (for now) to get a feel for latency in production mode.
We can determine the major curve (in view) in the first pass of all `Viz`s so drop the 2nd loop and thus the `mxmn_groups: dict`. Also simplifies logic for the case of only one (the major) curve in view.
Facepalm, obviously absolute array indexes are not going to necessarily align vs. time over multiple feeds/history. Instead use `np.searchsorted()` on whatever curve has the smallest support and find the appropriate index of intersection in time so that alignment always starts at a sensible reference. Also adds a `debug_print: bool` input arg which can enable all the prints when working on this.
use the new `do_overlay_scaling: bool` since we know each feed will have its own updates (cuz multiplexed by feed..) and we can avoid ranging/scaling overlays that will make their own calls. Also, pass in the last datum "brighter" color for ohlc curves as it was originally (and now that we can pass that styling bit through).
A super snappy `numpy.median()` calculator (per input range) which we slap an `lru_cache` on thanks to handy dunder method hacks for such things on mutable types XD
Massively speeds up scaling transform cycles (duh). Also includes a draft for an "overlay transform" type/api; obviously still a WIP 🏄..
In the (incrementally updated) display loop we have range logic that is incrementally updated in real-time by streams, as such we don't really need to update all linked chart's (for any given, currently updated chart) y-ranges on calls of each separate (sub-)chart's `ChartView.interact_graphics_cycle()`. In practise there are plenty of cases where resizing in one chart (say the vlm fsps sub-plot) requires a y-range re-calc but not in the OHLC price chart. Therefore we always avoid doing more resizing then necessary despite it resulting in potentially more method call overhead (which will later be justified by better leveraging incrementally updated `Viz.maxmin()` and `media_from_range()` calcs).
It's way faster since it uses a uniform time arithmetic to narrow the `numpy.searchsorted()` range before actually doing the index search B)
This was a subtle logic error when building the `plots: dict` we weren't adding the "main (ohlc or other source) chart" from the `LinkedSplits` set when interacting with some sub-chart from `.subplots`.. Further this tries out bypassing `numpy.median()` altogether by just using `median = (ymx - ymn) / 2` which should be nearly the same?
In situations where clients are (dynamically) subscribing *while* broadcasts are starting to taking place we need to handle the `set`-modified-during-iteration case. This scenario seems to be more common during races on concurrent startup of multiple symbols. The solution here is to use another set to take note of subscribers which are successfully sent-to and then skipping them on re-try. This also contains an attempt to exception-handle throttled stream overruns caused by higher frequency feeds (like binance) pushing more quotes then can be handled during (UI) client startup.
Since we pretty much always want the 'bottom' and any side that is not declared by the caller move the axis hides into this method. Lets us drop the same calls in `.ui._fsp` and `._display`. This also disables the auto-ranging back-linking for now since it doesn't seem to be working quite yet?
This finally seems to mitigate all the "smearing" and "jitter" artifacts when using Qt's "coordinate cache" graphics-mode: - whenever we're in a mouse interaction (as per calls to `ChartView.start/signal_ic()`) we simply disable the caching mode (set `.NoCache` until the interaction is complete. - only do this (for now) during a pan since it doesn't seem to be an issue when zooming? - ensure disabling all `Viz.graphics` and `.ds_graphics` to be agnostic to any case where there's both a zoom and a pan simultaneously (not that it's easy to do manually XD) as well as solving the problem whenever an OHLC series is in traced-and-downsampled mode (during low zoom). Impl deatz: - rename `ChartView._ic` -> `._in_interact: trio.Event` - add `.ChartView._interact_stack: ExitStack` which we use to open. and close the `FlowGraphics.reset_cache()` mngrs from mouse handlers. - drop all the commented per-subtype overrides for `.cache_mode: int`. - write up much better doc strings for `FlattenedOHLC` and `StepCurve` including some very basic ASCII-art diagrams.
Instead delegate directly to `Viz.default_view()` throughout charting startup and interaction handlers. Also add a `ChartPlotWidget.reset_graphics_caches()` context mngr which resets all managed graphics object's cacheing modes on enter and restores them on exit for simplified use in interaction handling code.
Previously when very zoomed out and using the `'r'` hotkey the interaction handler loop wouldn't trigger a re-(up)sampling to get a more detailed curve graphic and instead the previous downsampled (under-detailed) graphic would show. Fix that by ensuring we yield back to the Qt event loop and do at least a couple render cycles with paired `.interact_graphics_cycle()` calls. Further this flips the `.start/signal_ic()` methods to use the new `.reset_graphics_caches()` ctr-mngr method.
When the target pinning curve (by default, the dispersion major) is shorter then the pinned curve, we need to make sure we find still find the x-intersect for computing returns scalars! Use `Viz.i_from_t()` to accomplish this as well and, augment that method with a `return_y: bool` to allow the caller to also retrieve the equivalent y-value at the requested input time `t: float` for convenience. Also tweak a few more internals around the 'loglin_ref_to_curve' method: - only solve / adjust for the above case when the major's xref is detected as being "earlier" in time the current minor's. - pop the major viz entry from the overlay table ahead of time to avoid a needless iteration and simplify the transform calc phase loop to avoid handling that needless cycle B) - add much better "organized" debug printing with more clear headers around which "phase"/loop the message pertains and well as more explicit details in terms of x and y-range values on each cycle of each loop.
There's been way too many issues when trying to calculate this dynamically from the input array, so just expect the caller to know what it's doing and don't bother with ever hitting the error case of calculating and incorrect value internally.
As per the change to `slice_from_time()` this ensures this `Viz` always passes its self-calculated time indexing step size to the time slicing routine(s). Further this contains a slight impl tweak to `.scalars_from_index()` to slice the actual view range from `xref` to `Viz.ViewState.xrange[1]` and then reading the corresponding `yref` from the first entry in that array; this should be no slower in theory and makes way for further caching of x-read-range to `ViewState` opportunities later.
Again, as per the signature change, never expect implicit time step calcs from overlay processing/machinery code. Also, extend the debug printing (yet again) to include better details around "rescale-due-to-minor-range-out-of-view" cases and a detailed msg for the transform/scaling calculation (inputs/outputs), particularly for the cases when one of the curves has a lesser support.
For the purposes of eventually trying to resolve last-step indexing synchronization (an intermittent but still existing) issue(s) that can happen due to races during history frame query and shm writing during startup. In fact, here we drop all `hist_viz` info queries from the main display loop for now anticipating that this code will either be removed or improved later.
Not sure how i missed this (and left in handling of `list.remove()` and it ever worked for that?) after the `samplerd` impl in 5ec1a72 but, this adjusts the remove-broken-subscriber loop to catch the correct `set.remove()` exception type on a missing (likely already removed) subscription entry.
This is particularly more "good looking" when we boot with a pair that doesn't have historical 1s OHLC and thus the fast chart is empty from outset. In this case it's a lot nicer to be already zoomed to a comfortable preset number of "datums in view" even when the history isn't yet filled in. Adjusts the chart display `Viz.default_view()` startup to explicitly ensure this happens via the `do_min_bars=True` flag B)
Solve this by always scaling the y-range for the major/target curve *before* the final overlay scaling loop; this implicitly always solve the case where the major series is the only one in view. Tidy up debug print formatting and add some loop-end demarcation comment lines.
jaredgoldman
approved these changes
Mar 13, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Redo of #455 which was mistakenly merged to a dev branch 😂
See that original PR for details and the follow up in #461.