Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add max_price_impact parameter to KellyBettingStrategy #433

Merged
merged 44 commits into from
Oct 3, 2024

Conversation

gabrielfior
Copy link
Contributor

No description provided.

Copy link

coderabbitai bot commented Sep 25, 2024

Walkthrough

The pull request introduces a comprehensive suite of unit tests for the KellyBettingStrategy class, focusing on validating price impact calculations in various market conditions. Additionally, it enhances the match_bets_with_langfuse_traces.py file with mean squared error (MSE) calculations and integrates a new HttpxCachedClient class for managing HTTP requests. The KellyBettingStrategy class is also updated to include price impact considerations, while the logging mechanism is standardized across several files.

Changes

File Path Change Summary
tests_integration/markets/omen/test_kelly.py - Introduced tests for KellyBettingStrategy with functions: test_kelly_price_impact_calculation1, test_kelly_price_impact_calculation2, test_kelly_price_impact_works_large_pool, and test_kelly_price_impact_works_small_pool.
- Added helper functions: assert_price_impact_converges and assert_price_impact.
examples/monitor/match_bets_with_langfuse_traces.py - Added classes: MSEProfit and HttpxCachedClient.
- Updated get_outcome_for_trace method signature.
- Expanded main execution block for KellyBettingStrategy and MaxExpectedValueBettingStrategy with new parameters.
- Initialized strat_mse_profits dictionary and modified simulation loop for MSE calculations.
- Enhanced output with MSE values and updated markdown report generation.
prediction_market_agent_tooling/deploy/betting_strategy.py - Updated KellyBettingStrategy constructor to include max_price_impact parameter.
- Added methods for price impact calculations: calculate_price_impact_for_bet_amount, calculate_bet_amount_for_price_impact.
- Modified calculate_trades method to incorporate price impact logic.
- Updated __repr__ method to include max_price_impact.
prediction_market_agent_tooling/markets/polymarket/api.py - Updated logger import statement to use prediction_market_agent_tooling.loggers.
prediction_market_agent_tooling/tools/httpx_cached_client.py - Introduced HttpxCachedClient class with caching capabilities.
- Added get_client method.
prediction_market_agent_tooling/tools/is_predictable.py - Updated logger import statement to use prediction_market_agent_tooling.loggers.
prediction_market_agent_tooling/tools/tavily_storage/tavily_models.py - Updated logger import statement to use prediction_market_agent_tooling.loggers.
- Refined type annotations for include_domains and exclude_domains fields in TavilyResponseModel class.
- Modified from_model method to align with updated field types.
prediction_market_agent_tooling/markets/agent_market.py - Added enum values: HIGHEST_LIQUIDITY and LOWEST_LIQUIDITY to SortBy class.
prediction_market_agent_tooling/markets/omen/omen_subgraph_handler.py - Updated _build_sort_params and get_omen_binary_markets_simple methods to support new sorting options for liquidity.

Possibly related PRs

Suggested reviewers

  • evangriffiths

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between 74d524e and 8cd4818.

📒 Files selected for processing (1)
  • tests_integration/markets/omen/test_kelly.py (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests_integration/markets/omen/test_kelly.py

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@gabrielfior
Copy link
Contributor Author

Results from this simulation look promising - already better than Kelly(max_bet_amount=2)

strategy bet_amount bet_profit roi start_balance end_balance
original 78.1787 -2.07655 -2.65616 nan nan
MaxAccuracyBettingStrategy(bet_amount=1) 77 0.66571 0.864559 50 50.6657
MaxAccuracyBettingStrategy(bet_amount=2) 154 -3.57625 -2.32224 50 46.4237
MaxAccuracyBettingStrategy(bet_amount=25) 1925 -421.166 -21.8787 50 -371.166
KellyBettingStrategy(max_bet_amount=1) 36.1606 4.60697 12.7403 50 54.607
KellyBettingStrategy(max_bet_amount=2) 67.3328 6.85947 10.1874 50 56.8595
KellyBettingStrategy(max_bet_amount=25) 299.67 -19.1948 -6.40532 50 30.8052
MaxAccuracyWithKellyScaledBetsStrategy(max_bet_amount=1) 56.028 -1.92566 -3.43697 50 48.0743
MaxAccuracyWithKellyScaledBetsStrategy(max_bet_amount=2) 108.325 -7.37903 -6.81197 50 42.621
MaxAccuracyWithKellyScaledBetsStrategy(max_bet_amount=25) 604.156 -207.427 -34.3333 50 -157.427
MaxExpectedValueBettingStrategy(bet_amount=1) 77 -3.35646 -4.35904 50 46.6435
MaxExpectedValueBettingStrategy(bet_amount=2) 154 -12.0145 -7.80159 50 37.9855
MaxExpectedValueBettingStrategy(bet_amount=25) 1925 -544.073 -28.2636 50 -494.073
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.01) 18.6526 2.04804 10.9799 50 52.048
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.05) 47.5707 6.10747 12.8387 50 56.1075
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.1) 57.1876 8.02994 14.0414 50 58.0299
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.15) 61.8607 8.59814 13.8992 50 58.5981
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.2) 64.5801 8.41843 13.0356 50 58.4184
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.25) 66.4781 7.71421 11.6041 50 57.7142
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.3) 67.2288 6.96345 10.3578 50 56.9635
KellyMaxSlippageBettingStrategy(max_bet_amount=5)(max_slippage=0.1) 106.891 -1.61437 -1.51029 50 48.3856
KellyMaxSlippageBettingStrategy(max_bet_amount=5)(max_slippage=0.15) 117.23 4.04472 3.45024 50 54.0447
KellyMaxSlippageBettingStrategy(max_bet_amount=5)(max_slippage=0.2) 121.595 6.69145 5.50306 50 56.6915
KellyMaxSlippageBettingStrategy(max_bet_amount=25)(max_slippage=0.1) 299.67 -19.1948 -6.40532 50 30.8052
KellyMaxSlippageBettingStrategy(max_bet_amount=25)(max_slippage=0.15) 299.67 -19.1948 -6.40532 50 30.8052
KellyMaxSlippageBettingStrategy(max_bet_amount=25)(max_slippage=0.2) 266.499 -23.351 -8.76214 50 26.649

Note that
-> KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.15) has end_balance 58.59813977 whereas KellyBettingStrategy(max_bet_amount=2) has 56.85946612
-> KellyMaxSlippageBettingStrategy(max_bet_amount=5)(max_slippage=0.2) has end_balance 56.69145435 (~ same as Kelly(2)) but volume of 120 xDAI -> larger volumes == market odds that better reflect agent's estimate_p_yes, while profit remains the same
-> KellyBettingStrategy(max_bet_amount=25) and KellyMaxSlippageBettingStrategy(max_bet_amount=25)(max_slippage=0.1) quite similar (tiny advantage to vanilla Kelly), but code improvements (see Newton optimization) should improve this.

Next steps
-> Improve Newton optimization
-> Overall tidying up of codebase

@gabrielfior
Copy link
Contributor Author

Some updated simulations

Highlights:
KellyMaxSlippageBettingStrategy(max_bet_amount=5)(max_slippage=0.6) -> 100.11 |
| KellyMaxSlippageBettingStrategy(max_bet_amount=5)(max_slippage=0.7) -> 101.613 |
| KellyBettingStrategy(max_bet_amount=2) -> 91.0167

strategy bet_amount bet_profit roi start_balance end_balance
original 116.179 13.2415 11.3975 nan nan
MaxAccuracyBettingStrategy(bet_amount=1) 115 16.7231 14.5418 50 66.7231
MaxAccuracyBettingStrategy(bet_amount=2) 230 19.7104 8.56975 50 69.7104
MaxAccuracyBettingStrategy(bet_amount=25) 2875 -588.529 -20.4706 50 -538.529
KellyBettingStrategy(max_bet_amount=1) 55.6312 25.9623 46.6687 50 75.9623
KellyBettingStrategy(max_bet_amount=2) 103.602 41.0167 39.5908 50 91.0167
KellyBettingStrategy(max_bet_amount=25) 510.551 16.9832 3.32645 50 66.9832
MaxAccuracyWithKellyScaledBetsStrategy(max_bet_amount=1) 85.7558 12.9484 15.0991 50 62.9484
MaxAccuracyWithKellyScaledBetsStrategy(max_bet_amount=2) 165.941 13.4283 8.09225 50 63.4283
MaxAccuracyWithKellyScaledBetsStrategy(max_bet_amount=25) 996.573 -295.252 -29.6267 50 -245.252
MaxExpectedValueBettingStrategy(bet_amount=1) 115 15.3165 13.3187 50 65.3165
MaxExpectedValueBettingStrategy(bet_amount=2) 230 15.5621 6.76613 50 65.5621
MaxExpectedValueBettingStrategy(bet_amount=25) 2875 -701.591 -24.4032 50 -651.591
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.01) 52.4206 18.0567 34.4458 50 68.0567
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.05) 69.3675 20.1455 29.0418 50 70.1455
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.1) 82.548 26.2711 31.8253 50 76.2711
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.15) 90.7457 30.9359 34.0907 50 80.9359
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.2) 95.5983 33.7643 35.3189 50 83.7643
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.25) 98.3001 35.654 36.2705 50 85.654
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.3) 100.078 37.055 37.0261 50 87.055
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.4) 102.184 38.4914 37.6688 50 88.4914
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.5) 103.196 40.1389 38.896 50 90.1389
KellyMaxSlippageBettingStrategy(max_bet_amount=2)(max_slippage=0.7) 103.602 41.0167 39.5908 50 91.0167
KellyMaxSlippageBettingStrategy(max_bet_amount=5)(max_slippage=0.1) 131.799 29.7057 22.5387 50 79.7057
KellyMaxSlippageBettingStrategy(max_bet_amount=5)(max_slippage=0.15) 154.398 37.7778 24.4678 50 87.7778
KellyMaxSlippageBettingStrategy(max_bet_amount=5)(max_slippage=0.2) 169.063 41.868 24.7647 50 91.868
KellyMaxSlippageBettingStrategy(max_bet_amount=5)(max_slippage=0.3) 184.052 45.8879 24.932 50 95.8879
KellyMaxSlippageBettingStrategy(max_bet_amount=5)(max_slippage=0.5) 196.458 48.659 24.7682 50 98.659
KellyMaxSlippageBettingStrategy(max_bet_amount=5)(max_slippage=0.6) 199.354 50.1099 25.1361 50 100.11
KellyMaxSlippageBettingStrategy(max_bet_amount=5)(max_slippage=0.7) 201.304 51.6126 25.6392 50 101.613
KellyMaxSlippageBettingStrategy(max_bet_amount=25)(max_slippage=0.1) 195.265 29.1344 14.9204 50 79.1344
KellyMaxSlippageBettingStrategy(max_bet_amount=25)(max_slippage=0.2) 282.734 42.5541 15.0509 50 92.5541
KellyMaxSlippageBettingStrategy(max_bet_amount=25)(max_slippage=0.3) 342.181 43.2821 12.6489 50 93.2821
KellyMaxSlippageBettingStrategy(max_bet_amount=25)(max_slippage=0.5) 434.687 44.0739 10.1392 50 94.0739
KellyMaxSlippageBettingStrategy(max_bet_amount=25)(max_slippage=0.7) 460.249 34.1844 7.42738 50 84.1844

else new_yes / (new_yes + new_no)
)
# print(f"actual_price {actual_price} actual_price2 {actual_price_2}")
s = (actual_price - expected_price) / expected_price
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sign of s is the opposite to what you'd get from the formula in this comment. Does it make a difference?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-> If actual_price > expected_price, you paid more for the same amount of tokens, hence slippage should be > 0.
-> If actual_price < 'expected_price`, you paid less, hence slippage < 0.

So I guess the formula is correct per the above, or would you disagree?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also note that

-> By the way, one correction ref slippage - in the "normal case" (see example here https://docs.gnosis.io/conditionaltokens/docs/introduction3/#an-example-with-cpmm), slippage is >0 (new_price_yes = 0.8, old_price = 0.5, hence positive slippage) (edited)
-> Now, in the example above, let's say you place a bet NO with the same 10 xDAI. If you do the math, you derive that you got 23 .33 NO tokens (invariant becomes 15 Y * 6.67 No = 100), but you expected 20 tokens (since original price was 0.2). Hence this is a <0 slippage (you paid cheaper than expected)
12:32
So slippage can be negative!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, this makes sense now. And it also makes sense to me that this works as expected when called in slippage_diff 😄. Lots of not very intuitive things for me to get my head around

try:
optimized_bet_amount = minimize_scalar(
slippage_diff,
bounds=(min(yes_outcome_pool_size, no_outcome_pool_size) / 10, 100),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If self.max_slippage is very small, then the root might be smaller than the lower bound. Similarly, if the pool is very large, the root could be bigger than 100. Is [0, inf] okay?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-> 0 is not OK since get_buy_outcome_token_amount breaks for 0

    if ending_outcome_balance <= 0:
        raise ValueError("must have non-zero balances")

For that reason I used a small size as bound, which might be too large already.

-> inf works, I can try with sys.maxsize or similar

Copy link
Contributor Author

@gabrielfior gabrielfior Oct 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@evangriffiths please take a look the updated versions now, should address the concerns above.

I don't see any more logs with deviating values. I also changed the log to an Error, so it's obvious when values are wrong.

…track-experiments-kelly

# Conflicts:
#	examples/monitor/match_bets_with_langfuse_traces.py
#	poetry.lock
]

storage = hishel.FileStorage(ttl=3600)
controller = hishel.Controller(force_cache=True)
httpx_client = hishel.CacheClient(storage=storage, controller=controller)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's interesting, what about cutting it out to some helper function like get_cached_httpx_client?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Abstracted it away (see class HttpxCachedClient)

@@ -1,5 +1,10 @@
from abc import ABC, abstractmethod

import numpy as np
from loguru import logger
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from loguru import logger
from prediction_market_agent_tooling.loggers import logger

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You didn't commit this

slippage, self.max_price_impact, self.max_price_impact / 100
):
logger.info(
f"Slippage {slippage} deviates too much from self.max_slippage {self.max_price_impact}, market_id {market.id}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deviates too much

It sounds pretty bad, yet it's logged as info, and the code continues. Can it be just ignored?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see this happen a lot if I run the script on this branch. If this is ever hit, does that mean there's a bug in calculate_bet_amount_for_price_impact?

Copy link
Contributor Author

@gabrielfior gabrielfior Oct 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed the minimize_scalar was complaining - I adjusted bounds and now it looks better. Please give it a go and let me know if you find issues!
Please note that this is now a ValueError since I want to enforce these boundaries.

# We just don't want Kelly size to extrapolate price_impact - hence we take the min.
kelly_bet_size = min(kelly_bet.size, max_slippage_bet_amount)

self._check_price_impact_ok_else_log(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it's possible it's not ok, but it just continues?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check revised version - now it's a ValueError.

kongzii
kongzii previously approved these changes Sep 30, 2024
def calculate_bet_amount_for_price_impact(
self, market: AgentMarket, kelly_bet: SimpleBet, fee: float
) -> float:
def calculate_price_impact_deviation_from_target_price_impact(b: xDai) -> float:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this need to be a function within the function?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't have to be, it just felt simpler to have this here since it's not needed anywhere else in the class.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that's almost never done in Python. Can you just make it a normal function, please?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dunno, I think it's fine in this context, especially as it's being used as a Callable, passed as an arg to minimize_scalar

ChatGPT agrees:

in python is it good practice to define a function inside a function?
...
If the inner function is being passed as a callable to another function, this can be a valid and clean design choice, especially if the callable depends on the state of the outer function

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also I did the same thing here

def f(r: float) -> float:
R = r / (1 - fee)
first_term = other_holdings - R
second_term = holdings + shares_to_sell - R
third_term = holdings * other_holdings
return (first_term * second_term) - third_term
amount_to_sell = newton(f, 0)
😄

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay then, I'm standing overvoted 😄 For sure it's not a blocker.

@@ -0,0 +1,18 @@
r_a = 10
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In PMAT, all scripts have been helpful in executing some actions so far.

This feels more for the examples folder. 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

kelly: KellyBettingStrategy,
):
# expect
expected_price = yes / (yes + no) # p_yes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this p_no?

1 - self.outcomeTokenAmounts[self.yes_index] / sum(self.outcomeTokenAmounts)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed yes, no = no, yes if buy_direction is False.
Tests taken from https://docs.gnosis.io/conditionaltokens/docs/introduction3/#an-example-with-cpmm pass.

@kongzii kongzii dismissed their stale review September 30, 2024 11:20

Sorry, accidentaly clicked on Approve while submitting

@kongzii
Copy link
Contributor

kongzii commented Sep 30, 2024

Some updated simulations

Can you post these for all agents we have available? No need for the full table, just KellyBettingStrategy(max_bet_amount=2) and KellyMaxSlippageBettingStrategy(max_bet_amount=5)(max_slippage=0.7), to keep the table readable.

What I'm wondering is if this strategy is more profitable on all of them. To check if this isn't an over-fit for this particular agent.

@gabrielfior
Copy link
Contributor Author

Some more results

Notes:

  • 3 agents this time, KellyWithSlippage always at least as good as KellyWithoutSlippage
  • KnownOutcomeAgent -> KellyWithSlippage is not better than MaxAccuracy (as seen previously)

DeployablePredictionProphetGPT4TurboFinalAgent

23 bets

strategy bet_amount bet_profit roi start_balance end_balance
original 160.777 18.0077 11.2004 nan nan
MaxAccuracyBettingStrategy(bet_amount=2) 316 28.8172 9.11937 50 78.8172
KellyBettingStrategy(max_bet_amount=1)(max_price_impact=None) 73.4929 40.7539 55.4529 50 90.7539
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=None) 137.211 62.8255 45.7875 50 112.825
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=None) 691.641 34.9588 5.05448 50 84.9588
MaxAccuracyWithKellyScaledBetsStrategy(max_bet_amount=2) 228.797 21.5993 9.44039 50 71.5993
MaxExpectedValueBettingStrategy(bet_amount=1) 158 14.6483 9.27107 50 64.6483
MaxExpectedValueBettingStrategy(bet_amount=2) 316 9.11322 2.88393 50 59.1132
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.3) 133.024 57.5977 43.2988 50 107.598
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.4) 135.62 60.0017 44.2424 50 110.002
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.5) 136.805 61.9476 45.2817 50 111.948
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.6) 137.211 62.8255 45.7875 50 112.825
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.7) 137.211 62.8255 45.7875 50 112.825
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.3) 249.345 69.1817 27.7454 50 119.182
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.4) 258.642 73.4124 28.3838 50 123.412
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.5) 264.741 75.871 28.6586 50 125.871
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.6) 268.425 78.2573 29.1543 50 128.257
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.7) 270.949 80.4069 29.6761 50 130.407
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.1) 260.365 40.6729 15.6215 50 90.6729
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.2) 386.327 58.2634 15.0814 50 108.263
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.3) 466.331 58.3095 12.5039 50 108.31
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.5) 581.951 55.7225 9.57512 50 105.722
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.7) 620.596 49.179 7.92448 50 99.179

DeployablePredictionProphetGPT4TurboPreviewAgent

23 bets

strategy bet_amount bet_profit roi start_balance end_balance
original 104 -7.92499 -7.62019 nan nan
MaxAccuracyBettingStrategy(bet_amount=2) 208 -20.0945 -9.66083 50 29.9055
KellyBettingStrategy(max_bet_amount=1)(max_price_impact=None) 40.2936 8.26246 20.5056 50 58.2625
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=None) 74.7609 11.3541 15.1873 50 61.3541
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=None) 408.998 -57.7451 -14.1187 50 -7.74514
MaxAccuracyWithKellyScaledBetsStrategy(max_bet_amount=2) 156.919 -25.1377 -16.0195 50 24.8623
MaxExpectedValueBettingStrategy(bet_amount=1) 104 -6.5439 -6.29221 50 43.4561
MaxExpectedValueBettingStrategy(bet_amount=2) 208 -24.2854 -11.6757 50 25.7146
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.3) 73.8451 11.4346 15.4846 50 61.4346
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.4) 74.7609 11.3541 15.1873 50 61.3541
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.5) 74.7609 11.3541 15.1873 50 61.3541
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.6) 74.7609 11.3541 15.1873 50 61.3541
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.7) 74.7609 11.3541 15.1873 50 61.3541
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.3) 130.923 8.68103 6.63065 50 58.681
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.4) 139.854 9.47468 6.77468 50 59.4747
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.5) 145.182 8.883 6.11852 50 58.883
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.6) 148.609 7.45913 5.0193 50 57.4591
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.7) 150.303 6.90822 4.5962 50 56.9082
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.1) 168.902 -0.609285 -0.360734 50 49.3907
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.2) 228.025 -3.20583 -1.40591 50 46.7942
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.3) 264.118 -2.60361 -0.985773 50 47.3964
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.5) 307.953 -7.14671 -2.32072 50 42.8533
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.7) 326.901 -12.2237 -3.73927 50 37.7763

DeployableOlasEmbeddingOAAgent

23 bets

strategy bet_amount bet_profit roi start_balance end_balance
original 147 55.1104 37.4901 nan nan
MaxAccuracyBettingStrategy(bet_amount=2) 294 89.8477 30.5604 50 139.848
KellyBettingStrategy(max_bet_amount=1)(max_price_impact=None) 81.5521 56.4293 69.1942 50 106.429
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=None) 155.032 89.2523 57.5703 50 139.252
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=None) 735.673 71.1136 9.66647 50 121.114
MaxAccuracyWithKellyScaledBetsStrategy(max_bet_amount=2) 206.069 83.4645 40.5031 50 133.465
MaxExpectedValueBettingStrategy(bet_amount=1) 147 54.1975 36.8691 50 104.198
MaxExpectedValueBettingStrategy(bet_amount=2) 294 83.8231 28.5113 50 133.823
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.3) 144.999 86.1892 59.4414 50 136.189
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.4) 150.062 89.6193 59.7214 50 139.619
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.5) 152.866 90.0891 58.9335 50 140.089
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.6) 154.416 89.5823 58.0135 50 139.582
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.7) 154.842 89.1623 57.5829 50 139.162
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.3) 248.758 110.487 44.4157 50 160.487
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.4) 268.008 116.515 43.4744 50 166.515
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.5) 277.564 120.036 43.2461 50 170.036
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.6) 283.943 122.643 43.193 50 172.643
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.7) 288.367 123.353 42.7763 50 173.353
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.1) 233.914 63.782 27.2673 50 113.782
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.2) 362.385 104.051 28.7129 50 154.051
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.3) 439.988 125.579 28.5415 50 175.579
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.5) 550.385 137.681 25.0155 50 187.681
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.7) 614.055 126.629 20.6218 50 176.629

DeployableKnownOutcomeAgent

23 bets

strategy bet_amount bet_profit roi start_balance end_balance
original 280.535 -40.0711 -14.2838 nan nan
MaxAccuracyBettingStrategy(bet_amount=2) 194 3.76331 1.93985 50 53.7633
KellyBettingStrategy(max_bet_amount=1)(max_price_impact=None) 83.7098 9.78298 11.6868 50 59.783
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=None) 156.494 11.3861 7.27578 50 61.3861
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=None) 983.003 -228.228 -23.2174 50 -178.228
MaxAccuracyWithKellyScaledBetsStrategy(max_bet_amount=2) 156.494 11.3861 7.27578 50 61.3861
MaxExpectedValueBettingStrategy(bet_amount=1) 97 6.03838 6.22514 50 56.0384
MaxExpectedValueBettingStrategy(bet_amount=2) 194 3.76331 1.93985 50 53.7633
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.3) 151.358 9.17158 6.05951 50 59.1716
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.4) 154.249 11.5487 7.48705 50 61.5487
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.5) 155.413 11.9903 7.71515 50 61.9903
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.6) 155.792 12.0879 7.75899 50 62.0879
KellyBettingStrategy(max_bet_amount=2)(max_price_impact=0.7) 156.02 11.8595 7.60122 50 61.8595
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.3) 301.286 -9.02099 -2.99416 50 40.979
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.4) 310.59 -9.20958 -2.96519 50 40.7904
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.5) 316.442 -8.8274 -2.78958 50 41.1726
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.6) 320.954 -7.62369 -2.37532 50 42.3763
KellyBettingStrategy(max_bet_amount=5)(max_price_impact=0.7) 325.251 -6.46563 -1.98789 50 43.5344
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.1) 305.309 -9.48895 -3.10798 50 40.511
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.2) 464.043 -35.4952 -7.64912 50 14.5048
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.3) 575.063 -65.691 -11.4233 50 -15.691
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.5) 705.942 -115.614 -16.3773 50 -65.6142
KellyBettingStrategy(max_bet_amount=25)(max_price_impact=0.7) 767.354 -140.574 -18.3193 50 -90.5738

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Outside diff range and nitpick comments (3)
prediction_market_agent_tooling/deploy/betting_strategy.py (3)

150-174: New method: assert_price_impact_lt_max_price_impact_else_raise

This method ensures that the price impact of a bet doesn't exceed the specified maximum. The implementation looks correct, but there are a few points to consider:

  1. The method name is quite long. Consider a shorter name like assert_price_impact_within_limit.
  2. The use of np.isclose with a relative tolerance is a good approach to handle floating-point comparisons.
  3. The error message in the ValueError is informative, which is good for debugging.

Consider renaming the method to improve readability:

def assert_price_impact_within_limit(
    self,
    buy_direction: bool,
    bet_size: float,
    market: AgentMarket,
) -> None:
    # ... rest of the method remains the same

204-216: Updated calculate_trades method with price impact consideration

The modifications to include price impact calculations in the calculate_trades method are well-implemented. However, there are a few points to consider:

  1. The comment "Adjust amount" could be more descriptive.
  2. The hardcoded 0 passed to calculate_bet_amount_for_price_impact for the fee parameter might not be correct for all markets.

Consider updating the comment and using the market's fee:

# Calculate bet amount considering max price impact
max_slippage_bet_amount = self.calculate_bet_amount_for_price_impact(
    market, kelly_bet, market.fee
)

229-244: New method: calculate_price_impact_for_bet_amount

This method calculates the price impact based on the bet amount and market conditions. The implementation looks correct, but there's a minor issue with a comment:

The comment "price_impact should always be > 0" is not always true. Price impact can be negative in certain market conditions. Consider updating the comment to reflect this:

# Note: price_impact can be positive or negative depending on market conditions
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between 06be0d6 and 04ec90d.

📒 Files selected for processing (1)
  • prediction_market_agent_tooling/deploy/betting_strategy.py (5 hunks)
🔇 Additional comments (5)
prediction_market_agent_tooling/deploy/betting_strategy.py (5)

3-5: New imports added for price impact calculations

The additions of numpy, logger, and minimize_scalar are appropriate for the new price impact calculation functionality. These libraries will be useful for numerical operations, logging, and optimization respectively.


18-20: Import of get_buy_outcome_token_amount function

The import of get_buy_outcome_token_amount from the Omen module is now explicitly stated. This is good for clarity and helps in understanding the dependencies of the price impact calculations.


146-148: Updated constructor with max_price_impact parameter

The addition of the max_price_impact parameter with a default value of None is a good approach. It allows for backwards compatibility while introducing the new price impact functionality.


290-290: Updated __repr__ method

The __repr__ method has been updated to include the max_price_impact parameter. This is good for debugging and logging purposes.


Line range hint 1-290: Overall assessment of changes

The modifications to the KellyBettingStrategy class significantly enhance its functionality by incorporating price impact considerations. The new methods for calculating and asserting price impact are well-implemented, and the updates to the calculate_trades method effectively integrate these new features.

There are a few minor suggestions for improvement:

  1. Consider renaming the assert_price_impact_lt_max_price_impact_else_raise method for better readability.
  2. Ensure that market fees are correctly used in price impact calculations.
  3. Update comments to accurately reflect the behavior of price impact calculations.

These changes provide a more sophisticated betting strategy that should perform better in real-world market conditions by considering the impact of bets on market prices.

Comment on lines 246 to 287
def calculate_bet_amount_for_price_impact(
self,
market: AgentMarket,
kelly_bet: SimpleBet,
fee: float,
) -> float:
def calculate_price_impact_deviation_from_target_price_impact(
bet_amount: xDai,
) -> float:
price_impact = self.calculate_price_impact_for_bet_amount(
kelly_bet.direction,
bet_amount,
yes_outcome_pool_size,
no_outcome_pool_size,
fee,
)
# We return abs for the algorithm to converge to 0 instead of the min (and possibly negative) value.

max_price_impact = check_not_none(self.max_price_impact)
return abs(price_impact - max_price_impact)

if not market.outcome_token_pool:
logger.warning(
"Market outcome_token_pool is None, cannot calculate bet amount"
)
return kelly_bet.size

yes_outcome_pool_size = market.outcome_token_pool[
market.get_outcome_str_from_bool(True)
]
no_outcome_pool_size = market.outcome_token_pool[
market.get_outcome_str_from_bool(False)
]

optimized_bet_amount = minimize_scalar(
calculate_price_impact_deviation_from_target_price_impact,
bounds=(min(yes_outcome_pool_size, no_outcome_pool_size) / 1000, 1000),
method="bounded",
tol=1e-11,
options={"maxiter": 10000},
)
return float(optimized_bet_amount.x)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

New method: calculate_bet_amount_for_price_impact

This method uses numerical optimization to find the bet amount that achieves the target price impact. The implementation is generally good, but there are a few points to consider:

  1. The use of minimize_scalar with bounded optimization is appropriate.
  2. The inner function calculate_price_impact_deviation_from_target_price_impact is well-structured.
  3. The error handling for missing outcome_token_pool is good.

However, there's a potential issue:

The fee parameter is passed to this method but not used in the price impact calculation. This could lead to inaccurate results if the market has a non-zero fee. Consider updating the inner function to use the fee parameter:

price_impact = self.calculate_price_impact_for_bet_amount(
    kelly_bet.direction,
    bet_amount,
    yes_outcome_pool_size,
    no_outcome_pool_size,
    fee,  # Use the fee parameter here
)

price_impact = self.calculate_price_impact_for_bet_amount(
buy_direction,
bet_size,
outcome_token_pool["Yes"],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somewhere down the codes you used

market.outcome_token_pool[market.get_outcome_str_from_bool(True)]

and

market.outcome_token_pool[market.get_outcome_str_from_bool(False)]

That's more future-proof for no extra costs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point - implemented.

@@ -0,0 +1,50 @@
import numpy as np
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These test are in the omen directory but they don't have anything specific about Omen

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - moved.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Outside diff range and nitpick comments (7)
tests/test_kelly.py (3)

6-13: Enhance test coverage and readability.

While the test case is based on Gnosis documentation, consider the following improvements:

  1. Add assertions within the function body to verify the initial state of the KellyBettingStrategy instance.
  2. Include comments explaining the expected behavior or outcome of the test.
  3. Consider parameterizing the test to cover multiple scenarios easily.

Here's a suggested improvement:

def test_kelly_slippage_calculation1() -> None:
    # Test case based on https://docs.gnosis.io/conditionaltokens/docs/introduction3/#an-example-with-cpmm
    kelly = KellyBettingStrategy(max_bet_amount=1, max_price_impact=0.5)
    
    # Verify initial state
    assert kelly.max_bet_amount == 1
    assert kelly.max_price_impact == 0.5

    # Equal probability scenario
    yes = 10
    no = 10
    bet_amount = 10
    buy_direction = True

    # Expected outcome: Price impact should be calculated correctly for a buy direction in an equal probability scenario
    assert_price_impact(bet_amount, buy_direction, yes, no, kelly)

16-24: Enhance test coverage and readability.

Similar to the first test case, consider the following improvements:

  1. Add assertions within the function body to verify the initial state of the KellyBettingStrategy instance.
  2. Include comments explaining the expected behavior or outcome of the test.
  3. Consider parameterizing the test to cover multiple scenarios easily.

Here's a suggested improvement:

def test_kelly_slippage_calculation2() -> None:
    # Test case based on https://docs.gnosis.io/conditionaltokens/docs/introduction3/#an-example-with-cpmm
    kelly = KellyBettingStrategy(max_bet_amount=1, max_price_impact=0.5)
    
    # Verify initial state
    assert kelly.max_bet_amount == 1
    assert kelly.max_price_impact == 0.5

    # Scenario after a bet of 10 xDAI on Yes
    yes = 5
    no = 20
    bet_amount = 10
    buy_direction = False

    # Expected outcome: Price impact should be calculated correctly for a sell direction after a previous bet
    assert_price_impact(bet_amount, buy_direction, yes, no, kelly)

1-51: Overall good start, but room for improvement in test coverage and structure.

The file provides a good foundation for testing the KellyBettingStrategy class, particularly focusing on price impact calculations. However, there are several areas where the tests could be enhanced:

  1. Increase test coverage by adding more test cases with different scenarios.
  2. Consider using parameterized tests to easily cover multiple input combinations.
  3. Add more assertions within each test function to verify the state of the KellyBettingStrategy instance and intermediate calculations.
  4. Improve code readability by adding more comments explaining the expected behavior and outcomes of each test.
  5. Consider organizing the tests into a test class, which could help with setup and teardown of common test fixtures.

To improve the overall structure and maintainability of the tests, consider the following:

  1. Create a TestKellyBettingStrategy class to group related tests.
  2. Use pytest.fixture for common setup code.
  3. Implement parameterized tests using pytest.mark.parametrize for testing multiple scenarios efficiently.

Example structure:

import pytest
from prediction_market_agent_tooling.deploy.betting_strategy import KellyBettingStrategy

class TestKellyBettingStrategy:
    @pytest.fixture
    def kelly_strategy(self):
        return KellyBettingStrategy(max_bet_amount=1, max_price_impact=0.5)

    @pytest.mark.parametrize("yes,no,bet_amount,buy_direction,expected_impact", [
        (10, 10, 10, True, ...),  # Fill in expected impact
        (5, 20, 10, False, ...),  # Fill in expected impact
        # Add more test cases here
    ])
    def test_price_impact_calculation(self, kelly_strategy, yes, no, bet_amount, buy_direction, expected_impact):
        price_impact = kelly_strategy.calculate_price_impact_for_bet_amount(
            buy_direction, bet_amount=bet_amount, yes=yes, no=no, fee=0
        )
        assert pytest.approx(price_impact, rel=1e-2) == expected_impact

    # Add more test methods as needed

This structure would make it easier to add and maintain tests as the KellyBettingStrategy class evolves.

prediction_market_agent_tooling/deploy/betting_strategy.py (4)

146-148: New parameter added for price impact control.

The addition of max_price_impact parameter enhances the flexibility of the KellyBettingStrategy. This is a good improvement that allows for more fine-grained control over the betting behavior.

Consider using Optional[float] instead of float | None for better compatibility with Python versions below 3.10:

from typing import Optional

def __init__(self, max_bet_amount: float, max_price_impact: Optional[float] = None):

150-174: New method to assert price impact is within limits.

This method provides a crucial check to ensure that the price impact of a bet doesn't exceed the specified maximum. The use of np.isclose for floating-point comparison is a good practice.

Consider improving the error handling by providing more context in the error message:

raise ValueError(
    f"Price impact {price_impact:.4f} exceeds max_price_impact {max_price_impact:.4f} "
    f"by {(price_impact - max_price_impact) / max_price_impact:.2%}. "
    f"Market ID: {market.id}"
)

This change provides more detailed information about the extent of the violation, which could be helpful for debugging and monitoring.


229-244: New method to calculate price impact.

This method provides a crucial calculation for determining the price impact of a bet. The logic seems correct and uses appropriate market functions.

However, the comment "price_impact should always be > 0" might not be accurate in all cases. Price impact can be negative if the actual price is lower than the expected price. Consider updating the comment to:

# Note: price_impact can be positive or negative depending on market conditions

This change acknowledges that price impact can vary based on market dynamics.


246-287: New method for optimizing bet amount based on price impact.

This method effectively uses numerical optimization to find the optimal bet amount that achieves the target price impact. The use of minimize_scalar is appropriate for this task.

Consider the following improvements:

  1. Add a docstring to explain the purpose and parameters of the method.
  2. Handle the case when self.max_price_impact is None more explicitly.
  3. Consider making the optimization bounds more flexible or configurable.

Here's a suggested improvement:

def calculate_bet_amount_for_price_impact(
    self,
    market: AgentMarket,
    kelly_bet: SimpleBet,
    fee: float,
) -> float:
    """
    Calculate the optimal bet amount for a given price impact using numerical optimization.
    
    Args:
        market (AgentMarket): The market to calculate the bet amount for.
        kelly_bet (SimpleBet): The Kelly bet object containing direction and size.
        fee (float): The fee percentage.
    
    Returns:
        float: The optimal bet amount that achieves the target price impact.
    """
    if self.max_price_impact is None:
        return kelly_bet.size

    if not market.outcome_token_pool:
        logger.warning("Market outcome_token_pool is None, cannot calculate bet amount")
        return kelly_bet.size

    yes_outcome_pool_size = market.outcome_token_pool[market.get_outcome_str_from_bool(True)]
    no_outcome_pool_size = market.outcome_token_pool[market.get_outcome_str_from_bool(False)]

    lower_bound = min(yes_outcome_pool_size, no_outcome_pool_size) / 1000
    upper_bound = max(1000, kelly_bet.size * 2)  # Adjust upper bound based on Kelly bet size

    # ... rest of the method remains the same

These changes improve the method's clarity, error handling, and flexibility.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between 04ec90d and b8884de.

📒 Files selected for processing (2)
  • prediction_market_agent_tooling/deploy/betting_strategy.py (5 hunks)
  • tests/test_kelly.py (1 hunks)
🔇 Additional comments (3)
tests/test_kelly.py (1)

1-3: LGTM: Imports are appropriate for the tests.

The import statements are correct and necessary for the tests being performed. numpy is used for floating-point comparison, and KellyBettingStrategy is the class being tested.

prediction_market_agent_tooling/deploy/betting_strategy.py (2)

3-5: New imports added for enhanced functionality.

The additions of numpy, logger, and minimize_scalar from scipy suggest new mathematical operations and optimization techniques are being introduced. This is a good practice for expanding the capabilities of the betting strategy.


289-290: Updated string representation to include max_price_impact.

The __repr__ method has been correctly updated to include the max_price_impact parameter. This change ensures that the string representation of the class accurately reflects its current state, including the new parameter.

Comment on lines 27 to 51
def assert_price_impact(
bet_amount: float,
buy_direction: bool,
yes: float,
no: float,
kelly: KellyBettingStrategy,
) -> None:
price_impact = kelly.calculate_price_impact_for_bet_amount(
buy_direction, bet_amount=bet_amount, yes=yes, no=no, fee=0
)

# Calculation is done assuming buy_direction is True. Else, we invert the reserves.
if not buy_direction:
yes, no = no, yes

expected_price_yes = no / (yes + no)
k = yes * no
tokens_bought_yes = (yes + bet_amount) - (k / (no + bet_amount)) # 23.333 # x*y = k
actual_price_yes = bet_amount / tokens_bought_yes
expected_price_impact = (actual_price_yes - expected_price_yes) / expected_price_yes
####

assert np.isclose(price_impact, expected_price_impact, rtol=0.01)

print(price_impact)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codebase verification

Unnecessary print statements found in test files.

The following print statements are present:

  • tests/test_kelly.py: print(price_impact)
  • tests/markets/omen/test_omen.py: print(position) # For extra test coverage

Consider removing these print statements or replacing them with proper logging to avoid cluttering test outputs.

🔗 Analysis chain

Minor improvements and clarification needed.

The assert_price_impact function looks well-implemented overall. However, consider the following suggestions:

  1. Add type hints for the return value of calculate_price_impact_for_bet_amount.
  2. Consider using constants for magic numbers like 0.01 in the np.isclose call.
  3. The print statement at the end might not be necessary for automated tests. If it's for debugging, consider using a logger or removing it.

Here's a suggested improvement:

import logging
from typing import Any

RELATIVE_TOLERANCE = 0.01

def assert_price_impact(
    bet_amount: float,
    buy_direction: bool,
    yes: float,
    no: float,
    kelly: KellyBettingStrategy,
) -> None:
    price_impact: float = kelly.calculate_price_impact_for_bet_amount(
        buy_direction, bet_amount=bet_amount, yes=yes, no=no, fee=0
    )

    # ... (rest of the function remains the same)

    assert np.isclose(price_impact, expected_price_impact, rtol=RELATIVE_TOLERANCE)

    logging.debug(f"Price impact: {price_impact}")

Could you clarify the purpose of the print statement at the end of the function? Is it intended for debugging or some other purpose?

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for other print statements in test files
rg --type python 'print\(' tests/

Length of output: 66


Script:

#!/bin/bash
# Search for other print statements in Python test files
rg --type py 'print\(' tests/

Length of output: 154

Comment on lines 204 to 217
kelly_bet_size = kelly_bet.size
if self.max_price_impact:
# Adjust amount
max_slippage_bet_amount = self.calculate_bet_amount_for_price_impact(
market, kelly_bet, 0
)

# We just don't want Kelly size to extrapolate price_impact - hence we take the min.
kelly_bet_size = min(kelly_bet.size, max_slippage_bet_amount)

self.assert_price_impact_lt_max_price_impact_else_raise(
kelly_bet.direction, kelly_bet_size, market
)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Bet size adjustment based on price impact.

The addition of price impact consideration in the bet size calculation is a significant improvement. It ensures that the Kelly bet size doesn't exceed the maximum allowed price impact.

However, the hardcoded value of 0 for the fee parameter in calculate_bet_amount_for_price_impact could lead to inaccurate calculations if the market has a non-zero fee. Consider using the actual market fee:

max_slippage_bet_amount = self.calculate_bet_amount_for_price_impact(
    market, kelly_bet, fee=market.fee if hasattr(market, 'fee') else 0
)

This change ensures that the fee is correctly accounted for in the price impact calculations.

# We just don't want Kelly size to extrapolate price_impact - hence we take the min.
kelly_bet_size = min(kelly_bet.size, max_slippage_bet_amount)

self.assert_price_impact_lt_max_price_impact_else_raise(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed to run every time in the core code? Looks like a sanity check that could be done in a pytest, and removed from here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Refactored this as an integration test (see test_kelly).


optimized_bet_amount = minimize_scalar(
calculate_price_impact_deviation_from_target_price_impact,
bounds=(min(yes_outcome_pool_size, no_outcome_pool_size) / 1000, 1000),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I commented on these bounds before, but find it. But these don't seem super robust, e.g. if used for markets with v large or v small liquidity (or non-Omen markets that use a different currency where it's normal to see larger/smaller values). Shouldn't the bounds be [0, inf]?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah here's the previous comment #433 (comment)

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Outside diff range and nitpick comments (7)
prediction_market_agent_tooling/tools/tavily_storage/tavily_models.py (1)

20-20: LGTM! Consider updating the import style for consistency.

The change from using loguru to an internal logging module is a good practice for maintaining control over the logging implementation. This change is unlikely to cause any functional issues as the usage of logger.debug() is consistent with both implementations.

For consistency with other imports in the file, consider using an absolute import:

from prediction_market_agent_tooling.loggers import logger

This style is more explicit and less prone to potential circular import issues.

prediction_market_agent_tooling/deploy/betting_strategy.py (6)

146-149: LGTM: New parameter added for price impact control

The addition of the max_price_impact parameter with a default value of None is a good improvement. It allows for more flexible betting strategies while maintaining backward compatibility.

Consider adding a type hint for max_price_impact to improve code readability:

def __init__(self, max_bet_amount: float, max_price_impact: Optional[float] = None):

Don't forget to import Optional from typing if you make this change.


150-174: LGTM: New method to enforce maximum price impact

This method effectively ensures that the price impact doesn't exceed the maximum allowed value. The use of check_not_none for null checks and np.isclose for float comparisons are good practices.

Consider adding a docstring to explain the purpose and parameters of this method. For example:

def assert_price_impact_lt_max_price_impact_else_raise(
    self,
    buy_direction: bool,
    bet_size: float,
    market: AgentMarket,
) -> None:
    """
    Assert that the price impact of a bet is less than the maximum allowed price impact.

    Args:
        buy_direction (bool): True if buying, False if selling.
        bet_size (float): The size of the bet.
        market (AgentMarket): The market to check the price impact against.

    Raises:
        ValueError: If the price impact exceeds the maximum allowed value.
    """

204-217: LGTM: Bet size adjustment based on price impact

The addition of price impact considerations in the bet size calculation is a significant improvement. It ensures that the Kelly bet size doesn't exceed the maximum allowed price impact.

The hardcoded value of 0 for the fee parameter in calculate_bet_amount_for_price_impact could lead to inaccurate calculations if the market has a non-zero fee. Consider using the actual market fee:

max_slippage_bet_amount = self.calculate_bet_amount_for_price_impact(
    market, kelly_bet, fee=market.fee if hasattr(market, 'fee') else 0
)

229-244: LGTM: New method for calculating price impact

The calculate_price_impact_for_bet_amount method correctly implements the price impact calculation. The use of get_buy_outcome_token_amount from the Omen market is appropriate.

The comment "price_impact should always be > 0" might not always be true. Price impact can be negative if the actual price is lower than the expected price, which can happen in certain market conditions. Consider updating the comment to:

# Note: price_impact can be positive or negative depending on market conditions

246-287: LGTM: New method for optimizing bet amount based on price impact

The calculate_bet_amount_for_price_impact method effectively uses numerical optimization to find the optimal bet amount. The use of scipy.optimize.minimize_scalar with bounded optimization is appropriate, and the error handling for missing outcome_token_pool is good.

Consider making the optimization bounds more flexible:

  1. The lower bound min(yes_outcome_pool_size, no_outcome_pool_size) / 1000 might be too large for markets with very small liquidity.
  2. The upper bound of 1000 might be too small for markets with very large liquidity or when using a different currency.

You could make these bounds configurable or use a wider range. For example:

lower_bound = max(1e-6, min(yes_outcome_pool_size, no_outcome_pool_size) / 1e6)
upper_bound = max(1000, max(yes_outcome_pool_size, no_outcome_pool_size) * 10)

optimized_bet_amount = minimize_scalar(
    calculate_price_impact_deviation_from_target_price_impact,
    bounds=(lower_bound, upper_bound),
    method="bounded",
    tol=1e-11,
    options={"maxiter": 10000},
)

This change would make the method more robust across different market conditions and currencies.


Line range hint 1-290: Overall assessment: Significant improvement in betting strategy

The changes to the KellyBettingStrategy class represent a substantial improvement in the betting strategy by incorporating price impact considerations. The new methods for calculating and optimizing bet amounts based on price impact are well-implemented and add valuable functionality to the strategy.

Key improvements:

  1. Addition of max_price_impact parameter for flexible control.
  2. New methods for calculating and asserting price impact.
  3. Integration of price impact considerations in the calculate_trades method.
  4. Use of numerical optimization to find optimal bet amounts.

These changes should lead to more sophisticated and potentially more profitable betting strategies, especially in markets where price impact is a significant factor.

To further improve the code:

  1. Consider extracting the price impact calculation logic into a separate class or module. This would improve modularity and make it easier to test and maintain the price impact logic independently of the betting strategy.
  2. Implement unit tests for the new methods, especially calculate_price_impact_for_bet_amount and calculate_bet_amount_for_price_impact, to ensure their correctness across various market conditions.
  3. Consider adding logging statements in key parts of the code (e.g., when adjusting bet sizes or when price impact assertions fail) to aid in debugging and monitoring the strategy's performance.
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between b8884de and 6b7b6a7.

📒 Files selected for processing (4)
  • prediction_market_agent_tooling/deploy/betting_strategy.py (5 hunks)
  • prediction_market_agent_tooling/markets/polymarket/api.py (1 hunks)
  • prediction_market_agent_tooling/tools/is_predictable.py (1 hunks)
  • prediction_market_agent_tooling/tools/tavily_storage/tavily_models.py (1 hunks)
✅ Files skipped from review due to trivial changes (1)
  • prediction_market_agent_tooling/markets/polymarket/api.py
🔇 Additional comments (3)
prediction_market_agent_tooling/tools/is_predictable.py (1)

Line range hint 1-185: LGTM! Ensure thorough testing of logging functionality.

The change in the logger import doesn't appear to affect the rest of the file. The main functions is_predictable_binary and is_predictable_without_description should continue to work as expected. However, it's crucial to ensure that the logging functionality remains intact.

To verify the logging functionality, please run the following test cases:

  1. Test the error logging in is_predictable_binary:
import sys
sys.path.append('path/to/your/project')
from prediction_market_agent_tooling.tools.is_predictable import is_predictable_binary

# Simulate a scenario where langchain is not installed
import sys
sys.modules['langchain'] = None

result = is_predictable_binary("Test question")
# Check if the error was logged correctly
  1. Test the error logging in is_predictable_without_description:
from prediction_market_agent_tooling.tools.is_predictable import is_predictable_without_description

# Simulate a scenario where langchain is not installed
import sys
sys.modules['langchain'] = None

result = is_predictable_without_description("Test question", "Test description")
# Check if the error was logged correctly

These tests will help ensure that the logging functionality works as expected with the new logger.

prediction_market_agent_tooling/deploy/betting_strategy.py (2)

3-7: New imports added for price impact calculations

The additions of numpy, scipy.optimize, and get_buy_outcome_token_amount are appropriate for the new price impact calculations. These libraries provide the necessary tools for numerical optimization and financial calculations.


289-290: LGTM: Updated __repr__ method

The __repr__ method has been correctly updated to include the max_price_impact parameter. This change provides a more complete representation of the object's state, which is consistent with the addition of max_price_impact in the __init__ method.

@evangriffiths evangriffiths changed the title Add Slippage Kelly strategy Add max_price_impact parameter to KellyBettingStrategy Oct 3, 2024

optimized_bet_amount = minimize_scalar(
calculate_price_impact_deviation_from_target_price_impact,
bounds=(min(yes_outcome_pool_size, no_outcome_pool_size) / 1000, 1000),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you get some exception with max bound of sys.maxsize. A simple way of making it more robust would be just to set the upper bound relative to total pool size.

I tried

Suggested change
bounds=(min(yes_outcome_pool_size, no_outcome_pool_size) / 1000, 1000),
bounds=(0, 1000 * (yes_outcome_pool_size + no_outcome_pool_size)),

and it seems to work fine, and is more robust imo.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me as well. Implemented.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (6)
prediction_market_agent_tooling/deploy/betting_strategy.py (6)

146-149: LGTM: New parameter for price impact control.

The addition of max_price_impact parameter allows for configurable price impact limits, which is a good enhancement to the strategy.

Consider using Optional[float] instead of float | None for better compatibility with Python versions below 3.10:

-def __init__(self, max_bet_amount: float, max_price_impact: float | None = None):
+from typing import Optional
+
+def __init__(self, max_bet_amount: float, max_price_impact: Optional[float] = None):

150-174: LGTM: Price impact assertion method.

This method effectively ensures that the price impact doesn't exceed the specified limit. The use of np.isclose is a good practice for floating-point comparisons.

Consider the following improvements:

  1. Make the tolerance configurable, e.g., as a class attribute.
  2. Enhance the error message to include the actual difference between the price impact and the max_price_impact.
class KellyBettingStrategy(BettingStrategy):
    def __init__(self, max_bet_amount: float, max_price_impact: Optional[float] = None, price_impact_tolerance: float = 0.01):
        # ... existing code ...
        self.price_impact_tolerance = price_impact_tolerance

    def assert_price_impact_lt_max_price_impact_else_raise(
        self,
        buy_direction: bool,
        bet_size: float,
        market: AgentMarket,
    ) -> None:
        # ... existing code ...
        if price_impact > max_price_impact and not np.isclose(
            price_impact, max_price_impact, atol=max_price_impact * self.price_impact_tolerance
        ):
            raise ValueError(
                f"Price impact {price_impact:.6f} exceeds max_price_impact {max_price_impact:.6f} "
                f"by {price_impact - max_price_impact:.6f} (tolerance: {max_price_impact * self.price_impact_tolerance:.6f}), "
                f"market_id {market.id}"
            )

204-217: LGTM: Bet size adjustment based on price impact.

The addition of price impact consideration in the bet size calculation is a significant improvement. It ensures that the Kelly bet size doesn't exceed the maximum allowed price impact.

Consider the following improvements:

  1. Remove the redundant assertion after adjustment, as calculate_bet_amount_for_price_impact should already ensure the price impact is within limits.
  2. Rename max_slippage_bet_amount to max_price_impact_bet_amount for consistency with the new terminology.
  3. Pass the market fee to calculate_bet_amount_for_price_impact instead of hardcoding 0.
 if self.max_price_impact:
     # Adjust amount
-    max_slippage_bet_amount = self.calculate_bet_amount_for_price_impact(
-        market, kelly_bet, 0
+    max_price_impact_bet_amount = self.calculate_bet_amount_for_price_impact(
+        market, kelly_bet, market.fee
     )

     # We just don't want Kelly size to extrapolate price_impact - hence we take the min.
-    kelly_bet_size = min(kelly_bet.size, max_slippage_bet_amount)
+    kelly_bet_size = min(kelly_bet.size, max_price_impact_bet_amount)

-    self.assert_price_impact_lt_max_price_impact_else_raise(
-        kelly_bet.direction, kelly_bet_size, market
-    )

229-244: LGTM: Price impact calculation method.

The calculate_price_impact_for_bet_amount method correctly calculates the price impact based on the expected and actual prices.

Update the comment about price impact always being > 0, as it might not be accurate in all cases:

-# price_impact should always be > 0
+# Note: price_impact can be positive or negative depending on market conditions
 price_impact = (actual_price - expected_price) / expected_price

246-288: LGTM: Optimal bet amount calculation for price impact.

The calculate_bet_amount_for_price_impact method effectively uses numerical optimization to find the bet amount that achieves the target price impact.

Consider the following improvements:

  1. Make the optimization bounds more flexible:
-bounds=(0, 1000 * (yes_outcome_pool_size + no_outcome_pool_size)),
+bounds=(0, 10 * (yes_outcome_pool_size + no_outcome_pool_size)),
  1. Add a docstring explaining the method's purpose and parameters.
  2. Consider extracting the inner function calculate_price_impact_deviation_from_target_price_impact as a separate method for better readability.
  3. Add error handling for cases where the optimization fails to converge.

Example implementation:

def calculate_bet_amount_for_price_impact(
    self,
    market: AgentMarket,
    kelly_bet: SimpleBet,
    fee: float,
) -> float:
    """
    Calculate the optimal bet amount that achieves the target price impact.

    Args:
        market (AgentMarket): The market to calculate the bet amount for.
        kelly_bet (SimpleBet): The Kelly bet object containing direction and size.
        fee (float): The market fee.

    Returns:
        float: The optimal bet amount that achieves the target price impact.

    Raises:
        ValueError: If the optimization fails to converge.
    """
    if not market.outcome_token_pool:
        logger.warning("Market outcome_token_pool is None, cannot calculate bet amount")
        return kelly_bet.size

    yes_outcome_pool_size = market.outcome_token_pool[market.get_outcome_str_from_bool(True)]
    no_outcome_pool_size = market.outcome_token_pool[market.get_outcome_str_from_bool(False)]

    result = minimize_scalar(
        self._calculate_price_impact_deviation,
        args=(kelly_bet, yes_outcome_pool_size, no_outcome_pool_size, fee),
        bounds=(0, 10 * (yes_outcome_pool_size + no_outcome_pool_size)),
        method="bounded",
        tol=1e-11,
        options={"maxiter": 10000},
    )

    if not result.success:
        raise ValueError(f"Failed to find optimal bet amount: {result.message}")

    return float(result.x)

def _calculate_price_impact_deviation(
    self,
    bet_amount: float,
    kelly_bet: SimpleBet,
    yes_outcome_pool_size: float,
    no_outcome_pool_size: float,
    fee: float,
) -> float:
    price_impact = self.calculate_price_impact_for_bet_amount(
        kelly_bet.direction,
        bet_amount,
        yes_outcome_pool_size,
        no_outcome_pool_size,
        fee,
    )
    max_price_impact = check_not_none(self.max_price_impact)
    return abs(price_impact - max_price_impact)

Line range hint 1-291: Overall assessment: Significant enhancement to KellyBettingStrategy

The changes introduce price impact considerations to the KellyBettingStrategy, which is a valuable improvement. The implementation is generally well-structured and efficient, using numerical optimization techniques to calculate optimal bet amounts.

Key improvements:

  1. Addition of max_price_impact parameter for configurable price impact limits.
  2. New methods for calculating and asserting price impacts.
  3. Adjustment of bet sizes based on price impact constraints.

While the implementation is sound, consider the following general suggestions:

  1. Improve error handling and logging throughout the new methods.
  2. Add comprehensive unit tests for the new functionality, especially edge cases.
  3. Consider adding more detailed documentation, including examples of how to use the new price impact features.

To further enhance the strategy's robustness and flexibility, consider:

  1. Implementing a factory method or configuration system to easily create different variants of the KellyBettingStrategy with various price impact settings.
  2. Exploring the possibility of dynamically adjusting the max_price_impact based on market conditions or historical data.
  3. Implementing a simulation framework to test the strategy's performance under various market scenarios and price impact settings.
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between 6b7b6a7 and 6f014a6.

⛔ Files ignored due to path filters (2)
  • poetry.lock is excluded by !**/*.lock, !**/*.lock
  • pyproject.toml is excluded by !**/*.toml
📒 Files selected for processing (1)
  • prediction_market_agent_tooling/deploy/betting_strategy.py (5 hunks)
🔇 Additional comments (2)
prediction_market_agent_tooling/deploy/betting_strategy.py (2)

3-4: LGTM: New imports for numerical operations.

The addition of numpy and scipy.optimize.minimize_scalar is appropriate for the new price impact calculations implemented in the KellyBettingStrategy class.


291-291: LGTM: Updated string representation.

The __repr__ method now includes the max_price_impact parameter, providing a more complete representation of the strategy's configuration.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Outside diff range and nitpick comments (8)
tests_integration/markets/omen/test_kelly.py (5)

16-23: LGTM: Well-documented test case for price impact calculation.

The test case is clear and references the Gnosis documentation, which is excellent for traceability. The use of assert_price_impact is appropriate for verifying the calculation.

Consider adding a brief comment explaining the expected outcome of this test case to enhance readability.


26-34: LGTM: Consistent follow-up test case for price impact calculation.

This test case logically follows the previous one and continues to reference the Gnosis documentation. The use of assert_price_impact is consistent with the previous test.

For consistency with the previous test, consider adding a brief comment explaining the expected outcome of this specific scenario.


37-50: LGTM: Well-structured parameterized test for large pool price impact.

The use of pytest.mark.parametrize is excellent for testing multiple scenarios efficiently. The test appropriately retrieves a large market for realistic testing.

Consider the following improvements:

  1. Add error handling for potential exceptions from OmenSubgraphHandler.get_omen_binary_markets_simple().
  2. Include a brief comment explaining the purpose of testing with a large pool and the expected outcomes for different parameter combinations.

69-108: LGTM: Well-structured helper function for price impact convergence.

The function effectively calculates and asserts the convergence of price impact. The use of np.isclose for floating-point comparison is appropriate.

Consider the following improvements:

  1. Add error handling for potential exceptions from external function calls (e.g., get_kelly_bet_full, calculate_bet_amount_for_price_impact).
  2. Consider adding a docstring to explain the function's purpose and parameters.
  3. The tolerance in the assertion (0.001 * max_price_impact) might be too strict for some cases. Consider making it a parameter or using a relative tolerance instead.

111-135: LGTM: Well-implemented helper function for price impact assertion.

The function effectively calculates and asserts the correctness of price impact for both buy and sell directions. The logic is clear and the use of np.isclose for comparison is appropriate.

Consider the following improvements:

  1. Remove the print statement at the end of the function (line 135) as it's not necessary for a test function and might clutter the test output.
  2. Add a docstring to explain the function's purpose and parameters.
  3. Consider using a constant for the relative tolerance in np.isclose (currently 0.01) to make it easier to adjust if needed.
prediction_market_agent_tooling/deploy/betting_strategy.py (2)

145-147: LGTM: New max_price_impact parameter added

The addition of the max_price_impact parameter with a default value of None is a good improvement. It allows for more flexible betting strategies while maintaining backward compatibility.

Consider adding a docstring to explain the purpose and usage of the max_price_impact parameter.


215-257: New method for calculating bet amount based on price impact is well-implemented

The calculate_bet_amount_for_price_impact method is a good addition, using numerical optimization to find the bet amount that achieves the target price impact.

Consider the following improvements:

  1. Add a docstring explaining the purpose and parameters of the method.
  2. Add a comment explaining the rationale behind the chosen optimization bounds.
  3. Consider making the bounds more flexible or configurable, especially for markets with very large or very small liquidity pools.

Example docstring:

def calculate_bet_amount_for_price_impact(
    self,
    market: AgentMarket,
    kelly_bet: SimpleBet,
    fee: float,
) -> float:
    """
    Calculate the optimal bet amount that achieves the target price impact.

    Args:
        market (AgentMarket): The market to calculate the bet amount for.
        kelly_bet (SimpleBet): The Kelly bet object containing direction and size.
        fee (float): The market fee.

    Returns:
        float: The optimal bet amount that achieves the target price impact.
    """
prediction_market_agent_tooling/markets/omen/omen_subgraph_handler.py (1)

322-331: Summary: New liquidity-based sorting options added.

The changes introduce new sorting options for HIGHEST_LIQUIDITY and LOWEST_LIQUIDITY in the _build_sort_params method. These additions enhance the functionality of the Omen subgraph handler by allowing markets to be sorted based on their liquidity.

To ensure these changes are fully integrated:

  1. Verify that the get_omen_binary_markets_simple method properly handles these new sorting options.
  2. Update any relevant documentation or API references to include these new sorting options.
  3. Consider adding unit tests to verify the behavior of these new sorting options.
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between 6f014a6 and 74d524e.

📒 Files selected for processing (4)
  • prediction_market_agent_tooling/deploy/betting_strategy.py (5 hunks)
  • prediction_market_agent_tooling/markets/agent_market.py (1 hunks)
  • prediction_market_agent_tooling/markets/omen/omen_subgraph_handler.py (1 hunks)
  • tests_integration/markets/omen/test_kelly.py (1 hunks)
🔇 Additional comments (8)
tests_integration/markets/omen/test_kelly.py (2)

1-14: LGTM: Import statements are well-organized and appropriate.

The import statements are logically organized and include all necessary modules for testing the KellyBettingStrategy. Good job on following the import order convention (standard library, third-party, local).


1-135: Overall: Well-structured test file with room for minor improvements

This test file for the KellyBettingStrategy is generally well-organized and covers various scenarios for price impact calculations. The use of parameterized tests and helper functions promotes code reuse and maintainability.

Key points for improvement:

  1. Address the inconsistency in the test_kelly_price_impact_works_small_pool function.
  2. Add error handling for potential exceptions from external API calls and function invocations.
  3. Enhance documentation with docstrings for helper functions and brief comments explaining expected outcomes for test cases.
  4. Remove unnecessary print statements and consider parameterizing tolerance values for easier adjustment.

Addressing these points will further improve the robustness and clarity of the tests.

prediction_market_agent_tooling/markets/agent_market.py (2)

Line range hint 1-32: Overall assessment: Changes look good, but seem to be part of a larger implementation.

The additions to the SortBy enum are well-implemented and consistent with the existing code. However, these changes alone don't fully implement the max_price_impact parameter mentioned in the PR objectives. It would be helpful to see the complete set of changes across all affected files to fully understand how this enhancement to the Kelly betting strategy is being implemented.

To get a better understanding of the full scope of changes, please run the following script:

#!/bin/bash
# Description: List all modified files in this PR and show a summary of changes

# List all modified files
echo "Modified files in this PR:"
git diff --name-only origin/main

# Show a summary of changes for each file
echo "\nSummary of changes:"
git diff --stat origin/main

This will help reviewers understand the full context of the changes and how they relate to the PR objectives.


31-32: LGTM! Consider adding documentation for the new sorting options.

The addition of HIGHEST_LIQUIDITY and LOWEST_LIQUIDITY to the SortBy enum is a good enhancement. These new sorting options align well with the PR's objective of improving the Kelly betting strategy by considering market conditions.

Consider adding a brief comment above the SortBy class to explain the purpose of these sorting options, especially the new liquidity-based ones. This would help other developers understand the intent and usage of these options.

To ensure these changes are properly integrated, please run the following script to check for any usage of SortBy that might need updating:

✅ Verification successful

Verification Successful

All usages of the SortBy enum, including the new HIGHEST_LIQUIDITY and LOWEST_LIQUIDITY values, have been reviewed and integrated correctly across the codebase.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check for usage of SortBy enum and potential areas that need updating

# Search for SortBy usage
echo "Searching for SortBy usage:"
rg --type python "SortBy\." -A 5

# Search for potential switch statements or if-else blocks that might need updating
echo "\nSearching for potential switch statements or if-else blocks using SortBy:"
rg --type python "if.*SortBy\..*:.*elif.*SortBy\." -A 10

Length of output: 1588


Script:

#!/bin/bash
# Description: Check for usage of SortBy enum and potential areas that need updating

# Search for SortBy usage
echo "Searching for SortBy usage:"
rg --type python "SortBy\." -A 5

# Search for potential switch statements or if-else blocks using SortBy
echo -e "\nSearching for potential switch statements or if-else blocks using SortBy:"
rg --type python "if\s+.*SortBy\..*:" -A 10

Length of output: 370


Script:

#!/bin/bash
# Description: Check for usage of SortBy enum and potential areas that need updating

# Search for SortBy usage
echo "Searching for SortBy usage:"
rg --type py "SortBy\." -A 5

# Search for potential if-else blocks using SortBy
echo -e "\nSearching for potential if-else blocks using SortBy:"
rg --type py "if\s+.*SortBy\..*:" -A 10

Length of output: 23556

prediction_market_agent_tooling/deploy/betting_strategy.py (3)

3-6: New imports added for price impact calculations

The new imports (minimize_scalar, xDai, and logger) are appropriate for the added functionality. The minimize_scalar function from SciPy will be used for optimizing the bet amount based on price impact.


260-260: LGTM: Updated __repr__ method

The __repr__ method has been correctly updated to include the max_price_impact parameter in the string representation of the KellyBettingStrategy object.


198-213: ⚠️ Potential issue

New method for calculating price impact looks good

The calculate_price_impact_for_bet_amount method is well-implemented and correctly calculates the price impact for a given bet amount.

However, the comment "price_impact should always be > 0" on line 211 is not always true. Price impact can be negative if the actual price is lower than the expected price. Consider updating the comment to reflect this possibility:

-# price_impact should always be > 0
+# Note: price_impact can be positive or negative depending on market conditions

Likely invalid or redundant comment.

prediction_market_agent_tooling/markets/omen/omen_subgraph_handler.py (1)

322-331: New sorting options added for market liquidity.

The changes introduce two new sorting options: HIGHEST_LIQUIDITY and LOWEST_LIQUIDITY. These additions enhance the flexibility of the market querying functionality, allowing users to sort markets based on their liquidity.

A few observations:

  1. The implementation is consistent with the existing pattern.
  2. The new options use the liquidityMeasure field, which is appropriate for sorting by liquidity.
  3. The changes don't break existing functionality.

@gabrielfior gabrielfior merged commit dff7c35 into main Oct 3, 2024
14 checks passed
@gabrielfior gabrielfior deleted the gabriel/track-experiments-kelly branch October 3, 2024 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants