Skip to content

Commit a0cef03

Browse files
SunsetWolfyou-n-g
SunsetWolf
and
you-n-g
authored
update python version (#1868)
* update python version * fix: Correct selector handling and add time filtering in storage.py * fix: convert index and columns to list in repr methods * feat: Add Makefile for managing project prerequisites * feat: Add Cython extensions for rolling and expanding operations * resolve install error * fix lint error * fix lint error * fix lint error * fix lint error * fix lint error * update build package * update makefile * update ci yaml * fix docs build error * fix ubuntu install error * fix docs build error * fix install error * fix install error * fix install error * fix install error * fix pylint error * fix pylint error * fix pylint error * fix pylint error * fix pylint error E1123 * fix pylint error R0917 * fix pytest error * fix pytest error * fix pytest error * update code * update code * fix ci error * fix pylint error * fix black error * fix pytest error * fix CI error * fix CI error * add python version to CI * add python version to CI * add python version to CI * fix pylint error * fix pytest general nn error * fix CI error * optimize code * add coments * Extended macos version * remove build package --------- Co-authored-by: Young <[email protected]>
1 parent 7acb4f3 commit a0cef03

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

63 files changed

+462
-428
lines changed

‎.github/workflows/python-publish.yml

+4-24
Original file line numberDiff line numberDiff line change
@@ -12,43 +12,23 @@ jobs:
1212
runs-on: ${{ matrix.os }}
1313
strategy:
1414
matrix:
15-
os: [windows-latest, macos-13]
15+
os: [windows-latest, ubuntu-20.04, ubuntu-22.04, macos-13, macos-14, macos-latest]
1616
# FIXME: macos-latest will raise error now.
1717
# not supporting 3.6 due to annotations is not supported https://stackoverflow.com/a/52890129
18-
python-version: [3.7, 3.8]
18+
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
1919

2020
steps:
2121
- uses: actions/checkout@v2
22-
# This is because on macos systems you can install pyqlib using
23-
# `pip install pyqlib` installs, it does not recognize the
24-
# `pyqlib-<version>-cp38-cp38-macosx_11_0_x86_64.whl` and `pyqlib-<veresion>-cp38-cp37m-macosx_11_0_x86_64.whl`.
25-
# So we limit the version of python, in order to generate a version of qlib that is usable for macos: `pyqlib-<veresion>-cp38-cp37m
26-
# `pyqlib-<version>-cp38-cp38-macosx_10_15_x86_64.whl` and `pyqlib-<veresion>-cp38-cp37m-macosx_10_15_x86_64.whl`.
27-
# Python 3.7.16, 3.8.16 can build macosx_10_15. But Python 3.7.17, 3.8.17 can build macosx_11_0
2822
- name: Set up Python ${{ matrix.python-version }}
29-
if: matrix.os == 'macos-11' && matrix.python-version == '3.7'
30-
uses: actions/setup-python@v2
31-
with:
32-
python-version: "3.7.16"
33-
- name: Set up Python ${{ matrix.python-version }}
34-
if: matrix.os == 'macos-11' && matrix.python-version == '3.8'
35-
uses: actions/setup-python@v2
36-
with:
37-
python-version: "3.8.16"
38-
- name: Set up Python ${{ matrix.python-version }}
39-
if: matrix.os != 'macos-11'
4023
uses: actions/setup-python@v2
4124
with:
4225
python-version: ${{ matrix.python-version }}
4326
- name: Install dependencies
4427
run: |
45-
python -m pip install --upgrade pip
46-
pip install setuptools wheel twine
28+
make dev
4729
- name: Build wheel on ${{ matrix.os }}
4830
run: |
49-
pip install numpy
50-
pip install cython
51-
python setup.py bdist_wheel
31+
make build
5232
- name: Build and publish
5333
env:
5434
TWINE_USERNAME: __token__

‎.github/workflows/test_qlib_from_source.yml

+14-80
Original file line numberDiff line numberDiff line change
@@ -19,25 +19,15 @@ jobs:
1919
# If you want to use python 3.7 in github action, then the latest macos system version is macos-13,
2020
# after macos-13 python 3.7 is no longer supported.
2121
# so we limit the macos version to macos-13.
22-
os: [windows-latest, ubuntu-20.04, ubuntu-22.04, macos-13]
22+
os: [windows-latest, ubuntu-20.04, ubuntu-22.04, macos-13, macos-14, macos-latest]
2323
# not supporting 3.6 due to annotations is not supported https://stackoverflow.com/a/52890129
24-
python-version: [3.7, 3.8]
24+
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
2525

2626
steps:
2727
- name: Test qlib from source
2828
uses: actions/checkout@v3
2929

30-
# Since version 3.7 of python for MacOS is installed in CI, version 3.7.17, this version causes "_bz not found error".
31-
# So we make the version number of python 3.7 for MacOS more specific.
32-
# refs: https://github.com/actions/setup-python/issues/682
3330
- name: Set up Python ${{ matrix.python-version }}
34-
if: (matrix.os == 'macos-latest' && matrix.python-version == '3.7') || (matrix.os == 'macos-13' && matrix.python-version == '3.7')
35-
uses: actions/setup-python@v4
36-
with:
37-
python-version: "3.7.16"
38-
39-
- name: Set up Python ${{ matrix.python-version }}
40-
if: (matrix.os != 'macos-latest' || matrix.python-version != '3.7') && (matrix.os != 'macos-13' || matrix.python-version != '3.7')
4131
uses: actions/setup-python@v4
4232
with:
4333
python-version: ${{ matrix.python-version }}
@@ -47,7 +37,7 @@ jobs:
4737
python -m pip install --upgrade pip
4838
4939
- name: Installing pytorch for macos
50-
if: ${{ matrix.os == 'macos-13' || matrix.os == 'macos-latest' }}
40+
if: ${{ matrix.os == 'macos-13' || matrix.os == 'macos-14' || matrix.os == 'macos-latest' }}
5141
run: |
5242
python -m pip install torch torchvision torchaudio
5343
@@ -63,95 +53,41 @@ jobs:
6353
6454
- name: Set up Python tools
6555
run: |
66-
python -m pip install --upgrade cython
67-
python -m pip install -e .[dev]
56+
make dev
6857
6958
- name: Lint with Black
70-
# Python 3.7 will use a black with low level. So we use python with higher version for black check
71-
if: (matrix.python-version != '3.7')
7259
run: |
73-
pip install -U black # follow the latest version of black, previous Qlib dependency will downgrade black
74-
black . -l 120 --check --diff
60+
make black
7561
7662
- name: Make html with sphinx
7763
# Since read the docs builds on ubuntu 22.04, we only need to test that the build passes on ubuntu 22.04.
7864
if: ${{ matrix.os == 'ubuntu-22.04' }}
7965
run: |
80-
cd docs
81-
sphinx-build -W --keep-going -b html . _build
82-
cd ..
83-
84-
# Check Qlib with pylint
85-
# TODO: These problems we will solve in the future. Important among them are: W0221, W0223, W0237, E1102
86-
# C0103: invalid-name
87-
# C0209: consider-using-f-string
88-
# R0402: consider-using-from-import
89-
# R1705: no-else-return
90-
# R1710: inconsistent-return-statements
91-
# R1725: super-with-arguments
92-
# R1735: use-dict-literal
93-
# W0102: dangerous-default-value
94-
# W0212: protected-access
95-
# W0221: arguments-differ
96-
# W0223: abstract-method
97-
# W0231: super-init-not-called
98-
# W0237: arguments-renamed
99-
# W0612: unused-variable
100-
# W0621: redefined-outer-name
101-
# W0622: redefined-builtin
102-
# FIXME: specify exception type
103-
# W0703: broad-except
104-
# W1309: f-string-without-interpolation
105-
# E1102: not-callable
106-
# E1136: unsubscriptable-object
107-
# References for parameters: https://github.com/PyCQA/pylint/issues/4577#issuecomment-1000245962
108-
# We use sys.setrecursionlimit(2000) to make the recursion depth larger to ensure that pylint works properly (the default recursion depth is 1000).
66+
make docs-gen
67+
10968
- name: Check Qlib with pylint
11069
run: |
111-
pylint --disable=C0104,C0114,C0115,C0116,C0301,C0302,C0411,C0413,C1802,R0401,R0801,R0902,R0903,R0911,R0912,R0913,R0914,R0915,R1720,W0105,W0123,W0201,W0511,W0613,W1113,W1514,E0401,E1121,C0103,C0209,R0402,R1705,R1710,R1725,R1735,W0102,W0212,W0221,W0223,W0231,W0237,W0612,W0621,W0622,W0703,W1309,E1102,E1136 --const-rgx='[a-z_][a-z0-9_]{2,30}$' qlib --init-hook "import astroid; astroid.context.InferenceContext.max_inferred = 500; import sys; sys.setrecursionlimit(2000)"
112-
pylint --disable=C0104,C0114,C0115,C0116,C0301,C0302,C0411,C0413,C1802,R0401,R0801,R0902,R0903,R0911,R0912,R0913,R0914,R0915,R1720,W0105,W0123,W0201,W0511,W0613,W1113,W1514,E0401,E1121,C0103,C0209,R0402,R1705,R1710,R1725,R1735,W0102,W0212,W0221,W0223,W0231,W0237,W0246,W0612,W0621,W0622,W0703,W1309,E1102,E1136 --const-rgx='[a-z_][a-z0-9_]{2,30}$' scripts --init-hook "import astroid; astroid.context.InferenceContext.max_inferred = 500; import sys; sys.setrecursionlimit(2000)"
113-
114-
# The following flake8 error codes were ignored:
115-
# E501 line too long
116-
# Description: We have used black to limit the length of each line to 120.
117-
# F541 f-string is missing placeholders
118-
# Description: The same thing is done when using pylint for detection.
119-
# E266 too many leading '#' for block comment
120-
# Description: To make the code more readable, a lot of "#" is used.
121-
# This error code appears centrally in:
122-
# qlib/backtest/executor.py
123-
# qlib/data/ops.py
124-
# qlib/utils/__init__.py
125-
# E402 module level import not at top of file
126-
# Description: There are times when module level import is not available at the top of the file.
127-
# W503 line break before binary operator
128-
# Description: Since black formats the length of each line of code, it has to perform a line break when a line of arithmetic is too long.
129-
# E731 do not assign a lambda expression, use a def
130-
# Description: Restricts the use of lambda expressions, but at some point lambda expressions are required.
131-
# E203 whitespace before ':'
132-
# Description: If there is whitespace before ":", it cannot pass the black check.
70+
make pylint
71+
13372
- name: Check Qlib with flake8
13473
run: |
135-
flake8 --ignore=E501,F541,E266,E402,W503,E731,E203 --per-file-ignores="__init__.py:F401,F403" qlib
74+
make flake8
13675
137-
# https://github.com/python/mypy/issues/10600
13876
- name: Check Qlib with mypy
13977
run: |
140-
mypy qlib --install-types --non-interactive || true
141-
mypy qlib --verbose
78+
make mypy
14279
14380
- name: Check Qlib ipynb with nbqa
14481
run: |
145-
nbqa black . -l 120 --check --diff
146-
nbqa pylint . --disable=C0104,C0114,C0115,C0116,C0301,C0302,C0411,C0413,C1802,R0401,R0801,R0902,R0903,R0911,R0912,R0913,R0914,R0915,R1720,W0105,W0123,W0201,W0511,W0613,W1113,W1514,E0401,E1121,C0103,C0209,R0402,R1705,R1710,R1725,R1735,W0102,W0212,W0221,W0223,W0231,W0237,W0612,W0621,W0622,W0703,W1309,E1102,E1136,W0719,W0104,W0404,C0412,W0611,C0410 --const-rgx='[a-z_][a-z0-9_]{2,30}$'
82+
make nbqa
14783
14884
- name: Test data downloads
14985
run: |
15086
python scripts/get_data.py qlib_data --name qlib_data_simple --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn
15187
python scripts/get_data.py download_data --file_name rl_data.zip --target_dir tests/.data/rl
15288
15389
- name: Install Lightgbm for MacOS
154-
if: ${{ matrix.os == 'macos-13' || matrix.os == 'macos-latest' }}
90+
if: ${{ matrix.os == 'macos-13' || matrix.os == 'macos-14' || matrix.os == 'macos-latest' }}
15591
run: |
15692
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Microsoft/qlib/main/.github/brew_install.sh)"
15793
HOMEBREW_NO_AUTO_UPDATE=1 brew install lightgbm
@@ -161,11 +97,9 @@ jobs:
16197
brew unlink libomp
16298
brew install libomp.rb
16399
164-
# Run after data downloads
165100
- name: Check Qlib ipynb with nbconvert
166101
run: |
167-
# add more ipynb files in future
168-
jupyter nbconvert --to notebook --execute examples/workflow_by_code.ipynb
102+
make nbconvert
169103
170104
- name: Test workflow by config (install from source)
171105
run: |

‎.github/workflows/test_qlib_from_source_slow.yml

+4-16
Original file line numberDiff line numberDiff line change
@@ -19,41 +19,29 @@ jobs:
1919
# If you want to use python 3.7 in github action, then the latest macos system version is macos-13,
2020
# after macos-13 python 3.7 is no longer supported.
2121
# so we limit the macos version to macos-13.
22-
os: [windows-latest, ubuntu-20.04, ubuntu-22.04, macos-13]
22+
os: [windows-latest, ubuntu-20.04, ubuntu-22.04, macos-13, macos-14, macos-latest]
2323
# not supporting 3.6 due to annotations is not supported https://stackoverflow.com/a/52890129
24-
python-version: [3.7, 3.8]
24+
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
2525

2626
steps:
2727
- name: Test qlib from source slow
2828
uses: actions/checkout@v3
2929

30-
# Since version 3.7 of python for MacOS is installed in CI, version 3.7.17, this version causes "_bz not found error".
31-
# So we make the version number of python 3.7 for MacOS more specific.
32-
# refs: https://github.com/actions/setup-python/issues/682
3330
- name: Set up Python ${{ matrix.python-version }}
34-
if: (matrix.os == 'macos-latest' && matrix.python-version == '3.7') || (matrix.os == 'macos-13' && matrix.python-version == '3.7')
35-
uses: actions/setup-python@v4
36-
with:
37-
python-version: "3.7.16"
38-
39-
- name: Set up Python ${{ matrix.python-version }}
40-
if: (matrix.os != 'macos-latest' || matrix.python-version != '3.7') && (matrix.os != 'macos-13' || matrix.python-version != '3.7')
4131
uses: actions/setup-python@v4
4232
with:
4333
python-version: ${{ matrix.python-version }}
4434

4535
- name: Set up Python tools
4636
run: |
47-
python -m pip install --upgrade pip
48-
pip install --upgrade cython numpy
49-
pip install -e .[dev]
37+
make dev
5038
5139
- name: Downloads dependencies data
5240
run: |
5341
python scripts/get_data.py qlib_data --name qlib_data_simple --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn
5442
5543
- name: Install Lightgbm for MacOS
56-
if: ${{ matrix.os == 'macos-13' || matrix.os == 'macos-latest' }}
44+
if: ${{ matrix.os == 'macos-13' || matrix.os == 'macos-14' || matrix.os == 'macos-latest' }}
5745
run: |
5846
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Microsoft/qlib/main/.github/brew_install.sh)"
5947
HOMEBREW_NO_AUTO_UPDATE=1 brew install lightgbm

‎.gitignore

+2-1
Original file line numberDiff line numberDiff line change
@@ -48,4 +48,5 @@ tags
4848
*.swp
4949

5050
./pretrain
51-
.idea/
51+
.idea/
52+
.aider*

‎MANIFEST.in

+6-1
Original file line numberDiff line numberDiff line change
@@ -1 +1,6 @@
1-
include qlib/VERSION.txt
1+
exclude tests/*
2+
include qlib/*
3+
include qlib/*/*
4+
include qlib/*/*/*
5+
include qlib/*/*/*/*
6+
include qlib/*/*/*/*/*

0 commit comments

Comments
 (0)