Description
Before install ms-swift, i have env:
python3.10, vllm==0.6.6.post1, torch==2.5.1, cuda_version is 12
Then in installed ms-swift: pip install ms-swift[all] -U, it showed an error:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. cnstd 1.2.3 requires opencv-python>=4.0.0, which is not installed. transparent-background 1.3.2 requires opencv-python>=4.6.0.66, which is not installed.
It seems not related to ms-swift and vllm, and pip show ms-swift represent the ms-swift==3.1.0, so i just passed this error.
Then i run inference.py(using ms-swift and vllm to load model)
from swift.llm import ModelType, get_vllm_engine
model_type = ModelType.llama3_70b_instruct
vllm_engine = get_vllm_engine(model_type=model_type,
model_id_or_path=model_path,
tensor_parallel_size=args.tensor_parallel_size,
max_num_seqs=args.max_num_seqs,
engine_kwargs={
'enable_prefix_caching': args.enable_prefix_caching,
})
it showed an error:
ImportError: cannot import name 'get_vllm_engine' from 'swift.llm'
After that, i reinstalled vllm pip install vllm -U, it installed the latest vllm ==0.7.1, but the upon ImportError also exist.
Could you guide the best practice of ms-swift and vllm installation???
I checked related issues: not work.