Ac kubernetes

[Acribbs]

No description provided.

[Acribbs]

@snsansom @IanSudbery @nickilott @david-cgat @jscaber (and all). This is a big change. I have refactored the code to introduce a new base class called the Executor that is inherited by Slurm, Torque, SGE, Kubernetes and local (it replaces make_runner()). The main aim was to implement kubernetes functonality which I couldnt do with the current framework. I introduced a new class function in base_executor.py that is inherited in executor.py (There is also a kubernets.py which I will move into executor.py before merging). I have tested it locally and on slurm on my end and it works, but if others could give it a go that would be great. It also defaults to LocalExecutor if you dont specify it to run locally, but now I think about it, I think im going to make it raise an error as that keeps it consistent with the current behaviour when it complains if DRMMA API isnt found if its ran for cluster config but is ran locally. Thinking of implementing a lot more functionalities in my free time (when i travel on trains and planes) so would be happy to accept suggestions on what to implement next.

[nickilott]

Thanks Adam for keeping going with cgat-core! I'm happy to test it out on existing pipelines and will let you know here how it goes

[Acribbs]

Planning on merging this soon

[nickilott]

Hey Adam, I just pulled the branch and tried to run a pipeline and got the following error:

File /gpfs3/well/powrie/users/utu691/devel/venv/Python-3.10.8-GCCcore-12.2.0/skylake/lib/python3.10/site-packages/cgatcore/pipeline/executors.py, line 69, in run
self.monitor_job_completion(job_id)
AttributeError: 'SlurmExecutor' object has no attribute monitor_job_completion

I haven'[t had a chance to look into the source of the error in any detail at all but wanted to flag before the merge...

[Acribbs]

Thanks for this! I have no idea why the tests are not picking this up. Will investigate further. May need some more tests to capure this

[Acribbs]

@nickilott This was my fault, I had placeholder comments to include code for creating the monitor_job_completion function but I never did it, not sure how this wasn't picked up in testing, but have explicitly wrote tests to now check this. I tested the code on my end though and still dont get why i didnt get the same error as you. Anyway, its now fixed and should be fine. If you have time to check this out, it would be much appreciated.

[nickilott]

Hi Adam,
I have pulled these changes and will test in the next couple of days if that's ok?

[Acribbs]

yeh no problem, thanks!

[nickilott]

Hi Adam,

I ran a pipeline with this code and it seemed to work fine for me :)

[Acribbs]

Awesome, thanks! I'm going to merge given i now have tests in place for capturing the issue you raised earlier and it works my side.