Review autosubmit.job.template use when loading jobs from older versions

[kinow]

Blocker for 4.1.16.

@LuiggiTenorioK found this one while testing master. From Slack:

It happens that the pkl file generated by that version (4.1.16) is not readable by Autosubmit 4.1.11 (it raises a ModuleNotFoundError: No module named 'autosubmit.job.template') which is hard locked as a dependency in the API because is the one able to read all the versions of Autosubmit until now.

A possible action plan:

• Review/think if there's any other way to make it compatible
  • if not, then remove the use of autosubmit.job.template, without deleting that code or tests, then adjust the code to revert just that part
  • bash subshell should be still used in the updated version to make sure we get _STAT files updated (or we'll get intermittent failures, or issues in operationals/other workflows occasionally)
• Write tests to prevent that

---

Tests

We don't have a test like this yet, but I think it's doable to:

• Create a dummy workflow (maybe use the default dummy workflow of AS?)
• For a list of AS versions dating back to something like 4.0.98 repeat: create, run, then save pkl as as-$version.pkl
• Save all pkl files produced in a folder under our integration tests folder

TODO: check with @dbeltrankyl & @LuiggiTenorioK best strategy here to complete these tests, we could load older pickle files with the current version, or have a container where we pip install autosubmit==$VERSION, and try to load all pickle files.

[kinow]

p.s. if this is reverted, inform @dbeltrankyl about the revert (or better, maybe apply the fix to Dani's branch if that's simple).

[LuiggiTenorioK]

We don't have a test like this yet, but I think it's doable to:

• Create a dummy workflow (maybe use the default dummy workflow of AS?)
• For a list of AS versions dating back to something like 4.0.98 repeat: create, run, then save pkl as as-$version.pkl
• Save all pkl files produced in a folder under our integration tests folder

I was thinking of doing something similar. However, we don't have to recreate pkl files of already released versions; just the one that is in master has to be generated every time.

Also, more than a test, this feels more like a compatibility report since it doesn't test production code. Then, the only code snippet that needs to pass successfully to mark it as compatible is:

with open(pkl_path, "rb") as f:
    return pickle.load(f, encoding="latin1")

The final result could be an N x N table that could answer: i row version can read j column version?

[LuiggiTenorioK]

I'll start working on this compatibility report and try to automate it in a github action to output the result as an artifact.

[kinow]

I'll start working on this compatibility report and try to automate it in a github action to output the result as an artifact.

Oh, if you could do that, it will be awesome! @mcastril , @dbeltrankyl I think a test like this will save us a lot of time and headaches in development/release process/deploying in DestinE/EDITO/etc.

Thanks @LuiggiTenorioK !!!

[LuiggiTenorioK]

Btw, I was listing the versions for the compatibility report, and I remembered that there were some versions that were released but failing. Just by looking at the release date (close ones), I guess those versions are:

• 4.1.7 (fixed by 4.1.8)
• 4.0.102, 4.0.103, 4.0.104 (fixed by 4.0.105)
• 4.0.93, 4.0.94 (fixed by 4.0.95)

I have seen more, but I think we should yank failed releases to have a trace of which versions are not supposed to be used. Wdyt? @kinow @dbeltrankyl

[LuiggiTenorioK]

After a quick (and a little manual) check. I've found that the current version in master fixed compatibility problems that appeared in version 4.1.12. Probably, the best solution is to upgrade the version used in the API to the incoming version of Autosubmit.

(columns are the autosubmit version used to read, rows are the version used to generate the pkl file)

@kinow it seems that there will be no need to revert changes for now. Even so, I'll continue inspecting for issues 🐛

[LuiggiTenorioK]

Really bad news! I was testing using the master branch version of Autosubmit inside the API, and I found that it is not able to read pkl files from 4.1.11 or lower. This happens because this code snippet is in the current Job class:

    @property
    @autosubmit_parameter(name='status')
    def status(self):
        return self._status

This property function didn't exist in version 4.1.11 since the status was obtained directly from the status property without getting it from the private _status property, which didn't exist yet.

So, when using the master branch version of Autosubmit to read a 4.1.11 pkl file, it will try to get data from the _status property, which didn't exist in the 4.1.11 Job object, raising an AttributeError: 'Job' object has no attribute '_status'.

I'll update the compatibility report to also test the property getters, since this is also an issue that cannot be solved by adding more code to the API.

@kinow it seems that we will need to do the revert you mentioned after all.

[LuiggiTenorioK]

Here is the compatibility report with the stricter test:

The easier solution should be to make Autosubmit 4.1.16 generated pkl files readable when having Autosubmit 4.1.11 installed 😞

[LuiggiTenorioK]

@kinow after a deep dive into Python's pickle code and documentation. I managed to do an ad-hoc patch to read the newest pkl file in this PR: BSC-ES/autosubmit-api#228.

However, this works just like a band-aid, and replacing the pkl file with a DB is still a priority for future versions.

[dbeltrankyl]

What is the status of this one? @LuiggiTenorioK , @kinow

[LuiggiTenorioK]

I already merged BSC-ES/autosubmit-api#228, which makes it possible to read the pkl files of the newest version. However, I'll need to do a final check before releasing 4.1.16 just to be sure. So, let me know when the release is going to be, to do that check.