Rewrite and segregation of H5Parm

[Joshuaalbert]

Hi Fra,

I have come across many bugs in the recent while with losoto.h5parm. Partly it's from py3 compatibility and partly from pytables updates and partly I found more efficient ways of doing things. For example, trying to set the soltab values was occasionally zeroing out all the values in the soltab. Occasionally, the file would get corrupted. Deleting solset and soltabs was not working. The BLOSC compression is not used. This actually results in far faster IO if it's used at some computational cost to save data. I have rewritten it from scratch, keeping the same conventions as you use, and depreciating some variable names. It is also extended to use astropy for unit conversion, better handling of variable dimension order, has a context manager (so you can use with ...: to auto close), and added other features. I'm opening this issue to suggest a segregation of the h5parm code into it's own repo, and I can put my version there. That way if a project wants to use it without all the numerical aspects of losoto it's simple. They only need pytables and astropy installed. What are your thoughts on this? The rewrite can be seen here: https://github.com/Joshuaalbert/bayes_filter/blob/master/bayes_filter/datapack.py

[revoltek]

Hi Josh, that's a very impressive effort, and you are definitely right that there's a number of things that can be re-organised/optimized in that lib. It has been in my todo list for a long time. I quickly checked your new libs and we need to discuss how backward compatible it is. We can try to make it the new h5parm.py but we need to check that all operations works fine with it. Shall we plan an extensive discussion on that during the lofar meeting?

[Joshuaalbert]

That's a good plan. Let's meet then. If you have a set of must-haves that you could send me before hand that would be best. Cheers, Josh

…

On 23.4.2019 14:15, Francesco de Gasperin wrote: Hi Josh, that's a very impressive effort, and you are definitely right that there's a number of things that can be re-organised/optimized in that lib. It has been in my todo list for a long time. I quickly checked your new libs and we need to discuss how backward compatible it is. We can try to make it the new h5parm.py but we need to check that all operations works fine with it. Shall we plan an extensive discussion on that during the lofar meeting? -- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or mute the thread [2]. Links: ------ [1] #89 (comment) [2] https://github.com/notifications/unsubscribe-auth/ADQ676ATQ4ZMSO56WQAAUODPR34WJANCNFSM4HHWP6DA

[Joshuaalbert]

Which date seems best to do a session to work this out? I'm flexible.

[Joshuaalbert]

Requires unit tests to see which functionality breaks when we change things. Closing. Made milestone #91