Json changes int dictionary indexes to string

[Joelgranados]

I'm having trouble working with settings retrieved from a json file because of thie issue. FYI http://stackoverflow.com/questions/1450957/pythons-json-module-converts-int-dictionary-keys-to-strings. though I can get around this, its worth asking ourselves if Json is really what we need, or would we prefer to cPickle the entire TimeStream object.

[kdm9]

You know my preferences for json in terms of cross-language abilities.

An alternative is MSGPACK. It's more compact, at the cost of human readablity. But very cross platform.

I'm (reluctantly) happy to accept cPickle as the One True Format for now, so long as you're happy to port the entire perstance codebase to something cross plaform when we move the performance critical parts to C++ (which we should start doing once most features have been implemented, in the longer term).

K

[Joelgranados]

For now I address this by comparing with int and string for things that come from json. Its "fixed" for now... I guess :)

[kdm9]

That's a bit hacky IMHO, and it would be good to avoid using anything but the stock pack/unpack routines of whatever we settle on (msgpack, json, pickle, etc...). But i'll leave this closed, it's not important

[Joelgranados]

I know, I know.... I spend way to much time re-opening issues. please bare with me :)

Please correct me if I'm wrong: you are reluctant to use cPickle because we won't be able to access the data once we move the performance critical parts to C++? is this what you meant with your previous comments?

If this is in fact what you meant, please consider the following:

1. Can you give me a concrete example (with the code base we have now) of a place where we would replace python with C/C++ for performance purposes and we would have no other choice but to access the stuff in Json directly?

2. We already have most if not all critical parts in C/C++. Whenever we call cv2 or numpy we move down to very fast code.

3. Lets assume for the sake of argument that there is a case where we need to replace a piece of code with C/C++ and we cannot do it with opencv nor numpy nor scipy nor pandas.... If we are trying to be efficient, why do something as expensive as access a file? Why do it with C/C++ when we can do it in python and pass it on to C/C++.

[kdm9]

My point is less about "we need to optimise and move to C" than it is "we're pushing the timestream (file format) as a standard, others may use R/C/C++/Matlab/Julia/whatever, and we should use cross plaform data standards".

I use optimising to C as an example of the need to be cross platform, you're correct in saying that it's probably not going to need to happen any time soon. But the preference (not requirement) for cross platform data interchange formats still stands.

[Joelgranados]

Have you worked with jsonpickle? If you have, do you think its an option? https://github.com/jsonpickle/jsonpickle.

This could be the middle ground we have been searching for.