Open .ksy files directly

[abitrolly]

http://doc.kaitai.io/lang_python.html doesn't specify how exactly to import .ksy files from Python. I would expect something like this.

import kaitai
Gif = kaitai.compile('git.ksy')
g = Gif.from_file('some.gif')

If I understand it right, direct loading of .ksy files is not supported, because .ksy parser is not written in Python.

[dgelessus]

You're correct, there's no built-in support for importing .ksy files directly from Python. The usual workflow is to compile the .ksy files into Python modules using ksc (as described in the KS Python notes) and put the compiled Python modules into a location where they can be imported - if you're writing a library, into a package in your library code, or if you're writing a script, in the same directory as the script.

@KOLANICH has written a library called kaitaiStructCompile.py, which (among other things) lets you import .ksy files as Python modules without manually compiling them first. When you import a KSY using kaitaiStructCompile, it automatically calls ksc to generate the Python module, and imports that. But I would only recommend this for code for your own personal use, not for libraries/tools meant to be used by others - otherwise every user of your code needs to have the correct version of ksc installed.

[abitrolly]

@dgelessus good start. It is technically feasible to ship compiler binary in a Python wheel. I made a fork in https://github.com/abitrolly/ksykaitai with a roadmap, but no ETA how it could be done.

[abitrolly]

Two hours later still can't get it running.

$ python ksykaitai.py
Traceback (most recent call last):
  File "ksykaitai.py", line 18, in <module>
    compile('../kaitai_struct_visualizer/squashfs_superblock.ksy')
  File "ksykaitai.py", line 9, in compile
    backend = clibackend.init(ICompilerModule,
  File "/home/anatoli/h/snapsquash/ksykaitai/kaitaiStructCompile/backend/cmdline.py", line 87, in init
    class CLIPrefsStorage(ICompilerModule.IPrefsStorage):
AttributeError: type object 'ICompiler' has no attribute 'IPrefsStorage'

7566f4e committed https://github.com/abitrolly/ksykaitai

@KOLANICH I committed backed to repo to avoid installation hassle, and yet it doesn't work.

[KOLANICH]

type object 'ICompiler' has no attribute 'IPrefsStorage'

IPrefsStorage is in ICompiler module, not class (the module also contains an eponymous class).

[KOLANICH]

It is technically feasible to ship compiler binary in a Python wheel.

It is an extremily bad idea. KSC depends on JVM. You are not going to also ship JVM as a wheel, are you?

So it should be handled either manually, or by a system package manager.

If you want to make Pip able to install JVM using system package manager, you may want to look at open-source-ideas/ideas#50

[abitrolly]

It is an extremily bad idea. KSC depends on JVM. You are not going to also ship JVM as a wheel, are you?

Going. Exactly. :D https://github.com/dgiagio/warp#quickstart-with-java

IPrefsStorage is in ICompiler module, not class (the module also contains an eponymous class).

So how to import ICompiler module? The class import in __init__.py shadows it.

[KOLANICH]

Going. Exactly

🤦‍♂️

So how to import ICompiler module? The class import in init.py shadows it.

1. yes, it shadows. Probably I should rename that module.
2. It is not meant to be imported. It is injected into selected backends. The backends are discovered from entry points and usually are not needed to be constructed manually. If you created an alternative impl of a backend (i.e. using a server + memory maps, or maybe even modules for other impls of KSC, such as the ones in rust and nim) and registered it properly using entry points mechanism, it will also work (the backends have priorities, see the code of backendSelector for the logic on selecting a backend, a user can also explicitly select the backend he needs).

[abitrolly]

Going. Exactly.

kaitai-io/kaitai_struct#807

yes, it shadows. Probably I should rename that module.

Answering your style - you should remove that module and the module with exception. Just saying. 🙊

Anyway, fixed in abitrolly/ksykaitai@293c892

Right now the problem I am dealing with is the path to compiler binary. The Python lib expects it to be a directory structure, and I am running the compiler using a wrapper script (which crates secure environment).

[KOLANICH]

The Python lib expects it to be a directory structure, and I am running the compiler using a wrapper script (which crates secure environment).

You may be able to set KAITAI_STRUCT_ROOT to point inside that environment (is it still a dir?)

[abitrolly]

With the latest commit I successfully compile my .ksy from Python (after downloading compiler, unpacking to current dir and stripping version suffix). The code works! Although currently I am not sure how it is better than a direct subprocess call. :D

[abitrolly]

Added automatic import of generated file.

Also committed data to be used as test and proof of concept. Except compiler, because it is too big. The rest of the work is removing hardcoded filenames and creating a wheel with compiler bundled for amd64 Linux and accessing bundled data files with https://importlib-resources.readthedocs.io/en/latest/index.html