[RFE] Rewrite rpmautospec in a compiled language

[sgallagher]

rpmautospec was rewritten to run inside the mock environment for several good reasons, not least of which being that it's the only realistic way to ensure that the version of rpm that it's using for parsing matches the version on the target OS. However, as things have progressed, we've discovered several problems.

1. rpmautospec needs to be re-bootstrapped for every new Python release
2. Every new dependency makes the bootstrap more complicated
3. The dependency on Python and numerous Python packages makes it difficult to create rpmautospec for new EPEL releases

This is largely solvable by rewriting rpmautospec in a compiled language, consuming libgit2 (possibly bundled, to avoid bootstrapping issues due to its erratic backwards-compatibility history) and librpm.

I propose that we make a plan for "rpmautospec 1.0" to be written in C, since that will require no additional bindings for libgit2 or librpm and aim to implement this no later than May 2025 (so as to avoid the Python 3.14 bootstrapping problem).

cc @hroncok @praiskup

[hroncok]

Wouldn't a memeory-safe language be a better choice for a new project in 2025?

Even if it is in Rust and uses several dependencies, those are statically compiled and the tools would still have no (or very little) runtime dependencies. There is a precedence for using Rust for such things, https://github.com/keszybz/add-determinism.

There's https://src.fedoraproject.org/rpms/rust-git2 and possibly https://github.com/rpm-software-management/librpm.rs (which does not seem quite there yet)

cc @decathorpe @keszybz

[hroncok]

Alternatively, if this thing would be written as a single no-dependency Python script possibly using ctypes to interact with libgit2 and librpm, it would still solve the problem + it might be easier to hack on by casual contributors.

[decathorpe]

"rpmautospec 1.0" to be written in C

[sgallagher]

Alternatively, if this thing would be written as a single no-dependency Python script possibly using ctypes to interact with libgit2 and librpm, it would still solve the problem + it might be easier to hack on by casual contributors.

Maybe easier to hack on, sure. I don't think it entirely solves the original problem, however. It still creates the potential for a bootstrapping issue with Python itself (or we disallow the python interpreter from using rpmautospec).

Regarding the use of rust: while I'm not necessarily opposed to it, there's a huge learning curve there; I'm currently volunteering to do this rewrite, but I am not versed in rust. I cannot commit to having the necessary cycles to both learn a new language and completely rewrite this project in it.

[hroncok]

It still creates the potential for a bootstrapping issue with Python itself.

It doesn't. When we build Python 3.N+1 as python3 for the first time, we still have Python 3.N as python3.

I'm currently volunteering to do this rewrite...

That was not obvious to me from I propose that we make a plan for "rpmautospec 1.0" to be written in C...

...but I am not versed in rust. I cannot commit to having the necessary cycles to both learn a new language and completely rewrite this project in it.

I understand this argument. However, it also means you will be the only one available to maintain this¹ . While if we actually choose the Python route, perhaps more folks can contribute (I know I certainly can).

Footnotes

1. Anecdotal evidence: rpminspect is written in C and is mostly only ever contributed to by the author. ↩

[hroncok]

Either way, we should probably also figure out how to provide the importable Python module that is now available in rpmautospec-core and used by various Fedora Python projects (mock, rpkg, copr) trough the Python interface.

Would it continue to exist (and not be used by the new "standalone" rpmautospec at all)?

Similarly, would this "standalone" rpmautospec offer perks for packagers (such as the convert command or bash completion) or would that live in a separate Python project built on top of it?

[sgallagher]

It still creates the potential for a bootstrapping issue with Python itself.

It doesn't. When we build Python 3.N+1 as python3 for the first time, we still have Python 3.N as python3.

Sorry, I meant there's still a bootstrapping loop with Python 3.N+1 and python-rpmautospec. While it's not the end of the world, it would be nice if we could use %autorelease and %autochangelog in rpmautospec itself.

I'm currently volunteering to do this rewrite...

That was not obvious to me from I propose that we make a plan for "rpmautospec 1.0" to be written in C...

Yeah, sorry. That information was lost in rewrites as I reworked the initial description.

[hroncok]

It still creates the potential for a bootstrapping issue with Python itself.

It doesn't. When we build Python 3.N+1 as python3 for the first time, we still have Python 3.N as python3.

Sorry, I meant there's still a bootstrapping loop with Python 3.N+1 and python-rpmautospec.

I am sorry, but I still don't see it.

While it's not the end of the world, it would be nice if we could use %autorelease and %autochangelog in rpmautospec itself.

We could.

[sgallagher]

Either way, we should probably also figure out how to provide the importable Python module that is now available in rpmautospec-core and used by various Fedora Python projects (mock, rpkg, copr) trough the Python interface.

rpmautospec-core is already pure-python and should have no other dependencies. All it does is, basically, run a regex across the specfile to see if %autorelease or %autochangelog appears in the file anywhere. It doesn't even use RPM parsing to do this. I don't see any real need to rewrite it at all.

Would it continue to exist (and not be used by the new "standalone" rpmautospec at all)?

Its behavior is so trivial that I'd probably reimplement it in the compiled rpmautospec and leave the python module there for convenience.

Good questions!

Similarly, would this "standalone" rpmautospec offer perks for packagers (such as the convert command or bash completion) or would that live in a separate Python project built on top of it?

I'll confess that I haven't made a final determination on that yet; it may come down to the available time I have. I may just write librpmautospec in C (or, I guess Rust if I'm feeling ambitious...) and then split off the existing CLI functionality into a python-rpmautospec-cli tool to avoid rewriting it. That split was already being considered to trim dependencies, so it might be best to do that first in any case.

[sgallagher]

It still creates the potential for a bootstrapping issue with Python itself.

It doesn't. When we build Python 3.N+1 as python3 for the first time, we still have Python 3.N as python3.

Sorry, I meant there's still a bootstrapping loop with Python 3.N+1 and python-rpmautospec.

I am sorry, but I still don't see it.

python-rpmautospec has a runtime requirement on python(abi) = 3.13. It also uses %autorelease in the Release: field, which means that rpmautospec-core will tell mock to pull in rpmautospec. Once we build python3.14 into a buildroot, we won't be able to satisfy the python(abi) = 3.13 dependency that would be required to rebuild python-rpmautospec against python3.14, unless I'm missing something. (Yes, we could do a bootstrap dance and build it once without %autorelease, but I'd prefer to avoid special handling where possible).

[hroncok]

python-rpmautospec has a runtime requirement on python(abi) = 3.13.

Yes, it currently has. If we rewrite it to standalone Python script to be installed in /usr/bin/ (as if it was written in C or Rust or whatever), it wouldn't have.

It also uses %autorelease in the Release: field, which means that rpmautospec-core will tell mock to pull in rpmautospec. Once we build python3.14 into a buildroot, we won't be able to satisfy the python(abi) = 3.13 dependency that would be required to rebuild python-rpmautospec against python3.14, unless I'm missing something. (Yes, we could do a bootstrap dance and build it once without %autorelease, but I'd prefer to avoid special handling where possible).

Yes, this is in essence what I reported as #169

We are trying to find a solution. You said "rewrite it in C", I said "rewrite it in Rust or a standalone Python script". All of the three mentioned solutions remove the python(abi) requirement, solving #169