Update Feature

[Korusuke]

So after trying a bunch of things like diff-match-patch, normal diff and vaguely trying regex following is the technique that works best at least for situations that I tried.

Let's say we have a Portfile which we need to update and we have decided that we will update version and checksums field.
So first we extract that fields from existing Portfile (yet to be automated).
and make a file which looks like this:

version             0.7

checksums           sha256  7de9a0cc9d55c175b611ed748ee7ea572ec308411934769e2e647d1d26ba2eea \
                    rmd160  6dc1b42ac5760f8225c81b1cd9539332b33b1b4h \
                    size    24223

and then we generate a new file which looks like this:

version             0.9

checksums           sha256  b967cda25e8e5a634a450886e98ebeaf5aa51220e8888a17a34d78d001f96035 \
                    rmd160  aac9b9c2f52ac75eec0700ac996a87f0151923f9 \
                    size    25969

Notice the blank line at end of file, that is apparently important for this to work.

Now we generate a diff for above two files and then apply patch on the original file.
This even updates a file that maybe modified and has different order of fields than what upt usually has.

For example from this:

# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4

PortSystem          1.0
PortGroup           python 1.0

name                py-upt
version             0.7
revision            0

maintainers         {@korusuke somaiya.edu:karan.sheth} openmaintainer
description         Package software from any package manager to any distribution
long_description    ${description}

platforms           darwin
supported_archs     noarch

homepage            https://framagit.org/upt/upt
master_sites        pypi:[string index ${python.rootname} 0]/${python.rootname}
distname            ${python.rootname}-${version}

license             BSD
checksums           sha256  7de9a0cc9d55c175b611ed748ee7ea572ec308411934769e2e647d1d26ba2eea \
                    rmd160  6dc1b42ac5760f8225c81b1cd9539332b33b1b4h \
                    size    24223

python.versions     37

if {${name} ne ${subport}} {
    depends_lib-append \
                    port:py${python.version}-spdx-lookup \
                    port:py${python.version}-setuptools

    depends_run-append \
                    port:py${python.version}-upt-macports \
                    port:py${python.version}-upt-cpan \
                    port:py${python.version}-upt-pypi

    test.run        yes
    test.cmd        ${python.bin} -m unittest
    test.target
    test.env        PYTHONPATH=${worksrcpath}/build/lib

    post-destroot {
        set docdir ${prefix}/share/doc/${subport}
        xinstall -d ${destroot}${docdir}
        xinstall -m 0644 -W ${worksrcpath} README.md LICENSE CHANGELOG \
            ${destroot}${docdir}
    }

    livecheck.type  none
}

to this:

# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4

PortSystem          1.0
PortGroup           python 1.0

name                py-upt
version             0.9
revision            0

maintainers         {@korusuke somaiya.edu:karan.sheth} openmaintainer
description         Package software from any package manager to any distribution
long_description    ${description}

platforms           darwin
supported_archs     noarch

homepage            https://framagit.org/upt/upt
master_sites        pypi:[string index ${python.rootname} 0]/${python.rootname}
distname            ${python.rootname}-${version}

license             BSD
checksums           sha256  b967cda25e8e5a634a450886e98ebeaf5aa51220e8888a17a34d78d001f96035 \
                    rmd160  aac9b9c2f52ac75eec0700ac996a87f0151923f9 \
                    size    25969

python.versions     37

if {${name} ne ${subport}} {
    depends_lib-append \
                    port:py${python.version}-spdx-lookup \
                    port:py${python.version}-setuptools

    depends_run-append \
                    port:py${python.version}-upt-macports \
                    port:py${python.version}-upt-cpan \
                    port:py${python.version}-upt-pypi

    test.run        yes
    test.cmd        ${python.bin} -m unittest
    test.target
    test.env        PYTHONPATH=${worksrcpath}/build/lib

    post-destroot {
        set docdir ${prefix}/share/doc/${subport}
        xinstall -d ${destroot}${docdir}
        xinstall -m 0644 -W ${worksrcpath} README.md LICENSE CHANGELOG \
            ${destroot}${docdir}
    }

    livecheck.type  none
}

[mojca]

CC-ing also @satraul.

If you plan to update just the two fields, there's also his new port bump tool available.
What I would like to see, if possible, would also be the "diff" of dependencies between the existing and the new version (I hope one could get that list from pypi?). Maybe it would be sufficient to output just some kind of log for now, since the information is already pretty unreliable.

[Korusuke]

So after thinking about how the interface between upt and backend will be like and what will go where following is what I have come up with:

Frontend parses upt ->
Backend reads the existing file ->
Backend extracts information to be updated (example for macports for now it can be version line and checksums line)....When I say extract means not the value but just the lines itself as we don't need the exact info as demonstrated above ->
Backend generates update file using a update template ->
upt generates patch file using both of this ->
upt passes patch file, diff file, new file to backend ->
Backend saves required files.

So in psuedo code something like:
upt-core: (Sorry about the naming)

class Backend(object):
    def update_package(self, path_to_file, pkg):
        versions = self.package_versions(self.name)
        if versions.latest == self.version:
            return
        old = self.read_old_file(path_to_file)
        to_update = self.extract_update(old)
        updated = self.extract_update(pkg)
        file_diff = diff(old, pkg)  # os diff call here
        patch = diff(to_update, updated)  # os diff call here
        new = apply_patch(patch)  # os patch call here
        self.save_update(patch, file_diff, new)
       

Backend will have to implement :
extract_update()
read_old_file()(this can be done in upt-core but I am not sure about it)
save_update upt-core cannot handle this due to naming preference of different backend

and also this whole function can be overridden by backends in case they want to do something different, for example as mojca suggested for MacPorts we can use port bump tool which is already doing the same job using regex and in an interactive manner.

What do you think about this?

[reneeotten]

I think the general outline looks fine; at least from my perspective, but I'll defer to @Steap for everything touching upt itself and about what should be done in the core/backend.

Ideally, I would prefer to skip all the intermediate "extract_update" stuff and just do

1. read_old_file
2. generate_file_by_upt
3. make_diff_or_patch_directly_whatever_fields_we_want

but again, I don't know how feasible that is. I'll need to play around with some of the tools like diff-match-patch a bit more (soon-ish I hope) and see how that works.

[Steap]

Hello,

On 2019-07-11 19:33, Karan Sheth wrote: So after thinking about how the interface between upt and backend will be like and what will go where following is what I have come up with: Frontend parses upt -> Backend reads the existing file -> Backend extracts information to be updated (example for macports for now it can be version line and checksums line)....When I say extract means not the value but just the lines itself as we don't need the exact info as demonstrated above -> Backend generates update file using a update template ->

So basically these are the small files that only contain (for now) "versions" and "checksums", right?

upt generates patch file using both of this -> upt passes patch file, diff file, new file to backend -> Backend saves required files.

This honestly seems overly complicated. We need to extract data from an existing Portfile, do some work "on the side", re-inject everything into the backend and have it handle a merge. It seems difficult to implement (for MacPorts, let alone other backends) and error-prone. Or maybe I'm misunderstanding something?

[Steap]

If you plan to update just the two fields, there's also his new port bump tool available.

What can port bump do, exactly?

What I would like to see, if possible, would also be the "diff" of dependencies between the existing and the new version (I hope one could get that list from pypi?). Maybe it would be sufficient to output just some kind of log for now, since the information is already pretty unreliable.

So, my initial idea for update was to parse CPAN/PyPI/Rubygems/etc. for 2 different versions and generate a high-level diff ("these are the new requirements", "these requirements are no longer needed", etc.). At first, it seemed doable for PyPI:

https://pypi.org/pypi/requests/0.10.0/json <- an old version
https://pypi.org/pypi/requests/json <- the current version

It seemed doable for CPAN as well:

https://fastapi.metacpan.org/v1/release/NEILB/AppConfig-1.70 <- an old version
https://fastapi.metacpan.org/v1/release/AppConfig <- the current version

But I do not think Rubygems (and other frontends to be added in the future) will provide info about all versions of a package.

Also, it should be noted that some info is missing from these REST API. For instance, the test requirements for "requests" do not show up on PyPI (for various reasons that I won't go into here).

Wouldn't the best way to update just be to create a "Portfile.new" file, and then have the user run diff/git diff/whatever?

[mojca]

port bump basically does just the following (for any given port):

• check which is the latest version
• download that version, calculate checksums
• replace version and checksums (by searching for the exact checksum string, and replaces just sha256 itself, no spacing or anything else)

I wrote this and forgot to hit send before I read your replies:

I just want to reiterate: updating just port version and checksums is something that already port bump has done (testing and feedback greatly appreciated), and is relatively easy to do in general, but of course you miss a whole bunch of stuff.

What I would really like to see is at least some rudimentary support to tell you which dependencies changed between the two versions. I don't mind as much if I need to add those manually for a moment, but figuring out the exact dependencies is really the most time-consuming step which I very strictly :) always ignore when I need to do some mass update. We should be able to extract both old and new dependencies from the two versions, compare them, and at least tell the user:

• what officially changed inbetween based on official package data
• how dependencies declared upstream differ

[Korusuke]

But I do not think Rubygems (and other frontends to be added in the future) will provide info about all versions of a package.

The v2 of rubygems api seems to support this:
https://rubygems.org/api/v2/rubygems/acts_as_list/versions/0.9.18.json
https://rubygems.org/api/v2/rubygems/acts_as_list/versions/0.9.19.json

But yeah you are right maybe all frontends doesn't support this.

I agree that the whole process is a bit complicated so maybe we can divide that into two parts and do the following:

• Parse new version and save as 'Filename'.new
• Save a diff file for the same (I guess we should do this as this is something everyone will do)
  So all of the above can be done easily in all backends.

Now optionally backends can implement apply_update feature which is backend specific:

• Example for macports that will implement port bump for now.
• and if we want to and "can" update other things via some different way for example the diff method I described here

Is this fine?

[reneeotten]

I agree that Cyril initial idea to "generate a high-level diff" would be a good one, but indeed probably not doable for all frontends. Generating a new Portfile can and should be done for sure, in addition to making a diff between the two files; no problem there.

I also like Karan's idea of implementing an optional "apply_update" function (or whatever name) in the backend to do something more specific. I am not a fan of doing port bump for the MacPorts backend, that might for now work do update the version and checksums but is not scalable to anything more complicated for dependencies and such. So I think we need to come up with a way to "swap" the information into the new Portfile and should try to get that to work for simple things like "version/checksums" first and once that works it should be possible to extent.

Again, I don't like making separate intermediate diff files to patch the Portfile. This should be something people have done before and I would suggest you to investigate things like diff-match-patch or other approaches already being used.

@Steap can we first agree on how/what should be implemented in upt and what should be done in the backend and take it from there?

[Steap]

So, here is an updated (pun indented) idea of how to do things:

1. We get the current version of PKG using the recently introduced backend.package_versions() method. The current version is probably the "highest" version of the returned list.

2. We ask the frontend to parse both the "old version" and the "latest version".

3. We create a "package diff":

class PackageDiff(object):
    def __init__(self, old_pkg:upt.Package, new_pkg:upt.Package):
        self.version = new_pkg.version

    def new_requirements(self):
        pass

    def deleted_requirements(self):
        pass

upt.PackageDiff.new_requirements would be similar to upt.Package.requirements, but would only return the new requirements.

TODO: figure out what else we might want to access from a PackageDiff.

4. upt.Backend has a "create_package" method, it should have an update package as well.

class MacPortsBackend(upt.Backend):
    def update_package(self, pkgdiff:upt.PackageDiff):
        ...

Now, it's up to the backend to do the right thing. Can the backend easily make the required modifications? If not, maybe the abstraction I've proposed is not really good.

[Korusuke]

Sounds great, I have a few doubts(not even sure they are doubts)...so I will first code this up and if anything goes wrong then I will ask.

[Korusuke]

Ok so all of what @Steap suggested is done more or less...now before coming to upt-macports.
Till now I haven't used file_path input which we had discussed to take as input for --update.

For upt-macports backend, this are the things I think we will do:

• Save the new package file as Portfile.new to same path with --output is enabled else just print it.
• Print new requirements and the deleted ones if any
• Run port bump on existing portfile

Anything else that you think we should be doing?

[reneeotten]

@Steap I fully agree with your latest proposal! I don't understand though how this differs from you first "high-level diff" that seemed not supported by all the frontends. Anyway, if you think this will work I think it's a great way of doing this, so let's go for it!

@Korusuke as I said earlier I would prefer not to do port bump on the existing Portfile. This will only work for version and checksums and not for anything else. This is a major, and very important, part of your GSoC project for MacPorts and if it takes time to find a way to do this, this is time well spent.

Of course, we could decide to finalize the pieces needed for upt and start with what you suggested above as an intermediate solution. Sometimes the drawback of such an approach is though that the ideal solution might never get implemented...

[Korusuke]

Ok my bad, I somehow tested diff match patch wrongly or missed out..... but testing again with all the output that current idea is giving, it works unbelievably amazingly well :)
Will push the code tomorrow.

[Steap]

@reneeotten Yes, this is the same "high-level diff" approach. If some frontends do not give us access to all versions of a package, well, too bad. We could always ask the backend to parse their own files if needed, by the way.

Maybe we could improve "port bump" instead of implementing everything in upt-macports? I'm not sure exactly how "port bump" works, though.

[reneeotten]

Maybe we could improve "port bump" instead of implementing everything in upt-macports? I'm not sure exactly how "port bump" works, though.

Perhaps, but it's written in Tcl, which is pretty much voodoo to me ;) Looking at the man-page it will only update the checksums and reset the revision, after updating the version number of a port by the maintainer.

If we first need to update the version number and then run port bump I see no use-case for this when using upt, it appears to me that we have all the required information ourselves already.