Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow allow prefixing deltas and objects with different hostname/URL and be stored as top level files on that URL #1699

Closed
nurupo opened this issue Aug 3, 2018 · 5 comments
Closed

Allow allow prefixing deltas and objects with different hostname/URL and be stored as top level files on that URL #1699

nurupo opened this issue Aug 3, 2018 · 5 comments

Comments

@nurupo
Copy link

nurupo commented Aug 3, 2018 •
edited
Loading

I want my ostree to be on https://example.com/repo/, but the objects and deltas -- the most heavyweight files -- on https://cnd.example.com/repo/. Moreover, I want everything on https://cnd.example.com/foobar/ to be a file with no directory hierarchy, e.g. what would have been https://example.com/repo/objects/01/cf30656a692c23c76e73bae7950bdcc7ff6316e7cb7ebcd7a337c1ac45af57.dirtree should be something along the lines of this: https://cnd.example.com/foobar/objects_01_cf30656a692c23c76e73bae7950bdcc7ff6316e7cb7ebcd7a337c1ac45af57.dirtree.

So I want an option to specify objects and deltas prefix URL, potentially different host, and an option to flatten the directory structure so that they are top level files on that URL.

This can be very easy done server-side with a custom URL rewriting rule that does 301 HTTP redirect, but I host my repo on GitHub Pages, which doesn't provide any way to do server-side redirects. You can only do client-side redirects on GitHub Pages with <!DOCTYPE html><head><meta http-equiv="refresh" content="0;URL=https://cnd.example.com/foobar/objects_01_cf30656a692c23c76e73bae7950bdcc7ff6316e7cb7ebcd7a337c1ac45af57.dirtree"></head><body></body></html>, but those work only in browsers, neither wget nor curl follow those.

Anyway, my actual use case is to host a Flatpak repo on GitHub such that small metadata files would be hosted on GitHub Pages static website hosting, but heavyweight files like objects and deltas would be hosted on GitHub Releases. I want this due to limitations: GitHub Pages has a limit of 100MB per file, total repository size not should not exceed 1GB, and it allows only 100GB/month bandwidth, when GitHub Releases has a limit of 2GB per file, and no limit for the total size of all files combined and no limit on the bandwidth, so it makes more sense to host the heavyweight objects and deltas on Releases over Pages.

Just to give an example URLs closer to real-world use, given my ostree is at http://nurupo.github.io/test-repo/repo/, instead of ostree looking for http://nurupo.github.io/test-repo/repo/objects/01/cf30656a692c23c76e73bae7950bdcc7ff6316e7cb7ebcd7a337c1ac45af57.dirtree I want it to look for https://github.com/nurupo/test-repo/repo/releases/download/some-git-tag/objects_01_cf30656a692c23c76e73bae7950bdcc7ff6316e7cb7ebcd7a337c1ac45af57.dirtree.

I wanted to open this issue against Flatpak initially, but then I realized that there is little they can do about it given that they just use OSTreee for the repository and it's something that OSTree would need to support itself.
@nurupo
Copy link
Author

nurupo commented Aug 11, 2018

Any input on this? It's somewhat related to #1541, I also want to host objects/ and deltas/ on a separate CDN host, except I have no luxury of issuing HTTP 302 redirect as I have no control over the webserver as I use static page hosting (GitHub Pages), so I need to somehow signal the client that objects/ and deltas/ are elsewhere. (I also need an option for flattened file paths as the GitHub Releases CDN doesn't allow directories).

@cgwalters
Copy link
Member

Offhand it sounds potentially reasonable; not clear exactly how you're proposing this be configured though. Unless your app is huge though you can probably get away with just hosting "from scratch" deltas for new versions right? Also I guess integration with github releases could go all the way to libostree having some awareness of parsing the github release metadata API.

@nurupo
Copy link
Author

nurupo commented Aug 15, 2018

Offhand it sounds potentially reasonable; not clear exactly how you're proposing this be configured though.

One way to do this might be to add couple of options to repo/config file, e.g.

object_flat_path=true
deltas_flat_path=true
object_url_prefix=https://github.com/nurupo/test-repo/repo/releases/download/some-git-tag/
deltas_url_prefix=https://github.com/nurupo/test-repo/repo/releases/download/some-git-tag/

so that whenever someone tries to use my repo, their OSTree client program will see those config options set and will know that it needs to flatten the

objects/01/cf30656a692c23c76e73bae7950bdcc7ff6316e7cb7ebcd7a337c1ac45af57.dirtree

to something like

objects_01_cf30656a692c23c76e73bae7950bdcc7ff6316e7cb7ebcd7a337c1ac45af57.dirtree

and then also prefix it with whatever is in object_url_prefix option, making it

https://github.com/nurupo/test-repo/repo/releases/download/some-git-tag/objects_01_cf30656a692c23c76e73bae7950bdcc7ff6316e7cb7ebcd7a337c1ac45af57.dirtree

Also I guess integration with github releases could go all the way to libostree having some awareness of parsing the github release metadata API.

I don't think the OSTree software needs to use any GitHub API in order to implement the suggested. Although we are trying to work around GitHub limitations here, OSTree doesn't even have to be aware that it's GitHub, if some other service has the same limitations it would work with that too.

@nurupo
Copy link
Author

nurupo commented Aug 15, 2018

Here is an example of how I expect it to work.

I have two repositories on GitHub. One is for a software called FooBar, and another is for Flatpak repository that uses GitHub Pages. Whenever I make a new FooBar software release, I update the Flatpak repo locally with the new version of FooBar, run a bash script to flatten objects and deltas paths the way OSTree expects, create a new GitHub Release in the Flatpak repo, upload the flattened objects and deltas there, update the URL prefixes in the repo/config file to point to that new GitHub Release, push my local Flatpak repo to GitHub with all the changes (objects anddeltas dirs are empty or don't even exist) and I delete the previous GitHub Release the Flatpak repo had before (so there is just one GitHub Release containing the objects and deltas ). If I wanted, could automate this process with a python script that interacts with GitHub API - to create new release, flatten files, upload them to the created release, etc. and even automate it further so that it would run on Travis-CI every time I push a new git tag to FooBar repository.

@nurupo nurupo mentioned this issue Oct 19, 2018
Open
@nurupo nurupo changed the title [Feature Request] Allow prefixing deltas and objects with different hostname/URL and be stored as top level files on that URL [Feature Request] Allow hosting ostree on GitHub Releases (allow prefixing deltas and objects with different hostname/URL and be stored as top level files on that URL) Mar 7, 2020
@nurupo nurupo changed the title [Feature Request] Allow hosting ostree on GitHub Releases (allow prefixing deltas and objects with different hostname/URL and be stored as top level files on that URL) Allow allow prefixing deltas and objects with different hostname/URL and be stored as top level files on that URL Mar 7, 2020
@nurupo
Copy link
Author

nurupo commented Mar 7, 2020

Reworded the issue better in #2020.

@nurupo nurupo closed this as completed Mar 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cgwalters @nurupo