Move native binaries into separate NuGet package

[bording]

As discussed in #974, here is what I have done so far.

I'm currently hosting a manually created version of the native binaries package on a myget feed, which I've temporarily added as a solution-level package source, which should let package restore find the package without needing anyone to manually add the package source.

There are still several things that need to done still. The primary thing is deciding what to do with how handle the things that were previously being done by the UpdateLibgit2ToSha.ps1 script:

• Updating Core\NativeDllName.cs (used for the DllImport attributes)
• Updating libgit2_hash.txt (used by Version.cs)

These both are related to how naming is currently handled for the native binaries, mainly trying to use the commit hash as part of the name. Since the binary would be coming from outside the project, keeping this aspect in place might be more trouble than it's worth. It might make more sense to just standardize the name.

I'd also like to look into changing the way NativeMethods is looking for the binary file. Right now, on windows it's digging down into the NativeBinaries subfolder and then the architecture folder. For linux and osx, it's not doing anything special, just letting mono find it in the standard places.

Is there a reason for needing the subfolders on windows? Why not have libgit2-x86.dll and libgit2-x64.dll and let them live side by side with libgit2sharp.dll? It looks like there would be some mono-specific workaround code that could be removed from NativeMethods if we just let all the native binaries live in the same folder!

[bording]

Hmm, looks like the Travis CI builds didn't find the myget feed to get the NativeBinaries package, but AppVeyor didn't have a problem.

Looking into it...

[bording]

Well, now it's working on Windows and Mac, so that's progress at least!

[nulltoken]

Well, now it's working on Windows and Mac, so that's progress at least!

Indeed. How about tweaking build.libgit2sharp.sh so that it considers the output of uname and decides to either build libgit2 or not?

Regarding the naming of the binaries, I'd very like to keep the short sha suffix. Besides the troubleshooting help, it also allows two different version of LibGit2Sharp to be ran side by side in the same process, each of them leveraging a different version of libgit2 (eg. Visual Studio extensions).

[ethomson]

Indeed. How about tweaking build.libgit2sharp.sh so that it considers the output of uname and decides to either build libgit2 or not?

👍

Regarding the naming of the binaries, I'd very like to keep the short sha suffix. Besides the troubleshooting help, it also allows two different version of LibGit2Sharp to be ran side by side in the same process, each of them leveraging a different version of libgit2 (eg. Visual Studio extensions).

👍

[bording]

The updated package should have fixed the linux build, but nuget.org seems to be returning an error for the latest version of my package. All the builds failed because the couldn't restore the package. I'm looking into what's up with that now.

The reason the linux Travis CI build was failing was that the linux library I included in the 0.0.2 version of my package was expecting the http-parser lib to be on the system. I rebuilt it using the internal fallback dependency, so once this nuget.org problem gets resolved, I expect all the builds to pass.

[bording]

If you look at the build log for the linux travis CI build for 093593b, you'll see that it's failing on the shadow copy test.

As best I can tell, mono seems to have a problem with DLLImport and shadow copy assemblies, but only on linux. OS X appears to work fine.

As a workaround, I'm about to commit a change to build.libgit2sharp.sh that should work around the problem for now.

[bording]

Is there a reason for needing the subfolders on windows? Why not have libgit2-x86.dll and libgit2-x64.dll

I realized why this is needed now: the path windows is looking for can be changed at runtime, but DLLImport, being an attribute, needs a single compile-time value. So, yeah don't see a simple way to remove this. 😞

[bording]

Regarding the naming of the binaries, I'd very like to keep the short sha suffix. Besides the troubleshooting help, it also allows two different version of LibGit2Sharp to be ran side by side in the same process, each of them leveraging a different version of libgit2 (eg. Visual Studio extensions).

That makes sense. I hadn't thought about needing multiple versions in the same process!

Keeping that around means that libgit2_hash.txt and Core\NativeDllName.cs will have to be updated in a new way.

The NativeBinaries package could include libgit2_hash.txt and the props file could automatically include that as embedded resource, making that process automatic. The down side to doing that would mean that everyone who installs the package would get that file as an embedded resource in their assembly and it wouldn't be immediately obvious why because items added via a props file don't show up in Solution Explorer. This doesn't seem like a terrible problem though.

Updating Core\NativeDllName.cs automatically is a bit more difficult. Imposing an embedded resource on everyone is one thing, but I doubt that forcing everyone to have a class added to their project via the NativeBinaries package would be a good idea.

I had been thinking that the NativeDllName class could be changed to use libgit2_hash.txt somehow, but that can't happen since it needs to be const for the DllImport attributes. 😭

So, this is what the scenario around updating LibGit2Sharp to a new version of libgit2 would look like:

• Run a script in the NativeBinaries repo to update the libgit2 submodule and create a new libgit2_hash.txt
Build native windows, linux, and osx binaries and commit them to theNativeBinaries` repo
• Push out a new version of NativeBinaries nuget package
• Update LibGit2Sharp to the new version of the package, manually tweaking `Core\NativeDllName.cs in the process

[bording]

And of course the linux build fails again... but this time it looks like nuget.org timed out so it couldn't restore the packages successfully. 😢

The 0.0.5 version of the package is the first version I've built from a nuspec instead of just manually putting it together. 😄

The repo I have where this is coming from is here

It's close to being done, so take a look! I still need to add scripts to build and pack from linux/osx, but otherwise I think it's about done.

My thought was that we could take what I have there and either copy or fork it over to an "official" repo once this is all done.

[nulltoken]

You're doing an amazing work! ❤️

And of course the linux build fails again... but this time it looks like nuget.org timed out so it couldn't restore the packages successfully. 😢

I've tried to restart the build, but it looks like I can't connect to Travis now. I'll try again later.

Updating Core\NativeDllName.cs automatically is a bit more difficult. Imposing an embedded resource on everyone is one thing, but I doubt that forcing everyone to have a class added to their project via the NativeBinaries package would be a good idea.

Couldn't we rely on some compile time trickery in the LibGit2Sharp side and dynamically regenerate the file (à la #438)?

[nulltoken]

Regarding the NativeBinaries package content, would there be a way to rely on the CI servers to actually build them? We're already doing sort of this for LibGit2Sharp with AppVeyor (on merge, the NuGet package is published as a build artifact).

Provided we'd do the same for the Windows builds of libgit2 (and find a way to retrieve the Travis Linux/OS produced binaries), the only manual operation would be to download the Windows only package, the Linux/Mac Os X binaries, update the package with those and publish it.

[nulltoken]

@bording Just restarted the Mac Os X build

Regarding the native package version, how about applying the same LibGit2Sharp scheme and end up with something like 0.23.0-pre[BuildTimeStamp]?

[bording]

Couldn't we rely on some compile time trickery in the LibGit2Sharp side and dynamically regenerate the file (à la #438)?

Thanks for linking to that! I had wondered what the UniqueIdentifier.targets was about, but hadn't had time to investigate yet. Any idea if that CLR bug ever got fixed in a public release?

I think using a custom task like that for NativeDllName.cs should work. I'm playing around with something now and I believe it should be doable. The trick is that it's not just generating a GUID with no input needed, but actually needs the path of the libgit2_hash.txt passed to it, and will need to open that file up to figure out the short hash.

After adding another task to the CustomBuildTasks.dll, I think we've got another good candidate for a separate NuGet package. One more binary we can get out of the repo!

Provided we'd do the same for the Windows builds of libgit2 (and find a way to retrieve the Travis Linux/OS produced binaries), the only manual operation would be to download the Windows only package, the Linux/Mac Os X binaries, update the package with those and publish it.

That sounds like a good idea to me. Getting a windows Nuget package out of Appveyor and the other binaries out of a Travis build and manually adding them in would mean that we wouldn't need to commit the binaries to the repo at all. I like it! 😄

Regarding the native package version, how about applying the same LibGit2Sharp scheme and end up with something like 0.23.0-pre[BuildTimeStamp]?

Sure, should be possible. At the moment the package build command just takes any version string from the command line, which I've just been manually bumping up. I'll look into this as well.

[bording]

FYI, I have the NativeDllName custom build task working! I just need to get it cleaned up a bit before it's ready to be pushed up.

[bording]

Wow, Travis sure seems flaky, the nuget package restore failed again. 😢

[nulltoken]

Wow, Travis sure seems flaky, the nuget package restore failed again. 😢

Duh, I cannot connect to Travis.

@svenfuchs Any idea?

[nulltoken]

Build has been restarted

[nulltoken]

Shouldn't this be libgit2_hash.txt?

[nulltoken]

Hmmm. scratch this last comment.

[bording]

Hmm, I had commented on this, but seems like it I must not have saved it. oops! Like I'm guessing you noticed, the %(Filename) metadata doesn't include the extension. I could go ahead and add %(Ext) in there too, just to make the comparison more complete.

[bording]

I've made a bit more progress on my NativeBinaries repo. I've split the updatelibgit2 script out into a script that updates the submodule and related files, and now have a separate script that builds the windows binaries.

I've also added a script for building the linux and osx binaries, but I still need to get some more logic in place to copy them to the correct subfolders.

I suppose that might not matter though, if we can just push them out as build artifacts from the CI server.

I still need to write travis.yml and appveyor.yml files for that repo, but otherwise I think this is close to being ready!

[nulltoken]

If we start ignoring it, shouldn't we also drop it from the index?

[bording]

Hmm, thought 76eaeea included that, or is there something more that you're wanting me to do here?

[bording]

I've got some more changes in NativeBinaries:

• buildpackage.ps1 now accepts a -pre paramter to add the prerelease timestamp
• The binary files don't need to be committed at all now! 🎉

buildpackage.ps1 now makes sure the binary folder structure exists and adds placeholder files to make sure the folders are in the package. Manually adding in the binaries and deleting the placeholders should be pretty easy since the folders will already exist. I had to add the placeholder files because NuGet won't add empty folders to the package 😦

Things that still need to be done:

• Finish up the bash build script
• Add CI files
• Maybe make buildpackage.ps1 a bit less verbose and warn when there are missing binaries
• Change the package to the "real" name and some other nuspec metadata
• Update LibGit2Sharp's packages.config to point to the "real" package name

I think that covers everything I've got on my TODO list, but I do have a few questions.

1. build.msbuild Do we want to make any more changes here, like removing aspects that are only relevant for Windows, which doesn't use it?
2. Do we want to update the mono version used in the LibGit2Sharp's CI builds? It looks like the linux build is already pulling down 3.12, but the osx build is still grabbing 3.6. If we do update, we should be able to get rid of the mozroots stuff, since they do that on install with 3.12 now.
3. For the linux and osx builds, I've currently changed the build configuration to Release instead of RelWithDebInfo. It makes a massive difference in the size of the files, linux in particular. Like 980KB vs 4.5MB. Any reason we can't stick with that change?

[bording]

I have finalized the unix build script, and I've been playing with CI setups, with free accounts pointing at my repo.

I've added a travis.yml file, and it seems to be working fine, though apparently I can only get linux builds. 😦

To get artifacts pushed out of Travis, it looks like we have to set up some sort of 3rd party service, the default being an S3 account.

I haven't added an AppVeyor.yml yet, because I've still been directly using the settings from the site. The problem I'm running into there is that the hardware I get with a free account must be very slow. The builds are timing out after running for 40 minutes! It's possible that something else is wrong, but I'm not getting much to go on yet.

[bording]

Just pushed up some more commits to my NativeBinaries repo. That cleans up the output of the buildpackage.ps1 script and renames everything needed to make the real package, LibGit2Sharp.NativeBinaries.

Aside from finishing up the CI stuff, and the issues I mentioned above with that, this is just about done.

@nulltoken Have you had chance to look over the stuff I've done recently? What about the questions I posted above?

[bording]

And, I just got the AppVeyor issue I was having sorted out, so I'll have an appveyor.yml ready to go soon also! Should be able to get it finalized and pushed up tomorrow.

[nulltoken]

@bording Sorry for the delay in answering.

I've just created the https://github.com/libgit2/libgit2sharp.nativebinaries repository. Could you please submit your work over there? I've requested multi os support from Travis for this repository.

Regarding S3, @carlosmn seems to think that we may have other alternatives.

[carlosmn]

This neither a dummy value nor a bug in mono. It is not mono who is interpreting this value, but the system's loader, which is responsible for loading code into the application's memory space, and which may behave differently across OSs. On OS X, the current directory is searched by default, on Linux it is not. By setting an (or really, two) empty path as one of the paths to search, you're asking the loader to also search in the current path. Take a look at your system's dlopen(3) manpage for details about how it behaves on your platform.

[bording]

I hate to be contrary, but all the research I've done so far seems to indicate something more is going on!

If I comment out that LD_LIBRARY_PATH line, the native binaries are still loaded by mono, up until the shadow copy test runs.

This mono documentation points out that mono looks in the referencing assembly's directory before it then relies on the system's loader to find it:

If a library location has not been explicitly specified in a DllMap entry in an application or assembly .config file, Mono will search for a library in a few places:

• The directory where the referencing image was loaded from.
• In any place the system’s dynamic loader is configured to look for shared libraries. For example on Linux this is specified in the $LD_LIBRARY_PATH environment variable and the /etc/ld.so.conf file. On windows the $PATH environment variable is used, instead.

I've seen that exact behavior also, on both osx and linux. The native libraries are found and used without needing any sort of LD_LIBRARY_PATH/DYLD_LIBRARY_PATH overrides set. The only time this is failing is on linux when the managed assemblies have been shadow copied.

I even took at look at mono's loader code, and found the section where they check the for the directory where the managed assembly lives before they go relying on dlopen to find it.

[bording]

I've been playing around with this a bit more, and I think I see what's going on. I wasn't aware that setting LD_LIBRARY_PATH to : would be treated as the current working directory, so adding that bit of info, combined with what I mentioned above, just might fully explain all the behavior I've been seeing!

When no shadow copying is going on, mono's built-in lookup option is finding the native library next to libgit2sharp.dll and loading it, without needing LD_LIBRARY_PATH.

However, when shadow copying is involved, the built-in lookup fails because it's looking in the shadow copy directory, and the native library isn't there.

It then has to fall back to the system-level loading, which is where the difference between OS X and linux that I was seeing comes into play. OS X does include the current working directory in its search path, but linux does not.That is why OS X works without having anything extra set.

Linux, however, needs some extra help, which is where my inadvertent setting of the current working directory value comes in, making it work there too.

Now, for all of this to be true, that means xbuild has to be changing the current working directory as it is running, and then resetting once it's finished, because the current working directory would have to be set to the directory where the native libraries are sitting for this scenario to be feasible.

I wasn't aware that it would be doing that, but I suppose it could be. If it is, then all of the behavior I've been seeing is explained.

[carlosmn]

My command about S3 was more about that simply being the default, but yeah, it may make sense to do something else. Who is supposed create the package with the native code? My understanding was that a libgit2sharp build of vNext/master would create the native packages and upload them, so I don't quite see why you want to store the results in some other place beforehand.

[bording]

@Therzok The build scripts haven't been removed, they've just been moved over to the native binaries repo instead.

The entire libgit2 submodule has been removed from this repo and moved there, so keeping the build scripts here doesn't really make sense.

I'd be happy to help get the SSH PR up and working over there, but the whole point of this PR is get the native stuff out of the LibGit2Sharp repo! 😄

[Therzok]

Oh, great. That's fine by me. :D

[bording]

Just rebased this to vNext in preparation for updating it to use the almost ready real native binaries package!

[bording]

Ok, this is now using the real package! Unless there is any other feedback, I think this is ready to be squashed and merged!

[nulltoken]

Ok, this is now using the real package!

Yeah! Thanks to you. You've done an impressive job!

Unless there is any other feedback, I think this is ready to be squashed and merged!

Looks perfect on my side. Please squash those commits.

However, if that's ok with you, before merging this, I'd just like to wait until #1049 passes in order to ensure that everything is 👍.

[bording]

Squashed and rebased to vNext!

However, if that's ok with you, before merging this, I'd just like to wait until #1049 passes in order to ensure that everything is 👍.

👍

[nulltoken]

@bording Small nitpick

$ git diff c131238 c131238^ -- Lib/CustomBuildTasks/GenerateNativeDllNameTask.cs
diff --git a/Lib/CustomBuildTasks/GenerateNativeDllNameTask.cs b/Lib/CustomBuildTasks/GenerateNativeDllNameTask.cs
deleted file mode 100644
index f687249..0000000
Binary files a/Lib/CustomBuildTasks/GenerateNativeDllNameTask.cs and /dev/null differ

Is there any strange non printable char in the file so that it's detected as binary?

[bording]

@nulltoken I noticed that too, and it turns out the Lib folder has a .gitattributes file that make it treat everything in there as binary.

[nulltoken]

@bording Yep. That's it! I forgot about that (and I may be the guy that put it here in the first place 😉).

BTW, not willing to derail this PR but while reading http://www.cazzulino.com/xplat-msbuild.html I was starting to wonder if we shouldn't try and got the same way.

• Stop relying on xbuild, but rather using the MSBuild NuGet package
• Instead of embedding binary versions of our tasks, rather migrate them to Inline Code Tasks (https://msdn.microsoft.com/en-us/library/dd722601.aspx)

Thoughts?

/cc @Therzok

[bording]

@nulltoken The idea of being able to use the Inline Code Tasks would be a good option for getting rid of the CustomBuildTasks binary in the lib folder.

I wonder if the cross platform MSBuild stuff is far enough along to actually be relied on since they haven't officially released anything yet.

Another option would be to move CustomBuildTasks out into another nuget package, and LibGit2Sharp could use it as a development-only dependency.

Thinking a bit further ahead, as I mentioned in #1019, I've started a branch playing around with moving to the new dnx-style project format, and trying to get a coreCLR build working. There are some pretty cool aspects to it, though several things will have to be revised. MSBuild tasks, for example, aren't going to work.

[nulltoken]

I wonder if the cross platform MSBuild stuff is far enough along to actually be relied on since they haven't officially released anything yet.

I'd say that MSBuild has probably a good history of being "production ready" on Windows. 😉 With @kzu working for Xamarin, there might be a good chance it's been tested as well on *nix paltforms.

The only thing that worries me a bit is that the NuGet package is actually maintained by @kzu and not someone from the MSBuild project.

/cc @AndyGerlicher

[akoeplinger]

FYI, a PR was merged to the MSBuild repo yesterday that adds NuGet packages: dotnet/msbuild#95. They are now used to build dotnet/corefx on Unix (but I think they are only on corefx's myget feed and not on nuget.org yet).

[nulltoken]

Fixes #739

[nulltoken]

@bording Another fine piece of software engineering 💟

[nulltoken]

Published as NuGet pre-release package LibGit2Sharp 0.22.0-pre20150529055949