pachi 12.70 dcnn #123060

lemonsqueeze · 2023-02-12T16:56:09Z

Hi,

I'm Pachi's current maintainer.
Following up on pachi recent build issue, i took at the pachi.rb formula and found it could use an update : currently it does a nodcnn build which is really unfortunate, and downloads deprecated data files which are no longer used.
See original comment for details.

@chenrui333 suggested i open a PR to try a new build, so here it is.

I don't have access to a mac, but hopefully should work out with CI testing ...
This is my first PR to homebrew-core so if anything's out of place please let me know.

chenrui333 · 2023-02-12T17:32:52Z

Thanks @lemonsqueeze, there are some style issues which can be fixed with brew style --fix pachi

chenrui333 · 2023-02-12T17:34:19Z

Formula/pachi.rb

+  depends_on "caffe"
+
+  resource "datafiles" do
+    url "https://launchpad.net/~lemonsqueeze/+archive/ubuntu/pachi/+sourcefiles/pachi-go-data/12.70/pachi-go-data_12.70.orig.tar.xz"


why not host this file in the github release?

The dcnn files (detlef54.*) are too big to keep in git, that's why they are missing from the github source release. They are included in the binary releases (windows, linux-static) though, so could get them from github instead of launchpad if that's an issue. The good thing with this launchpad resource is it's got just the datafiles, the bad : it usually takes me a day or two to update it after a release. So could pull everything from github if that's preferrable.

So where are they kept inbetween binary releases?

chenrui333 · 2023-02-12T18:06:16Z

we have actually deprecate caffe

  pachi:
    * Dependency 'caffe' is deprecated but has un-deprecated dependents. Either
      un-deprecate 'caffe' or deprecate it and all of its dependents.

rewrote recipe to build Pachi with dcnn support, since homebrew already builds caffe. (Pachi with dcnn support has a lot more value) take advantage of DESTDIR and DATADIR build variables so we can just install with "make install". removed old caveat notice about datafiles having to be in current directory, no longer the case. no longer downloads old patterns files (not used since Pachi 12.10) and opening book from external site. get missing dcnn datafiles from launchpad (could get them from Pachi binary release too).

lemonsqueeze · 2023-02-12T19:45:01Z

  pachi:
    * Dependency 'caffe' is deprecated but has un-deprecated dependents. Either
      un-deprecate 'caffe' or deprecate it and all of its dependents.

Yeah, this is unfortunate. Is there anything i can do about it ?
Or should i ask caffe to be un-deprecated ?

lemonsqueeze · 2023-02-12T19:49:23Z

Fixed style issues, looks like it passes now.

chenrui333 · 2023-02-12T21:26:56Z

  pachi:
    * Dependency 'caffe' is deprecated but has un-deprecated dependents. Either
      un-deprecate 'caffe' or deprecate it and all of its dependents.
Yeah, this is unfortunate. Is there anything i can do about it ? Or should i ask caffe to be un-deprecated ?

see discussions in this thread, #119433

carlocab

Thanks for a PR. A few comments.

carlocab · 2023-02-13T03:45:07Z

Formula/pachi.rb

-    url "https://sainet-dist.s3.amazonaws.com/pachi_patterns.zip"
-    sha256 "73045eed2a15c5cb54bcdb7e60b106729009fa0a809d388dfd80f26c07ca7cbc"
-  end
+  depends_on "caffe"


Is caffe required? We no longer allow caffe to be used as a dependency because it is deprecated.

Yes, currently Pachi depends on caffe to evaluate its neural network. It's possible to do a nodcnn build, in which case caffe is not needed, but really unfortunate for users (Pachi loses much of its value without dcnn).

I think that's the trade-off we'll have to make here. We can use dcnn once it works with software that is not deprecated.

carlocab · 2023-02-13T03:45:31Z

Formula/pachi.rb

  end

  def install
+    ENV["DESTDIR"] = HOMEBREW_PREFIX


We don't actually want to install directly into HOMEBREW_PREFIX, I think.

Yeah, was not sure about this one. Maybe i should set PREFIX = HOMEBEW_PREFIX instead, this is what i intended to do.

lemonsqueeze · 2023-02-13T08:26:30Z

I took a look at #119433. Homebrew is not alone in dropping caffe : it's not in debian testing anymore and already it's gone from ubuntu 22.04, so i have to build it in my ppa. To be honest i don't really understand why everybody drops it. Software engineering is a strange place sometimes, in the real world nobody would think of tearing down a house that's just finished building just because the builders have gone in maintenance mode and there are other taller, shinier buildings being built. The house is built, it's stable, mature and useful, might as well use it !

In this case it provides an efficient way to evaluate a neural network in production code while keeping code footprint small. I don't want to pull in tensorflow, pytorch or another huge framework just for that, it would be ridiculous. If someone wants to recommend another library that does that i'll consider migrating. Otherwise if caffe gets dropped completely i guess i'll have to include a caffe build in Pachi, which is a bit ridiculous when there's already a working caffe formula ...

I'm curious, from Homebrew's point of view does it really hurt that much to keep caffe around ?
It's not like it's so broken that it doesn't build anymore or something like that, is it ?

SMillerDev · 2023-02-14T08:14:17Z

Formula/pachi.rb

  end

  def install
+    ENV["DESTDIR"] = HOMEBREW_PREFIX
+    ENV["DATADIR"] = pkgshare
+    ENV["GENERIC"] = "1"                # make cpu-generic binary (intel)


So what happens on ARM?

Yes, should be intel only. I guess i should put this on a switch.

lemonsqueeze · 2023-02-14T19:48:15Z

As it is this formula can't succeed (caffe deprecated).
I'll see about including a caffe build in Pachi and open a new PR, i guess that's the only way it could build in homebrew.
Thanks for all the useful feedback.

SMillerDev · 2023-02-15T08:26:46Z

I'll see about including a caffe build in Pachi and open a new PR, i guess that's the only way it could build in homebrew.

No, that would still be disallowed. We don't want to vendor deprecated software. The only way to build it in Homebrew is to switch away from deprecated dependencies.

lemonsqueeze · 2023-02-15T12:39:19Z

Closing per above comment.

SMillerDev · 2023-02-15T13:05:21Z

To be honest i don't really understand why everybody drops it. Software engineering is a strange place sometimes, in the real world nobody would think of tearing down a house that's just finished building just because the builders have gone in maintenance mode and there are other taller, shinier buildings being built. The house is built, it's stable, mature and useful, might as well use it !

But people redo the paint with new paint, replace the windowsills and remove asbestos to make sure the house is still safe to live in. If the builders promise you to maintain the house and then just disappear for a long time you'd be worried about the state of your house too.

lemonsqueeze · 2023-02-16T12:17:24Z

No, that would still be disallowed. We don't want to vendor deprecated software. The only way to build it in Homebrew is to switch away from deprecated dependencies.

Looks like you misunderstood what i wrote. There wouldn't be any deprecated dependency then, Pachi would depend on Openblas, Boost, Protobuf, Glog, Gflags and the Standard C++ Library. Are any of these deprecated as well ?

While i certainly respect Homebrew's decision to not package Caffe anymore, individual projects remain free to make whatever technical choices best suit their needs. Homebrew should certainly respect that, and it is none of its business what these choices even are are as long as the software builds and functions properly. A comment like this where someone tries to force his political decisions down a project's throat is concerning to say the least. I may be wrong in assuming that your views reflect that of Homebrew's of course.

SMillerDev · 2023-02-16T12:29:12Z

There wouldn't be any deprecated dependency then, Pachi would depend on Openblas, Boost, Protobuf, Glog, Gflags and the Standard C++ Library. Are any of these deprecated as well ?

It will include a version of Caffe then right? Because that would be vendoring a deprecated dependency.

individual projects remain free to make whatever technical choices best suit their needs.

They certainly are and so is Homebrew when it decides what to package.

force his political decisions down a project's throat

Nobody is forcing any project to do something. I'm just outlining the requirements for software to be included in Homebrew.

lemonsqueeze · 2023-02-16T13:34:42Z

It will include a version of Caffe then right? Because that would be vendoring a deprecated dependency.

Like i said, Pachi would have its own neural network code and link against Boost and Openblas. It wouldn't ship any Caffe library binary, nor would there be any trace of one on the system. So as far as homebrew is concerned once you're done deprecating Caffe it's effectively gone from the system. Isn't it all that matters ?

lemonsqueeze · 2023-02-18T21:47:35Z

Well, I guess we'll leave it at that.
First encounter with homebrew police, i must say i'm impressed.
Maybe it'd be good to put badges on the avatars, that would make things clear.
For some time i thought i was talking to another software engineer, hence the complete misunderstanding.

BrewTestBot added the deprecated license Formula uses a deprecated SPDX license which should be updated label Feb 12, 2023

lemonsqueeze mentioned this pull request Feb 12, 2023

pachi 12.70 build failure pasky/pachi#153

Open

lemonsqueeze force-pushed the pachi branch from 23b141f to 87b8d86 Compare February 12, 2023 17:12

chenrui333 mentioned this pull request Feb 12, 2023

pachi 12.70 #122989

Closed

chenrui333 reviewed Feb 12, 2023

View reviewed changes

lemonsqueeze force-pushed the pachi branch from e828cf0 to 839585f Compare February 12, 2023 19:37

carlocab reviewed Feb 13, 2023

View reviewed changes

SMillerDev reviewed Feb 14, 2023

View reviewed changes

lemonsqueeze closed this Feb 15, 2023

lemonsqueeze mentioned this pull request Feb 15, 2023

pachi: removal #123301

Closed

github-actions bot added the outdated PR was locked due to age label Mar 21, 2023

github-actions bot locked as resolved and limited conversation to collaborators Mar 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pachi 12.70 dcnn #123060

pachi 12.70 dcnn #123060

lemonsqueeze commented Feb 12, 2023

chenrui333 commented Feb 12, 2023

chenrui333 Feb 12, 2023

lemonsqueeze Feb 12, 2023

SMillerDev Feb 14, 2023

chenrui333 commented Feb 12, 2023

lemonsqueeze commented Feb 12, 2023

lemonsqueeze commented Feb 12, 2023

chenrui333 commented Feb 12, 2023

carlocab left a comment

carlocab Feb 13, 2023

lemonsqueeze Feb 13, 2023

SMillerDev Feb 14, 2023

carlocab Feb 13, 2023

lemonsqueeze Feb 13, 2023

lemonsqueeze commented Feb 13, 2023

SMillerDev Feb 14, 2023

lemonsqueeze Feb 14, 2023

lemonsqueeze commented Feb 14, 2023

SMillerDev commented Feb 15, 2023

lemonsqueeze commented Feb 15, 2023

SMillerDev commented Feb 15, 2023

lemonsqueeze commented Feb 16, 2023

SMillerDev commented Feb 16, 2023

lemonsqueeze commented Feb 16, 2023

lemonsqueeze commented Feb 18, 2023

pachi 12.70 dcnn #123060

pachi 12.70 dcnn #123060

Conversation

lemonsqueeze commented Feb 12, 2023

chenrui333 commented Feb 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenrui333 commented Feb 12, 2023

lemonsqueeze commented Feb 12, 2023

lemonsqueeze commented Feb 12, 2023

chenrui333 commented Feb 12, 2023

carlocab left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lemonsqueeze commented Feb 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lemonsqueeze commented Feb 14, 2023

SMillerDev commented Feb 15, 2023

lemonsqueeze commented Feb 15, 2023

SMillerDev commented Feb 15, 2023

lemonsqueeze commented Feb 16, 2023

SMillerDev commented Feb 16, 2023

lemonsqueeze commented Feb 16, 2023

lemonsqueeze commented Feb 18, 2023