Good afternoon!
In the effort to package some of my own python projects, it was frustrating me I couldn't include 'rgbmatrix' as a dependency.
I've worked up a fork with a pyproject.toml definition, build script, etc. I've tested it on my development Pi3 and it all seems to work fine.
Due to versioning limitations it can't be published to PyPi, but I've tested installing with pip straight from git and it works. Fork is here and hope it's helpful.