mtl module is member of hidden package

[yaitskov]

Hi,

I noticed that modules from some packages are not available inside interpreter, while rest packages are loadable. I don't see why behavour is different.
In my case mtl package is in trouble but text package is not!
Both packages are on the same dependency list.

Could not load module \8216Control.Monad.State.Strict\8217\n It is a member of the hidden package \8216mtl-2.2.2\8217.\n You can run \8216:set -package mtl\8217 to expose it.\n (Note: this unloads all the modules in the current scope.)\n Use -v (or :set -vin ghci) to see a list of the files searched for."}]

{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE ScopedTypeVariables #-}

module Hello where

import Prelude (String, (++), Char, Int)
import Control.Monad.State.Strict
import Data.Functor.Identity
import Data.Text

countDown :: StateT Int Identity ()
countDown = do
  i <- get
  if (i > 0)
  then do
    put (i - 1)
    countDown
  else pure ()

strFromText :: String
strFromText = unpack ("HELLO TEXT" :: Text)

ghc 8.10.2

[gelisam]

Both packages are on the same dependency list.

Which dependency list? hint doesn't know anything about your .cabal file. It doesn't know anything about stack or cabal either. Since hint is a wrapper around the ghc library which is intended to make that library easier to use, it probably should cooperate with both stack and cabal, but at the moment it does not (I've just filed #129 to track that).

Instead, hint uses ghc-pkg, the package manager which comes with ghc. It stores packages in various folders called "package databases". stack and cabal both make use of ghc-pkg, and call ghc with various command-line arguments and environment variables which tell ghc which package databases to use. It is also possible to tell ghc to hide all the packages from the package databases it knows about, and to only unhide a select few packages. You can see the flags which do this using the --verbose flag:

$ cabal repl --verbose
...
/Users/gelisam/.ghcup/bin/ghc --interactive \
  ... \
  -no-user-package-db \
  -package-db /Users/gelisam/.cabal/store/ghc-9.0.1/package.db \
  -package-db /Users/gelisam/working/haskell/toy/dist-newstyle/packagedb/ghc-9.0.1 \
  -hide-all-packages \
  -package-id base-4.15.0.0 \
  -package-id containers-0.6.4.1 \
  -package-id mtl-2.2.2
GHCi, version 9.0.1: https://www.haskell.org/ghc/  :? for help
Loaded GHCi configuration from /Users/gelisam/.ghci
[1 of 1] Compiling Main             ( src/Main.hs, interpreted )
Ok, one module loaded.
>

Your error message says that your module is a "member of the hidden package mtl", so the problem is definitely that your build system is somehow passing -hide-all-packages but not -package-id mtl-2.2.2 to hint.

Now, in order to pass those flags to hint, you'd need to use the unnecessarily-scarily-named unsafeRunInterpreterWithArgs, but I assume you're using the ordinary runInterpreter, which doesn't pass any flags. So I am guessing that there is some other mechanism, like an environment variable or a .ghc-something file which ghc is reading and which your build system is setting.

[gelisam]

I couldn't reproduce with stack run, stack install && ~/.local/bin/my-exe, cabal run, nor cabal install && ~/.cabal/bin/my-exe. How are you building and running your code?

module Main where

import qualified Language.Haskell.Interpreter as Hint

test :: IO (Either Hint.InterpreterError ())
test = Hint.runInterpreter $ do
  Hint.loadModules ["Hello.hs"]

main :: IO ()
main = print =<< test

[gelisam]

The output of ghc-pkg list would also be quite useful.

[gelisam]

Aha! I've managed to reproduce the problem with cabal install && ~/.cabal/bin/my-exe by switching to ghc-8.10.3; I didn't think the version of ghc would matter all that much, but it does!

[gelisam]

I can also reproduce with plain ghci. Therefore, this is not a bug in hint.

$ ghci
GHCi, version 8.10.3: https://www.haskell.org/ghc/  :? for help
Loaded package environment from /Users/gelisam/.ghc/x86_64-darwin-8.10.3/environments/default
Loaded GHCi configuration from /Users/gelisam/.ghci
> import Data.Text
> import Control.Monad.State

<no location info>: error:
    Could not load module ‘Control.Monad.State’
    It is a member of the hidden package ‘mtl-2.2.2’.
    You can run ‘:set -package mtl’ to expose it.
    (Note: this unloads all the modules in the current scope.)
> :set -package mtl
> import Control.Monad.State
Leaving GHCi.

The line Loaded package environment from /Users/gelisam/.ghc/x86_64-darwin-8.10.3/environments/default is interesting. That file contains:

clear-package-db
global-package-db
package-db /Users/gelisam/.cabal/store/ghc-8.10.3/package.db
package-id ghc-8.10.3
package-id base-4.14.1.0
[...]
package-id text-1.2.4.1

but it does not include the line package-id mtl-2.2.2. I don't know what is generating this file, but it is clearly the source of the problem.

Another interesting line is ​the recommendation to run :set -package mtl. Unfortunately hint doesn't support ghci's :cmd syntax, so it's not possible to use the same workaround in hint; however, it is possible to specify -package mtl using unsafeRunInterpreterWithArgs:

import qualified Language.Haskell.Interpreter as Hint
import qualified Language.Haskell.Interpreter.Unsafe as Hint

test :: IO (Either Hint.InterpreterError ())
test = Hint.unsafeRunInterpreterWithArgs ["-package", "mtl"] $ do
  Hint.setImports ["Control.Monad.State"]

It would probably be a good idea to add a function to hint which behaves like ghci's :set -package <package-name> (I've filed #130 to track this).

[yaitskov]

Wow! Thanks for premium support ;)
I agree GHC version matters a lot.
In my case 8.10.2 and 8.10.4 has issues, but 8.10.3 and 9.0.1 is working!
Which is kind of not consistent with your version (8.10.3 breaks).
I was changing stack.yaml:

resolver: lts-17.9
#compiler: ghc-9.0.1
#compiler: ghc-8.10.2
compiler: ghc-8.10.3
compiler-check: match-exact

unsafeRunInterpreterWithArgs and package arguments have no effect for me in stack demo project, but worked out in a nix project.

[yaitskov]

I discovered additional info. ghc-pkg is not the only place for resolving dependencies.

The project, I develop, uses nix and it has been upgraded to from ghc-8.10.2 to ghc-8.10.4 and I noticed new problem. Probably the existed before I just wasn't able to notice them.

cabal builds my program. The program needs to use its own module inside hint (cabal library)

inside nix-shell:

cabal build

once I run a Haskell script imporintg MyPackage I get runtime error:

GhcException "libHSmylib-0.3.8-2b517a72dcdee51aa9773d0e2dad33ae67d9079beaf1f792beef208cb6cd1f07.so: cannot open shared object file: No such file or directory"

cabal install --lib solves the issue, mean while ghc-pkg list doesn't show any difference!

I started looking for the magic artifact.
The cabal install library command creates library in

~/.cabal/store/ghc-8.10.4/mylib-0.3.8-2b517a72dcdee51aa9773d0e2dad33ae67d9079beaf1f792beef208cb6cd1f07/lib/libHSmylib-0.3.8-2b517a72dcdee51aa9773d0e2dad33ae67d9079beaf1f792beef208cb6cd1f07-ghc8.10.4.so

Reference to so file is not inside elf file.

grep -a -o  -E -e 'HSmylib[A-Za-z0-9. _-]+' ./dist-newstyle/build/x86_64-linux/ghc-8.10.4/mylib-0.3.8/x/mylib/build/mylib/mylib

HSmylib-0.3.8-inplace

there is no hash suffix.

ldd doesn't show anything either.

inplace word grabbed my attention.

https://cabal.readthedocs.io/en/latest/nix-local-build.html

A local package is one that is listed explicitly in the packages,
optional-packages or extra-packages field of a project. Usually, these
refer to packages whose source code lives directly in a folder in your
project. But you can list an arbitrary Hackage package in packages to
force it to be treated as local.

Local packages, as well as the external packages (below) which depend
on them, are built inplace, meaning that they are always built
specifically for the project and are not installed globally. Inplace
packages are not cached and not given unique hashes, which makes them
suitable for packages which you want to edit and recompile.

Then I tried to remove lilbHSmylib.so from cabal store and put into current folder with

export LD_LIBRARY_PATH=$PWD

then app fails with new error:

GhcException "/nix/store/ll08x08c93j8g17qvjlxf2mip6r0m49m-ghc-8.10.4/lib/ghc-8.10.4/ghc-prim-0.6.1/libHSghc-prim-0.6.1-ghc8.10.4.so: undefined symbol: stg_gc_unpt_r1

build directory contains:

./dist-newstyle/build/x86_64-linux/ghc-8.10.4/mylib-0.3.8/build/libHSmylib-0.3.8-inplace-ghc8.10.4.so

Definitely inplace cause the problem. It would nice to know how to reference program library as any other one.

After further experimenting I detected that libHSmylib.so file is not need at all. It is not sufficient due missing RTS symbol stg_gc_unpt_r1, but next to this file I noticed static library libHSmylib-0.3.8-2b517a72dcdee51aa9773d0e2dad33ae67d9079beaf1f792beef208cb6cd1f07.a.

The static lib make the difference. How come? Static library rescues runtime?!
The question is why the static library should be exactly in .caba/store and not in current folder.

[gelisam]

cabal install --lib solves the issue, mean while ghc-pkg list doesn't show any difference!

As the ghc-pkg documentation explains, ghc-pkg list only lists the packages in the "global" database:

$ ghc-pkg --help
[...]
  ghc-pkg list [pkg]
    List registered packages in the global database, and also the
    user database if --user is given. If a package name is given
    all the registered versions will be listed in ascending order.
    Accepts the --simple-output flag.
[...]

In my case, this "global database" is /Users/gelisam/.ghcup/ghc/8.10.3/lib/ghc-8.10.3/package.conf.d. This is the package database which is installed by ghc itself, not the package database to which stack and cabal install their packages. When you install a library via cabal install --lib, cabal installs the library somewhere inside ~/.cabal, which has its own package database. On my computer, I can see the list of packages installed by cabal using $ ghc-pkg --package-db="$HOME/.cabal/store/ghc-8.10.3/package.db" list.

Definitely inplace cause the problem. It would nice to know how to reference program library as any other one.

Remember, I don't recommend using hint to load module X from module X.

The static lib make the difference. How come? Static library rescues runtime?!

I'm not sure about that. I do know, however, that each package (or is it each module?) is compiled in two different forms: a compiled form and an interpreted form. ghci and hint both distinguish between loading a module and importing a module; the loaded modules are interpreted, whereas for the module which are imported but not loaded, the compiled version is used. Presumably there is some kind of shim in between which allows the interpreted code to call compiled functions and vice-versa. Maybe the .so and the .a files are holding the interpreted and compiled versions of the package? Or maybe vice-versa?

[gelisam]

to load module X from module X.

...or even a module from package my-package from a module of my-package.