Nested `.` separated flags with common prefix not working with pflag in Viper >= `1.18.0`

[shani1998]

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.
• I have checked the troubleshooting guide for my problem, without success.

Viper Version

1.18.0

Go Version

1.22.6

Config Source

Flags

Format

Other (specify below)

Repl.it link

No response

Code reproducing the issue

package main

import (
	"fmt"

	"github.com/spf13/pflag"
	"github.com/spf13/viper"
)

var (
	_ = pflag.String("gateway.endpoint.v1", "some/path/v1", "endpoint of gateway server v2")
	_ = pflag.String("gateway.endpoint", "some/path/", "endpoint of gateway server v3")
)

func init() {
	pflag.Parse()
	viper.BindPFlags(pflag.CommandLine)
	viper.AutomaticEnv()
}

func main() {
	fmt.Println("viper.GetString(\"gateway.endpoint\"):", viper.GetString("gateway.endpoint"))
	fmt.Println("viper.GetString(\"gateway.endpoint.v1\"):", viper.GetString("gateway.endpoint.v1")) // getting empty 
}

got output for this:
viper.GetString("gateway.endpoint"): some/path/
viper.GetString("gateway.endpoint.v1"):

Expected Behavior

it should print:

~ go run test.go                                                                    
viper.GetString("gateway.endpoint"): some/path/
viper.GetString("gateway.endpoint.v1"): some/path/v1

Actual Behavior

in v1.17.0
it is working as expected
returning output as

~ go run test.go                                                                    
viper.GetString("gateway.endpoint"): some/path/
viper.GetString("gateway.endpoint.v1"): some/path/v1

Steps To Reproduce

 ✗ go env                                                                                  
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/spahk/Library/Caches/go-build'
GOENV='/Users/spahk/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/spahk/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/spahk/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.22.4/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.22.4/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.4'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/Users/spahk/projects/test/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/jx/dpklpdss291clqs9t6xrh0c00000gp/T/go-build851702384=/tmp/go-build -gno-record-gcc-switches -fno-common'

go mod

module test

go 1.22.4

require (
	github.com/spf13/pflag v1.0.5
	github.com/spf13/viper v1.18.0
)

require (
	github.com/fsnotify/fsnotify v1.7.0 // indirect
	github.com/hashicorp/hcl v1.0.0 // indirect
	github.com/magiconair/properties v1.8.7 // indirect
	github.com/mitchellh/mapstructure v1.5.0 // indirect
	github.com/pelletier/go-toml/v2 v2.2.2 // indirect
	github.com/sagikazarmark/locafero v0.4.0 // indirect
	github.com/sagikazarmark/slog-shim v0.1.0 // indirect
	github.com/sourcegraph/conc v0.3.0 // indirect
	github.com/spf13/afero v1.11.0 // indirect
	github.com/spf13/cast v1.6.0 // indirect
	github.com/subosito/gotenv v1.6.0 // indirect
	go.uber.org/atomic v1.9.0 // indirect
	go.uber.org/multierr v1.9.0 // indirect
	golang.org/x/exp v0.0.0-20230905200255-921286631fa9 // indirect
	golang.org/x/sys v0.18.0 // indirect
	golang.org/x/text v0.14.0 // indirect
	gopkg.in/ini.v1 v1.67.0 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
)

Additional Information

Updated:
I can see it is working with another nested flags like _ = pflag.String("a.b.c.d.v1", "some/path/abc", "endpoint of gateway server v4")
but i am wondering what is the special with key gateway.endpoint.*

[github-actions]

👋 Thanks for reporting!

A maintainer will take a look at your issue shortly. 👀

In the meantime: We are working on Viper v2 and we would love to hear your thoughts about what you like or don't like about Viper, so we can improve or fix those issues.

⏰ If you have a couple minutes, please take some time and share your thoughts: https://forms.gle/R6faU74qPRPAzchZ9

📣 If you've already given us your feedback, you can still help by spreading the news,
either by sharing the above link or telling people about this on Twitter:

https://twitter.com/sagikazarmark/status/1306904078967074816

Thank you! ❤️

[akshay8]

I am also facing the same issue. I found a pattern where in two flags registered with same common prefix, then it is returning empty for the child flags. For example:

	_ = pflag.String("akshay.barik.v1", "akshay barik v1 value", "some v1 description")
	_ = pflag.String("akshay.barik", "akshay barik value", "some description")
	
	fmt.Println(viper.GetString("akshay.barik.v1"))         // getting empty
	fmt.Println( viper.GetString("akshay.barik"))       

[shani1998]

I am also facing the same issue. I found a pattern where in two flags registered with same common prefix, then it is returning empty for the child flags. For example:

	_ = pflag.String("akshay.barik.v1", "akshay barik v1 value", "some v1 description")
	_ = pflag.String("akshay.barik", "akshay barik value", "some description")
	
	fmt.Println(viper.GetString("akshay.barik.v1"))         // getting empty
	fmt.Println( viper.GetString("akshay.barik"))       

Thanks, Akshay, for noticing this pattern. I see the same pattern now. I initially thought it was specific to gateway.endpoint, but it seems like multiple nested flags with a common prefix are having issues in Viper 1.18.0. Hopefully, we’ll get an update on this soon.

[chengxilo]

Maybe read this part would be helpful. "gateway.endpoint.v1" was shadowed by "gateway.endpoint"

Here is the source code cause this 'problem'. Pay attention to the v.isPathShadowedInFlatMap(path, v.pflags) would be helpful.

viper/viper.go

Lines 1202 to 1204 in 8b223a4

	if nested && v.isPathShadowedInFlatMap(path, v.pflags) != "" {
	return nil
	}

viper/viper.go

Lines 633 to 675 in 8b223a4

	// isPathShadowedInFlatMap makes sure the given path is not shadowed somewhere
	// in a sub-path of the map.
	// e.g., if "foo.bar" has a value in the given map, it “shadows”
	//
	// "foo.bar.baz" in a lower-priority map
	func (v *Viper) isPathShadowedInFlatMap(path []string, mi any) string {
	// unify input map
	var m map[string]interface{}
	switch miv := mi.(type) {
	case map[string]string:
	m = castMapStringToMapInterface(miv)
	case map[string]FlagValue:
	m = castMapFlagToMapInterface(miv)
	default:
	return ""
	}
	// scan paths
	var parentKey string
	for i := 1; i < len(path); i++ {
	parentKey = strings.Join(path[0:i], v.keyDelim)
	if _, ok := m[parentKey]; ok {
	return parentKey
	}
	}
	return ""
	}
	// isPathShadowedInAutoEnv makes sure the given path is not shadowed somewhere
	// in the environment, when automatic env is on.
	// e.g., if "foo.bar" has a value in the environment, it “shadows”
	//
	// "foo.bar.baz" in a lower-priority map
	func (v *Viper) isPathShadowedInAutoEnv(path []string) string {
	var parentKey string
	for i := 1; i < len(path); i++ {
	parentKey = strings.Join(path[0:i], v.keyDelim)
	if _, ok := v.getEnv(v.mergeWithEnvPrefix(parentKey)); ok {
	return parentKey
	}
	}
	return ""
	}

It should not be treated as a bug.

[chengxilo]

But I do think there should be some error when we use Pflag.String to set those keys, telling us that we should not use shadowed key.

[github-actions]

Issues with no activity for 30 days are marked stale and subject to being closed.

[shani1998]

But I do think there should be some error when we use Pflag.String to set those keys, telling us that we should not use shadowed key.

Agree, that would helpful to user to know upfront.