My Niche Use Cases

[nelsam]

This is kinda a maintenance question. I was a maintainer on testify and gorp for a long time, so I know how many headaches you can get maintaining all sorts of features for niche use cases. How do you (i.e. the maintainers on this repo) feel about supporting super niche cases?

The tl;dr is that I wrote some features into a fork of toml intending to submit them as PRs, then felt the maintenance burden in my soul when I looked at the diff. I don't want to put that on you folks unless you think it's worth your time. Do you want to support my weird use cases? I'm happy to make a few PRs, but I don't want to delve too deep in the weeds if it's just going to waste your time.

My niche case(s):

1. I need to pass in pre-filled data, i.e. filled with default values. When I unmarshal a file to that data, I only want it to overwrite data in the fields if the file contains data for those fields. Currently, structs work this way, but maps don't. One of my data structures requires a map, and the default values in that map get wiped out when I use this repo to unmarshal data.
2. I need to unmarshal several files from a conf.d directory to the same destination. Values from each file overwrite values from the previous files. This kind of rolls up into the issue from [1] - structs work fine, maps do not. Primitives only store the most recently decoded file.
3. I need to support lazy decoders - types that do not yet know their child structure. I can almost get there with Primitive, but since I am unmarshaling several files, the Primitive's context and undecoded values lose track of data from previous files. They need to know which MetaData they got those values from (and probably track multiple MetaData values) in order to decode the right data.
4. I need to pause lazy decoding part of the way through in order to save the current data. I have two folders - a dflt.d storing defaults from previous versions, and a conf.d storing user-customized data. I don't want new defaults to overwrite previous defaults, but I do want to save defaults for any settings that didn't exist in the previous version. So I load previous defaults from a dflt.d directory and then save the current config struct to a file. After saving that file, I resume decoding the user-defined files. MetaData plus Primitive is almost enough, but the changes needed to handle it are non-trivial.

I solved [1] and most of [2] in #491, but as I solved [3] it started to look like a lot of code changes ... and [4] just made me think that I'm probably going to be wasting your time. I wrote my own (semi-naive) toml parser to handle my niche cases instead. But before I plow forward with that long-term, I wanted to open a discussion. How do you folks feel about supporting niche use cases like that? If you think there's value to these features in this repo, I can shift my changes to a PR.

But it feels like a lot of features for a single user (it doesn't seem like anyone else has hit these yet), so I am also happy to maintain my own parser that just handles my specific use case.

[arp242]

It should be compatible with encoding/json. That's the least surprising thing to do – it can't fully do that though because this library goes back to before Go 1.0 and some things were added before it was added to encoding/json (but in a different way).

A v2 version should be compatible with the encoding/json/v2 due in Go 1.27. I haven't really looked at this in-depth yet to see what it can and can't do. I'm hesitant to add extra features on top because that how we ended up in trouble with incompatibilities in the first place.

So if encoding/json/v2 can do any of that then that'll end up being supported here too sooner or later. If not ... well, I'm not so sure if it should be added.

I'm not super enthusiastic in making large changes for the current (v1) to be honest unless it really fixes bugs or (serious) encoding/json incompatibilities as I don't have a huge amount of time to work on it.

[nelsam]

Yeah, makes sense. WRT [1] and [2] of my cases:

• json (v1) in go 1.26 replaces sub-values in maps, the same as toml (playground link)
• json/v2 in go tip does not replace sub-values (playground link)

WRT [3] and [4], json.Unmarshaler in v1 covers those cases already, since it passes raw json []byte values to the Unmarshaler. I can store them and later resume decoding using json.Unmarshal. This is (very roughly) how I would do it with json:

type LazyJSON struct {
    raw [][]byte
}

func (l *LazyJSON) UnmarshalJSON(raw []byte) error {
    l.raw = append(l.raw, raw)
    return nil
}

func (l *LazyJSON) Unmarshal(v any) error {
    for _, raw := range l.raw {
        if err := json.Unmarshal(raw, v); err != nil {
            return fmt.Errorf("lazy: failed to lazy unmarshal json value: %w", err)
        }
    }
    return nil
}

The problem with toml.Unmarshaler in this case is that toml passes me a []any or map[string]any - data that is mostly decoded already. If it were a []byte, then I could later call toml.Unmarshal again. But as is, there's no way to ask toml to resume decoding later on like there is with json.Unmarshaler. That's why I started out using toml.Primitive instead - I thought I could track all of my MetaData values and later resume decoding with MetaData.PrimitiveDecode.

I see why toml.Unmarshaler is different from json.Unmarshaler, though - in toml data, sub-keys can be in multiple places in the file, so there isn't necessarily a single contiguous byte slice for a given value (unlike in json). It was pretty non-trivial to solve that in my own rudimentary toml parser that I tossed together - I settled on just calling my unmarshaler multiple times. That's probably not an ideal solution.

From my perspective, the difference between json.Unmarshaler and toml.Unmarshaler would have caught me less off-guard if toml had something like the following:

// UnmarshalUndecoded takes an undecoded value (e.g. from `Unmarshaler.UnmarshalTOML`) and unmarshals
// it to v.
func UnmarshalUndecoded(undecoded, v any) error {
    return MetaData{}.unify(undecoded, reflect.ValueOf(v))
}

But I definitely get your reluctance to change v1. I'm happy to keep using the quick and dirty parser I wrote for the time being.