Let JsonFormatter implement Serializer

[laeubi]

Once in a while I need to write json output in some of my plugins as well but it is surprisingly hard to do so as the API to do so is all internal and non trivial to implement.

On the other hand the JsonFormatter already has everything we need here, so this lets JsonFormatter now implement Serializer to conviently support this use-case

@mpkorstanje what do you think? I just (again) encountered the problem that my custom hack for this broke the Eclipse-Cucumber plugin, having a semi official API for that would be great and it does not exposes to much of the internals but still allow to reliable write out the JSON (e.g. to a file or websocket) in custom plugins.

[mpkorstanje]

Would the MessageFormatter be something that you could use?

Though admittedly I'm a bit confused about what your use case is exactly.

Once in a while I need to write json output in some of my plugins as well but it is surprisingly hard to do so as the API to do so is all internal and non trivial to implement.

Here it sounds like you want to write arbitrary json. But the Serializer interface only permits writing cucumber messages (in an envelope) to json. So as proposed it seems the pull request doesn't meet your own requirements.

[laeubi]

Would the MessageFormatter be something that you could use?

Good catch, that would be a better place to implement that interface

But it has the same limitations as JsonFormatter:

1. No way to write an Envelope directly I need to somehow "trick" it by injecting an EventPublisher to get access to the actual write method
2. The code takes control over the OutputStream (e.g. closing it in some cases)
3. There is no way to write individual frames unless I create a new MessageFormatter + Output stream for each message (what would work but feels a bit wasted)
4. No way to extend as everything is (package) private

Here it sounds like you want to write arbitrary json.

No, I want exactly that:

But the Serializer interface only permits writing cucumber messages (in an envelope) to json.

So this hopefully makes it more clear:

1. I have an endpoint (actually a socket) that requires a special frame format (here the number of bytes as an int + a payload that is a Json encoded cucumber Envelope)
2. I start cucumber with a special plugin that simply listen to message (like MessageFormatter does) and when I receive one I convert it to the Json into a byte array and send the actual size+bytes over the wire
3. Then on the endpoint side I decode it again (for what yet I have also not found a good way in cucumber directly)

This all is used to observer the test-run of a forked cucumber test in eclipse-cucumber plugin.

[mpkorstanje]

Unfortunately Java doesn't come with a builtin JSON serializer/deserializer. The MessageToNdjsonWriter and NdjsonToMessageIterable are designed with that in mind by requiring that users bring their own. And eventually Cucumber will require that too (cucumber/messages-ndjson#2).

I don't mean to XY-problem you, but it really seems like you need a JSON serializer/deserializer. What is the reason you can't bring your own serializer?

I start cucumber with a special plugin that simply listen to message (like MessageFormatter does) and when I receive one I convert it to the Json into a byte array and send the actual size+bytes over the wire

I did see you were working on something similar for JUnit (junit-team/junit-framework#5096, ota4j-team/open-test-reporting#728). There it seems writing to an output stream is sufficient?

But for what it is worth, the MessageFormatter writes '\n delimited output. You could create an OutputStream that buffers the output until a \n is seen. Something like:

ByteArrayOutputStream delegate = new ByteArrayOutputStream();

OutputStream outputStream = new OutputStream() {
    private final ByteArrayOutputStream buffer = new ByteArrayOutputStream();

    @Override
    public void write(int b) throws IOException {
        if (b == '\n') {
            delegate.write(buffer.size());
            buffer.writeTo(delegate);
            buffer.reset();
        } else {
            buffer.write(b);
        }
    }
};

MessageFormatter messageFormatter = new MessageFormatter(outputStream);

[mpkorstanje]

Alternatively would it help if messages implemented the serializable interface?

[laeubi]

Unfortunately Java doesn't come with a builtin JSON serializer/deserializer. The MessageToNdjsonWriter and NdjsonToMessageIterable are designed with that in mind by requiring that users bring their own.

That's why I found it quite smart if the ones here implement the Serializer interface (same for HTml of course by the way).

I don't mean to XY-problem you, but it really seems like you need a JSON serializer/deserializer. What is the reason you can't bring your own serializer?

The plugin should be as minimal as possible (as I don't know what people are using) and I don't want to bring in something new, and as cucumber already support this and I want exactly this it seems wasted effort anyways.

I did see you were working on something similar for JUnit (junit-team/junit-framework#5096, ota4j-team/open-test-reporting#728). There it seems writing to an output stream is sufficient?

The protocol requires an ACK (so I can hold on the remote process for debugging steps), the formatters only write and forget (what is good!) so I really need to know the boundaries. Also i probably want to send other commands over the channel and the writer possibly do not give me enough control (e.g about flush).

But for what it is worth, the MessageFormatter writes '\n delimited output. You could create an OutputStream that buffers the output until a \n is seen. Something like:

So as said its all possible somehow but not very convenient and I'm bound to how the internal implementation works and I'm not sure I want to put to much burden on it. To be concrete, would you say it will always be the case now and in the future?! And always a single \n ... so implementing an official interface from the messages bundle seems a much more stable and future proof contract to me.

Alternatively would it help if messages implemented the serializable interface?

Not really as then it would require exact same class versions on client/server.

[mpkorstanje]

Would this be sufficient?

public final class ObjectToJsonWriter {
    
    @Override
    public void writeValue(Writer writer, Object value) throws IOException {
        Jackson.OBJECT_MAPPER.writeValue(writer, value);
    }
}

[laeubi]

@mpkorstanje yes that would of course be suitable as well, I just noticed that in this case an output stream might be more flexible? (and one maybe want to use that in the Message/Jsonformater then as well for consistency).

it would of course be great to have the reverse as well for symmetry:

public class JsonReader {
	public <T> T readValue(Class<T> type, byte[] buffer, int offset, int length) throws IOException {
		return Jackson.OBJECT_MAPPER.readerFor(type).readValue(buffer, offset, length);
	}

	public <T> T readValue(Class<T> type, InputStream inputStream) throws IOException {
		return Jackson.OBJECT_MAPPER.readerFor(type).readValue(inputStream);
	}
}

Thinking further in the sense of the API consistency it might be better to simply have a class implement

• io.cucumber.messages.MessageToNdjsonWriter.Serializer
• io.cucumber.messages.NdjsonToMessageIterable.Deserializer

this would also make clear that we write/get an Envelope here...

[mpkorstanje]

I'm going to think about it for a bit.

Providing a json serializer/deserializer has a lot of complexity attached. And I'm worried that it could open a maintaince sinkhole.

[laeubi]

I'm going to think about it for a bit.

Providing a json serializer/deserializer has a lot of complexity attached. And I'm worried that it could open a maintaince sinkhole.

Any progress on this? Because I noticed that the Jackson.OBJECT_MAPPER now is missing addModule(new ParameterNamesModule(Mode.PROPERTIES)) to de-serialize values I tried using jackson directly... but this clashes with other parts of cucumber classloader. This worked before because the messages shade jackson in an own package but I really want to avoid having yet another shading here...

Can you explain what maintenance problem you see here? The mapper is already so it does not feel anything is specifically needed here except the proposed class that almost never changes!

[mpkorstanje]

Any progress on this?

Not really. I'm in the middle of upgrading everything to Java 17 at the moment.

Can you explain what maintenance problem you see here? The mapper is already so it does not feel anything is specifically needed here except the proposed class that almost never changes!

There are multiple problems to solve:

1. A generic (json <-> Object) json de/serializer has more API surface than I care to take in consideration when making changes. It might not need to change often, but the potential impact is bigger than I care to think about.

2. Jackson 3 was released recently so changes will arrive sooner than you'd think. Likewise the module system poses its own set of challenges.

3. An argument can be made for providing a Envelope -> json API as part of cucumber-core. Cucumber would use that. But the json -> Envelope API would be unused in cucumber-core. It could however be used by many other parts of Cucumber, especially in tests.

4. In the long term I don't want to shade Jackson at all. I'd rather see that users bring Jackson 2, Jackson 3, or Gson themselves and Cucumber then utilizes what is available.

This all points towards creating a dedicated dependency for dealing with serialization and deserialization of messages. But I'm in the middle of upgrading everything to Java 17 at the moment so I haven't exactly gotten around to that yet.

[laeubi]

A generic (json <-> Object) json de/serializer has more API surface than I care to take in consideration when making changes.

I don't think it should be generic just for Envelope as this is the main "transport" type of cucumber messages.

Jackson 3 was released recently so changes will arrive sooner than you'd think. Likewise the module system poses its own set of challenges.

That's why I would love it to be abstracted here, I really don't mind how it works (e.g. jsonb would be my personal preference but as long as it works I don'T mind the technique).

So if there is a implementation agnostic way to convert an envelope to / from json I could use that without have to care much about the used technique.

An argument can be made for providing a Envelope -> json API as part of cucumber-core. Cucumber would use that. But the json -> Envelope API would be unused in cucumber-core. It could however be used by many other parts of Cucumber, especially in tests.

I would say if is use JSonFormatter to print the messages I want to have something to read it... what would then the purpose of it anyways? Sure I can use any json parser but as the structure is quite complex it seems natural to reuse the same lib to parse it into an object model that was used to produce the json.

In the long term I don't want to shade Jackson at all. I'd rather see that users bring Jackson 2, Jackson 3, or Gson themselves and Cucumber then utilizes what is available.

If you want that jakarta json / jsonb might be a good choice as it defines the API that then one can plug in any of the providers. Anyways it would be perfectly possible to still provide such reader/writer specifically targeting envelope so users do not need to know the details.

This all points towards creating a dedicated dependency for dealing with serialization and deserialization of messages. But I'm in the middle of upgrading everything to Java 17 at the moment so I haven't exactly gotten around to that yet.

I can wait a bit longer, but using json as the transport for cuumber events has just become a crucial part when interfacing different endpoint version so it is really powerful for the cucumber-eclipse so removing the need for replicate what cucumber already does would be a big win!

[mpkorstanje]

Cheers! Superseded by cucumber/messages-ndjson#1.

If you want that jakarta json / jsonb might be a good choice as it defines the API that then one can plug in any of the providers. Anyways it would be perfectly possible to still provide such reader/writer specifically targeting envelope so users do not need to know the details.

I had a look at that in cucumber/messages-ndjson#2 but jsonb isn't adopted by Gson or Jackson at the moment. And I'm not sure I'm up to writing my own wrapper. 😄

[mpkorstanje]

@laeubi you can now use

• https://github.com/cucumber/messages-ndjson for serialization and deserialisation.

In conjunction with:

• https://github.com/cucumber/messages/blob/main/java/src/main/java/io/cucumber/messages/MessageToNdjsonWriter.java
• https://github.com/cucumber/messages/blob/main/java/src/main/java/io/cucumber/messages/NdjsonToMessageIterable.java
• https://github.com/cucumber/messages/blob/main/java/src/main/java/io/cucumber/messages/NdjsonToMessageReader.java

For the latter you'll have to pass new Deserialiser()::readValue. Due to version constraints, that can't make it into messages-ndjson yet.

[laeubi]

jsonb isn't adopted by Gson or Jackson at the moment

Yeah that's too bad. Sadly even though json is very popular there is almost no common denominator for an API (yet) leading to all kind of brittle problems with dependency.

@laeubi you can now use
https://github.com/cucumber/messages-ndjson for serialization and deserialisation.

I'll take a look! Created

• Evaluate and migrate to messages-ndjson cucumber-eclipse#628

Due to version constraints, that can't make it into messages-ndjson yet.

I'm a bit confused where new Deserialiser is located?! It seems not to be part of cucumber-jvm either.

[mpkorstanje]

I'm a bit confused where new Deserialiser is located?! It seems not to be part of cucumber-jvm either.

It's located in the io.cucumber:messages-ndjson artifact, in the io.cucumber.messages.ndjson package.

It will be a transitive dependency of cucumber-core once published.

But I just realized I forgot to put the OSGI module name in. I'll do that soonish.