4.x: Add advanced shard awareness

[Bouncheck]

Adds advanced shard awareness to the 4.x version of the driver.
Added integration test that checks if reconnections are sufficiently rare with advanced shard awareness.
should_struggle_to_fill_pools and should_not_struggle_to_fill_pools can be run locally to see the difference between the driver with and without this feature.

Fixes #515

[Bouncheck]

I'll probably introduce some small checks to port allocator (for example the range should not be too small) and some unit tests but the gist of the PR will probably look as it is now.

[dkropachev]

@Bouncheck , I realy don't like idea of PortAllocator, there is a gap between when you allocate port and create a socket, if by then port become busy, then connection attempt fails.
I have pushed one commit to address that, please take a look.

[Bouncheck]

@Bouncheck , I realy don't like idea of PortAllocator, there is a gap between when you allocate port and create a socket, if by then port become busy, then connection attempt fails. I have pushed one commit to address that, please take a look.

There always is that possibility. PortIterator suffers from the same thing. The only solution is to try again.

From what I see PortIterator does not check if returned port was free at all.
Multiple instances use independent counters, which can result in more situations where connections snatch ports from each other. Note that PortAllocator uses global atomic counter, so subsequent calls will not return the same port number unless a very small range of free ports is given.
I think out of the two I prefer the allocator.

[dkropachev]

@Bouncheck , I realy don't like idea of PortAllocator, there is a gap between when you allocate port and create a socket, if by then port become busy, then connection attempt fails. I have pushed one commit to address that, please take a look.

There always is that possibility. PortIterator suffers from the same thing. The only solution is to try again.

From what I see PortIterator does not check if returned port was free at all. Multiple instances use independent counters, which can result in more situations where connections snatch ports from each other. Note that PortAllocator uses global atomic counter, so subsequent calls will not return the same port number unless a very small range of free ports is given. I think out of the two I prefer the allocator.

It does that when socket is actually allocated, and no, it doesn't suffer from it, it will retry on next port if port is busy

[dkropachev]

@Bouncheck , also another reason to stick to PortIterator is that it keeps behavior inline with OS and driver socket configurations:

1. DefaultDriverOption.SOCKET_REUSE_ADDRESS
2. UnixChannelOption.SO_REUSEPORT
3. sysctl net.ipv4.tcp_tw_reuse

[Bouncheck]

@Bouncheck , I realy don't like idea of PortAllocator, there is a gap between when you allocate port and create a socket, if by then port become busy, then connection attempt fails. I have pushed one commit to address that, please take a look.

There always is that possibility. PortIterator suffers from the same thing. The only solution is to try again.
From what I see PortIterator does not check if returned port was free at all. Multiple instances use independent counters, which can result in more situations where connections snatch ports from each other. Note that PortAllocator uses global atomic counter, so subsequent calls will not return the same port number unless a very small range of free ports is given. I think out of the two I prefer the allocator.

It does that when socket is actually allocated, and no, it doesn't suffer from it, it will retry on next port if port is busy

Both solutions are just black boxes that return an int when asked. In the time between they return an int and bootstrap.connect(...) is called the port with the same number can be stolen. So both do suffer from it and it cannot be really solved other than trying the whole process another time.

The additional bind exception handling you've added is separate from portIterator and can be done with any other port number picking black box as well, but it does not solve this problem (does not prevent stealing the port).

[Bouncheck]

@Bouncheck , also another reason to stick to PortIterator is that it keeps behavior inline with OS and driver socket configurations:

1. `DefaultDriverOption.SOCKET_REUSE_ADDRESS`

2. `UnixChannelOption.SO_REUSEPORT`

3. sysctl net.ipv4.tcp_tw_reuse

I don't understand this part

[dkropachev]

@Bouncheck , also another reason to stick to PortIterator is that it keeps behavior inline with OS and driver socket configurations:

1. `DefaultDriverOption.SOCKET_REUSE_ADDRESS`

2. `UnixChannelOption.SO_REUSEPORT`

3. sysctl net.ipv4.tcp_tw_reuse

I don't understand this part

When you create a socket or bind it, the OS is generally expected to return an error if the specified port and/or address is already in use.
However, there are exceptions to this behavior, and they are influenced by two main factors:

Socket options:

SO_REUSEPORT and SO_REUSEADDR (exposed in Netty via ChannelOption.SO_REUSEADDR and UnixChannelOption.SO_REUSEPORT).

At least one of these is also configurable through the driver using DefaultDriverOption.SOCKET_REUSE_ADDRESS.

OS-level settings:

For example, on Linux, the sysctl setting net.ipv4.tcp_tw_reuse.

The outcome depends on the combination of SO_REUSEPORT, SO_REUSEADDR, net.ipv4.tcp_tw_reuse, and the current state of the address/port.
These rules differ between bind and listen operations.

Since you're calling listen first, the behavior will follow the rules specific to listen.

[dkropachev]

LGTM, one more test needed and and couple of doc issues.

[dkropachev]

Does it ? I don't see any difference in this regard between regular shard awarness and advanced, if anything it will reduce number of reconnections needed to fullfill connection pool.

[Bouncheck]

If it decreases the number of reconnections needed then it lessens the load, no? Of course all driver applications will need to use it for best results.
Maybe I'm using a wrong term here?

[dkropachev]

Can you please refrase it so that it does not confuse.

[Bouncheck]

Sure, but which part? I don't know which part is the problem

[dkropachev]

# Results in smaller connection storms in multi-client settings. -> # Enabling it reduces amount of reconnections driver does to get connection pool filled.

[Bouncheck]

changed to Reduces number of reconnections driver needs to fully initialize connection pool.

[Bouncheck]

Addresed docs comments and added 1 change to port allocator. It will follow reuse address setting from the config.
Reasoning is that in theory port allocator using false all the time could prevent the driver from using some port that would actually work if reuse setting was respsected.

[Bouncheck]

Squashed fixes into appropriate commits and added missing @RunWith(DataProviderRunner.class)