[CALCITE-6652] RelDecorrelator can't decorrelate query with limit 1

[suibianwanwank]

This PR handles decorrelate since the query contains limit 1. There are two main types of conversions:

1. If sorted with no OFFSET and FETCH = 1, and only one collation field, rewrite the Sort as Aggregate using MIN/MAX function.

Example:
  Sort(sort0=[$0], dir0=[ASC], fetch=[1])
    input
Rewrite to:
  Aggregate(group=(corVar), agg=[min($0))

2. For the more general case,rewrite the sort to be

Aggregate(group=(corVar.. , field..))
  project(first_value(field) over (partition by corVar order by (sort collation)))
     input

[caicancai]

If no one reviews it, I will start reviewing it this weekend, it may take some time.

[suibianwanwank]

If no one reviews it, I will start reviewing it this weekend, it may take some time.

Thank you. If you have any questions or need further clarification, feel free to ask.

[mihaibudiu]

I will try to review some more later, after I understand the relationship between order by and min/max.
Perhaps it's enough to check that the orderby does null handling in the expected way.

[mihaibudiu]

"if the input has 0 rows."

I think that the references to the other calcite issue should not be in this comment - they will become obsolete if the issue is closed, and they don't really clarify what is going on here.

[mihaibudiu]

the check for fetch == 1 is not here (so you can mention that it's enforced by the caller).
Moreover, for symmetry, should this comment be within the if branch?

[mihaibudiu]

I am wondering: sorting has collation, but min/max do not.
For example, sorting can specify what happens to nulls.

[rubenada]

This was my exact same thought after a quick look at the PR... is the proposed conversion 100% equivalent in all cases, also when nulls are involved? What about when the original Sort is nulls-first and the relevant field has null values?

[suibianwanwank]

Yes, as mentioned in JIRA, I missed this. This conversion is only valid if nulls-last.

[rubenada]

Could we have some unit tests with nulls-first, to see the expected result being the conversion not being applied?

[mihaibudiu]

I would think that MAX works for NULL LAST only, and MIN for NULL FIRST.
Am I wrong?

[suibianwanwank]

For MIN/MAX, null values are ignored in the databases I know (I didn't find a corresponding description in the sql standard). So I think both asc and desc should be null last, i.e.return null only if all values are null.

[suibianwanwank]

@mihaibudiu @rubenada Thanks for your feedback! I've updated the code and added null value test, thanks to review when you have time.

[rubenada]

Maybe I'm a bit too paranoid, but just to be on the safe side, I'd propose the new methods (decorrelateFetchOneSort and decorrelateSortAsAggregate) to be protected instead of private; the reason is that if, for whatever reason, a downstream project wants to "switch off" this conversion, they could simply extend the RelDecorreletor with their own child class and override these methods to simply return null, right?

[suibianwanwank]

I agree that we need a safe way to avoid errors. Usually the user uses the decorrelate by calling decorrelateQuery. So I think is it possible to add something like SqlToRelConverter#CONFIG (which can be combined with [CALCITE-6674]) to choose whether to enable it or not. WDYT

[rubenada]

It may be a cleaner solution, but IMO maybe a bit overkill for this particular purpose. I think protected methods would suffice to provide a way to "fallback". Let's see if there are other opinions on this subject.

[suibianwanwank]

Got it, I've made the changes. Thanks for the suggestion.

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
91.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[mihaibudiu]

I am assuming that the quidem test results have been checked with another database as well.