Introduce `Value_Type.Null`, fix edge cases for `read_many` #11737

radeusgd · 2024-12-02T19:01:44Z

Pull Request Description

Closes Introduce a Value_Type.Null. #6281

Important Notes

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

The documentation has been updated, if necessary.
Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
All code follows the
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
Unit tests have been written where possible.
If meaningful changes were made to logic or tests affecting Enso Cloud integration in the libraries,
or the Snowflake database integration, a run of the Extra Tests has been scheduled.
- If applicable, it is suggested to paste a link to a successful run of the Extra Tests.

radeusgd · 2024-12-03T17:37:26Z

Changes affect Snowflake so I'm running the Extra suite:

I expect it to fail because it is failing on develop. Yet I want to see if there are no more failures than what is expected.

radeusgd · 2024-12-03T17:44:18Z

Followup to fix Nothing * Nothing: #11751 and Nothing in expressions in DB #11757.

GregoryTravis · 2024-12-03T22:23:15Z

distribution/lib/Standard/Database/0.0.0-dev/src/Internal/SQLite/SQLite_Dialect.enso

@@ -628,6 +626,9 @@ make_custom_cast column target_value_type type_mapping =
    result = Ref.new Nothing


I find this logic hard to follow. The return value would be either Nothing or the return value of result.put expr -- but isn't the the return value of Ref.put the old value?

The return value of put is discarded.

The function returns ref.get at the very end.

I agree it is not ideal. This is a hack to get multiline if without adding a ton of indentation to the else branches. There's just too many branches here (and while sometimes too many branches is bad, here it is just intrinsic to this function) to make it readable.

Once we get multiline if (#6408) I will be more than happy to replace this with just:

if cond1 then ... else if cond2 then ... else Nothing

Do you think I should rewrite it with increasing indentation?

I'd rather keep it as is as IMO it is readable once the reasons are explained.

I see, it makes more sense now.

distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Value_Type_Helpers.enso

radeusgd · 2024-12-19T14:05:31Z

Followup to fix Nothing * Nothing: #11751 and Nothing in expressions in DB #11757.

I think we could merge this PR and then improve the missing bits from these 2 tickets in a follow-up PR. Unless there is any opposing views and we want to do it as a single change?

GregoryTravis · 2025-01-02T16:36:47Z

distribution/lib/Standard/Database/0.0.0-dev/src/Internal/SQLite/SQLite_Dialect.enso

@@ -628,6 +626,9 @@ make_custom_cast column target_value_type type_mapping =
    result = Ref.new Nothing


I see, it makes more sense now.

GregoryTravis · 2025-01-10T21:23:12Z

distribution/lib/Standard/Table/0.0.0-dev/src/Value_Type.enso

+                if bucket_1.is_nothing || bucket_2.is_nothing then False else
+                    bucket_1 == bucket_2


Suggested change

if bucket_1.is_nothing || bucket_2.is_nothing then False else

bucket_1 == bucket_2

bucket_1 == bucket_2

Since we know either_type_is_not_ordered is false here.

GregoryTravis · 2025-01-10T21:31:02Z

std-bits/table/src/main/java/org/enso/table/data/column/operation/unary/NotOperation.java

@@ -29,6 +31,10 @@ public ColumnStorage apply(
      return boolStorage.makeNegated();
    }

+    if (storage.getType() instanceof NullType) {
+      return new NullStorage(Math.toIntExact(storage.getSize()));


Is there a reason getSize() can return long but NullStorage can only accept int? Is an exception here possible?

getSize is the Enso-facing one so it returns long because that is the default storage for integers in Enso.

All of our storages take int size because most of them are backed by arrays (Null is inded an exception) so they cannot hold more than int can index.

Thanks for pointing this out, making me rethink this - long getSize() is just wrapper for int size() that 'translates' to the more directly Enso-friendly long value, but I should have just used size() here and avoided the conversion.

Hmm after further checking - this function gets ColumnStorage which only has the long returning one in its interface...

I guess I'll just keep the toIntExact conversion. It shouldn't happen as long as our storages are int based. Once they aren't - the other code will need to be updated anyway. Does that seem ok?

# Conflicts: # std-bits/table/src/main/java/org/enso/table/aggregations/Sum.java # std-bits/table/src/main/java/org/enso/table/data/column/builder/Builder.java # std-bits/table/src/main/java/org/enso/table/data/column/builder/TypedBuilderImpl.java # std-bits/table/src/main/java/org/enso/table/data/column/operation/cast/ToBigDecimalConverter.java # std-bits/table/src/main/java/org/enso/table/data/column/operation/cast/ToBigIntegerConverter.java # std-bits/table/src/main/java/org/enso/table/data/column/operation/cast/ToBooleanStorageConverter.java # std-bits/table/src/main/java/org/enso/table/data/column/operation/cast/ToDateStorageConverter.java # std-bits/table/src/main/java/org/enso/table/data/column/operation/cast/ToDateTimeStorageConverter.java # std-bits/table/src/main/java/org/enso/table/data/column/operation/cast/ToIntegerStorageConverter.java # std-bits/table/src/main/java/org/enso/table/data/column/operation/cast/ToTextStorageConverter.java # std-bits/table/src/main/java/org/enso/table/data/column/operation/cast/ToTimeOfDayStorageConverter.java # std-bits/table/src/main/java/org/enso/table/data/table/Column.java

# Conflicts: # std-bits/table/src/main/java/org/enso/table/data/column/builder/Builder.java # std-bits/table/src/main/java/org/enso/table/data/column/builder/ObjectBuilder.java # std-bits/table/src/main/java/org/enso/table/data/table/Column.java

radeusgd · 2025-01-21T18:56:34Z

std-bits/table/src/main/java/org/enso/table/data/table/Column.java

  public static Column fromRepeatedItem(
      String name, Value item, int repeat, ProblemAggregator problemAggregator) {
-    if (repeat < 0) {
-      throw new IllegalArgumentException("Repeat count must be non-negative.");
-    }
-
-    Object converted = Polyglot_Utils.convertPolyglotValue(item);
-
-    if (converted == null) {
-      var builder = Builder.getForType(AnyObjectType.INSTANCE, repeat, problemAggregator);
-      builder.appendNulls(repeat);
-      return new Column(name, builder.seal());
-    }
-
-    StorageType storageType = StorageType.forBoxedItem(converted);
-    Builder builder = Builder.getForType(storageType, repeat, problemAggregator);
-    Context context = Context.getCurrent();
-    for (int i = 0; i < repeat; i++) {
-      builder.append(converted);
-      context.safepoint();
-    }
-
-    return new Column(name, builder.seal());
+    return new Column(name, Storage.fromRepeatedItem(item, repeat, problemAggregator));


@jdunkerley I've allowed myself to overwrite some of your cleanups from the other PR in this particular method with a fully custom Storage::fromRepeatedItem.

It also relies on Builder.getForType for the generic case but also has 'specializations' that create a more compact storage directly (NullStorage and LongConstantStorage that are constructed in O(1) instead of O(N)).

I think we do want this 'optimization'. In the future we can even add more ConstantStorage kinds.

The major question is if Storage class is the right place for this method. I imagine once we start adding more backends it may not necessarily be. But for now I think it is relatively good place. So I'd keep it here and move it when it makes sense.

Does that seem OK?

Yes fine to do. Will rebase and cope :)

radeusgd · 2025-01-22T19:38:32Z

New Extra tests run after the new changes:

✅ https://github.com/enso-org/enso/actions/runs/12915770991

jdunkerley · 2025-01-24T10:31:35Z

std-bits/table/src/main/java/org/enso/table/data/table/Column.java

  public static Column fromRepeatedItem(
      String name, Value item, int repeat, ProblemAggregator problemAggregator) {
-    if (repeat < 0) {
-      throw new IllegalArgumentException("Repeat count must be non-negative.");
-    }
-
-    Object converted = Polyglot_Utils.convertPolyglotValue(item);
-
-    if (converted == null) {
-      var builder = Builder.getForType(AnyObjectType.INSTANCE, repeat, problemAggregator);
-      builder.appendNulls(repeat);
-      return new Column(name, builder.seal());
-    }
-
-    StorageType storageType = StorageType.forBoxedItem(converted);
-    Builder builder = Builder.getForType(storageType, repeat, problemAggregator);
-    Context context = Context.getCurrent();
-    for (int i = 0; i < repeat; i++) {
-      builder.append(converted);
-      context.safepoint();
-    }
-
-    return new Column(name, builder.seal());
+    return new Column(name, Storage.fromRepeatedItem(item, repeat, problemAggregator));


Yes fine to do. Will rebase and cope :)

radeusgd added the CI: No changelog needed Do not require a changelog entry for this PR. label Dec 2, 2024

radeusgd self-assigned this Dec 2, 2024

radeusgd force-pushed the wip/radeusgd/6281-value-type-null branch 3 times, most recently from c57057b to 22a2371 Compare December 3, 2024 17:22

radeusgd marked this pull request as ready for review December 3, 2024 17:23

radeusgd requested review from jdunkerley, GregoryTravis, AdRiley and marthasharkey as code owners December 3, 2024 17:23

GregoryTravis reviewed Dec 3, 2024

View reviewed changes

radeusgd added 16 commits December 19, 2024 15:12

Adding Value Type Null logic

e993a0f

fixes

6927176

wip

b2b606e

forgot to commit file

f647799

NullStorag

9c72ebb

WIP integrating NullStorage

fc63d39

fixing edge cases with Null type

bc43d3d

updating tests

55dfc54

can cast Null to anything

b1b8209

union with null should be possible for every type

df1db00

updating tests, check that Null is accepted in Data.read_many

6100b86

allow null col

f268d1b

typo

98a5aa8

postgres

d8068b9

re enable some union tests

95e4c73

re enable some agg tests

539a8fd

Merge branch 'develop' into wip/radeusgd/6281-value-type-null

e9fbef0

GregoryTravis approved these changes Jan 2, 2025

View reviewed changes

GregoryTravis approved these changes Jan 10, 2025

View reviewed changes

Merge branch 'develop' into wip/radeusgd/6281-value-type-null

75d51b6

AdRiley approved these changes Jan 15, 2025

View reviewed changes

radeusgd added 9 commits January 16, 2025 19:30

Merge branch 'develop' into wip/radeusgd/6281-value-type-null

ad380f2

fmt after merge

27bc84d

align after merge

b9a7758

mark builders final

562c3f3

mark ctor package private

0f535fc

fix usage

ec97093

re-add change lost in merge

4b45d7e

radeusgd commented Jan 21, 2025

View reviewed changes

radeusgd added 7 commits January 21, 2025 20:05

fix after merge

a8aeb7c

add a test for better coverage

c073786

revert fake bug introduced for testing purposes

71c9785

Merge branch 'develop' into wip/radeusgd/6281-value-type-null

777566b

re-enable the disabled tests

46575be

update tests: we do not expect Null column support is needed in DB

38efa05

Merge branch 'develop' into wip/radeusgd/6281-value-type-null

b29b3b3

radeusgd mentioned this pull request Jan 22, 2025

What is the desired behaviour for Database backends on Nothing * Nothing #11751

Closed

Merge branch 'develop' into wip/radeusgd/6281-value-type-null

b4c47ad

jdunkerley approved these changes Jan 24, 2025

View reviewed changes

CR

9b0136e

radeusgd added the CI: Ready to merge This PR is eligible for automatic merge label Jan 24, 2025

mergify bot merged commit bc32447 into develop Jan 24, 2025
42 of 44 checks passed

mergify bot deleted the wip/radeusgd/6281-value-type-null branch January 24, 2025 18:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce `Value_Type.Null`, fix edge cases for `read_many` #11737

Introduce `Value_Type.Null`, fix edge cases for `read_many` #11737

radeusgd commented Dec 2, 2024 •

edited

Loading

radeusgd commented Dec 3, 2024 •

edited

Loading

radeusgd commented Dec 3, 2024 •

edited

Loading

GregoryTravis Dec 3, 2024

radeusgd Dec 4, 2024 •

edited

Loading

radeusgd Dec 4, 2024

GregoryTravis Jan 2, 2025

radeusgd commented Dec 19, 2024 •

edited

Loading

GregoryTravis Jan 2, 2025

GregoryTravis Jan 10, 2025

GregoryTravis Jan 10, 2025

radeusgd Jan 13, 2025

radeusgd Jan 16, 2025

radeusgd Jan 21, 2025 •

edited

Loading

jdunkerley Jan 24, 2025

radeusgd commented Jan 22, 2025 •

edited

Loading

jdunkerley Jan 24, 2025

		@@ -628,6 +626,9 @@ make_custom_cast column target_value_type type_mapping =
		result = Ref.new Nothing

		if bucket_1.is_nothing \|\| bucket_2.is_nothing then False else
		bucket_1 == bucket_2

Introduce Value_Type.Null, fix edge cases for read_many #11737

Introduce Value_Type.Null, fix edge cases for read_many #11737

Conversation

radeusgd commented Dec 2, 2024 • edited Loading

Pull Request Description

Important Notes

Checklist

radeusgd commented Dec 3, 2024 • edited Loading

radeusgd commented Dec 3, 2024 • edited Loading

Choose a reason for hiding this comment

radeusgd Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

radeusgd commented Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

radeusgd Jan 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

radeusgd commented Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

Introduce `Value_Type.Null`, fix edge cases for `read_many` #11737

Introduce `Value_Type.Null`, fix edge cases for `read_many` #11737

radeusgd commented Dec 2, 2024 •

edited

Loading

radeusgd commented Dec 3, 2024 •

edited

Loading

radeusgd commented Dec 3, 2024 •

edited

Loading

radeusgd Dec 4, 2024 •

edited

Loading

radeusgd commented Dec 19, 2024 •

edited

Loading

radeusgd Jan 21, 2025 •

edited

Loading

radeusgd commented Jan 22, 2025 •

edited

Loading