feat: Improve support for Decimal DType

[FBruzzesi]

Description

Adds precision and scale parameters to Decimal dtype.

TODO: Decide on stable API

What type of PR is this? (check all applicable)

• 💾 Refactor
• ✨ Feature
• 🐛 Bug Fix
• 🔧 Optimization
• 📝 Documentation
• ✅ Test
• 🐳 Other

Related issues

• Closes [Enh]: Improve support for Decimal dtype #3375
• Blocking Enh: Add support for precision and scale keywords for Decimal dtype FBruzzesi/anyschema#101

Checklist

• Code follows style guide (ruff)
• Tests added
• Documented the changes

[FBruzzesi]

Regarding the stable API, how do we feel for this feature? I would say that keeping it backward compatible for v2 might be a stretch, but for v1 it would be ok. cc @MarcoGorelli

[MarcoGorelli]

thanks for your pr!

i was under the impression that precision was going to be removed, but it looks like that didn't happen, i'll study pola-rs/polars#19784 and get back to this

[dangotbanned]

thanks for your pr!

i was under the impression that precision was going to be removed, but it looks like that didn't happen, i'll study pola-rs/polars#19784 and get back to this

To save you a long read, there was nothing conclusive in that issue.
But the PR that closed it does have a summary that's more digestible

• feat: Use fixed-scale Decimals pola-rs/polars#24542

On the Python-side, all that's changed is precision=38 gets assigned if you don't provide one to Decimal.

There are a lot more changes on the Rust-side (who would've guessed that?)
It appears that precision is here to stay, but there are a lot of (positive?) behavioral changes which we may have a tough time trying to align between backends.

I'm not pitching we do/don't do that - just trying to summarize the old-fashioned way 😉

[MarcoGorelli]

thanks @FBruzzesi !

[MarcoGorelli]

not sure what i'm missing but why not

precision: int = 38

?

[dangotbanned]

I suppose if we got an old pl.Decimal, that could have a None already?

Otherwise - yeah if we can avoid introducing the contructor with a None - I'm on board 😄

[FBruzzesi]

It's solely to mirror polars signature: polars.datatypes.Decimal and code

    def __init__(
        self,
        precision: int | None = None,
        scale: int = 0,
    ) -> None:
        if precision is None:
            precision = 38


        self.precision = precision
        self.scale = scale

[FBruzzesi]

@MarcoGorelli thanks for reviewing this. I have a couple of questions that might need to be discussed before merging:

1. Stable API - What's your take here?
2. Polars Decimal allows for any value of precision and scale at datatype initialization, however it raises an InvalidOperationError at runtime if scale is greater than precision. See example below. I would argue that to standardize the error message, we could raise such error during initialization.

Example:

import polars as pl

dtype = pl.Decimal(2, 3)  # Ok so far
pl.Series([1.2]).cast(dtype)  
> InvalidOperationError: scale must be less than or equal to precision

[dangotbanned]

@FBruzzesi

RE: (#3377 (comment)), (#3377 (comment))
I thought Decimal was marked as unstable in narwhals, but it looks like it wasn't? 🤔

@MarcoGorelli did you mean to introduce it as stable in #1571?
polars didn't stablize until a month after pola-rs/polars#24542 in pola-rs/polars#25020

[dangotbanned]

2. Polars Decimal allows for any value of precision and scale at datatype initialization, however it raises an InvalidOperationError at runtime if scale is greater than precision.
   See example below. I would argue that to standardize the error message, we could raise such error during initialization.

@FBruzzesi I agree, let's do that

Note

Sorry for blowing up CI with pandas>=3 warnings 🫣
https://github.com/narwhals-dev/narwhals/pull/3377/checks?sha=075ceb86353c6a511c068ea2bbf5986d83e3ccd9