Initial support for multiple languages #6

dkuku · 2022-12-03T10:49:54Z

Closes #4
I prepared initial support for multiple languages.
This can be enabled in config by specifying the languages key.
It uses the translations provided by timex but can be switched to anything else.
I tested it with pl and I left the tests commented out because it need's to be enabled for the test to work.

dbernheisel · 2022-12-03T17:12:39Z

Looking good! However I don't want to expand using Timex due to #1. I can help by creating an IANA file parser to provide the timezone abbreviations, and maybe use/borrow ex_cldr for translating months.

What do you think?

dkuku · 2022-12-03T17:16:59Z

Sure, I just used timex because it was already installed. I have problems using it in projects due to the max version pinning.

…

On Sat, 3 Dec 2022, 17:12 David Bernheisel, ***@***.***> wrote: Looking good! However I don't want to expand using Timex due to #1 <#1>. I can help by creating an IANA file parser to provide the timezone abbreviations, and maybe use/borrow ex_cldr for translating months. What do you think? — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG4X46EPHZJ7GK352XWUJTWLN5RHANCNFSM6AAAAAASSV2MV4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

dkuku · 2022-12-03T20:43:58Z

@dbernheisel I'm struggling to find in ex_cldr and the child libraries any functions that returns list of months and weekdays with the abbreviations. I may use some kind of hacks to get the values by formatting some fake date with only weekday or month but it's not the way to go

dbernheisel · 2022-12-03T21:23:10Z

@dkuku I tried this in Livebook, and it seems to work ok. You might conditionally check if the user has Cldr available, have the user pass in their Cldr backend, and date_time_parser use it to compile available translations. If the user doens't have Cldr in their system, I'm not sure I want to offer translations other than English.

ExCLDR Translation

Mix.install([
  {:ex_cldr_dates_times, "~> 2.0"}
])

Config

defmodule MyApp.Cldr do
  use Cldr,
    locales: ["en", "fr", "es"],
    default_locale: "en",
    providers: [Cldr.Number, Cldr.Calendar, Cldr.DateTime]
end

locales = MyApp.Cldr.known_locale_names()
anchor = ~D[2022-01-02]

for month <- 1..12 do
  date = Date.new!(anchor.year, month, anchor.day)

  trans =
    Enum.map(locales, fn l ->
      {l,
       [
         short: Cldr.DateTime.Formatter.month(date, 3, l, MyApp.Cldr),
         long: Cldr.DateTime.Formatter.month(date, 4, l, MyApp.Cldr)
       ]}
    end)

  {month, trans}
end

[
  {1,
   [
     en: [short: "Jan", long: "January"],
     es: [short: "ene", long: "enero"],
     fr: [short: "janv.", long: "janvier"]
   ]},
  {2,
   [
     en: [short: "Feb", long: "February"],
     es: [short: "feb", long: "febrero"],
     fr: [short: "févr.", long: "février"]
   ]},
  {3,
   [
     en: [short: "Mar", long: "March"],
     es: [short: "mar", long: "marzo"],
     fr: [short: "mars", long: "mars"]
   ]},
  {4,
   [
     en: [short: "Apr", long: "April"],
     es: [short: "abr", long: "abril"],
     fr: [short: "avr.", long: "avril"]
   ]},
  {5,
   [
     en: [short: "May", long: "May"],
     es: [short: "may", long: "mayo"],
     fr: [short: "mai", long: "mai"]
   ]},
  {6,
   [
     en: [short: "Jun", long: "June"],
     es: [short: "jun", long: "junio"],
     fr: [short: "juin", long: "juin"]
   ]},
  {7,
   [
     en: [short: "Jul", long: "July"],
     es: [short: "jul", long: "julio"],
     fr: [short: "juil.", long: "juillet"]
   ]},
  {8,
   [
     en: [short: "Aug", long: "August"],
     es: [short: "ago", long: "agosto"],
     fr: [short: "août", long: "août"]
   ]},
  {9,
   [
     en: [short: "Sep", long: "September"],
     es: [short: "sept", long: "septiembre"],
     fr: [short: "sept.", long: "septembre"]
   ]},
  {10,
   [
     en: [short: "Oct", long: "October"],
     es: [short: "oct", long: "octubre"],
     fr: [short: "oct.", long: "octobre"]
   ]},
  {11,
   [
     en: [short: "Nov", long: "November"],
     es: [short: "nov", long: "noviembre"],
     fr: [short: "nov.", long: "novembre"]
   ]},
  {12,
   [
     en: [short: "Dec", long: "December"],
     es: [short: "dic", long: "diciembre"],
     fr: [short: "déc.", long: "décembre"]
   ]}
]

for day <- 0..6 do
  date = Date.new!(anchor.year, anchor.month, anchor.day + day)

  trans =
    Enum.map(locales, fn l ->
      {l,
       [
         short: Cldr.DateTime.Formatter.day_name(date, 2, l, MyApp.Cldr),
         long: Cldr.DateTime.Formatter.day_name(date, 4, l, MyApp.Cldr)
       ]}
    end)

  {day, trans}
end

[
  {0,
   [
     en: [short: "Sun", long: "Sunday"],
     es: [short: "dom", long: "domingo"],
     fr: [short: "dim.", long: "dimanche"]
   ]},
  {1,
   [
     en: [short: "Mon", long: "Monday"],
     es: [short: "lun", long: "lunes"],
     fr: [short: "lun.", long: "lundi"]
   ]},
  {2,
   [
     en: [short: "Tue", long: "Tuesday"],
     es: [short: "mar", long: "martes"],
     fr: [short: "mar.", long: "mardi"]
   ]},
  {3,
   [
     en: [short: "Wed", long: "Wednesday"],
     es: [short: "mié", long: "miércoles"],
     fr: [short: "mer.", long: "mercredi"]
   ]},
  {4,
   [
     en: [short: "Thu", long: "Thursday"],
     es: [short: "jue", long: "jueves"],
     fr: [short: "jeu.", long: "jeudi"]
   ]},
  {5,
   [
     en: [short: "Fri", long: "Friday"],
     es: [short: "vie", long: "viernes"],
     fr: [short: "ven.", long: "vendredi"]
   ]},
  {6,
   [
     en: [short: "Sat", long: "Saturday"],
     es: [short: "sáb", long: "sábado"],
     fr: [short: "sam.", long: "samedi"]
   ]}
]

dkuku · 2022-12-03T22:44:17Z

Thanks-I was thinking about this but I was not sure about it, it felt a bit hacky. I'll try to do it tomorrow.

…

On Sat, 3 Dec 2022, 21:23 David Bernheisel, ***@***.***> wrote: @dkuku <https://github.com/dkuku> I tried this in Livebook, and it seems to work ok. You might conditionally check if the user has Cldr available, have the user pass in their Cldr backend, and date_time_parser use it to compile available translations. If the user doens't have Cldr in their system, I'm not sure I want to offer translations other than English. ------------------------------ ExCLDR Translation Mix.install([ {:ex_cldr_dates_times, "~> 2.0"} ]) Config defmodule MyApp.Cldr do use Cldr, locales: ["en", "fr", "es"], default_locale: "en", providers: [Cldr.Number, Cldr.Calendar, Cldr.DateTime] end locales = MyApp.Cldr.known_locale_names() for month <- 1..12 do trans = Enum.map(locales, fn l -> Cldr.DateTime.Formatter.month(Date.new!(2022, month, 1), 4, l, MyApp.Cldr) end) {month, trans} end [ {1, ["January", "enero", "janvier"]}, {2, ["February", "febrero", "février"]}, {3, ["March", "marzo", "mars"]}, {4, ["April", "abril", "avril"]}, {5, ["May", "mayo", "mai"]}, {6, ["June", "junio", "juin"]}, {7, ["July", "julio", "juillet"]}, {8, ["August", "agosto", "août"]}, {9, ["September", "septiembre", "septembre"]}, {10, ["October", "octubre", "octobre"]}, {11, ["November", "noviembre", "novembre"]}, {12, ["December", "diciembre", "décembre"]} ] — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG4X43CD6LUX5MZUUDTSTTWLO24RANCNFSM6AAAAAASSV2MV4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

dkuku · 2022-12-04T21:16:58Z

@dbernheisel It's working but I have problems to test it because the code is either compiled in one version or another.
I can switch between the implementations but I don't have any idea how to test it - easiest way would be to have 2 separate ci or refactor the code and have date_time_parser and date_time_parser_cldr packages.

dkuku · 2022-12-05T11:38:37Z

I'm thinking of some optimisation - instead of matching January and Jan we can try to just match all strings that are starting with Jan this should work out of the box with the current logic and reduce the amount of generated code.
I'm Polish and the month name can have multiple endings but it always starts the same. In cldr you have month and standalone_month which for english are the same but for other languages may not always be true - the abbrviation will work for all cases.

dbernheisel · 2022-12-22T14:35:59Z

@dkuku Once we have some tests added, let's get this merged! It's looking good.

dkuku · 2022-12-22T14:38:23Z

@dkuku Once we have some tests added, let's get this merged! It's looking good.

👍🏻 I can look into it during holidays.

dkuku · 2022-12-27T22:54:00Z

I can compile it and run the tests with mix tests - note the s at the end.
But I had to disable the warnings as errors check in mix.exs

dbernheisel · 2023-02-28T14:34:54Z

I have not forgotten about this :) I'm going to incorporate this in with some of my changes and get this going. I appreciate your work and your patience.

dkuku · 2023-02-28T14:38:21Z

That's why the 'allow edits from maintainers' checkbox is here - looking to see the next release :)

dbernheisel · 2023-03-09T14:42:59Z

Unfortunately there's an issue with this approach. In the combinators.ex.eex, we're using @month_map which gets compiled into the real combinators file by Nimble Parsec.

Where this is ok is in a function like vocal_month_to_numeric_month since that's called during runtime and the map is copied over into the final file, but the @months_map is also used by Nimble Parsec to construct the functions to know what to look for; this means that whenever a user adds :date_time_parser to their app, they'd have to re-compile combinators.ex.eex into combinators.ex in order to benefit from their Cldr.Backend which provides the translations.

Maybe this is ok, but it would mean that NimbleParsec would have to be a non-dev dependency and will be a 2-phase compilation again.

dkuku · 2023-03-09T15:08:13Z

Yes, you're right. But this would pe only needed if someone want's to configure the parser for custom language. By default it's not needed and the dependency would only be needed in dev envs for recompilation.

…

On Thu, 9 Mar 2023, 14:43 David Bernheisel, ***@***.***> wrote: Unfortunately there's an issue with this approach. In the combinators.ex.eex, we're using @month_map which gets compiled into the real combinators file by Nimble Parsec. Where this is ok is in a function like vocal_month_to_numeric_month since that's called during runtime and the map is copied over into the final file, but the @months_map is also used to determine which strings to look for and is inlined by NimbleParsec into functions; this means that whenever a user adds :date_time_parser to their app, they'd have to re-compile combinators.ex.eex into combinators.ex in order to benefit from their Cldr.Backend which provides the translations. Maybe this is ok, but it would mean that NimbleParsec would have to be a non-dev dependency and will be a 2-phase compilation again. — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG4X43ZJZPVG7ISLSBCG2LW3HT75ANCNFSM6AAAAAASSV2MV4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

dbernheisel · 2023-12-06T19:41:37Z

I'm going to close for now, mostly because of length of time open, but also there's an order-of-operations issue with this approach in that it requires cldr languages to be present during the build process of the library, and not the build process of the app using the library. We'll need to re-think the approach.

support_multiple_languges

cef304e

dkuku force-pushed the support_multiple_languges branch from 5b235d2 to cef304e Compare December 3, 2022 10:56

use cldr when available

41eb436

dkuku added 2 commits December 4, 2022 22:19

remove jason from deps

993e090

reset generated file

27e55f8

enable tests with cldr

00d4b21

dbernheisel added 4 commits March 9, 2023 02:32

Merge branch 'main' into support_multiple_languges

f13bed5

Update mix.exs

bb111ed

Update mix.exs

67be992

Merge branch 'main' into support_multiple_languges

f88ddff

dbernheisel closed this Dec 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial support for multiple languages #6

Initial support for multiple languages #6

dkuku commented Dec 3, 2022 •

edited

Loading

dbernheisel commented Dec 3, 2022

dkuku commented Dec 3, 2022 via email

dkuku commented Dec 3, 2022

dbernheisel commented Dec 3, 2022 •

edited

Loading

dkuku commented Dec 3, 2022 via email

dkuku commented Dec 4, 2022

dkuku commented Dec 5, 2022 •

edited

Loading

dbernheisel commented Dec 22, 2022

dkuku commented Dec 22, 2022

dkuku commented Dec 27, 2022

dbernheisel commented Feb 28, 2023

dkuku commented Feb 28, 2023

dbernheisel commented Mar 9, 2023 •

edited

Loading

dkuku commented Mar 9, 2023 via email

dbernheisel commented Dec 6, 2023

Initial support for multiple languages #6

Initial support for multiple languages #6

Conversation

dkuku commented Dec 3, 2022 • edited Loading

dbernheisel commented Dec 3, 2022

dkuku commented Dec 3, 2022 via email

dkuku commented Dec 3, 2022

dbernheisel commented Dec 3, 2022 • edited Loading

ExCLDR Translation

Config

dkuku commented Dec 3, 2022 via email

dkuku commented Dec 4, 2022

dkuku commented Dec 5, 2022 • edited Loading

dbernheisel commented Dec 22, 2022

dkuku commented Dec 22, 2022

dkuku commented Dec 27, 2022

dbernheisel commented Feb 28, 2023

dkuku commented Feb 28, 2023

dbernheisel commented Mar 9, 2023 • edited Loading

dkuku commented Mar 9, 2023 via email

dbernheisel commented Dec 6, 2023

dkuku commented Dec 3, 2022 •

edited

Loading

dbernheisel commented Dec 3, 2022 •

edited

Loading

dkuku commented Dec 5, 2022 •

edited

Loading

dbernheisel commented Mar 9, 2023 •

edited

Loading