Initial support for Dash.jl component generation

[rpkyle]

This PR proposes to contribute modifications to dash-generate-components which enable generation of components for use with the Julia version of the Dash framework.

• Support for generating individual Julia components with docstrings
• Changes to the dash-generate-components argument parser
• Add Julia reserved keywords list to Dash, and omit any properties which match an element in this list
• Generation of module metadata (e.g. Project.toml)
• Updating component template to reflect any changes to the API required for general parity with Dash for Python & Dash for R

For the moment, the proposed syntax to generate Julia components includes a prefix, as with the R component generator (there are proposals to allow importing Julia modules with aliases, as in Python, though I don't believe this is available yet, based on JuliaLang/julia#1255).

dash-generate-components ./src/components dash_html_components -p package-info.json --r-prefix 'html' --jl-prefix 'html'

Closes #1189.

@Marc-Andre-Rivet @waralex

[mbauman]

There's nothing particularly special about Project.toml — and you don't need to use Julia to create it. The only challenge is the creation of the package UUID. It must remain constant between versions and must be distinct from all other UUIDs. I think what you could do is something like:

import uuid
u = uuid.UUID('c3754af0-4ebb-44ab-a1a8-173fddf15295')
uuid.UUID(hex=u.hex[:-12] + hex(hash(__name__))[-12:])

That is, use a pre-computed (and shared) UUID, but replace the last 12 hex digits with some bits from the hashed package name.

[mbauman]

For the moment, the proposed syntax to generate Julia components includes a prefix, as with the R component generator (there are proposals to allow importing Julia modules with aliases, as in Python, though I don't believe this is available yet, based on JuliaLang/julia#1255)

Yeah, we don't yet have a nice import DashCoreComponents as DCC syntax, but it's as easy as:

import DashCoreComponents
const DCC = DashCoreComponents

Personally, I'd be happy doing a using DashCoreComponents and just using dropdowns and inputs directly (without qualification).

[rpkyle]

Thanks @mbauman, these responses are most helpful.

I was mostly worried about the potential for collisions between existing Julia functions and the current component interface (particularly those from imported modules).

To avoid conflicts like the one between the Graph component within dashCoreComponents and dashDesignKit (using the R syntax here), we preface with the abbreviation, my inclination would just be to do the same here.

Realistically, I'm not as comfortable with this language as I am with R, so perhaps my concerns are misplaced. For now, it seems workable, curious what others think.

@Marc-Andre-Rivet @chriddyp @alexcjohnson

[waralex]

In terms of referencing things like input, my main concern is the potential for collisions between existing Julia functions and the current component interface (particularly those from imported modules).

It seems to me that the more significant problem is the name conflict in different modules. For example label is present in html and bootstrap component modules and I think not only in them

[rpkyle]

In terms of referencing things like input, my main concern is the potential for collisions between existing Julia functions and the current component interface (particularly those from imported modules).

It seems to me that the more significant problem is the name conflict in different modules. For example label is present in html and bootstrap component modules and I think not only in them

I believe we'd address this by declaring html_label and dbc_label (assuming dbc is the abbreviated form of DashBootstrapComponents.jl as it often is when importing dash-bootstrap-components in Python), but maybe I'm missing something here.

[rpkyle]

@waralex is currently in the process of updating the component generator templates for Julia, and he'll commit them to this branch when ready for review.

[mbauman]

Instead of simply putting these into deps, it'd be really awesome to register these files (js/css/maps) as Artifacts. The main motivation for me would be that it'd allow the Julia PackageCompiler — that is the tool that can compile Julia code into redistributable binaries — to find and include them in its binaries. It's really helpful for deployments.

[waralex]

Does it make sense to create Artifacts for data that can only be used with a certain package and are meaningless for everyone else? I haven't worked with Artifacts yet, but from the documentation, it seemed to me that this is a solution for data that is independent of concrete packages.

And another question - is there any approximate data how many people have already switched to 1.3 or 1.4? For example, a month ago I was asked to make DataFrameDBs compatible with 1.2

[waralex]

Theoretically, we may not register components as packages at all. Instead, we can create artifacts with resources and a JSON file with component descriptions. And we could add a function or macros to Dash that generates functions for components based on the artifact name and adds resources to the index page. But this makes the Dash interface different from the interface in Python and R.

Most likely, you can even make the interface as similar as possible to what it is. Something like

using Dash
@using_components dash_html_components
@using_components dash_core_components

@rpkyle as your opinion, can it be worth trying to do so, to see how it will look?

[rpkyle]

Hi @mbauman Does "deployments" here refer to deploying a Dash.jl web application? Or just deploying collections of Julia-related data into containerized environments?

@waralex Having read through https://julialang.org/blog/2019/11/artifacts/ briefly, I could envision scenarios in which registering data for Dash component libraries as artifacts does make sense.

For example, the dashBio package in R has sample data, which I think would be worth registering as an artifact instead of downloading/rebundling within the Juila package itself. (Assuming I understand what Matt is getting at, of course.)

[mbauman]

Yes, I mean deploying the Dash.jl web app to a service. For example, in my POC demo, I used PackageCompiler to compile Dashboards.jl (and all other dependencies) in order to improve its performance on the slow free Heroku dynos... but I had to keep the source around since that's where these assets lived. If these were artifacts, I could essentially create a much lighter-weight binary executable.

And yes, we're actively working on workflows for improving data-as-artifacts as it can then provide complete reproducibility (since the artifacts themselves are content-hashed and versioned) — and allow you to remove large datasets from Git repos.

[staticfloat]

Hello! I'm the author of the Artifacts system; @mbauman asked me to come in and discuss artifacts with you all.

I haven't worked with Artifacts yet, but from the documentation, it seemed to me that this is a solution for data that is independent of concrete packages.

This is true in some senses, but not true in others. The data is independen" in that it is stored in a separate directory, but it is not independent in that the code of your package can be as reliant on the artifact data as you want. Examples of artifact usage in the ecosystem right now include:

• Precompiled dynamic libraries
• Large data files that shouldn't live inside a git repository
• Small pieces of data that many packages all need to access

Note that Artifacts are immutable, we don't allow writing to them. This is part of our master plan to increase reliability and reproducibility within the Julia package ecosystem; we want to eventually set package directories to be read-only by default, and disallow packages the ability to modify themselves. To this day, a disheartening percentage of the bug reports we get have to do with local state getting wedged somehow, and package authors not designing their packages such that it can recover from a half-finished tarball extraction into their deps folder, for example. To that end, whenever we see new packages downloading things into deps/ we immediately jump to Artifacts as a good solution.

A side-benefit, as @mbauman pointed out, is that Artifacts implicitly perform their file path calculating at runtime, which avoids embedding file paths into precompilation files. This allows a compiled form of your code to be relocatable; and enables users to create and distribute applications that bundle Dash.

I'm happy to help out if there are any questions/concerns about using Artifacts.

[alexcjohnson]

I'm not sure if there are functional consequences of this, but at least from a documentation perspective might it be better to include more number types? Float64 | Int64 looks like it covers the output of JSON2.read, ie values to be received by callback functions. But values you can send back as callback returns are broader, likely Real.

[waralex]

This is only used in documentation. Yes, it makes sense to replace types with abstract ones.
Abstract types for numbers looks like this:

abstract type Number end
abstract type Real     <: Number end
abstract type AbstractFloat <: Real end
abstract type Integer  <: Real end
abstract type Signed   <: Integer end
abstract type Unsigned <: Integer end

[alexcjohnson]

OK good, let's go with Real.

[mbauman]

There's also the any=lambda: "Bool | Float64 | String | Dict | Array" below. I remember finding these annotations very helpful when looking at the online Python docs. And we can express this directly in Julia with Union{...}s.

In fact, it looks like we can use the union lambda below to print out real Julian Union{}s using your existing architecture. I'm afraid it'd take further restructuring to make an options block like this format like the below for Julia, but this would be closer to how we'd naturally document these things:

• label::Union{String, Real} (required): The checkbox's label
• value::Union{String, Real} (required): The value of the checkbox. This value corresponds to the items specified in the value property.
• disabled::Bool (optional): If true, this checkbox is disabled and can't be clicked on.

All that said, this sort of strict notation doesn't work as well for arrays and dicts, though, since we support far more than just Array and Dict. The builtin Array is just one of many possible AbstractArrays, and JSON2 also supports serializing Tuples and Sets the same way. Similarly, the builtin Dict is just one of many dictionary-like things; JSON2 also supports NamedTuples and even your own structs. That said, I don't want to lean too heavily on what the particular JSON library does. This gets into the nitty gritty, but JSON2 should probably serialize all AbstractDicts as JSON dicts; it doesn't right now. It may be good to move to JSON3 in the future which has a far more robust manner of flagging these types and is even speedier, so tying yourself to a particular implementation isn't the best.

One way we distinguish between the concrete implementations (like Array and Dict) and their general behaviors in our own documentation is just to use lowercase without code formatting, simply: array and dictionary. And if I were using these sorts of more relaxed terms, I wouldn't use the Union{} notation at all.

So all that to say, maybe don't uppercase any of these guys? Any maybe keep the (x | y | z) union printing?

[waralex]

The point is that these types are actually mapping what JS expects and the question of linking them to Julia types is fairly conditional. I don't know how to write this in the documentation to make it clear. There is no verification of these types at the Julia level. To json dict also mapped NamedTuple, for example and Tuple mapped to json array.

[alexcjohnson]

these are types at the js level, not at the Julia level

Yes, but the Julia app developer needs to know two things:

• What type(s) can they expect to receive in callbacks?
• What type(s) can they provide as prop values, either as callback return values or in layout components?

The more precision we can provide around these two issues the better. The first is easier, as the set of possible inputs from a JSON string is fairly limited. And then I assume that JSON2 is at least symmetric in the sense that JSON2.write(JSON2.read(s))==s - ie if you provide a return value consistent with the argument value we would provide, it will always work.

Given that, I'd prefer that we base our component documentation (ie this generated code) on this more restricted set of types you get from de-serializing JSON, and then somewhere in our general dash.jl docs we can describe the other types that you can provide as prop values - and we can frame this explicitly as "Anything that JSON2 (or JSON3 if we switch to that) will serialize the same way as type X."

In Dash for Python we've found that folks get confused when these types don't match - for example the DCC date pickers can be initialized using Python datetime objects, because they have a standard JSON serialization, but when you create a callback it will receive the date as a string. So we document this as a string type and explicitly stringify it in our main example:

date=str(dt(2017, 8, 25, 23, 59, 59))

(though I notice in later examples you'll see a datetime passed in directly - we should probably switch them all to strings, and/or make an explicit note about serializing datetime objects!)

[waralex]

You should also keep in mind that each argument is converted using Front.to_dash before converting to JSON. By default, this is an identical conversion. But it can be overloaded with third-party packages. For example you can use it to pass a PlotlyBase object instead of dcc_graph

[rpkyle]

@alexcjohnson changed to Real in fd24e50

[alexcjohnson]

As @mbauman mentioned we should also update this in any

[rpkyle]

fixed in 7eb5022

[rpkyle]

@alexcjohnson @waralex My understanding is that we're holding this PR until we settle on a final configuration for the core packages, and a suitable method for ensuring users always have the right combination of Dash + core component libraries, and that installing Dash installs them at the same time -- as described in more detail in plotly/Dash.jl#22.

[alexcjohnson]

The one remaining issue we've discussed here is that the generated Julia components end up in src alongside JavaScript source files - the actual source. This seems to be a hard requirement of Julia's package system, at least until Julia 1.5 and it'll probably be some time after that comes out before we're willing to require it.

For the time being we'll just accept the messiness of combined src directories, but in the future we may either (a) move the JS source to a different dir, or (b) generate the Julia package into a separate repo. In fact we might even do (a) for dash-component-boilerplate and our non-core packages, to make it easier for 3rd-party packages to support Julia, and (b) for the core packages. But this can all happen later.