Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Working with auxiliary data files #20

Merged
merged 8 commits into from
Mar 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 10 additions & 9 deletions docs/src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,13 @@ The code is available at:

The method is described in the following articles:

* Barth, A., Alvera-Azcárate, A., Licer, M., & Beckers, J.-M. (2020). DINCAE 1.0: a convolutional neural network with error estimates to reconstruct sea surface temperature satellite observations. Geoscientific Model Development, 13(3), 1609–1622. https://doi.org/10.5194/gmd-13-1609-2020
* Barth, A., Alvera-Azcárate, A., Ličer, M., & Beckers, J.-M. (2020). DINCAE 1.0: a convolutional neural network with error estimates to reconstruct sea surface temperature satellite observations. Geoscientific Model Development, 13(3), 1609–1622. https://doi.org/10.5194/gmd-13-1609-2020
* Barth, A., Alvera-Azcárate, A., Troupin, C., & Beckers, J.-M. (2022). DINCAE 2.0: multivariate convolutional neural network with error estimates to reconstruct sea surface temperature satellite and altimetry observations. Geoscientific Model Development, 15(5), 2183–2196. https://doi.org/10.5194/gmd-15-2183-2022

The neural network will be trained on the GPU. Note convolutional neural networks can require a lot of GPU memory depending on the domain size.
So far, only NVIDIA GPUs are supported by the neural network framework `Knet.jl` used in DINCAE (beside training on the CPU but which is prohibitively slow).

# So far, only NVIDIA GPUs are supported by the neural network framework `Knet.jl` used in DINCAE (beside training on the CPU but which is prohibitively slow).
[`Flux.jl`](https://github.com/FluxML/Flux.jl) supports NVIDIA GPUs as well as other brands (see https://fluxml.ai/Flux.jl/stable/gpu/ for details).
Training on the CPU can be performedi, but it is prohibitively slow.

## User API

Expand All @@ -43,34 +44,34 @@ Convolutional neural networks can require "a lot" of GPU memory. These parameter
* reduce the mini-batch size
* use fewer layers (e.g. `enc_nfilter_internal` = [16,24,36] or [16,24])
* use less filters (reduce the values of the optional parameter enc_nfilter_internal)
* use a smaller domain or lower resolution
* use a smaller domain or a lower resolution


## Troubleshooting

### Installation of cuDNN

If you get the warniong `Package cuDNN not found in current path` or the error `Scalar indexing is disallowed`:
If you get the warning `Package cuDNN not found in current path` or the error `Scalar indexing is disallowed`:

```
```julia
julia> using DINCAE
┌ Warning: Package cuDNN not found in current path.
│ - Run `import Pkg; Pkg.add("cuDNN")` to install the cuDNN package, then restart julia.
│ - If cuDNN is not installed, some Flux functionalities will not be available when running on the GPU.
```


You need to install and load cuDNN before calling a function in DINCAE.jl:
You need to install and load cuDNN before calling a function in `DINCAE.jl`:

``` julia
```julia
using cuDNN
using DINCAE
# ...
```

### Dependencies of DINCAE.jl

`DINCAE.jl` depends on `Flux.jl` and `CUDA.jl` which will automatically be installed.
`DINCAE.jl` depends on `Flux.jl` and `CUDA.jl`, which will automatically be installed.
If you have some problems installing these package you might consult the
[documentation of `Flux.jl`](http://fluxml.ai/Flux.jl/stable/#Installation) or
[`CUDA.jl`](https://cuda.juliagpu.org/stable/installation/overview/).
12 changes: 10 additions & 2 deletions src/points.jl
Original file line number Diff line number Diff line change
Expand Up @@ -711,13 +711,21 @@ function reconstruct_points(
ds_.attrib["learning_rate_decay_epoch"] = learning_rate_decay_epoch
ds_.attrib["min_std_err"] = min_std_err
ds_.attrib["loss_weights_refine"] = Vector{Float64}(collect(loss_weights_refine))
ds_.attrib["auxdata_files"] = Vector{String}(collect(auxdata_files))
if length(auxdata_files) > 0
ds_.attrib["auxdata_filenames"] = getindex.(auxdata_files,:filename)
ds_.attrib["auxdata_varnames"] = getindex.(auxdata_files,:varname)
ds_.attrib["auxdata_errvarname"] = getindex.(auxdata_files,:errvarname)
else
@debug("No auxiliary data files")
ds_.attrib["auxdata_filenames"] = ""
ds_.attrib["auxdata_varnames"] = ""
ds_.attrib["auxdata_errvarname"] = ""
end
ds_.attrib["savesnapshot"] = Int(savesnapshot)
ds_.attrib["laplacian_penalty"] = laplacian_penalty
ds_.attrib["laplacian_error_penalty"] = laplacian_error_penalty
end
end

# Write
return losses
end
Loading