update docs

Author: LongxingTan

.gitignore

@@ -18,4 +18,5 @@ MYIDEA.md
 **/.ipynb_checkpoints/
 /examples/data/crawer_data.py
 /reference/*
-/models/*
+/models/variables/*
+/data/*

README.md

@@ -1,9 +1,10 @@
 # Time series prediction
 This repo implements the common methods of time series prediction, especially deep learning methods in TensorFlow2. 
-It's highly welcomed to contribute if you have better idea, just create a PR. If any question, feel free to open an issue.
+It's highly welcomed to contribute if you have any better idea, just create a PR. If any question, feel free to open an issue.
 
+#### Ongoing project, welcome to join
 
-<table style="width:100%">
+<table style="width:100%" align="center">
   <tr>
     <th>
       <p align="center">
@@ -14,6 +15,8 @@ It's highly welcomed to contribute if you have better idea, just create a PR. If
       <p align="center">
            <a href="./docs/arima.md" name="introduction">intro</a>             
       </p>
+     </th>
+    <th>
        <p align="center">           
            <a href="./deepts/models/arima.py" name="code">code</a>     
       </p>
@@ -28,7 +31,9 @@ It's highly welcomed to contribute if you have better idea, just create a PR. If
     <th>
       <p align="center">
         <a href="./docs/tree.md" name="introduction">intro</a> 
-      </p>   
+      </p>
+    </th>
+    <th>   
       <p align="center">
            <a href="./deepts/models/tree.py" name="code">code</a>     
       </p>
@@ -44,6 +49,8 @@ It's highly welcomed to contribute if you have better idea, just create a PR. If
       <p align="center">
            <a href="./docs/rnn.md" name="introduction">intro</a>      
       </p>
+    </th>
+    <th>
       <p align="center">
            <a href="./deepts/models/seq2seq.py" name="code">code</a>     
       </p>
@@ -59,6 +66,8 @@ It's highly welcomed to contribute if you have better idea, just create a PR. If
       <p align="center">
            <a href="./docs/wavenet.md" name="introduction">intro</a>      
       </p>
+    </th>
+    <th>
       <p align="center">
            <a href="./deepts/models/wavenet.py" name="code">code</a>     
       </p>
@@ -73,7 +82,9 @@ It's highly welcomed to contribute if you have better idea, just create a PR. If
     <th>
       <p align="center">
            <a href="./docs/transformer.md" name="introduction">intro</a>              
-      </p>    
+      </p>   
+    </th>
+    <th> 
       <p align="center">
            <a href="./deepts/models/transformer.py" name="code">code</a>     
       </p>      
@@ -89,6 +100,8 @@ It's highly welcomed to contribute if you have better idea, just create a PR. If
       <p align="center">
            <a href="./docs/unet.md" name="introduction">intro</a>     
       </p>
+    </th>
+    <th>
       <p align="center">
            <a href="./deepts/models/unet.py" name="code">code</a>     
       </p>      
@@ -104,6 +117,8 @@ It's highly welcomed to contribute if you have better idea, just create a PR. If
       <p align="center">
             <a href="./docs/nbeats.md" name="introduction">intro</a>     
       </p>
+    </th>
+    <th>
       <p align="center">
            <a href="./deepts/models/nbeats.py" name="code">code</a>     
       </p>      
@@ -119,6 +134,8 @@ It's highly welcomed to contribute if you have better idea, just create a PR. If
       <p align="center">
            <a href="./docs/gan.md" name="introduction">intro</a>      
       </p>
+    </th>
+    <th>
       <p align="center">
            <a href="./deepts/models/gan.py" name="code">code</a>     
       </p>      
@@ -136,7 +153,7 @@ pip install -r requirements.txt
 ```bash
 bash ./data/download_passenger.sh
 ```
-3. Train the model, set `custom_model_params` if you want
+3. Train the model, set `custom_model_params` if you want, and pay attention to your own feature engineering
 ```bash
 cd examples
 python run_train.py --use_model seq2seq
@@ -147,7 +164,9 @@ python run_test.py
 ```
 
 ## Further reading
-https://github.com/awslabs/gluon-ts/
+- https://github.com/awslabs/gluon-ts/
+- https://github.com/Azure/DeepLearningForTimeSeriesForecasting
 
 ## Contributor
 - [LongxingTan](https://longxingtan.github.io/)
+

deepts/layers/__init__.py

@@ -0,0 +1 @@
+#encodeing=utf-8

deepts/layers/attention_layer.py

@@ -215,7 +215,9 @@ def build(self,input_shape):
         super(PositionEncoding,self).build(input_shape)
 
     def get_config(self):
-        pass
+        return {
+            'max_len': self.max_len
+        }
 
     def call(self,x,masking=True):
         E = x.get_shape().as_list()[-1]  # static

deepts/layers/rnn_layer.py

@@ -9,4 +9,4 @@
 
 class RNNLayer(Layer):
     def __init__(self):
-        super(RNNLayer,self).__init__()
+        super(RNNLayer,self).__init__()

deepts/model.py

@@ -11,7 +11,7 @@
 from deepts.models.unet import Unet
 from deepts.models.nbeats import NBeatsNet
 from deepts.models.gan import GAN
-assert tf.__version__>"2.0.0"
+assert tf.__version__>"2.0.0", "Should you consider to use TensorFlow 2?"
 
 
 class Loss(object):
@@ -29,11 +29,11 @@ class Optimizer(object):
     def __init__(self,use_optimizer):
         self.use_optimizer=use_optimizer
 
-    def __call__(self,):
+    def __call__(self,learning_rate):
         if self.use_optimizer == 'adam':
-            return tf.keras.optimizers.Adam(lr=0.001)
+            return tf.keras.optimizers.Adam(lr=learning_rate)
         elif self.use_optimizer == 'sgd':
-            return tf.keras.optimizers.SGD(lr=0.001)
+            return tf.keras.optimizers.SGD(lr=learning_rate)
 
 
 class Model(object):
@@ -66,7 +66,7 @@ def __init__(self,params, use_model, use_loss='mse',use_optimizer='adam', custom
         self.use_loss = use_loss
         self.use_optimizer = use_optimizer
         self.loss_fn = Loss(use_loss)()
-        self.optimizer_fn = Optimizer(use_optimizer)()
+        self.optimizer_fn = Optimizer(use_optimizer)(learning_rate=params['learning_rate'])
         self.model = tf.keras.Model(inputs, outputs, name=use_model)
 
     def train(self, dataset, n_epochs, mode='eager', export_model=False):
@@ -105,25 +105,41 @@ def train_step(self, x, y):
 
     def eval(self, valid_dataset):
         for step,(x,y) in enumerate(valid_dataset.take(-1)):
-            metrics=self.test_step(x,y)
+            metrics=self.dev_step(x,y)
             print("=> STEP %4d Metrics: %4.2f"%(step, metrics))
 
-    def test_step(self, x, y):
+    def dev_step(self, x, y):
+        '''
+        evaluation step function
+        :param x:
+        :param y:
+        :return:
+        '''
         x=tf.cast(x, tf.float32)
         y=tf.cast(y, tf.float32)
-        y_pred=self.model(x,training=False)
+        try:
+            y_pred=self.model(x,training=False)
+        except:
+            y_pred=self.model((x,tf.ones([tf.shape(x)[0],self.params['output_seq_length'],1],tf.float32)))
         metrics=self.loss_fn(y, y_pred).numpy()
         return metrics
 
     def predict(self, x_test, model_dir, use_model='pb'):
+        '''
+        predict function, don't use self.model here, but saved checkpoint or pb
+        :param x_test:
+        :param model_dir:
+        :param use_model:
+        :return:
+        '''
         if use_model=='pb':
             print('Load saved pb model ...')
             model=tf.saved_model.load(model_dir)
         else:
             print('Load checkpoint model ...')
             model=self.model.load_weights(model_dir)
 
-        y_pred=model(tf.constant(x_test),True,None)  # To be clarified
+        y_pred=model(x_test,True,None)  # To be clarified, not sure why additional args are necessary here
         return y_pred
 
     def export_model(self):

deepts/models/tft.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 # @author: Longxing Tan, [email protected]
-# @date: 2020-05
+# @date: 2020-06
 # paper: https://arxiv.org/pdf/1912.09363v1.pdf
 # other implementations: https://github.com/google-research/google-research/blob/master/tft/libs/tft_model.py
 

deepts/models/transformer.py

@@ -3,6 +3,7 @@
 # @date: 2020-01
 # paper:
 # other implementations: https://github.com/maxjcohen/transformer
+#                        https://github.com/Trigram19/m5-python-starter
 
 
 import tensorflow as tf

deepts/models/wavenet.py

@@ -9,7 +9,6 @@
 import tensorflow as tf
 from tensorflow.keras.layers import Dense
 from deepts.layers.wavenet_layer import Dense3D, ConvTime
-#tf.config.experimental_run_functions_eagerly(True)  # ??
 
 
 params={

docs/arima.md

@@ -2,7 +2,7 @@
 
 
 ## Introduction
-ARIMA is a short for "Autoregressive Integrated Moving Average model", it's a traditional time-series-prediction model. Basically it's a linear model combined the auto regression model and moving average model.
+ARIMA is a short for "Auto-regressive Integrated Moving Average model", it's a traditional time-series-prediction model. Basically it's a linear model combined the auto regression model and moving average model.
 - Auto regression model is a linear regression model using the history data as its feature. The important hyper parameter is how many days from history are used.
 - Moving average model is a linear regression model using the history residual error as its feature. The important hyper parameter is also history data length.
 - arima

docs/feature.md

@@ -0,0 +1,17 @@
+# Feature in Time series
+
+## Introduction
+I learned a lot from the data competition especially Kaggle about feature engineering in time series. They are so good and creative at it.
+
+
+## Auto-regression
+
+## Statistical feature
+
+## Categorical feature
+
+## Embedding feature
+- https://zhuanlan.zhihu.com/p/144030067
+- https://zhuanlan.zhihu.com/p/142681935
+
+

docs/gan.md

@@ -1,7 +1,7 @@
 # GAN
 
 ## Introduction
-
+Generative Adversarial Network(GAN)
 
 ## Performance
 

docs/rnn.md

@@ -1,8 +1,10 @@
 # RNN
 
 ## Introduction
+RNN and its modification LSTM, GRU are good at modelling the sequence data. So it's natural to use them in time series application
 
 
 ## Performance
 
+
 ## Further reading

docs/smoothing.md

@@ -0,0 +1,13 @@
+# Smoothing in Time series
+
+## Introduction
+Smoothing can help reduce some noise.
+
+## Moving average
+
+## Exponential smoothing
+
+## Filter
+
+
+## Auto-encoder

docs/transformer.md

@@ -1,9 +1,10 @@
 # Transformer
 
 ## Introduction
-Transformer is introduced in [Attention is all your need]()
+Transformer is introduced in [Attention is all your need](https://arxiv.org/abs/1706.03762), it has become the most popular NLP model.
 
 ## Performance
 
+
 ## Further reading
 

docs/tree.md

@@ -1,11 +1,18 @@
 # GBDT
 
 ## Introduction of GBDT
+Gradient boosting decision tree is also the promising solutions for time series issues.
 
 ## Introduction of XGBoost
+XGBoost is introduced in [XGBoost: A Scalable Tree Boosting System](https://arxiv.org/abs/1603.02754)
 
 ## Introduction of LightGBM
+LightGBM is introduces in [LightGBM: A Highly Efficient Gradient Boosting Decision Tree](http://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision-tree.pdf)
 
 ## Performance
+GBDT could also be tuned into SOTA model in time series. I read some implementations of the competition to let me so sure that I'm not so good at tuning the parameters.
+So I believe the performance here could be further optimized.
 
-## Further reading
+## Further reading
+- https://www.kaggle.com/pureheart/1st-place-lgb-model-public-0-470-private-0-502
+- https://www.kaggle.com/plantsgo/solution-public-0-471-private-0-505

docs/unet.md

@@ -2,7 +2,10 @@
 
 ## Introduction
 U-Net is introduced in [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://arxiv.org/pdf/1505.04597)
-used for image segmentation.
+used for image segmentation, almost the most popular model for image segmentation.
+
+We can tune the model a little for time series prediction.
+
 
 ## Performance
 

docs/validation.md

@@ -0,0 +1,15 @@
+# Validation in Time series
+
+Validation is difficult in time series.
+
+Basically, the data can be split by time, or by examples.
+
+
+## Examples in Kaggle
+
+There are some popular time-series-prediction competitions in Kaggle and M-series.
+
+
+
+
+

docs/wavenet.md

@@ -1,21 +1,21 @@
 # Wavenet
 
 ## Introduction
-Wavenet is introduced in [WaveNet: A Generative Model for Raw Audio](https://arxiv.org/abs/1609.03499) by DeepMind, first used for audio generation. The main components use the causal dilated convolutional neutral network. The kernel of CNN layer share the same weights, so it can also be used to percept the seasonality of time series issue.
+Wavenet is introduced in [WaveNet: A Generative Model for Raw Audio](https://arxiv.org/abs/1609.03499) by DeepMind, first used in audio generation. Its main components use the causal dilated convolutional neutral network. The kernel of CNN layer share the same weights, so it can also be used to percept the seasonality of time series issue.
 
 The dilated causal convolutional layer
 ![wavenet](https://github.com/LongxingTan/Time-series-prediction/blob/master/docs/assets/wavenet.gif)
 
+It's become popular in time series application since sjv open-sourced his repo [web-traffic-forecasting](https://github.com/sjvasquez/web-traffic-forecasting)
+
 ## Some detail
-### casual dilated convolutional neutral network
+#### casual dilated convolutional neutral network
 Casual: make sure that the future information won't leak
 
-Normal convolution
-
 Dilated: extend the receptive field to track the long term dependencies
 Implementation:
 
-
+Normal convolution:
 
 
 ## Performance