Feature/week3

.pre-commit-config.yaml

@@ -1,3 +1,6 @@
+default_stages:
+- pre-commit
+
 exclude: ^tests/resources/
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks

README.md

@@ -121,6 +121,7 @@ week 2
 - In example code, the features generated at runtime were not used in the fe model
 > **Workaround**
 - Ran the feature engineering notebook from Databricks workspace, this resolved permissions issues
+    - Feature Engineering package limitations [documentation](https://docs.databricks.com/en/machine-learning/feature-store/python-api.html#limitations)
 - Ran the feature engineering feature function on the training and testing set and included the new features in the fe model
 -   ```python
     testing_set = fe.create_training_set(

configs/project_configs.yml

@@ -6,6 +6,13 @@ hyperparameters:
   n_estimators: 1000
   max_depth: 6
 
+ab_test_hyperparameters:
+  learning_rate_a: 0.02
+  learning_rate_b: 0.02
+  n_estimators: 1000
+  max_depth_a: 6
+  max_depth_b: 10
+
 processed_features:
   num_features:
     - Temperature

notebooks/week2/01_prepare_dataset.py

@@ -27,6 +27,13 @@
 # COMMAND ----------
 train_set.reset_index(inplace=True)
 test_set.reset_index(inplace=True)
+# Convert DateTime to Unix timestamp (milliseconds) and create ID column
+train_set['id'] = train_set['DateTime'].astype('int64') // 10**9
+test_set['id'] = test_set['DateTime'].astype('int64') // 10**9
+
+# Convert to string
+train_set['id'] = train_set['id'].astype(str)
+test_set['id'] = test_set['id'].astype(str)
 # COMMAND ----------
 data_processor.save_to_catalog(train_set=train_set, test_set=test_set, spark=spark)
 # COMMAND ----------