update samples from Release-169 as a part of 1.0.85 SDK release (Azure#742)

Co-authored-by: vizhur <[email protected]>

Author: harneetvirk

README.md

@@ -20,8 +20,8 @@ If you want to...
  * ...try out and explore Azure ML, start with image classification tutorials: [Part 1 (Training)](./tutorials/img-classification-part1-training.ipynb) and [Part 2 (Deployment)](./tutorials/img-classification-part2-deploy.ipynb).
  * ...learn about experimentation and tracking run history, first [train within Notebook](./how-to-use-azureml/training/train-within-notebook/train-within-notebook.ipynb), then try [training on remote VM](./how-to-use-azureml/training/train-on-remote-vm/train-on-remote-vm.ipynb) and [using logging APIs](./how-to-use-azureml/training/logging-api/logging-api.ipynb).
  * ...train deep learning models at scale, first learn about [Machine Learning Compute](./how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.ipynb), and then try [distributed hyperparameter tuning](./how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-pytorch/train-hyperparameter-tune-deploy-with-pytorch.ipynb) and [distributed training](./how-to-use-azureml/training-with-deep-learning/distributed-pytorch-with-horovod/distributed-pytorch-with-horovod.ipynb).
- * ...deploy models as a realtime scoring service, first learn the basics by [training within Notebook and deploying to Azure Container Instance](./how-to-use-azureml/training/train-within-notebook/train-within-notebook.ipynb), then learn how to [register and manage models, and create Docker images](./how-to-use-azureml/deployment/register-model-create-image-deploy-service/register-model-create-image-deploy-service.ipynb), and [production deploy models on Azure Kubernetes Cluster](./how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks.ipynb).
- * ...deploy models as a batch scoring service, first [train a model within Notebook](./how-to-use-azureml/training/train-within-notebook/train-within-notebook.ipynb), learn how to [register and manage models](./how-to-use-azureml/deployment/register-model-create-image-deploy-service/register-model-create-image-deploy-service.ipynb), then [create Machine Learning Compute for scoring compute](./how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.ipynb), and [use Machine Learning Pipelines to deploy your model](https://aka.ms/pl-batch-scoring).
+ * ...deploy models as a realtime scoring service, first learn the basics by [training within Notebook and deploying to Azure Container Instance](./how-to-use-azureml/training/train-within-notebook/train-within-notebook.ipynb), then learn how to [production deploy models on Azure Kubernetes Cluster](./how-to-use-azureml/deployment/production-deploy-to-aks/production-deploy-to-aks.ipynb).
+ * ...deploy models as a batch scoring service, first [train a model within Notebook](./how-to-use-azureml/training/train-within-notebook/train-within-notebook.ipynb), then [create Machine Learning Compute for scoring compute](./how-to-use-azureml/training/train-on-amlcompute/train-on-amlcompute.ipynb), and [use Machine Learning Pipelines to deploy your model](https://aka.ms/pl-batch-scoring).
  * ...monitor your deployed models, learn about using [App Insights](./how-to-use-azureml/deployment/enable-app-insights-in-production-service/enable-app-insights-in-production-service.ipynb).
 
 ## Tutorials

configuration.ipynb

@@ -103,7 +103,7 @@
       "source": [
         "import azureml.core\n",
         "\n",
-        "print(\"This notebook was created using version 1.0.83 of the Azure ML SDK\")\n",
+        "print(\"This notebook was created using version 1.0.85 of the Azure ML SDK\")\n",
         "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
       ]
     },

how-to-use-azureml/README.md

@@ -9,7 +9,6 @@ As a pre-requisite, run the [configuration Notebook](../configuration.ipynb) not
 * [train-on-amlcompute](./training/train-on-amlcompute): Use a 1-n node Azure ML managed compute cluster for remote runs on Azure CPU or GPU infrastructure.
 * [train-on-remote-vm](./training/train-on-remote-vm): Use Data Science Virtual Machine as a target for remote runs.
 * [logging-api](./track-and-monitor-experiments/logging-api): Learn about the details of logging metrics to run history.
-* [register-model-create-image-deploy-service](./deployment/register-model-create-image-deploy-service): Learn about the details of model management.
 * [production-deploy-to-aks](./deployment/production-deploy-to-aks) Deploy a model to production at scale on Azure Kubernetes Service.
 * [enable-app-insights-in-production-service](./deployment/enable-app-insights-in-production-service) Learn how to use App Insights with production web service.
 

how-to-use-azureml/automated-machine-learning/README.md

@@ -197,6 +197,17 @@ If automl_setup_linux.sh fails on Ubuntu Linux with the error: `unable to execut
 4) Check that the region is one of the supported regions: `eastus2`, `eastus`, `westcentralus`, `southeastasia`, `westeurope`, `australiaeast`, `westus2`, `southcentralus`
 5) Check that you have access to the region using the Azure Portal.
 
+## import AutoMLConfig fails after upgrade from before 1.0.76 to 1.0.76 or later
+There were package changes in automated machine learning version 1.0.76, which require the previous version to be uninstalled before upgrading to the new version.
+If you have manually upgraded from a version of automated machine learning before 1.0.76 to 1.0.76 or later, you may get the error:
+`ImportError: cannot import name 'AutoMLConfig'`
+
+This can be resolved by running:
+`pip uninstall azureml-train-automl` and then 
+`pip install azureml-train-automl`
+
+The automl_setup.cmd script does this automatically.
+
 ## workspace.from_config fails
 If the call `ws = Workspace.from_config()` fails:
 1) Make sure that you have run the `configuration.ipynb` notebook successfully.

how-to-use-azureml/automated-machine-learning/classification-bank-marketing-all-features/auto-ml-classification-bank-marketing-all-features.ipynb

@@ -92,6 +92,32 @@
         "from azureml.explain.model._internal.explanation_client import ExplanationClient"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Accessing the Azure ML workspace requires authentication with Azure.\n",
+        "\n",
+        "The default authentication is interactive authentication using the default tenant.  Executing the `ws = Workspace.from_config()` line in the cell below will prompt for authentication the first time that it is run.\n",
+        "\n",
+        "If you have multiple Azure tenants, you can specify the tenant by replacing the `ws = Workspace.from_config()` line in the cell below with the following:\n",
+        "\n",
+        "```\n",
+        "from azureml.core.authentication import InteractiveLoginAuthentication\n",
+        "auth = InteractiveLoginAuthentication(tenant_id = 'mytenantid')\n",
+        "ws = Workspace.from_config(auth = auth)\n",
+        "```\n",
+        "\n",
+        "If you need to run in an environment where interactive login is not possible, you can use Service Principal authentication by replacing the `ws = Workspace.from_config()` line in the cell below with the following:\n",
+        "\n",
+        "```\n",
+        "from azureml.core.authentication import ServicePrincipalAuthentication\n",
+        "auth = auth = ServicePrincipalAuthentication('mytenantid', 'myappid', 'mypassword')\n",
+        "ws = Workspace.from_config(auth = auth)\n",
+        "```\n",
+        "For more details, see [aka.ms/aml-notebook-auth](http://aka.ms/aml-notebook-auth)"
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,

how-to-use-azureml/automated-machine-learning/classification-bank-marketing-all-features/auto-ml-classification-bank-marketing-all-features.yml

@@ -2,3 +2,10 @@ name: auto-ml-classification-bank-marketing-all-features
 dependencies:
 - pip:
   - azureml-sdk
+  - azureml-train-automl
+  - azureml-widgets
+  - matplotlib
+  - interpret
+  - onnxruntime==1.0.0
+  - azureml-explain-model
+  - azureml-contrib-interpret

how-to-use-azureml/automated-machine-learning/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.ipynb

@@ -210,7 +210,6 @@
         "automl_settings = {\n",
         "    \"n_cross_validations\": 3,\n",
         "    \"primary_metric\": 'average_precision_score_weighted',\n",
-        "    \"preprocess\": True,\n",
         "    \"enable_early_stopping\": True,\n",
         "    \"max_concurrent_iterations\": 2, # This is a limit for testing purpose, please increase it as per cluster size\n",
         "    \"experiment_timeout_hours\": 0.2, # This is a time limit for testing purposes, remove it for real use cases, this will drastically limit ablity to find the best model possible\n",

how-to-use-azureml/automated-machine-learning/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.yml

@@ -2,3 +2,8 @@ name: auto-ml-classification-credit-card-fraud
 dependencies:
 - pip:
   - azureml-sdk
+  - azureml-train-automl
+  - azureml-widgets
+  - matplotlib
+  - interpret
+  - azureml-explain-model

how-to-use-azureml/automated-machine-learning/classification-text-dnn/auto-ml-classification-text-dnn.ipynb

@@ -275,7 +275,6 @@
         "automl_settings = {\n",
         "    \"experiment_timeout_minutes\": 20,\n",
         "    \"primary_metric\": 'accuracy',\n",
-        "    \"preprocess\": True,\n",
         "    \"max_concurrent_iterations\": 4, \n",
         "    \"max_cores_per_iteration\": -1,\n",
         "    \"enable_dnn\": True,\n",

how-to-use-azureml/automated-machine-learning/classification-text-dnn/auto-ml-classification-text-dnn.yml

@@ -2,3 +2,7 @@ name: auto-ml-classification-text-dnn
 dependencies:
 - pip:
   - azureml-sdk
+  - azureml-train-automl
+  - azureml-widgets
+  - matplotlib
+  - azurmel-train

how-to-use-azureml/automated-machine-learning/continuous-retraining/auto-ml-continuous-retraining.ipynb

@@ -350,7 +350,6 @@
         "    \"experiment_timeout_hours\": 0.2,\n",
         "    \"n_cross_validations\": 3,\n",
         "    \"primary_metric\": 'r2_score',\n",
-        "    \"preprocess\": True,\n",
         "    \"max_concurrent_iterations\": 3,\n",
         "    \"max_cores_per_iteration\": -1,\n",
         "    \"verbosity\": logging.INFO,\n",

how-to-use-azureml/automated-machine-learning/continuous-retraining/auto-ml-continuous-retraining.yml

@@ -2,3 +2,7 @@ name: auto-ml-continuous-retraining
 dependencies:
 - pip:
   - azureml-sdk
+  - azureml-train-automl
+  - azureml-widgets
+  - matplotlib
+  - azureml-pipeline

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/auto-ml-forecasting-beer-remote.yml

@@ -1,4 +1,10 @@
 name: auto-ml-forecasting-beer-remote
 dependencies:
+- fbprophet==0.5
+- py-xgboost<=0.80
 - pip:
   - azureml-sdk
+  - azureml-train-automl
+  - azureml-widgets
+  - matplotlib
+  - azureml-train

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/helper.py

@@ -76,9 +76,12 @@ def get_result_df(remote_run):
 def run_inference(test_experiment, compute_target, script_folder, train_run,
                   test_dataset, lookback_dataset, max_horizon,
                   target_column_name, time_column_name, freq):
-    train_run.download_file('outputs/model.pkl', 'inference/model.pkl')
-    train_run.download_file('outputs/conda_env_v_1_0_0.yml',
-                            'inference/condafile.yml')
+    model_base_name = 'model.pkl'
+    if 'model_data_location' in train_run.properties:
+        model_location = train_run.properties['model_data_location']
+        _, model_base_name = model_location.rsplit('/', 1)
+    train_run.download_file('outputs/{}'.format(model_base_name), 'inference/{}'.format(model_base_name))
+    train_run.download_file('outputs/conda_env_v_1_0_0.yml', 'inference/condafile.yml')
 
     inference_env = Environment("myenv")
     inference_env.docker.enabled = True
@@ -91,7 +94,8 @@ def run_inference(test_experiment, compute_target, script_folder, train_run,
                         '--max_horizon': max_horizon,
                         '--target_column_name': target_column_name,
                         '--time_column_name': time_column_name,
-                        '--frequency': freq
+                        '--frequency': freq,
+                        '--model_path': model_base_name
                     },
                     inputs=[test_dataset.as_named_input('test_data'),
                             lookback_dataset.as_named_input('lookback_data')],

how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/infer.py

@@ -232,20 +232,25 @@ def MAPE(actual, pred):
 parser.add_argument(
     '--frequency', type=str, dest='freq',
     help='Frequency of prediction')
+parser.add_argument(
+    '--model_path', type=str, dest='model_path',
+    default='model.pkl', help='Filename of model to be loaded')
 
 
 args = parser.parse_args()
 max_horizon = args.max_horizon
 target_column_name = args.target_column_name
 time_column_name = args.time_column_name
 freq = args.freq
+model_path = args.model_path
 
 
 print('args passed are: ')
 print(max_horizon)
 print(target_column_name)
 print(time_column_name)
 print(freq)
+print(model_path)
 
 run = Run.get_context()
 # get input dataset by name
@@ -267,7 +272,8 @@ def MAPE(actual, pred):
 y_lookback_df = lookback_dataset.with_timestamp_columns(
     None).keep_columns(columns=[target_column_name])
 
-fitted_model = joblib.load('model.pkl')
+fitted_model = joblib.load(model_path)
+
 
 if hasattr(fitted_model, 'get_lookback'):
     lookback = fitted_model.get_lookback()

how-to-use-azureml/automated-machine-learning/forecasting-bike-share/auto-ml-forecasting-bike-share.yml

@@ -1,4 +1,9 @@
 name: auto-ml-forecasting-bike-share
 dependencies:
+- fbprophet==0.5
+- py-xgboost<=0.80
 - pip:
   - azureml-sdk
+  - azureml-train-automl
+  - azureml-widgets
+  - matplotlib

how-to-use-azureml/automated-machine-learning/forecasting-energy-demand/auto-ml-forecasting-energy-demand.ipynb

@@ -253,7 +253,7 @@
       "source": [
         "# split into train based on time\n",
         "train = dataset.time_before(datetime(2017, 8, 8, 5), include_boundary=True)\n",
-        "train.to_pandas_dataframe().sort_values(time_column_name).tail(5).reset_index(drop=True)"
+        "train.to_pandas_dataframe().reset_index(drop=True).sort_values(time_column_name).tail(5)"
       ]
     },
     {
@@ -264,7 +264,7 @@
       "source": [
         "# split into test based on time\n",
         "test = dataset.time_between(datetime(2017, 8, 8, 6), datetime(2017, 8, 10, 5))\n",
-        "test.to_pandas_dataframe().head(5).reset_index(drop=True)"
+        "test.to_pandas_dataframe().reset_index(drop=True).head(5)"
       ]
     },
     {

how-to-use-azureml/automated-machine-learning/forecasting-energy-demand/auto-ml-forecasting-energy-demand.yml

@@ -2,3 +2,9 @@ name: auto-ml-forecasting-energy-demand
 dependencies:
 - pip:
   - azureml-sdk
+  - azureml-train-automl
+  - azureml-widgets
+  - matplotlib
+  - interpret
+  - azureml-explain-model
+  - azureml-contrib-interpret

how-to-use-azureml/automated-machine-learning/forecasting-grouping/auto-ml-forecasting-grouping.yml

@@ -2,3 +2,7 @@ name: auto-ml-forecasting-grouping
 dependencies:
 - pip:
   - azureml-sdk
+  - azureml-train-automl
+  - azureml-widgets
+  - matplotlib
+  - azureml-pipeline

how-to-use-azureml/automated-machine-learning/forecasting-grouping/deploy/deploy.py

@@ -1,9 +1,9 @@
 import argparse
 import json
 
-from azureml.core import Run, Model, Workspace
-from azureml.core.conda_dependencies import CondaDependencies
+from azureml.core import Run, Model
 from azureml.core.model import InferenceConfig
+from azureml.core.environment import Environment
 from azureml.core.webservice import AciWebservice
 
 
@@ -39,18 +39,16 @@
 run = Run.get_context()
 ws = run.experiment.workspace
 
+myenv = Environment.from_conda_specification(name="env", file_path=conda_env_file_name)
+
 deployment_config = AciWebservice.deploy_configuration(
     cpu_cores=1,
     memory_gb=2,
     tags={"method": "grouping"},
     description='grouping demo aci deployment'
 )
 
-inference_config = InferenceConfig(
-    entry_script=script_file_name,
-    runtime='python',
-    conda_file=conda_env_file_name
-)
+inference_config = InferenceConfig(entry_script=script_file_name, environment=myenv)
 
 models = []
 for model_name in model_list:

how-to-use-azureml/automated-machine-learning/forecasting-high-frequency/automl-forecasting-function.yml

@@ -1,4 +1,9 @@
 name: automl-forecasting-function
 dependencies:
+- fbprophet==0.5
+- py-xgboost<=0.80
 - pip:
   - azureml-sdk
+  - azureml-train-automl
+  - azureml-widgets
+  - matplotlib

how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/auto-ml-forecasting-orange-juice-sales.ipynb

@@ -631,9 +631,7 @@
       "outputs": [],
       "source": [
         "import json\n",
-        "# The request data frame needs to have y_query column which corresponds to query.\n",
         "X_query = X_test.copy()\n",
-        "X_query['y_query'] = np.NaN\n",
         "# We have to convert datetime to string, because Timestamps cannot be serialized to JSON.\n",
         "X_query[time_column_name] = X_query[time_column_name].astype(str)\n",
         "# The Service object accept the complex dictionary, which is internally converted to JSON string.\n",

how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales/auto-ml-forecasting-orange-juice-sales.yml

@@ -1,4 +1,9 @@
 name: auto-ml-forecasting-orange-juice-sales
 dependencies:
+- fbprophet==0.5
+- py-xgboost<=0.80
 - pip:
   - azureml-sdk
+  - azureml-train-automl
+  - azureml-widgets
+  - matplotlib

how-to-use-azureml/automated-machine-learning/local-run-classification-credit-card-fraud/auto-ml-classification-credit-card-fraud-local.ipynb

@@ -155,7 +155,6 @@
         "automl_settings = {\n",
         "    \"n_cross_validations\": 3,\n",
         "    \"primary_metric\": 'average_precision_score_weighted',\n",
-        "    \"preprocess\": True,\n",
         "    \"experiment_timeout_hours\": 0.2, # This is a time limit for testing purposes, remove it for real use cases, this will drastically limit ability to find the best model possible\n",
         "    \"verbosity\": logging.INFO,\n",
         "    \"enable_stack_ensemble\": False\n",

how-to-use-azureml/automated-machine-learning/local-run-classification-credit-card-fraud/auto-ml-classification-credit-card-fraud-local.yml

@@ -2,3 +2,8 @@ name: auto-ml-classification-credit-card-fraud-local
 dependencies:
 - pip:
   - azureml-sdk
+  - azureml-train-automl
+  - azureml-widgets
+  - matplotlib
+  - interpret
+  - azureml-explain-model

how-to-use-azureml/automated-machine-learning/regression-hardware-performance-explanation-and-featurization/auto-ml-regression-hardware-performance-explanation-and-featurization.ipynb

@@ -208,7 +208,7 @@
         "|**primary_metric**|This is the metric that you want to optimize. Regression supports the following primary metrics: <br><i>spearman_correlation</i><br><i>normalized_root_mean_squared_error</i><br><i>r2_score</i><br><i>normalized_mean_absolute_error</i>|\n",
         "|**experiment_timeout_hours**| Maximum amount of time in hours that all iterations combined can take before the experiment terminates.|\n",
         "|**enable_early_stopping**| Flag to enble early termination if the score is not improving in the short term.|\n",
-        "|**featurization**| 'auto' / 'off' / FeaturizationConfig Indicator for whether featurization step should be done automatically or not, or whether customized featurization should be used. Note: If the input data is sparse, featurization cannot be turned on.|\n",
+        "|**featurization**| 'auto' / 'off' / FeaturizationConfig Indicator for whether featurization step should be done automatically or not, or whether customized featurization should be used. Setting this enables AutoML to perform featurization on the input to handle *missing data*, and to perform some common *feature extraction*. Note: If the input data is sparse, featurization cannot be turned on.|\n",
         "|**n_cross_validations**|Number of cross validation splits.|\n",
         "|**training_data**|(sparse) array-like, shape = [n_samples, n_features]|\n",
         "|**label_column_name**|(sparse) array-like, shape = [n_samples, ], targets values.|"
@@ -244,7 +244,7 @@
       "source": [
         "featurization_config = FeaturizationConfig()\n",
         "featurization_config.blocked_transformers = ['LabelEncoder']\n",
-        "#featurization_config.drop_columns = ['ERP', 'MMIN']\n",
+        "#featurization_config.drop_columns = ['MMIN']\n",
         "featurization_config.add_column_purpose('MYCT', 'Numeric')\n",
         "featurization_config.add_column_purpose('VendorName', 'CategoricalHash')\n",
         "#default strategy mean, add transformer param for for 3 columns\n",

how-to-use-azureml/automated-machine-learning/regression-hardware-performance-explanation-and-featurization/auto-ml-regression-hardware-performance-explanation-and-featurization.yml

@@ -2,3 +2,10 @@ name: auto-ml-regression-hardware-performance-explanation-and-featurization
 dependencies:
 - pip:
   - azureml-sdk
+  - azureml-train-automl
+  - azureml-widgets
+  - matplotlib
+  - interpret
+  - azureml-explain-model
+  - azureml-explain-model
+  - azureml-contrib-interpret

how-to-use-azureml/automated-machine-learning/regression/auto-ml-regression.ipynb

@@ -198,7 +198,6 @@
         "automl_settings = {\n",
         "    \"n_cross_validations\": 3,\n",
         "    \"primary_metric\": 'r2_score',\n",
-        "    \"preprocess\": True,\n",
         "    \"enable_early_stopping\": True, \n",
         "    \"experiment_timeout_hours\": 0.3, #for real scenarios we reccommend a timeout of at least one hour \n",
         "    \"max_concurrent_iterations\": 4,\n",

how-to-use-azureml/automated-machine-learning/regression/auto-ml-regression.yml

@@ -2,3 +2,6 @@ name: auto-ml-regression
 dependencies:
 - pip:
   - azureml-sdk
+  - azureml-train-automl
+  - azureml-widgets
+  - matplotlib