Merge release 1.0.0a21 (#155)

* Use SqlToolsService built on .NET Core 2.0 and a build script updates (#131)

* Bump version to 1.0.0a19

* Use .NET Core 2.0 RTM built sqltoolsservice

* Add build script to upload to azure blob storage

* Upgrade to VS 2017

* Remove 3.3 as supported Python version

* Fix perf issue where main event loop takes 100% of CPU (#132)

Fix perf issue where main event loop takes 100% of CPU

We have a 2 threads:
Thread #1 runs in a loop polling the response queue
Thread #2 runs in a loop decoding responses from the sqltoolsservice over stdout and posting them to the response queue

Since thread #1 doesn't sleep, it's takes 100% CPU. In addition, running python 2.7 on windows, #2 doesn’t preempt the CPU due to #1 taking all of the CPU cycles, so no response is processed.

Fix is simple – thread #1 needs to sleep so thread #2 can get scheduled and get it’s work done.

* Refine event loop perf fix in main.py

Refine event loop perf fix in main.py

* Fixing regular expression

Previous regex would result in release:a1 and release_version: 12.
Modified the regex for part Release to only pick up lower case letters.

* Adding missing forward slash on test pypi url

* fixing typos/grammar (#138)

fixing typos/grammar.

* Updating to release version 1.0.0a20.

* Create doc for official msft docs page

* Updated documentation page with usage_guide

* Added link to download adventureworks

* Updated with sqlcmd usage

* Added in run and cloud shell support

* Update/consolidate linux install (#153)

* universal linux wheel gen and setup update.

* Updating version cfg.

* Updating sqltoolsservice container.

* Updating spacing for flake8.

* Updating team email. (#154)

Author: MrMeemus

.bumpversion.cfg

@@ -1,5 +1,6 @@
 [bumpversion]
-current_version = 1.0.0a20
+current_version = 1.0.0a21
+
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)((?P<release>[a-z]+))(?P<release_version>\d+)
 serialize = 
 	{major}.{minor}.{patch}{release}{release_version}

doc/documentation_page.md

@@ -0,0 +1,100 @@
+# Get Started with mssql-scripter
+
+mssql-scripter is the command line equivalent of the widely used Generate Scripts Wizard experience in SSMS.
+
+You can use mssql-scripter on Linux, macOS, Windows, and the Azure Cloud Shell to generate data definition language (DDL) and data manipulation language (DML) T-SQL scripts for database objects in SQL Server running anywhere, Azure SQL Database, and Azure SQL Data Warehouse. You can save the generated T-SQL script to a .sql file or pipe it to standard *nix utilities (for example, sed, awk, grep) for further transformations. You can edit the generated script or check it into source control and subsequently execute the script in your existing SQL database deployment processes and DevOps pipelines with standard multiplatform SQL command line tools such as sqlcmd.
+
+## Install
+
+For information about installation, please see the LINK/install guide/LINK.
+
+The examples in this guide use the Adventureworks sample database. You can download the sample [here](https://www.microsoft.com/en-us/download/details.aspx?id=49502).
+
+## Generate scripts
+Use mssql-scripter to generate scripts for database schema and/or data by including specific objects, targeting server versions/editions,  and piping the script to files.
+
+### Dump database object schema
+
+    # generate DDL scripts for all objects in the Adventureworks database and save the script to a file
+    mssql-scripter -S localhost -d AdventureWorks -U sa
+    
+    # alternatively, specify the schema only flag to generate DDL scripts for all objects in the Adventureworks database and save the script to a file
+    mssql-scripter -S localhost -d AdventureWorks -U sa -f ./adventureworks.sql
+
+### Dump database object data
+
+    # generate DDL scripts for all objects in the Adventureworks database and save the script to stdout.
+    mssql-scripter -S localhost -d AdventureWorks -U sa --data-only
+
+### Dump the database object schema and data
+
+    # script the database schema and data piped to a file.
+    mssql-scripter -S localhost -d AdventureWorks -U sa --schema-and-data  > ./adventureworks.sql
+
+    # execute the generated above script with sqlcmd
+    sqlcmd -S mytestserver -U sa -i ./adventureworks.sql
+    
+### Include database objects
+
+    # generate DDL scripts for objects that contain 'Employee' in their name to stdout
+    mssql-scripter -S localhost -d AdventureWorks -U sa --include-objects Employee
+
+    # generate DDL scripts for the dbo schema and pipe the output to a file
+    mssql-scripter -S localhost -d AdventureWorks -U sa --include-objects dbo. > ./dboschema.sql
+
+### Exclude database objects
+   
+    # generate DDL scripts for objects that do not contain 'Sale' in their name to stdout
+    mssql-scripter -S localhost -d AdventureWorks -U sa --exclude-objects Sale
+
+### Target server version
+    
+    # specify the version of SQL Server the script will be run against
+    mssql-scripter -S myServer -d AdventureWorks -U myUser –-target-server-version "AzureDB" > myData.sql
+
+### Target server edition
+
+    # specify the edition of SQL Server the script will be run against
+    mssql-scripter -S localhost -d AdventureWorks -U myUser –-target-server-edition "Enterprise" > myData.sql
+
+### Pipe a generated script to sed
+Note this example is for Linux and macOS usage.
+
+    # change a schema name in the generated DDL script
+    # 1) generate DDL scripts for all objects in the Adventureworks database
+    # 2) pipe generated script to sed and change all occurrences of SalesLT to SalesLT_test and save the script to a file
+    mssql-scripter -S localhost -d Adventureworks -U sa | sed -e "s/SalesLT./SalesLT_test./g" > adventureworks_SalesLT_test.sql 
+
+### Script data to a file
+   
+    # script all the data to a file.
+    mssql-scripter -S localhost -d AdventureWorks -U sa --data-only > ./adventureworks-data.sql 
+
+### Set environment variables
+You can set environment variables for your connection string through the following steps:
+
+
+    # set environment variable MSSQL_SCRIPTER_CONNECTION_STRING with a connection string.
+    export MSSQL_SCRIPTER_CONNECTION_STRING='Server=myserver;Database=mydb;User Id=myuser;Password=mypassword;'
+    mssql-scripter
+
+    # set environment variable MSSQL_SCRIPTER_PASSWORD so no password input is required.
+    export MSSQL_SCRIPTER_PASSWORD='ABC123'
+    mssql-scripter -S localhost -d AdventureWorks -U sa
+    
+ ## Generate and run scripts
+ In this example you will generate a script, send it to a file, and execute the script.
+ 
+ Generate a script and send it to a file using mssql-scripter.
+    # script all the data to a file.
+    mssql-scripter -S localhost -d AdventureWorks -U sa --data-only > ./adventureworks-data.sql
+    
+ Now that you have generated a script for your database objects, you can execute the script using sqlcmd such as in the example below.
+    
+    # execute the script from the file.
+    sqlcmd -S localhost -d AdventureWorks -U sa -i`./adventureworks-data.sql
+ 
+ You can find more details on using sqlcmd [here](https://docs.microsoft.com/en-us/sql/relational-databases/scripting/sqlcmd-use-the-utility).
+
+## Use mssql-scripter in the Cloud Shell
+You can use mssql-scripter in the Azure Cloud Shell to generate scripter for Azure SQL DB, Azure SQL DW, and SQL Server instances in Azure VMs. [Connect to the Azure Cloud Shell](https://docs.microsoft.com/en-us/azure/cloud-shell/overview?view=azure-cli-latest). Once connected, you can use mssql-scripter in the terminal as you would in a local terminal.

mssqlscripter/__init__.py

@@ -3,4 +3,6 @@
 # Licensed under the MIT License. See License.txt in the project root for license information.
 # --------------------------------------------------------------------------------------------
 
-__version__ = '1.0.0a20'
+
+__version__ = '1.0.0a21'
+

mssqltoolsservice/buildwheels.py

@@ -17,21 +17,16 @@
 from urllib.request import urlopen
 
 
-DOWNLOAD_URL_BASE = 'https://mssqlscripter.blob.core.windows.net/sqltoolsservice-08-16-2017/'
+
+DOWNLOAD_URL_BASE = 'https://mssqlscripter.blob.core.windows.net/sqltoolsservice-10-12-2017/'
 
 # Supported platform key's must match those in mssqlscript's setup.py.
 SUPPORTED_PLATFORMS = {
-    'CentOS_7': DOWNLOAD_URL_BASE + 'microsoft.sqltools.servicelayer-centos-x64-netcoreapp2.0.tar.gz',
-    'Debian_8': DOWNLOAD_URL_BASE + 'microsoft.sqltools.servicelayer-debian-x64-netcoreapp2.0.tar.gz',
-    'Fedora_23': DOWNLOAD_URL_BASE + 'microsoft.sqltools.servicelayer-fedora-x64-netcoreapp2.0.tar.gz',
-    'openSUSE_13_2': DOWNLOAD_URL_BASE + 'microsoft.sqltools.servicelayer-opensuse-x64-netcoreapp2.0.tar.gz',
-    'OSX_10_11_64': DOWNLOAD_URL_BASE + 'microsoft.sqltools.servicelayer-osx-x64-netcoreapp2.0.tar.gz',
-    'RHEL_7': DOWNLOAD_URL_BASE + 'microsoft.sqltools.servicelayer-rhel-x64-netcoreapp2.0.tar.gz',
-    'Ubuntu_14': DOWNLOAD_URL_BASE + 'microsoft.sqltools.servicelayer-ubuntu14-x64-netcoreapp2.0.tar.gz',
-    'Ubuntu_16': DOWNLOAD_URL_BASE + 'microsoft.sqltools.servicelayer-ubuntu16-x64-netcoreapp2.0.tar.gz',
-    'Windows_7_64': DOWNLOAD_URL_BASE + 'microsoft.sqltools.servicelayer-win-x64-netcoreapp2.0.zip',
-    'Windows_7_86': DOWNLOAD_URL_BASE + 'microsoft.sqltools.servicelayer-win-x86-netcoreapp2.0.zip'
-}
+    'Linux_64': DOWNLOAD_URL_BASE + 'Microsoft.SqlTools.ServiceLayer-linux-x64-netcoreapp2.0.tar.gz',
+    'OSX_10_11_64': DOWNLOAD_URL_BASE + 'Microsoft.SqlTools.ServiceLayer-osx-x64-netcoreapp2.0.tar.gz',
+    'Windows_7_64': DOWNLOAD_URL_BASE + 'Microsoft.SqlTools.ServiceLayer-win-x64-netcoreapp2.0.zip',
+    'Windows_7_86': DOWNLOAD_URL_BASE + 'Microsoft.SqlTools.ServiceLayer-win-x86-netcoreapp2.0.zip'
+
 
 CURRENT_DIRECTORY = os.path.abspath(os.path.join(os.path.abspath(__file__), '..'))
 BUILD_DIRECTORY = os.path.abspath(os.path.join(CURRENT_DIRECTORY, 'build'))
@@ -78,6 +73,7 @@ def build_sqltoolsservice_wheels(platforms):
         os.environ[u'MSSQLTOOLSSERVICE_PLATFORM'] = platform
 
         print(u'Calling setup bdist_wheel for platform:{}'.format(platform))
+        print(SUPPORTED_PLATFORMS[platform])
         download_and_unzip(SUPPORTED_PLATFORMS[platform], directory=TARGET_DIRECTORY)
         utility.exec_command(u'python setup.py check -r -s bdist_wheel', CURRENT_DIRECTORY)
 

mssqltoolsservice/mssqltoolsservice/__init__.py

@@ -9,7 +9,9 @@
 import os
 import platform
 
-__version__ = '1.0.0a20'
+
+__version__ = '1.0.0a21'
+
 
 
 def get_executable_path():

mssqltoolsservice/setup.py

@@ -12,7 +12,9 @@
 
 # This version number is in place in two places and must be in sync with
 # mssqlscripter's version in setup.py.
-MSSQLTOOLSSERVICE_VERSION = '1.0.0a20'
+
+MSSQLTOOLSSERVICE_VERSION = '1.0.0a21'
+
 
 # If we have source, validate version numbers match to prevent
 # uploading releases with mismatched versions.

setup.py

@@ -14,7 +14,9 @@
 
 # This version number is in place in two places and must be in sync with
 # mssqltoolsservice's version in setup.py.
-MSSQLSCRIPTER_VERSION = '1.0.0a20'
+
+MSSQLSCRIPTER_VERSION = '1.0.0a21'
+
 
 # If we have the source, validate our setup version matches source version.
 # This will prevent uploading releases with mismatched versions. This will
@@ -58,119 +60,12 @@
 
 MSSQLTOOLSSERVICE_PACKAGE_NAME = 'mssqltoolsservice-{}=={}'
 MSSQLTOOLSSERVICE_PACKAGE_SUFFIX = [
-    'CentOS_7',
-    'Debian_8',
-    'Fedora_23',
-    'openSUSE_13_2',
     'OSX_10_11_64',
-    'RHEL_7',
-    'Ubuntu_14',
-    'Ubuntu_16',
     'Windows_7_64',
-    'Windows_7_86'
+    'Windows_7_86',
+    'Linux_64'
 ]
 
-LINUX_DISTRO_NO_VERSION = {
-    'centos': 'CentOS_7',
-    'ol': 'CentOS_7',
-    'fedora': 'Fedora_23',
-    'opensuse': 'OpenSUSE_13_2',
-    'rhel': 'RHEL_7',
-    'debian': 'Debian_8',
-}
-
-LINUX_DISTRO_WITH_VERSION = {
-    'ubuntu':
-        {
-            '14': 'Ubuntu_14',
-            '16': 'Ubuntu_16'
-        },
-    'elementary':
-        {
-            '0.3': 'Ubuntu_14',
-            '0.4': 'Ubuntu_16'
-        },
-    'elementaryOS':
-        {
-            '0.3': 'Ubuntu_14',
-            '0.4': 'Ubuntu_16'
-        },
-    'linuxmint':
-        {
-            '18': 'Ubuntu_16'
-        },
-    'galliumos':
-        {
-            '2.0': 'Ubuntu_16'
-        },
-}
-
-
-def _get_runtime_id_helper(name, version):
-    """
-        Checks if linux distro name and version match to a supported package.
-    """
-    if name in LINUX_DISTRO_NO_VERSION:
-        return LINUX_DISTRO_NO_VERSION[name]
-
-    if name in LINUX_DISTRO_WITH_VERSION:
-        for supported_version in LINUX_DISTRO_WITH_VERSION[name]:
-            if version.startswith(supported_version):
-                return LINUX_DISTRO_WITH_VERSION[name][supported_version]
-    return None
-
-
-def _get_linux_distro_runtime_id(content):
-    """
-        Parse content for linux distro run time id.
-     """
-    name = None
-    version = None
-    id_like = None
-
-    # Try to find name, version and id_like best effort.
-    for line in content.splitlines():
-        key, value = line.rstrip().split('=')
-        value = value.strip('"')
-        if key == 'ID':
-            name = value
-        elif key == 'VERSION_ID':
-            version = value
-        elif key == 'ID_LIKE':
-            id_like = value.split(' ')
-        if name and version and id_like:
-            break
-    # First try the distribution name.
-    run_time_id = _get_runtime_id_helper(name, version)
-
-    # If we don't understand it, try the 'ID_LIKE' field.
-    if not run_time_id and id_like:
-        for name in id_like:
-            run_time_id = _get_runtime_id_helper(name, version)
-            if run_time_id:
-                break
-
-    return run_time_id
-
-
-def _get_linux_distro_from_file():
-    """
-        Find linux distro based on
-        https://www.freedesktop.org/software/systemd/man/os-release.html.
-    """
-    os_release_info_file = None
-
-    if os.path.exists('/etc/os-release'):
-        os_release_info_file = '/etc/os-release'
-    elif os.path.exists('/usr/lib/os-release'):
-        os_release_info_file = '/usr/lib/os-release'
-    else:
-        raise EnvironmentError('Error detecting Linux distro version.')
-
-    with io.open(os_release_info_file, 'r', encoding='utf-8') as os_release_file:
-        content = os_release_file.read()
-        return _get_linux_distro_runtime_id(content)
-
 
 def _get_runtime_id(
         system=_platform.system(),
@@ -191,7 +86,7 @@ def _get_runtime_id(
             run_time_id = 'OSX_10_11_64'
     elif system == 'Linux':
         if architecture == '64bit':
-            run_time_id = _get_linux_distro_from_file()
+            run_time_id = 'Linux_64'
 
     return run_time_id
 
@@ -244,7 +139,7 @@ def get_mssqltoolsservice_package_name(run_time_id=_get_runtime_id()):
     description='Microsoft SQL Scripter Command-Line Tool',
     license='MIT',
     author='Microsoft Corporation',
-    author_email='sqlxplatclieng@microsoft.com',
+    author_email='sqlcli@microsoft.com',
     url='https://github.com/Microsoft/sql-xplat-cli/',
     zip_safe=True,
     long_description=open('README.rst').read(),