Skip to content
This repository was archived by the owner on May 7, 2024. It is now read-only.

Commit be55629

Browse files
authored
Merging release 1.0.0a22 (#164)
* Use SqlToolsService built on .NET Core 2.0 and a build script updates (#131) * Bump version to 1.0.0a19 * Use .NET Core 2.0 RTM built sqltoolsservice * Add build script to upload to azure blob storage * Upgrade to VS 2017 * Remove 3.3 as supported Python version * Fix perf issue where main event loop takes 100% of CPU (#132) Fix perf issue where main event loop takes 100% of CPU We have a 2 threads: Thread #1 runs in a loop polling the response queue Thread #2 runs in a loop decoding responses from the sqltoolsservice over stdout and posting them to the response queue Since thread #1 doesn't sleep, it's takes 100% CPU. In addition, running python 2.7 on windows, #2 doesn’t preempt the CPU due to #1 taking all of the CPU cycles, so no response is processed. Fix is simple – thread #1 needs to sleep so thread #2 can get scheduled and get it’s work done. * Refine event loop perf fix in main.py Refine event loop perf fix in main.py * Fixing regular expression Previous regex would result in release:a1 and release_version: 12. Modified the regex for part Release to only pick up lower case letters. * Adding missing forward slash on test pypi url * fixing typos/grammar (#138) fixing typos/grammar. * Updating to release version 1.0.0a20. * Create doc for official msft docs page * Updated documentation page with usage_guide * Added link to download adventureworks * Updated with sqlcmd usage * Added in run and cloud shell support * Update/consolidate linux install (#153) * universal linux wheel gen and setup update. * Updating version cfg. * Updating sqltoolsservice container. * Updating spacing for flake8. * Updating team email. (#154) * Fixing resource warning for sqltoolsservice (#158) * Fixing resource warning for sqltoolsservice by closing stdout after killing process. * Shortening sleep time during shutdown. * Fixing missing bracket. * Updating doc's after repro rename (#160) * Updating files after repro rename. * Fixing flake8 issues. * Ron/platform wheels support (#161) * Merge release 1.0.0a21 (#155) * Use SqlToolsService built on .NET Core 2.0 and a build script updates (#131) * Bump version to 1.0.0a19 * Use .NET Core 2.0 RTM built sqltoolsservice * Add build script to upload to azure blob storage * Upgrade to VS 2017 * Remove 3.3 as supported Python version * Fix perf issue where main event loop takes 100% of CPU (#132) Fix perf issue where main event loop takes 100% of CPU We have a 2 threads: Thread #1 runs in a loop polling the response queue Thread #2 runs in a loop decoding responses from the sqltoolsservice over stdout and posting them to the response queue Since thread #1 doesn't sleep, it's takes 100% CPU. In addition, running python 2.7 on windows, #2 doesn’t preempt the CPU due to #1 taking all of the CPU cycles, so no response is processed. Fix is simple – thread #1 needs to sleep so thread #2 can get scheduled and get it’s work done. * Refine event loop perf fix in main.py Refine event loop perf fix in main.py * Fixing regular expression Previous regex would result in release:a1 and release_version: 12. Modified the regex for part Release to only pick up lower case letters. * Adding missing forward slash on test pypi url * fixing typos/grammar (#138) fixing typos/grammar. * Updating to release version 1.0.0a20. * Create doc for official msft docs page * Updated documentation page with usage_guide * Added link to download adventureworks * Updated with sqlcmd usage * Added in run and cloud shell support * Update/consolidate linux install (#153) * universal linux wheel gen and setup update. * Updating version cfg. * Updating sqltoolsservice container. * Updating spacing for flake8. * Updating team email. (#154) * Updating mssqltoolsservice to be integrated as a package of mssqlscripter. * Updating sqltoolsservice to be loaded from the repro instead of storage account. * Fix index file generation for daily storage account. * Fixing manylinux1 tag. * Updating platform tag for win x64. * Renaming sqltoolsservice win x64 folder. * Adding platform tags for win_amd64, manylinux1_x86_64, manylinux1_i686. * version bumping to 1.0.0a22. * Flake8 format fixes. * Erroring out when build receives invalid flag. * Fixing tag for win64 * Update libunwind8 install for CentOS * Ron/sqltoolsservice update (#163) * Updating sqltoolsservice with self contained version. * Refreshing sqltoolsservice again. * Making mssql-scripter executable and adding null checks in main.py * Adding clean up step to remove build directory after each build. * Fixing path for build directory. * Removing 'pypi' from upload step. * Flake 8 extra line fix.
1 parent 606053a commit be55629

33 files changed

+295
-541
lines changed

.bumpversion.cfg

+1-5
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[bumpversion]
2-
current_version = 1.0.0a21
2+
current_version = 1.0.0a22
33

44
parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)((?P<release>[a-z]+))(?P<release_version>\d+)
55
serialize =
@@ -14,9 +14,5 @@ values =
1414

1515
[bumpversion:file:setup.py]
1616

17-
[bumpversion:file:mssqltoolsservice/setup.py]
18-
1917
[bumpversion:file:mssqlscripter/__init__.py]
2018

21-
[bumpversion:file:mssqltoolsservice/mssqltoolsservice/__init__.py]
22-

.gitignore

+1-1
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ ENV/
9393

9494
# sqltoolsservice binaries
9595
/mssqlscripter/sqltoolsservice/*
96-
/mssqltoolsservice/mssqltoolsservice/bin/*
96+
/mssqlscripter/mssqltoolsservice/bin/*
9797

9898

9999
# VSCode configuration

MANIFEST.in

+2
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,4 @@
11
include LICENSE.txt
22
include README.rst
3+
include mssqlscripter/mssqltoolsservice/bin/*
4+
include mssqlscripter/mssqltoolsservice/bin/*/*

README.rst

+8-8
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,17 @@
1-
.. image:: https://travis-ci.org/Microsoft/sql-xplat-cli.svg?branch=dev
2-
:target: https://travis-ci.org/Microsoft/sql-xplat-cli
1+
.. image:: https://travis-ci.org/Microsoft/mssql-scripter.svg?branch=dev
2+
:target: https://travis-ci.org/Microsoft/mssql-scripter
33

44
.. image:: https://ci.appveyor.com/api/projects/status/vpm8p71265ijytqs/branch/dev?svg=true
5-
:target: https://ci.appveyor.com/project/MrMeemus/sql-xplat-cli
5+
:target: https://ci.appveyor.com/project/MrMeemus/mssql-scripter
66

7-
.. image:: https://codecov.io/gh/Microsoft/sql-xplat-cli/branch/dev/graph/badge.svg?token=M96uWrHOIu
8-
  :target: https://codecov.io/gh/Microsoft/sql-xplat-cli/branch/dev
7+
.. image:: https://codecov.io/gh/Microsoft/mssql-scripter/branch/dev/graph/badge.svg?token=M96uWrHOIu
8+
  :target: https://codecov.io/gh/Microsoft/mssql-scripter/branch/dev
99

1010
.. image:: https://badge.fury.io/py/mssql-scripter.svg
1111
:target: https://pypi.python.org/pypi/mssql-scripter
1212

1313
.. image:: https://img.shields.io/pypi/pyversions/mssql-scripter.svg
14-
:target: https://travis-ci.org/Microsoft/sql-xplat-cli
14+
:target: https://travis-ci.org/Microsoft/mssql-scripter
1515

1616
mssql-scripter
1717
===============
@@ -68,7 +68,7 @@ mssql-scritper is licensed under the `MIT license`_.
6868
.. _installation guide: doc/installation_guide.md
6969
.. _development guide: doc/development_guide.md
7070
.. _usage guide: doc/usage_guide.md
71-
.. _Issues: https://github.com/Microsoft/sql-xplat-cli/issues
71+
.. _Issues: https://github.com/Microsoft/mssql-scripter/issues
7272
.. _Microsoft Open Source Code of Conduct: https://opensource.microsoft.com/codeofconduct/
7373
.. _Code of Conduct FAQ: https://opensource.microsoft.com/codeofconduct/faq/
74-
.. _MIT license: https://github.com/Microsoft/sql-xplat-cli/blob/dev/LI
74+
.. _MIT license: https://github.com/Microsoft/mssql-scripter/blob/dev/LI

build.py

+113-60
Original file line numberDiff line numberDiff line change
@@ -6,23 +6,55 @@
66
# --------------------------------------------------------------------------------------------
77

88
from __future__ import print_function
9+
from azure.storage.blob import BlockBlobService, ContentSettings
910
import os
10-
import re
1111
import sys
12-
import tempfile
1312
import utility
14-
from azure.storage.blob import BlockBlobService, ContentSettings
13+
import mssqlscripter.mssqltoolsservice.external as mssqltoolsservice
1514

1615
AZURE_STORAGE_CONNECTION_STRING = os.environ.get('AZURE_STORAGE_CONNECTION_STRING')
1716
BLOB_CONTAINER_NAME = 'simple'
18-
UPLOADED_PACKAGE_LINKS = []
17+
UPLOADED_PACKAGE_LINKS = []
1918

2019

2120
def print_heading(heading, f=None):
2221
print('{0}\n{1}\n{0}'.format('=' * len(heading), heading), file=f)
2322

2423

25-
def upload_index_file(service, blob_name, title, links):
24+
def build(platform_names):
25+
"""
26+
Builds mssql-scripter package.
27+
"""
28+
print_heading('Cleanup')
29+
30+
# clean
31+
utility.clean_up(utility.MSSQLSCRIPTER_DIST_DIRECTORY)
32+
33+
print_heading('Running setup')
34+
35+
# install general requirements.
36+
utility.exec_command('pip install -r dev_requirements.txt', utility.ROOT_DIR)
37+
38+
# convert windows line endings to unix for mssql-cli bash script
39+
utility.exec_command('python dos2unix.py mssql-scripter mssql-scripter', utility.ROOT_DIR)
40+
41+
for platform in platform_names:
42+
utility.clean_up(utility.MSSQLSCRIPTER_BUILD_DIRECTORY)
43+
utility.cleaun_up_egg_info_sub_directories(utility.ROOT_DIR)
44+
45+
mssqltoolsservice.copy_sqltoolsservice(platform)
46+
47+
print_heading('Building mssql-scripter {} wheel package package'.format(platform))
48+
utility.exec_command('python --version', utility.ROOT_DIR)
49+
utility.exec_command(
50+
'python setup.py check -r -s bdist_wheel --plat-name {}'.format(platform),
51+
utility.ROOT_DIR,
52+
continue_on_error=False)
53+
54+
mssqltoolsservice.clean_up_sqltoolsservice()
55+
56+
57+
def _upload_index_file(service, blob_name, title, links):
2658
print('Uploading index file {}'.format(blob_name))
2759
service.create_blob_from_text(
2860
container_name=BLOB_CONTAINER_NAME,
@@ -37,23 +69,23 @@ def upload_index_file(service, blob_name, title, links):
3769
content_language=None,
3870
content_md5=None,
3971
cache_control=None
40-
)
72+
)
4173
)
4274

4375

44-
def gen_pkg_index_html(service, pkg_name):
76+
def _gen_pkg_index_html(service, pkg_name):
4577
links = []
4678
index_file_name = pkg_name+'/'
4779
for blob in list(service.list_blobs(BLOB_CONTAINER_NAME, prefix=index_file_name)):
4880
if blob.name == index_file_name:
4981
# Exclude the index file from being added to the list
5082
continue
5183
links.append(blob.name.replace(index_file_name, ''))
52-
upload_index_file(service, index_file_name, 'Links for {}'.format(pkg_name), links)
84+
_upload_index_file(service, index_file_name, 'Links for {}'.format(pkg_name), links)
5385
UPLOADED_PACKAGE_LINKS.append(index_file_name)
5486

5587

56-
def upload_package(service, file_path, pkg_name):
88+
def _upload_package(service, file_path, pkg_name):
5789
print('Uploading {}'.format(file_path))
5890
file_name = os.path.basename(file_path)
5991
blob_name = '{}/{}'.format(pkg_name, file_name)
@@ -62,63 +94,84 @@ def upload_package(service, file_path, pkg_name):
6294
blob_name=blob_name,
6395
file_path=file_path
6496
)
65-
gen_pkg_index_html(service, pkg_name)
66-
6797

68-
def build(options):
6998

70-
supported_actions = ['nightly']
71-
action = None
99+
def validate_package(platform_names):
100+
"""
101+
Install mssql-scripter wheel package locally.
102+
"""
103+
root_dir = os.path.abspath(os.path.join(os.path.abspath(__file__), '..'))
104+
# Local install of mssql-scripter.
105+
mssqlscripter_wheel_dir = os.listdir(utility.MSSQLSCRIPTER_DIST_DIRECTORY)
106+
current_platform = utility.get_current_platform()
72107

73-
if len(options) >= 1:
74-
if options[0] not in supported_actions:
75-
print('Please provide a supported action {}.'.format(supported_actions))
76-
return
77-
action = options[0]
108+
mssqlscripter_wheel_name = [pkge for pkge in mssqlscripter_wheel_dir if current_platform in pkge]
78109

79-
if action == 'nightly':
80-
assert AZURE_STORAGE_CONNECTION_STRING, 'Set AZURE_STORAGE_CONNECTION_STRING environment variable'
81-
82-
print_heading('Cleanup')
83-
84-
# clean
85-
utility.clean_up(utility.MSSQLSCRIPTER_DIST_DIRECTORY)
86-
utility.clean_up(utility.MSSQLTOOLSSERVICE_DIST_DIRECTORY)
87-
utility.cleaun_up_egg_info_sub_directories(utility.ROOT_DIR)
88-
utility.cleaun_up_egg_info_sub_directories(utility.MSSQLTOOLSSERVICE_DIRECTORY)
110+
# To ensure we have a clean install, we disable the cache as to prevent cache overshadowing actual changes made.
111+
utility.exec_command(
112+
'pip install --no-cache-dir --no-index ./dist/{}'.format(mssqlscripter_wheel_name),
113+
root_dir, continue_on_error=False)
89114

90-
print_heading('Running setup')
91-
92-
# install general requirements.
93-
utility.exec_command('pip install -r dev_requirements.txt', utility.ROOT_DIR)
94-
95-
print_heading('Running mssql-scripter tests')
96-
utility.exec_command('tox', utility.ROOT_DIR, continue_on_error = False)
97-
98-
print_heading('Building mssql-scripter pip package')
99-
utility.exec_command('python setup.py check -r -s sdist', utility.ROOT_DIR, continue_on_error = False)
100-
101-
print_heading('Building mssqltoolsservice pip package')
102-
utility.exec_command('python buildwheels.py', utility.MSSQLTOOLSSERVICE_DIRECTORY, continue_on_error = False)
103-
104-
if action == 'nightly':
105-
blob_service = BlockBlobService(connection_string=AZURE_STORAGE_CONNECTION_STRING)
106-
107-
print_heading('Uploading packages to blob storage ')
108-
for pkg in os.listdir(utility.MSSQLSCRIPTER_DIST_DIRECTORY):
109-
pkg_path = os.path.join(utility.MSSQLSCRIPTER_DIST_DIRECTORY, pkg)
110-
print('Uploading package {}'.format(pkg_path))
111-
upload_package(blob_service, pkg_path, 'mssql-scripter')
112-
113-
for pkg in os.listdir(utility.MSSQLTOOLSSERVICE_DIST_DIRECTORY):
114-
pkg_path = os.path.join(utility.MSSQLTOOLSSERVICE_DIST_DIRECTORY, pkg)
115-
pkg_name = os.path.basename(pkg_path).split('-')[0].replace('_', '-').lower()
116-
print('Uploading package {}'.format(pkg_name))
117-
upload_package(blob_service, pkg_path, pkg_name)
118115

119-
# Upload the final index file
120-
upload_index_file(blob_service, 'index.html', 'Simple Index', UPLOADED_PACKAGE_LINKS)
116+
def publish_daily(platforms_names):
117+
"""
118+
Publish mssql-scripter wheel package to daily storage account.
119+
"""
120+
print('Publishing to simple container within storage account.')
121+
assert AZURE_STORAGE_CONNECTION_STRING, 'Set AZURE_STORAGE_CONNECTION_STRING environment variable'
122+
123+
blob_service = BlockBlobService(connection_string=AZURE_STORAGE_CONNECTION_STRING)
124+
125+
print_heading('Uploading packages to blob storage ')
126+
for pkg in os.listdir(utility.MSSQLSCRIPTER_DIST_DIRECTORY):
127+
pkg_path = os.path.join(utility.MSSQLSCRIPTER_DIST_DIRECTORY, pkg)
128+
print('Uploading package {}'.format(pkg_path))
129+
_upload_package(blob_service, pkg_path, 'mssql-scripter')
130+
131+
# Upload index files
132+
_gen_pkg_index_html(blob_service, 'mssql-scripter')
133+
_upload_index_file(blob_service, 'index.html', 'Simple Index', UPLOADED_PACKAGE_LINKS)
134+
135+
136+
def publish_official(platforms_names):
137+
"""
138+
Publish mssql-scripter wheel package to PyPi.
139+
"""
140+
mssqlscripter_wheel_dir = os.listdir(utility.MSSQLSCRIPTER_DIST_DIRECTORY)
141+
# Run twine action for mssqlscripter.
142+
# Only authorized users with credentials will be able to upload this package.
143+
# Credentials will be stored in a .pypirc file.
144+
for mssqlscripter_wheel_name in mssqlscripter_wheel_dir:
145+
utility.exec_command(
146+
'twine upload {}'.format(mssqlscripter_wheel_name),
147+
utility.MSSQLSCRIPTER_DIST_DIRECTORY)
121148

122149

123150
if __name__ == '__main__':
124-
build(sys.argv[1:])
151+
action = 'build'
152+
supported_platforms = [
153+
'win32',
154+
'win_amd64',
155+
'win64',
156+
'macosx_10_11_intel',
157+
'manylinux1_x86_64',
158+
'manylinux1_i686']
159+
160+
targets = {
161+
'build': build,
162+
'validate_package': validate_package,
163+
'publish_daily': publish_daily,
164+
'publish_official': publish_official
165+
}
166+
167+
if len(sys.argv) > 1:
168+
action = sys.argv[1]
169+
170+
if len(sys.argv) > 2:
171+
supported_platforms = [sys.argv[2]]
172+
173+
if action in targets:
174+
targets[action](supported_platforms)
175+
else:
176+
print('{} is not a supported action'.format(action))
177+
print('Supported actions are {}'.format(list(targets.keys())))

dev_setup.py

+8-6
Original file line numberDiff line numberDiff line change
@@ -9,19 +9,21 @@
99
from __future__ import print_function
1010

1111
import os
12-
import setup
12+
import platform
1313
import utility
14+
import mssqlscripter.mssqltoolsservice.external as mssqltoolsservice
1415

1516
print('Running dev setup...')
1617
print('Root directory \'{}\'\n'.format(utility.ROOT_DIR))
1718

1819
# install general requirements.
1920
utility.exec_command('pip install -r dev_requirements.txt', utility.ROOT_DIR)
21+
run_time_id = utility.get_current_platform()
22+
23+
if run_time_id:
24+
mssqltoolsservice.copy_sqltoolsservice(run_time_id)
25+
else:
26+
print("This platform does not support mssqltoolsservice.")
2027

21-
# install mssqltoolsservice if this platform supports it.
22-
mssqltoolsservice_package_name = os.environ['MSSQLTOOLSSERVICE_PACKAGE_NAME']
23-
print('Installing {}...'.format(mssqltoolsservice_package_name))
24-
# mssqltoolsservice package name is retrieved from environment variable set by setup.py.
25-
utility.exec_command('pip install {}'.format(mssqltoolsservice_package_name), utility.ROOT_DIR)
2628

2729
print('Finished dev setup.')

doc/development_guide.md

+5-5
Original file line numberDiff line numberDiff line change
@@ -11,18 +11,18 @@ Development Guide
1111

1212
See the [Architecture Guide](architecture_guide.md) for details on how mssql-scripter is designed and implemented.
1313

14-
mssql-scripter sources are located on GitHub (https://github.com/Microsoft/sql-xplat-cli). In order to contribute to the project, you are expected to:
14+
mssql-scripter sources are located on GitHub (https://github.com/Microsoft/mssql-scripter). In order to contribute to the project, you are expected to:
1515
- Have a GitHub account. For Microsoft contributors, follow the guidelines on https://opensourcehub.microsoft.com/ to create, configure and link your account
16-
- Fork the https://github.com/Microsoft/sql-xplat-clirepository into your private GitHub account
17-
- Create pull requests against the https://github.com/Microsoft/sql-xplat-cli repository to get your code changes merged into the project repository.
16+
- Fork the https://github.com/Microsoft/mssql-scripterrepository into your private GitHub account
17+
- Create pull requests against the https://github.com/Microsoft/mssql-scripter repository to get your code changes merged into the project repository.
1818

1919
## <a name="Preparing_Machine"></a>1. Preparing your machine
2020
1. Install latest Python from http://python.org. Please note that the version of Python that comes preinstalled on OSX is 2.7. It is recommended to install both Python 2.7 and Python3.6 to ensure backwards compatibility for testing.
2121
#### Windows
2222
- The latest Python installation package can be downloaded from [here](https://www.python.org/downloads/).
2323
- During installation, select the 'Add Python to PATH' option. Python must be in the PATH environment variable.
2424

25-
2. Clone the repo from [https://github.com/Microsoft/sql-xplat-cli](https://github.com/Microsoft/sql-xplat-cli)
25+
2. Clone the repo from [https://github.com/Microsoft/mssql-scripter](https://github.com/Microsoft/mssql-scripter)
2626

2727
## <a name="Environment_Setup"></a>2. Environment Setup
2828
When developing on a Python project, it is recommended to do so in a virtual environment. A virtual environment is a sandbox that maintains a copy of all libraries necessary to run python in a isolated environment without interfering with the system or global python. For more information on virtual environments, go to [Virtual Environment Info](docs/virtual_environment_info.md).
@@ -78,7 +78,7 @@ General development steps that apply to both a virtual environment or a global e
7878
## <a name="Configure_IDE"></a>3. Configuring your IDE
7979
#### Visual Studio (Windows only)
8080
1. Install [Python Tools for Visual Studio](https://github.com/Microsoft/PTVS)
81-
2. Open the `<clone_root>\sql-xplat-cli.pyproj` project in Visual Studio
81+
2. Open the `<clone_root>\mssql-scripter.pyproj` project in Visual Studio
8282
8383
8484
#### Visual Studio Code (Any platform)

doc/installation_guide.md

+11-2
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ C:\> pip install mssql-scripter
9090

9191
# Troubleshooting
9292

93-
If you're having installation issues, please check the below known issues and workarounds. If you're having a different issue, please check the [issues](https://github.com/Microsoft/sql-xplat-cli/issues) page to see if the issue has already been reported. If you don't see your issue there, filing a new issue would be appreciated.
93+
If you're having installation issues, please check the below known issues and workarounds. If you're having a different issue, please check the [issues](https://github.com/Microsoft/mssql-scripter/issues) page to see if the issue has already been reported. If you don't see your issue there, filing a new issue would be appreciated.
9494

9595
## Error: No module named mssqlscripter
9696
If the installation was successful and this error message is encountered, this may be caused by different versions of python in the environment.
@@ -151,10 +151,19 @@ $ sudo apt-get update
151151
$ sudo apt-get install libunwind8
152152
```
153153

154+
### CentOS 7
155+
Run commands
156+
```shell
157+
## libunwind.x86_64 is in the epel-release repository
158+
## (Not necessary on GNOME, minimal install does not have this installed)
159+
$ sudo yum -y install epel-release
160+
$ sudo yum -y install libunwind
161+
```
162+
154163
## Installing the icu package
155164

156165
### RHEL 7.3
157166
Run commands
158167
```shell
159-
$ sudo sudo yum install icu
168+
$ sudo yum install icu
160169
```

0 commit comments

Comments
 (0)