Skip to content
This repository was archived by the owner on Oct 12, 2023. It is now read-only.

Commit f18e3a6

Browse files
author
Zeqi Cui
authored
Merge pull request #2 from Azure/setup-travis-ci
Setup Travis CI
2 parents 39d8fe2 + 9ebe84f commit f18e3a6

File tree

18 files changed

+336
-45
lines changed

18 files changed

+336
-45
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
*.class
22
*.log
33
*.iml
4+
target/
45

56
#IDE
67
.idea/*

‎.travis.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
language: java
2+
sudo: required
3+
4+
matrix:
5+
include:
6+
- os: linux
7+
jdk: oraclejdk8
8+
- os: osx
9+
osx_image: xcode8
10+
11+
script:
12+
- mvn clean package

‎README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# Spark connector for Azure SQL Databases and SQL Server
22

3+
[![Build Status](https://travis-ci.org/Azure/azure-sqldb-spark.svg?branch=master)](https://travis-ci.org/Azure/azure-sqldb-spark)
4+
35
The Spark connector for [Azure SQL Database](https://azure.microsoft.com/en-us/services/sql-database/) and [SQL Server](https://www.microsoft.com/en-us/sql-server/default.aspx) enables SQL databases, including Azure SQL Databases and SQL Server, to act as input data source or output data sink for Spark jobs. It allows you to utilize real time transactional data in big data analytics and persist results for adhoc queries or reporting.
46

57
Comparing to the built-in Spark connector, this connector provides the ability to bulk insert data into SQL databases. It can outperform row by row insertion with 10x to 20x faster performance. The Spark connector for Azure SQL Databases and SQL Server also supports AAD authentication. It allows you securely connecting to your Azure SQL databases from Azure Databricks using your AAD account. It provides similar interfaces with the built-in JDBC connector. It is easy to migrate your existing Spark jobs to use this new connector.

‎lib/scalastyle_config.xml

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
<scalastyle commentFilter="enabled">
2+
<name>Scalastyle standard configuration</name>
3+
<check level="warning" class="org.scalastyle.file.FileTabChecker" enabled="true"></check>
4+
<check level="warning" class="org.scalastyle.file.FileLengthChecker" enabled="true">
5+
<parameters>
6+
<parameter name="maxFileLength"><![CDATA[800]]></parameter>
7+
</parameters>
8+
</check>
9+
<check level="warning" class="org.scalastyle.file.HeaderMatchesChecker" enabled="true">
10+
<parameters>
11+
<parameter name="header"><![CDATA[/**
12+
* The MIT License (MIT)
13+
* Copyright (c) 2018 Microsoft Corporation
14+
*
15+
* Permission is hereby granted, free of charge, to any person obtaining a copy
16+
* of this software and associated documentation files (the "Software"), to deal
17+
* in the Software without restriction, including without limitation the rights
18+
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
19+
* copies of the Software, and to permit persons to whom the Software is
20+
* furnished to do so, subject to the following conditions:
21+
*
22+
* The above copyright notice and this permission notice shall be included in all
23+
* copies or substantial portions of the Software.
24+
*
25+
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
26+
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
27+
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
28+
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
29+
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
30+
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
31+
* SOFTWARE.
32+
*/]]></parameter>
33+
</parameters>
34+
</check>
35+
<check level="warning" class="org.scalastyle.scalariform.SpacesAfterPlusChecker" enabled="true"></check>
36+
<check level="warning" class="org.scalastyle.file.WhitespaceEndOfLineChecker" enabled="true"></check>
37+
<check level="warning" class="org.scalastyle.scalariform.SpacesBeforePlusChecker" enabled="true"></check>
38+
<check level="warning" class="org.scalastyle.file.FileLineLengthChecker" enabled="true">
39+
<parameters>
40+
<parameter name="maxLineLength"><![CDATA[160]]></parameter>
41+
<parameter name="tabSize"><![CDATA[4]]></parameter>
42+
</parameters>
43+
</check>
44+
<check level="warning" class="org.scalastyle.scalariform.ClassNamesChecker" enabled="true">
45+
<parameters>
46+
<parameter name="regex"><![CDATA[[A-Z][A-Za-z]*]]></parameter>
47+
</parameters>
48+
</check>
49+
<check level="warning" class="org.scalastyle.scalariform.ObjectNamesChecker" enabled="true">
50+
<parameters>
51+
<parameter name="regex"><![CDATA[[A-Z][A-Za-z]*]]></parameter>
52+
</parameters>
53+
</check>
54+
<check level="warning" class="org.scalastyle.scalariform.PackageObjectNamesChecker" enabled="true">
55+
<parameters>
56+
<parameter name="regex"><![CDATA[^[a-z][A-Za-z]*$]]></parameter>
57+
</parameters>
58+
</check>
59+
<check level="warning" class="org.scalastyle.scalariform.EqualsHashCodeChecker" enabled="true"></check>
60+
<check level="warning" class="org.scalastyle.scalariform.IllegalImportsChecker" enabled="true">
61+
<parameters>
62+
<parameter name="illegalImports"><![CDATA[sun._,java.awt._]]></parameter>
63+
</parameters>
64+
</check>
65+
<check level="warning" class="org.scalastyle.scalariform.ParameterNumberChecker" enabled="true">
66+
<parameters>
67+
<parameter name="maxParameters"><![CDATA[8]]></parameter>
68+
</parameters>
69+
</check>
70+
<check level="warning" class="org.scalastyle.scalariform.MagicNumberChecker" enabled="true">
71+
<parameters>
72+
<parameter name="ignore"><![CDATA[-1,0,1,2,3]]></parameter>
73+
</parameters>
74+
</check>
75+
<check level="warning" class="org.scalastyle.scalariform.NoWhitespaceBeforeLeftBracketChecker" enabled="false"></check>
76+
<check level="warning" class="org.scalastyle.scalariform.NoWhitespaceAfterLeftBracketChecker" enabled="true"></check>
77+
<check level="warning" class="org.scalastyle.scalariform.ReturnChecker" enabled="true"></check>
78+
<check level="warning" class="org.scalastyle.scalariform.NullChecker" enabled="false"></check>
79+
<check level="warning" class="org.scalastyle.scalariform.NoCloneChecker" enabled="true"></check>
80+
<check level="warning" class="org.scalastyle.scalariform.NoFinalizeChecker" enabled="true"></check>
81+
<check level="warning" class="org.scalastyle.scalariform.CovariantEqualsChecker" enabled="true"></check>
82+
<check level="warning" class="org.scalastyle.scalariform.StructuralTypeChecker" enabled="true"></check>
83+
<check level="warning" class="org.scalastyle.file.RegexChecker" enabled="true">
84+
<parameters>
85+
<parameter name="regex"><![CDATA[println]]></parameter>
86+
</parameters>
87+
</check>
88+
<check level="warning" class="org.scalastyle.scalariform.NumberOfTypesChecker" enabled="true">
89+
<parameters>
90+
<parameter name="maxTypes"><![CDATA[30]]></parameter>
91+
</parameters>
92+
</check>
93+
<check level="warning" class="org.scalastyle.scalariform.CyclomaticComplexityChecker" enabled="true">
94+
<parameters>
95+
<parameter name="maximum"><![CDATA[15]]></parameter>
96+
</parameters>
97+
</check>
98+
<check level="warning" class="org.scalastyle.scalariform.UppercaseLChecker" enabled="true"></check>
99+
<check level="warning" class="org.scalastyle.scalariform.SimplifyBooleanExpressionChecker" enabled="true"></check>
100+
<check level="warning" class="org.scalastyle.scalariform.IfBraceChecker" enabled="true">
101+
<parameters>
102+
<parameter name="singleLineAllowed"><![CDATA[true]]></parameter>
103+
<parameter name="doubleLineAllowed"><![CDATA[false]]></parameter>
104+
</parameters>
105+
</check>
106+
<check level="warning" class="org.scalastyle.scalariform.MethodLengthChecker" enabled="true">
107+
<parameters>
108+
<parameter name="maxLength"><![CDATA[70]]></parameter>
109+
</parameters>
110+
</check>
111+
<check level="warning" class="org.scalastyle.scalariform.MethodNamesChecker" enabled="true">
112+
<parameters>
113+
<parameter name="regex"><![CDATA[^[a-z][A-Za-z0-9]*$]]></parameter>
114+
</parameters>
115+
</check>
116+
<check level="warning" class="org.scalastyle.scalariform.NumberOfMethodsInTypeChecker" enabled="true">
117+
<parameters>
118+
<parameter name="maxMethods"><![CDATA[30]]></parameter>
119+
</parameters>
120+
</check>
121+
<check level="warning" class="org.scalastyle.scalariform.PublicMethodsHaveTypeChecker" enabled="false"></check>
122+
<check level="warning" class="org.scalastyle.file.NewLineAtEofChecker" enabled="true"></check>
123+
<check level="warning" class="org.scalastyle.file.NoNewLineAtEofChecker" enabled="false"></check>
124+
<check level="warning" class="org.scalastyle.scalariform.WhileChecker" enabled="false"></check>
125+
<check level="warning" class="org.scalastyle.scalariform.VarFieldChecker" enabled="false"></check>
126+
<check level="warning" class="org.scalastyle.scalariform.VarLocalChecker" enabled="false"></check>
127+
<check level="warning" class="org.scalastyle.scalariform.RedundantIfChecker" enabled="true"></check>
128+
<check level="warning" class="org.scalastyle.scalariform.TokenChecker" enabled="true">
129+
<parameters>
130+
<parameter name="regex"><![CDATA[println]]></parameter>
131+
</parameters>
132+
<customMessage>Avoid print line, use logger.</customMessage>
133+
</check>
134+
<check level="warning" class="org.scalastyle.scalariform.DeprecatedJavaChecker" enabled="true"></check>
135+
<check level="warning" class="org.scalastyle.scalariform.EmptyClassChecker" enabled="true"></check>
136+
<check level="warning" class="org.scalastyle.scalariform.ClassTypeParameterChecker" enabled="false">
137+
<parameters>
138+
<parameter name="regex"><![CDATA[^[A-Z_]$]]></parameter>
139+
</parameters>
140+
</check>
141+
<check level="warning" class="org.scalastyle.scalariform.UnderscoreImportChecker" enabled="false"></check>
142+
<check level="warning" class="org.scalastyle.scalariform.LowercasePatternMatchChecker" enabled="true"></check>
143+
<check level="warning" class="org.scalastyle.scalariform.MultipleStringLiteralsChecker" enabled="false">
144+
<parameters>
145+
<parameter name="allowed"><![CDATA[2]]></parameter>
146+
<parameter name="ignoreRegex"><![CDATA[^""$]]></parameter>
147+
</parameters>
148+
</check>
149+
<check level="warning" class="org.scalastyle.scalariform.ImportGroupingChecker" enabled="true"></check>
150+
</scalastyle>

‎pom.xml

Lines changed: 106 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -3,19 +3,27 @@
33
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
44
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
55
<modelVersion>4.0.0</modelVersion>
6-
7-
<groupId>com.microsoft.azure.sqldb.spark</groupId>
6+
<groupId>com.microsoft.azure</groupId>
87
<artifactId>azure-sqldb-spark</artifactId>
8+
<packaging>jar</packaging>
99
<version>1.0.0</version>
10-
10+
<name>${project.groupId}:${project.artifactId}</name>
11+
<description>Spark Connector for Microsoft Azure SQL Database and SQL Server</description>
1112
<licenses>
1213
<license>
1314
<name>MIT License</name>
1415
<url>http://www.opensource.org/licenses/mit-license.php</url>
1516
</license>
1617
</licenses>
17-
18+
<properties>
19+
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
20+
</properties>
1821
<dependencies>
22+
<dependency>
23+
<groupId>org.scala-lang</groupId>
24+
<artifactId>scala-library</artifactId>
25+
<version>2.11.8</version>
26+
</dependency>
1927
<dependency>
2028
<groupId>junit</groupId>
2129
<artifactId>junit</artifactId>
@@ -48,6 +56,11 @@
4856
<artifactId>adal4j</artifactId>
4957
<version>1.2.0</version>
5058
</dependency>
59+
<dependency>
60+
<groupId>com.microsoft.sqlserver</groupId>
61+
<artifactId>mssql-jdbc</artifactId>
62+
<version>6.4.0.jre8</version>
63+
</dependency>
5164
</dependencies>
5265
<developers>
5366
<developer>
@@ -58,6 +71,57 @@
5871
</developers>
5972
<build>
6073
<plugins>
74+
<plugin>
75+
<groupId>org.codehaus.mojo</groupId>
76+
<artifactId>build-helper-maven-plugin</artifactId>
77+
<version>3.0.0</version>
78+
<executions>
79+
<execution>
80+
<phase>generate-sources</phase>
81+
<goals>
82+
<goal>add-source</goal>
83+
</goals>
84+
<configuration>
85+
<sources>
86+
<source>src/main/java</source>
87+
<source>src/main/scala</source>
88+
</sources>
89+
</configuration>
90+
</execution>
91+
<execution>
92+
<id>add-test-source</id>
93+
<phase>generate-test-sources</phase>
94+
<goals>
95+
<goal>add-test-source</goal>
96+
</goals>
97+
<configuration>
98+
<sources>
99+
<source>src/test/java</source>
100+
<source>src/test/scala</source>
101+
</sources>
102+
</configuration>
103+
</execution>
104+
</executions>
105+
</plugin>
106+
<plugin>
107+
<groupId>org.apache.maven.plugins</groupId>
108+
<artifactId>maven-assembly-plugin</artifactId>
109+
<version>3.0.0</version>
110+
<configuration>
111+
<descriptorRefs>
112+
<descriptorRef>jar-with-dependencies</descriptorRef>
113+
</descriptorRefs>
114+
</configuration>
115+
<executions>
116+
<execution>
117+
<id>make-assembly</id>
118+
<phase>package</phase>
119+
<goals>
120+
<goal>single</goal>
121+
</goals>
122+
</execution>
123+
</executions>
124+
</plugin>
61125
<plugin>
62126
<groupId>org.scalastyle</groupId>
63127
<artifactId>scalastyle-maven-plugin</artifactId>
@@ -70,7 +134,6 @@
70134
<sourceDirectory>${project.basedir}/src/main/scala</sourceDirectory>
71135
<testSourceDirectory>${project.basedir}/src/test/scala</testSourceDirectory>
72136
<configLocation>${project.basedir}/lib/scalastyle_config.xml</configLocation>
73-
<outputFile>${project.basedir}/scalastyle-output.xml</outputFile>
74137
<outputEncoding>UTF-8</outputEncoding>
75138
</configuration>
76139
<executions>
@@ -83,50 +146,69 @@
83146
</plugin>
84147
<plugin>
85148
<artifactId>maven-compiler-plugin</artifactId>
149+
<version>3.7.0</version>
86150
<configuration>
87151
<source>1.8</source>
88152
<target>1.8</target>
89153
</configuration>
90154
</plugin>
91155
<plugin>
92-
<groupId>org.apache.maven.plugins</groupId>
93-
<artifactId>maven-assembly-plugin</artifactId>
94-
<version>3.0.0</version>
156+
<groupId>net.alchim31.maven</groupId>
157+
<artifactId>scala-maven-plugin</artifactId>
158+
<version>3.2.2</version>
95159
<configuration>
96-
<descriptorRefs>
97-
<descriptorRef>jar-with-dependencies</descriptorRef>
98-
</descriptorRefs>
160+
<checkMultipleScalaVersions>false</checkMultipleScalaVersions>
161+
<scalaVersion>2.11.8</scalaVersion>
162+
<args>
163+
<arg>-deprecation</arg>
164+
<arg>-feature</arg>
165+
</args>
99166
</configuration>
100167
<executions>
101168
<execution>
102-
<id>assemble-all</id>
103-
<phase>package</phase>
169+
<id>scala-compile-first</id>
170+
<phase>process-resources</phase>
104171
<goals>
105-
<goal>single</goal>
172+
<goal>add-source</goal>
173+
</goals>
174+
</execution>
175+
<execution>
176+
<id>scala-compile</id>
177+
<goals>
178+
<goal>compile</goal>
179+
</goals>
180+
<phase>compile</phase>
181+
</execution>
182+
<execution>
183+
<id>test-compile</id>
184+
<goals>
185+
<goal>testCompile</goal>
186+
</goals>
187+
<phase>test-compile</phase>
188+
</execution>
189+
<execution>
190+
<id>scala-doc</id>
191+
<phase>prepare-package</phase>
192+
<goals>
193+
<goal>doc</goal>
194+
<goal>doc-jar</goal>
106195
</goals>
107196
</execution>
108197
</executions>
109198
</plugin>
110-
<plugin>
111-
<groupId>org.apache.maven.plugins</groupId>
112-
<artifactId>maven-surefire-plugin</artifactId>
113-
<version>2.7</version>
114-
<configuration>
115-
<skipTests>true</skipTests>
116-
</configuration>
117-
</plugin>
118199
<plugin>
119200
<groupId>org.scalatest</groupId>
120201
<artifactId>scalatest-maven-plugin</artifactId>
121202
<version>1.0</version>
122203
<configuration>
123204
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
124205
<junitxml>.</junitxml>
125-
<filereports>WDF TestSuite.txt</filereports>
206+
<filereports>TestSuite.txt</filereports>
207+
<stdout>W</stdout>
126208
</configuration>
127209
<executions>
128210
<execution>
129-
<id>test</id>
211+
<id>scala-test</id>
130212
<goals>
131213
<goal>test</goal>
132214
</goals>

‎src/main/java/com/microsoft/azure/sqldb/spark/bulkcopy/SQLServerBulkDataFrameFileRecord.java

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,6 @@
2222
*/
2323
package com.microsoft.azure.sqldb.spark.bulkcopy;
2424

25-
import com.microsoft.azure.sqldb.spark.Logging;
2625
import com.microsoft.sqlserver.jdbc.ISQLServerBulkRecord;
2726
import com.microsoft.sqlserver.jdbc.SQLServerException;
2827
import com.microsoft.sqlserver.jdbc.SQLServerResource;
@@ -40,7 +39,7 @@
4039
/**
4140
* Class to allow SQLServerBulkCopy to write data to SQL Server Tables from Spark DataFrames
4241
*/
43-
public class SQLServerBulkDataFrameFileRecord extends Logging implements ISQLServerBulkRecord, java.lang.AutoCloseable {
42+
public class SQLServerBulkDataFrameFileRecord implements ISQLServerBulkRecord, java.lang.AutoCloseable {
4443

4544
private Iterator<Row> iterator;
4645

0 commit comments

Comments
 (0)