Skip to content

Commit c2f8ed5

Browse files
harishskfrank-dong-ms-zzmichaelgsharpmstfblfrank-dong-ms
authored
Merge arcade to master (dotnet#5525)
* Initial commit for Arcade migration * Added omitted files * Changed strong name signing to use the same key for shipping and test assemblies * arcade linux build (dotnet#5423) * arcade linux build * put file execution permission change into source control * The `-test` command for windows. Nuget packages (dotnet#5464) * working on testing * testing updates * tests almost working * build changes * all tests should be working * changes from PR comments * fixes for .net 3.1 * Fixed extension check. Removed <PackageId> where not needed * Removed pkg folder and updated paths. * Added test key. (dotnet#5475) * Added test key. * Update PublicKey.cs Removed extra newline. * Update ComponentCatalog.cs Fixed 3 spaces to 4. * Windows CI working (dotnet#5477) * ci testing changes * comments from pr * Added Linux & Mac changes for Arcade (dotnet#5479) * Initial Windows, Linux, Macos builds test * Add Linux/MacOS specific CI requirements * Run Arcade CI tests on MacOS/Linux * Fix final package building * Add benchmark download to benchmars .csporj file * Print detailed status of each unit test * Install CentOS & Ubuntu build dependencies * Use container names to differenciate between Ubuntu & CentOS * Remove sudo usage in CentOS * Fix Linux build dependencies * Add -y param to apt install * Remove installation of Linux dependencies * Minor additions * Rename Benchmarks to PerformanceTests for Arcade * Changes * Added benchmark doc changes * Pre-merge changes * Fixing failing Arcade Windows Builds (dotnet#5482) * Try Windows build single quote fix * Remove %20 * Added variable space value * Using variables for spacing * Added space values as job parameters * Try conditional variables again * fix official builds * Revert "fix official builds" This reverts commit 7dbbdc7. * fixing tensorflow rebase issue * Fixes for many of the CI builds. (dotnet#5496) * yml log changes * Fix NetFX builds by ensuring assembly version is set correctly and not to Arcade default of 42.42.42.42 (dotnet#5503) * Fixed official builds for Arcade SDK (dotnet#5512) * Added fixes for official builds * Make .sh files executable * fix mkl nuget issue Co-authored-by: Frank Dong <[email protected]> * fix code generator tests failure (dotnet#5520) * Added fixes for official builds * Make .sh files executable * fix mkl nuget issue * fix code generate test fails * only add necessary dependency Co-authored-by: Mustafa Bal <[email protected]> * Fixed memory leaks from OnnxTransformer (dotnet#5518) * Fixed memory leak from OnnxTransformer and related x86 build fixes * Reverting x86 build related fixes to focus only on the memory leaks * Updated docs * Reverted OnnxRuntimeOutputCatcher to private class * Addressed code review comments * Refactored OnnxTransform back to using MapperBase based on code review comments * Handle integration tests and nightly build testing (dotnet#5509) * Make -integrationTests work * Update .yml file * Added the TargetArchitecture properties * Try out -integrationTest * Missed -integrationTest flag * Renamed FunctionalTestBaseClass to IntegrationTestBaseClass * Missed rename * Modified tests to make them more stable * Fixed leak in object pool (dotnet#5521) Co-authored-by: frank-dong-ms <[email protected]> Co-authored-by: Michael Sharp <[email protected]> Co-authored-by: Mustafa Bal <[email protected]> Co-authored-by: Frank Dong <[email protected]> Co-authored-by: Michael Sharp <[email protected]> Co-authored-by: Antonio Velázquez <[email protected]>
1 parent d257b88 commit c2f8ed5

File tree

321 files changed

+14109
-1265
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

321 files changed

+14109
-1265
lines changed

Diff for: ‎.gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55

66
# Tool Runtime Dir
77
/[Tt]ools/
8+
/.dotnet/
9+
/.packages/
810

911
# User-specific files
1012
*.suo

Diff for: ‎.vsts-dotnet-ci.yml

+3-3
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ resources:
66
containers:
77
- container: CentosContainer
88
image: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-8bba86b-20190314145033
9-
9+
1010
- container: UbuntuContainer
1111
image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-mlnet-20200515184230-2c829e8
1212

@@ -29,7 +29,7 @@ jobs:
2929
_targetFramework: netcoreapp3.1
3030
innerLoop: true
3131
pool:
32-
name: Hosted Ubuntu 1604
32+
name: Hosted Ubuntu 1604
3333

3434
- template: /build/ci/job-template.yml
3535
parameters:
@@ -38,7 +38,7 @@ jobs:
3838
container: UbuntuContainer
3939
innerLoop: true
4040
pool:
41-
name: Hosted Ubuntu 1604
41+
name: Hosted Ubuntu 1604
4242

4343
- template: /build/ci/job-template.yml
4444
parameters:

Diff for: ‎Directory.Build.props

+46-96
Original file line numberDiff line numberDiff line change
@@ -1,117 +1,37 @@
1+
<?xml version="1.0" encoding="utf-8"?>
12
<Project>
2-
<!-- Directory.Build.props contains the common build settings for all projects in the repo. -->
33

4-
<Import Project="build/BranchInfo.props" />
5-
<Import Project="build/Dependencies.props" />
4+
<Import Project="Sdk.props" Sdk="Microsoft.DotNet.Arcade.Sdk" />
5+
6+
<Import Project="eng/BranchInfo.props" />
7+
<Import Project="eng/ExternalBenchmarkDataFiles.props" />
8+
<Import Project="eng/TensorflowMetaFiles.props" />
69

710
<PropertyGroup>
811
<CreateSymbolicLinksForPublishFilesIfPossible>true</CreateSymbolicLinksForPublishFilesIfPossible>
912
</PropertyGroup>
1013

14+
<PropertyGroup>
15+
<Copyright>$(CopyrightNetFoundation)</Copyright>
16+
<TreatWarningsAsErrors>True</TreatWarningsAsErrors>
17+
<DebugType>portable</DebugType>
18+
<DebugSymbols>true</DebugSymbols>
19+
<LangVersion>latest</LangVersion>
20+
</PropertyGroup>
21+
1122
<PropertyGroup>
1223
<Configuration Condition="'$(Configuration)'==''">Debug</Configuration>
1324
<Configurations>Debug;Release;Debug-netcoreapp3_1;Release-netcoreapp3_1;Debug-netfx;Release-netfx</Configurations>
1425
<Platform Condition="'$(Platform)'==''">AnyCPU</Platform>
1526
<TargetArchitecture Condition="'$(TargetArchitecture)' == ''">x64</TargetArchitecture>
1627
<NativeTargetArchitecture Condition="'$(NativeTargetArchitecture)' == ''">$(TargetArchitecture)</NativeTargetArchitecture>
1728
<PlatformConfig>$(Platform).$(Configuration)</PlatformConfig>
29+
<StrongNameKeyId>Open</StrongNameKeyId>
1830
</PropertyGroup>
1931

2032
<PropertyGroup>
21-
<RestoreSources>
22-
https://api.nuget.org/v3/index.json;
23-
https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json;
24-
https://dotnet.myget.org/F/dotnet-core/api/v3/index.json;
25-
https://dotnet.myget.org/F/roslyn-analyzers/api/v3/index.json;
26-
https://pkgs.dev.azure.com/dnceng/public/_packaging/MachineLearning/nuget/v3/index.json;
27-
https://pkgs.dev.azure.com/dnceng/public/_packaging/machinelearning-testdata/nuget/v3/index.json;
28-
</RestoreSources>
29-
</PropertyGroup>
30-
31-
<!-- Common repo directories -->
32-
<PropertyGroup>
33-
<RepoRoot>$(MSBuildThisFileDirectory)</RepoRoot>
34-
<SourceDir>$(RepoRoot)src/</SourceDir>
33+
<PackageAssetsPath>$(ArtifactsDir)pkgassets/</PackageAssetsPath>
3534
<PkgDir>$(RepoRoot)pkg/</PkgDir>
36-
37-
<!-- Output directories -->
38-
<BinDir Condition="'$(BinDir)'==''">$([MSBuild]::NormalizeDirectory('$(RepoRoot)', 'bin'))</BinDir>
39-
<BaseOutputPath Condition="'$(BaseOutputPath)'==''">$(BinDir)</BaseOutputPath>
40-
<ObjDir Condition="'$(ObjDir)'==''">$([MSBuild]::NormalizeDirectory('$(BinDir)', 'obj'))</ObjDir>
41-
<RootIntermediateOutputPath Condition="'$(RootIntermediateOutputPath)'==''">$(ObjDir)</RootIntermediateOutputPath>
42-
43-
<IntermediateOutputRootPath Condition="'$(IntermediateOutputRootPath)' == ''">$(RootIntermediateOutputPath)$(PlatformConfig)\</IntermediateOutputRootPath>
44-
<IntermediateOutputPath Condition="'$(IntermediateOutputPath)' == ''">$(IntermediateOutputRootPath)$(MSBuildProjectName)\</IntermediateOutputPath>
45-
<BaseIntermediateOutputPath Condition="'$(BaseIntermediateOutputPath)' == ''">$(IntermediateOutputPath)</BaseIntermediateOutputPath>
46-
47-
<OutputPath Condition="'$(OutputPath)'==''">$(BaseOutputPath)$(PlatformConfig)\$(MSBuildProjectName)\</OutputPath>
48-
49-
<PackageAssetsPath>$(ObjDir)/packages/</PackageAssetsPath>
50-
51-
<PackageOutputPath Condition="'$(PackageOutputPath)'=='' and '$(NonShippingPackage)' == 'true'">$(BinDir)packages_noship/</PackageOutputPath>
52-
<PackageOutputPath Condition="'$(PackageOutputPath)'==''">$(BinDir)packages/</PackageOutputPath>
53-
54-
<NativeOutputPath>$(BaseOutputPath)$(NativeTargetArchitecture).$(Configuration)\Native\</NativeOutputPath>
55-
56-
<!-- Input Directories -->
57-
<PackagesDir>$(DotNetRestorePackagesPath)</PackagesDir>
58-
<PackagesDir Condition="'$(PackagesDir)'==''">$(RepoRoot)packages/</PackagesDir>
59-
<RestorePackagesPath>$(PackagesDir)</RestorePackagesPath>
60-
<ToolsDir Condition="'$(ToolsDir)'==''">$(RepoRoot)Tools/</ToolsDir>
61-
</PropertyGroup>
62-
63-
<Import Project="$(ToolsDir)BuildVersion.targets"
64-
Condition="Exists('$(ToolsDir)BuildVersion.targets')" />
65-
66-
<!-- Version properties -->
67-
<PropertyGroup>
68-
<VersionPrefix>$(MajorVersion).$(MinorVersion).$(PatchVersion)</VersionPrefix>
69-
<BuildNumberMajor Condition="'$(BuildNumberMajor)' == ''">00001</BuildNumberMajor>
70-
<BuildNumberMinor Condition="'$(BuildNumberMinor)' == ''">0</BuildNumberMinor>
71-
<AssemblyFileVersion Condition="'$(AssemblyFileVersion)'==''">$(MajorVersion).$(MinorVersion).$(BuildNumberMajor).$(BuildNumberMinor)</AssemblyFileVersion>
72-
73-
<StabilizePackageVersion Condition="'$(StabilizePackageVersion)' == ''">false</StabilizePackageVersion>
74-
<IncludeBuildNumberInPackageVersion Condition="'$(IncludeBuildNumberInPackageVersion)' == '' and '$(StabilizePackageVersion)' != 'true'">true</IncludeBuildNumberInPackageVersion>
75-
76-
<VersionSuffix Condition="'$(IncludeBuildNumberInPackageVersion)' == 'true'">$(BuildNumberMajor)-$(BuildNumberMinor)</VersionSuffix>
77-
<NoPackageAnalysis>true</NoPackageAnalysis>
78-
</PropertyGroup>
79-
80-
<!-- SourceLink properties used by dotnet/buildtools - need to be set before importing $(ToolsDir)versioning.props -->
81-
<PropertyGroup>
82-
<UseSourceLink>true</UseSourceLink>
83-
<GitHubRepositoryName>machinelearning</GitHubRepositoryName>
84-
</PropertyGroup>
85-
86-
<!--
87-
Source code control properties used by the .NET Core SDK to inject SCC info into the NuGet package.
88-
In future versions, these will be used for SourceLink and to generate AssemblyInfo.
89-
-->
90-
<PropertyGroup>
91-
<PrivateRepositoryUrl>https://github.com/dotnet/$(GitHubRepositoryName)</PrivateRepositoryUrl>
92-
<PublishRepositoryUrl>true</PublishRepositoryUrl>
93-
<SourceRevisionId>$(LatestCommit)</SourceRevisionId>
94-
</PropertyGroup>
95-
96-
<Import Project="$(ToolsDir)versioning.props"
97-
Condition="Exists('$(ToolsDir)versioning.props') and '$(DisableImportVersioningProps)' != 'true'" />
98-
99-
<!-- Language configuration -->
100-
<PropertyGroup>
101-
<LangVersion Condition="'$(MSBuildProjectExtension)'=='.csproj'">8.0</LangVersion>
102-
<LangVersion Condition="'$(MSBuildProjectExtension)'=='.fsproj'">4.7</LangVersion>
103-
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
104-
</PropertyGroup>
105-
106-
<PropertyGroup>
107-
<RunningOnUnix Condition="'$(OS)'!='Windows_NT'">true</RunningOnUnix>
108-
</PropertyGroup>
109-
110-
<!-- Signing properties -->
111-
<PropertyGroup>
112-
<AssemblyOriginatorKeyFile Condition="'$(AssemblyOriginatorKeyFile)' == ''">$(ToolsDir)Open.snk</AssemblyOriginatorKeyFile>
113-
<SignAssembly>true</SignAssembly>
114-
<PublicSign Condition="'$(OS)' != 'Windows_NT'">true</PublicSign>
11535
</PropertyGroup>
11636

11737
<!-- Need to explicitly set these properties for the -netcoreapp3_1 or -netfx configurations becuase they are typically based off 'Debug' or 'Release' configs -->
@@ -125,4 +45,34 @@
12545
<Optimize>true</Optimize>
12646
</PropertyGroup>
12747

48+
<PropertyGroup>
49+
<!-- Optional: Publish the repository URL in the built .nupkg (in the NuSpec <Repository> element) -->
50+
<PublishRepositoryUrl>true</PublishRepositoryUrl>
51+
52+
<!-- Optional: Embed source files that are not tracked by the source control manager in the PDB -->
53+
<EmbedUntrackedSources>true</EmbedUntrackedSources>
54+
55+
<!-- Optional: Build symbol package (.snupkg) to distribute the PDB containing Source Link -->
56+
<IncludeSymbols>true</IncludeSymbols>
57+
<SymbolPackageFormat>snupkg</SymbolPackageFormat>
58+
</PropertyGroup>
59+
60+
<PropertyGroup Condition="'$(DotNetBuildFromSource)' == 'true'">
61+
<!--
62+
When building using source-build the process is:
63+
- Newtonsoft.Json versions 9.0.1 and 12.0.2 are built by source-build
64+
- Version 12.0.2 is written to Version.props
65+
- Arcade needs to use 9.0.1 so we need to override Version.props value here
66+
-->
67+
<NewtonsoftJsonVersion>9.0.1</NewtonsoftJsonVersion>
68+
</PropertyGroup>
69+
70+
<PropertyGroup Condition="'$(DotNetBuildOffline)' == 'true'">
71+
<!--
72+
Arcade has a special version prop for CodeAnalysis.CSharp in GenFacades
73+
to try to match the version loaded by msbuild. In the offline build, this
74+
is simply the source-built version.
75+
-->
76+
<MsbuildTaskMicrosoftCodeAnalysisCSharpVersion>$(MicrosoftCodeAnalysisCSharpVersion)</MsbuildTaskMicrosoftCodeAnalysisCSharpVersion>
77+
</PropertyGroup>
12878
</Project>

Diff for: ‎Directory.Build.targets

+35-47
Original file line numberDiff line numberDiff line change
@@ -1,51 +1,39 @@
1-
<Project InitialTargets="CheckForBuildTools">
2-
3-
<ItemGroup>
4-
<PackageReference Include="Microsoft.VisualStudio.Threading.Analyzers" Version="16.5.132" PrivateAssets="all" />
5-
</ItemGroup>
6-
7-
<Target Name="CheckForBuildTools">
8-
<Error Condition="!Exists('$(ToolsDir)')"
9-
Text="The tools directory [$(ToolsDir)] does not exist. Please run build in the root of the repo to ensure the tools are installed before attempting to build an individual project." />
10-
</Target>
11-
12-
<Target Name="CopyNativeAssemblies"
13-
BeforeTargets="PrepareForRun">
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<Project>
3+
<Import Project="Sdk.targets" Sdk="Microsoft.DotNet.Arcade.Sdk" />
144

5+
<Target Name="CopyNativeAssembiles" AfterTargets="CopyFilesToOutputDirectory">
156
<PropertyGroup>
16-
<LibPrefix Condition="'$(OS)' != 'Windows_NT'">lib</LibPrefix>
17-
<LibExtension Condition="'$(OS)' == 'Windows_NT'">.dll</LibExtension>
18-
<LibExtension Condition="'$(OS)' != 'Windows_NT'">.so</LibExtension>
19-
<LibExtension Condition="$([MSBuild]::IsOSPlatform('osx'))">.dylib</LibExtension>
20-
</PropertyGroup>
21-
22-
<ItemGroup>
23-
<NativeAssemblyReference>
24-
<FullAssemblyPath>$(NativeOutputPath)$(LibPrefix)%(NativeAssemblyReference.Identity)$(LibExtension)</FullAssemblyPath>
25-
</NativeAssemblyReference>
26-
</ItemGroup>
27-
28-
<Copy SourceFiles = "@(NativeAssemblyReference->'%(FullAssemblyPath)')"
29-
DestinationFolder="$(OutputPath)"
30-
OverwriteReadOnlyFiles="$(OverwriteReadOnlyFiles)"
31-
Retries="$(CopyRetryCount)"
32-
RetryDelayMilliseconds="$(CopyRetryDelayMilliseconds)"
33-
UseHardlinksIfPossible="$(CreateHardLinksForPublishFilesIfPossible)"
34-
UseSymboliclinksIfPossible="$(CreateSymbolicLinksForPublishFilesIfPossible)">
35-
<Output TaskParameter="DestinationFiles" ItemName="FileWrites"/>
36-
</Copy>
37-
7+
<LibPrefix Condition="'$(OS)' != 'Windows_NT'">lib</LibPrefix>
8+
<LibExtension Condition="'$(OS)' == 'Windows_NT'">.dll</LibExtension>
9+
<LibExtension Condition="'$(OS)' != 'Windows_NT'">.so</LibExtension>
10+
<LibExtension Condition="$([MSBuild]::IsOSPlatform('osx'))">.dylib</LibExtension>
11+
12+
<TargetArchitecture Condition="'$(Platform)' == ''">x64</TargetArchitecture>
13+
<NativeTargetArchitecture Condition="'$(NativeTargetArchitecture)' == ''">$(TargetArchitecture)</NativeTargetArchitecture>
14+
<BinDir Condition="'$(BinDir)'==''">$([MSBuild]::NormalizeDirectory('$(RepoRoot)', 'artifacts', 'bin'))</BinDir>
15+
<NativeOutputPath>$(BinDir)Native\$(NativeTargetArchitecture).$(Configuration)\</NativeOutputPath>
16+
17+
<Platform Condition="'$(Platform)'==''">AnyCPU</Platform>
18+
<PlatformConfig>$(Platform).$(Configuration)</PlatformConfig>
19+
<OutputPath Condition="'$(OutputPath)'==''">$(BinDir)$(MSBuildProjectName)\Debug</OutputPath>
20+
</PropertyGroup>
21+
22+
<ItemGroup>
23+
<NativeAssemblyReference>
24+
<FullAssemblyPath>$(NativeOutputPath)$(LibPrefix)%(NativeAssemblyReference.Identity)$(LibExtension)</FullAssemblyPath>
25+
</NativeAssemblyReference>
26+
</ItemGroup>
27+
28+
<Copy SourceFiles = "@(NativeAssemblyReference->'%(FullAssemblyPath)')"
29+
DestinationFolder="$(OutDir)"
30+
OverwriteReadOnlyFiles="$(OverwriteReadOnlyFiles)"
31+
Retries="$(CopyRetryCount)"
32+
RetryDelayMilliseconds="$(CopyRetryDelayMilliseconds)"
33+
UseHardlinksIfPossible="$(CreateHardLinksForPublishFilesIfPossible)"
34+
UseSymboliclinksIfPossible="$(CreateSymbolicLinksForPublishFilesIfPossible)">
35+
<Output TaskParameter="DestinationFiles" ItemName="FileWrites"/>
36+
</Copy>
3837
</Target>
39-
40-
<Import Project="$(ToolsDir)/versioning.targets" Condition="Exists('$(ToolsDir)/versioning.targets')" />
41-
42-
<!-- Workaround: AssemblyInfo.cs is not embedded -->
43-
<!-- https://github.com/dotnet/sourcelink/issues/572 -->
44-
<PropertyGroup>
45-
<TargetFrameworkMonikerAssemblyAttributesPath>$([System.IO.Path]::Combine('$(IntermediateOutputPath)','$(TargetFrameworkMoniker).AssemblyAttributes$(DefaultLanguageSourceExtension)'))</TargetFrameworkMonikerAssemblyAttributesPath>
46-
</PropertyGroup>
47-
<ItemGroup>
48-
<EmbeddedFiles Include="$(GeneratedAssemblyInfoFile)"/>
49-
</ItemGroup>
5038

51-
</Project>
39+
</Project>

0 commit comments

Comments
 (0)