Designing a build system for .NET solutions with Cake - 1: The build submodule & Versioning dependencies

There is no Cake in this issue, however we'll set the scene for things to come.

Contents

This is the first issue in a series around designing a build system for .NET projects

1. The build submodule & Versioning dependencies
2. Build, test and pack .NET projects by convention using Cake
3. Build and pack .NET applications ready for deployment
4. Introducing build configuration options and smart defaults
5. Versioning strategy using GitVersion and auto-tagging from your CI build
6. Templating new .NET solutions from our build submodule

The build submodule

Whilst GIT submodules can often be tricky (the amount of times I've ended up in a disconnected HEAD situation...), but they are also a really powerful concept. The idea is that you can embedded a pointer to another GIT repository, in another.

So, to get started, we need two repositories - the build submodule, and the parent solution (that will contain our code).

1. build - https://github.com/Antaris/build
2. The parent repo - https://github.com/Antaris/example-repo-with-build-system

So, the first steps, we need to initialise something into the build submodule, so let's provide a simple README:

# `build` submodule

An  example build system, implemented using Cake and MSBuild.

After we commit this we need to add this as a submodule for our parent repo:

git submodule add git@github.com:Antaris/build.git build
git submodule update --init

We can see that the submodule materialises as a local folder under our repo. The benefits of this, is we can update our build submodule entirely independently of the parent repository. When we need to 'update' the submodule, its simply a git pull within the build submodule itself.

What is committed to the parent repository, is a pointer to a specific commit in the submodule.

An important thing to consider, is the build submodule is a GIT repo in itself, which means you can branch, push, pull, rebase, all of the standard operations you can perform in GIT. This also means you need to make sure you commit your changes (and push) in your submodule before commiting changes to your parent repo - otherwise the commit for the build submodule will not change.

Next, let's figure out how we're going to manage our dependencies.

For the purpose of the blog series going forward, I will make all of my changes to the build GIT repo, from the context of the parent repo where I have added the submodule.

Versioning dependencies

.NET has long now had a solution for dependencies - NuGet, and this has evolved in the last couple of years to become more streamlined. What started out as an external solution (using packages.config and a manual NuGet restore), it has become integrated into the MSBuild toolchain. Gone is the need to maintain packages.config files - now we can utilise the new <PackageReference /> project element to express our dependencies. The important thing to remember at this point, is that dependencies are expressed at the project level.

NuGet does not currently have a solution (ahem) for expressing dependencies at the solution-level. Sure, Visual Studio provides UI for managing versions (and the Consolidate tab is very useful), but it still comes down to individual <PackageReference /> elements for each project, each with their own (potentially different) versions. i.e., I could quite easily get into a situation where I depend on <PackageReference Include="Newtonsoft.Json" Version="10.0.0" /> and <PackageReference Include="Newtonsoft.Json" Version="12.0.3" />.

One of the design goals of Paket was to provide solution-level tooling for managing dependency versions. We can utilise this same concept to centralise our dependency versions across our repo.

For our build submodule approach, we're going to first define a new MSBuild file, dependencies.targets where we will define our dependencies.

You can name this dependencies.props or dependencies.targets, however the convention is that props are imported first, and targets are imported last - our dependencies need to be imported after they are defined as part of an MSBuild project

<Project>
    <ItemGroup Label="Third Party">
        <PackageReference Update="Newtonsoft.Json" Version="12.0.3" />
    </ItemGroup>
</Project>

I prefer to group my dependencies based on provider, namely Microsoft packages I keep separate from other third party packages.

You'll hopefully notice something different here, the <PackageReference /> contains an Update and not an Include attribute. This tells MSBuild to update the attributes of a given node, in this case any package references where we Include="Newtonsoft.Json". If your project does not contain a reference to this package, this operation is a noop.

We need to create one more file before we create a new .NET project. This file is called Directory.Build.targets, and it will sit at the root of your parent repository. Directory.Build.(props|targets) are special files to MSBuild - it will detect and automatically import the closest instance to where your .NET project is defined. Having this file at the root means it will be automatically imported into your project.

If you are nesting Directory.Build.(props|targets), you will need to import a parent instance of the file, as MSBuild only imports the closest file.

Within this file, we need to import our dependencies.targets file:

<Project>
    <Import Project=".\build\dependencies.targets" />
</Project>

Now we have the basics of the versioning elements complete, we can add a .NET project and test that our references are resolved.

mkdir src
cd src
dotnet new classlib --name ExampleLib;

cd ../
dotnet new sln --name Example;
dotnet sln add ../src/ExampleLib/ExampleLib.csproj

With the above shell commands, we're creating a project structure as follows:

/Example.sln
/src/ExampleLib/ExampleLib.csproj

So, let's go ahead an edit the project file to include a reference to Newtonsoft.Json:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup Label="Packages">
    <PackageReference Include="Newtonsoft.Json" />
  </ItemGroup>

</Project>

We can now run a dotnet restore on either the project or the solution and it should happily restore the specific version of Newtonsoft.Json: 12.0.3. If you load this solution up in Visual Studio, the IDE should confirm this:

Troubleshooting

If the restore action gives you the following error, or similar to it, it means that the <PackageReference Update="Newtonsoft.Json" Version="12.0.3" /> node was not being picked up by MSBuild.

Project dependency Newtonsoft.Json does not contain an inclusive lower bound. Include a lower bound in the dependency version to ensure consistent restore results.

Package 'Newtonsoft.Json 3.5.8' was restored using '.NETFramework,Version=v4.6.1, .NETFramework,Version=v4.6.2, .NETFramework,Version=v4.7, .NETFramework,Version=v4.7.1, .NETFramework,Version=v4.7.2, .NETFramework,Version=v4.8' instead of the project target framework '.NETStandard,Version=v2.0'. This package may not be fully compatible with your project.

You'll need to check a few things:

• Package name is spelled correctly in build.dependencies and your project file
• You are correctly referencing build\dependencies.targets from the root Directory.Build.targets file
• Your root Directory.Build.targets file is being imported into your MSBuild project correctly.

The MSBuild Log Viewer is a great tool for debugging issues with MSBuild projects.

Committing our changes

Lastly, we need to commit our changes to both the build submodule, and also our parent repository.  Let's run a git status in our parent repo and see what has changed:

And within the build submodule:

Let's handle the build submodule changes first:

cd build
git add .\dependencies.targets
git commit -m 'Added dependencies.targets'
git push

Now we can commit our changes to our parent repo, but first, we probably should create a .gitignore in our parent repo so we can filter out some noise. Head over to gitignore.io or perhaps use this preconfigured one.

Let's stage our changes and commit:

cd ../
git add -A
git commit -m 'Added Example project and solution'
git push

Takeaways and next steps

There was a fair amount to go through in this issue. Let's recap:

• We created two repositories, one which will be our parent, and another which will be our submodule.
• We used GIT's git submodule add command to include the build GIT repository within our parent repository.
• We added a file, build\dependencies.targets which is an MSBuild file containing a list of <PackageReference /> nodes which Update instead of Include
• We added a file, Directory.Build.targets at the root of our parent repository which will be automatically imported by MSBuild projects
• We created a new .NET Core class library and a solution file. Within that project file, we added a <PackageReference Include="Newtonsoft.Json" /> with no Version attribute.
• We performed a dotnet restore and it successfully restored our packages with the correct version.

Next up in the series, we'll add the Cake build framework to our build submodule, and have it building, testing and packing our code by convention.