Skip to content

Commit b431045

Browse files
crummelcrummel
authored
Add guidance for repo owners (#2726)
* Add guidance for new dependencies * Instructions for upating dependencies. * Guidance for new features * Add general build info guidance. * Add index. * Add onboarding new repo documentation * Address code review comments. * Markdown linter clean.
1 parent 6f20c55 commit b431045

File tree

7 files changed

+611
-0
lines changed

7 files changed

+611
-0
lines changed

‎Documentation/sourcebuild-in-repos/README.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Repo owner's handbook for source-build
2+
3+
These docs aim to be a collection of useful information for repo owners who work
4+
with and participate in source-build.
5+
6+
## Index
7+
8+
* [Adding new dependencies](new-dependencies.md)
9+
* [Updating dependencies](update-dependencies.md)
10+
* [Considerations when adding features](adding-features.md)
11+
* [General notes on the source-build build](build-info.md)
12+
* [Adding new repositories](new-repo.md)

‎Documentation/sourcebuild-in-repos/adding-features.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# New features
2+
3+
New features are great! They also have source-build considerations though.
4+
.NET is no longer just multi-platform but also multi-distribution: there's
5+
a Linux SDK produced by Microsoft, but there's also a Linux SDK produced
6+
by Red Hat, one produced by Fedora, and one that anyone in the community
7+
can build himself from source.
8+
9+
## Things to consider
10+
11+
New features, or expansions of current features, should act sensibly
12+
across Windows, Linux, and OSX. This also involves taking into account
13+
the limitations and conventions of the different platforms - for instance,
14+
on Linux, it's typical for the .NET SDK to be installed by root and
15+
therefore be unchangeable by normal users, so installing global tools
16+
needs to take into account the fact that the user might not be able to
17+
add anything to the SDK directories (see
18+
[this docker global tools issue](https://github.com/dotnet/dotnet-docker/issues/520)
19+
and [a similar installer issue](https://github.com/dotnet/installer/issues/7069)).
20+
21+
New features also need to be compatible across all distributions of
22+
.NET - for instance, Fedora and Debian cannot distribute *any*
23+
non-open-source code, even samples, tests, or minor additions to
24+
licenses like restricting the field that the code can be used in.
25+
This includes any code or packages that are used to build product
26+
code as well. Microsoft generally prefers the MIT license.
27+
28+
One example of a licensing issue that source-build found was with
29+
dotnet-check and dotnet-format: these tools were brought into the
30+
product build but had a dependency that was not licensed compatibly -
31+
even without the source-build aspect. We had to scramble to replace
32+
the dependency before the release. Dependencies need to be carefully
33+
checked for license compatibility as well.
34+
35+
## Resources
36+
37+
Fedora's approved open-source licenses can be found
38+
[on their wiki](https://fedoraproject.org/wiki/Licensing:Main#Good_Licenses),
39+
or you can check on the [OSI-approved list of licenses](https://opensource.org/licenses/alphabetical).
40+
41+
If you would like some of our distribution maintainer partners to
42+
review your new feature you can ping @dotnet/distro-maintainers
43+
on a PR or issue.

‎Documentation/sourcebuild-in-repos/build-info.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Source-build build info
2+
3+
This is a collection of notes about how source-build can differ in general
4+
from your repo's build and what kind of issues that can create.
5+
6+
## Single-version and single-RID build
7+
8+
Source-build is required to build on a single machine with no internet
9+
access. This means that we build targeting a single RID, usually the
10+
non-portable RID for the build machines (like rhel.7-x64). We do
11+
support building portable (linux-x64) as well - this is useful for
12+
bootstrapping new distributions.
13+
14+
Source-build cannot build with any *prebuilts*. This is our term for
15+
any package that comes from *outside the current source-build*. This means
16+
that everything that ships out of source-build and everything that is used to
17+
build those shipping products must come from the source-build in progress.
18+
Packages from nuget.org, Microsoft builds, or other unrelated source-builds
19+
cannot be used in source-build except in a limited bootstrapping process.
20+
21+
Source-build supplies a *previously-source-built* set of packages for this
22+
bootstrapping process. This is one way we have of breaking cycles in the
23+
build. However, none of these packages can make it to the final build output.
24+
This also means that your repo should be buildable with the immediately
25+
previous version of the SDK than you are building for; i.e., if you are
26+
building for 6.0.103, everything should be buildable with the 6.0.102 SDK.
27+
28+
We also only build one version of each repo. This means that if your repo
29+
turns production of some packages on and off, for instance, if you only
30+
produce packages if they are changed, source-build will need a workaround
31+
to force all packages to be produced. Additionally, we can only supply
32+
one version of each package to a repo. This is injected into the
33+
`$({PackageName}PackageVersion)` variables, e.g. SystemReflectionMetadataPackageVersion.
34+
One exception is reference-only packages -
35+
[dotnet/source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages)
36+
produces multiple versions and these can be hard-coded or use a
37+
`$(<PackageName>ReferenceVersion)` property or similar if you don't
38+
need source-build to change them.
39+
40+
## Platform-specific packages
41+
42+
Packages that require components or packages only available on some other
43+
operating system than the building OS cannot be built in source-build.
44+
These should use `<ExcludeFromSourceBuild>true</ExcludeFromSourceBuild>` or
45+
other options to be excluded from the source-build. For instance, if a
46+
project depends on a package that can only be built on Windows, it will need
47+
to be disabled or worked around in source-build. As an example,
48+
[Roslyn removes](https://github.com/dotnet/roslyn/blob/b999a65c8b0feeccb2b58da3d7a6e80e5f08feab/src/Workspaces/Core/Portable/Storage/PersistentStorageExtensions.cs#L23)
49+
a small performance improvement when building for source-build because it
50+
requires a component that isn't available.

‎Documentation/sourcebuild-in-repos/img/ci-job.png‎

42.2 KB
Loading

‎Documentation/sourcebuild-in-repos/new-dependencies.md‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
# Adding a new dependency
2+
3+
## Basics
4+
5+
When adding a new dependency, there are a few steps common between any type
6+
of dependency. If you use darc, `darc add-dependency` will take care of
7+
this process for you. Otherwise:
8+
9+
1. Add the package and a default version to eng/Versions.props. The default
10+
version property should be named `<PackageName>PackageVersion`,
11+
e.g. `SystemCollectionsImmutablePackageVersion`.
12+
1. Add the package and a repo reference to eng/Version.Details.xml. This
13+
should include a SourceBuild metadata entry, looking something like:
14+
15+
```xml
16+
<Dependency Name="Microsoft.Net.Compilers.Toolset"
17+
Version="4.2.0-3.22205.5"
18+
CoherentParentDependency="Microsoft.NET.Sdk">
19+
<Uri>https://github.com/dotnet/roslyn</Uri>
20+
<Sha>0167599e0e1634ea3ed8d0e41390a3c0d9b3e4e9</Sha>
21+
<SourceBuild RepoName="roslyn" ManagedOnly="true" />
22+
</Dependency>
23+
```
24+
25+
1. Set up dependency flow from the foreign repo to your repo.
26+
27+
In general, you should aim to use one version of each package. If you are
28+
using a package as reference-only, it is possible to use multiple versions,
29+
but only one implementation version of each package will be used -
30+
source-build will override it to the version that is being built in this
31+
version of the SDK.
32+
33+
The source-build metadata is important - this tells source-build which repo
34+
package contains the specific nupkg you want.
35+
36+
Another uncommon case that can cause problems is when the repo `<Dependency>`
37+
version and the source-built intermediate package version don't match.
38+
In this case, a direct dependency should be added to the source-build
39+
intermediate package and the CoherentParentDependency attribute should be
40+
set to the repo that consumes the dependency. For an example,
41+
see [installer's F# dependency](https://github.com/dotnet/installer/blob/ba1739a2363b1062f03ea386ec67174c6468d3b2/eng/Version.Details.xml#L128).
42+
You can find the version needed by running `darc get-build`
43+
or using [BAR](https://aka.ms/bar).
44+
45+
## Adding dependencies
46+
47+
Source build classifies dependencies in the following ways
48+
49+
1. .NET - a dependency on a component from the .NET org - e.g. dotnet/runtime
50+
1. Microsoft - a dependency on a component from the Microsoft org - e.g. microsoft/vstest
51+
1. External - a dependency on a component outside of Microsoft/.NET - e.g. JamesNK/Newtonsoft.Json
52+
53+
The following checklist can be used to determine how to handle each type of
54+
dependency and the nuances it may have.
55+
56+
1. Are you already using a package from the same repo and is the new
57+
dependency already source-built? You can determine this by checking if
58+
the repo has a SourceBuild leg building, e.g. runtime's is
59+
`runtime-dev-innerloop (Build Linux x64 release SourceBuild)`.
60+
61+
1. This is the simplest case. You can add new dependencies from repos
62+
you are already using freely. Note that you need to use the same version
63+
for all packages from the repo.
64+
1. Is the repo already built in source-build including the specific
65+
package you want?
66+
1. Add the dependency using `darc add-dependency` as normal, then
67+
add the [source-build metadata](#Basics) as above.
68+
1. Is the repo already built in source-build but the specific package is not?
69+
1. There's probably an issue with source-building this package. Please
70+
talk to a [source-build team member](https://github.com/orgs/dotnet/teams/source-build-internal)
71+
about why that is and whether we can fix it.
72+
1. Is this a repo that uses Arcade to build?
73+
1. Does the foreign repo depend on your repo, directly or indirectly?
74+
i.e. would adding the dependency create a cycle?
75+
1. This isn't necessarily a deal-breaker - it can sometimes be worked
76+
around with reference-only packages. Please contact a
77+
[source-build team member](https://github.com/orgs/dotnet/teams/source-build-internal)
78+
to discuss.
79+
1. Does the foreign repo publish to BAR?
80+
1. If not, please contact them to get them publishing to BAR
81+
in an appropriate channel.
82+
1. If neither of these caveats apply you should be in good shape.
83+
Follow the instructions under "Basics" above.
84+
1. Dependencies that have no code (e.g. SDKs with just props and targets) can
85+
usually be added using the [source-build text-only-package process](https://github.com/dotnet/source-build-reference-packages/tree/main/src/textOnlyPackages/src).
86+
If the package is not already included there, you can [open a PR](https://github.com/dotnet/source-build-reference-packages/pulls)
87+
or [file an issue](https://github.com/dotnet/source-build/issues/new/choose)
88+
to include it.
89+
1. Source-build has in the past used Arcade shims to allow non-Arcade repos
90+
to build appropriate packages for source-build. Please
91+
[log an issue](https://github.com/dotnet/source-build/issues/new/choose)
92+
to determine if this is a workable approach for your foreign repo.
93+
1. We build some external dependencies in the [dotnet/source-build-externals](https://github.com/dotnet/source-build-externals)
94+
repo. Good targets for this generally have very few if any dependencies and
95+
very simple build processes.
96+
[Please log an issue](https://github.com/dotnet/source-build/issues/new/choose)
97+
to get the process started.
98+
99+
## Deciding what version to use
100+
101+
- If you are using the package as reference-only and want the version to be
102+
pinned, use a literal version number in the csprojs that reference the
103+
project. You can also set up a reference-only package version variable
104+
in eng/Versions.props, for instance `<PackageNameReferenceOnly>1.2.3</PackageNameReferenceOnly>`
105+
in addition to `<PackageNamePackageVersion>4.5.6</PackageNamePackageVersion>`.
106+
Also verify that the package is available in
107+
[source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages),
108+
and if not, [file a source-build issue](https://github.com/dotnet/source-build/issues).
109+
- If you are using the package in the actual build or want the version
110+
to be updated whenever the foreign repo publishes to your channel, use
111+
the version number property set up in eng/Versions.props. When performing
112+
a source-build, the version number will get updated to the current
113+
source-built version.
114+
- If you are using an external or non-Arcade package, please coordinate as
115+
much as possible with other teams using that package. Each
116+
package-version is essentially maintained as a separate concern, so
117+
something like repo A requiring Newtonsoft.Json 9.0.1 and repo B requiring
118+
12.0.2 essentially doubles the source-build team's work.

0 commit comments

Comments
 (0)