Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(referrers): support referrers API #180

Merged
merged 47 commits into from
Feb 27, 2025
Merged
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
e3e1217
push manifest with subject
Nov 18, 2024
846e173
add unit tests
Nov 19, 2024
7fca0cb
add unit tests
Nov 19, 2024
ec93662
Merge branch 'main' into feature/pushManifestWithSubject
Nov 19, 2024
7809549
resolve merge conflicts
Nov 19, 2024
c7e4418
add tests
Nov 19, 2024
a6d552f
add comments
Nov 19, 2024
685ab70
add unit test
Nov 20, 2024
1b9ade3
add unit tests
Nov 20, 2024
5041592
resolve comments
Nov 21, 2024
c26ccb6
add SetReferrersSupportLevel func and unit tests
Nov 21, 2024
ddbf048
remove NoReferrerUpdateException and update tests accordingly
Nov 22, 2024
d6e8499
add Index constructor
Nov 24, 2024
cf431f0
add license header
Nov 26, 2024
50e696e
resolve merge conflicts
Nov 26, 2024
856c0ef
add lock on SetReferrerState
Nov 29, 2024
fcb121e
simplify ApplyReferrerChanges
Nov 29, 2024
2b24e07
resolve comments
Dec 3, 2024
7df36ca
Merge branch 'main' into feature/pushManifestWithSubject
Dec 3, 2024
c30d6b2
resolve comments
Dec 13, 2024
b20a202
resolve comments
Dec 20, 2024
07d8e06
resolve comments
Dec 23, 2024
27382ae
resolve comments
Dec 23, 2024
b858999
Merge branch 'main' into feature/pushManifestWithSubject
Dec 23, 2024
404337e
resolve comments
Dec 23, 2024
1041bc0
resolve comments
Dec 23, 2024
85435cc
list referrers
Jan 10, 2025
cd05c4b
fix merge conflict
Jan 12, 2025
e17c50c
add unit tests
Jan 13, 2025
db582c2
add unit tests
Jan 15, 2025
966b2ad
add comments
Feb 3, 2025
037b092
format codes
Feb 3, 2025
71fcfcb
add headers
Feb 3, 2025
dfb1291
Merge branch 'main' into feature/listReferrers
Feb 5, 2025
447a280
add invalidResponseException tests
Feb 5, 2025
f668c22
Merge branch 'main' into feature/listReferrers
Feb 10, 2025
7f0553f
Merge branch 'main' into feature/listReferrers
Feb 18, 2025
7724d7f
address comments
Feb 19, 2025
9f1cc15
address comments
Feb 19, 2025
c64338f
verify returned manifest
Feb 24, 2025
9880d45
change callback function to return IAsyncEnumerable
Feb 25, 2025
703df70
resolve conflicts
Feb 25, 2025
b923532
resolve comments
Feb 26, 2025
5a51f41
resolve comments
Feb 26, 2025
b98b303
resolve comments
Feb 26, 2025
8c7b38e
resolve comments
Feb 26, 2025
28d3be2
resolve comments
Feb 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
resolve comments
Signed-off-by: Patrick Pan <panjiaxuan@microsoft.com>
Patrick Pan committed Dec 3, 2024
commit 2b24e071f95ff3c0f741dce50cb51b517d949f7a
2 changes: 1 addition & 1 deletion src/OrasProject.Oras/Oci/Descriptor.cs
Original file line number Diff line number Diff line change
@@ -72,7 +72,7 @@ public static Descriptor Create(Span<byte> data, string mediaType)

internal BasicDescriptor BasicDescriptor => new BasicDescriptor(MediaType, Digest, Size);

internal static bool IsEmptyOrNull(Descriptor? descriptor)
internal static bool IsEmptyOrInvalid(Descriptor? descriptor)
{
return descriptor == null || descriptor.Size == 0 || string.IsNullOrEmpty(descriptor.Digest) || string.IsNullOrEmpty(descriptor.MediaType);
}
Original file line number Diff line number Diff line change
@@ -107,7 +107,7 @@ public static void VerifyContentDigest(this HttpResponseMessage response, string
/// </summary>
/// <param name="response"></param>
/// <param name="repository"></param>
public static void CheckOCISubjectHeader(this HttpResponseMessage response, Repository repository)
internal static void CheckOCISubjectHeader(this HttpResponseMessage response, Repository repository)
{
if (response.Headers.TryGetValues("OCI-Subject", out var values))
{
14 changes: 7 additions & 7 deletions src/OrasProject.Oras/Registry/Remote/ManifestStore.cs
Original file line number Diff line number Diff line change
@@ -28,7 +28,7 @@

public class ManifestStore(Repository repository) : IManifestStore
{
public Repository Repository { get; init; } = repository;

Check warning on line 31 in src/OrasProject.Oras/Registry/Remote/ManifestStore.cs

GitHub Actions / Analyze (8.0.x)

Parameter 'Repository repository' is captured into the state of the enclosing type and its value is also used to initialize a field, property, or event.

Check warning on line 31 in src/OrasProject.Oras/Registry/Remote/ManifestStore.cs

GitHub Actions / Analyze (8.0.x)

Parameter 'Repository repository' is captured into the state of the enclosing type and its value is also used to initialize a field, property, or event.

Check warning on line 31 in src/OrasProject.Oras/Registry/Remote/ManifestStore.cs

GitHub Actions / build (8.0.x)

Parameter 'Repository repository' is captured into the state of the enclosing type and its value is also used to initialize a field, property, or event.

/// <summary>
/// Fetches the content identified by the descriptor.
@@ -184,7 +184,7 @@
return;
}

var contentBytes = await content.ReadAllAsync(expected, cancellationToken);
var contentBytes = await content.ReadAllAsync(expected, cancellationToken).ConfigureAwait(false);
using (var contentDuplicate = new MemoryStream(contentBytes))
{
// Push the manifest when ReferrerState is Unknown or NotSupported
@@ -202,11 +202,11 @@
// 1. Index the referrers list using referrers tag schema when manifest contains a subject field
// And the ReferrerState is not supported
// 2. Or do nothing when the manifest does not contain a subject field when ReferrerState is not supported/unknown
await ProcessReferrersAndPushIndex(expected, contentDuplicate, cancellationToken);
await ProcessReferrersAndPushIndex(expected, contentDuplicate, cancellationToken).ConfigureAwait(false);
}
break;
default:
await DoPushAsync(expected, content, reference, cancellationToken);
await DoPushAsync(expected, content, reference, cancellationToken).ConfigureAwait(false);
break;
}
}
@@ -241,11 +241,11 @@
desc.Annotations = imageManifest.Annotations;
break;
default:
return;

Check warning on line 244 in src/OrasProject.Oras/Registry/Remote/ManifestStore.cs

Codecov / codecov/patch

src/OrasProject.Oras/Registry/Remote/ManifestStore.cs#L244

Added line #L244 was not covered by tests
}

Repository.SetReferrersState(Referrers.ReferrersState.ReferrersNotSupported);
await UpdateReferrersIndex(subject, new Referrers.ReferrerChange(desc, Referrers.ReferrerOperation.ReferrerAdd), cancellationToken);
await UpdateReferrersIndex(subject, new Referrers.ReferrerChange(desc, Referrers.ReferrerOperation.ReferrerAdd), cancellationToken).ConfigureAwait(false);
}

/// <summary>
@@ -265,7 +265,7 @@
{
// 1. pull the original referrers index list using referrers tag schema
var referrersTag = Referrers.BuildReferrersTag(subject);
var (oldDesc, oldReferrers) = await PullReferrersIndexList(referrersTag, cancellationToken);
var (oldDesc, oldReferrers) = await PullReferrersIndexList(referrersTag, cancellationToken).ConfigureAwait(false);

// 2. apply the referrer change to referrers list
var (updatedReferrers, updateRequired) =
@@ -287,7 +287,7 @@
}
}

if (repository.Options.SkipReferrersGC || Descriptor.IsEmptyOrNull(oldDesc))
if (repository.Options.SkipReferrersGC || Descriptor.IsEmptyOrInvalid(oldDesc))
{
// Skip the delete process if SkipReferrersGC is set to true or the old Descriptor is empty or null
return;
@@ -310,11 +310,11 @@
{
try
{
var (desc, content) = await FetchAsync(referrersTag, cancellationToken);
var (desc, content) = await FetchAsync(referrersTag, cancellationToken).ConfigureAwait(false);
var index = JsonSerializer.Deserialize<Index>(content);
if (index == null)
{
throw new JsonException("null index manifests list");

Check warning on line 317 in src/OrasProject.Oras/Registry/Remote/ManifestStore.cs

Codecov / codecov/patch

src/OrasProject.Oras/Registry/Remote/ManifestStore.cs#L316-L317

Added lines #L316 - L317 were not covered by tests
}
return (desc, index.Manifests);
}
24 changes: 14 additions & 10 deletions src/OrasProject.Oras/Registry/Remote/Referrers.cs
Original file line number Diff line number Diff line change
@@ -52,6 +52,11 @@
/// <returns>The updated referrers list, updateRequired</returns>
internal static (IList<Descriptor>, bool) ApplyReferrerChanges(IList<Descriptor> oldReferrers, ReferrerChange referrerChange)
{
if (Descriptor.IsEmptyOrInvalid(referrerChange.Referrer))
{
return (oldReferrers, false);
}

// updatedReferrers is a list to store the updated referrers
var updatedReferrers = new List<Descriptor>();
// updatedReferrersSet is a HashSet to store unique referrers
@@ -60,7 +65,7 @@
var updateRequired = false;
foreach (var oldReferrer in oldReferrers)
{
if (Descriptor.IsEmptyOrNull(oldReferrer))
if (Descriptor.IsEmptyOrInvalid(oldReferrer))
{
// Skip any empty or null referrers
updateRequired = true;
@@ -84,19 +89,18 @@
updatedReferrersSet.Add(basicDesc);
}

if (!Descriptor.IsEmptyOrNull(referrerChange.Referrer))

var basicReferrerDesc = referrerChange.Referrer.BasicDescriptor;
if (referrerChange.ReferrerOperation == ReferrerOperation.ReferrerAdd)
{
var basicDesc = referrerChange.Referrer.BasicDescriptor;
if (referrerChange.ReferrerOperation == ReferrerOperation.ReferrerAdd)
if (!updatedReferrersSet.Contains(basicReferrerDesc))
{
if (!updatedReferrersSet.Contains(basicDesc))
{
// Add the new referrer only when it has not already existed in the updatedReferrersSet
updatedReferrers.Add(referrerChange.Referrer);
updatedReferrersSet.Add(basicDesc);
}
// Add the new referrer only when it has not already existed in the updatedReferrersSet
updatedReferrers.Add(referrerChange.Referrer);
updatedReferrersSet.Add(basicReferrerDesc);
}
}


// Skip unnecessary update
if (!updateRequired && updatedReferrersSet.Count == oldReferrers.Count)
@@ -106,9 +110,9 @@
{
var basicDesc = oldReferrer.BasicDescriptor;
if (!updatedReferrersSet.Contains(basicDesc))
{
updateRequired = true;
break;

Check warning on line 115 in src/OrasProject.Oras/Registry/Remote/Referrers.cs

Codecov / codecov/patch

src/OrasProject.Oras/Registry/Remote/Referrers.cs#L113-L115

Added lines #L113 - L115 were not covered by tests
}
}

@@ -116,7 +120,7 @@
{
return (updatedReferrers, false);
}
}

Check warning on line 123 in src/OrasProject.Oras/Registry/Remote/Referrers.cs

Codecov / codecov/patch

src/OrasProject.Oras/Registry/Remote/Referrers.cs#L123

Added line #L123 was not covered by tests
return (updatedReferrers, true);
}
}
2 changes: 1 addition & 1 deletion tests/OrasProject.Oras.Tests/Remote/ManifestStoreTest.cs
Original file line number Diff line number Diff line change
@@ -100,7 +100,7 @@
var cancellationToken = new CancellationToken();
var store = new ManifestStore(repo);
var (receivedDesc, receivedManifests) = await store.PullReferrersIndexList("test", cancellationToken);
Assert.True(Descriptor.IsEmptyOrNull(receivedDesc));
Assert.True(Descriptor.IsEmptyOrInvalid(receivedDesc));
Assert.Empty(receivedManifests);
}

@@ -306,7 +306,7 @@

byte[]? receivedManifestContent = null;
byte[]? receivedIndexContent = null;
var referrersTag = Referrers.BuildReferrersTag(firstExpectedManifest.Subject);

Check warning on line 309 in tests/OrasProject.Oras.Tests/Remote/ManifestStoreTest.cs

GitHub Actions / Analyze (8.0.x)

Possible null reference argument for parameter 'descriptor' in 'string Referrers.BuildReferrersTag(Descriptor descriptor)'.

Check warning on line 309 in tests/OrasProject.Oras.Tests/Remote/ManifestStoreTest.cs

GitHub Actions / Analyze (8.0.x)

Possible null reference argument for parameter 'descriptor' in 'string Referrers.BuildReferrersTag(Descriptor descriptor)'.

Check warning on line 309 in tests/OrasProject.Oras.Tests/Remote/ManifestStoreTest.cs

GitHub Actions / build (8.0.x)

Possible null reference argument for parameter 'descriptor' in 'string Referrers.BuildReferrersTag(Descriptor descriptor)'.
var oldIndexDeleted = false;
var firstIndexDeleted = false;
var mockHttpRequestHandler = async (HttpRequestMessage req, CancellationToken cancellationToken) =>
@@ -504,7 +504,7 @@
};

byte[]? receivedManifestContent = null;
var referrersTag = Referrers.BuildReferrersTag(expectedManifest.Subject);

Check warning on line 507 in tests/OrasProject.Oras.Tests/Remote/ManifestStoreTest.cs

GitHub Actions / Analyze (8.0.x)

Possible null reference argument for parameter 'descriptor' in 'string Referrers.BuildReferrersTag(Descriptor descriptor)'.

Check warning on line 507 in tests/OrasProject.Oras.Tests/Remote/ManifestStoreTest.cs

GitHub Actions / Analyze (8.0.x)

Possible null reference argument for parameter 'descriptor' in 'string Referrers.BuildReferrersTag(Descriptor descriptor)'.

Check warning on line 507 in tests/OrasProject.Oras.Tests/Remote/ManifestStoreTest.cs

GitHub Actions / build (8.0.x)

Possible null reference argument for parameter 'descriptor' in 'string Referrers.BuildReferrersTag(Descriptor descriptor)'.

var mockHttpRequestHandler = async (HttpRequestMessage req, CancellationToken cancellationToken) =>
{

Unchanged files with check annotations Beta

public class BlobStore(Repository repository) : IBlobStore, IMounter
{
public Repository Repository { get; init; } = repository;

Check warning on line 30 in src/OrasProject.Oras/Registry/Remote/BlobStore.cs

GitHub Actions / Analyze (8.0.x)

Parameter 'Repository repository' is captured into the state of the enclosing type and its value is also used to initialize a field, property, or event.

Check warning on line 30 in src/OrasProject.Oras/Registry/Remote/BlobStore.cs

GitHub Actions / Analyze (8.0.x)

Parameter 'Repository repository' is captured into the state of the enclosing type and its value is also used to initialize a field, property, or event.

Check warning on line 30 in src/OrasProject.Oras/Registry/Remote/BlobStore.cs

GitHub Actions / build (8.0.x)

Parameter 'Repository repository' is captured into the state of the enclosing type and its value is also used to initialize a field, property, or event.
public async Task<Stream> FetchAsync(Descriptor target, CancellationToken cancellationToken = default)
{
Referrers.ReferrerOperation.ReferrerAdd
);
var (updatedReferrers, updateRequired) = Referrers.ApplyReferrerChanges(oldReferrers, referrerChange);

Check warning on line 251 in tests/OrasProject.Oras.Tests/Remote/ReferrersTest.cs

GitHub Actions / Analyze (8.0.x)

Argument of type 'List<Descriptor?>' cannot be used for parameter 'oldReferrers' of type 'IList<Descriptor>' in '(IList<Descriptor>, bool) Referrers.ApplyReferrerChanges(IList<Descriptor> oldReferrers, ReferrerChange referrerChange)' due to differences in the nullability of reference types.

Check warning on line 251 in tests/OrasProject.Oras.Tests/Remote/ReferrersTest.cs

GitHub Actions / Analyze (8.0.x)

Argument of type 'List<Descriptor?>' cannot be used for parameter 'oldReferrers' of type 'IList<Descriptor>' in '(IList<Descriptor>, bool) Referrers.ApplyReferrerChanges(IList<Descriptor> oldReferrers, ReferrerChange referrerChange)' due to differences in the nullability of reference types.

Check warning on line 251 in tests/OrasProject.Oras.Tests/Remote/ReferrersTest.cs

GitHub Actions / build (8.0.x)

Argument of type 'List<Descriptor?>' cannot be used for parameter 'oldReferrers' of type 'IList<Descriptor>' in '(IList<Descriptor>, bool) Referrers.ApplyReferrerChanges(IList<Descriptor> oldReferrers, ReferrerChange referrerChange)' due to differences in the nullability of reference types.
Assert.Single(updatedReferrers);
for (var i = 0; i < updatedReferrers.Count; ++i)
{
internal Referrers.ReferrersState ReferrersState
{
get => (Referrers.ReferrersState) _referrersState;
private set => _referrersState = (int) value;

Check warning on line 55 in src/OrasProject.Oras/Registry/Remote/Repository.cs

Codecov / codecov/patch

src/OrasProject.Oras/Registry/Remote/Repository.cs#L55

Added line #L55 was not covered by tests
}
internal static readonly string[] DefaultManifestMediaTypes =