prefetching dependencies refactoring #200

slimreaper35 · 2024-12-19T13:54:18Z

No description provided.

github-actions · 2024-12-19T13:55:38Z

🚀 Preview is available at https://pr-200--konflux-docs.netlify.app

brunoapimentel · 2024-12-19T19:19:39Z

I think we could greatly simplify this page if we changed the structure so we have a single initial section describing how to se the prefetch-input parameter?, instead of having it repead for each individual package manager. There are some sections that are important to be kept, IMO:

Configuring your pip project to build from source (how to generate a requirements-build.txt)
Prefetching pip dependencies from custom index servers
Generating the rpms.lock.yml file for RPM prefetch
Creating a artifacts.lock.yaml file for generic artifact prefetching

All the rest, can probably be inferred and is described in Cachi2's README.

github-actions · 2025-01-03T09:22:10Z

🚀 Preview is available at https://pr-200--konflux-docs.netlify.app

github-actions · 2025-01-03T09:35:24Z

🚀 Preview is available at https://pr-200--konflux-docs.netlify.app

github-actions · 2025-01-03T09:41:23Z

🚀 Preview is available at https://pr-200--konflux-docs.netlify.app

github-actions · 2025-01-03T10:02:58Z

🚀 Preview is available at https://pr-200--konflux-docs.netlify.app

github-actions · 2025-01-03T15:48:33Z

🚀 Preview is available at https://pr-200--konflux-docs.netlify.app

github-actions · 2025-01-03T15:54:15Z

🚀 Preview is available at https://pr-200--konflux-docs.netlify.app

github-actions · 2025-01-08T16:05:42Z

🚀 Preview is available at https://pr-200--konflux-docs.netlify.app

arewm · 2025-01-08T16:10:06Z

docs/modules/ROOT/pages/how-tos/configuring/prefetching-dependencies.adoc

+. Configure the hermetic pipeline by adding the following parameters in both `.yaml` files:
+
+
+[source,yaml]
+----
+pipelineSpec:
+  params:
+    ...
+    - name: hermetic
+      type: string
+      description: Execute the build with network isolation
+      default: "true"
+----


This belongs on the hermetic page, but we can refer to that page.

This was originally split into two pages because each can be done independently. We can still increase the accuracy of the SBOM by prefetching even if the build isn't done in a hermetic mode, right?

arewm · 2025-01-08T16:12:24Z

docs/modules/ROOT/pages/how-tos/configuring/prefetching-dependencies.adoc

+      value: '{"type": "<package_manager>", "path": "."}' <1>
+----
+NOTE: The prefetch-input parameter specifies the path to the directory of the project. In this example, the `.` indicates the repository root. If you have multiple directories, you can provide the path to those directories in the JSON array format: `[{"type": "<package_manager>", "path": "."}, {"type": "<package_manager>", "path": "subpath/to/the/other/directory"}]`.


You have the inline <1> here but are using a NOTE comment instead.

arewm · 2025-01-08T16:13:09Z

docs/modules/ROOT/pages/how-tos/configuring/prefetching-dependencies.adoc

+      value: '{"type": "<package_manager>", "path": "."}' <1>
+----
+NOTE: The prefetch-input parameter specifies the path to the directory of the project. In this example, the `.` indicates the repository root. If you have multiple directories, you can provide the path to those directories in the JSON array format: `[{"type": "<package_manager>", "path": "."}, {"type": "<package_manager>", "path": "subpath/to/the/other/directory"}]`.


It is also possible to use a tekton array, specifying a different package manager configuration per line. Is that more usable? Should we suggest that or at least mention it (it is definitely harder to give an inline example like you did here)?

docs/modules/ROOT/pages/how-tos/configuring/prefetching-dependencies.adoc

arewm · 2025-01-08T16:15:20Z

docs/modules/ROOT/pages/how-tos/configuring/prefetching-dependencies.adoc

+    - name: dev-package-managers
+      value: "true"
+----
+NOTE: You won't find `dev-package-managers` as a param on the `prefetch-dependencies` task. You have to add it, and set it to true. This is because Cachi2 hasn't declared stable support for the feature yet. Use it at your own risk.


If we think that many Konflux users will leverage this, should we start exposing it to reduce the amount of manual work that needs to be done?

I think this section should temporarily be nested under the RPM section only (we only need it for RPMs). And then I think the next step is to remove RPM from dev-package-managers.

arewm · 2025-01-08T16:16:15Z

docs/modules/ROOT/pages/how-tos/configuring/prefetching-dependencies.adoc

+
+== Verification
+* From the {ProductName} *Applications* view, go to *Activity > Pipeline runs*.
+** Go to the pipeline run with *Build* in the *Type* column and confirm that the `prefetch-dependencies` stage displays a green checkmark. This indicates that the build process successfully fetched all dependencies.


Is it possible for the task to show green if input has been provided but not prefetched (i.e. bad configuration which doesn't correspond to a package manager or skipped fetching)?

What is the color of the prefetch task, if the hermetic pipelineSpec param is set to false, but prefetch-input is not empty ?

Prefetch can work on its own regardless of the hermetic parameter. If successful, then I think it should indicate a green check mark if the process worked.

brunoapimentel · 2025-01-08T19:31:00Z

I still think we should get completely rid of the individual package manager sections, unless they're really necessary. I don't think the "Prerequisites" section adds much valeu, and the rest is essentially repetition.

This is a rough suggestion on how I see the sections (note that after "Troubleshooting, all sections are in-depth/specific info)

Title
Procedure
Verification
Troubleshooting
Building Pip wheels from source
Configuring the repository for RPM prefetching
Enabling the prefetching of binary files (pip and bundler)
Prefetching generic artifacts

arewm · 2025-01-14T13:56:43Z

I like the idea of having some summary/representation of the prefetch configuration in this document but I don't think that we should repeat everything in the Cachi2 documentation. What would you think about having a table with a sample input to the prefetch dependency task for each package manager. We can then have sections later with specific things that need to be done (like enabling the dev package managers, binary file prefetchign, etc.). The table can then point users to the specific documentation for the prefetch elsewhere (i.e. cachi2 docs) and/or the the sections later on the page.

slimreaper35 · 2025-01-20T10:35:36Z

I am back. Let's finish this off 🚀

I don't think that we should repeat everything in the Cachi2 documentation.

I believe we all agree on that.

The table can then point users to the specific documentation for the prefetch elsewhere (i.e. cachi2 docs)

👍

This is a rough suggestion on how I see the sections (note that after "Troubleshooting, all sections are in-depth/specific info)

Currently, there is a Title, Procedure, Verification, and Troubleshooting. I thought we could have something like Tutorials where we would add those specific things.

Signed-off-by: Michal Šoltis <[email protected]>

Cachi2 documentation is referenced multiple times in the document. At the beginning [1], there is a link to main README.md file which has all references needed for a user. --- [1]: https://github.com/konflux-ci/docs/blob/main/docs/modules/ROOT/pages/how-tos/configuring/prefetching-dependencies.adoc#prefetching-package-manager-dependencies-for-hermetic-builds Signed-off-by: Michal Šoltis <[email protected]>

Signed-off-by: Michal Šoltis <[email protected]>

github-actions · 2025-01-31T09:37:11Z

🚀 Preview is available at https://pr-200--konflux-docs.netlify.app

slimreaper35 · 2025-01-31T09:38:10Z

Let's not make the scope of this PR too big. I basically moved duplicate paragraphs for the Procedure, Verification, and Troubleshooting sections to the top of the page.

arewm

Sorry for the delay. I think it looks like an improvement. We can iterate with future PRs to improve further.

arewm · 2025-02-03T15:07:21Z

docs/modules/ROOT/pages/how-tos/configuring/prefetching-dependencies.adoc

-== Additional resources
-
-* To troubleshoot any issues you might experience when you enable prefetch builds for `pip` or `pip` with source dependencies, see link:https://github.com/containerbuildsystem/cachi2/blob/main/docs/pip.md#troubleshooting[Troubleshooting].
-* For more information about Cachi2, see link:https://github.com/containerbuildsystem/cachi2/blob/main/docs/usage.md[Cachi2].


Do we not want to link to these anymore?

The link to pip troubleshooting is already in pip section:
https://github.com/slimreaper35/konflux-ci-docs/blob/prefetching/docs/modules/ROOT/pages/how-tos/configuring/prefetching-dependencies.adoc?plain=1#L210

The link to cachi2 is at the beginning of the page:
https://github.com/slimreaper35/konflux-ci-docs/blob/prefetching/docs/modules/ROOT/pages/how-tos/configuring/prefetching-dependencies.adoc?plain=1#L3

That's why I deleted those.

docs/modules/ROOT/pages/how-tos/configuring/prefetching-dependencies.adoc

Signed-off-by: Michal Šoltis <[email protected]>

github-actions · 2025-02-04T12:19:56Z

🚀 Preview is available at https://pr-200--konflux-docs.netlify.app

arewm

Thanks for these changes!

slimreaper35 · 2025-02-04T14:32:48Z

NP! Thank you for reviewing it.

slimreaper35 force-pushed the prefetching branch from 2f00c0e to 01c67f9 Compare January 3, 2025 09:34

slimreaper35 force-pushed the prefetching branch from 01c67f9 to 6022999 Compare January 3, 2025 09:40

slimreaper35 force-pushed the prefetching branch from 6022999 to 714e944 Compare January 3, 2025 10:01

slimreaper35 force-pushed the prefetching branch from 714e944 to 3410739 Compare January 3, 2025 15:47

slimreaper35 force-pushed the prefetching branch from 3410739 to f4818a0 Compare January 3, 2025 15:52

slimreaper35 force-pushed the prefetching branch from f4818a0 to 476bc30 Compare January 8, 2025 16:04

arewm reviewed Jan 8, 2025

View reviewed changes

slimreaper35 added 4 commits January 31, 2025 10:02

prefetching-dependencies: Remove duplicate troubleshooting sections

91c1456

Signed-off-by: Michal Šoltis <[email protected]>

prefetching-dependencies: Remove duplicate verification sections

69c40de

Signed-off-by: Michal Šoltis <[email protected]>

prefetching-dependencies: Drop invalid image reference

fec8dd0

Signed-off-by: Michal Šoltis <[email protected]>

slimreaper35 force-pushed the prefetching branch from 476bc30 to 7276419 Compare January 31, 2025 09:35

slimreaper35 requested review from arewm and brunoapimentel January 31, 2025 09:39

arewm reviewed Feb 3, 2025

View reviewed changes

prefetching-dependencies: Add one shared procedure section

255dd28

Signed-off-by: Michal Šoltis <[email protected]>

slimreaper35 force-pushed the prefetching branch from 7276419 to 255dd28 Compare February 4, 2025 12:18

arewm approved these changes Feb 4, 2025

View reviewed changes

arewm merged commit 3e45341 into konflux-ci:main Feb 4, 2025
3 checks passed

slimreaper35 deleted the prefetching branch February 4, 2025 14:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

prefetching dependencies refactoring #200

prefetching dependencies refactoring #200

slimreaper35 commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

brunoapimentel commented Dec 19, 2024

github-actions bot commented Jan 3, 2025

github-actions bot commented Jan 3, 2025

github-actions bot commented Jan 3, 2025

github-actions bot commented Jan 3, 2025

github-actions bot commented Jan 3, 2025

github-actions bot commented Jan 3, 2025

github-actions bot commented Jan 8, 2025

arewm Jan 8, 2025

arewm Jan 8, 2025

arewm Jan 8, 2025

arewm Jan 8, 2025

brunoapimentel Jan 8, 2025

arewm Jan 8, 2025

slimreaper35 Jan 20, 2025

arewm Jan 20, 2025

brunoapimentel commented Jan 8, 2025

arewm commented Jan 14, 2025

slimreaper35 commented Jan 20, 2025

github-actions bot commented Jan 31, 2025

slimreaper35 commented Jan 31, 2025

arewm left a comment

arewm Feb 3, 2025

slimreaper35 Feb 4, 2025 •

edited

Loading

github-actions bot commented Feb 4, 2025

arewm left a comment

slimreaper35 commented Feb 4, 2025

prefetching dependencies refactoring #200

prefetching dependencies refactoring #200

Conversation

slimreaper35 commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

brunoapimentel commented Dec 19, 2024

github-actions bot commented Jan 3, 2025

github-actions bot commented Jan 3, 2025

github-actions bot commented Jan 3, 2025

github-actions bot commented Jan 3, 2025

github-actions bot commented Jan 3, 2025

github-actions bot commented Jan 3, 2025

github-actions bot commented Jan 8, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brunoapimentel commented Jan 8, 2025

arewm commented Jan 14, 2025

slimreaper35 commented Jan 20, 2025

github-actions bot commented Jan 31, 2025

slimreaper35 commented Jan 31, 2025

arewm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slimreaper35 Feb 4, 2025 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Feb 4, 2025

arewm left a comment

Choose a reason for hiding this comment

slimreaper35 commented Feb 4, 2025

slimreaper35 Feb 4, 2025 •

edited

Loading