Generate source code SBOM in 'context' mode #1430

sergii-ssh · 2023-03-16T18:12:16Z

Create SBOM even if the docker output set to context just skip scanning tars.
Add BillOfMaterialsURI property to overwrite default location for SBOM

Change-Id: Idf989734e10c4329d703fb9295c18efe39043ae3

istio-policy-bot · 2023-03-16T18:12:19Z

😊 Welcome @sergii-ssh! This is either your first contribution to the Istio release-builder repo, or it's been
awhile since you've been here.

You can learn more about the Istio working groups, code of conduct, and contributing guidelines
by referring to Contributing to Istio.

Thanks for contributing!

Courtesy of your friendly welcome wagon.

linux-foundation-easycla · 2023-03-16T18:12:20Z

The committers listed above are authorized under a signed CLA.

✅ login: sergii-ssh (34d5faa)

kfaseela · 2023-03-16T18:19:15Z

pkg/build/sbom.go

@@ -26,42 +26,40 @@ import (
 	"istio.io/release-builder/pkg/util"
 )

-// Sbom generates Software Bill Of Materials for istio repo in an SPDX readable format.
+const (
+	sourceSpdx    string = "istio-source.spdx"


i was wondering if we should have version appended to the file name... looks like all other artifacts have this. @ericvn thoughts?

Also, is the spdx file created on a per-artifact basis or for the project as a whole? If per-artifact, maybe the name should be included as infix as well.

Good idea, I'll add the version name to the file name.
spdx is create for build/project, i.e combined spdx is generated for all docker tars

We don't version manifest.yaml or a few others fwiw. Its already in a per-version folder (https://gcsweb.istio.io/gcs/istio-release/releases/1.16.0).

but I am fine with it as well

updated, ptal

Change-Id: If2c6a41b38c6b387f48be7b500770f05e61057e6

Change-Id: I2e22fb501ff2e44fdc1fcfb561b509f292613a37

kfaseela · 2023-03-17T18:59:14Z

pkg/build/sbom.go

-		return fmt.Errorf("couldn't generate sbom for istio release artifacts: %v", err)
+	// For Docker output in 'context' mode we will not produce SBOM.
+	// `bom` can produce bill only for tar and remote images.
+	if manifest.DockerOutput == model.DockerOutputTar {


i am wondering where we are running SBOM in 'context' mode as the PR title says.

Currently we don't call bom at all if context mode is specified. I am enabling it here https://github.com/istio/release-builder/pull/1430/files#diff-e3115b8ca9fdf611ae3dc61dc235e39aeeddbca61036b803ba3c0826f25cc3c9L91

For this snippet. I am skipping sbom generation for docker artifacts since bom doesn't support local context/registries but generate one for the source code.

OK, so for source code we need to generate SBOM, as docker output mode does not matter in that case. Previously we were fully skipping. @puerco is this correct, that bom cannot support local registries? I remember you mentioning that it also should work.

@puerco is it possible to generate bom for local registers without exporting to tar?

Could you please clarify how the PR title "[Run SBOM in 'context' mode]" makes sense now? You run it in context mode, but the SBOM is generated only for source code right?

Updated the title to highlight that explicitly.

puerco · 2023-03-17T19:03:52Z

pkg/build/sbom.go

+	// For Docker output in 'context' mode we will not produce SBOM.
+	// `bom` can produce bill only for tar and remote images.


@sergii-ssh What happens when you build in "context" mode? Happy to learn about the use case if you think a feature is missing. Or is it for local development?

hi @puerco In context mode images we build are stored in local registry/context, i.e. gcr.io/istio-release/pilot:1.17.2. Now if I run bom generate -o istio-release.spdx --image gcr.io/istio-release/pilot:1.17.2 it will fail because image hasn't been pushed to remote registry yet.

FATA generating doc: creating SPDX document: generating SPDX package from image ref gcr.io/istio-release/pilot:1.17.2: while downloading images to archive: fetching remote descriptor: GET https://gcr.io/v2/istio-release/pilot/manifests/1.17.2: MANIFEST_UNKNOWN: Failed to fetch "1.17.2" from request "/v2/istio-release/pilot/manifests/1.17.2".

Workaround is to docker save and then run with --image-archive but it would be nice to have something like --image-local flag to avoid doing so.

Can we not point to the images that were created when in context (that's a bad name) mode? They won't been the final location. Or is the SBOM going to end up with inaccurate data? Maybe it would be accurate if we set sbomOutputURI to the correct value.

SBOM location is configurable in manifest.BillOfMaterialsURI and filenames are stamped with version. Accuracy shouldn't be an issue unless spdx report overwriten on purpose.

ericvn · 2023-03-17T19:45:35Z

pkg/build/sbom.go

@@ -26,42 +26,38 @@ import (
 	"istio.io/release-builder/pkg/util"
 )

-// Sbom generates Software Bill Of Materials for istio repo in an SPDX readable format.
+const (
+	sbomOutputURI string = "https://storage.googleapis.com/istio-release/releases"


We generate images to multiple places: We may also put them in istio-prerelease-testing or istio-testing depending on what we are building. That said I see we seem to use a single URI in the replaced code.

It was hardcoded so I am keeping it not to break anything. This is the default unless manifest.BillOfMaterialsURI is specified.
I suspect sbom location needs to be specified here as well? https://github.com/istio/release-builder/blob/master/release/build.sh#L38

Updated build script. Now we'll upload SBOMs to the same bucket as other artifacts.

ericvn · 2023-03-17T20:08:10Z

pkg/build/sbom.go

+	// For Docker output in 'context' mode we will not produce SBOM.
+	// `bom` can produce bill only for tar and remote images.


Can we not point to the images that were created when in context (that's a bad name) mode? They won't been the final location. Or is the SBOM going to end up with inaccurate data? Maybe it would be accurate if we set sbomOutputURI to the correct value.

Change-Id: I2d3b13b82e23dd80d46898d9a0e48c05e66b8e39

kfaseela · 2023-03-29T10:38:12Z

LGTM

ericvn · 2023-05-25T16:02:22Z

pkg/build/build.go

-	if manifest.DockerOutput == model.DockerOutputContext {
-		log.Warnf("Docker output in 'context' mode; will not produce SBOM.")
-	} else if manifest.SkipGenerateBillOfMaterials {
+	if manifest.SkipGenerateBillOfMaterials {


We create a new option, SkipGenerateBillOfMaterials, and we output a message if set, but I don't see that we actually act on it. I would have expected some check around the actual BOM generation to not run if this value is set.

ericvn · 2023-05-25T16:04:08Z

pkg/build/sbom.go

-		"--image-archive", strings.Join(dockerImages, ","), "--output", releaseSbomFile).Run(); err != nil {
-		return fmt.Errorf("couldn't generate sbom for istio release artifacts: %v", err)
+	// For Docker output in 'context' mode we will not produce SBOM.
+	// `bom` can produce bill only for tar and remote images.


We used to generate a message that BOM generation wouldn't happen if we were in context mode and removed that above. If we are in context mode and have not specified to skip the generation, we will not have generated an SBOM, and there would have been no messages in the log that SBOM generation was skipped.

ericvn · 2023-05-25T16:10:03Z

release/build.sh

@@ -36,6 +36,7 @@ fi

 PRERELEASE_DOCKER_HUB=${PRERELEASE_DOCKER_HUB:-gcr.io/istio-prerelease-testing}
 GCS_BUCKET=${GCS_BUCKET:-istio-prerelease/prerelease}
+SBOM_OUTPUT_URI="https://storage.googleapis.com/${GCS_BUCKET}/releases"


Do we want this value to be overwrite able as it isn't as written here? If not, then we could include it in the URI below and save a line.

Run SBOM for 'context' mode

34d5faa

Change-Id: Idf989734e10c4329d703fb9295c18efe39043ae3

sergii-ssh requested a review from a team as a code owner March 16, 2023 18:12

istio-testing added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Mar 16, 2023

kfaseela reviewed Mar 16, 2023

View reviewed changes

sergii-ssh added 2 commits March 16, 2023 16:13

fix lint

c4467a3

Change-Id: If2c6a41b38c6b387f48be7b500770f05e61057e6

Add version prefix

ea29811

Change-Id: I2e22fb501ff2e44fdc1fcfb561b509f292613a37

kfaseela reviewed Mar 17, 2023

View reviewed changes

puerco reviewed Mar 17, 2023

View reviewed changes

ericvn reviewed Mar 17, 2023

View reviewed changes

Add sbom uri to build.sh

d9d505f

Change-Id: I2d3b13b82e23dd80d46898d9a0e48c05e66b8e39

sergii-ssh changed the title ~~Run SBOM in 'context' mode~~ Generate source code SBOM in 'context' mode Mar 27, 2023

ericvn reviewed May 25, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate source code SBOM in 'context' mode #1430

Generate source code SBOM in 'context' mode #1430

sergii-ssh commented Mar 16, 2023

istio-policy-bot commented Mar 16, 2023

linux-foundation-easycla bot commented Mar 16, 2023 •

edited

kfaseela Mar 16, 2023

stewartbutler Mar 16, 2023

sergii-ssh Mar 16, 2023

howardjohn Mar 16, 2023

howardjohn Mar 16, 2023

sergii-ssh Mar 17, 2023

kfaseela Mar 17, 2023

sergii-ssh Mar 17, 2023

kfaseela Mar 17, 2023

sergii-ssh Mar 24, 2023

kfaseela Mar 27, 2023

sergii-ssh Mar 27, 2023

puerco Mar 17, 2023

sergii-ssh Mar 17, 2023

ericvn Mar 17, 2023

sergii-ssh Mar 17, 2023

ericvn Mar 17, 2023

sergii-ssh Mar 17, 2023

sergii-ssh Mar 24, 2023

ericvn Mar 17, 2023

kfaseela commented Mar 29, 2023

ericvn May 25, 2023

ericvn May 25, 2023

ericvn May 25, 2023

		// For Docker output in 'context' mode we will not produce SBOM.
		// `bom` can produce bill only for tar and remote images.

Generate source code SBOM in 'context' mode #1430

Are you sure you want to change the base?

Generate source code SBOM in 'context' mode #1430

Conversation

sergii-ssh commented Mar 16, 2023

istio-policy-bot commented Mar 16, 2023

linux-foundation-easycla bot commented Mar 16, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kfaseela commented Mar 29, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

linux-foundation-easycla bot commented Mar 16, 2023 •

edited