[DISCUSS] Move towards more reproducible builds?

[chrisdutz]

Hi all,

trying out GitHub Discussions this time.

So a lot of attention has recently been given to Open-Source and it's security and safety.
I always liked the idea of reproducible build, but I learned that there are many factors influencing this:

• Operating System
• Java version
• System Settings
  So effectively it seems that in order to get reproducible build easily, you would need to ship the computer the RM produced the release on.

No my idea was: How about we use Docker to produce our releases?
I propose to create a Dockerfile that is as simple as absolutely possible (So people can review it)
In this dockerfile, we configure what is needed in order to build PLC4X fully.

We could then add convenience scripts to our source repo, that allow people to download a release source tar ball and to build exactly this in Docker and compare the produced output with the official binary tar ball.

It would theoretically also allow us to have similar scripts for validating RCs, which could co-sign the RCs (Right now we only have one signature on the bundles. This way we could increase trust by having multiple PMC members sign the releases)

What do you folks think?

[ottlukas]

I really like the way how Apache StreamPipes builds from source with docker and if we can create reproducible builds with a dockerfile that would really be awesome and would enable PLC4X to be much more compliant and secure.

[chrisdutz]

Yeah ... however would I not want us to become lazy and simply rely on Docker for building. We should never let go of being able to build on real hardware. Unfortunately I have seen this happen way too often at the ASF. Initially a build works fine, till someone comes up with the idea of "using Docker to build" or to rely on a "Official Project build VM" and after a while, that is the only way you can build it.

[sruehl]

I quite like using Docker to define the build environment as it helps to have a clear reference environment. But as Chris said it should always be possible to build without it

[sruehl]

the "without it" we can achieve by having actions like we do now. I would even say let's keep them as is and add an additional workflow / Jenkinsfile which also builds with docker for reference.

[splatch]

Reproducible builds include also a repeatable checksums produced out same commit. In this regard Maven needs some tweaks, we might also remove timestamps from generated sources (if they are there).

With regard to docker - we don't need it for maven, however making it possible will bring benefit for non-java folks who run docker, but have no java environment. We just need to be careful since I've seen projects suffering from limits imposed by docker hub.

[chrisdutz]

Well I've already put quite a bit of effort towards the timestamps and the line-endings seveal years ago, when I setup the build.

However the problem is, that you still need to provide the OS as well as the Java version ... With using Docker, we could ensure the "system" is configured identically and that everything is configured to build all the parts of PLC4X (I mean ... most problems with broken builds we had in the last few months were related to the person merging not running with all "with-xyz" profiles enabled (Ok ... all build failures except the ones where the build failed because the build failed ;-) )

[ottobackwards]

IRL we have are moving from old fashioned Jenkins builds and build agent vms to Pipelines and docker and it is really great. Do we have the option to do pipelines?

[sruehl]

No sure what "Pipelines" are but I used to work with Gitlab and there we put every builder in Docker containers and I quite like that approach as we would have a pretty good defined environment.
Here is Github we also use github actions and those are what I mostly look to check if CI is ok. Jenkins has just to great of a latency. At the moment it acts as our golden builder.

[chrisdutz]

Ok ... so I've moved the old Dockerfile in the root of the project to an example in the plc4j section and created a new Dockerfile. Here I left away everything super-fancy as the Dockerfile should be understandable by anyone and optimizing everything (like pre-fetching the dependencies and building a customized JVM) only makes this more complicated.

I tried it on my systems and I was able to get a build-success for building all parts.

So in general this Dockerfile could be used by anyone to validate his changes didn't break anything without having to setup anything besides Docker and Java.

[chrisdutz]

I guess in next steps we might even do an experiment, wehre I create a RC (which we will not be releasing) but we could try to get reproducible builds configured and documented.

Ideally a RM creates a RC using this Dockerfile and stages that ... then ideally others could download the source archive and run the Dockerfile on their machines ... if the produced binaries match the ones staged, then this could be used as an additional checkbox in the release vote email template ... we could also probably have something in place to have PMC members co-sign the release artifacts if the test is successful (Might be good to have with all the legislation changes in europe and the US).

[sruehl]

Just stumbled over that https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers

[chrisdutz]

That could be cool for simplifying development without needing to setup stuff locally ... right? I think Jetbrains offers the same, right? How would this look like? Are there some default files that we need to add to enable such a feature?

Would be awesome if we could add all the config needed and as soon as someone uses VSCode or IntelliJ (Or any of the other Jetbrains IDEs) it could just "start development container".

[sruehl]

I think to your last point that might be that: https://youtrack.jetbrains.com/issue/IDEA-292050/Support-for-.devcontainer

[sruehl]

and this https://youtrack.jetbrains.com/issue/IDEA-317576/Devcontainer.json-support-for-IDEA-and-Gateway

[chrisdutz]

But has anyone tested the new Dockerfile that I created? And does anyone have an idea how to copy the output that the container builds into the local filesystem? I guess that's my most pressing problem with it right now.

I'd love to have some shell-scripts that you run, and for example one would take the current project, build it and then have the output in "target/docker" directories ...

[chrisdutz]

Ok ... so I've made a bit of progress on this :-)

So I've removed the maven build stuff from the Dockerfile. So now the Dockerfile just prepares the container for running the build. I've then created a docker-compose.yaml that runs the maven build based on that container. The main difference being, that I'm mounting the current directory in the container and having it locally store maven downloads in a directory ".repository" and deploy the build artifacts and deploy them to "./.local-snapshots-dir".

When running the docker compose command, you get the built artifacts in your local "target" directories, and beyond that what is deployed in the "./.local-snapshots-dir" ... we then should be able to compare the content here with the content in the staging repo and hereby confirm that the artifacts are correctly built from the given sources.