RFC: new way of loading certificate files #785

f1-outsourcing · 2020-07-29T12:49:19Z

nothing searches in private
#bind 0.0.0.0:8080 ssl crt "${HAPROXY_CRT_DIR}"/dev.marathon.mesos.crt ssl-load-extra-files key "${HAPROXY_KEY_DIR}"/dev.marathon.mesos.key
#bind 0.0.0.0:8080 ssl crt "${HAPROXY_CRT_DIR}"/dev.marathon.mesos.crt ssl-load-extra-files key
bind 0.0.0.0:8080 ssl crt "${HAPROXY_CRT_DIR}"/dev.marathon.mesos.crt
my key does not end on crt.key, just key (It is obvious from the name in front that it is already the key for that domain)
Why can't I specify like any other application my own crt and key file name? I don't get why you even want to automate this. Also the certs and private directories are different on distributions.

[ALERT] 210/124445 (30) : parsing [/etc/haproxy/haproxy.cfg:52] : 'bind 0.0.0.0:8080' : No Private Key found in '/etc/ssl/certs/dev.marathon.mesos.crt' or '/etc/ssl/certs/dev.marathon.mesos.crt.key'.

Is there an config option to specify a key dir and a cert dir?

wlallemand · 2020-07-29T13:24:34Z

Hello,

This is not a bug and works as designed. Indeed the "crt" option takes a basename which will be used to search other files by adding suffixes (".issuer", ".ocsp", ".key" etc.) It won't try to remove the extension of your basename. https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#ssl-load-extra-files

Is there an config option to specify a key dir and a cert dir?

You can only do that with a cert dir with 'crt-base' but we could probably add a 'key-base' too, that could be backported in 2.2.

The problem with a "key" directive is that it will apply on the whole bind line, and that's not a good idea because it will apply on all crt instances. Only a "key" directive in a crt-list file makes sense.

f1-outsourcing · 2020-07-29T13:40:38Z

Yes would be very nice, I am now scripting to auto generate the pems. I

If you have key and cert dir and specifiy those files, you do not need to do anything more. Except for ca validation.

You should also not force people the use of names. Although most follow standards, it is just never 'one size fits all'. I don't think I have ever seen any application, forcing me to use basename.key.

I think it is even the opposite of logical what is now implemented. In my case all the file names are the same with a nice 3 letter different extension. Furthermore your basename usage is also inconsistent, logically you should apply .crt to it. So in my case it would be dev.marathon.mesos.crt.crt so I could configure it as dev.marathon.mesos and crt and key are appended.
But then you have the 3rd argument that your configuration file values are not exactly the same as files on disk.

So I would rethink this ssl configuration, stick to file names easier configuration, less coding, less options for mistakes.

wlallemand · 2020-07-29T14:44:01Z

I agree with you regarding the use of names, but we had some limitations:

You can have several "crt" directives on a bind line
The directives on a bind line are supposed to apply on the whole line, not in declaration order

This was the original design of the SSL configuration parsing in HAProxy 1.5, I honestly hates it as most people use separate files and directories instead of PEM. When the OCSP was implemented, we couldn't add a "ocsp" directive because of (2), so a .ocsp file is looked up using the basename in "crt". And every features followed the same design after that, becoming worse with the bundles, which could look after ".ecdsa.ocsp" for example.

I won't be able to change the way we look at files because of backward compatibility. But the introduction of the "ssl-load-extra-files" directive was a first step to disable the auto discover and be able to specify the files with new directives in the future.

Specifying the extra files manually by certificate won't be compatible with the "crt" directive, so we will need to use the "crt-list" keyword for that.

A lot of things changed internally in HAProxy, we now have the concept of certificate storage, which is not really visible in the configuration. But what HAProxy does is that it will store all files related to a "crt' in the certificate storage and then load this storage for each new bind line it is used. So maybe what we need is to be able to declare a new certificate storage in a dedicated section. I don't have any definitive proposal for this, I'll probably open a new ticket with some thoughts.

f1-outsourcing · 2020-07-29T16:36:19Z

Good, interesting background info!

wlallemand · 2020-09-21T14:32:43Z

After discussing this with several people here are more thoughts about this:

the ssl-load-extra-files feature is convenient, people like being able to auto-load a .key but adding this at this end of the filename is not convenient. We could add an option to remove the current extension before looking for the .key.
some people want to be able to search a key in a separate directory, so the "key-base" will be implemented
other people would like to put their filenames manually without any auto-discovery, with a "key" and "chain" keyword. However this is not compatible with bind lines. I thought the crt-list was a good way to do it, but after some discussions on future improvements we conclude that it will make the configuration worse. What we need is a way to describe a certificate, and a new section will be dedicated to this.

This is an example of what it could be:

certificates
    crt foobar.crt key foobar.key chain foobar.chain
    crt /etc/ssl/certs/site1.crt key /etc/ssl/private/site1.key

listen
    bind *:443 ssl crt foobar.crt # use the crt described in the certificates section

WRMSRwasTaken · 2020-09-21T14:40:49Z

How about using the resolvers section as a model to be able to define a name per certificates section?

certificates foobar
    crt /etc/ssl/certs/foobar.crt
    key /etc/ssl/private/foobar.key

So you'd just have to pass foobar to other directives (like for example making it compatible with crt-list or being able to specify multiple named certificates to the bind line)

wlallemand · 2020-09-21T14:54:22Z

We thought about it first, but that's not really convenient for people having thousands of certificates.

What I suggested will be compatible with crt-list, however if we want to use a different name we could use an optional "name" parameter, for example:

certificates
    crt foobar.crt key foobar.key chain foobar.chain
    crt /etc/ssl/certs/site1.crt key /etc/ssl/private/site1.key name site1

listen
    bind *:443 ssl crt foobar.crt crt site1 # use the crt described in the certificates section

wtarreau · 2020-09-22T08:58:28Z

I'm overall fine with this but I'm convinced that we really need to be able to name the section and probably to make it mandatory, like this:

certificates mysite
    crt foobar.crt key foobar.key chain foobar.chain
    crt /etc/ssl/certs/site1.crt key /etc/ssl/private/site1.key name site1

listen
    bind *:443 ssl crt foobar.crt crt @mysite:site1 # use the crt described in this certificates section

This would allow to easily load all the certs mentioned in such a group (e.g. crt @mysite), and more importantly will ease config management at scale. I don't think it adds any complexity at all, and should avoid difficulties imposed by a centralized list when it comes to multiple applications having their own certs. Typically an application that needs to be deployed could simply be shipped with a single "certificates" section mentioning the various paths and certs.

What I like with this format is that we can later think about extending it, for example by adding:

import : loads a file into the section
crt-path : preset the path to load all "crt" files
key-path : preset the path to load all "key" files
etc.

wlallemand · 2020-09-24T18:56:00Z

We talked about this, face to face with @wtarreau and decided to change a little bit the format:


# we declare a "mysite" scope
crt-store mysite
    crt foobar.crt key foobar.key chain foobar.chain
    crt /etc/ssl/certs/site1.crt key /etc/ssl/private/site1.key name site1

# or use the default scope
crt-store
    crt cert1.crt key cert1.key name cert1  

# the @:cert1 is used to specify the certificate "cert1" in the global scope
# @mysite:foobar.crt take the "mysite" scope
listen
    bind *:443 ssl crt @mysite:foobar.crt crt @mysite:site1 crt @:cert1

In issue #785, users are reporting that it's not convenient to load a ".crt.key" when the configuration contains a ".crt". This option allows to remove the extension of the certificate before trying to load any extra SSL file (.key, .ocsp, .sctl, .issuer etc.) The patch changes a little bit the way ssl_sock_load_files_into_ckch() looks for the file.

In issue haproxy#785, users are reporting that it's not convenient to load a ".crt.key" when the configuration contains a ".crt". This option allows to remove the extension of the certificate before trying to load any extra SSL file (.key, .ocsp, .sctl, .issuer etc.) The patch changes a little bit the way ssl_sock_load_files_into_ckch() looks for the file. (cherry picked from commit 8e8581e) Signed-off-by: William Lallemand <wlallemand@haproxy.org>

serimin · 2020-10-23T08:37:28Z

It works well.
But this option does not seem to be compatible with "set ssl cert, etc" command.

wlallemand · 2020-10-23T09:48:02Z

The problem is that the path of each file is not stored, only the path of the "crt" is indexed. So when you are doing a commit this is not a problem, but if you do a "set" with a ".key", haproxy is not able to know what was the extension of the corresponding crt.

So it's a little bit difficult to fix with the current architecture, you will still need to set ".key" at the end of the current path, and we will have the same problem with the new "crt-store" keyword.

I'll really don't want to introduce this in a 2.2 or 2.3 release without having a clean solution, so I'll revert the patch before it become a problem for the backward compatibility of the "set ssl" command.

wlallemand · 2020-10-23T14:00:48Z

What I could do is change a little bit the behavior, we could force the "crt" file to have a ".crt" extension as a requirement for this feature. It's less flexible but it make more sense and it's easy to replace the extension by ".crt" before the lookup

In order to be compatible with the "set ssl cert" command of the CLI, this patch restrict the ssl-load-extra-del-ext to files with a ".crt" extension in the configuration. Related to issue #785. Should be backported where 8e8581e ("MINOR: ssl: 'ssl-load-extra-del-ext' removes the certificate extension") was backported.

wlallemand · 2020-10-23T16:43:08Z

The above commit should fix this, I also added a reg-test which test this with "set ssl cert".

wlallemand · 2020-10-23T17:17:46Z

Unfortunately I won't backport this to 2.2, the CLI still has the bundle code and I don't want to risk to break it, so I'm still reverting this in 2.2.

hholst80 · 2021-03-16T20:04:27Z

The current workflow is also inconsistent with the highly opinionated K8s workflow. kubectl will by default create two keys called "tls.crt" and "tls.key" for a TLS secret.. Not "tls.crt.key", thus not working out of the box.

Workaround (bash shell):

kubectl create secret tls haproxy --cert=<(cat fullchain.pem privkey.pem) --key=privkey.pem

hholst80 · 2021-03-16T20:36:52Z

For public information, why can't we just have a "key" keyword in the bind statement, explicitly specifying a key file?

The directives on a bind line are supposed to apply on the whole line, not in declaration order

I do not know the internal magic how the suffixes are resolved but I do not think the above statement is true or at least how it is incompatible with the suggested syntax below.

    bind [...] ssl crt /tls/fullchain.pem key /tls/privkey.pem crt /etc/tls/another.crt

This would resolve 2 certificates (fullchain.pem and another.crt) and 2 keys (privkey.pem and another.{key,crt.key} [must exists if another.crt is not a cert bundle]).

Another alternative could be to introduce a new keyword (which intuitively seems backwards but it gets the job done)

    bind [...] ssl crtkey /tls/fullchain.pem /tls/privkey.pem crt /etc/tls/another.crt

wlallemand · 2021-03-17T01:28:45Z

The current workflow is also inconsistent with the highly opinionated K8s workflow. kubectl will by default create two keys called "tls.crt" and "tls.key" for a TLS secret.. Not "tls.crt.key", thus not working out of the box.

This was fixed in 2.3 with the "ssl-load-extra-del-ext" keyword.

For public information, why can't we just have a "key" keyword in the bind statement, explicitly specifying a key file?

The directives on a bind line are supposed to apply on the whole line, not in declaration order

I do not know the internal magic how the suffixes are resolved but I do not think the above statement is true or at least how it is incompatible with the suggested syntax below.
    bind [...] ssl crt /tls/fullchain.pem key /tls/privkey.pem crt /etc/tls/another.crt
This would resolve 2 certificates (fullchain.pem and another.crt) and 2 keys (privkey.pem and another.{key,crt.key} [must exists if another.crt is not a cert bundle]).

Unfortunately there is no way to know which key is associated with which crt using this syntax, because there is no "declaration order" on a bind line, all options are applied for the whole line. That means if you change the order of the crt and key options it must behave the same way, so that won't work if you have multiple key and crt keywords on the same line.

Another alternative could be to introduce a new keyword (which intuitively seems backwards but it gets the job done)
    bind [...] ssl crtkey /tls/fullchain.pem /tls/privkey.pem crt /etc/tls/another.crt

I think it's kind of confusing but it's an interesting idea, we tend to avoid having multiple arguments to a keyword but we should maybe reconsider something like this.

WRMSRwasTaken · 2021-03-17T11:20:40Z

If you plan to implement the said crtkey <certfile> <keyfile> or something in this direction, having an alternative to crt-list, for example: crtkey-list <file> with two file paths per line would be a good idea aswell imho.

wlallemand · 2021-03-17T12:42:32Z

I don't think think adding a new crtkey-list keyword is a good idea, each line generates only one SSL context, you can't have multiple key and crt on the same line, so just having a key keyword that works only on a crt-list section could be worth considering. It's not the same problem as a bind line.

rustyx · 2021-06-28T10:18:31Z

Can't bind options order be preserved? In that case the key keyword can be required to be either never specified or specified exactly as many times as the crt keyword. Then the correspondence when it is specified is always 1-to-1, in the same order, e.g. crt a key ka crt b key kb or key ka crt a key kb crt b or crt a crt b, but never crt a key ka crt b.

wlallemand · 2021-06-28T12:57:57Z

That would be the best option if the options order was preserved from the start, but it does not make sense to preserve order only for one keyword, it would be confusing. The key keyword is only part of the problem, people will eventually ask to set other files or options, for a specific certificate. The best "simple" way to implement this without breaking anything would be to do it in a crt-list file, because each line contains a certificate, but this would be kind of overkill for someone who just have one or two certificates to use.

But your comment make me thing of something, but it would need to be discussed with the other maintainers because it's kind of touchy.
What if the SSL options are applied in their own scope, which could be defined with an ssl keyword (or a new one if it breaks things).

bind *:443 ssl crt /etc/ssl/certs/foobar.pem key /etc/ssl/private/foobar.key ocsp /etc/ssl/ocsp/foobar.ocsp ssl crt /etc/ssl/certs/site1.pem key /etc/ssl/private/site1.key ocsp /etc/ssl/privte/site1.ocsp

This way we could emit an error if there are multiple "key" or "ocsp" keywords in the same scope. It will be kind of a crt-list, but instead of a newline character, it's an "ssl" keyword. If it's confusing for backward compatibility we could decide of a new keyword to do it (what about tls ?)

nakoo · 2022-05-10T17:34:59Z

bind *:443 ssl crt /etc/ssl/certs/foobar.pem key /etc/ssl/private/foobar.key ocsp /etc/ssl/ocsp/foobar.ocsp ssl crt /etc/ssl/certs/site1.pem key /etc/ssl/private/site1.key ocsp /etc/ssl/privte/site1.ocsp

This way we could emit an error if there are multiple "key" or "ocsp" keywords in the same scope. It will be kind of a crt-list, but instead of a newline character, it's an "ssl" keyword. If it's confusing for backward compatibility we could decide of a new keyword to do it (what about tls ?)

+1 for this. I'm trying to use docker secret to store my TLS certificates and ocsp files. But since docker secret only supports file, without folder support, it was impossible to provide ocsp as static file unless using socat which needs additional packages that I want to avoid.

wlallemand · 2022-05-10T18:47:32Z

Unfortunately this would break a lot of configuration so that is not possible, we will probably proceed this way:

add the "key", "ocsp" etc keywords to crt-list (since it's only one SSL context per line it is possible easily)
add a support for "inline" crt-list (so we could use the same syntax without breaking the old one)

nakoo · 2022-05-10T20:20:23Z

Unfortunately this would break a lot of configuration so that is not possible, we will probably proceed this way:

add the "key", "ocsp" etc keywords to crt-list (since it's only one SSL context per line it is possible easily)

add a support for "inline" crt-list (so we could use the same syntax without breaking the old one)

Thank you for the answer. crt-list syntax is much cleaner. Can I expect it to be working like this? I checked the docs just in case but seems doesn't support yet.

In crt-list file

example.com.pem [key example.com.pem.key ocsp example.com.pem.ocsp]

In haproxy.cfg

bind *:443,[::]:443 ssl crt-list /etc/haproxy/certs/crt-list strict-sni

wlallemand · 2022-05-10T21:15:30Z

example.com.pem [key example.com.pem.key ocsp example.com.pem.ocsp]

It could work like this indeed, but nothing was pushed in the master repository yet, we'll probably try to have at least these two keywords for 2.6.

wlallemand · 2022-05-18T13:09:40Z

Unfortunately the changes are too deep in the configuration parser, and will require to change a lot of things, it's a bit too late to be integrated into 2.6.

sigv · 2022-11-16T19:37:22Z

Is this expected to land in 2.7?

quality-leftovers · 2023-04-11T06:47:26Z

I'd like to add here that the ability to specifiy the key as seperate configuration itemg would enable supporting hardware security modules via OpenSSL key engine (which is deprecated) or OpenSSL 3 store API with just a few lines of code.

When changing the configuration please bear in mind that the OpenSSL 3 API supports URIs for key locations. So it would be great if URIs would be supported in addition to "plain" file paths.

See #71

scryptio · 2023-04-28T09:39:37Z

What is the current state about this topic? Its been 3 years and unfortunately I cannot find new / further information about this. I once read it was coming in 2.6, then that it was planned for 2.7. Now 2.8 is on the horizon. An official statement would be great about whether this topic is still on the list, being worked on or not considered.

wlallemand · 2023-04-28T13:35:48Z

We made multiple experiments around this, but it was implying a lot of breakage on the CLI and the configuration. Nothing was merged yet and it's a bit too late for 2.8 since we are freezing the code. This change has big implications in the configuration system even if it seems like a simple configuration option.

The topic is still on the list and we are conscious of the current limitation with the old openssl pkcs11 engine (#71) and people that need to modify their deployment scripts to concatenate the PEM.

Also, don't forgot that even if you are not able to specify an absolute path, HAProxy is still able to load separated .crt and .key files.

The "load" keyword in the "crt-store" section allows to define which files you want to load for a specific certificate definition. Ex: crt-store load name "site1" crt "site1.crt" key "site1.key" load crt "site2.crt" key "site2.key" frontend in bind *:443 ssl crt "site1" crt "site2.crt" This is part of the certificate loading which was discussed in #785.

wlallemand · 2024-02-13T16:26:48Z

For those interested there was no possible solution with the crt-list system which was going to work, so we stopped trying to make it work with an architecture which was going to be hackish...

The "crt-store" section solution was kept, and a first version was pushed in a feature branch https://github.com/haproxy/haproxy/tree/20240208-split-key-cert . See the documentation 26ac95c

There are still keywords to be implemented and changes to be done on the CLI, but this will basically looks like this:

    crt-store
        load name "site1" crt "site1.crt" key "site1.key"
        load crt "site2.crt" key "site2.key"

    frontend in
        bind *:443 ssl crt "site1" crt "site2.crt"

hholst80 · 2024-02-16T20:50:35Z

so "name" is optional or derived how?

edit: nm, i saw that documentation was already added to the commit. thanks looking forward to using tihs!

26ac95c

tmccombs · 2024-02-16T23:01:32Z

would this help with being able to use a PKCS#11 URL for the key?

wlallemand · 2024-02-19T08:20:11Z

@hholst80 the name is either defined or it will use the crt parameter like it was done previously. but this could change in the future before it reach the master branch.

@tmccombs this will provide more flexibility in the certificate configuration, it will probably need some adaption for the PCKS#11 but I think that will help.

'crt-store' is a new section useful to define the struct ckch_store. The "load" keyword in the "crt-store" section allows to define which files you want to load for a specific certificate definition. Ex: crt-store load name "site1" crt "site1.crt" key "site1.key" load crt "site2.crt" key "site2.key" frontend in bind *:443 ssl crt "site1" crt "site2.crt" This is part of the certificate loading which was discussed in #785.

wlallemand · 2024-02-27T15:05:50Z

I pushed a new version in the https://github.com/haproxy/haproxy/tree/20240227-crtlist-crtstore branch. It introduces the new concept of "inline" crt-list. This allow to declare a crt-list directly in the configuration, it can also uses keyword from the crt-store section.

Example:

    crt-store
        load name "site0" crt "site0.crt" key "site0.key"

    crt-list list2
        use crt "site0"

    crt-list list1
        use crt "site1.crt"
        use crt "site2.crt" key "site2.key" ocsp "site2.ocsp"
        use crt "site3.crt" ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256"

    frontend in
        bind *:443 ssl crt-list "list1" crt-list "list2"

'crt-store' is a new section useful to define the struct ckch_store. The "load" keyword in the "crt-store" section allows to define which files you want to load for a specific certificate definition. Ex: crt-store load crt "site1.crt" key "site1.key" load crt "site2.crt" key "site2.key" frontend in bind *:443 ssl crt "site1.crt" crt "site2.crt" This is part of the certificate loading which was discussed in #785.

wlallemand · 2024-05-22T09:37:10Z

Small update about the actual state in 3.0.

The documentation about crt-store was updated here: https://github.com/haproxy/haproxy/blob/04a42a92f4cf31adf66a0796309bcb429e28de2f/doc/configuration.txt#L4783

the crt-store section was merged into 3.0, it supports an anonymous section which is "global" or a scoped one which can be called elsewhere in the configuration with the @/ syntax (see the doc)
the crt-store allows 3 keywords "load", "crt-base", "key-base"
the load keyword allows the "crt", "key", "ocsp", "issuer", "sctl" and "ocsp-update" parameters
the crt-list allows the usage of crt-store load parameters (https://github.com/haproxy/haproxy/blob/04a42a92f4cf31adf66a0796309bcb429e28de2f/doc/configuration.txt#L16049)
no "HSM" tests were done yet, feel free to test it with it, a keyword for an HSM URI could probably be added
the inline crt-list and crt-list V2 are postponed for now, it doesn't provide any feature but only a simplified syntax, so I prefer to wait a little bit before merging it in master.

I'm closing this ticket, for HSM features please refer to ticket #71.

WRMSRwasTaken · 2024-06-01T11:03:07Z

the crt-list allows the usage of crt-store load parameters (https://github.com/haproxy/haproxy/blob/04a42a92f4cf31adf66a0796309bcb429e28de2f/doc/configuration.txt#L16049)

Going to ask this here as I can't figure it out right now (running HAProxy version 3.0.0 2024/05/29 - https://haproxy.org/):

Having

bind [::]:443 v6only name fe_https_ipv6 tfo ssl curves X448:X25519:P-256 alpn h2,http/1.1 crt-list /etc/haproxy/certlist.txt

and the certlist.txt now (just one line extracted):

load crt /path/to/fullchain.cer key /pyth/to/privkey.key

HAProxy errors out with:

'bind [::]:443' in section 'frontend' : 'crt-list' : unable to stat SSL certificate from file 'load' : No such file or directory.

Removing load from the certlist.txt also errors out with unable to stat SSL certificate from file 'crt'.

Am I overlooking something here?

f1-outsourcing added status: needs-triage This issue needs to be triaged. type: bug This issue describes a bug. labels Jul 29, 2020

wlallemand added subsystem: ssl This issue is within the SSL / TLS subsystem. type: feature This issue describes a feature request / wishlist. and removed type: bug This issue describes a bug. labels Jul 29, 2020

wtarreau removed the status: needs-triage This issue needs to be triaged. label Jul 31, 2020

wlallemand mentioned this issue Sep 4, 2020

Support certificate and private key PEM in separate files #221

Closed

wlallemand changed the title ~~haproxy 2.2 looks in certs dir for keys?, should be looking private of course~~ RFC: new way of loading certificate files Sep 24, 2020

jmagnin mentioned this issue Oct 18, 2020

If splitting certificate and key, check for a file with a .key extension #906

Closed

This comment has been minimized.

Sign in to view

quality-leftovers mentioned this issue Apr 11, 2023

PKCS#11 support with HAProxy #71

Open

wlallemand self-assigned this Feb 14, 2024

wlallemand pinned this issue Mar 27, 2024

wlallemand closed this as completed May 22, 2024

wlallemand added the status: fixed This issue is a now-fixed bug. label May 22, 2024

wlallemand unpinned this issue May 22, 2024

RFC: new way of loading certificate files #785

RFC: new way of loading certificate files #785

Comments

f1-outsourcing commented Jul 29, 2020 • edited

wlallemand commented Jul 29, 2020 • edited

f1-outsourcing commented Jul 29, 2020

wlallemand commented Jul 29, 2020

f1-outsourcing commented Jul 29, 2020

wlallemand commented Sep 21, 2020 • edited

WRMSRwasTaken commented Sep 21, 2020 • edited

wlallemand commented Sep 21, 2020 • edited

wtarreau commented Sep 22, 2020

wlallemand commented Sep 24, 2020

This comment has been minimized.

serimin commented Oct 23, 2020

wlallemand commented Oct 23, 2020

wlallemand commented Oct 23, 2020 • edited

wlallemand commented Oct 23, 2020

wlallemand commented Oct 23, 2020

hholst80 commented Mar 16, 2021 • edited

hholst80 commented Mar 16, 2021 • edited

wlallemand commented Mar 17, 2021

WRMSRwasTaken commented Mar 17, 2021

wlallemand commented Mar 17, 2021

rustyx commented Jun 28, 2021

wlallemand commented Jun 28, 2021

nakoo commented May 10, 2022

wlallemand commented May 10, 2022

nakoo commented May 10, 2022 • edited

wlallemand commented May 10, 2022

wlallemand commented May 18, 2022

sigv commented Nov 16, 2022

quality-leftovers commented Apr 11, 2023

scryptio commented Apr 28, 2023

wlallemand commented Apr 28, 2023

wlallemand commented Feb 13, 2024 • edited

hholst80 commented Feb 16, 2024 • edited

tmccombs commented Feb 16, 2024

wlallemand commented Feb 19, 2024 • edited

wlallemand commented Feb 27, 2024

wlallemand commented May 22, 2024

WRMSRwasTaken commented Jun 1, 2024 • edited

f1-outsourcing commented Jul 29, 2020 •

edited

wlallemand commented Jul 29, 2020 •

edited

wlallemand commented Sep 21, 2020 •

edited

WRMSRwasTaken commented Sep 21, 2020 •

edited

wlallemand commented Sep 21, 2020 •

edited

wlallemand commented Oct 23, 2020 •

edited

hholst80 commented Mar 16, 2021 •

edited

hholst80 commented Mar 16, 2021 •

edited

nakoo commented May 10, 2022 •

edited

wlallemand commented Feb 13, 2024 •

edited

hholst80 commented Feb 16, 2024 •

edited

wlallemand commented Feb 19, 2024 •

edited

WRMSRwasTaken commented Jun 1, 2024 •

edited