bug report: setup script alway fail with "setup email list" related on the user name

[mattn]

📝 Preliminary Checks

• I tried searching for an existing issue and followed the debugging docs advice, but still need assistance.

👀 What Happened?

I don't make sure this is really related on the username but:

j5 is ok

setup email add j5@example.com x
setup email list

* j5@example.com ( 0 / ~ ) [0%]

but j5v is not ok

setup email add j5v@example.com x
setup email list

[  ERROR  ]  Supplied non-number argument '' to '_bytes_to_human_readable_size()'
[  ERROR  ]  Aborting
[  ERROR  ]  Supplied non-number argument '' to '_bytes_to_human_readable_size()'
[  ERROR  ]  Aborting
* j5v@example.com (  /  ) [%]

👟 Reproduction Steps

No response

🐋 DMS Version

v13.3.1

💻 Operating System and Architecture

Linux docker-mailserver-7b4bd5cdf9-n5vk9 5.15.0-205.149.5.1.el8uek.x86_64 #2 SMP Fri Apr 5 12:44:45 PDT 2024 x86_64 GNU/Linux

⚙️ Container configuration files

I guess this is not a bug related on configuration file but if you want, I'll put my configuretion file here.

📜 Relevant log output

This is not an error of e-mail server.

Improvements to this form?

No response

[glime-ai]

Please note that this response was created using generative AI, leveraging previously available information from the repository. This information may be helpful. However, it is our strong recommendation that you carefully consider the information before blindly applying it.

---

It appears that the issue you're encountering with the setup email list command might be related to how the script handles usernames and their associated data. The error messages suggest a problem with converting byte values to a human-readable format, which could be triggered by specific conditions related to the username or the mailbox data.

To troubleshoot this issue, you can try the following steps:

1. Check Mailbox Data: Ensure that the mailbox for j5v@example.com is properly initialized and contains valid data. Sometimes, issues can arise if the mailbox is empty or not properly set up.

2. Review Script Functions: Look into the script that handles the setup email list command, particularly the function _bytes_to_human_readable_size(). There might be a bug in how it handles empty or unexpected values.

3. Permissions and Ownership: Verify that the permissions and ownership of the mailbox files are correct. Incorrect permissions can sometimes lead to unexpected errors when scripts try to access these files.

4. Logs and Debugging: Since the standard logs didn't provide enough information, consider enabling more detailed logging for the script operations. This might give you more insight into what exactly fails when the command is run.

5. Update and Patch: Ensure that your Docker Mailserver installation is up to date. If this is a known issue, it might have been addressed in a more recent release or there could be patches available.

6. Community and Support: If the issue persists, consider reaching out to the Docker Mailserver community or support channels. It's possible others have encountered and resolved similar issues.

If you need specific guidance on how to access or modify the script functions, or if you need help with enabling detailed logging, please let me know!

This comment was generated by Glime.

[mattn]

Something wrong.

$ setup dovecot-master add j5v@example.com x
$ doveadm quota get -u j5v@example.com
doveadm(j5v@example.com): Error: User doesn't exist
Quota name Type Value Limit                               

[polarathene]

$ docker run --rm -itd --hostname mail.example.test --name dms mailserver/docker-mailserver

# You have 120 sec to provision `/tmp/docker-mailserver/postfix-accounts.cf` with a user:
$ docker exec -it dms bash
$ setup email add j5@example.com x
$ setup email add j5v@example.com x

$ setup email list
* j5@example.com ( 0 / ~ ) [0%]

* j5v@example.com ( 0 / ~ ) [0%]

No problem it's working fine 👍

---

Error: User doesn't exist
[  ERROR  ]  Supplied non-number argument '' to '_bytes_to_human_readable_size()'
[  ERROR  ]  Aborting
[  ERROR  ]  Supplied non-number argument '' to '_bytes_to_human_readable_size()'
[  ERROR  ]  Aborting

This error though when running setup email list is due to running it too quickly, I was able to reproduce it.

Basically setup email add ... will update postfix-accounts.cf file with the new account. Then we have a check every 2 seconds if the file is updated, and if it is then create the new accounts in Dovecot. This can take a little bit longer, but it should all be done within about 10 seconds tops? This is just a disadvantage of using text file for accounts instead of an actual database unfortunately.

[mattn]

Thanks. I still have problem. Please do not close. I'm thinking I got this problem.

https://serverfault.com/questions/260488/dovecot-user-lookup-fails-when-using-usernamedomain-format

[mattn]

At least I can reproduce this problem even after 2 seconds have passed.

In my case, half of days are passed.

[polarathene]

even after 2 seconds have passed.

You misread. Every 2 seconds the file is checked for new accounts, if there is any it will trigger the Dovecot update which takes longer.

If you add an account, and this check happens, then you add another account after the check, while Dovecot is being updated, this process will start again to update Dovecot. You need adequate time to pass but it should work once Dovecot UserDB is updated.

---

Thanks. I still have problem. Please do not close. I'm thinking I got this problem.

$ doveadm user j5v@example.com

field   value
uid     5000
gid     5000
home    /var/mail/example.com/j5v/home
mail    maildir:/var/mail/example.com/j5v

I cannot reproduce with the basic instructions I provided. Likely your problem is you did not wait long enough before running the command.

[mattn]

@polarathene At least, half of days are passed after my "setup email add" command. I have hidden my domain name for security reasons, but it seems that this problem related on the domain name or hash value of userdb.
Could you please try some usernames randomly?

[mattn]

I could reproduce this with

$ setup email add j5F@example.com x
$ sleep 10
$ setup email list
[  ERROR  ]  Supplied non-number argument '' to '_bytes_to_human_readable_size()'
[  ERROR  ]  Aborting
[  ERROR  ]  Supplied non-number argument '' to '_bytes_to_human_readable_size()'
[  ERROR  ]  Aborting
* j5F@example.com (  /  ) [%]

[polarathene]

I have hidden my domain name for security reasons, but it seems that this problem related on the domain name or hash value of userdb.
Could you please try some usernames randomly?

Just look at the commands I gave you please. Run them yourself with any adjustments.

You can change the hostname to whatever you're using, and create accounts for whatever mail address you like. This runs perfectly fine locally to test and prove you can reproduce the problem.

If you are concerned about privacy, try reproduce with some domain that is not yours, like I have done with example.com.

---

UPDATE:

I could reproduce this with

Reproduced, thank you.

$ docker run --rm -it --hostname mail.example.test --name dms mailserver/docker-mailserver
$ docker exec -it dms bash -c 'setup email add j5F@example.com x && sleep 10 && setup email list'

doveadm(j5F@example.com): Error: User doesn't exist
[  ERROR  ]  Supplied non-number argument '' to '_bytes_to_human_readable_size()'
[  ERROR  ]  Aborting
[  ERROR  ]  Supplied non-number argument '' to '_bytes_to_human_readable_size()'
[  ERROR  ]  Aborting
* j5F@example.com (  /  ) [%]

This is due to uppercase usage in the email address, which AFAIK is being normalized to lowercase somewhere in DMS, but configuration in both postfix-accounts.cf and /etc/dovecot/userdb including the created mailbox at /var/mail is using the original uppercase input.

EDIT: Here we go, the default is using the %L modifier to lowercase the user (%u variable that is the full username and domain):

docker-mailserver/target/dovecot/10-auth.conf

Line 51 in 016d6b5

#auth_username_format = %Lu

Interestingly no one has reported this thus far, so not many have been attempting to use upper-case letters in their email addresses.

Personally it should remain lowercased like this which avoids any deployments that automate user account creation where allowing mixed casing would allow a fake account to exist with any uppercase letter to get mail instead (and various other UX issues). We already prevent creating another account like this via setup email add ... by checking for case-insensitive existing account:

docker-mailserver/target/scripts/helpers/database/manage/postfix-accounts.sh

Lines 19 to 20 in 016d6b5

	# Fail early before requesting password:
	[[ ${ACTION} == 'create' ]] && _account_should_not_exist_yet

docker-mailserver/target/scripts/helpers/database/manage/postfix-accounts.sh

Lines 72 to 77 in 016d6b5

	function _account_should_not_exist_yet() {
	__account_already_exists && _exit_with_error "'${MAIL_ACCOUNT}' already exists"
	if [[ -f ${DATABASE_VIRTUAL} ]] && grep -q "^${MAIL_ACCOUNT}" "${DATABASE_VIRTUAL}"; then
	_exit_with_error "'${MAIL_ACCOUNT}' is already defined as an alias"
	fi
	}

docker-mailserver/target/scripts/helpers/database/db.sh

Lines 126 to 135 in 016d6b5

	# Internal method for: _db_operation
	function __db_list_already_contains_value() {
	# Avoids accidentally matching a substring (case-insensitive acceptable):
	# 1. Extract the current value of the entry (`\1`),
	# 2. Value list support: Split values into separate lines (`\n`+`g`) at V_DELIMITER,
	# 3. Check each line for an exact match of the target VALUE
	sed -ne "s/^${KEY_LOOKUP}\+\(.*\)/\1/p" "${DATABASE}" \
	| sed -e "s/${V_DELIMITER}/\n/g" \
	| grep -qi "^${_VALUE_}$"
	}

But we don't normalize to lowercase, which affects the files and mailbox location mentioned. We probably should resolve this by normalizing to lowercase on account creation instead, especially since we know that there's already this issue with Dovecot not acknowledging such accounts it would be less invasive.

---

Do you have a specific need for uppercase mail addresses? Or is normalizing to lowercase acceptable?

[mattn]

There is no strong need for me. I was experimenting with the WordPress email post-submission feature. In doing so, WordPress recommends including a random string of characters as part of the email address so that it cannot be submitted by anyone else.
That is what caused this problem.

[polarathene]

There is no strong need for me.

👍

WordPress recommends including a random string of characters as part of the email address so that it cannot be submitted by anyone else.

I'm not familiar with the advice there, but I do want to point out that by default setup email add user@example.com is not specifically an email address. It's the DMS account with an email address as the username, but by default DMS will allow anyone who authenticates with these credentials to send email with any sender address they like.

I can only imagine that the advice from WP is to prevent guessing common usernames easily to try various passwords with 🤷‍♂️

• If this is a concern from an attacker with access to the container already, they can easily find the DMS accounts to use for credentials though, but they'll still need to guess the password.
• If DMS can be reached publicly on ports 587 or 465, authentication can be attempted by guessing the username, so you might have some added benefit there if that's a concern. However you may be better to just add the security improvement to the password if this is only personal accounts, high entropy there will work just as well if not better.

For high entropy password, look at this site, by selecting 48-bit, here is a passphrase example generated "detailed snail summons slim lab coat" (yes that's a very secure password!). This would at a minimum take 6 years to attack due to the hashing involved for someone with a fairly decent GPU. They can speed that up with more compute but that's still quite expensive (and as a remote login attack there is notable latency involved to the point it's irrelevant), plus with Fail2Ban active any remote login attack should quickly be blocked.

If you do want to restrict who can submit mail with your sender addresses, you can use SPOOF_PROTECTION=1. This restricts the sender address to the DMS account username, so if any service like WP was compromised locally by some malware for example, it can only use the address it was permitted to use. And since WP would be in a separate container it shouldn't have access to DMS data for account info or making any changes there (unless the Docker host itself is compromised, but that's a bigger problem).

[mattn]

The WordPress email post-submission feature does not authenticate. Just simple. Everything is published once the email arrives at that email address. So the email address must be strongly secret.
I don't plan to use this feature, but at least the docker-mailserver's setup script should display some kind of friendly error message like "Do not use upper case letters in mail addresses". 👍

[polarathene]

The WordPress email post-submission feature does not authenticate. Just simple. Everything is published once the email arrives at that email address. So the email address must be strongly secret.

DMS won't allow that?

• You cannot relay mail through DMS without authentication / trust.
• Port 25 doesn't support authentication. It is only for receiving mail inbound, or sending mail already submitted outbound.
• Ports 587 and 465 are required to submit mail you want to send outbound. These enforce authentication.

You can relax this of course with PERMIT_DOCKER, but that is discouraged for security reasons mentioned in my previous comment.

Or I've misunderstood, this isn't about WP sending mail through DMS, just reading mail DMS has already received via IMAP/POP3, in which case I understand now 👍

---

I don't plan to use this feature, but at least the docker-mailserver's setup script should display some kind of friendly error message like "Do not use upper case letters in mail addresses". 👍

The issue was re-opened, with plans to instead normalize to lowercase on account creation for the next release. That should remove any UX issues, thus no warning needed :)

[mattn]

The write of the article write and submit the text of the article to a specific (veriy secret) e-mail address.
Apart from that, WordPress checks for new emails triggered by cron and publishes the body text as an article. So DMS is used for just e-mail spooler.

[mattn]

[polarathene]

So the email address must be strongly secret.

For this then, keep in mind that you'll have only lowercase letters (due to planned normalization with DMS) and numbers, 36 values (aka base 36).

To avoid malicious abuse from guessing this by sending mail to many addresses, you would want this suffix to have plenty of entropy. The same advice for the passphrase would work here (eg: wordpress.detailed-snail-summons-slim-lab-coat@example.com, but obviously don't use the same value as a password).

The equivalent with base 36 is roughly 47-bit (log2(36^9) = 46.53) and would look like wordpress.zqy2cs5c5. If my math is right, that would require sending over 250k email addresses every second 24/7 to reach halfway of the possible email addresses (((36^9 / 250e3) / (60*60*24*365)) / 2_) within the same 6 years target (this assumes they already know the wordpress. part). There's over 100 trillion possible values in 36^9.

log2(52^8) (upper + lowercase letters) as shown in the image example from WP is 45.6 bits of entropy, so this is roughly equivalent of what they're suggesting. For another point of context log2(36^7) = 36, just dropping down to 7 characters from 9 is now 36-bit, and if the attacker could only send at a much slower rate of 5k mails per second constantly, that 50% point (average rate of guessing successfully) is now 90 days (((36^7 / 5e3) / (60*60*24)) / 2).

Hope that helps translate the WP advice to adjust for security equivalence in DMS 👍

[georglauterbach]

FYI: This is now a blocker for v14.0.0 - I thought it'd be good to resolve this before we release v14.0.0.

[georglauterbach]

I will have a look at this later today, just FYI.