I've just found something interesting while checking the email header.

It reads like this:

From: Text@dd39012.kasserver.com, Forum@dd39012.kasserver.com,
nicht =?UTF-8?Q?=C3=B6ffentlich=2C=20nur=20f=C3=BCr=20Mitg?= =?UTF-8?Q?lieder?= <Forum <info@domain.de>>

However, the email in the settings reads:

Forum <info@domain.de>

The title of the Forum is set to:

Forum, nicht öffentlich, nur für Mitglieder

After changing this to just "Forum" it seems to work.

What am I missing here?