The Alerts are broken. They don't work at all and need to be fixed.
Alerts are created but text and emails are never sent. I spoke with Microsoft support on this, and they acknowledged the problem, including the fact that it was broken for them, as well, but did not have any solution or workaround. They suggested I post feedback here in addition to them logging the call on their side.