The UI for adding followers to a task should be easier to use, similar to the "Assign" UI.
The current UI is buried in a menu, hard to get to, and hard to add people while the "assign" UI is easy and accessible. The end result is it encourages people to add too many "assignees" who aren't responsible for the task.
It could function a bit like a "CC" on an email (need to be aware, no action needed) or an "informed" designation on a RACI matrix.