Map marker actions are disassociated

Why not use Shift/Ctrl/Alt+Left Click to add/remove? Why add a destination via the map, then use the global notification system to remove?

This system is stupid.