Fire the manager for one of your fiefs. This option also allows a player who is a manager of a fief to quit as manager.