Spend a certain amount of your time looking after fief affairs. Depending on your personal characteristics, this can have a significant effect on improving your fiefs prosperity and loyalty.