An analogy for goal design · King Midas Problem
Midas wanted everything he touched to turn to gold. He did not account for this including his food and drink. Having his wish granted became his downfall.
Applied to AI, the story illustrates the difference between a formal objective and a human intention. A system can find a solution that meets a success metric without providing the intended benefit. That is why constraints must also be specified and real-world effects checked.
Takeaway
A well-defined objective specifies both the desired outcome and unacceptable ways of achieving it.
Source published: