Why do past participles that take 'avoir' sometimes agree with the noun?

The answer is the preceding direct object.

Take the sentence the boy ate the apple.

Here we have a subject (the boy), followed by the verb (ate) and then the object (the apple).

So the sentence would be as follows.....

Le garçon a mangé la pomme.

HOWEVER

If the object comes before the verb, that is it precedes the subject which does the action to it, then the past participle has to agree in gender (masculine and feminine) and in number (singular or plural) with the object.

So it would become.........

La pomme a été mangée par le garçon.

Because pomme is feminine the past participle 'mangé' has an extra 'e' to make it agree.

The preceding direct object is useful to show off your knowledge of the french language, and also allows you to place the emphasis on the object in a sentence, by placing it at the start, rather than the subject.