Group 5 and 6 have a different electronic structure, with group 6 having one extra electron that group 5.
If we look at Hunds rule, which states that electrons must occupy orbitals individually before pairing up. This is crucial in the P orbitals in which there are 3. (X, Y and Z) taking oxygen and nitrogen for example nitrogen is in group 5. It has 3 electrons in its P orbitals, thus one electron is each orbital. Oxygen is in group 6, with 4 electrons in the P orbitals. The first orbital has 2 electrons and the other two have 1 electron.
We know electrons are negative so in that first orbital of oxygen they must repel. This makes it easier to remove this electron so the energy required decreases. The general trend of ionisation energies is for it to increase across a period.