How exactly can new species evolve from within a population, and how can we classify them as such?

There are two main mechanisms that contribute towards the process of speciation: natural selection, and genetic drift. The result of both of these mechanisms is phenotypic and genetic change as a population ages through the generations. Natural selection will select for specific phenotypic traits and their underlying genetic causes as a response to a selection pressure, such as predation or food resources, and genetic drift will cause a shift in allele (a specific form of a gene) frequency in a population over time. These processes can then lead to a speciation event in one of two ways: allopatric or sympatric speciation (although you could bear in mind that there is not always a clear line between the two, and speciation may occur in a situation that lies somewhere in the middle). Allopatric speciation is when the population is split geographically, such as by a body of water, or a mountain chain. Because of this physical barrier, the species will begin to diverge- either through random genetic drift changing the proportions of alleles in the populations, or through natural selection acting on the populations as a result of different environmental pressures, such as a different terrain or food source. Sympatric speciation occurs in the absence of this physical barrier. This means that a population must be naturally diverging while occupying the same environment, and may be because of the presence of two or more potential niches that are available for the organisms to 'move into'. It is important to also consider how natural selection may take forms such as sexual selection in a situation like this, where certain females are attracted to certain traits in males (or vice versa), and so the genes that push an individual to be attracted to a certain feature become linked to the genes for these features themselves. However, to have speciation, you must be able to define a point at which two organisms no longer belong to the same species. This is an inexact concept, as any definition will always have exceptions (just consider bacteria for example!), however we often use the idea that individuals in a species must be able to reproduce to form fertile offspring. Consider donkeys and horses, which are able to breed, giving mules, which are unable to reproduce themselves, indicating that donkeys and horses are different species.