A CPU (Central Processing Unit) is comprised of a few large processing cores, each of which is capable of performing complicated tasks on its own. This added complexity also enables CPUs to be efficent sequential processors, queueing up tasks in a complex form that greately speeds up the processing time of the computer. As most tasks relating to the day-to-day running of a computer are quite complex, the CPU is the primary processor used by the vast majority of computers.
A GPU (Graphics Processing Unit), by contrast, is made up of many smaller cores. On their own these cores can only perform the simplest of tasks, however a GPU has a huge number of them (often a thousand times the number of a CPU of a similar cost). This number of cores enables the GPU to perform many simple tasks in parallel, allowing it to quickly perform tasks that would take a CPU a huge amount of time (for example, rendering graphics requires millions of simple calculations, making it perfect for a GPU). The GPU is not used in modern computers most of the time, but instead it is signaled by the CPU when the processor encounters a task that suits the capabilities of the GPU.