This problem is all about using integration by parts, so let's start by quoting the formula for integration by parts:
∫u*(dv/dx)dx = uv - ∫v*(du/dx)dx
To get the integral we want on the left hand side we can use the subtitutions u = dv/dx = ln(x). This means that we will have to find ∫ln(x)dx, this is also done using integration by parts:
To find ∫ln(x)dx we can use the substitutions u = ln(x) and dv/dx = 1. Using the formula above will then give us:
∫ln(x)1dx = ln(x)x - ∫x(1/x)dx
= xln(x) - ∫dx = xln(x) - x = x(ln(x)-1)
Using this we can now use our original substitutions in the formula to get:
∫ln(x)ln(x)dx = ln(x)x(ln(x)-1) - ∫x(ln(x)-1)(1/x)dx
= xln(x)(ln(x)-1) - ∫(ln(x)-1)dx
= xln(x)(ln(x)-1) - x(ln(x)-1) + x + c
Now we just have to tidy this up to get our final answer:
∫(ln(x))^2dx = x[(ln(x)+1)^2 + 1] + c