"Of the order x^2" means the dominant behavior matches that of x^2 as x gets large. For polynomial functions, the dominant behavior is that of the highest-degree term.
For other functions, the dominant behavior will typically be governed in some other way. Here, the rate of growth of the x·log(x) function is determined by log(x), which has decreasing slope as x increases.
Only answer selection B has a highest-degree term of x^2, so only that one exhibits O(x^2) behavior.
