"Of the order x^2" means the dominant behavior matches that of x^2 as x gets large. For polynomial functions, the dominant behavior is that of the highest-degree term.
For other functions, the dominant behavior will typically be governed in some other way. Here, the rate of growth of the x·log(x) function is determined by log(x), which has decreasing slope as x increases.
Only answer selection B has a highest-degree term of x^2, so only that one exhibits O(x^2) behavior.
You write down the decimal fraction like say .75 you put .75/1. Then you multiply both sides by 100 so .75/1 would turn into 75/100. Then all you have to do is simplify the fraction.