Following the example given where "square, then subtract 5" refers to squaring the variable x and then subtracting 5 from it.
f(x) = x² - 5
The "rule" we must then express in function notation is "subtract 6, then square". The rule appears to suggest that the number 6 is being subtracted from the variable x. Then the entire expression is squared, resulting in the following function: