The notation
h: x --> ax+b
is another way of saying h(x) = ax+b
The input is x, the output is h(x) = ax+b
The composite function notation h^2 is the same as (h o h)(x) or h(h(x)). I prefer h(h(x)) as it is the most descriptive of the three notation styles. The square notation is easily confused with actual squaring (when instead we want composite notation of a function with itself)
----------------
h(x) = ax+b
h(x) = a( x )+b
h(h(x)) = a( h(x) )+b ... replace every x with h(x)
h(h(x)) = a( ax+b )+b .... replace the h(x) on the RHS with ax+b
h(h(x)) = a*ax + ab + b
h(h(x)) = a^2*x + (ab + b)
-----------------
Since h(h(x)) = 36x-35, this means we can equate the two expressions to find a and b
a^2*x + (ab+b) = 36x-35
we see that
a^2 = 36, so a = 6 (keep in mind a > 0)
since a = 6, we know that
ab+b = -35
6b+b = -35
7b = -35
b = -35/7
b = -5
<---- Not the answer
<----- Not the answer
<---- Your answer