Let
![P=(x,y,z)](https://tex.z-dn.net/?f=P%3D%28x%2Cy%2Cz%29)
be an arbitrary point on the surface. The distance between
![P](https://tex.z-dn.net/?f=P)
and the given point
![(7,11,0)](https://tex.z-dn.net/?f=%287%2C11%2C0%29)
is given by the function
![d(x,y,z)=\sqrt{(x-7)^2+(y-11)^2+z^2}](https://tex.z-dn.net/?f=d%28x%2Cy%2Cz%29%3D%5Csqrt%7B%28x-7%29%5E2%2B%28y-11%29%5E2%2Bz%5E2%7D)
Note that
![f(x)](https://tex.z-dn.net/?f=f%28x%29)
and
![f(x)^2](https://tex.z-dn.net/?f=f%28x%29%5E2)
attain their extrema, if they have any, at the same values of
![x](https://tex.z-dn.net/?f=x)
. This allows us to consider the modified distance function,
![d^*(x,y,z)=(x-7)^2+(y-11)^2+z^2](https://tex.z-dn.net/?f=d%5E%2A%28x%2Cy%2Cz%29%3D%28x-7%29%5E2%2B%28y-11%29%5E2%2Bz%5E2)
So now you're minimizing
![d^*(x,y,z)](https://tex.z-dn.net/?f=d%5E%2A%28x%2Cy%2Cz%29)
subject to the constraint
![z^2=xy](https://tex.z-dn.net/?f=z%5E2%3Dxy)
. This is a perfect candidate for applying the method of Lagrange multipliers.
The Lagrangian in this case would be
![\mathcal L(x,y,z,\lambda)=d^*(x,y,z)+\lambda(z^2-xy)](https://tex.z-dn.net/?f=%5Cmathcal%20L%28x%2Cy%2Cz%2C%5Clambda%29%3Dd%5E%2A%28x%2Cy%2Cz%29%2B%5Clambda%28z%5E2-xy%29)
which has partial derivatives
![\begin{cases}\dfrac{\mathrm d\mathcal L}{\mathrm dx}=2(x-7)-\lambda y\\\\\dfrac{\mathrm d\mathcal L}{\mathrm dy}=2(y-11)-\lambda x\\\\\dfrac{\mathrm d\mathcal L}{\mathrm dz}=2z+2\lambda z\\\\\dfrac{\mathrm d\mathcal L}{\mathrm d\lambda}=z^2-xy\end{cases}](https://tex.z-dn.net/?f=%5Cbegin%7Bcases%7D%5Cdfrac%7B%5Cmathrm%20d%5Cmathcal%20L%7D%7B%5Cmathrm%20dx%7D%3D2%28x-7%29-%5Clambda%20y%5C%5C%5C%5C%5Cdfrac%7B%5Cmathrm%20d%5Cmathcal%20L%7D%7B%5Cmathrm%20dy%7D%3D2%28y-11%29-%5Clambda%20x%5C%5C%5C%5C%5Cdfrac%7B%5Cmathrm%20d%5Cmathcal%20L%7D%7B%5Cmathrm%20dz%7D%3D2z%2B2%5Clambda%20z%5C%5C%5C%5C%5Cdfrac%7B%5Cmathrm%20d%5Cmathcal%20L%7D%7B%5Cmathrm%20d%5Clambda%7D%3Dz%5E2-xy%5Cend%7Bcases%7D)
Setting all four equation equal to 0, you find from the third equation that either
![z=0](https://tex.z-dn.net/?f=z%3D0)
or
![\lambda=-1](https://tex.z-dn.net/?f=%5Clambda%3D-1)
. In the first case, you arrive at a possible critical point of
![(0,0,0)](https://tex.z-dn.net/?f=%280%2C0%2C0%29)
. In the second, plugging
![\lambda=-1](https://tex.z-dn.net/?f=%5Clambda%3D-1)
into the first two equations gives
![\begin{cases}2(x-7)+y=0\\2(y-11)+x=0\end{cases}\implies\begin{cases}2x+y=14\\x+2y=22\end{cases}\implies x=2,y=10](https://tex.z-dn.net/?f=%5Cbegin%7Bcases%7D2%28x-7%29%2By%3D0%5C%5C2%28y-11%29%2Bx%3D0%5Cend%7Bcases%7D%5Cimplies%5Cbegin%7Bcases%7D2x%2By%3D14%5C%5Cx%2B2y%3D22%5Cend%7Bcases%7D%5Cimplies%20x%3D2%2Cy%3D10)
and plugging these into the last equation gives
![z^2=20\implies z=\pm\sqrt{20}=\pm2\sqrt5](https://tex.z-dn.net/?f=z%5E2%3D20%5Cimplies%20z%3D%5Cpm%5Csqrt%7B20%7D%3D%5Cpm2%5Csqrt5)
So you have three potential points to check:
![(0,0,0)](https://tex.z-dn.net/?f=%280%2C0%2C0%29)
,
![(2,10,2\sqrt5)](https://tex.z-dn.net/?f=%282%2C10%2C2%5Csqrt5%29)
, and
![(2,10,-2\sqrt5)](https://tex.z-dn.net/?f=%282%2C10%2C-2%5Csqrt5%29)
. Evaluating either distance function (I use
![d^*](https://tex.z-dn.net/?f=d%5E%2A)
), you find that
![d^*(0,0,0)=170](https://tex.z-dn.net/?f=d%5E%2A%280%2C0%2C0%29%3D170)
![d^*(2,10,2\sqrt5)=46](https://tex.z-dn.net/?f=d%5E%2A%282%2C10%2C2%5Csqrt5%29%3D46)
![d^*(2,10,-2\sqrt5)=46](https://tex.z-dn.net/?f=d%5E%2A%282%2C10%2C-2%5Csqrt5%29%3D46)
So the two points on the surface
![z^2=xy](https://tex.z-dn.net/?f=z%5E2%3Dxy)
closest to the point
![(7,11,0)](https://tex.z-dn.net/?f=%287%2C11%2C0%29)
are
![(2,10,\pm2\sqrt5)](https://tex.z-dn.net/?f=%282%2C10%2C%5Cpm2%5Csqrt5%29)
.