Question

find a homogeneous linear system of two equations in three unknowns whose solution space consists of those vectors in R3 that are orthogonal to a=(1,1,1) and b=(-2,3,0)

Answer 1

$\mathbf a=(1,1,1)^\top=1\,\mathbf i+1\,\mathbf j+1\,\mathbf k$
$\mathbf b=(-2,3,0)^\top=-2\,\mathbf i+3\,\mathbf j$
$\mathbf a\times\mathbf b=\begin{vmatrix}\mathbf i&\mathbf j&\mathbf k\\1&1&1\\-2&3&0\end{vmatrix}=-3\,\mathbf i-2\,\mathbf j+5\,\mathbf k=(-3,-2,5)^\top$

Basically, you're looking for a matrix $\mathbf A$ such that

$\mathbf A(\mathbf a\times\mathbf b)=\mathbf 0$

i.e. a matrix $\mathbf A$ whose nullspace with basis vector $\mathbf a\times\mathbf b$.

By the rank-nullity theorem, the rank of $\mathbf A$ and the dimension of its nullspace must add up to the number of columns, so

$\mathrm{rank}\mathbf A+\underbrace{\mathrm{null}\mathbf A}_1=3\implies\mathrm{rank}\mathbf A=2$

One easy choice for a row would be $\begin{bmatrix}1&1&1\end{bmatrix}$, since

$(1,1,1)(-3,-2,5)^\top=0$

Now you only need to find another combination such that the second row of $\mathbf A$ is independent of the first. An easy choice for this is to let the first element be 0, and the next be 1. Then the last element must be $\dfrac25$, as

$\left(0,1,\dfrac25\right)(-3,-2,5)^\top=0$

So,

$\underbrace{\begin{bmatrix}1&1&1\\0&1&\frac25\end{bmatrix}}_{\mathbf A}\begin{bmatrix}-3\\-2\\5\end{bmatrix}=\begin{bmatrix}0\\0\end{bmatrix}$

is one possible solution.