I don't usually do calculus on Brainly and I'm pretty rusty but this looked interesting.
We have to turn K into the limits of integration on our integrals.
Clearly 0 is the lower limit for all three of x, y and z.
Now we have to incorporate
x+y+z ≤ 1
Let's do the outer integral over x. It can go the full range from 0 to 1 without violating the constraint. So the upper limit on the outer integral is 1.
Next integral is over y. y ≤ 1-x-z. We haven't worried about z yet; we have to conservatively consider it zero here for the full range of y. So the upper limit on the middle integration is 1-x, the maximum possible value of y given x.
Similarly the inner integral goes from z=0 to z=1-x-y
We've transformed our integral into the more tractable
![\displaystyle \int_0^1 \int_0^{1-x} \int _0^{1-x-y} (x^2-z^2)dz \; dy \; dx](https://tex.z-dn.net/?f=%5Cdisplaystyle%20%5Cint_0%5E1%20%5Cint_0%5E%7B1-x%7D%20%5Cint%20_0%5E%7B1-x-y%7D%20%28x%5E2-z%5E2%29dz%20%5C%3B%20dy%20%5C%3B%20dx)
For the inner integral we get to treat x like a constant.
![\displaystyle \int _0^{1-x-y} (x^2-z^2)dz = (x^2z - z^3/3)\bigg|_{z=0}^{z= 1-x-y}=x^2(1-x-y) - (1-x-y)^3/3](https://tex.z-dn.net/?f=%5Cdisplaystyle%20%5Cint%20_0%5E%7B1-x-y%7D%20%28x%5E2-z%5E2%29dz%20%3D%20%28x%5E2z%20-%20z%5E3%2F3%29%5Cbigg%7C_%7Bz%3D0%7D%5E%7Bz%3D%201-x-y%7D%3Dx%5E2%281-x-y%29%20-%20%281-x-y%29%5E3%2F3)
Let's expand that as a polynomial in y for the next integration,
![= y^3/3 +(x-1) y^2 + (2x+1)y -(2x^3+1)/3](https://tex.z-dn.net/?f=%3D%20y%5E3%2F3%20%2B%28x-1%29%20y%5E2%20%2B%20%282x%2B1%29y%20-%282x%5E3%2B1%29%2F3)
The middle integration is
![\displaystyle \int_0^{1-x} ( y^3/3 +(x-1) y^2 + (2x+1)y -(2x^3+1)/3)dy](https://tex.z-dn.net/?f=%5Cdisplaystyle%20%5Cint_0%5E%7B1-x%7D%20%28%20y%5E3%2F3%20%2B%28x-1%29%20y%5E2%20%2B%20%282x%2B1%29y%20-%282x%5E3%2B1%29%2F3%29dy)
![= y^4/12 + (x-1)y^3/3+ (2x+1)y^2/2- (2x^3+1)y/3 \bigg|_{y=0}^{y=1-x}](https://tex.z-dn.net/?f=%3D%20y%5E4%2F12%20%2B%20%28x-1%29y%5E3%2F3%2B%20%282x%2B1%29y%5E2%2F2-%20%282x%5E3%2B1%29y%2F3%20%5Cbigg%7C_%7By%3D0%7D%5E%7By%3D1-x%7D%20)
![= (1-x)^4/12 + (x-1)(1-x)^3/3+ (2x+1)(1-x)^2/2- (2x^3+1)(1-x)/3](https://tex.z-dn.net/?f=%3D%20%281-x%29%5E4%2F12%20%2B%20%28x-1%29%281-x%29%5E3%2F3%2B%20%282x%2B1%29%281-x%29%5E2%2F2-%20%282x%5E3%2B1%29%281-x%29%2F3)
Expanding, that's
![=\frac{1}{12}(5 x^4 + 16 x^3 - 36 x^2 + 16 x - 1)](https://tex.z-dn.net/?f=%3D%5Cfrac%7B1%7D%7B12%7D%285%20x%5E4%20%2B%2016%20x%5E3%20-%2036%20x%5E2%20%2B%2016%20x%20-%201%29)
so our outer integral is
![\displaystyle \int_0^1 \frac{1}{12}(5 x^4 + 16 x^3 - 36 x^2 + 16 x - 1) dx](https://tex.z-dn.net/?f=%5Cdisplaystyle%20%5Cint_0%5E1%20%5Cfrac%7B1%7D%7B12%7D%285%20x%5E4%20%2B%2016%20x%5E3%20-%2036%20x%5E2%20%2B%2016%20x%20-%201%29%20dx)
That one's easy enough that we can skip some steps; we'll integrate and plug in x=1 at the same time for our answer (the x=0 part doesn't contribute).
![= (5/5 + 16/4 - 36/3 + 16/2 - 1)/12](https://tex.z-dn.net/?f=%3D%20%285%2F5%20%2B%2016%2F4%20-%2036%2F3%20%2B%2016%2F2%20-%201%29%2F12)
![=0](https://tex.z-dn.net/?f=%3D0)
That's a surprise. You might want to check it.
Answer: 0