Since you mentioned calculus, perhaps you're supposed to find the area by integration.
The square is circumscribed by a circle of radius 6, so its diagonal (equal to the diameter) has length 12. The lengths of a square's side and its diagonal occur in a ratio of 1 to sqrt(2), so the square has side length 6sqrt(2). This means its sides occur on the lines
and
.
Let
be the region bounded by the line
and the circle
(the rightmost blue region). The right side of the circle can be expressed in terms of
as a function of
:

Then the area of this circular segment is


Substitute
, so that 


Then the area of the entire blue region is 4 times this, a total of
.
Alternatively, you can compute the area of
in polar coordinates. The line
becomes
, while the circle is given by
. The two curves intersect at
, so that


so again the total area would be
.
Or you can omit using calculus altogether and rely on some basic geometric facts. The region
is a circular segment subtended by a central angle of
radians. Then its area is

so the total area is, once again,
.
An even simpler way is to subtract the area of the square from the area of the circle.
