So this passage in question is chapter 23 pg 304
Leading into what he says, jem explains to scout that there are four kinds of folks, which leads them to talk about their family's background leading to the conversation of learning how to read and write where scout goes on and tells him "no everybody's gotta learn, nobody's born knowin'. That walter's as smart as he can be, he just gets held back sometimes because he has to stay out and help his daddy. Nothin's wrong with him. Naw, jem, i think there's just one kind of folks. Folks."
Which leads us to Jem saying "thats what i thought, too when i was your age. If there's just one kind of folks, why can't they get along with each other? if they're all alike, why do they go out of their way to despise each other? Scout, I think I'm beginning to understand why Boo Radley's stayed shut up in the house all this time...its because he want to stay inside."
My interapation of this is that Jem means the he thought that the trial wasn't equal. The thought that the world isn’t as good as he thought and believed. The Tom Robinson’s trial made it clear which made Jem come upon this relization. That the staying inside maybe meant he (boo radley) maybe was scared it may not be a very pleasant world, not fair and judgemental towards him
Hope this helps :)