Torvald means that if he lets his wife does what she wants or get away with certain acts he will become the laughing stock of the town. For the time it was crucial that the man was the leader of the household so if it looks like he has lost that authority it will damage his reputation.
