The phrases that reflect mercilessness of the sea are:
1-"Their eyes glanced level, and were fastened upon the waves that swept toward them."
They were glancing at the level of the sea changing by the waves, and fastened to fell the security of grabbing to something.
2-"These waves were of the hue of slate, save for the tops, which were of foaming white, and all of the men knew the colors of the sea."
Describing the color of the waves as hue and mentioning that they all knew the colors of the sea, we can determine that these dark waves are pretty scary.
3-"The horizon narrowed and widened, and dipped and rose, and at all times its edge was jagged with waves that seemed thrust up in points like rocks. "
The description of how the movement of the boat and the sea changed the view of the horizon, as well as the comparison made of them thrusting like rocks let's the reader know how rough the sea is.
4-"These waves were most wrongfully and barbarously abrupt and tall, and each froth-top was a problem in small-boat navigation."
The description of how the waves were abrupt and tall let's the reader determine how harsh it is, as well as how it was uneasy to navigate with the froth-top, having a small boat.