Answer:
Technically, the largest ever marine predator known thus far - the blue whale - is, for now, alive and well, but I know what you mean. You’re not talking about filter-feeders, or fish-eaters, you mean macropredators.
The last truly enormous macropredator in our oceans was probably Carcharocles megalodon, which went extinct about 3.6 million years ago. Other recent examples include the terrifying sperm whale Livyatan melvillei (which disappeared 8.9 mya) and other megatoothed sharks like C. chubutensis.
So, why did the giants go away? The answer, in truth, is likely multi-faceted. Here are some major promising hypotheses, or at least contributing factors:
Whales have declined. Giant sea carnivores like C. megalodon and Livyatan relied on marine mammals - specifically baleen whales. During the Pliocene, the Americas connected and the Central American Seaway was closed. This seems to have triggered a significant decline in the diversity of tropical whales, and would thus limit food availability for huge, energy-hungry predators.
Things got too cold. The extinction of megalodon and other giant ocean predators roughly coincides with the trend of decreasing temperatures the Pliocene brought. Giant sharks in particular, not being fully warm-blooded, would suffer from this, and it would also mean a decline in food supply. You can even see a sudden cold snap just about when Carcharocles went extinct, 3.6 mya.
Smaller predators outcompeted them. The niche of Carcharocles and the macroraptorial sperm whales is arguably still filled today - by the killer whale. Many orca pods specialize in hunting whales or other marine mammals. However, killer whales and their ancestors were perhaps more opportunistic, fast, intelligent and indeed adaptable than the massive, solitary hunters.
Ironically, the largest ever marine predator owes its existence to the demise of the largest ever marine macropredators. Since the extinction of macroraptorial sperm whales and Carcharocles megalodon, baleen whales have been gradually increasing in size, for they no longer need to be as fast and agile. Because of this, we have the privilege of sharing our world with the largest animal known to have ever lived - the blue whale.
