Same deal as fathom: one root quietly fanning out into things that look unrelated until you trace them. Last time the surprising payoff was a word for understanding. With "swell," it's slang.
The anchor is Old English swellan, a strong verb meaning to grow, expand, rise up, be inflated. It's solidly Germanic: German schwellen, Dutch zwellen, Old Norse svella all mean the same thing. Unlike fathom, though, nobody has pinned down a clean Indo-European root underneath it. Past Germanic, etymologists mostly shrug and write "origin unknown." So the bedrock sense is just that: to get bigger, to puff up.
The sea
The ocean swell is the easy one. It's that verb turned into a noun. The sea swells, heaves, rises in long rolling motions, so "a swell" is a rising body of water. Nothing metaphorical is happening here. The sea is doing the literal thing the word always meant. That noun sense settles in around the 1600s.
The dandy
Here's where it gets fun, because the word travels through fashion and class to reach us. By the late 1700s and early 1800s, British slang had "a swell" meaning a stylishly dressed, high-status person: a dandy, one of the well-to-do set. The logic is "puffed up." Someone inflated with fine clothes and self-importance, literally swollen with grandeur. The same image still lives in "a swollen head," or in being "swollen with pride."
From that noun came an adjective. "Swell" started to mean fashionable, elegant, first-rate, the kind of thing a swell would own: a swell carriage, a swell party. Then American English sanded the class connotation off and kept the approval. Swell came to mean plain old great, fine, excellent, which is the sense that peaked in early-1900s usage. "A swell guy." "Things are going swell."
So the thread running through all three is rising and puffing up. The sea heaves upward, a person inflates with finery and status, and something being "swell" still carries that leftover whiff of grandeur: good in the sense of impressive, elevated, a cut above. The dandy strutting around in his fine coat and the calm rolling sea are, etymologically, doing the exact same thing your ankle does when you twist it.