Coveleski , the younger brother of pitching star Stan Coveleski, made his major league debut with the Phillies on September 10, 1907, in a playoff series against the New York Giants. Over the course of five days, he notched three victories, including one against Christy Mathewson, earning the nickname "The Giant Killer".The next season, he developed trouble with his arm (and, reportedly, with his appetite for drink) and was traded to the Cincinnati Reds that January. He had another disappointing year, walking 16 batters in one game.

Rick Woodward bought Coveleski's contract from the Reds for $1,000 and put him on the rotation for his 1910 squad, managed by Carlton Molesworth, that would showcase his newly-built Rickwood Field. Covelski was frank with local sportswriters about his disappointment at a move that he considered a demotion, but won over fans with a July 17 no-hitter that ended up as a loss in extra innings. He followed with a no-hit victory on July 31, beginning a streak of wins that dropped his ERA to 1.55. Coveleski got the start for Rickwood's August 18 opening day game against the Montgomery Climbers, and avoided a 2-1 loss on a ninth-inning rally. He finished the season with 21 wins, securing his last against Southern Association champion New Orleans Pelicans. He held their star slugger Shoeless Joe Jackson hitless in four plate appearances for the 1-0 shutout.

Woodward traded the problem-prone star to Chattanooga for the 1911 season, and Coveleski seemed to justify the trade by going 25-37 in his first two seasons there. He regained his composure in 1913, notching a league-leading 28 victories.

Coveleski joined the Detroit Tigers in the 1914 season and pitched over 300 innings, completed 23 of his 36 games, and won 22 games, second in the American League only to Walter Johnson. In four of his five seasons with the Tigers, Coveleski's ERA was under 3.00, and his 2.34 ERA with the Tigers is still the franchise's all-time career record. He played his last game in the majors on June 23, 1918, ending his career with an 81-55 major league record, a career 2.39 ERA, and 511 total strikeouts.

In a 1976 Esquire article, sportswriter Harry Stein published an "All Time All-Star Argument Starter," consisting of five imaginary baseball teams divided by ethnicity. Harry Coveleski was the left-handed pitcher on Stein's Polish team.