The Blues were without their top scorer after a training ground dispute over his fitness, amid reports he has been offered a lucrative move to China.
That saga did not affect them at the King Power Stadium however, with Eden Hazard setting up Alonso to fire home after only six minutes.
The Spanish wing-back added his second early in the second half when his shot from the edge of the area was deflected in.
Pedro made sure of the points with a deft header from Willian’s cross.
Leicester, lining up in a new-look 3-5-2 formation, struggled to find a way back into the game.
Ahmed Musa had tested Chelsea keeper Thibaut Courtois before his side fell behind, but the defending champions did not manage another shot on target.
Before kick-off, Chelsea manager Antonio Conte had played down any row involving Costa, claiming he was not in his squad because of a back injury.
Whatever the truth behind Costa’s absence, the Blues showed again that they can win without the Spain striker, who has scored 14 goals and provided five assists for them this season.
Chelsea initially had to work harder for the points than they did when Costa was suspended in their 3-0 victory over Bournemouth on Dec. 26.
It was the only other league game he has missed in 2016/2017.
Again they used Hazard to lead the line rather than turn to £33m summer signing Michy Batshuayi, who was left on the bench and still awaits his first league start.
Alonso, rather than Hazard, will get the headlines on this occasion, but Chelsea’s dominant performance was further evidence it will be hard to dislodge them from top spot.
This was Chelsea’s 12th clean sheet in 21 games and, even without Costa’s belligerent presence upfront, they remain difficult to break down and dangerous when they come forward.
Conte’s side had seen their 13-game winning streak ended by Tottenham in their previous Premier League game, but that remains their only defeat since the end of September.
Spurs’ emphatic win over West Brom earlier on Saturday had reduced the Blues’ lead to four points, but if Chelsea are feeling any pressure in the title race they did not show it here.
Their form is becoming ominous for the chasing pack, and Liverpool — their nearest rivals before the weekend — can ill afford to lose any more ground.
They face Manchester United at Old Trafford on Sunday.
Foxes manager Claudio Ranieri, named FIFA’s coach of the year this week, attempted to match Chelsea’s formation by playing with wing-backs of his own.
But his experiment did not pay off as Albrighton and Ben Chilwell failed to make the same impact as their opposite numbers, Alonso and Victor Moses.
To compound their woes, the Foxes also struggled to create chances.
England striker Jamie Vardy was back from suspension to lead their attack, but he made little impression apart from playing one dangerous cross and did not manage a single shot.
Leicester are still six points above the relegation zone in spite of this defeat.
But, with Riyad Mahrez, Islam Slimani and Daniel Amarty away at the African Cup of Nations (AFCON), Ranieri must find a better blend with the players still at his disposal.
Chelsea host struggling Hull on Jan. 22, while Leicester travel to Southampton on the same day, hoping for their first away win of a disappointing domestic season.