The Kings started brilliantly through Bjorn Basson’s individual effort but Edinburgh soon fired back through centre Chris Dean.
The Scots dominated the middle portion of the match and went ahead with a second-half penalty try, followed up with a score for hooker Ross Ford.
But the Kings had other ideas, and, trailing by eight with five minutes to play, the South Africans struck back through Bader-Werner Pretorius and Penxe’s try-of-the-season contender to snatch victory.
The home side started like a train and got the first try in the opening minutes as Basson executed a wonderful bit of skill down the left flank.
Attacking from just inside the Edinburgh half Basson took a pass out wide, chipped in-field to beat the defence and then touched down for an excellent score.
Masixole Banda kicked the extras but Edinburgh would fire back almost immediately, going through phase after phase in the opposition 22 before a delightful inside pass found centre Dean to do the rest.
Van der Walt kicked the conversion, followed by a penalty at the other end, with Edinburgh enjoying most of the possession for the rest of the half only for Banda to kick three more points before the break.
The Kings led 13-7 at the interval but Edinburgh continued to press in the second half, almost getting over from short range through Pietro Ceccarelli only for a knock-on to be given.
The Kings defence held out impressively but was dealt a blow on the hour as blindside CJ Velleman saw yellow and Edinburgh were awarded a penalty.
The South African side had made 180 tackles at this stage as the Scots opted for a scrum, eventually being rewarded with a penalty try to put them 14-13 ahead.
Edinburgh’s dominance continued and they used forward power for their third try, driving off a lineout until Ford was able to snipe over the line from short distance.
The Kings though weren’t done there and attacked in the final ten minutes, with replacement Pretorius powering over with just four minutes on the clock.
Heading into the final minutes the try of the game was still to come, with the Kings receiving the restart and immediately passing wide to another substitute – Meli Rokoua.
Rokoua stepped inside to beat the defence and flew to the halfway line before offloading out the back of his hand to Penxe, who stepped up the pace a notch further and blasted in to the tryline for a famous win.