Rossi, back this season after successive knee injuries effectively side-lined him for two years, led Fiorentina's revival as they overturned a 2-0 deficit, leaving Juventus coach Antonio Conte seething over his team's collapse.
Inter had goalkeeper Samir Handanovic sent off in the fifth minute, then saw replacement Juan Pablo Carrizo save a penalty with his first touch. They twice fell behind, hit back to lead 3-2 only to be denied a win by last-minute free kick at Torino.
There was plenty going on elsewhere with nine penalties awarded in Sunday's seven games and a first-ever top-flight win for Serie A debutants Sassuolo, who splashed their way past Bologna 2-1 on a waterlogged pitch.
Sampdoria converted two penalties, the second in the sixth minute of stoppage time, for a 2-1 win at Livorno, their first of the season, while Brazilian Jorginho converted a brace of spot kicks for Hellas Verona as they came from behind to beat Parma 3-2.
AS Roma, who made it eight wins out of eight by beating Napoli 2-0 on Friday, lead the table with 24 points, followed by Napoli and Juventus on 19. Surprise package Hellas Verona are fourth (16) followed by Inter and Fiorentina (15).
"We suffered a 10-minute nightmare," bemused Juventus coach Conte told reporters. "I thought it was all over bar the shouting. We dominated the game for long periods and it all went down the drain in 10 minutes."
Juventus had dropped only two points all season and went ahead when Carlos Tevez collapsed dramatically after minimal contact from Gonzalo Rodriguez before firing home a penalty in the 37th minute. Three minutes later, Juan Cuadrado's mis-hit clearance gifted Paul Pogba a second.
Fiorentina's fightback began with another controversial penalty in the 66th minute when Matias Fernandez was bundled over and Rossi scored from the spot.
Juventus then collapsed, conceding three goals in five minutes. Rossi levelled in the 76th minute and two minutes later, the Juventus defence left Joaquin Sanchez all alone and the Spaniard swept home the equaliser.
Then Colombian international Cuadrado led a counter-attack and slipped the ball to Rossi who drove home his third.
"This could be the most beautiful win (of my career)," said Fiorentina coach Vincenzo Montella. "In the first half, we didn't look like scoring but, fortunately, football is a marvellous sport and everything can change in a breath.
Inter's adventures began when Handanovic was dismissed for hauling down Alessio Cerci in the area. Mateo Kovacic had to make way for Carrizo and the Argentine made a spectacular entrance when he stopped Cerci's penalty with his first touch.
There was nothing Carrizo could do when Alexander Farnerud put Torino ahead in the 21st minute but Inter were reprieved when Fredy Guarin levelled before the break with a bicycle kick.
Torino regained the lead through Ciro Immobile but were stunned by two goals from Rodrigo Palacio. With Inter about to celebrate a remarkable win, Nicola Bellomo saved the home team's blushes when his intended cross from a free kick floated over Carrizo and into the goal in the 90th minute.
Sassuolo set up their win when Domenico Berardi converted a penalty and Antonio Floro Flores scored from long range in the first 20 minutes.
Alessandro Diamanti pulled one back for winless Bologna with another penalty in the 34th minute but their hopes of equalising were thwarted by the torrential rain which made the pitch virtually unplayable.