Tamil Nadu Chief Minister J. Jayalalithaa ordered five ministers to rush to the accident spot to oversee rescue operations.
A police officer in Virudhunagar district told IANS on telephone that the accident happened around 12.30 p.m. in one of the biggest factories in the area.
He said nearly 40 of the 48 sheds in the sprawling warehouse had been destroyed in the blaze.
Thirteen fire tenders battled the flames while more than 25 ambulances ferried the injured to hospitals.
The police officer said thick smoke and continuing blasts in the factory area hindered rescue efforts.
According to police, the fire from one of the sheds quickly spread to the rest of the factory, thanks to a huge stock of raw materials/firecrackers being readied for the upcoming festival season.
Hours after the tragedy, a local official told journalists that around 100 people were still trapped in the factory premises.
According to reports, only 53 workers are known to have come out of the factory after the fire.
Ministers R. Vaithilingam, O. Panneerselvam, Natham R. Viswanathan, K.P. Munusamy and K.T. Rajenthra Balaji have rushed to Sivakasi.
Balaji, minister for information and special programme implementation, told IANS that the chief minister had ordered five ministers to go to the spot and also asked Virudhunagar district officials to act fast in providing medical assistance and rescue efforts.
"We have asked government and private doctors to take all efforts to reduce casualty. All fire tenders and the ambulance vans in the district have been sent to the spot," Balaji said.
Senior district officials and relatives of the factory workers also reached the accident spot.
Tamil television channels showed intermittent explosions in the area, and thick smoke rising to the sky.
Villagers were seen carrying away the injured in their arms or in cloth sheets in the absence of stretchers.
Police said the exact numbers inside the factory was not known. Normally, there are 300 workers in a firecracker factory.
Crackers are produced in small rooms with space for not more than four people. The rooms have four doors so that workers can rush out in case of any emergency.
Cracker units allegedly employ inexperienced workers during the Diwali season in order to meet the demand, which in turn results in such accidents.
Sivakasi is one of the oldest and biggest firecracker manufacturing hubs in India.
Nearly 90 percent of the country's fireworks are made in Sivakasi along with 80 percent of the safety matches.