The bus, which was stationary at the time on a busy road, was buried under the rubble in Gwangju, some 270 kilometers (168 miles) southwest of the capital, Seoul.
A further eight people were injured, the ministry added.
The 16 demolition workers on the site are safe and did not sustain any injuries as the building collapsed.
Approximately 190 firefighters were deployed for search and rescue, and police joined the rescue effort, the ministry said.
The reason for the collapse was unclear, Reuters reported, citing officials at the Gwangju Fire and Safety Headquarters. The Ministry of Land, Infrastructure and Transport will investigate the cause of the collapse.
South Korea has struggled with a bad safety record of infrastructure — a department store collapse killed more than 500 people in 1995, and a bridge collapse in 1994 killed 49 people. In recent years, the country has tried to improve its safety record.
This story has been updated to reflect the number of injuries.