The men working for Hawkeswood Metal in the Nechells area of the city where crushed when a large retaining wall used to contain tonnes of metal collapsed.
A sixth man managed to scramble free from the rubble and was taken to Heartlands Hospital with a broken leg.
Officers were called by West Midlands Ambulance Service at 8.45am to recycling centre in Aston Church Road.
Emergency services attended but the men could not be saved and they were pronounced dead at the scene.
A specialist team of firefighters is now on the scene with heavy lifting gear and cutting equipment to recover the bodies of the men.
An investigation into what happened is now underway. The Health and Safety Executive has been notified and are assisting officers at the scene.
Detective Superintendent Mark Payne, from Force CID, said: “We are still in the very early stages of this investigation to determine exactly what happened this morning.
The scene in Aston Church Road, Nechells
“Sadly five people have lost their lives and we have several teams of specialist officers supporting their families at this time.
“This has been a very traumatic incident for those who were at the site this morning and we will continue to work alongside the HSE and the other emergency services at the scene.”