The Home Office said the arrested men presented themselves as Albanian, Indian, Ukrainian and Kazakhstani.
Local reports said immigration officers clashed with protesters chanting “stop deportations” during the high profile raid.
Circus Street is a major mixed-use project being developed by U+I with Henry Construction acting as main contractor.
A spokeswoman for U+I told The Argus: “We are committed to ethical practice and our contracts with construction contractors require that they abide by employment and immigration law and that all site personnel are qualified to come on site.
“We are aware that Home Office officials have visited the site this morning, and we are unable to provide any further details until their investigation is complete.’’
Henry Construction told the BBC it had “fully complied” with the Home Office investigation.