The partners blamed soaring costs and the new second staircase rule for the decision to abandon the Perceval House project.
Instead, the council now plans to retain and retrofit the building to make it an office building and community space.
Council leader Peter Mason said the decision was the right decision and removed years of uncertainty and new risk to the council.
He said that since the redevelopment project was originally proposed in 2014, significant events had substantially changed the risks and viability of the scheme.
These included an increase in materials and construction costs and most recently, changes required to the design and structure of the planned buildings to ensure that they complied with the latest fire safety regulations.
He said that the second staircase changes were very difficult to incorporate into the existing designs and therefore significantly affected the financial viability of the scheme.
A Vistry spokesperson said: “The economic and technical conditions effecting the project have been exceptional and we are grateful to have worked closely with the council and Greater London Authority to attempt to mitigate these.
“We fully support the council’s new strategy which will enable fast delivery of important facilities for the council and community.”
Work to make Perceval House a modern civic centre, will start by the end of 2023.