The plans for the 220-acre Somerdale site had been passed by the Bath and North-East Somerset Council planning committee in September.
Now council officials have signed off final conditions allowing work to start.
Taylor Wimpey’s Bristol office will oversea the construction of the homes in several multi-storey blocks, alongside a sports and leisure club, primary school and healthcare centre.
Work is already under way with the demolition of one of the existing factory blocks, known locally as Block D.
These works will pave the way for the construction and early delivery of the Fry Club, the development’s new sports and social facility.
The entire project is expected to take five years to complete and will create 1,000 jobs during construction.
The factory was closed in 2011 following US company Kraft’s acquisition of Cadbury in 2010.
Housing plan for Somerdale site