“I always said you should never trust a bank with property, or a property developer with money,” says Peter Rees. The former chief planner of the City of London should know about such things, having presided over the results of both. Over the last 30 years, he has ushered in a menagerie of their monuments, from the Gherkin and Cheesegrater to the Walkie-Talkie and Heron Tower, during which time he has seen a significant shift in the balance of power. “When I arrived in the job in the 1980s, the big banks were in control of London,” he says. “But now it’s the big house-builders. We’ve gone from being ruled by Barclay’s bank to being controlled by Berkeley homes.”

Left unchecked, the banks went off the rails in spectacular fashion, as they sprayed money into the great mortgage mirage. And now property developers have been allowed to follow suit. Fuelled by the dazzling wealth of investors from Russia, China and the Middle East, who they turned to when the banks stopped lending, their steroidal schemes are causing irreparable harm to our cities.

Across the country – and especially in superheated London, where stratospheric land values beget accordingly bloated developments – authorities are allowing planning policies to be continually flouted, affordable housing quotas to be waived, height limits breached, the interests of residents endlessly trampled. Places are becoming ever meaner and more divided, as public assets are relentlessly sold off, entire council estates flattened to make room for silos of luxury safe-deposit boxes in the sky. We are replacing homes with investment units, to be sold overseas and never inhabited, substituting community for vacancy. The more we build, the more our cities are emptied, producing dead swathes of zombie town where the lights might never even be switched on.

Developers have bounced back from the crash with bigger plans than ever before, acquiring vast areas of land with the ambition to operate like the great estates of yore. Framed with the cuddly terminology of “long-term stewardship” and “adding value”, they are merely mimicking those aristocratic fiefdoms, recasting the city as a network of privatised enclaves. The landed families of Grosvenor, Portman and Cadogan have been joined by a breed of corporate giants like Lend Lease, CapCo and Ballymore. The latter is overseeing the £2bn transformation of Nine Elms into a high-security zone of luxury flats around the new American embassy, that will apparently “draw inspiration from the attractive residential and commercial estates which evolved over time in cities like New York and Boston”. CapCo is building its £8bn kingdom across a 30-hectare swathe of Earls Court, while Lend Lease is ruling Elephant and Castle, Argent is reshaping Kings Cross, and most of Victoria is now controlled by Land Securities. The list goes on.

They have been accompanied, and often outbid, by a newer kind of international development force, supercharged by the untold riches of sovereign wealth funds, national pension funds and the gushing pump of petrodollars. The Qataris, who bailed out the Shard and snapped up the Olympic Village, have been joined by the growing appetite of Malaysian and Chinese investors. Malaysian consortium SP Setia acquired Battersea power station for significantly more than its competitors could muster, while China’s recent property supermarket sweep includes such sites as Wandsworth’s Ram Brewery and a £1bn deal for the Royal Docks. These inflated land deals, with foreign buyers ready to pay over the odds, are spawning a new form of equally oversized and exclusive developments.

Bankers have faced our collective wrath, but what about developers? The economy goes in fickle booms and busts, cycling merrily through bubbles and crises, but cities, built in concrete and steel, generally stay put. What we are making now, we will all have to live with for a very long time. The iniquities of the banking crash have been intricately unpicked, but the wilful destruction of the places where we live and work remains something of a mystery. We may rant and rage against ugly additions to the skyline, but what of the mechanisms that are allowing it to happen? How did it come to this?

The principal reason can be traced to the fact that awarding planning permission in the UK comes down to a Faustian pact. If the devil is in the detail, then the detail is Section 106 of the Town and Country Planning Act 1990, a clause which formalised “planning gain”, making it in the local authorities’ interests to allow schemes to balloon beyond all reason, in the hope of creaming off the fat of developers’ profits for the public good.

Introduced as a negotiable levy on new development, Section 106 agreements entail a financial contribution to the local authority, intended to be spent on offsetting the effects of the scheme on the local area. The impact of a hundred new homes might be mitigated by money for extra school places, or traffic calming measures. In practice, since council budgets have been so viciously slashed, Section 106 has become a primary means of funding essential public services, from social housing to public parks, health centres to highways, schools to play areas. The bigger the scheme, the fatter the bounty, leading to a situation not far from legalised bribery – or extortion, depending on which side of the bargain you are on. Vastly inflated density and a few extra storeys on a tower can be politically justified as being in the public interest, if it means a handful of trees will be planted on the street.

“Council chief executives will allow schemes to be pumped up as much as they can go before they get political push-back from councillors,” says one planning officer from a London borough that has suffered from a recent a spate of towers. “And the worst schemes happen when there is no political resistance at all.”

It is a system that is all too open to political pressure, given that any officer who advises against a new development can be conveniently framed as “anti-growth”, heartlessly preventing a promised tidal wave of new public amenities from flooding into the borough. Based on negotiation and discretion, the result is entirely down to the individual planning officer’s ability to squeeze out as good a deal as they can get, a battle that all too often ends in the developer’s favour.

The results of such botched bargaining can be seen sprouting up across London’s “regeneration” hot-spots, such as Elephant and Castle, where the council is attempting to transform the maligned mess of the roundabout into an “exciting destination”. With shimmering golden fins rising into the skies, the 37-storey tower of One the Elephant promises to “set new standards for contemporary London living”. It is one of the flagship projects by Australian developer Lend Lease in the £3bn transformation of the area. But take a closer look, and it seems the new standards it is setting comprise an impressive ability to avoid providing any affordable housing at all. Such second-class accommodation would of course require its own “poor door” entrance and circulation and, according to a council report: “The cost of construction would increase with the introduction of a further lift, as well as separate access and servicing arrangements.”

Bypassing Southwark’s requirement for 35% affordable housing – which would have meant around 100 units – Lend Lease has instead contributed £3.5m in lieu towards the construction of a community leisure centre next door, which will cost £20m to build. A triumph for the public good, you might think, until you realise that the equivalent cost of building 100 affordable units would have been around £10m, three times what the developer paid. Pressure group 35 Percent – which campaigns for the borough-wide policy of 35% affordable housing to be enforced in Elephant and Castle – estimates that, in the six biggest schemes in the area, developers have avoided paying £265m in off-site affordable housing tariff payments required by policy. And of the 4,282 new homes being built, just 79 will be social rented (ie. managed by registered providers for those on low incomes).

The same story is repeated the other side of town, where Haringey awaits the momentous arrival of Tottenham Hotspur’s new £400m football stadium. This bulbous mothership was promised to bring 200 new homes, half of which would be “affordable”, and an abundance of public benefits to the area. But, once again, the affordable component has been mysteriously waived, replaced with 285 flats for solely private sale, while the Section 106 contribution has been reduced from an agreed £16m to just £477,000 – a token contribution towards transport improvements.

The system has spawned a whole industry of S106 avoidance, with consultancies set up specifically to help developers get out of paying for affordable housing at all scales of development. Section 106 Management, set up by solicitor-turned-developer Robin Furby, is one such company that offers a service to small-scale developers, promising “to establish the profitability of your project and thereby reveal unviable Section 106 obligations”. Its website displays a list of case studies proudly showing how much they have helped developers dodge, and boasting of planning permissions achieved “without any contribution towards affordable housing” at all, saving “tens, if not hundreds of thousands of pounds”.

So what exactly does it mean when a property developer pleads poverty? “If the profit margin for your scheme is pushed to below 17.5% by Section 106 payments, you should talk to us,” says the website. Other consultants promise to safeguard 20% profit margins and upwards, before any Section 106 contributions are even considered. If a scheme is declared “unviable”, it simply means “we’re not getting our 20% profit so why should we bother”.

The power of the policy to leverage affordable housing has been further eroded since the introduction of community infrastructure levy (CIL) in 2010. A non-negotiable fixed-rate tax on new development, CIL was intended to introduce more transparency and give developers a level of certainty about how much they would be expected to contribute towards infrastructural improvements. But, in reality, it has provided another excuse to dodge Section 106 obligations. A further change to the town planning act last year has made Section 106 agreements renegotiable, allowing review and appeal of all existing obligations, in a misguided attempt to promote growth – which simply makes it easier for developers to wriggle out of their promises, as happened in Tottenham and elsewhere.

“Not surprisingly, developers are now even keener to renegotiate the S106 after they’ve got planning permission, finding they can’t negotiate the CIL,” says Peter Rees. “In most cases, they manage to prove that they can no longer afford to pay for the affordable housing that they agreed – it’s simply ‘not viable’ any more.” One planning officer puts it succinctly: “There has never been a worse time to give schemes consent, in terms of securing public benefit.”

In all cases, how developers prove what they can afford to pay for comes down to the dark art of “viability”. The silver bullet of planning applications, the viability appraisal explains, through impenetrable pages of spreadsheets and fastidious appendixes, exactly how a project stacks up financially. It states, in carefully worded sub-clauses, just why it would be impossible for affordable housing to be provided, why the towers must of course be this height, why no ground-floor corner shop or surgery can be included, why workspace is out of the question; indeed, why it is inconceivable for the scheme to be configured in any other form. Presented as a precise science, viability is nothing of the sort; it is a form of bureaucratic alchemy, figures fiddled with spreadsheet spells that can be made to conjure any outcome desired.

“Councils just don’t have the expertise to challenge viability reports,” says one senior planning officer. “We can’t argue back.” Instead, they can commission viability assessments, produced by the same consultants that work for developers, to determine whether the report is accurate – but not to propose an alternative. The figures may well stack up, but it doesn’t mean the scheme could not be designed in a different way, which would still guarantee the developer’s 20% profit margin.

“You only have to modify one of the variables very slightly to get completely different outcome,” says one planning consultant. “You can very easily go from something being rip-roaringly viable to completely unviable by tweaking something very modestly. If a planner doesn’t understand that, they’re not going to do very well.”

Evidence suggests that is all too often the case, judging by the number of planning officers’ reports that diligently conclude a scheme would simply be unviable if it was obliged to fulfil the policy objectives. With calculations often undisclosed for reasons of commercial confidentiality, councils are forced to blindly accept the developers’ figures as the ultimate de facto truth, allowing their own policies to be flagrantly breached.

“I’ve never been confident in reports that I’ve received on viability,” says one planning officer, describing how the big property consultancies operate as something of a cabal, with one wary of challenging another’s figures. “Every consultant that’s advising a local authority is hoping to advise a developer tomorrow. If they put the boot in on a big development scheme, they simply won’t be hired again.”

A relatively new field, viability has been given increasing weight by the government’s National Planning Policy Framework, introduced in 2012, which slashed 1,300 pages of policy down to 65, as part of the coalition’s triumphant bonfire of red tape. The NPPF introduced a “presumption in favour of sustainable development”, which sounds innocuous enough – but as Rees points out, “the definition of ‘sustainable’ has nothing to do with green issues or energy at all. It means one thing: commercially viable.”

Immune from public scrutiny, viability assessments have rightly come under fire for clouding the accountability and transparency of what should be a statutory public process. Their confidentiality is closely guarded, in order to preserve developers’ trade secrets, but where the sale of public assets is concerned, there is increasing pressure for the books to be opened.

One such case recently ended in victory for housing campaigners, when after two years of fighting, which culminated at a tribunal, Southwark Council was ordered to disclose the viability assessment produced by Lend Lease over its controversial redevelopment of the Heygate Estate. The 15-year project is seeing more than 1,200 mainly social-rented homes on the post-war estate replaced with over 2,300 units, only 25% of which will be classed as affordable, with just 79 flats for social rent. Many leaseholders were subject to compulsory purchase orders so low they have been forced to move to the far reaches of outer London, their decent-sized two-bed flats valued at under £150,000, while the new homes of “Elephant Park” will be sold for prices reaching £420,000 for a one-bed apartment.

The figures explaining why this was the only feasible way to develop the site were safely locked away in the viability appraisal, which Southwark fought tooth and nail to keep secret. The borough has been particularly keen to keep financial details under wraps since it accidentally disclosed it had sold the entire nine-hectare site for just £50m, having spent £44m on moving residents out – while estimating its gross development value at £990m.

“Without some commercially sensitive information remaining private, developers could simply refuse to work with councils, leaving boroughs without the housing and regeneration we all need,” says a spokeswoman for Southwark Council. The borough brought a legal challenge against a decision by the Information Commissioner’s Office last year ordering the council to disclose the full details of the viability report, after a freedom of information request was denied. Southwark argued that full disclosure would “damage regeneration”, while Lend Lease, in a defence that verged on farce, pleaded the human right to “peaceful enjoyment of its possessions”, arguing that disclosing the viability assessment would amount to “unjustified interference with this enjoyment”.

The tribunal concluded that the information must be disclosed, stating that “the importance … of local people having access to information to allow them to participate in the planning process outweighs the public interest in maintaining the remaining rights of Lend Lease”. It sets an encouraging precedent for campaign groups battling similar situations elsewhere, from Greenwich Peninsula to Earls Court – where the information commissioner has supported further disclosure of viability assessments on gargantuan regeneration schemes.

The Heygate decision comes after increased scrutiny of Southwark council’s cosy relationship with Lend Lease, following reports in Private Eye of perks enjoyed by Peter John, the Labour leader of the borough, at the expense of the Australian giant. From a pair of £1,600 Olympic opening ceremony tickets to a £1,250 trip to the lavish Mipim property fair in Cannes, these sponsored outings were reported to have joined a lengthy list from the previous year of Proms tickets and dinners at the Ivy, paid for by at least 10 other companies.

Developers getting into bed with local authorities might usually happen behind closed doors, but at Mipim the conspicuous chumminess was proudly on show along the Croisette for all to see. In the wake of headlines decrying public money being spent on councils attending the champagne-soaked jamboree, their private “development partners” have been more than willing to step in and foot the bill. With a borough’s presence at Mipim costing up to £500,000, developers happily pay for glistening city models, trade show booths and yachts, where cakes iced with their logos are handed out by mayors. More than 20 local authorities took part this year, with developers sponsoring everything from a “Croydon on the beach” cocktail party to an entire “Manchester bar”, where public-private relationships could be cemented by free-flowing booze.

“The boroughs might be proud that they’re not here at the public’s expense,” says housing campaigner Jake Freeland, who held a protest in Cannes this year. “But that’s precisely the problem. They’re in the pockets of the investors, and they’ve come here to sell off our city.”

Developers have long thrown parties and funded foreign trips as a way of lubricating their plans through the system, but the quest for permissions now extends into the statutory planning process itself, through the rise of deals known as planning performance agreements. Introduced to help fast-track large, unwieldy schemes through the system, PPAs see the applicant pay for a new dedicated position in the council’s planning team to focus solely on their application, guaranteeing a faster turnaround and a better “bespoke” service.

Capital and Counties Properties (Capco) paid over £2m to Hammersmith and Fulham council under a PPA to have its £8bn redevelopment of Earls Court assessed, while similar deals were reached for Westfield and Hammerson’s £1bn plans for another mega-mall in Croydon, as well as Argent’s £2bn redevelopment of King’s Cross.

“There’s nothing wrong with planning performance agreements,” says one planning officer. “It’s just like allowing people to travel club class. You pay for a better service.” Quite whether club-class planning should be offered by a statutory pubic service is questionable, but developers have few qualms about throwing money at an authority, spitting out as many applications and fees as are necessary to see a project through. “We pay vast sums of money to have things determined quickly,” says the director of one major development company. “We pay the planner’s salary, cover their lawyers’ fees and everything, but we wouldn’t expect preferential treatment. It’s not a bribe.”

Under the coalition’s localism agenda, the wheels for private-sector encroachment into public planning have been further oiled, with the introduction of neighbourhood plans. Presented as a means of empowering communities, they have in fact left the door wide open for canny developers to move in, host a few community coffee mornings with felt-tips and post-it notes, and engineer a plan to their own advantage. There is no requirement for those who draw up the plan to even reside in the neighbourhood and, although they need a 50% “yes” vote at referendum, there is no requisite minimum turnout.

But such a tactic would require at least cursory engagement with the community and the council, something which many developers are increasingly choosing to bypass altogether. Since the introduction of the NPPF, there has been a sharp rise in the number of planning applications won on appeal, as many applicants choose to go straight to the inspectorate, conscious of the new “presumption in favour” of development.

Rather than being the last resort option, after negotiations with the local authority have broken down, the process of planning by appeal has become a tactic in itself. One developer is particularly candid on the matter: “Planning decisions are so often the result of political wrangling at committee anyway,” he says. “Why would you waste months negotiating something to get the planning officer on side, when they can’t guarantee delivery at planning committee?” On appeal, it comes down to a battle between planning lawyers, the judgement often determined by who can afford the best representation. When the Rolls Royce legal team of the private developer meets the quivering case officer of the emasculated public sector, its not hard to guess the outcome.

Developers with bigger ambitions are choosing to bypass the local authority in a different way, by going straight to the top and playing for a “call-in” – waving their schemes under the nose of the mayor of London or secretary of state. Such a situation has emerged at Mount Pleasant in north London, where the Royal Mail Group has proposed a fortress-like scheme of 700 flats, only 12% of which will be affordable. The site straddles the boundary between Camden and Islington, both of which have a target of 50% affordable housing. Boris Johnson ignited local fury when he called the scheme in earlier this year to be determined by his planning team, describing it as a “beautiful design … and a wonderful place to live” before the local boroughs had even turned it down.

“It is hardly surprising that the mayor has called this in,” says Duncan Bowie, lecturer in planning at the University of Westminster. “The mayoral planning process is based entirely on achieving the maximum number of housing units on any given site, aimed at selling to an international market. The London-wide target of building 42,000 new units per year is predicated on a lot of very high density developments that don’t even comply with the mayor’s own policies on density.”

The same thing happened at Convoys Wharf in Deptford, where a £1bn proposal for 3,500 units (of which just 15% will be affordable), in the form of three towers rising up to 40 storeys, was called in by the mayor after the Hong-Kong based developer wrote a blustering letter complaining of planning delays. The scheme was approved in April, against the advice of the local authority and the cries of heritage groups.

“It’s common practice to play the mayor off the borough,” says one senior planning officer. “We recently had one vastly oversized scheme that we’d spent months trying to tame, then we had a meeting with the GLA planning team, and their first response was ‘why not make it taller?’” Driven by tick-box housing targets, the GLA merrily rubber-stamps whatever comes its way, yet most of these schemes are doing nothing to help the housing crisis, given the fraction of “affordable” homes they include are still out of reach of most, at up to 80% of market rent.

“Developers have quickly latched on to the fact that, even if they can’t get local authorities to approve schemes, they can get them through the mayor or the government,” says Peter Rees. “The bigger the better. And they know that they’ll happily allow towers to be built outside designated clusters.”

As deputy prime minister, John Prescott personally approved both the Shard and the Vauxhall Tower, the latter against the decision of the planning inspector and after strong warnings from his advisers that it “could set a precedent for the indiscriminate scattering of very tall buildings across London”. How right they were. With a 50-storey shaft already on the skyline, the council was in no position to refuse further skyward ambitions. The GLA, keen to seem “strategic”, quickly declared the area a “cluster”, beckoning in a thicket of towers and opening the floodgates for the emerging Dubai on Thames. A wall of glass stumps is beginning to sprawl along the river from Wandsworth and Battersea to Nine Elms and Vauxhall – and beyond.

“It is an absolute fiasco,” says Mark Brearley, professor at Cass Cities and former director of Design for London. “It is the outcome of not really taking much notice of plans and being fairly relaxed about negotiating the best outcome, and not placing too many obligations on developers. Nothing hangs together as a result, nothing makes sense at ground level. As a piece of city it’s a farce.”

A similarly galumphing form of urbanism is appearing across London, from the gauntlet of City Road to Stratford High Street. Many of the worst offenders are the result of our slippery two-stage planning system, in which general outline permission can be given, while further details are postponed to a later “reserved matters” application. In a system based entirely on negotiation, it is a fair way of allowing developers to test the water and see what they can get away with, before spending money on detailed work. Yet it also allows crucial elements, like ground-floor uses, the location of entrances, the nature of materials and even massing and bulk, to slip through the net, allowing designs to be watered down to pale imitations of what had been agreed. And the hands of the local authority are hopelessly tied.

“Once an outline permission is granted, it makes it very difficult for us to refuse a scheme further down the line,” says one officer. In Stratford City’s “International Quarter”, part of the promised spoils of the Olympic legacy, consented tower proposals have recently gained a substantial number of extra storeys. Similarly in Wandsworth, a proposed pair of towers have put on a growth spurt and lost their planned mix of uses, reverting entirely to high-end flats.

Conditions that have been agreed are relentlessly renegotiated at reserved matters stage. Good architects are employed to win outline planning, then ditched for a cheaper alternative; high-quality materials are substituted for flimsy plastic panels – all in the name of viability.

Just like the banking crisis, the problem of botched urban development has long been encouraged by a system that is open to exploitation and all too susceptible to careless regulation. But it is also not something that can be easily fixed. “There’s only so much mileage in vilifying developers or planners,” says Brearley. “Making cities is imperfect and messy, and has been for thousands of years. But we should be able to do better than this.”

It comes down, he thinks, to the fact the UK planning system is overly reliant on individual negotiation between private developer and public servant, which is usually far from a level playing field. “It makes a very opaque and confusing system that relies on having people that are very sophisticated at brokering deals,” Brearley adds. “And those people will generally settle in places where they’ll earn more money. The people negotiating on behalf of the public are simply not sophisticated enough.”

One former planning officer is frank about the reality of the imbalance in our confrontational system. “If you throw enough resources at a planning application, you’re going to manage to tire everyone out,” he says. “The documentation gets more and more extensive, the phone calls get more frequent and more aggressive, the letters ever more litigious. The weight of stuff just bludgeons everyone aside, and the natural inclination is to say, ‘Oh yeah okay, I’ve had enough of this one,’ and just let it through. It’s like a war of attrition.”

And it is a war in which the side representing the public interest has been systematically drained of expertise. The number of architects employed in the public sector has fallen from over 60% to less than 10% over the last 30 years, while planners have been relegated to third- and fourth-tier officers, with some boroughs contracting the service out altogether. As part of the Farrell Review into architecture and the built environment, a “Plan First” initiative has been proposed, by GLA regeneration manager Finn Williams, on the model of Teach First, to try to lure the best graduates into planning. But it faces an uphill struggle to overturn the years of neglect and transform a system that is fundamentally anti-plan-making.

“To this day our planning system is the wrong way around,” says Rees. “It evolved to protect the countryside from the encroachment of the towns, rather than to make the cities better. It isn’t about building great places, it’s about protecting non-places.” And in the process, it has allowed our cities to cannibalise themselves and become those non-places it set out to protect.

Bullied and undermined, planning authorities have been left castrated and toothless, stripped of the skills and power they need to regulate, and sapped of the spatial imagination to actually plan places. As one house-builder puts it simply, “The system is ripe for sharp developers to drive a bulldozer right through.” And they will continue to do so with supercharged glee, squeezing the life out of our cities and reaping rewards from the ruins, until there is something in the way to stop them.