How S.F. became the least affordable city in America, and what we can do about it now.

My friends keep moving to Oakland. Gone from San Francisco for greener pastures and cheaper rents, because it’s just gotten too hard, by which I really mean too expensive. Their move signals that something has gone terribly wrong in this most progressive of American cities.

In some ways, we came by the problem innocently. San Francisco had the good fortune to be one of the very few 19th century industrial cities to successfully make the transition to a new, post-industrial economic base. It wasn’t just bohemians who set up shop here—all kinds of entrepreneurs and creative business people decided to call San Francisco home. As wave after wave of older industrial jobs moved out of town, new types of work were created to replace them.

At the same time, San Francisco was a great place to live. Partly from historical inheritance and partly from the work of activists who chose to make the city the focus of their activism, the city remained a walkable, urban paradise compared to most of America.

A great quality of life and a lot of high-paying professional jobs meant that a lot of people wanted to live here. And they still do.

But the city did not allow its housing supply to keep up with demand. San Francisco was down-zoned (that is, the density of housing or permitted expansion of construction was reduced) to protect the “character” that people loved. It created the most byzantine planning process of any major city in the country. Many outspoken citizens did—and continue to do—everything possible to fight new high-density development or, as they saw it, protecting the city from undesirable change.

Unfortunately, it worked: the city was largely “protected” from change. But in so doing, we put out fire with gasoline. Over the past two decades, San Francisco has produced an average of 1,500 new housing units per year. Compare this with Seattle (another 19th century industrial city that now has a tech economy), which has produced about 3,000 units per year over the same time period (and remember it’s starting from a smaller overall population base). While Seattle decided to embrace infill development as a way to save open space at the edge of its region and put more people in neighborhoods where they could walk, San Francisco decided to push regional population growth somewhere else.

Whatever the merits of this strategy might be in terms of preserving the historic fabric of the city, it very clearly accelerated the rise in housing prices. As more people move to the Bay Area, the demand for housing continues to increase far faster than supply.

There’s a lot of housing under construction now, and for the next couple of years, we’ll see more built. But a few years of strong housing production, building out neighborhood plans that the city has worked on for the last two decades, is going to be too little, too late to undo the larger trend. Absent any transformative approaches, new housing construction is likely to return to its normally low levels after the current round of building is finished.

Railing against Google buses, fancy restaurants or new condos—the visible signs of gentrification—will do nothing to stop San Francisco from becoming more expensive. These are not causes of the rising rents; they are symptoms. The root cause is that many people have chosen to live in San Francisco, and we are now all competing with one another to bid up the rents. As long as this remains a desirable place to live in a region that is producing a lot of jobs — while at the same time we fail to produce enough housing to accommodate the demand — then housing prices will continue to rise.

San Francisco needs more affordable housing. The problem is, subsidized, below-market-rate units are too expensive to build to help very many people. It costs around $250,000 in government subsidy per unit. You can get a sense of the scale of the cost based on how many people you want to help. Subsidizing affordable homes for 10,000 families comes at a price of tag of $2.5 billion. To subsidize affordable homes for 100,000 people would cost $25 billion.

So yes, we should build as much subsidized affordable housing as we can. But most people will never get to live in one of these units. It is not a strategy that will have an effect on the housing costs for the vast majority of the people trying to make a go of it in San Francisco. If we want to actually make the city affordable for most people—a place where a young person or an immigrant can move to pursue their dreams, a place a parent can raise kids and not have to spend every minute at work—we have to fix the supply problem.

There is room for San Francisco to grow for many more decades. Our estimate is that if we could produce 5,000 units a year for a sustained period of time, that would be enough to make a real impact on affordability. This kind of infill development—if it is well-designed and well-located—would be good not just for housing costs, but also for moving us toward a society that drives less.

There may also be some more innovative approaches that are worth experimenting with, to try to create more housing specifically targeted to middle-income households. Perhaps we should pick one neighborhood with great transit where we say, all housing produced here has to be car-free. Perhaps we can get some more neighborhoods to allow micro-units and new secondary units (sometimes called “in-law” units) as “naturally affordable,” un-subsidized types of housing. And maybe there are tweaks to the city’s “inclusionary housing” program that can generate more middle-income units.

The rise in housing costs is having big effects on the broader region as well. We should be thinking about San Francisco in the context of the rest of the urbanized Bay Area as we look for ways to make life more affordable. As people get priced out of their first-choice neighborhoods, they move to new ones, and they in turn make it safe for subsequent waves of gentrifiers. This process has now moved from San Francisco to Oakland, and perhaps other parts of the Bay Area as well.

Whether the gentrification process is good or bad for neighborhoods, and for the lower-income people who live there, is something that can be debated endlessly. But what is strikingly different about the Bay Area in contrast to a place like New York is the fact that New York has so many more walkable, pre-war neighborhoods located on rail transit, within easy commuting distance of Manhattan. When New York neighborhoods like Soho and the Village got too expensive, for example, the Lower East Side became a major center for artists and other members of the cultural avant-garde. When the Lower East Side got too expensive, people went across the East River to Williamsburg. Next came Fort Green, Dumbo, Red Hook and other neighborhoods in Brooklyn that were still cheap. But as every spot in Brooklyn with a good rail connection to the city gets more expensive, there still is Queens, the Bronx, Newark, the towns up the Hudson — walkable neighborhoods in every direction.

As expensive as Manhattan is, and as far along into the gentrification process as the many surrounding communities are, there are still many places to go within the New York orbit to have an affordable, urban way of life.

In the Bay Area, there are far fewer options that fit the criteria of walkable, transit-proximate and affordable. For many of my friends, there is just one: Oakland. This is what people mean when they say Oakland is the Brooklyn of the Bay Area. It’s the next stop on the train, it’s cool, it’s where young people go now.

If we were one city, San Francisco could spend some of its incredible wealth on the things Oakland needs, like hiring more cops and teachers, not to mention more transit connections between the two cities. This is not an argument for annexation but a call to think about the answers to our problems from a regional perspective. We can’t solve affordable housing or transit access within the limits of any one city.

Oakland is perched in a paradoxical and highly charged position right now, dealing with problems of serious poverty—not enough economic opportunity for its residents, not enough money to cover basic public services—while at the same time facing fears of rising rents and displacement if too many members of the San Francisco exodus move there. In the Oakland neighborhoods that are near BART or have good schools, or both, the housing prices are already climbing quickly.

San Francisco can’t do it alone, but it needs to do its part. The three big cities of the region (San Francisco, San Jose, and Oakland) have disproportionately more opportunity and more responsibility to absorb a major percentage of the region’s population growth for the simple reason that they have the room and they have the existing transit infrastructure. But smaller cities should be asked to do their part, too. If San Jose succeeded in becoming more urban, and if smaller cities such as Palo Alto and Berkeley were willing to grow more, some of the pressure would be relieved from San Francisco. We need our own “metropolitan” strategy that ties the region together in better ways, and creates walkable, diverse communities in more locations.