2013 was an extraordinary year for those of us who are interested in privacy and data protection. What was previously seen as the domain of paranoid nitpickers has exploded into the public consciousness, shaking international ties and making many people re-evaluate how they live their lives online.

But it would be a mistake to see 2013 purely in terms of Edward Snowden and his revelations. Indeed, months before the former NSA contractor downloaded what he could and fled to Hong Kong, the year began as it meant to go on: by demonstrating the power and dangerous potential of our increasingly open, plugged-in nature.

We, the searchable

In mid-January, Facebook(s fb) introduced Graph Search, a way for the average user to tap into the social network’s web of interconnected human intelligence. Suddenly it was easy to enter queries like “My friends who like Rihanna” — handy for when you’re buying concert tickets, perhaps. It also became trivial for anyone to search for things like “Men who live in Iran and are interested in men.”

As my colleague Mathew Ingram argued at the time, this issue was about the death of obscurity. Any gay person in the homophobic Islamic Republic of Iran who makes their sexuality clear online has already forfeited their own privacy, but they might have been forgiven for thinking that this information would remain partially obscured from the authorities’ gaze.

Instead, Facebook automated the process of querying its users’ relationships with one another, their tastes and everything else with a “public” setting. We were all suddenly that much more readable. At the same time, we were able to get an unprecedentedly hands-on understanding of the links that are so easily drawn between our myriad data points.

Advertisers were given an even more detailed view. In April, Facebook introduced “partner categories,” letting advertisers target users not only on the basis of their “likes” on the site, but also by correlating that with data about what they buy through other web services. “OK, now I’m convinced Facebook is trying to be creepy,” my colleague Derrick Harris wrote.

Limited glimpses

Publicly posted information is one thing, but what about private posts and communications? Most people were aware that law enforcement agencies and governments ordered web companies to give up data on individuals of interest, but the scale of this trend has always been difficult to nail down.

On this front, Google(s goog) has always led the way, and the January 2013 edition of its semi-annual Transparency Report showed the numbers of such requests had increased 70 percent between 2009 and 2012. Twitter, too, gave users an indication of how often the feds knocked on its door. Facebook said at the time that it had no plans to release such information to the public, and Microsoft(s msft) initially hummed its way through calls for a Skype transparency report before giving in to pressure from privacy groups.

However, while Google covered itself with more glory than most on this aspect of transparency, it found itself increasingly under fire in Europe over other privacy-related issues. The company was fined by a German privacy regulator for the 2010 siphoning of people’s passwords and communications by Google’s Street View cars. And it continued to get into all sorts of trouble over its unified privacy policy.

In the eyes of Europe’s data protection authorities, the unification of Google’s disparate services represented an attempt to do new and serious things with people’s information without true permission, connecting previously unconnected pockets of data without making this obvious to users or giving them a meaningful way to say no. A lot of people were unhappy about Google Search and Maps — and Google+ — knowing about their YouTube commenting habits.

However, although European privacy laws are significantly stronger than those in the U.S., a February survey showed that Americans are nearly as keen as Europeans on protecting their online privacy. The Ovum study also demonstrated an overwhelming lack of trust in internet companies’ honesty about data protection; which is interesting, because those companies clearly assumed the public had a very high tolerance for privacy invasion.

In May, Microsoft(s msft) said its Xbox One console would have an always-on microphone, constantly listening to whatever people say near it. Google said Glass would gain the same functionality, which also became a feature of the Moto X smartphone. Apps got in on the act. All listening, all the time.

Glass wasn’t the only wearable computer with eavesdropping potential. At our 2013 Structure Data conference, the CIA’s tech chief, Gus Hunt, said the new breed of fitness trackers were both light on security and heavy enough on sensor data to betray the user’s gender, rough height and weight, and more.

“What’s really most intriguing is that you can be 100 percent guaranteed to be identified by simply your gait – how you walk. Now this could be a really good thing.

Just as disturbingly, a study demonstrated how easy it was to identify someone from just a handful of time and location-based data points. We are that predictable; that easy to pick out from a crowd, based on factors we barely recognize.

Boom

On the 5th of June, Britain’s Guardian newspaper ran a story alleging that the U.S. carrier Verizon(s vz) was handing over call records to the National Security Agency, America’s signals intelligence operation. These records — describing who called who, when, where, for how long — are also known as metadata, a term with which everyone would subsequently become very familiar. My colleague Stacey Higginbotham nailed it right away:

“There are so many questions and angles to this story, but from my perspective the most pressing issue is that as citizens we need to understand that times have changed. Many don’t recognize that our digital data — from cell phones, connected devices and our social media profiles — combined with powerful computing and analytics can create detailed histories of our lives, our habits and our actions.”

The next story was even more explosive. It alleged that the NSA had direct access to the servers of the big U.S. tech firms, such as Google, Microsoft and Apple. This didn’t just affect Americans — now all these companies’ users and customers, anywhere in the world, were clearly at risk of being spied upon by the U.S. authorities.

It was immediately apparent that there would be severe fallout abroad, particularly in Europe, with its strict privacy laws. Within days, European data protection regulators and activists were demanding to know what was happening with EU citizens’ data held on and transmitted through American web services. As well they might. While Snowden had ignited a debate in the U.S. about the constitutional protections that U.S. citizens were supposed to be enjoying in their home country, the American constitution doesn’t grant those protections to anyone else.

As far as the NSA is concerned, there are no limits when it comes to spying on the citizenry of the outside world. And as it turned out, the U.S. had even secretly lobbied for changes in EU legislation to make sure it could keep on spying there.

The European response proved to be chaotic because, while EU-level law trumps country-level law on most things, national security isn’t one of them. It also gradually became clear that many of those countries’ intelligence agencies were in fact in on it — to varying degrees — with the chief culprit being the United Kingdom, whose GCHQ agency is at least as active as the NSA when it comes to tapping the world’s communications.

So, while the European Parliament expressed outrage and wanted to wave a big stick in Washington’s direction, the European Commission knew there wasn’t much it could do, and did little more than talk about “rebuilding trust.”

A few days after the first stories broke, Snowden himself broke cover. His personal circumstances became of intense public interest, which was fair: this was the extraordinary tale of a young man on the run from Hawaii to Hong Kong, who became stranded in Putin’s Russia after the U.S. cancelled his passport. Later, journalist Glenn Greenwald and his partner David Miranda — and even the Guardian (see disclosure below) — would also become stories in themselves, thanks to some rather disturbing harassment from the British authorities.

All juicy stuff. But this element of the story in many ways diverted public attention from what it was Snowden was revealing: that, for us as citizens who use digital communications, everything was not as it had seemed. Since the end of World War II, mass communications have been subject to surveillance by a network of friendly, English-speaking intelligence agencies. With the advent of the web, that data-gathering activity took on new dimensions. And now we knew about it.

Sure, there had been whispers before, but knowledge trumps suspicion.

Slow clarity

Early surveys showed most Americans weren’t that bothered about the NSA recording their cellphone metadata. But the revelations kept on coming, at times shocking even industry experts. As of late December, we have apparently seen the key points of just one percent of the documentation Snowden snuck out of Hawaii. It’s taken a while to digest that relatively small amount of information, and conceptions of the truth have evolved along the way, but here’s a rough approximation of how we now understand things:

• The NSA and its international partners are monitoring most of the world’s communications, mainly but not only by tapping the fiber-optic cables that link everything together. They are doing this with a degree of industry collusion, primarily from the cable operators.
• The purpose of the exercise is to monitor everything, so that when terrorists or other nefarious individuals are identified, they can be easily tracked in detail. Targeted tracking can involve a whole different level of surveillance, involving malware and hacking into desktop and mobile devices. It can also involve knocking on the door of Google or Facebook.
• Even without the firepower that can be brought to a targeted investigation, mass surveillance results in a searchable map of millions of people’s links, who they know, where they are and have been, and whether they have ever entered a search term that raises a red flag. Anyone up to three degrees of separation from a “person of interest” – potentially hundreds of thousands of people per target – is open to surveillance.
• The NSA says it’s not interested in most people and that’s plausible, but many would argue that indiscriminately recording all this data and making it easily searchable constitutes a severe and widespread invasion of privacy. A U.S. federal judge has taken this view in relation to the Verizon metadata.
• There are at least a few recorded cases of NSA employees using their power to stalk crushes and ex-lovers — but these are just the cases where the culprits turned themselves in. Hundreds of thousands of analysts and contractors have access to these systems. The NSA has itself fessed up to thousands of “unintentional” compliance violations.
• Nothing can be trusted. In one of the most shocking revelations, it turned out the NSA has been betraying its partners in the computer security industry and at least trying to weaken everyone’s digital locks, rather than just building better lockpicks. The NSA has even targeted Tor, a U.S.-funded tool that’s supposed to protect the anonymity of dissidents in other countries and other users who value their privacy.
• It’s fair to assume any unencrypted communications are open to monitoring. Strong encryption is probably still secure, but the trust system that governs web security has integral flaws that need addressing. As the closures of Lavabit and the Silent Mail service showed, firms that make encryption easier by managing the user’s keys also make the user less secure by becoming a target for the authorities.

Now rewind. When Snowden’s “direct access” revelation hit — which we now suspect to be a fiber-related affair — American technology companies freaked out. Google, Microsoft, Apple(s aapl) and others appeared utterly complicit with the NSA, as though they were happily inviting them into their data centers for a look-around, and perhaps a nice martini. We still don’t have the full picture, but there’s currently reason to give some of the tech companies at least a limited break. The companies, or certainly their high-level management, didn’t need to know for all this to go on.

However, even if it turns out they were entirely ignorant of the NSA’s fiber-tapping ways, that doesn’t mean they haven’t been complicit.

U.S. antiterror and anticrime laws, from CALEA to FISA to the Patriot Act, have always made it abundantly clear that U.S. authorities can commandeer whatever U.S.-based communications services they want. Knowing that, many of the big web companies were scarcely encrypting the data they held in storage, let alone the connections between their data centers. They encouraged people to give up more and more, and they didn’t protect what everyone gave up to the best of their ability.

On the plus side, the likes of Facebook and Apple are now a whole lot more keen about transparency when it comes to government data requests, perhaps in order to rescue their public image. The big U.S. tech firms have also finally banded together — seven months after the Snowden revelations — to demand a change in U.S. intelligence tactics.

Fallout

The post-Snowden months have seen a rash of interest in privacy-protecting plug-ins, search engines and anonymous surfing appliances, many of which come out of the U.S. Overall though, due to U.S. laws, the likelihood is that non-U.S. companies will prove the beneficiaries of the surveillance scandal, particularly those that are savvy enough to push their privacy-friendliness.

Early warnings from IBM(s ibm) and Cisco(s csco) indicate that some big U.S. tech firms are already seeing a significant drop-off in orders and contracts from abroad, particularly in emerging markets such as Brazil and China. A desire to avoid American services has also led Brazil’s government and Germany’s big telcos to consider the merits of keeping local-to-local internet traffic within their borders. This has led to fears of a “balkanization” of the web, with unpleasant censorship potential. However, the web’s globally interconnected nature makes this a tall order at best.

What does look set to happen is a legal and technical reinforcement of online privacy. The United Nations is working on a resolution affirming that human rights apply online as well as offline, and on the technical side the web may soon be encrypted by default.

But what of the tech firms whose services have been so successfully hijacked by the NSA and its partners?

Facebook is still quietly doing what it can to stop users from protecting their privacy. And Google, which delighted privacy advocates in July by releasing an Android feature called App Ops that made it possible to turn off specific tracking functionalities in individual apps, pulled that feature in a later Android update, claiming it had been included by accident.

The online ad industry is also doing its best to ensure everyone remains trackable. At the end of September, Stanford privacy advocate Jonathan Mayer quit the working group that has been steering the abortive Do Not Track standard, a browser feature that’s supposed to dissuade websites from tracking internet use with cookies. Rather than doing what it says on the tin, Do Not Track now mostly comes turned off by default. The ad industry, which lives off tracking people, won the day. Of course, we now know those same cookies can be hijacked by the NSA.

And all over the world, new privacy challenges are appearing. In the U.K. recently, someone tried setting up a scheme whereby smart trashcans scoop up identifying information from passing smartphone users, and retail chain Tesco started scanning customers’ faces as they stand in line to pay. And then we have the profound privacy implications of the emerging internet of things. Each case needs to be evaluated on its own merits, but they are all now colored by the knowledge that collected data may at some point be targeted by intelligence agencies and other authorities.

New beginnings

In summary, 2013 felt like both an end and a beginning.

It was the end of an age of innocence. Sure, many suspected and some knew that the internet is a giant monitoring system, but anyone paying the slightest bit of attention must now realize that everything they do online — and increasingly offline too — is open to tech-enabled surveillance. Anyone carrying a mobile device should now understand that they are being constantly tracked.

Now we must address the fundamental questions of our time. Is it possible, as some suggest, to accept technological trends such as “big data” while also giving people the option of privacy? Can agencies such as the NSA continue to track terrorists without tracking everyone? Can we continue to see commercial surveillance as separate from state surveillance? Can we create popular internet business models that don’t make the user a well-described product? Are we heading into a world of data-driven authoritarianism?

These are questions we are only now asking with seriousness and urgency. Perhaps we are too late, perhaps we are just in time – but either way, 2013 has at least helped us understand the choices we all face.

Disclosure: Guardian News & Media is an investor in the parent company of Gigaom.