AllSecurityLawsBest PracticesCOVID-19Culture

Defining the Transparency and Privacy Gray Area

June 11, 2018
animated spiderman surrounded by text: With Great Data Comes Great Responsibility

The increased ease of access for government records has caused a trend in issues related to the gray-space of what constitutes a public record. In an effort to disseminate how various governments walk the line between transparency and privacy we gathered a panel of Bay Area government personnel, all of whom hold roles that touch public records or open data, to hear their real-world examples on how they balance what information is made public. The 45-minute panel, moderated by our NextRequest COO and co-founder, Reed Duecy-Gibbs, was presented to a packed room of Code for America Summit attendees.


  1. Alex Katz, Chief of Staff at the Office of Oakland City Attorney
  2. Eugene Park, the Deputy Port Attorney at the Port of Oakland
  3. Jason Lally, the Data Services Manager at DataSF within the City and County of San Francisco

An overall consensus from each panelists was that each of their agencies is on the higher end of the transparency spectrum compared to agencies nationwide---meaning each of their agencies attempts to proactively release public records and data sets after they’ve been vetted for privacy concerns. Alex Katz noted that the City of Oakland’s “community and the press are very sophisticated compared to other places who knows what rights they have and what types of information they are entitled to.”

Three main themes emerged during the panel:

  1. Ways to better navigate the gray area of public records
  2. Public records aren’t personal
  3. Technology can help solve the tension between transparency and privacy

There are ways to better navigate the gray area of public records

Jason Lally started the conversation off by describing how his organization, DataSF, created a toolkit to help staff assess those fuzzy, gray areas of government records. The toolkit is used by San Francisco staff to:

  1. Identify sensitive information
  2. Perform a risk assessment
  3. Figure out ways to reduce any risk
  4. Perform a secondary assessment on whether or not data can be re-identified

The re-identification of public data sets is a huge gray area for governments. Jason gave the example of juvenile probation records. By law these records can be published by dropping all of the identifiable information, but if you have access to certain demographic records you can actually cross-reference who an individual juvenile is. This is where Jason says that laws start to fail us and why it’s so important to rigorously evaluate data-sets.

Another factor that Alex Katz raised was how time constraints and resources come into play in the gray area. A misconception of just how long it takes for a request to be processed is often the case. This can be frustrating to both the requester and to the agency. What often doesn’t get communicated to the requester is that an agency can be dealing with a multitude of resource issues, from a lack of personnel to sudden increase in request complexities, and these issues create friction for requests being processed in a timely manner. More times than not agency constraints are not relayed to the requester, which makes it look like the agency isn’t being as transparent as it should be. The additional effort to determine what is public and releasable versus an individual’s private information compounds resource constraints and is often where governments fall behind.

Eugene Park reminded us all that sometimes active transparency can have unexpected side effects. As an example, he explained how his agency’s relationship with journalists has shifted. The Port of Oakland’s recent initiative to post more records requests online has actually had the journalists asking to delay records posted over concerns that a published request would result in a story getting scooped.

Public records aren’t personal

Wait, what?

Updating internal records management procedures within an agency is no small task but is the first key step to increasing an agency’s transparency. Eugene put it best with, “A lot of people say don’t talk about politics or religion at the workplace—I would add records to that.” He’s found that his government staff can get emotional about records. Mind you, these are public records---not theirs. So why would they be so protective? When discussing records management, specifically records destruction, Eugene says that people can get very defensive and alarmist, saying “You’re going to destroy all my files, but they’re very important”.

He went on to ask the panel audience to imagine someone redoing their email folders or Slack channels and how that would turn their lives upside-down. Eugene suggests, because many people resists change, the key is to let them know that you don’t want to turn their work life upside-down, but instead make a small portion better. These changes to internal procedures and workflows can pay huge dividends in the overall transparency of an agency.

The key to breaking down the information silos that get built up in every type of organization is to build up the “institutional muscle” that empowers agency staff and makes them feel that their work is valued.

Technology can help solve the tension between transparency and privacy

The overwhelming consensus from the panel was that the problem affecting transparency and privacy isn’t due to a lack of technology. The real problem is that the laws haven’t caught up. The Freedom of Information Act was enacted in 1967 when public records were paper. Most public records lived in folders, boxes, or filing cabinets. Nowadays we have dashboard cameras, emails, and text messages--and these public records live in different repositories, databases, and devices.  The sheer number of mediums that can now be considered a record has also mushroomed. Technology may be giving us better ways to manage transparency and privacy, but it’s also giving us a lot more to manage.

The City of Oakland recently started releasing police officer uniform camera videos. Alex gave the examples of a police videos recorded at someone’s house or if a child is in the video. In cases like those, there’s a lot of work required to determine if the video should be blurred or entire sections should be redacted. This work adds up.

Jason summed it up by saying, “We’re not going to pump the brakes on technology. We can try, and people do, but it’s not gonna happen.”

As governments adopt new technologies, additional follow-on technologies are often needed to manage and audit the new work conducted and information produced. On top of all this it’s clear that technology isn’t a panacea and as Jason Lally put it “cannot be separated from the technology user”. In order for governments to successfully balance transparency and privacy of their records and data in the digital age; law changes, good policy and processes, and relationship building are equally if not more critical.

You can watch the full panel here.