How do I build tech products that guard against gender-based violence?
The New York Cyber Abuse Task Force offers a framework for humane technology product design.
By Tanuja Jain Gupta, on behalf of the New York Cyber Abuse Task Force
The New York Cyber Abuse Task Force is a coalition of legal and non-legal professionals, survivors, and technology workers to fight tech-facilitated gender-based violence in all its forms. Our findings consistently show that the most dangerous and persistent forms of tech abuse occur in the context of intimate partner violence and image-based abuse.
A coalition of legal and non-legal professionals, survivors, and technology workers came together, as the New York Cyber Abuse Task Force, to fight technology-facilitated gender-based violence in all its forms. Technology accelerates the speed and magnifies the scale of existing illegal sexually-abusive behavior such as stalking, spying, spoofing, impersonation, sextortion and non-consensual distribution of sexually explicit images and videos beyond imagination. Lawyers and law enforcement support the survivors of these crimes who are forced to retreat from a digital life - or life altogether. We published a “Manual for Advocates” to help lawyers navigate the courts for the issues that plague abuse victims, but we still need more. We need the engineers, designers, product managers and analysts - the ones with the technical know-how and a desire to use technology for building a better future - to change the industry from within.
We have also created a framework for product designers and engineers to build future products that guard against facilitating gender-based violence.
Five intersecting groups prove to be the most vulnerable to the harms of technology: children, women, members of the LGBTQIA+ community, people of color and low-income individuals. When product teams consider these groups as primary users rather than edge-cases, all users benefit. Point in case: updates to air tag tracking devices, which help domestic violence survivors avoid being stalked by their abusers … but also help all users better protect their privacy. Our task force has found that ‘humane’ product design boils down to a product team being able to answer ‘NO’ to the following three questions:
1) Could your product let someone exert P.O.W.E.R. over another person?
2) Is that power exerted without the subject’s C.O.N.S.E.N.T.?
3) Does anything prevent your company from providing law enforcement the right E.V.I.D.E.N.C.E. needed to hold an abuser accountable?
The three acronyms of P.O.W.E.R., C.O.N.S.E.N.T. and E.V.I.D.E.N.C.E. address twenty factors of online abuse that can help protect survivors long before the lawyers are involved.
Whether you’re writing the PRD, presenting wireframes or mocks, architecting the database, writing the technical design doc, determining and implementing API calls, or performing a security or privacy review etc, you can ask these questions below to determine how humane your product design is for vulnerable populations (download one-sheet here):
1st Question: Could your product let someone exert P.O.W.E.R. over another person?
P: Does the success of your product depend on the propagation of content your company did not create?
O: Do you treat behavior in your product differently than if it happened offline? Is there a chance the product could reflect activity online differently than it is occurring offline?
W: Could anyone who is watching, changing settings or asking for help on the user’s account differ from the user herself in your product?
E: Does your product entangle the user’s account to either another account belonging to the same person or another person all together?
R: Could your product be used by one person to malign the reputation of another person?
If YES to any of the above, move to the second question.
If NO to all of the above, first check if others on your team would answer the same way. If they all say NO, great - move to the third question.
2nd Question: Is that power exerted without the subject’s C.O.N.S.E.N.T.?
C: How does a user determine if their account or device has been compromised in real-time in your product?
O: Is the default feature setting opt-in or opt-out in your product? Is the user aware?
N: How does the user negotiate their presence in your product? Does she have an opportunity to understand how her data will be used and where it’s going?
S: How does the user screenshot, store, save and send proof of unconsented-to activity to authorities in your product? Conversely, can a user prevent someone else from screenshotting/saving material intended for them to receive, but not to be distributed to others?
E: How do you monitor points of egress (and ingress, for that matter) for anomalies in your product?
N: If you use social features, how does a user notify her network that she's cut ties with someone, and warn of potential impersonation in your product? Conversely, is there a way for the user to know if someone reaching out to her is tied to the network of the person who she blocked?
T: Can a user report harm in-product in a timely fashion? Do you respond to the subject’s reports of harassment in a timely manner? Are your revocations of access and removals of content timely? What does it take to trigger your break-glass plan in a timely manner? (ex)
If you don’t have an answer for one or more of these sub-questions, consider whether your product’s features deny options of privacy and consent.
3rd Question: Does anything prevent your company from providing law enforcement the right E.V.I.D.E.N.C.E. needed to hold an abuser accountable?
E: What is your team’s plan for how to work with law enforcement to decrypt encrypted messages?
V: How do you provide verification of identity and data authenticity as required in court? Can you connect the actor to the activity?
I: How does your product back-end integrate with other databases of evidence from previous cyber abuse violations / bad actors to learn and detect future abuse?
D: Enumerate the data logs of user activity with protocols for when that data should be stored, for how long, etc.
E: How do you ensure that your data logs have plain descriptions of each field to explain the meaning of each field, its metadata values and what could be a sign of manipulation?
N: How do current legal protections cover the next iteration of your product or feature?
C: How does your product understand when crimes and confessions are being aired or live streamed?
E: How quickly can the above data be exported to law enforcement in a human readable format in a timely fashion?
If you don’t have an answer for one or more of these sub-questions, consider whether your product is ready to release if you can’t hold abusers of your product accountable.
A handful of examples to get your minds going:
P.O.W.E.R. means possessing control, authority, or influence over someone or something. Could your product let someone exert P.O.W.E.R. over another person? Consider if that person is a historically underrepresented minority and/or a child.
Propagation - Does the success of your product depend on the propagation of content your company did not create? (Propagation is the action of widely spreading and promoting an idea, theory, etc.) Another way of asking this is “Can you hit your GEMS metrics without propagating user-generated content?” For those not in tech, GEMS stands for growth, engagement, monetization, and satisfaction (ex: information satisfaction, regulatory compliance, etc). So can you grow your number of users, lengthen the time they spend in/on a product, increase the revenue you make on them or satisfy their needs without having to propagate content your company didn’t create? If not, you’re inherently incentivized to build a product that encourages fast, frictionless propagation of content - which is at odds with the measures like a cooling period normally associated with rational, deliberate decisions (much less give your product team time to determine if the content adheres to your product policies). For example, social media companies rely on a robust content ecosystem to have users scroll infinitely through a feed. Search engines need content to fulfill user queries (just look up a query like [track my girlfriend] or [stalkerware]). App stores rely on apps like Dream Zone to satisfy some men with ads that gamify rape. Maybe this kind of issue isn’t inherently at odds with your goals, but it’s a consideration.
Offline Parity - Do you treat behavior in your product differently than if it happened offline? Is there a chance the product could reflect activity online differently than it is occurring offline? Showing your genitals to someone without consent is sexual violence. Sending a dick pic to someone should be treated the same.
E.V.I.D.E.N.C.E.
Verification of identity and data authenticity: Can you provide verification of identity and data authenticity as required in court? Can you connect the actor to the activity? Directly from a prosecutor: “At trial, the main hurdle is often proving that a specific perpetrator sent a specific transmission. Offenders tend to use new devices and public Wi-Fi when distributing the photos/videos. Services exist to mask IP addresses. Some may also use throwaway devices and/or a virtual private network (VPN) to make it seem as if the distribution originated from China or Russia. Getting logs and connection data from a foreign VPN provider (if the logs even exist) is difficult and tedious. Defendants will commonly argue that they themselves were hacked. A well-organized evidence chart can be used to show that only that perpetrator would have the motive and ability to create the campaign of cyber sexual abuse your client endured … but that is usually directly at odds with the internal privacy mandate of a company.”
How do current legal protections cover the next iteration of your product or feature? Here’s a common clause used in temporary restraining orders and orders of protection: The Respondent is not to post, transmit, or maintain, or cause a third party to post, transmit, or maintain, any images, pictures, or other media, depicting the Petitioner in a naked state or participating in any sexual act OR threaten to do the same. The Respondent is to refrain from using Petitioner’s likeness or impersonating Petitioner on any social media. If your product counsel can’t fit your feature into that language, how will you communicate to lawyers and legislators that legal protections need to be updated?
You get it.
The members of our task force, and the technology industry professionals who partnered with us to create this framework, all realize that some of these questions may be addressed at varying seniority levels. We know this framework may be more useful for some types of technologies than others. And in some countries with more authoritarian governments, turning over evidence to law enforcement may require different considerations than this framework considers. But good, iterative product design does not permit perfection to be the enemy of progress - so let’s get started. Instead of doom-scrolling through your social media feeds while watching the presidential debates this week, listen to the candidates’ plans to see if they come close to addressing the groups most vulnerable to tech-enabled abuse. And then maybe take a moment to ask yourself these questions of power and consent in the products you build.
What will be your plan to build more humane technology products in the future?
Tanuja Jain Gupta is currently a Tom Henderson Civil Rights Fellow at Sanford Heisler Sharp McKnight and a former senior engineering program manager of twenty years, with eleven of those years at Google. During this time, she also advocated for workers’ rights in the form of leading a global walkout against sexual harassment in 2018 and successfully lobbying for Google to end its policy of forced arbitration in March 2019. Gupta was a key advocate for HR 4445, which became law in March of 2022, bringing together survivors of sexual harassment around the country to end forced arbitration at the federal level. For this work, she received the 2019 American Association for Justice Steven J. Sharp Public Service Award. While managing a large team at Google and working on some of its highest profile engineering and regulatory initiatives, Tanuja built a diversity, equity and inclusion program that was replicated by several teams within the company. During this time, Gupta also chaired the Board of the Crime Victims Treatment Center from 2017 - 2023. She joined the NY Cyber Abuse Task Force to channel her tech expertise for the benefit of survivors, and hopes her former colleagues in the industry will do the same. Deep thanks to the multiple engineers and trust & safety analysts who contributed to the near year-long development of this framework.