Inkwell: Authors and Artists

    
Inkwell welcomes Virginia Eubanks, author of  Automating Inequality:
How High-Tech Tools Profile, Police, and Punish the Poor. Virginia
is an Associate Professor of Political Science at the University at
Albany, SUNY. She is also the author of Digital Dead End: Fighting
for Social Justice in the Information Age; and co-editor, with
Alethia Jones, of Ain’t Gonna Let Nobody Turn Me Around: Forty Years
of Movement Building with Barbara Smith. Her writing about
technology and social justice has appeared in The American Prospect,
The Nation, Harper’s and Wired. For two decades, Eubanks has worked
in community technology and economic justice movements. Today, she
is a founding member of the Our Data Bodies Project and a Fellow at
New America. She lives in Troy, NY.

Leading the interview is Ari Davidow, who has a background in
community planning and software/web development. Ari has been
working with online community for many years, including a long
history as community host here on the WELL. He has also been a host
at other online communities, mailing lists, blogs and websites.
  

    
Virginia's most recent book is "Automating Inequality," which was 
recommended a few months ago here on the WELL. In the book, she covers 
three primary case studies showing how we are using big data and AI in 
ways that, as the book title "automate inequality."

The book hit me harder than I expected. Since the 2016 election, I have 
been focused on ways that technology can help us organize more 
effectively. I have been slow to realize that both inadvertently and 
otherwise, we have long been making the lives of the underserved more 
difficult, while also letting ourselves be seduced by the apparent 
"all-knowingness" of big data and machine learning to do even worse.

Perhaps the best way to start things off, though, is to let Eubanks talk 
about how she got involved in doing this work--both her work with the "Our 
Data Bodies" project (http://www.odbproject.org), and how she came to 
write this book.
  

    
For those who haven't heard of the book, here are some resources
that cover the book and the issues surrounding it:

NPR story, Feb 19, 2018, "'Automating Inequality': Algorithms In
Public Services Often Fail The Most Vulnerable":
<https://n.pr/2GsR6Vy>

Discussion w/Virgina Eubanks on PBS, Jan 16, 2018:
<https://to.pbs.org/2CoiFSL>

Discussion between Virginia Eubanks and Kathryn Edin, author of "$2
a Day," on CSPAN, Mar 29, 2018: <https://cs.pn/2wNTeUF>
  

    
I'm very excited to see this discussion, because I've been
interested in this book for a while but haven't gotten a chance to
read it.

I was just watching the PBS show, and I was really struck by this
comment, at 23:00. Eubanks is asked what we need to do to solve this
problem, and she says that the first thing is "get our souls right
around poverty in the United States, because as long as we believe
that folks are poor because they've made bad choices, as long as we
believe that poverty is an aberration and not the majority
experience int he United States, we're going to produce these
systems that -- no matter how hard we try to be fair -- reproduce
politics as usual."

As most folks here know, I've done a lot of work around poorly
understood illnesses, and I'm struck by the same dynamics in the
conversation there. Sick people, and most especially people who have
the nerve to get sick with illnesses that are not 100% accepted, are
seen as victims of their own bad choices. That's the gist of the
argument that these illnesses are "all in our heads." And our
experience is othered -- if we're sick because we're actually crazy,
then normal people need not worry that they'll ever get sick like
us. In a similar way, seeing poverty as an aberration makes those
who aren't poor feel safe -- just be a normal person and you'll be
fine.

Sorry to come in early with a comment that's a bit to the side of
her main argument, but her way of formulating the problem was useful
to me, so I wanted to throw that out there.
  

    
Nonetheless, Julie, a good point to consider and a valuable parallel
(imnho). Virginia should be online this evening and I expect things
to start barreling along.
  

    
Virginia Eubanks was just interviewed for the Plutopia News Network
podcast:
https://www.plutopia.io/2018/09/06/virginia-eubanks-automating-inequality/ Suggest giving a listen as background for this discussion. 

You can share a world readable version of this conversation at
http://bit.ly/automating-inequality

If you're reading this conversation and are not a member of the
WELL, you can still ask a question or contribute a comment via email
to inkwell at well.com.
  

    
Like Ari, I found myself hit harder by this book than I expected.

I was fuming about brittle, ungiving systems that allow no
flexibility on the part of ordinary humans, whether recipients or
officials. I knew that these systems don't *have* to be this way.
With enough initial investment these problems were and are
foreseeable, and when they're not fixable, you go back to parts of
the old system (as the Indiana state system was finally forced to, a
"hybrid" system").  

But the larger truth is that in their rigidity and parsimoniousness,
they reflect exactly the mind-set of the people who ordered up and
implemented them. 

Julie, your point about poorly understood illnesses is well-taken.
  

    
Hi all! I'm so excited to be here. It means a lot to hear that the
book has resonated with folks emotionally. That was important to me,
and a big reason that I chose for the book to be a work of
journalism rather than academic scholarship. More on that soon,
though... ?

To Ari's question, I usually talk about one particular experience as
the origin moment for the book -- I come from a background in the
community technology and welfare rights/economic justice movements.
In the late 1990s and early 2000s, I worked a lot with a community
of women living in a residential YWCA in my home town of Troy NY. 

I was sitting in a community computer lab one morning with a young
mom on public assistance -- she goes by the pseudonym "Dorothy
Allen" in my first book -- and shooting the breeze about technology.
In the course of that conversation, I asked what she thought about
her EBT (Electronic Benefits Transfer) card, which is the ATM-like
card that now delivers SNAP and cash benefits. They were fairly new
at the time.

She said that it was a little more convenient, but that her
caseworker now had access to her purchase records, and used it to
track all her movements and spending. 

I must have looked really (naively) shocked, because she laughed at
me for a while, and then said, "Oh Virginia! You all [meaning
professional middle-class people] should pay attention to what's
happening to us. You're next!" That was in 2000 - so eighteen years
ago. I think she was remarkably prescient.

Dorothy's voice is always in my head when I begin a writing project
about technology. She reminds me how important it is to start with
people most impacted by new tools. Not only because we should care
about justice and equity, but because they have the best information
about them. 

That old saw by William Gibson, "The future is already here, it's
just not very evenly distributed," is true. But I think I mean it
the opposite way that he meant it. For Gibson, the rich and
privileged get the future first. But when we looked at tools of
digital surveillance and social control, the future comes first to
the marginalized and exploited: poor and working-class communities,
people of color, migrants (especially those who are without legal
status), people who live in war zones, etc.

My goal with the book was to go out and ask poor and working-class
families what their experiences were like as *targets* of new
systems being integrated into social assistance: automated
decisions, matching systems, and predictive analytics in welfare,
homeless services, and child protective services. My instinct was
that the stories they told would be very different from the stories
we tend to hear in the closed circles of technology policy (and even
a lot of technology-focused activism). 
  

    
And to Julie and Pamela:

Thanks for joining the conversation! You've hit on key pieces of the
argument of the book. 

We often treat new tech as outside of history and context -- like
the monolith in 2001: A Space Odyssey, it just appears from nowhere,
lands on blank ground, and changes everything. But really, tech
grows out of culture and loops back to influence it. 

I spend the first chapter of *Automating Inequality* establishing
that we've had largely punitive, moralistic, and paternalistic
poverty policy in the US, at least since the county poorhouse of the
early 1800s. County poorhouses were brick-and-mortar institutions
for incarcerating poor and working-class people who asked for public
aid. There were more than 1,000 of them across the U.S., and the one
in my home town -- the Rensselaer County House of Industry -- was
open until 1954. In order to enter the poorhouse, you had to give up
your rights to vote and hold office (if you had them), marry, and,
often, you had to give up your kids. The death rates at these
institutions was as high as 30% annually.

What's most important about them for the purpose of the book is that
they represented an important political choice we made in the US,
that our social service system would make the conditions of
receiving support so terrifying that only the most desperate would
ever ask for help. We decided social assistance should be more moral
thermometer -- judging who is worthy and unworthy -- rather than a
universal floor under us all. 

I use the metaphor of the "digital poorhouse" to capture how our new
high-tech tools are part of this history, while still offering new
challenges and possibilities. I think of it as the "deep social
programming" that goes into many of the new tools in social
services.  
  

    
Thank you for your responses, Virginia. One of the reasons the book
resonates is that, unlike, say, "Weapons of Math Destruction" (which
I regard as critical reading on its on terms) you aren't talking as
much about the oversights and lack of critical thinking behind
deploying AI and Big Data systems--you are talking about deliberate
continuity in our spurning of the poor (something that seems to go
back to the the early European settlements on this continent, and is
also at the root of American racism). 

As you write, the issue isn't something new, but rather, new tools
to =automate= inequality. Before we go on to how much of this
affects "the rest of us," can we dig more deeply into what is
happening with the "digital poorhouse"--how it is used to deny
services, but also, as you wrote earlier, how the tracking
mechanisms and the denial of privacy rights may ultimately affect
all of us?
  

    
May? Do already. But we're middle class and not so much at risk. 

That's why I so admire that opening, where Virginia talks about a
medical emergency that comes at a particularly inopportune time (new
job, etc.) and how she, as a well-educated woman with flexible time
from work still is run ragged. It's such a telling incident. 
  

    
Welcome, Virginia!

I'm struck by the differences and the overlap between what you wrote
about and what Cathy O'Neil wrote about in "Weapons of Math
Distraction". She wrote about errors that grew out of bad technique
in interpreting data, largely by assigning unjustifiable meanings to
the data. You write about the use of technology as a tool and as an
excuse for pursuing antisocial agendas.

You may have gone too easy on the insurance company in your
introduction. Our experience is that they repeatedly use "technical
error" as an excuse to deny benefits, and try to wear down
policyholders even over small benefits.
  

    
Thanks, Virtual Sea Monkey.

If only I could have validated my hunches about the insurance
company!! I do think that the repeated denial of service when you
are at your most vulnerable -- like when I was caring for my injured
and traumatized partner -- is a pretty clear (and callous) example
of diversion. It's particularly frustrating to have them blame it on
a "technical error" when you are someone who has been studying
technological sleights-of-hand for two decades. 

I absolutely see that if I had even one more challenge, or a little
less support, during that time, I could have easily given up on
getting the health care that we deserved. I was so tired,
demoralized, and vulnerable. I don't think this ended up in the
book, but the day that my employer finally got our health insurance
re-instated, I came down with strep throat. I was bed-ridden for
something like three weeks.

I like *Weapons of Math Destruction* so much. I read it avidly and
refer people to it all the time. It's particularly useful to have a
quant like O'Neil -- an insider -- willing to go on record with her
insights about systemic flaws in these systems. But I wanted to
spend time on a slightly different goal: developing the historical
context and human cost of these tools so that we can understand that
there bad outcomes, even when they are not intentional, are
predictable and avoidable if you take equity and justice seriously. 
  

    
> Before we go on to how much of this
> affects "the rest of us," can we dig more deeply into what is
> happening with the "digital poorhouse"--how it is used to deny
> services, but also, as you wrote earlier? 

I write in the book about three different data-based systems being
integrated into social service programs across the US: an attempt to
automate and privatize all the eligibility process for welfare
programs in Indiana; an electronic registry of the unhoused in Los
Angeles County that is supposed to match the most vulnerable
homeless with the most appropriate available housing resources; and
a statistical model in Allegheny County, Pennsylvania that is
supposed to predict which children might be victims of abuse or
neglect in the future.

Indiana is a pretty stark example of how (perhaps unconscious)
assumptions about poor and working-class people -- for example, that
they are prone to fraud, lazy, shiftless, addicted, etc -- get baked
into technological design and implementation with devastating
effects. For example, the automated eligibility system interpreted
every mistake in an application as a willful "failure to cooperate
in establishing eligibility," and used errors as reasons to deny
people access to benefits they were eligible for and needed to keep
their families safe and healthy.

Los Angeles raises, among other things, a concern about how the
criminalization of poverty shapes the new tools of poor relief. The
coordinated entry system in LA County was designed and built by very
smart, well-intentioned people who care deeply about the well-being
of the people their agencies serve. But the system is built on top
of a very invasive survey called the VI-SPDAT (Vulnerability Index
and Service Prioritization Decisions Assistance Tool) that asks, for
example, if you are currently trading sex for drugs or if there is
an open warrant on you. And federal data regulations then allow some
of the information coordinated entry collects to be shared with law
enforcement based just on an oral request -- no warrant, no
oversight at all. This leaves many unhoused people feeling that they
have to incriminate themselves in exchange for a slightly better
lottery ticket for housing.

And finally, the Allegheny Family Screening Tool (AFST) raises what
I think might be the book's most troubling question: what does it
mean that we will spend millions of dollars to try and predict how
poor parents might behave in the future, but we won't spend what is
necessary to ensure poor families' basic human rights to shelter,
medical care, and food? For me, in the context of the US' unique
social assistance system, punishment and prediction all are tied up
in each other, leading parents to tell me that they felt like the
prediction tool confuses parenting while poor with poor parenting.  

For each story, I began with the point of view of those people most
impacted by the new tool -- welfare applicants who lost their
benefits during the Indiana automation, unhoused people who have
been surveyed with the VI-SPDAT in Skid Row and South Los Angeles,
and child-welfare involved parents in Allegheny County. I don't stop
there, of course. I did more than a hundred interviews for the book,
and they included designers, data scientists, policy makers,
organizers and activists as well as poor and working families. But
starting with the most impacted allowed me to tell deeper, richer
stories.  
  

    
>And federal data regulations then allow some
of the information coordinated entry collects to be shared with law
enforcement based just on an oral request -- no warrant, no
oversight at all.

Wow.  
  

    
Each episode had its own horror. Yet I couldn't help feeling, as I
said above, that these outcomes represent some basic societal
hostility to poor people. 

The term "deserving poor" has been with us for a very long time, but
the last forty years of metrics in almost any field have been around
money, which makes anybody poor "undeserving." 
  

    
Pamela, are you saying that the root of the problem is the social
attitude, not the tech? I get that...but part of the argument I'm
trying to make is that the technology embodies -- and often
amplifies or intensifies -- social attitudes like our fear of
poverty and hatred of the poor.
  

    
Virginia, you last comment also raises obvious questions. First off,
how do we change the use of technology, in the short term, to
dehumanize the poor--is this as simple as getting states to review
procedures? (Obviously not--even when the goals are seemingly
positive, as in Massachusetts' "Health Care Connector," navigating
the system is still a nightmare (and suspect--the system launched
with horrific bugginess).

And longer term, have you had success--has anyone had
success--addressing that fear and hatred of the poor? Does the
problem get a bit easier since the great recession given how much
more widespread the effects of income inequality have become, and
how many more of us know people directly affected?
  

    
Yes, I'm saying the root of the problem is the social attitude. The
tech, as you say, embodies those social attitudes.

Thus the tech itself is ill-planned (good planning takes money and
flexibility to fix when things inevitably go wrong). I had visions
of lowest-bid services getting the contract, and government entities
getting what they paid for. I was not entirely surprised that the
Indiana vendor was IBM, and a bit surprised that the firm prevailed
in court when things blew up.

Your point that technology also amplifies or intensifies social
attitudes is spot on, and I got that too. It's a point well worth
making (shouting from the rooftops).

Big programs always have bugs, sometimes lots of them. Payrolls,
which we've been doing for more than half a century with computers,
are finally pretty stable. But note the modifier: "pretty." That's
why Ronald Reagan's Brilliant Pebbles/aka Star Wars, was so
hopeless. It had to work right the first time. That's why our
anti-ballistic missile programs are so piss-poor in their
performances. (Those are different sides of the
amplification/intensification phenomenon.)

The homeless program in Los Angeles seemed to me a slightly
different issue. People really wanted to be helpful to the homeless.
How could they have done it better?
  

    
Part of the problem in LA was  , as someone in the book says, a
carpentry problem. All the computer-matching in the world could not
compensate for the loss of what was it, 10,000+ units?

And in Indiana it seemed that, beyond all the bugs, the technology
was doing what it was designed to do: reducing the headcount and
cost of public services. 

Victoria, thank you for joining us and I'll agree it was a brutally
hard book to read, but important. 

It was very interesting to read how this traced back decades, and
about the inherent contradiction between wanting to support the
genuinely needy while not making it tempting for the lazy. Somehow,
today, this contradiction manifests as actively penalizes people who
are receiving assistance for trying in any way to improve their
situation. And the massively intrusive surveillance enforces this. 
  

    
My career working with technology began when I was working
eligibility for poverty programs (Food Stamps, AFDC, Medicaid) in
the 80s and 90s. Texas had developed an electronic version of the
worksheet we used for eligibility determination. It worked
surprisingly well given the complexity of the regulations we were
implementing. Our system was pretty objective. 

Virginia wrote about Indiana's concern that caseworkers might
collude with their clients - this was a driver for greater reliance
on automation. In Texas, the caseworker's job didn't really change,
the automation was created to support the eligibility determination,
but the worker's judgement was still involved. However we didn't see
collusion with clients - in fact some caseworkers had more of a
police mentality, looking patterns suggesting fraud. 

I think they implemented a new system years after I was gone, and
that it was more like Indiana's.

Getting back to complexity of the regulations - the impact of that
complexity was often underestimated, and it's effect on operations
was sometimes interpreted from the outside as inefficiency. I recall
that a team of developers was brought in at some point, confident
that they could build a new, more efficient version of the system.
In the beginning they were confident, but as they continued working
their confidence waned - they realized that the tool we had
developed was actually about as good as it was going to get.

From what Virginia has written, I think the problem in Indiana was
probably attributable to mismanagement, and to in correct
assumptions and expectations on the part of the outside vendors. In
Texas, our software was developed by internal teams working with
subject matter experts, and the result was quite functional and
useable.
  

    
From what Virginia wrote about Indiana I see mismanagement,
unrealistic expectations, and bad faith.
  

    
I think "mismanagement" is too kind. The point is that we think that
there is something wrong with people who require aid, which leads us
to use these systems to further prune eligibility. 

We reinforce the animus against the poor by also assaulting their
privacy. If you are poor, you are forced to submit to a significant
loss of privacy. If you use a card to purchase food, the government
knows what you are purchasing. But, more generally, the applications
to get aid require you to reveal personal details that would never
be required of people not requiring government aid--they are clearly
far too intrusive.

This all leads to my big question: How do we change attitudes (and
aid is one part of this--criminal justice reform is another major
place where we are likelier to use big data to perpetuate
inequality, rather than improve outcomes)? How do we change our
perspective from "they're all cheats, so cut benefits, lock them up,
deny services" to "when people need help, we need to find ways to
ensure that they have the help they need"?

As Utah Phillips used to complain, we're good at attacking the minor
miscreants, and all-too-eager to let those "job creator" folks who
have done well in our system pay a fraction of their fair share.
That may well be far beyond the scope of this particular book, but
it is very much on my mind.

Getting back to the book, Virginia, what are the outcomes you were
looking for when you wrote the book? More awareness? Public
discussion on how we "automate inequality" and what that means?
Specific changes to programs and legislation?
  

    
Ari, in the Texas system at the time I was working there, there was
an avoidance of judgemental bias, and we were also taking privacy
seriously. I can't say whether that's been the case in the many
years since I left, but my point is that the technology itself had
been developed avoiding "animus against the poor." I guess I'm
saying that the use of technology in and of itself is not the
problem.
  

    
I'm deeply suspicious of the "it's not the technology, it's the
people" line of argument because technology IS people. It is
designed and implemented by humans and holds human values and
decisions. I think it is dangerous to separate human values from
their manifestation in technology. It's like the snake biting its
tail -- technology emerges out of a certain set of values and
doubles back to shape and reshape society. 

So often, people try to establish the neutrality of tech by arguing
that it is "just a tool." One of the odder jobs I hold is as a brick
mason. And I have at least SIX different trowels. I'm barely an
amateur -- I mostly repoint and repair historic brick -- so I have a
very limited kit. But every one of my tools is essential for a
different task, and uniquely shaped to it. I can't use a 1/4 inch
tuck pointing trowel to smooth an expanse of concrete, for example.
I can't even use a 1/4 inch tuck pointing trowel to repoint a
building with narrower or wider joints. 

What I'm trying to say is that there is nothing neutral or blank
about tools. They evolve over time to fit a specific purpose and
while you can use a hammer to paint a barn, you'll do a terrible
job.

Similarly, I believe that the deep social programming of our newest
high-tech tools -- rooted as they are in hundreds of years of
punitive and criminalizing poverty policy in the US -- makes them
ill-suited to social justice. It's not impossible to bend them
towards better outcomes -- just like it's not impossible to paint a
barn with a hammer. But it is really, really hard.
  

    
There can be a fine line, I think, between the thoughtless
interfaces of, say, common online forms--filled in a credit card
number or phone number online lately?--and the numbing, bewildering
experience of the systems we have put in place of people as the
front line (or only line) to provide access to needed services. But,
one is stupid and inconvenient; the latter is used to winnow out all
but the best at filling out online forms and denies needed services
to those who don't succeed. If we saw that help as a necessary part
of who we are as a society, we would be using different tools in
different ways.