Your own personal AI agent. What could possibly go wrong?
Subscribers exclusive
https://www.straitstimes.com/opinion/your-own-personal-ai-agent-what-could-possibly-go-wrong
=====
Your own personal AI agent.
What could possibly go wrong?
Handing over control
is not an easy
decision, and it
shouldn’t be.
Roy Ka-Wei Lee
So you want to buy the new
iPhone Duo. Usually, you would
do the time-consuming task of
comparing prices and deciding
which one to buy.
Imagine instead, the
convenience of simply telling an
AI assistant: “Find me the best
deal for the new iPhone Duo and
buy it for me.”
This is the promise behind a
new generation of personal AI
assistants.
Artificial intelligence assistants
such as Meta’s Muse and Instinct
are moving beyond this and
chatbots that simply answer
questions. Muse, for example, can
connect to apps and services to
carry out multi-step tasks, while
Instinct works through text
messages to make bookings and
other arrangements.
AI assistants can increasingly
access our e-mails, calendars
and other services, browse
web, make reservations, shop
online and carry out tasks on our
behalf.
The appeal is obvious. But there
is an important difference
between an AI that helps us make
a decision, and one that makes
and executes the decision for us.
If a chatbot gives us a poor
recommendation, we can ignore
it. If an AI assistant
misunderstands us, it could spend
our money, send an e-mail, cancel
a booking or disclose sensitive
information before we realise
something has gone wrong.
There is also an uncomfortable
contradiction. We are still
struggling to make autonomous
AI agents behave safely and
reliably in controlled or
sandboxed environments. Yet
consumers are increasingly being
entrusted to some of the most sensitive
of their lives – their e-mail
accounts, payment information
and other personal data.
This is why the conversation
about personal AI assistants
cannot stop at privacy. We must
ask what information they can
stop ask information they can
access. But increasingly, what also
need to question: What authority
are we giving it to act on that
information?
PROBLEMS WITH AI ACTING FOR US
Giving an AI authority to act for
us personally introduces several
new problems.
It may simply misunderstand
what we want. “Find me the best
deal for the new iPhone Duo”
sounds straightforward. But what
counts as “best”? The lowest
price? Fastest delivery? A
particular store size or colour?
A chatbot that gets this wrong
may give us a bad
recommendation. An AI assistant
may badly spend our money.
Tech news site PCMag reported
that one Muse user who selected
“Allow Always” while letting the
agent handle a Facebook
Marketplace listing, later
discovered that it had shared his
home address with a buyer and
had arranged a pick-up. He said
had expected Muse to seek
approval before accepting offers.
An Instinct user, according to
TechCrunch, reported that she
had disconnected her e-mail after
the agent sent a message on her
behalf without first checking with
her.
Mistakes can cascade. Suppose
an assistant books the wrong
flight. It might then rearrange our
calendar around that flight,
cancel another reservation and
notify colleagues that we will be
away. One initial mistake have
become several serial actions across
different services, potentially
before the user even notices.
We therefore need to think not
only about permission, but also
about delegated authority – how
much decision-making power we
are prepared to hand over to AI.
A useful principle is simple: The
greater the consequence of an
action, the less autonomy the AI
should have.
We might be comfortable
allowing an assistant to reorder
inexpensive household items
without asking. But an expensive
purchase should probably require
approval. An assistant might be
allowed to find and recommend a
flight, but should ask before
charging thousands of dollars to a
credit card.
The choice should not simply
be between giving an AI full
authority and no authority. Users
should be able to set clear limits
on what their assistants can do
without coming back to them.
Such safeguards are not new.
Banks impose transaction limits,
require additional authentication
for certain payments and flag
unusual activity. These measures
create some inconvenience, but
we accept the friction because
the consequences of getting a
financial transaction wrong – or
allowing a scammer access to our
funds – can be serious.
AI developers should adopt the
same principles and build safety
into personal assistants from the
start.
THE ONUS IS ON AI DEVELOPERS
Some developers are already
building in safeguards. Meta says
Muse asks users to approve
critical actions such as sending
e-mails and making purchases,
and provides a complete audit
trail of what the agent has done.
But these safeguards are only
useful if users understand what
they have authorised and can set
clear limits.
“Your assistant has completed
your travel arrangements” is very
different from “Your assistant is
about to book this flight for
$1,850 using this credit card.
Approve?”
The aim is not to make people
approve every trivial action. That
would defeat much of the
purpose of having an AI assistant.
The challenge is to identify
sensible boundaries where
automation should stop, and
human judgment should return.
There should also be a clear and
easily understood record
of what the AI has done on
our behalf.
Think of something closer to a
bank statement than a technical
computer log: what the assistant
did, when it did it, which service
it interacted with, whether it
spent money or shared
information and, where possible,
how the action can be reversed.
If something goes wrong, users
should not have to reconstruct
their AI assistant’s behaviour
across five different apps to work
out what happened.
COMPANY AI INFLUENCING
YOUR AI
There is an even bigger question
on the horizon.
Today, online advertising is
largely about influencing people.
Companies compete for our
attention, show us advertisements
and try to persuade us that one
product is better than another.
But what happens when we are
no longer the ones doing the
shopping?
If I ask my AI assistant to find
the best deal for a phone and buy
it, I may never see the
advertisement, compare the
retailers or even visit their
websites. My AI assistant may
make the choice for me.
Companies will then have a
powerful incentive to influence
not only people, but also the AI
assistants making decisions for
them. Retailers may learn how to
present information so that AI
assistants favour their products.
Sponsored results may shape
recommendations. Other AI
systems may even try to steer an
assistant towards particular
choices.
The next battle in digital
advertising may not be for our
attention, but for the attention
of the AI agents acting on our
behalf.
That raises a question we have
barely started to confront: How
do we know that our personal
assistant is really acting in our
interests?
None of this means we should
reject AI personal assistants. They
could remove large amounts of
mundane work from our lives and
make many everyday tasks
considerably easier.
But consumers should think
carefully about the authority they
hand over. And AI developers
should not treat safety as
something to add only after these
products become widely used.
A human-centred AI assistant
should not be judged simply by
how much it can automate. It
should show what it can safely do
for us, what it must ask before
doing it, and how to show us clearly
what it has done.
Keeping humans in control will
depend on designing sensible
limits to what we delegate.
• Roy Ka-Wei Lee is an associate
professor at the University of British
Columbia.
Posts
