2018 AI Summit San Francisco Keynote Microsoft AI CTO Joseph Sirosh

>> I'm going to tell you
about the three key trends in
AI that are really
powerful that you probably
haven't really heard about.
Now, let me start
with an example.
This is an arm that can see.
It's 3D printed, it has a camera
in the palm of his hand,
it is connected to
a service in the cloud.
The cloud service can
trigger the movement
of the fingers,
based on what the arm
actually sees.
Let's take a look at
it in this video.
So, watch the arm.
Now, as someone brings
it over a keychain,
the camera and the palm
recognizes the keychain
and on the right,
you see the classification of
the object and
a pincer grip was selected.
With the flexion of a muscle,
with the muscle sensor,
I can close the grip and you pick
up that and then you
can put it back down.
Now, watch as we bring
it to another object,
in this case, a wine glass.
The classification is
for a palmar action,
closing all the fingers together,
and with a flexor of a muscle,
I can pick that up and
I can put that back.
All it takes is a
few off the shelf components
like a Raspberry Pi,
an Arduino board, servomotors,
a 3D printed arm.
In fact, inside of this are
fish lines that pull
the fingers closed,
but of course, the magic is
the Cloud AI service behind it.
An AI service in
the cloud that can
recognize what
the camera in the palm
sees and then match
it to the grip action
that should be taken so
that the right grip action
can be performed.
That's trainable.
It's adaptable.
It really is something
you can set up,
something that
others could set up,
in the service in the cloud,
personalized prosthetics.
That's very powerful.
So, that leads me to
the most important macro trend,
which is that a cloud AI service
behind every device,
it might be a prosthetic,
it might be any device that
you use in your house.
Of course, your apps on
your phone have AI services
behind them eventually,
some of them already have
AI, but others as well.
Everything in the world that
is connected with Wi-Fi or
Internet connectivity
can now be backed
up by an AI service.
That's very powerful and
profound when you think about it.
Now, think about this one,
the grip classification.
How it works is there's
a muscle sensor that I've
attached to my arm here,
there's a camera in the hand.
So, through the electronics,
it goes to an Azure
Custom Vision Service,
where our classification model
has been set up,
a deep learned model
that recognizes object,
classifies it to the right
action and then that triggers
the appropriate grip
classification in
the servo motors connected
to an Arduino board in the arm.
Two undergraduates built this.
Hamayal Choudhry from
the University of Ontario
Institute of Technology
and Khan from University
of Toronto. Samin Khan.
They did this for the
Microsoft Imagine Cup.
They were the winners in 2018.
Building this took
them a few weeks.
Of course, then the magic
was provided by
a cloud AI service to be
able to make this device
intelligent.
That's a power. Even
an undergraduate can
build something as
powerful as this today.
So, why is this revolutionary?
Step back and think
about this device.
Look, there are over
a million amputations per year.
That's an amputation
every 30 seconds.
WHO estimates that
30-100 million people
in the world live with limb loss.
Only five to 15 percent
of these have
access to Prosthetics.
Even though prosthetic
devices have
been around since
the Egyptian times,
that what you see on the left is
a toe on an Egyptian mummy.
You can see this in the Egyptian
Museum and then you see
the iron hand of a knight
from medieval era,
his arm was cut off
and he got one.
Even though these devices
have been there,
they have been purely
physical devices
and very severely limited.
Limited by cost.
The bionic arms that you
have heard about today,
they cost tens of thousands
of dollars and
it takes a lot of effort
to fit them on you.
They're limited by availability,
very few people
have access to it,
and they're limited
by the interface
you can attach to the body.
Above all, they're limited
by the nervous system that
we have because
we've got to train
ourselves to use that device.
In fact, literally, we had
to force our will into
these devices to be able
to use them effectively.
How could we change all of that?
What could change us
from having to wrestle
with physical devices?
How could we break these limits?
The answer is an AI or a cloud
AI service backing it up.
Think about this, what if you
had low-cost electronics
to build with it?
What if we could
change the game of
availability with 3D printing?
So, you can print these
things anywhere in the world.
What if you had a Cloud
AI service behind it that
provided the ability to
recognize things and
make the movements?
What if it could be personalized?
What if it could be adapted?
What if other people,
your friends could train
your arm to make
the right kind of movements,
in the right kind
of environments?
How could you have
customizability of all types?
What if you could tap
into the knowledge
of the world beyond
our senses through
the cloud service
so that you can
keep improving it?
What if all of these
things came together for
a very low cost like the
$100 it took for
this arm to be built?
That would be
revolutionary, right?
Imagine, now every prosthetic in
the world or orthosis
in the world which is,
let's say you break your arm and
[inaudible] sling and
you need assistance?
What if you could get
something very cheap that
you could move around but it's
controlled by a Cloud
AI service and all you
have to do is express your intent
to that Cloud AI service
somehow and it
does the more complex task
of actually doing the grasp?
See, this is the difference
that the services can make.
What you do is you express
your intents and
your constraints,
and the service generates
the behavior you need.
So, it's a generative service.
The behavior is generated
but from high-level intention
that you communicate.
So, the future is
affordable, intelligent,
cloud-powered, personalized,
prosthetic devices and
really devices of every type.
That's hugely revolutionary.
So, let me keep this here
and now talk about
the next trend.
So, you realize how
empowering AI can be.
Now, with all this power,
we have 3D printing. We have AI.
You're going to be able to
revolutionize every aspect of
your life and potentially
for millions of people
who are disabled,
that could be
a new lease on life.
So, now let's talk about
how these things are built.
What we're seeing
is a huge explosion
of APIs in the cloud
that democratize AI,
so that every developer
can tap into
this incredibly sophisticated AI
without knowing AI.
Now, this is
a standard common trend
in computing by the way.
Incredibly sophisticated
algorithms are wrapped
up in functions that are so
simple you just call them.
When you call a sort function
in your programming.
Well, there might be an extremely
sophisticated implementation
of quicksort behind it,
but you don't have
to worry about it.
You learn to build it.
Same thing is happening with AI.
So now, there are cloud APIs
with machine learning in it.
I call them AutoML.
So, let's look at some
of the current trends.
There are APIs for perception.
There are APIs for comprehension.
So, perception vision
is being solved,
and a lot of vision tasks
are being solved.
There are capabilities
like face recognition,
identifying a face and
you can train them.
Computer vision,
meaning put an image,
get a caption or
a description of it.
Custom vision, where
you can upload
your own images with
class labels and train
them to classify.
Speech, speech recognition.
All of you know about it
but it's trainable now.
You with the right
language model,
with audio environment
and text to speech,
text to generating voice.
Then comprehension,
the world of language.
Language understanding.
So, you can train a system with
the kind of language
that you might see
and it will recognize
the intent that's
expressed and call the right
functions to execute them.
Filtering objectionable
content or
translating text or
analytics on text.
Then, the whole power of
search engines like
the Bing search engine,
including customizing
the search to
different domains or
doing search with images.
All of that is available as APIs.
These are just the start.
A lot more APIs like
this are coming.
What's important about these APIs
is they're not just algorithms,
they are built with
proprietary data,
so it brings the power of
the company that is
building it behind it,
whether it be a Microsoft
or a Google or an Amazon.
They're bringing data
and algorithms and
all of those things together
to build these APIs.
Very sophisticated ones.
So, here's an example of
a custom vision thing,
called free customization models.
You upload images with labels.
You train it. You deploy
it as a rest API.
You can even take those models as
containers and deploy them in
your software application.
So, what's an example
of an application?
Here is a fun example.
That image, by the way, is
from a real customer of ours.
They asked us if we could
understand all those images
and catalog and organize them.
It happened to be
the Ministry of Justice
of a country, by the way.
We quite couldn't get
access to all of that data
for security reasons,
but we asked ourselves "Hey,
how would we go about
solving such a challenge?"
I want to now show that
with a fun example.
In November 22nd, of 1963,
John F. Kennedy was
assassinated by
a lone gunman in the streets
of Dallas or so,
they lead us to believe, right?
Well, this topic was so
controversial that
Congress mandated that
all the documents associated with
the Kennedy assassination be
released to the public by 2018.
So, end of 2017 came out
all these documents,
lots of PDF scans.
If you pile them up on the stage,
it would be four huge
tax seven feet tall.
So, how would we understand
all of these documents?
How would we categorize,
organize, discover
who killed JFK?
All other controversies
around it.
So, our software engineers
took this challenge on.
So, they've created this thing
called cognitive search,
is actually a service
in Azure which allows
you ingest all types of
documents with the majors
were taxed and all of that.
You then apply these cognitive
skills that I talked about.
You enrich it and then you put
a search engine on
top of it to explore.
So, let me show
you the JFK files.
I'll actually show
you a fun demo.
So now, switching.
So, this is our website,
live website that you can
actually go to
jfkdemoazurewebsites.net.
I'm going to just search for
Oswald and let's
see what comes up.
Here's a PDF document.
It did OCR and recognized
Oswald in here.
Even more interestingly,
you see something here.
This is an handwritten
document and
OCR allowed you to recognize
terms like Oswald in here.
Right there. Then,
I can even go down,
take a picture of Oswald.
The custom vision,
the vision service
actually captioned it.
It's a Lee Harvey Oswald
posing for the camera.
Now, he's not really posing for
the camera but close enough.
It even recognizes
the OCR numbers here.
Very interesting.
So, now I can even see
relationships between them.
I can see Oswald is connected to
lots of interesting people
like Sylvia Duran.
As I go look through this,
I see things like Cuba in here.
So, what's Cuba
doing in JFK files?
So, let me show you.
This is a fun thing.
We search for
Castro operation in here
and we found all of this by
just building this application.
You see Castro operation
and you see, apparently,
in around that time
in the late 1960's,
the CIA in an operation
called Operation Mongoose had
hired the Chicago mafia
to poison Fidel Castro
with poison pills.
Fun thing. No one
knew but apparently,
the pills took a whole day to
dissolve in
Fidel Castro's coffee.
So, our test coffee.
So, Chicago mafia got
cold feet and backed
out of the whole thing.
So, out of the fun thing.
So, now let me show
you another thing.
Like, when a government releases
this kind of very
classified documents,
you hope your name
is not in there.
Now, my name is not in there,
but the name of one of
Microsoft's products is in
the JFK files. SQL Server.
Well, SQL Server didn't kill JFK.
But, we found that
SQL Server was selected
as the platform for
the secure classified
information facility by the CIA
when they built it
and Lotus Notes
from IBM was selected as
a medium of communication.
They even gave us
a whole architecture for
how these things will look.
You've been a complete with
dial-up lines and so on.
So, really fun story.
The amazing thing again,
is this kind of things
can be built by
an engineer in
a very short time period.
In this particular case,
it took about three weeks
for an engineer to
build it using these APIs
and all of that.
So, let me just get
back to my slide here.
These are incredibly useful.
What is really useful is that
you can take pretty much any data
in an enterprise,
like legal contracts,
or engineering plants, or
extract form information,
connect all of these things up,
understand it in
a cognitive sense,
you think it's
cognitive APIs and apply it.
Which then leads me to
the third big trend.
AI Enables Natural
User Interfaces.
Well, all of you know about
bots and speech interfaces,
there are even neural
interfaces emerging,
behind all of these things is AI,
and AI is enabling completely
new types of interfaces.
Now, one that you may not
be as familiar with is Ink,
Digital Ink, using a pen.
So, let me show you some
examples of the power of ink.
Look, all of these are
drawn by ink, and the pen.
There's this famous saying,
the pen is mightier
than the sword.
Try and type any of these things,
you can't quite create that.
But with the power of
a digital pen and a Digital Ink,
backed by a Cloud AI service,
you can now start capturing
these creative experiences,
and even go beyond.
So, let me show
you some examples.
Now, we have Digital
Inking as a service
in the Cloud behind PowerPoint
and Word and Office 365.
Here's an example of
what you can do in
PowerPoint, you can write,
you can turn that into text,
you can now draw boxes like this,
especially on a touch screen,
you got all of that,
and yes Lasso it
with a circle and
then you can turn it into
actual printed letters,
you can make those boxes
look much cleaner,
and you can even draw
lines between them, right?
So, now you've created
something new.
Same thing with Word,
you can edit in Word with a pen.
So, you can put an arrow there,
you can write what you
want like brand and then
it'll get inserted right
there in that resume, right?
You can cut out a line
and that will clear up.
So, all of these interactive
experiences that you're
seeing can be done with
the power of the pen.
So, let's keep going,
what if I had
handwriting like this?
I can make it look prettier
using a Cloud AI service,
this is ink beautification.
So, that's my handwriting,
and you will see it
getting cleaned up.
This is beautified, original,
beautified, original,
you see that it's improved,
my handwriting became better.
Let me give you another example,
what if I'm actually
drawing diagrams?
These diagrams are not as clean.
By the way, this
enables speed as well.
I can quickly draw something and
then let the AI service
clean it up for me.
So, this is the original, this
is beautified,
original, beautified.
Now over time by the way,
we can keep improving
these things,
and it'll become
better and better,
and your interactions with
these devices will
become very powerful.
It doesn't stop there.
Now, here's another example
that I'm going to
show where you're
drawing on a whiteboard,
and a picture is done,
and then you can focus with
your hand on
the right portions of
the whiteboard and then
touch any of those.
>> Zoom catcher,
eliminates scenario
for selection in
an extremely lightweight manner.
The user can then
act on the strokes,
such as to recognize.
But only what areas the user
wants and only when
the user chooses to do so.
>> Cool. Right. So, you saw
that interactive power.
So, this is a progression
of Ink in Microsoft.
It's been a journey, but around
2017 is where the magic
started happening,
where we saw a big step
change improvement with
the power of more data and AI,
and I wanted to show that to you.
Really, up until 2017
we were using a shallow machine
learning models,
limited data, limited accuracy
and a client API.
But then starting 2017,
we started using DNS,
and we started using
much more data.
We had a Cloud AI
service behind it.
We had a Cloud service
that draw the
country's improvement,
significant improvement
in the capabilities
and all of the endpoints to
which you could bring them.
Now, I want to end
with a final story.
So, this is a story of
an application called Helpicto.
Helpicto was built by
a French developer.
A French developer who just used
the Cloud AI services to create
an application to communicate
with autistic children.
Now, communicating to
autistic children,
mothers, fathers,
communicating, that's
always a challenge.
The standard of care has been you
bring up a picture book,
you take pictures from it,
compose pictorial conversation at
the same time as you speak.
So, the child hears you and at
the same time sees the picture
and that increases comprehension.
But of course, this is
incredibly unwieldy.
So, the developer
ask the question,
why can this be on
a mobile phone?
Why can't this whole thing
just recognize my speech,
make that conversation
happen pictorially on
a mobile phone and so
the child can be shown that,
and it just improves the speed
at which you can do this,
and you don't have to carry
a book around with you.
Let me play the video and look at
the subtitles so you can
understand it, it's in French.
>> [FOREIGN]
>> AI powered Natural Interfaces
can be very empowering.
So, AI is the new normal.
It is an incredibly
empowering technology,
and Microsoft, by the way,
is about empowering others
by creating platforms on
top of which all of you can
build these types of
powerful applications.
So, I hope you go
away from this event,
inspired by the power of
what AI can do for you,
and build on top
of this to change
the world and to change
your communities,
and make it the next
technology that empowers us
all. Thank you very much.