[00:03.480 --> 00:04.180]  Thanks, everyone.
[00:05.860 --> 00:10.400]  This is a picture of my desk as it looks... well, as of yesterday or so.
[00:11.040 --> 00:13.820]  So, you can see a bunch of crazy stuff there.
[00:13.920 --> 00:20.060]  I've got some microphones, I've got an ergonomic Gold Touch keyboard and a trackball and so on.
[00:20.850 --> 00:24.830]  And this is not really what a normal programmer's setup looks like, I suppose.
[00:25.300 --> 00:27.080]  But I was forced into it, unfortunately.
[00:28.960 --> 00:33.200]  I had an illness starting around a little more... a little under two years ago.
[00:34.760 --> 00:35.710]  Some kind of virus.
[00:36.020 --> 00:36.890]  My body overreacted.
[00:37.210 --> 00:42.300]  Doctor's not really sure what happened, but the end result is I couldn't type for a long time.
[00:42.500 --> 00:44.340]  I couldn't... as it says, I couldn't lift a pencil.
[00:44.500 --> 00:46.960]  I couldn't... I couldn't open a door with keys.
[00:47.100 --> 00:50.220]  I mean, it was very little hand strength.
[00:50.220 --> 00:56.780]  And then later on, it started springing to my legs and I couldn't... couldn't walk for a long period.
[00:56.980 --> 00:57.900]  Sometimes I couldn't even stand.
[00:58.640 --> 01:02.120]  And I couldn't sit, for sure, unless the chair was very well padded.
[01:02.600 --> 01:06.100]  So, that was somewhat of a disaster for me.
[01:06.180 --> 01:09.640]  As you saw from the first slide, I'm a PhD student at Columbia University.
[01:09.880 --> 01:11.680]  A PhD in computer science.
[01:13.800 --> 01:17.560]  Job description involves, you know, typing 10 or 12 hours a day, normally.
[01:19.340 --> 01:21.500]  So, fortunately, my advisor didn't kick me out right away.
[01:24.040 --> 01:27.840]  And... and... I... I worked around it with these microphones and stuff.
[01:27.960 --> 01:28.720]  And you'll hear all about that.
[01:29.400 --> 01:32.200]  And this is a very strange affliction.
[01:32.260 --> 01:33.300]  I don't think anyone else really has it.
[01:33.400 --> 01:35.960]  But what's a lot more common is repetitive strain injury.
[01:36.500 --> 01:44.340]  And this is just, you know, your muscles get tired, particularly the myelin sheaths around your nerves, get tired of doing small motions all the time, all day, every day.
[01:45.760 --> 01:48.800]  So, it's very common for programmers and musicians, especially, to develop this.
[01:49.340 --> 01:55.200]  You can get carpal tunnel here, you can get tennis elbow, you can... all of your nerves and your shoulders and neck and so on.
[01:55.340 --> 01:57.760]  You can get upset with a lot of repetitive motions.
[02:00.600 --> 02:02.100]  So, I was in this situation.
[02:02.780 --> 02:05.220]  I thought, okay, what can I do about this?
[02:05.220 --> 02:06.200]  So, clearly, I can't type.
[02:06.520 --> 02:08.380]  What should I... what should I do?
[02:08.730 --> 02:10.660]  So, here there'd be dragons.
[02:11.020 --> 02:16.700]  And actually, quite literally, because the main speech recognition software is called Dragon Naturally Speaking.
[02:18.730 --> 02:24.120]  I thought... there's no way this will work, but I saw a video of a guy who was actually able to code with this.
[02:24.200 --> 02:25.420]  So, I thought, well, maybe I'll give a shot.
[02:25.560 --> 02:26.240]  You'll see more about that.
[02:26.940 --> 02:32.360]  So, it's a... Dragon Naturally Speaking is a... it's a dictation and a command and control system for Windows.
[02:32.360 --> 02:37.140]  So, dictation meaning you can say English, like you would write an email, and it will type it out for you.
[02:37.360 --> 02:44.280]  And command and control meaning you can open Windows, you can say, open Firefox, select the search bar, type google.com, whatever.
[02:45.740 --> 02:46.840]  You know, it's kind of expensive.
[02:47.060 --> 02:51.440]  It's great for English and for writers and for, you know, if you're just doing normal English.
[02:51.580 --> 02:53.760]  But it's really pretty terrible for programming.
[02:53.900 --> 02:57.440]  I mean, it will insert commas and periods and capitalization and...
[02:57.920 --> 02:59.680]  Try to say, printf, hello world.
[02:59.960 --> 03:01.380]  It'll take you, like, a week.
[03:01.660 --> 03:04.900]  So, this is the problem.
[03:05.120 --> 03:06.480]  How to hack your Dragon.
[03:09.340 --> 03:11.500]  And there's actually quite a long history of this.
[03:11.620 --> 03:13.620]  So, Dragon Naturally Speaking has been around for many years.
[03:13.800 --> 03:15.100]  It's the biggest commercial product.
[03:15.240 --> 03:17.480]  It's had many millions of dollars poured into it.
[03:18.320 --> 03:23.260]  NatLink, as in Naturally Speaking, Link, I suppose, was the first attempt at hacking Dragon.
[03:23.260 --> 03:27.120]  And it was created, actually, by an employee of Dragon Systems.
[03:27.360 --> 03:31.740]  And he wanted to be able to write Python macros for this unwieldy speech recognition system.
[03:31.960 --> 03:33.880]  So, he wrote this crazy thing.
[03:34.000 --> 03:35.060]  And this is the actual hack, right?
[03:35.120 --> 03:39.660]  It goes in, pretends to be a Windows SAPI client, speech API client.
[03:40.880 --> 03:43.120]  And, you know, communicates with Dragon.
[03:43.260 --> 03:44.580]  Dragon's like, cool, you're a program.
[03:45.660 --> 03:46.820]  I'll start interacting with you.
[03:48.160 --> 03:49.560]  Yeah, so he wrote that a long time ago.
[03:49.560 --> 03:59.680]  And then, in 2008, Dragonfly was sort of like a compiler-compiler for these Python grammars.
[03:59.820 --> 04:02.000]  Or, like, a way of writing a much higher-level Python grammars.
[04:02.080 --> 04:09.020]  You don't have to say every individual thing and write it out like you would a, I don't know, like a regular expression or something like that.
[04:09.780 --> 04:10.900]  So, that was amazing.
[04:11.780 --> 04:14.120]  And this is the talk that I mentioned that inspired me.
[04:14.440 --> 04:16.080]  Tavis Rudd gave a talk at PyCon.
[04:16.080 --> 04:17.140]  You should look it up.
[04:17.320 --> 04:20.520]  It's called Using Python to Code by Voice.
[04:20.670 --> 04:21.580]  And it's very inspiring.
[04:22.200 --> 04:27.260]  So, I'm going to talk... I'm going to do a little bit about what he's doing and then I'm going to talk more about the open-source side of that.
[04:28.090 --> 04:31.380]  And Aenea is a reimplementation of what Tavis Rudd did.
[04:31.520 --> 04:34.090]  His system was so hacked together, he was never able to release the source code.
[04:34.290 --> 04:36.550]  But someone else reproduced it and released it online.
[04:37.640 --> 04:38.790]  And this is what Aenea looks like.
[04:38.900 --> 04:41.320]  And this is what I'm using, and you'll see a demo of it in a minute.
[04:42.380 --> 04:43.210]  It's very complicated.
[04:43.210 --> 04:46.640]  First of all, Dragon is a proprietary software and it only runs on Windows.
[04:46.940 --> 04:50.550]  You can kind of hack it with Wine, but it's very flaky.
[04:51.410 --> 04:52.230]  So, you need Dragon.
[04:52.500 --> 04:53.790]  You need NatLink to hack Dragon.
[04:54.110 --> 04:55.770]  You need Dragonfly to write better grammars.
[04:55.960 --> 04:58.910]  You need Aenea to do all this stuff.
[04:59.200 --> 05:00.050]  You write a grammar.
[05:01.050 --> 05:03.650]  And all of that is happening inside a Windows Virtual Machine.
[05:04.000 --> 05:06.170]  I have one running on this laptop with KVM.
[05:06.260 --> 05:09.790]  You can use VirtualBox or VMware, whatever you choose.
[05:11.320 --> 05:15.500]  So, the end result is you have speech coming in through USB microphone and virtual USB.
[05:15.900 --> 05:17.640]  Tons of recognition and crazy stuff happening.
[05:18.270 --> 05:20.790]  And then it sends the keystrokes back out to your Linux virtual machine.
[05:20.960 --> 05:22.550]  Or your Linux host machine.
[05:24.060 --> 05:24.460]  So...
[05:25.650 --> 05:26.050]  Yeah.
[05:27.400 --> 05:29.110]  How does this really help you for coding?
[05:30.820 --> 05:33.290]  I'm just going to show you a very basic voice grammar design.
[05:33.840 --> 05:36.460]  What would you do from the ground up if you had to design a voice grammar?
[05:36.460 --> 05:37.790]  So, first you need to be able to say letters.
[05:38.040 --> 05:39.610]  And most people start with the NATO alphabet.
[05:40.100 --> 05:40.760]  Alpha, Bravo, Charlie.
[05:41.420 --> 05:44.820]  And after a while you get tired of saying alpha and you shorten it to arch.
[05:45.020 --> 05:45.840]  Or I did anyway.
[05:46.200 --> 05:48.460]  And you, you know, develop a slight variation on that.
[05:49.400 --> 05:50.960]  You need to be able to say symbols and characters.
[05:52.020 --> 05:54.170]  So, space and one, two, three is easy.
[05:54.320 --> 05:56.100]  But you have to make up words for other things.
[05:56.550 --> 06:01.230]  And ideally you want to make up a word that is not common in normal English dictation.
[06:01.340 --> 06:03.790]  You don't want it to be confused with a normal word.
[06:03.790 --> 06:06.320]  So, I say act for escape rather than saying the whole word escape.
[06:08.380 --> 06:09.790]  Symbols, I developed a system for that.
[06:10.080 --> 06:11.480]  There's a lot of other people that do different things.
[06:11.600 --> 06:15.160]  But here I say either an L word or an R word if it's the left or right.
[06:15.480 --> 06:18.730]  And then parentheses, brackets, angle braces, whatever.
[06:19.040 --> 06:22.660]  So, len, ren, lace, race, langle, wrangle, that sort of thing.
[06:24.850 --> 06:26.440]  And, okay, now you can do individual symbols.
[06:26.550 --> 06:30.480]  But you really don't want to be coding all day by saying, India, November, tango, space.
[06:31.420 --> 06:34.700]  So, you want to be able to say some English words as well for variable names if nothing else.
[06:36.320 --> 06:42.110]  So, what you generally do is you say, alright, this is an escape word and it's going to start typing English words.
[06:42.350 --> 06:43.380]  And it will adjust the formatting.
[06:43.540 --> 06:44.980]  So, this is what Dragon doesn't let you do, right?
[06:45.200 --> 06:47.540]  So, I'll say sentence if I want it to be capitalized.
[06:48.040 --> 06:50.670]  I'll say score if I want there to be underscores, like a variable name.
[06:50.900 --> 06:52.580]  I can say camel if I want camel case.
[06:53.540 --> 06:55.470]  Proper camel if I want the beginning to be uppercase.
[06:56.320 --> 06:57.080]  And that kind of thing.
[06:57.920 --> 06:59.940]  So, now you can say English words.
[07:01.220 --> 07:02.900]  You can put commas and periods.
[07:03.040 --> 07:03.600]  You can write emails.
[07:03.920 --> 07:07.610]  You can maybe do some basic voice coding, but if you want to use some of these characters.
[07:08.500 --> 07:10.850]  And the other very important thing is you need to be able to say sequences.
[07:12.040 --> 07:17.260]  So, most voice systems are designed so you say one command and then it processes it and comes back.
[07:17.260 --> 07:20.170]  So, when I first wrote my voice grammar, I would say, you know, arch.
[07:20.380 --> 07:21.610]  And I have to wait half a second.
[07:21.760 --> 07:24.320]  Because if I say arch delta, it's going to be like, what's arch delta?
[07:25.540 --> 07:27.380]  So, and that puts a lot of complexity in the grammar.
[07:27.500 --> 07:28.100]  But you have to do it.
[07:28.760 --> 07:29.080]  Absolutely.
[07:29.290 --> 07:29.720]  So, chaining.
[07:30.110 --> 07:30.610]  Very important.
[07:32.640 --> 07:32.960]  Yes.
[07:33.160 --> 07:34.220]  I'm going to give a demo of this.
[07:34.380 --> 07:35.380]  So, I'm going to put this mic down.
[07:36.080 --> 07:36.520]  No, wait.
[07:36.580 --> 07:37.280]  Maybe I can do both at once.
[07:37.350 --> 07:37.580]  Let's see.
[07:39.850 --> 07:42.200]  Actually, one word about this before I go into it.
[07:44.340 --> 07:46.640]  Microphone is very, very important for speech recognition.
[07:46.640 --> 07:51.290]  So, if you try to use the built-in microphone on your laptop, even if it's a nice laptop, it's really not going to work.
[07:52.380 --> 07:53.600]  I tried various microphones.
[07:53.730 --> 07:56.220]  These are all the ones that I have and sort of use on a semi-regular basis.
[07:57.220 --> 07:58.580]  The USB microphones are okay.
[07:59.920 --> 08:00.640]  XLR microphones.
[08:00.730 --> 08:05.460]  XLR is the professional audio shielded connection.
[08:05.660 --> 08:06.350]  And that's the best.
[08:06.820 --> 08:09.320]  I'm going to be using the Shure microphone here to do the demonstration.
[08:25.970 --> 08:31.390]  So, as I mentioned, this is the commercial system, drag naturally speaking combined with Enya.
[08:32.430 --> 08:33.090]  Wake up.
[08:33.950 --> 08:34.870]  Desk left to.
[08:36.390 --> 08:37.210]  Wake up.
[08:38.110 --> 08:39.170]  Desk left to.
[08:40.550 --> 08:41.530]  Open new terminal.
[08:43.290 --> 08:45.530]  Charlie delta space slash tangle mic pop a slap.
[08:47.690 --> 08:49.770]  Line Sierra space minus line arch slap.
[08:51.450 --> 08:52.370]  Command vim.
[08:52.710 --> 08:53.710]  Word hello dit Charlie.
[08:55.330 --> 08:56.090]  Go to sleep.
[08:56.910 --> 09:02.310]  So, you see, I'm using some additional things like command vim or, you know, open new terminal.
[09:02.530 --> 09:03.570]  And those are things...
[09:03.570 --> 09:05.830]  Once you have the basic grammar, you start adding things that you do every day.
[09:06.230 --> 09:09.930]  I can also say everything by hand like alt F2, type this, type that.
[09:11.090 --> 09:11.670]  Wake up.
[09:12.830 --> 09:13.210]  Slap.
[09:14.370 --> 09:16.450]  India hash word include space langle.
[09:17.430 --> 09:18.290]  Sierra delta.
[09:18.810 --> 09:19.870]  Scratch tango delta.
[09:20.910 --> 09:22.870]  India Oscar dit hotel wrangle slap slap.
[09:24.530 --> 09:26.470]  India November tango space word main.
[09:27.950 --> 09:30.330]  Len Ren space lace slap to race up one tab.
[09:30.510 --> 09:32.110]  Word return space number zero semicolon.
[09:34.310 --> 09:35.990]  You guys, you messed it up.
[09:39.470 --> 09:40.550]  Let's try that again.
[09:41.610 --> 09:45.970]  Len Ren space lace slap to race up one tab.
[09:46.130 --> 09:47.830]  Word return space number zero semicolon.
[09:48.530 --> 09:50.030]  Act sky Oscar tab.
[09:50.550 --> 09:51.350]  Word print.
[09:52.090 --> 09:53.750]  Act visual line langle.
[09:54.650 --> 09:55.930]  Sky arch fox.
[09:56.850 --> 09:57.910]  Len double quote.
[09:59.210 --> 10:00.550]  Len double quote.
[10:01.610 --> 10:03.830]  Sentence hello comm space word world bang.
[10:04.390 --> 10:06.770]  Backslash November double quote Ren semicolon act.
[10:07.830 --> 10:08.930]  Vim save and quit.
[10:09.930 --> 10:12.170]  Golf charlie charlie space word hello tab.
[10:13.430 --> 10:15.750]  Golf charlie charlie space word hello tab.
[10:16.990 --> 10:18.890]  Minus oscar space word hello slap.
[10:20.130 --> 10:22.010]  Dit slash word hello slap.
[10:23.630 --> 10:24.530]  Go to sleep.
[10:33.870 --> 10:36.110]  Yeah, so it seems unwieldy at first.
[10:36.210 --> 10:37.810]  And at first you're like, oh man, I have to spell everything out.
[10:37.810 --> 10:39.950]  But it becomes second nature very quickly.
[10:40.070 --> 10:43.330]  Like now if someone asks me to spell something, I'll spell it delta arch tango arch.
[10:44.070 --> 10:45.790]  Like that's how I think of letters.
[10:46.450 --> 10:50.270]  And so I think it's roughly around half the speed of typing with your hands.
[10:50.390 --> 10:56.890]  And if you can use, if you can use your hands a little bit, like not fully, but you can use them for things that are slower with voice, then you know it's not much slower.
[10:56.890 --> 11:00.130]  I wrote a full system with this for my research.
[11:00.270 --> 11:06.010]  I wrote a full paper with this, which unfortunately was not accepted, but nothing to do with the voice recognition.
[11:06.910 --> 11:08.310]  So, yeah, it's very powerful.
[11:10.790 --> 11:20.050]  And I really encourage, like I think this is right now the best way to continue to code if you are unable to use your hands, like using the Aeneas setup and all this complexity.
[11:20.230 --> 11:26.270]  And it's quite hard to set up, but there's a great mailing list, so if anyone is actually, like actually can't type, this is where you should go.
[11:27.030 --> 11:29.790]  Go to this mailing list, buy a good microphone like this one.
[11:29.990 --> 11:33.210]  It doesn't have to be super expensive, this is around $100, $80 or something.
[11:34.790 --> 11:36.070]  And you can really get some work done.
[11:38.710 --> 11:44.430]  So, I had been using this for a year, and I was like, this is so cool, I wish this was more of a thing.
[11:44.570 --> 11:47.790]  You know, no one thinks about using voice input for a computer, not for programming at least.
[11:49.030 --> 11:51.590]  So, I looked into how could I make this easier for people.
[11:51.810 --> 11:54.130]  So the main thing with Aeneas, it's very difficult to set up.
[11:54.250 --> 11:56.790]  So there's actually people on the mailing list that try and try and they never get it.
[11:56.890 --> 11:58.130]  And they write their own thing instead.
[11:58.310 --> 12:05.990]  Just because there's so many moving pieces, and they can't distribute a working virtual machine because it has lots of proprietary, excuse me, lots of proprietary software in there.
[12:06.110 --> 12:08.690]  You know, there's Dragon, there's Windows, there's maybe VMware.
[12:09.150 --> 12:10.510]  And so, it's quite complicated.
[12:11.690 --> 12:12.690]  So, I thought, okay.
[12:14.270 --> 12:15.090]  How can I do this?
[12:16.450 --> 12:17.870]  I think I mentioned most of this already.
[12:18.890 --> 12:19.950]  Some people never get it working.
[12:20.650 --> 12:28.550]  The other thing is that people, like me, use it for years, and they write their own custom grammar, and they're like, my grammar can do everything and slice cheese and open Emacs and whatever.
[12:29.590 --> 12:33.110]  But it's very hard, it's very hard to combine different people's grammars.
[12:33.450 --> 12:35.770]  You know, I use my coding style and my grammar.
[12:36.270 --> 12:39.070]  You can use my Vim grammar, but you can't combine it with your Firefox grammar.
[12:39.210 --> 12:39.830]  It's just not possible.
[12:40.410 --> 12:44.630]  So, some people have tried to make like a standard toolbox for all this, but no one really uses it.
[12:44.670 --> 12:49.290]  So when I was starting out, I hacked together four different sources until I figured out how to write my own voice grammars.
[12:50.350 --> 12:51.270]  So, that's a bit of a problem.
[12:51.390 --> 12:52.850]  And you need significant computing power.
[12:52.950 --> 12:54.590]  And this is a brand new laptop with a Skylake processor.
[12:55.530 --> 12:59.990]  Previously, I had to upgrade my RAM, I bought a separate server, I put it under my desk, I ran the speech recognition there.
[13:00.090 --> 13:00.810]  There was no other way.
[13:01.530 --> 13:03.730]  So, I was, okay, is there an easier way of doing this?
[13:05.950 --> 13:12.390]  I thought I could buy Dragon Instances myself, run them in the cloud, then people could at least try it out before they installed it on their own system.
[13:14.330 --> 13:15.030]  Really not possible.
[13:15.130 --> 13:19.170]  I called up the Nuance director of sales and I said, you know, can I do this?
[13:19.170 --> 13:19.810]  Buy your license agreement.
[13:19.950 --> 13:23.270]  And he's like, yeah, but you'll have to buy our SDK and that's like five or ten thousand.
[13:23.550 --> 13:24.350]  And I was like, okay, never mind.
[13:26.430 --> 13:28.350]  So, and there's also scripting issues.
[13:28.470 --> 13:29.310]  Like Dragon will crash.
[13:29.850 --> 13:30.870]  Every now and then it just hangs.
[13:31.070 --> 13:33.610]  I think it misses audio frames from the microphone.
[13:34.650 --> 13:36.790]  And you have to go into the UI and fiddle around with it.
[13:36.890 --> 13:42.270]  And so there's no real way to do that across the cloud unless I gave, you know, remote access to the people who are using it.
[13:42.330 --> 13:43.910]  Like a remote desktop or something like that.
[13:44.790 --> 13:47.530]  And speaking of remote, remote microphone is very difficult to do.
[13:47.530 --> 13:50.690]  When I said I bought a server, I plugged the USB directly into the server.
[13:51.010 --> 13:54.010]  Because USB virtualization is extremely high bandwidth.
[13:54.850 --> 13:57.610]  Audio streaming, fine with VLC or something.
[13:57.890 --> 13:59.270]  But Dragon doesn't open it.
[13:59.370 --> 14:02.110]  It's like, this is not a real microphone if you create a virtual microphone.
[14:02.330 --> 14:03.370]  So, okay.
[14:04.050 --> 14:06.810]  You can use Microsoft RDP, have fun with that on Linux.
[14:08.310 --> 14:12.070]  Anyway, I thought I could get it to work for around $5 a month by buying Dragon off eBay.
[14:12.070 --> 14:14.570]  Obviously not the best business plan.
[14:18.250 --> 14:19.970]  So I looked into other kinds of speech recognition.
[14:20.290 --> 14:23.330]  Okay, there's not just Dragon, although that does have around like 80% of the market share.
[14:25.010 --> 14:27.310]  There's a lot of cloud-based speech recognition for smartphones.
[14:27.510 --> 14:29.230]  You know, Siri, Google.
[14:31.510 --> 14:32.850]  And it's very hard to get an API.
[14:32.970 --> 14:34.630]  I noticed that Google just released a new API.
[14:34.710 --> 14:36.410]  This was not available when I was trying this.
[14:36.410 --> 14:37.730]  So I might have gone that route.
[14:37.990 --> 14:42.910]  But the dedicated APIs like Hound and Nuance, they actually charge a lot of money.
[14:43.070 --> 14:44.170]  So I think Hound...
[14:45.310 --> 14:47.170]  And maybe I shouldn't say this because I'm not sure how public this is.
[14:47.270 --> 14:50.730]  But they give you a small number of recognitions per month.
[14:50.830 --> 14:53.190]  And then after that, they charge you a certain number of cents per recognition.
[14:53.450 --> 14:57.110]  So they're expecting you to have a mobile phone and you use it, you know, every hour or something like that.
[14:57.170 --> 14:57.990]  Not every 10 seconds.
[14:58.910 --> 14:59.730]  So that's no good.
[15:01.290 --> 15:02.670]  Local recognition on your smartphone.
[15:02.930 --> 15:04.610]  To me, this is the ideal because everyone has a smartphone.
[15:04.850 --> 15:06.150]  If you could get that to work, it would be amazing.
[15:06.150 --> 15:07.770]  There's some papers from Google Research.
[15:07.910 --> 15:09.010]  They're just coming out about this.
[15:09.110 --> 15:11.130]  So I think they'll probably try and do that relatively soon.
[15:12.450 --> 15:13.390]  Others, Amazon Echo.
[15:13.510 --> 15:15.410]  There's a Kickstarter project for Arduino Shields.
[15:16.610 --> 15:17.310]  So, okay.
[15:17.490 --> 15:18.530]  Nothing really fit what I needed.
[15:18.610 --> 15:20.010]  Time to reinvent the wheel.
[15:20.190 --> 15:20.550]  Maybe.
[15:22.250 --> 15:23.170]  I'm a student at Columbia.
[15:23.410 --> 15:23.990]  I looked around.
[15:24.090 --> 15:25.270]  There's a speech recognition class.
[15:25.370 --> 15:26.350]  So I thought, well, cool.
[15:26.510 --> 15:27.010]  I'll take that.
[15:27.290 --> 15:30.450]  And I learned about some open-source speech recognition toolkits.
[15:30.630 --> 15:31.290]  There's a bunch.
[15:32.390 --> 15:33.470]  HDK used to be pretty popular.
[15:33.630 --> 15:34.210]  CMU Sphinx.
[15:34.210 --> 15:38.790]  There was a huge research group at Carnegie Mellon that was doing voice recognition for a long time.
[15:38.910 --> 15:39.810]  And they made Pocket Sphinx.
[15:39.870 --> 15:41.030]  They made in Java.
[15:41.170 --> 15:42.650]  And they made a C++ CMU Sphinx.
[15:43.870 --> 15:45.470]  Which is really best in the world for a long time.
[15:45.570 --> 15:47.790]  But all those guys graduated, got very high paying jobs.
[15:47.790 --> 15:50.450]  And now, there's no one at Carnegie Mellon doing that.
[15:51.130 --> 15:51.530]  So...
[15:52.190 --> 15:53.270]  The price of success.
[15:53.630 --> 15:55.450]  And now, the big project...
[15:55.450 --> 15:57.710]  Or the big open-source system that people use is called Caldee.
[15:58.150 --> 16:00.010]  And that's out of John Hopkins University.
[16:00.170 --> 16:02.450]  And there's a very large research group around that.
[16:03.030 --> 16:04.030]  Led by Dan Povey.
[16:04.230 --> 16:04.710]  Great guy.
[16:06.230 --> 16:06.990]  And so, yeah.
[16:07.070 --> 16:08.390]  Most current research is happening on Caldee.
[16:08.470 --> 16:09.430]  So that's what I decided to use.
[16:09.870 --> 16:13.370]  And just really briefly, I'm going to show you a bit about what speech recognition entails.
[16:13.370 --> 16:14.310]  Because I think it's pretty cool.
[16:14.670 --> 16:15.150]  There's...
[16:15.150 --> 16:18.570]  It's a very broad range of stuff to get this all to work.
[16:18.690 --> 16:19.410]  There's signal processing.
[16:20.050 --> 16:21.130]  There's acoustic modeling.
[16:21.290 --> 16:22.010]  And there's language modeling.
[16:22.310 --> 16:23.590]  So the signal processing is just...
[16:23.590 --> 16:24.930]  You have an input sound wave.
[16:24.970 --> 16:26.810]  How do you extract features that you can do machine learning on?
[16:27.370 --> 16:28.970]  Acoustic modeling is that machine learning.
[16:29.330 --> 16:34.790]  And language modeling is trying to interpret those A's and B's and C's to something meaningful.
[16:35.670 --> 16:36.730]  So here's signal processing.
[16:37.030 --> 16:42.210]  This is me saying horse and horse into my microphone.
[16:42.210 --> 16:43.990]  So you can see a little bit of what's going on.
[16:44.450 --> 16:45.310]  Just some quick notes.
[16:45.690 --> 16:48.510]  Speech is always recognized at 16kHz.
[16:48.630 --> 16:50.570]  16k samples and 16k range.
[16:51.030 --> 16:53.810]  And because of folding, you can measure up to 8k.
[16:53.970 --> 16:57.530]  And that's all human speech basically is less than 8kHz.
[16:59.590 --> 17:03.110]  Except phone signals which only measure up to 4kHz.
[17:03.230 --> 17:05.850]  So they do 8k sampling and because of folding it goes down to 4k.
[17:06.290 --> 17:10.630]  So if you look at this thing here, there's some stuff that's above 4k.
[17:11.590 --> 17:16.470]  Particular fricative sounds like S or T or TH, those kinds of things.
[17:16.690 --> 17:18.230]  They're higher than 4k.
[17:18.470 --> 17:22.950]  And when you speak over a cell phone, or a traditional cell phone line anyway, it's going to get cut off.
[17:23.110 --> 17:26.850]  And that's why you have so much trouble saying to someone over the phone, it's THS.
[17:26.930 --> 17:27.590]  Did you say...?
[17:27.590 --> 17:27.770]  Yeah.
[17:28.370 --> 17:30.390]  So because they actually cut off that.
[17:30.470 --> 17:31.550]  And that was just to save some bandwidth.
[17:31.650 --> 17:34.930]  Because they knew that speech recognition really needs this 8k.
[17:34.930 --> 17:36.310]  But they're like, no, we can't do that.
[17:37.630 --> 17:40.950]  And the other interesting thing here is vowels have formats.
[17:41.110 --> 17:43.610]  And these are like... what do you call it?
[17:43.630 --> 17:45.350]  They're like harmonics in music, right?
[17:45.410 --> 17:50.910]  So you can see when I said or in horse, there's that white patch that's a very resonant sound.
[17:51.310 --> 17:53.590]  And there's the first format, which is the white one.
[17:53.850 --> 17:55.690]  Second, third, maybe even the fourth format.
[17:55.690 --> 18:01.270]  And that makes it very easy to recognize in speech recognition.
[18:01.970 --> 18:05.330]  You're kind of like recognizing the vowels and piecing together everything else.
[18:09.870 --> 18:11.810]  And yeah, I won't go into this.
[18:11.950 --> 18:12.230]  What else?
[18:12.390 --> 18:12.490]  Oh yeah.
[18:13.030 --> 18:14.530]  The frames that you...
[18:14.530 --> 18:17.070]  So you slice this speech out, right?
[18:17.130 --> 18:18.330]  Into 10 millisecond frames.
[18:18.590 --> 18:21.530]  Because most phones that we speak are around 30 milliseconds.
[18:21.850 --> 18:25.590]  So I'm saying horse, and I say, you know, three phones or something like that.
[18:25.590 --> 18:26.790]  It's 30 milliseconds for each one.
[18:27.250 --> 18:29.350]  So think about that from a computer's perspective.
[18:29.490 --> 18:30.290]  That's forever, right?
[18:31.570 --> 18:33.290]  Some of them are five milliseconds, like stops.
[18:34.210 --> 18:34.650]  Like...
[18:35.930 --> 18:36.650]  What's the example?
[18:36.970 --> 18:37.210]  Like...
[18:37.210 --> 18:38.430]  Or something like that.
[18:38.670 --> 18:39.910]  Those are fricatives as well, but anyway.
[18:41.530 --> 18:42.430]  Yeah, so you get...
[18:42.430 --> 18:44.470]  You're extracting what's called...
[18:45.310 --> 18:45.750]  Kepstrel...
[18:46.210 --> 18:48.370]  Mel frequency Kepstrel coefficients from this.
[18:48.470 --> 18:49.850]  And that's what you use for your machine learning features.
[18:51.170 --> 18:52.190]  They're kind of like derivatives.
[18:52.590 --> 18:55.650]  You get an infinite series of them with, you know, higher and higher degree.
[18:56.230 --> 18:57.230]  And they're actually...
[18:57.230 --> 18:57.930]  The ratio...
[18:58.690 --> 19:00.050]  I wrote it down there, but...
[19:00.050 --> 19:02.150]  They're the ratio of different quadrants.
[19:02.230 --> 19:04.570]  So the zeroth one is just the mean value.
[19:05.010 --> 19:08.270]  The first one is the ratio of above and below.
[19:09.170 --> 19:13.730]  The second one is the ratio of the even quadrants, if you divide into four, to the odd quadrants.
[19:14.170 --> 19:17.430]  And then you divide into eight quadrants, and you look at the even ones versus the odd ones.
[19:17.530 --> 19:21.390]  And this kind of gives you a notion of whether the signal's going up or down.
[19:21.390 --> 19:25.790]  You can't use a normal derivative because all these signals are periodic.
[19:25.990 --> 19:30.630]  If you just use a normal one, at any point it looks like it's going up when it's not actually going up.
[19:31.090 --> 19:35.390]  So some guys in the 80s at IBM developed this, and no one's found something better.
[19:35.610 --> 19:36.650]  So it's very complicated.
[19:36.730 --> 19:42.030]  You take the Fourier transformation, then you do the Mel scaling, which sort of emphasizes the frequencies that the human ear can hear.
[19:42.450 --> 19:45.030]  You take the log, you take the cosine transformation, whatever.
[19:45.410 --> 19:46.950]  But yeah, that's what the machine learning uses.
[19:48.110 --> 19:51.090]  And acoustic modeling, so, okay, this is the hard part, right?
[19:51.270 --> 20:03.330]  You have to go out there and listen or record people speaking for hundreds of hours, and you have to generate a transcript of what they said, so that the speech recognition can come and say, okay, this guy said horse, and here are the characters H-O-R-S-E,
[20:03.470 --> 20:08.790]  and so, oh, when you say an O after an H but before an R, it kind of sounds like this.
[20:09.610 --> 20:17.190]  You have to take a lot of context into account, because we don't say an O the same way based on, you know, it's heavily dependent on what you said before and what you said after.
[20:19.190 --> 20:28.990]  Yeah, so you can model the acoustics with Gaussian mixer models, or more recently, deep neural networks, and you have good, you have good efficacy with that.
[20:30.410 --> 20:37.090]  And then you create a hidden Markov model that will, that will use these, like, smaller acoustic model pieces, and figure out what you're actually saying.
[20:37.330 --> 20:40.030]  The problem is training this is very computationally expensive.
[20:40.330 --> 20:45.050]  So, think about it, you have hundreds of hours of speech, you're creating, like, you're training deep neural networks.
[20:45.370 --> 20:53.110]  I had a 24 core server with a lot of RAM, and, you know, it took, I don't know, two weeks or something like that to finish.
[20:53.170 --> 20:56.210]  I didn't actually use the model I trained, I just grabbed some ones from online.
[20:57.130 --> 20:57.770]  Much easier.
[20:58.770 --> 21:02.710]  So, you know, the researchers that had already trained it on a cluster rather than just one server.
[21:03.350 --> 21:06.790]  So, and then third part, language modeling.
[21:06.990 --> 21:13.390]  So, okay, I could say, I could say English, English, and I use nouns and verbs, and I string them together in a way that makes sense.
[21:13.570 --> 21:16.150]  Or I could say complete nonsense, but that's significantly less likely.
[21:16.650 --> 21:20.730]  So, with a three gram, it's like, any three words, what's the probability of that happening?
[21:21.150 --> 21:28.570]  So, you write down a bunch of sequ-, you know, you scan Wikipedia, and hundreds of other websites, and you say, this happens, this happens, this happens, with this probability.
[21:29.110 --> 21:37.510]  And that way, the speech recognition system can say, oh, I didn't really understand what you heard, but, what you said, but I got this part, and this part, and the most common thing in the middle is that.
[21:38.150 --> 21:42.870]  So, the system that I'm using, the open-source one, you'll see later, is a three gram-based one.
[21:43.410 --> 21:45.770]  Google actually has created a five gram, which is insane.
[21:45.970 --> 21:47.010]  They just have so much data.
[21:47.610 --> 21:51.190]  And Dragon used to use four gram, and now is using a five gram as well.
[21:52.070 --> 21:55.270]  The five gram, their five gram is incredibly slow, and you can't really run it in a virtual machine.
[21:55.690 --> 21:57.870]  So, we use the four gram for that kind of thing.
[21:57.970 --> 21:58.090]  Yes?
[21:58.170 --> 21:59.390]  Is that a sliding sequence?
[21:59.930 --> 22:00.490]  Yes, it is.
[22:00.710 --> 22:04.850]  So, you have ten words, you look at three gram, and then you slide over one word, and so on.
[22:05.030 --> 22:09.390]  And a lot of the, I mean, most of the word combinations are not there, there's zero, right?
[22:10.390 --> 22:12.810]  And so, you can infer some of them from two grams.
[22:12.990 --> 22:15.050]  So, like, a three gram model will have two grams, and we'll have one grams.
[22:15.230 --> 22:18.410]  So, if you see something usually, you just fall back to the two gram or the one gram model.
[22:20.170 --> 22:23.370]  Anyway, in our case, we want to be able to speak voice commands.
[22:23.430 --> 22:24.950]  So, we need to change this language model.
[22:25.690 --> 22:28.170]  We have different statistics on our commands.
[22:28.270 --> 22:29.750]  We don't just have English statistics.
[22:31.510 --> 22:34.110]  So, that's part three, open sourcing this.
[22:35.890 --> 22:37.710]  This is the intro slide for my speech system.
[22:37.810 --> 22:38.530]  It's called Silvius.
[22:39.310 --> 22:43.090]  That's because Silvius was the son of Aeneas in Virgil's epic.
[22:44.930 --> 22:49.430]  And I wanted to be open-sourced for sure with freely available speech models.
[22:49.850 --> 22:58.950]  I wanted you to be able to run it locally or in the cloud for the greatest flexibility, because there's a lot of computing requirements, and to be able to use a custom speech grammar, obviously.
[22:59.830 --> 23:03.590]  And also, to do speech recognition with minimum hassle.
[23:03.690 --> 23:04.490]  I mean, that's the point of this.
[23:04.830 --> 23:14.070]  Aenea already does it well, but it's not very hackable, and it's, you know, very difficult to set up, and so people are not going to try it until they absolutely can't type.
[23:14.550 --> 23:16.190]  So, yeah.
[23:16.330 --> 23:22.190]  And the ultimate goal here is, can we have a speech recognition system that needs no software installation on your computer at all?
[23:22.350 --> 23:24.210]  And we're actually almost there, and you'll see that.
[23:26.910 --> 23:30.930]  So, the question is, how do you use a custom grammar?
[23:32.910 --> 23:35.970]  The three-gram thing that I mentioned is good when you can say almost absolutely anything.
[23:36.170 --> 23:40.590]  If you have a lot of structure to your grammar, you might instead try a rule-based language model.
[23:40.810 --> 23:44.510]  This is what you might see in Bison or something like that if you wrote a programming language grammar.
[23:44.710 --> 23:49.610]  You could say, oh, you can say int, and then you can say a variable, and then you can say parentheses, parentheses, you know, that kind of thing.
[23:50.770 --> 23:55.130]  It's not great for voice, because whatever you say, it will try to fit into one of those branches.
[23:56.230 --> 23:56.690]  Pardon me.
[23:57.070 --> 24:06.450]  So, if I say int caterpillar fox charlie, or something random, it will be like int, you must have meant main, you must have meant parentheses.
[24:07.490 --> 24:12.730]  So, yeah, if you make a mistake or you say something wrong, it'll just be like, it'll interpret it.
[24:12.870 --> 24:13.550]  So you don't want that.
[24:14.990 --> 24:17.330]  You can attempt to merge two language models together.
[24:17.490 --> 24:19.450]  And if you have two natural languages, this is actually good.
[24:20.030 --> 24:23.730]  Baidu's created a Mandarin and English speech recognition.
[24:23.990 --> 24:24.550]  Quite a feat.
[24:24.710 --> 24:26.430]  It took them 10,000 hours of training data.
[24:26.590 --> 24:30.930]  And I don't want to know how many deep neural networks had to throw away before they got one that worked.
[24:31.430 --> 24:32.550]  So, okay.
[24:32.730 --> 24:34.890]  You can also retrain with command words interspersed.
[24:34.970 --> 24:39.490]  So you can take your original Wikipedia source, and you can just put random words in the middle, wherever they happen to be.
[24:39.590 --> 24:43.030]  And then it gets used to seeing delta in the middle of random stuff.
[24:43.390 --> 24:44.810]  So that's a lot of work, though.
[24:44.870 --> 24:48.310]  So I did the simplest thing, which is a linear combination of two grammars.
[24:48.750 --> 24:57.370]  I extract an n-gram from whatever the user has specified as a grammar.
[24:57.570 --> 25:00.510]  I take the n-gram from the English grammar, and I just combine them.
[25:00.610 --> 25:03.370]  80% from the English one, 20% from the command one.
[25:03.650 --> 25:04.810]  And that seems to work fairly well.
[25:05.270 --> 25:11.750]  Of course, the grammar that the user writes, I have to be able to iterate over it and generate the n-grams, because the user's not going to sit there and write their own n-grams.
[25:12.210 --> 25:14.130]  So how do we do that?
[25:14.890 --> 25:17.450]  Well, the Sylvius grammars are written in Python.
[25:17.630 --> 25:18.710]  All of this is written in Python, by the way.
[25:18.790 --> 25:24.270]  For some reason, Python became the language of speech stuff after, back in 91, when Natlik was created, I suppose.
[25:24.450 --> 25:26.250]  All the grammars, everything has been written in Python.
[25:27.610 --> 25:30.850]  So we use the, I use the Spark parsing toolkit, which is kind of cool.
[25:30.990 --> 25:32.910]  It's like writing Bison or something like that.
[25:33.010 --> 25:34.550]  You write a compiler-compiler.
[25:34.950 --> 25:44.730]  But when you, once you've written your compiler-compiler, your source code is now available as Python meta-objects and you can do introspection on it.
[25:45.530 --> 25:55.370]  So, I mean, a compiler-compiler's job is to make, as you can see on the, on the diagram there, Spark is creating a parser that can take text input and make an abstract syntax tree, a parse tree.
[25:55.710 --> 26:02.090]  But the Spark code itself has a tree, and that represents the, the syntax of the abstract syntax tree, sort of.
[26:02.390 --> 26:09.290]  And you can iterate over that, which is, I don't think anyone ever really needs to do this, but as a side effect of the design, I can do this.
[26:10.410 --> 26:13.210]  So, from that, we can automatically generate the n-gram statistics.
[26:13.510 --> 26:17.110]  We can say, I don't know with what probability you'll say which word.
[26:17.230 --> 26:23.850]  I just say, okay, every word is equally likely, but after this word, you can say this, after this word, you can say that, take the chaining into account, all that stuff.
[26:25.430 --> 26:29.130]  So, this is what Silvius looks like when you're running it.
[26:29.670 --> 26:34.270]  There's the Spark parser on the left that I mentioned in your grammar written on top of it.
[26:34.330 --> 26:41.130]  I extract the n-grams, and create a command language model, merge it with the English language model, and that's fed to Caldy.
[26:41.390 --> 26:44.250]  And that's the third component of your speech recognition.
[26:45.690 --> 26:49.270]  From the client side, we have a microphone and remote streaming is going on.
[26:49.390 --> 26:53.530]  I mean, you can run this on one computer if you want, but there's remote streaming going on, sends packets.
[26:53.750 --> 27:03.330]  This is compressed, it's not like crazy USB virtualization like before, and it goes to a web server, which passes off to Caldy, sends the stuff back to your Spark parser so it can do the normal parsing that it's designed to do.
[27:03.450 --> 27:08.570]  Take the text input, and produce a parse tree which you can iterate over and execute commands.
[27:10.150 --> 27:11.130]  Yeah, and a huge thanks.
[27:11.390 --> 27:16.250]  A lot of this infrastructure was created by Tano Alamai for his, I guess it's his PhD research.
[27:16.510 --> 27:20.650]  He created this remote streaming stuff, and I leveraged that a lot for this project.
[27:22.370 --> 27:23.590]  So, here's what you can do with Silvius.
[27:24.270 --> 27:28.610]  You can run the recognition completely locally, and you need a lot of RAM for that, a lot of CPU.
[27:29.130 --> 27:30.270]  You can use cloud servers.
[27:30.590 --> 27:36.630]  I currently have two free servers running in the server that I mentioned under my desk since then, you know, it's there.
[27:36.870 --> 27:44.690]  And I think I can provide this for, I think I can provide this for like three or four dollars a month if people are interested in, or if there's interest in deploying this at scale.
[27:46.610 --> 27:48.230]  And you can run it on embedded systems.
[27:50.370 --> 27:52.170]  You know, if you're lucky, maybe a phone.
[27:52.270 --> 27:53.750]  Right now it needs x86, but I'm hopeful.
[27:54.230 --> 27:55.730]  Or you can do a voice box or something.
[27:55.930 --> 27:59.470]  One of the demos you'll see is this little computer right here, which you might recognize.
[27:59.690 --> 28:01.650]  It's an MSI QB.
[28:01.870 --> 28:09.730]  It has a Broadwell, so 5000 series, i3 processor, which is more than you need for Silvius, but I want it to be on the safe side.
[28:11.050 --> 28:14.890]  It costs around $300 to assemble and put a mini SSD in and stuff like that.
[28:14.890 --> 28:16.570]  I can do the recognition on there.
[28:17.870 --> 28:27.110]  And like I said, the ultimate goal is to be able to use the ultimate, or the recognition results on this computer without me having to install any software on this computer.
[28:28.670 --> 28:31.090]  Sounds strange, but how do you do that?
[28:31.190 --> 28:31.650]  With this.
[28:33.570 --> 28:35.330]  Which is something my brother has created.
[28:36.590 --> 28:40.110]  It is a Bluetooth to fake USB dongle.
[28:40.510 --> 28:43.930]  So what happens is this thing recognizes my speech.
[28:44.050 --> 28:45.070]  The microphone is plugged into this thing.
[28:45.250 --> 28:50.490]  This thing recognizes my speech, has a Bluetooth connection to this dongle, and says, I want you to press D.
[28:50.950 --> 28:55.450]  And then it goes, oh, if I was a USB keyboard, I would send this in order to press D.
[28:55.630 --> 28:56.290]  And it does that.
[28:58.470 --> 28:58.830]  So...
[29:00.930 --> 29:01.450]  Thank you.
[29:02.730 --> 29:11.090]  So I can have this little voice box, plug a microphone into it, come up to any computer, plug in the USB, and start speaking at it.
[29:11.410 --> 29:12.950]  There's no software installation necessary.
[29:13.890 --> 29:18.410]  So, unfortunately, this is on the other side of the country, in Vancouver, so I can't show you that right now.
[29:18.590 --> 29:21.670]  This is my attempt to recreate it for this talk, but it didn't quite finish.
[29:22.450 --> 29:23.710]  Didn't quite work out so well.
[29:25.070 --> 29:29.690]  But, nevertheless, I will show you this in a second demo.
[29:30.050 --> 29:33.570]  So, for demonstration purposes, this is also broadcasting...
[29:34.150 --> 29:36.710]  By the way, this has been on the whole time, so it's been listening to everything I've been saying.
[29:37.490 --> 29:40.150]  This is broadcasting an unsecured wireless network.
[29:40.510 --> 29:41.290]  Unsecured, be nice.
[29:41.770 --> 29:46.370]  Unsecured wireless network called VoxCube, because that's maybe what I'm calling it.
[29:46.570 --> 29:54.330]  And if you connect to VoxCube, you can see the recognition results that it's generating.
[29:55.110 --> 29:56.090]  Let's see here.
[30:02.590 --> 30:07.190]  And if my computer is not connecting to it, come on.
[30:08.670 --> 30:13.270]  So, it's just running a little web server in there, and it lets you look at the recognition results in the log and whatnot.
[30:14.870 --> 30:15.830]  Did that work or not?
[30:17.410 --> 30:23.570]  Well, I'll reboot it, and give it a second attempt at greatness.
[30:25.670 --> 30:28.530]  Meanwhile, I'll show you Sylvia's running locally on my computer.
[30:30.190 --> 30:34.310]  So, like I said before, we can run it locally, we just need a lot of computing power for it.
[30:34.510 --> 30:37.310]  This is Ania, by the way, the little Windows virtual machine.
[30:39.450 --> 30:40.230]  Where is it?
[30:40.310 --> 30:40.690]  Here it is.
[30:40.850 --> 30:45.290]  So, I have this master process running, and a single worker process.
[30:45.290 --> 30:48.050]  I mean, I could have multiple, but I'm going to be the only one using this.
[30:50.710 --> 30:55.610]  And normal recognition looks something like this.
[30:56.370 --> 30:57.290]  Hello, over there.
[30:59.150 --> 31:00.970]  Open-source speech recognition at its best.
[31:06.390 --> 31:13.450]  So, what I'm doing here is I'm using my internal laptop microphone, which is, by and large, very poor quality.
[31:14.890 --> 31:16.010]  But it gets the job done.
[31:16.630 --> 31:18.550]  And it's just printing the results.
[31:19.690 --> 31:24.610]  If I wanted to actually execute commands, I would pipe this through the Spark Python grammar.
[31:28.160 --> 31:30.700]  So, it's just two pieces, like this, one executes the other.
[31:31.160 --> 31:32.760]  So, whoops.
[31:36.040 --> 31:37.100]  Charlie delta space.
[31:38.580 --> 31:40.020]  Slash tango mic pop a slap.
[31:41.620 --> 31:43.300]  Slash tango mic pop a slap.
[31:44.620 --> 31:46.920]  Line Sierra space minus line arch slap.
[31:48.500 --> 31:49.960]  Victor India mic space.
[31:51.440 --> 31:52.620]  Hotel tab slap.
[31:54.440 --> 31:55.220]  Vim quit.
[31:56.540 --> 31:57.420]  Vim quit.
[31:58.680 --> 31:59.680]  Colon queen slap.
[32:01.160 --> 32:02.020]  Up one.
[32:02.540 --> 32:04.640]  It doesn't actually understand vim, I just forgot.
[32:07.980 --> 32:09.140]  Dot charlie slap.
[32:11.660 --> 32:12.380]  Down three.
[32:13.920 --> 32:15.080]  Yankee yankee papa.
[32:16.720 --> 32:18.380]  Fox double quote, right.
[32:20.080 --> 32:21.260]  Fox sky hotel.
[32:22.540 --> 32:27.160]  Forgot to mention, uppercase letters, people say shift or whatever, I say sky because it's short and it's up there.
[32:29.140 --> 32:30.720]  Charlie tango backslash.
[32:32.560 --> 32:35.620]  Sentence open-source speech recognition at its best.
[32:37.220 --> 32:38.580]  Hey, that actually worked.
[32:47.000 --> 32:48.440]  Anyway, you get the idea.
[32:50.560 --> 32:57.440]  So, what I've done with this is, I've also fed it the command grammar and I have...
[32:59.440 --> 33:01.340]  Yeah, like I said, I merged the models together.
[33:01.520 --> 33:07.840]  So, I've trained it so that Charlie, tango, delta, those words are a lot more frequent than they would be in normal English.
[33:08.200 --> 33:15.720]  So, and that process takes a long time and a lot of RAM, but you can actually introduce new words whenever you want to, because it recognizes full English words.
[33:15.840 --> 33:18.520]  You just have to be, you know, careful when you say those words or whatever.
[33:19.560 --> 33:24.560]  So, yeah, you can iterate quickly on it and then once you've found a grammar that you like, you can go and lock it in.
[33:25.220 --> 33:27.340]  And train the new language model.
[33:28.780 --> 33:33.220]  Let's see if this one is up to its full capacity.
[33:33.340 --> 33:33.680]  Oh, there we go.
[33:36.260 --> 33:37.240]  So, one moment.
[33:39.160 --> 33:42.140]  And you should be able to enter anything here in a redirect, but I'm not going to count on that.
[33:44.460 --> 33:48.380]  So, the whole time, this has been listening to what I've been saying.
[33:52.220 --> 33:52.960]  Oh, yeah, sorry.
[33:57.640 --> 34:00.000]  You could just connect from your phone to watch what's happening here.
[34:00.160 --> 34:01.500]  You know, trust my unsecured network.
[34:03.780 --> 34:07.280]  So, this is a mini USB microphone.
[34:11.220 --> 34:13.100]  Well, it doesn't have USB, but oh well.
[34:14.840 --> 34:17.200]  And I used this for recognition for a while.
[34:20.830 --> 34:22.670]  It's very good for podcasting and so on.
[34:22.670 --> 34:25.850]  It's not fantastic for speech recognition, but it's very cheap.
[34:26.030 --> 34:28.190]  I got it for like $40 or something like that.
[34:28.750 --> 34:32.610]  And luckily, it outputs 16K audio natively.
[34:32.810 --> 34:40.390]  A lot of other microphones, you need to, you know, do some conversion on your own to get the to get the right frequency.
[34:41.210 --> 34:45.590]  So, right now, I'm relying on system stuff to do that, like pulse audio can do that.
[34:45.750 --> 34:48.590]  But it also requires an X server, which this doesn't have, so you know, whatever.
[34:49.750 --> 34:52.270]  Yeah, so this is a demonstration of what I'm trying to create.
[34:52.590 --> 34:55.630]  Of course, normally, I would be saying delta tango, delta tango.
[34:55.990 --> 34:59.630]  And then getting that typed on my computer with that little hardware dongle.
[34:59.710 --> 35:04.990]  Which, by the way, we can produce for around $20, $25 for shipping, you know, something like that.
[35:05.530 --> 35:10.010]  So, yeah, that's the, that's the demo of the Sylvia stuff.
[35:18.860 --> 35:20.920]  And, like all good projects, it has a website.
[35:21.220 --> 35:23.540]  I registered voxhub.io.
[35:25.440 --> 35:26.760]  And, you know, you can go there.
[35:26.840 --> 35:27.460]  There's a set of instructions.
[35:27.600 --> 35:29.020]  It says, get clone this, do that.
[35:29.420 --> 35:33.160]  And by default, it will just connect to the servers that are running under my desk.
[35:34.260 --> 35:36.600]  Because it's a bit more of a pain to actually download the thing.
[35:36.600 --> 35:39.320]  I have a tarball, which I should update this, but it's not there yet.
[35:39.500 --> 35:40.840]  So, it's coming.
[35:41.820 --> 35:42.960]  And I want this site.
[35:43.060 --> 35:45.000]  And there it is in big, in case you didn't see it before.
[35:45.260 --> 35:46.000]  Voxhub.io.
[35:47.080 --> 35:51.480]  I want this to be first to provide Sylvia servers that people can just use.
[35:51.580 --> 35:52.640]  They can be like, oh, I heard about this.
[35:52.700 --> 35:53.200]  I want to go there.
[35:53.280 --> 35:53.660]  I want to try it.
[35:54.260 --> 35:55.780]  I want to have a grammar database.
[35:55.880 --> 35:57.400]  I want to have a standard for the grammars.
[35:57.660 --> 36:00.200]  So that it's not like a NIA and there's a lot of fragmentation going on.
[36:00.680 --> 36:03.280]  And I also wanted to have a hardware configuration database.
[36:03.440 --> 36:05.500]  I mean, I tried a lot of different pieces of hardware.
[36:05.500 --> 36:08.720]  I know that this works fine.
[36:08.780 --> 36:10.020]  It's two gigahertz broad well.
[36:10.780 --> 36:16.520]  I know that a 4700HQ locked at 1.2 gigahertz is fine.
[36:16.620 --> 36:18.540]  But locked at 1.0 gigahertz is not quite fine.
[36:18.900 --> 36:24.360]  So, you know, I want to write all these things down so people can figure it out and buy the cheapest hardware that will do what they need.
[36:25.500 --> 36:27.100]  And of course, you can always run it on your computer.
[36:27.220 --> 36:31.040]  But when you're doing software development, things use CPU.
[36:31.200 --> 36:32.060]  You're compiling something.
[36:32.280 --> 36:33.560]  You know, you're searching for something.
[36:33.560 --> 36:34.460]  You're launching some programs.
[36:34.620 --> 36:36.140]  You're running a find in the background.
[36:36.480 --> 36:40.660]  And then you suddenly get lag spikes on your input.
[36:40.700 --> 36:42.500]  And it feels like your computer's freezing, right?
[36:42.520 --> 36:43.880]  You're pressing Ctrl-C and nothing's happening.
[36:44.060 --> 36:47.020]  Really, the recognitions just can't keep up.
[36:47.440 --> 36:48.740]  And it's just very frustrating.
[36:48.860 --> 36:52.020]  And so that's why I tend to want to offload it to another server if possible.
[36:53.900 --> 37:01.060]  Yeah, so in summary, when you can't type, grab your trusty dragon harness and try and code by voice.
[37:01.100 --> 37:02.160]  It actually is possible.
[37:02.360 --> 37:05.520]  And go and watch Travis Rudd's presentation on this.
[37:05.600 --> 37:06.540]  He goes into it in more detail.
[37:07.220 --> 37:13.640]  And if you find all of this interesting, I hope that Silverstone make it easy to experiment and build new ways of doing voice interaction with computers.
[37:13.640 --> 37:15.260]  Because the whole thing is open-source.
[37:15.500 --> 37:16.720]  You can go in and plug in your own grammars.
[37:17.120 --> 37:19.380]  Someone asked me earlier whether you can use things other than English.
[37:19.640 --> 37:20.500]  And I haven't tried.
[37:20.800 --> 37:23.500]  But there are a lot of free models online.
[37:23.560 --> 37:26.740]  This is the standard Cali acoustic model format.
[37:28.760 --> 37:30.620]  Yeah, so just some quick acknowledgments here.
[37:32.480 --> 37:34.700]  Tano LMI's streamer, amazing.
[37:35.280 --> 37:38.140]  Means that I didn't have to do the serialization aspects.
[37:39.740 --> 37:44.900]  Professor Homo Yun-Beiji, who did the speech class at Columbia, and told me that this was possible, that I should try it.
[37:46.220 --> 37:49.960]  And of course, the Cali speech recognition toolkit, which is really amazing.
[37:50.100 --> 37:54.260]  I wouldn't be surprised if it surpasses Dragon in quality at some point, because there's just so much active research going on there.
[37:54.560 --> 37:56.680]  Right now, it's, you know, they're just hacking on it.
[37:56.740 --> 37:57.260]  They're making it work.
[37:57.380 --> 37:58.800]  But eventually, it's going to be pretty great.
[37:59.320 --> 38:03.760]  And the Spark toolkit that I mentioned, I found it, and then I later discovered my undergrad advisor had written it.
[38:05.480 --> 38:06.520]  How curious is that?
[38:07.080 --> 38:11.760]  And of course, a bunch of other people that have maintained all of this crazy software over the years.
[38:11.760 --> 38:13.420]  It's just giant hacks on giant hacks.
[38:13.600 --> 38:17.840]  But, you know, Dragon releases a new version, and a couple days later, NatLink works on it.
[38:17.980 --> 38:19.520]  So, this is an amazing community.
[38:20.880 --> 38:21.800]  Yeah, so that's it.
[38:21.880 --> 38:22.240]  That's it.
[38:22.400 --> 38:22.880]  Thanks, everyone.
[38:35.380 --> 38:38.280]  So, it looks like we have seven minutes or so for questions.
[38:38.440 --> 38:41.320]  So, are there microphones for people?
[38:41.940 --> 38:42.860]  We just shut it out.
[38:42.960 --> 38:43.240]  Okay, go ahead.
[38:44.620 --> 38:45.060]  Yeah, it's you.
[38:45.080 --> 38:45.420]  Can I start?
[38:45.580 --> 38:45.740]  Okay.
[38:46.680 --> 38:47.900]  So, this is the grammar.
[38:48.060 --> 38:50.840]  Did you ever try to use this for generating stuff?
[38:50.840 --> 38:51.800]  Would that be possible?
[38:52.300 --> 38:52.940]  For generating stuff?
[38:53.020 --> 38:54.940]  Yeah, because the grammar, you can turn it around, right?
[38:55.200 --> 38:56.900]  I mean, that's kind of what I do with the n-grams.
[38:56.920 --> 38:57.500]  I walk through it.
[38:57.680 --> 38:59.560]  I generate all possible things you could possibly say.
[38:59.720 --> 39:02.140]  I deduplicate it in case there's cycles and stuff.
[39:03.560 --> 39:03.840]  Yeah.
[39:04.040 --> 39:07.100]  So, in some ways, you can blurt out, like, random things.
[39:07.100 --> 39:09.040]  Like, what the computer program would look like.
[39:09.100 --> 39:09.240]  Yeah.
[39:09.380 --> 39:18.440]  Because I played a lot with Markov chains, and then I tried them on program code, and had them output, arbitrary programs, and it's super funny, you should try it.
[39:18.720 --> 39:19.400]  Okay, I will.
[39:19.740 --> 39:20.040]  Yeah.
[39:20.540 --> 39:24.600]  So, a lot of you can do with these grammars, once you realize you can introspect on them.
[39:25.580 --> 39:26.280]  This is great.
[39:26.840 --> 39:27.400]  Sorry, go ahead.
[39:27.620 --> 39:33.420]  I was wondering if you could tell me a little bit more about the streaming and the transport that you used instead of the USB.
[39:33.740 --> 39:33.840]  Yeah.
[39:34.060 --> 39:44.680]  And if G-Streamer used speaks or something like that, and to what quality could you degrade before the Recognizer engine would have a tough time?
[39:44.820 --> 39:46.100]  Like, you have 16k audio.
[39:46.600 --> 39:51.020]  How bad can it get before it starts to not work as well?
[39:51.160 --> 39:53.180]  So, all the training is done on 16k.
[39:53.400 --> 39:57.160]  So, every input you feed it must be 16k, or the neural network has no idea what's going on.
[39:57.300 --> 39:58.620]  So, can you compress it on the way there?
[39:58.760 --> 39:59.220]  You can, of course.
[39:59.500 --> 40:01.020]  Can we expand it, and will that quality be...
[40:01.020 --> 40:02.580]  Yeah, you can do lossy compression.
[40:03.500 --> 40:08.540]  In general, voice recognition is very sensitive to little things in your speech that a human would not care about.
[40:08.720 --> 40:11.620]  Do you know what compression is used on the G-Streamer?
[40:11.660 --> 40:12.040]  I don't know.
[40:13.880 --> 40:14.840]  I get the audio.
[40:15.320 --> 40:16.420]  There is some compression going on.
[40:16.500 --> 40:17.460]  I think it's lossless compression.
[40:17.540 --> 40:22.520]  And I send the packets at 4k a second, and it's around like, it's very low bandwidth.
[40:22.640 --> 40:25.460]  I think it's 10 kilobytes, 5 kilobytes a second, something like that.
[40:25.960 --> 40:26.800]  Sorry, you in there?
[40:27.080 --> 40:27.420]  Yeah.
[40:27.420 --> 40:33.620]  My question is about acoustic modeling for programming languages specifically.
[40:33.620 --> 40:34.100]  Yeah.
[40:34.180 --> 40:46.060]  So, like, printf, we say printf, as opposed to like printf, or something, which is what, you know, an English acoustic model might think that's supposed to sound like.
[40:46.920 --> 40:48.880]  Is there any research in that respect?
[40:48.880 --> 40:53.080]  And like, someone's sitting there reading C code to the computer?
[40:55.220 --> 40:57.880]  No one's collected enough data on this to train something, that's for sure.
[40:58.600 --> 41:04.780]  There has been a bunch of research that was, they're trying to figure out how can you write a full voice, so I'm doing very low level stuff here, right?
[41:04.860 --> 41:06.900]  I'm dictating the words, I'm doing the variables myself.
[41:07.200 --> 41:16.660]  They were like, how can you write a grammar that's like, create new class foo that inherits this, create new member variable this, you know, and you can do that, but it's very limiting.
[41:16.840 --> 41:17.880]  It works in one language, right?
[41:17.980 --> 41:19.820]  This, this is, this is very generic.
[41:20.020 --> 41:26.560]  And in some ways, you're leveraging the, the keyboard input infrastructure that's already there, and you write macros on top of that.
[41:26.620 --> 41:35.840]  So I can, I can create a macro, and I actually have one, vim try that, which is vim save, switch window, make, enter, make run, or something like that, right?
[41:35.840 --> 41:39.900]  So it's just, it's pretty powerful that you can build on top of the, the keyboard stuff.
[41:40.260 --> 41:40.660]  Cool.
[41:41.560 --> 41:41.920]  Anyone else?
[41:42.140 --> 41:42.280]  Yeah?
[41:42.380 --> 41:44.200]  Just to respond to the question about compression.
[41:44.540 --> 41:44.620]  Yeah.
[41:44.740 --> 41:51.440]  If you insist on doing glossy compression, retrain your speech model, it'll take forever to retrain.
[41:51.620 --> 41:51.720]  Yeah.
[41:51.820 --> 41:56.000]  But you will get significantly better accuracy having trained on compressed data.
[41:56.260 --> 41:56.640]  I see.
[41:56.880 --> 41:57.900]  That's a good point, yeah.
[41:58.860 --> 41:59.640]  In the back there?
[42:00.260 --> 42:04.860]  So how do you deal with multiple accents, how many different accents can you actually recognize?
[42:04.860 --> 42:15.360]  So Dragon, which has a lot of funding, actually has like Indian, Canadian, believe it or not, British, American, they have like a lot of different accents.
[42:15.640 --> 42:22.920]  And the thing is, you just need to collect, in Dragon's case, probably 10 or 100,000 hours of speech in that, in that accent.
[42:23.040 --> 42:23.440]  Would it be its own model then?
[42:23.620 --> 42:23.940]  I'm sorry?
[42:24.200 --> 42:25.700]  Would it be its own model?
[42:25.780 --> 42:26.100]  Absolutely.
[42:26.420 --> 42:26.500]  Yeah.
[42:26.680 --> 42:28.400]  Its own 2 gigabyte model.
[42:30.600 --> 42:31.080]  Yes?
[42:31.320 --> 42:32.200]  So two things.
[42:32.200 --> 42:36.820]  First of all, Alex Roper says hi, we were live IRC-ing here and talked to him.
[42:36.940 --> 42:37.200]  Oh really?
[42:37.420 --> 42:37.700]  That's great.
[42:38.380 --> 42:40.400]  I'm really amused, especially by the name of the project.
[42:40.700 --> 42:41.060]  It's great.
[42:41.200 --> 42:41.380]  Great.
[42:42.200 --> 42:52.600]  The second thing I'm curious about is I noticed that you have to sort of speak for a while and then pause long enough for the system to be like, okay, that's one utterance, now I'm going to decode it and do something with it.
[42:52.660 --> 42:52.840]  Yeah.
[42:52.840 --> 42:54.020]  Could you get it to be more continuous?
[42:54.340 --> 42:54.780]  Yes, you could.
[42:55.020 --> 42:59.560]  So, Tanel LMI created a silence detection system and that's what Silvius is using.
[42:59.740 --> 43:01.740]  So it actually waits quite a while before it does the processing.
[43:02.000 --> 43:08.820]  We could definitely improve that because, I mean, when you're doing voice coding, you can kind of split it up anywhere and it will probably still work because your commands are quite short.
[43:10.600 --> 43:12.000]  And part of that is the CPU, right?
[43:12.060 --> 43:13.960]  The faster you run this on, the lower the latency is.
[43:13.980 --> 43:14.240]  Absolutely.
[43:14.660 --> 43:16.460]  If I'm running this on my server, much better.
[43:17.180 --> 43:17.220]  Yeah.
[43:17.780 --> 43:18.020]  Yes?
[43:18.020 --> 43:25.700]  If you have multiple streams, say two people, which parts of that would need to duplicate or would the entire thing be able to do?
[43:26.220 --> 43:27.020]  If you have...
[43:27.020 --> 43:27.540]  Can you repeat, sorry?
[43:27.880 --> 43:29.000]  Let's say you had two microphones.
[43:29.320 --> 43:30.600]  You and someone you were interviewing.
[43:30.880 --> 43:31.080]  Yes.
[43:31.300 --> 43:31.420]  Right?
[43:31.540 --> 43:32.360]  You wanted a transcript.
[43:32.700 --> 43:34.660]  So both of you would need to code.
[43:35.180 --> 43:39.760]  Would you have to run two complete separate instances of it or is there parts that could be shared between the two?
[43:40.080 --> 43:44.420]  Oh, so the biggest problem is you have a ton of in-resident RAM.
[43:44.560 --> 43:47.860]  You create this giant tri, multi-level tri, and that can absolutely be shared.
[43:47.860 --> 43:54.000]  So, and the CPU, you know, if I'm not speaking, the CPU is not being used.
[43:54.160 --> 44:00.100]  I worked out that you can have roughly three Silvius instances per two CPUs, even if they're being very heavily used.
[44:01.300 --> 44:04.600]  And right now the memory sharing is not happening, so I need two and a half gigs of RAM for each one.
[44:05.660 --> 44:09.700]  But yeah, I think if this was carefully designed to be a multi-user thing, it would be much better.
[44:09.900 --> 44:15.040]  I mean, you could even have two threads walking over the same trine memory and trying to stomp on each other with locks and whatnot.
[44:15.640 --> 44:18.320]  So, lots of room for improvement for sure.
[44:20.360 --> 44:21.140]  Any other questions?
[44:23.920 --> 44:24.360]  Yes?
[44:25.660 --> 44:36.120]  I mean, in terms of, like, using this as a developer aid, just in general, and not just for the people, you use it to solve your challenge of overcoming physical limitations.
[44:36.540 --> 44:52.220]  But what about, or would you see that there's a room for future work in things like tying it to a static analyzer or something like that to actually, you know, have another, something else part of the stream that can help you with, like, the grammar that you're writing or find errors in your syntax or something like that?
[44:52.220 --> 44:54.840]  You mean something that's listening that actually understands code?
[44:57.100 --> 45:05.740]  I mean, there's a very basic form of that, which is that most of these systems will create a log, and then you can search back through the log, and you can say, repeat that command, or that kind of thing.
[45:06.220 --> 45:13.220]  But, yeah, to me, a voice system, and I'm not there yet, but a voice system should be able to understand that you just created a variable.
[45:13.520 --> 45:16.680]  And then, later, you can refer back to that variable, and it will know what you're talking about.
[45:16.920 --> 45:19.500]  And it can be like, that's not the right type, you know, that kind of thing.
[45:19.880 --> 45:25.900]  So I think, yeah, similar to what an IDE can do for typing, yeah, you could definitely do that here.
[45:26.080 --> 45:28.860]  And the best thing is that you leverage everything that already exists, right?
[45:28.860 --> 45:30.340]  I can take this and use Eclipse with it.
[45:30.620 --> 45:32.700]  I'm not that crazy, but, yeah.
[45:33.480 --> 45:35.620]  Eclipse would use too much RAM, I would not have enough for this.
[45:37.200 --> 45:37.480]  Yeah.
[45:38.060 --> 45:39.080]  So yeah, a lot of possibilities.
[45:42.180 --> 45:42.680]  No one else?
[45:44.420 --> 45:44.780]  Okay.
[45:45.520 --> 45:46.300]  Oh, yeah, one more.
[45:46.420 --> 45:46.480]  What?
[45:46.480 --> 45:51.600]  Have you tried testing it on, like, a YouTube video for your, to, like, compare their closed captioning to your...
[45:52.260 --> 45:54.840]  Well, actually, this speech model is from Tedlium.
[45:55.120 --> 46:00.200]  You know, all the TED Talks ever done, it's trained on that speech, because someone actually went and typed the transcripts.
[46:00.540 --> 46:06.080]  So it's very good at recognizing an announcer's voice, and not very good at Delta Arch, Tango Arch, Delta Arch, Tango Arch.
[46:07.260 --> 46:08.100]  But that's okay.
[46:08.480 --> 46:09.140]  We'll get there.
[46:09.580 --> 46:09.740]  Yeah.
[46:10.440 --> 46:10.740]  All right.
[46:10.960 --> 46:11.620]  Thanks so much, everyone.
[46:11.620 --> 46:12.320]  Thanks for coming out.