[00:05.760 --> 00:07.080]  All right, hey guys.
[00:07.400 --> 00:18.520]  Now that I've been standing in front of you for like 10 minutes not saying anything, so first of all, thank you so much for coming to my talk on usability research.
[00:19.060 --> 00:29.980]  It thrills me that this is becoming something that's talked about more often in the context of, you know, crypto tools and surveillance circumstances.
[00:30.000 --> 00:31.460]  I mentioned tools and everything else.
[00:31.760 --> 00:35.720]  It's something that we need so, so badly.
[00:36.120 --> 01:05.100]  I mean, I think of the tools that I use on a daily basis like PGP, which I, even as someone who has a background in development, had to actually be taught how to use it because I was just sort of staring at my screen blankly and blinking and not really understanding how to go about getting myself set up with the key pair and getting an email sent out that's encrypted and that type of thing where
[01:05.100 --> 01:11.880]  it's just the average user is having a really tough time adapting to that type of technology.
[01:12.920 --> 01:35.020]  And especially with crypto tools and surveillance or convention tools in general, when users, when your average user is not only just an average user of technology, but also someone who might be in a situation where their lives are at stake, there's a lot of,
[01:35.040 --> 01:37.280]  there's a lot at stake for them.
[01:38.480 --> 01:51.120]  Being able to use these tools quickly and easily and be able to immediately open up a program, understand how to use it and teach themselves how to use it instead of having to be taught is super valuable.
[01:51.620 --> 01:56.240]  I'd also like to thank Edward Snowden for giving props to usability today in his talk.
[01:56.520 --> 01:57.380]  That was pretty sweet.
[01:57.520 --> 01:59.060]  I was totally geeking out about that.
[02:02.560 --> 02:07.860]  So that would be pretty useful to explain user experience first.
[02:08.200 --> 02:16.860]  So I've always sort of had this working definition of user experience as how a person feels when they're interacting with something.
[02:16.860 --> 02:17.940]  That's anything.
[02:18.180 --> 02:19.280]  It could be in the digital space.
[02:19.460 --> 02:20.580]  It could be something physical.
[02:21.980 --> 02:35.380]  And if that's user experience, and user experience design is sort of shaping the experience around data about those users to make that experience valuable to them and enjoyable and something that they want to use.
[02:36.800 --> 02:49.560]  So at the crux of that really is getting to know your users very, very well in order to serve up that technology that they're going to like to use and want to use and keep using.
[02:50.440 --> 03:01.600]  I've always sort of described usability research as like dating because it's basically the same thing to break down dating in a way that is totally robotic.
[03:02.540 --> 03:06.140]  You know, you meet someone that you like and then you go out on a couple of dates with them.
[03:06.300 --> 03:07.220]  You're asking questions.
[03:08.020 --> 03:11.280]  You know, you're gathering data to make sure they're a good match.
[03:12.060 --> 03:14.740]  That's basically the same thing that you're doing with usability research.
[03:17.140 --> 03:19.460]  So, except hopefully less awkward.
[03:21.920 --> 03:25.540]  So, you know, we talk about Alice and Bob a lot.
[03:26.620 --> 03:34.400]  And we use them as sort of a paradigm to explain how crypto tools work to the average user.
[03:34.620 --> 03:40.140]  But we also have to remember that Alice and Bob themselves have no clue what the f*ck's going on.
[03:41.860 --> 03:49.720]  They're not super clear on how to encrypt things or how to, you know, set up a proxy or whatever.
[03:49.720 --> 03:51.480]  They have no idea what's happening.
[03:51.700 --> 03:57.020]  They just know that they would need to talk to each other and they don't want Eve or Mallory in the situation either.
[03:57.380 --> 04:06.940]  So, but the most important thing with these types of tools is that Bob and Alice can also be a source and a journalist.
[04:08.520 --> 04:16.860]  Where the source could get into serious, serious trouble if they are, if their identity is revealed.
[04:17.720 --> 04:27.120]  If they want to do it in a way, if they want to pass on documents, a blow whistle in a way that's secure and keeps their identity safe.
[04:27.320 --> 04:34.240]  So that they know that they're, the information that they want to get out to the public is just getting out there without them being in danger.
[04:34.900 --> 04:38.740]  And the journalist, on the other hand, needs to protect that source.
[04:39.120 --> 04:45.320]  And we need to realize that, you know, these average users have, might have a lot at stake here.
[04:46.540 --> 04:54.780]  You know, it's the same thing with activists in, you know, in countries that are in crisis.
[04:54.780 --> 05:08.640]  If there's, you know, political uprisings, that type of thing where they want to mobilize and others can help, but they really don't need any government intervention reading their messages between each other.
[05:09.300 --> 05:22.320]  Again, this is why we need to make tools that are not only effective and developed really, really well, but also have a front end that people understand how to use immediately.
[05:24.620 --> 05:37.040]  So that sort of, this is going to probably be preaching to the choir to a lot of people here, but I thought it would be important to bring up the difference between quantitative and qualitative research, right?
[05:37.280 --> 05:42.360]  So if you have quantitative research, it's obviously incredibly valuable.
[05:42.360 --> 05:43.560]  You get lots of data.
[05:45.220 --> 05:57.820]  It's a really great way to measure overall behaviors to get a baseline of your users, to sort of understand them at their very core, because you have such a large sampling.
[05:58.940 --> 06:05.000]  The downsides, though, of quantitative research is that it can be a logistical nightmare getting that stuff set up.
[06:05.000 --> 06:10.800]  I mean, I've had issues with it before where it's just, it's awful and it's expensive.
[06:12.120 --> 06:14.340]  The questions are very rigid.
[06:14.680 --> 06:15.760]  And hey, hey guys.
[06:17.560 --> 06:23.500]  The questions tend to be very rigid because you're not really getting written responses in most cases.
[06:23.500 --> 06:34.620]  You just want the cold, hard data of yes or no questions or multiple choices, and you can't really do much else with it, but it obviously is valuable for a handful of reasons.
[06:34.620 --> 06:41.840]  It also sort of kind of makes you a little bit more like the NSA, because it's sort of just a blanket, lots and lots of data, we'll do something with it later.
[06:44.560 --> 06:57.120]  But obviously, quantitative is really, really, really important at the beginning of a project, but qualitative, I feel like, especially in this context, oftentimes gets ignored.
[06:58.140 --> 07:01.960]  And the great thing about qualitative research is that it's detailed.
[07:01.960 --> 07:17.720]  It's very, very flexible, meaning that you can have very broad focus and use it as a way to sort of find the next point of your project, you know, steer it in a way, help it steer you in a way that you weren't expecting to go.
[07:18.800 --> 07:38.540]  It's easy to arrange, and it's overall just you get this really great, juicy, meaty data out of it, because you're talking to actual users one-on-one in a way that's very personal, and you can really delve as deep into it as you see fit.
[07:38.540 --> 07:46.460]  And that's why I sort of am sitting here preaching the benefits of qualitative research.
[07:46.640 --> 08:04.540]  And that's why I think this guerrilla-style user interviews, yay, are the most valuable, are the most valuable form of qualitative research, because it's so easy to do, and anyone can do it who's involved in a project.
[08:05.040 --> 08:11.840]  They're absolutely magical, just like the unicorn that rode across the screen on a motorcycle.
[08:13.360 --> 08:18.920]  This is also, unfortunately, about as rare as that for crypto tools.
[08:19.200 --> 08:30.520]  So I'm hoping that sort of giving you this toolkit to use will help sort of push usability even more in this context.
[08:31.100 --> 08:46.100]  So I know that this can be really overwhelming, because you're going out and you're trying to find people that are willing to, you know, sit down with you for a while and look at your app, or look at, you know, your software, and play with it, and you don't know what to ask,
[08:46.160 --> 08:51.820]  and you don't know how long the interview should be, and it's very overwhelming.
[08:51.960 --> 09:01.380]  The first couple of times that I went out on either interviews, I was a mess, and I was asking stupid questions, and I was shaking, and it was awful.
[09:02.600 --> 09:15.660]  But if you make yourself do it once or twice, you get in the swing of things, and you can get easily, you can get five or six really great user interviews done in, like, an average workday.
[09:16.800 --> 09:27.900]  And if you do that, you know, twice a week, just are always going out and asking users what they think of your product, you're going to get really, really valuable feedback on it.
[09:28.840 --> 09:35.060]  But there are a lot of things that you have to consider before you're going into these situations where you're talking directly to your users.
[09:36.240 --> 09:40.140]  But thankfully, it's not as daunting as it may look.
[09:40.700 --> 09:47.060]  So, first, to prepare yourself for this, you need to do two very important things.
[09:47.300 --> 09:57.340]  You need to establish your priorities and determine what you actually need for your project, or for actually going out on the user interview.
[09:57.340 --> 10:19.660]  So, for establishing priorities, the reason you want to do that instead of just going out and winging it, which is sometimes actually fine, but most times you want to establish priorities in order to get really good and relevant data to a very specific problem that you're trying to solve.
[10:21.060 --> 10:32.680]  And there's also, I feel like there's this point where data can reach a critical mass, and you don't want, like, a data black hole where it's just sucking everyone in and everything's terrible.
[10:32.780 --> 10:37.060]  I hope there's no, like, physicists in here, because I don't know if that's analogous at all.
[10:39.600 --> 10:45.420]  But the thing is, when it reaches that critical mass and you have so much data, it can block progress.
[10:46.720 --> 10:58.860]  And if you take the time to prioritize before you go out and research, it really does help to sort of pare it down and make sure that you get the data that's the most relevant to you at that time.
[11:00.880 --> 11:03.840]  So, there are a couple of ways of going about doing this.
[11:04.240 --> 11:12.280]  I know that sort of establishing priorities can lead to sort of political battles between teams and that type of thing.
[11:12.280 --> 11:22.700]  But there's one sort of tool that I use a lot that's called KJ Analysis that I encourage everyone to look up.
[11:22.880 --> 11:24.360]  It's super valuable.
[11:24.600 --> 11:25.480]  It's very helpful.
[11:26.340 --> 11:35.220]  Everyone involved in the project, whether it be a stakeholder or a developer or a designer, anyone in the project can be involved in the KJ Analysis process.
[11:35.220 --> 11:41.560]  Basically, what happens there is that you have... you make sure that you can have a focus question.
[11:42.440 --> 11:45.040]  It can be as broad as what are users' goals.
[11:47.260 --> 11:57.980]  After you have that focus question, you have everyone who is involved in a room write down their opinions on that matter, one per post-it note.
[11:58.260 --> 12:00.380]  Write down post-it notes, one per post-it note.
[12:00.560 --> 12:04.980]  Stick it up on a whiteboard and then organize it.
[12:04.980 --> 12:08.100]  Get into categories of like answers.
[12:09.360 --> 12:12.500]  The category that has the most, that's your priority.
[12:12.940 --> 12:15.900]  And it's a totally democratic process and nobody's arguing.
[12:16.240 --> 12:18.120]  It's like clean cut right there.
[12:18.320 --> 12:19.680]  That's what we have to focus on.
[12:19.880 --> 12:28.080]  So I've always found that to be a nice, quick, easy way to get through that type of process and just keep this thing moving.
[12:30.340 --> 12:40.240]  So that being said, determining what's needed is obviously for actually going out into the field is really important.
[12:41.340 --> 12:50.880]  And I want to stress that when you're going out, you don't need to have a released version of whatever your software might be.
[12:50.880 --> 12:54.460]  You don't need something that's fully functional.
[12:55.080 --> 13:00.860]  It's absolutely 100% fine to go out with a prototype of any fidelity.
[13:01.740 --> 13:14.580]  I've gone out with paper prototypes where I'm literally cutting out things that look like buttons in an app and slapping it down the table and moving it around as the user actually taps on things.
[13:14.580 --> 13:18.900]  And that is just as valuable as taking out something that's already been released.
[13:23.360 --> 13:29.140]  Now getting into the good stuff is writing screening questions.
[13:29.320 --> 13:32.380]  Screening questions are valuable for a couple of reasons.
[13:33.000 --> 13:40.860]  So first of all, good screening questions will help you determine if the user that you're going up to is sort of the guerrilla style approach for you.
[13:40.860 --> 13:45.100]  You just sort of ambush somebody and say, hey, do you want to use my app for a minute?
[13:46.440 --> 13:52.000]  And you want to be able to screen them first to make sure that they're actually someone that's worth testing the product.
[13:52.760 --> 14:03.420]  Nothing against them, but if they're, you know, a technologist, maybe you move on to someone who's not as familiar with, you know, things much past the Internet.
[14:04.340 --> 14:07.060]  And use them instead as sort of your litmus.
[14:08.140 --> 14:13.960]  Then again, sometimes it's also valuable to have somebody who's an expert in something test it.
[14:14.140 --> 14:15.160]  But that's up to you.
[14:15.360 --> 14:19.880]  It totally depends on your priorities and what you need for the project.
[14:22.120 --> 14:26.840]  I've also found that they're really great for serving as a record of demographics.
[14:27.830 --> 14:32.100]  So as you're screening and you're saying, oh, what do you do for a living and blah, blah, blah.
[14:32.240 --> 14:37.040]  Then you can also ask for age ranges and, you know, gender, that type of thing.
[14:37.160 --> 14:42.520]  So you get a little bit more of a broader picture of what that user looks like.
[14:43.660 --> 14:53.440]  And that can help in the end when you're doing your analysis to see sort of what users had difficulties with some things and didn't with others.
[14:53.680 --> 14:59.220]  And you can sort of compare to see where that issue might be coming from.
[15:01.060 --> 15:06.980]  So you also want to be asking and writing really broad and open questions before you go out.
[15:07.780 --> 15:12.920]  Open questions being things like, what are your main concerns with regards to privacy?
[15:13.180 --> 15:16.080]  Or what steps have you taken to protect your identity online?
[15:16.860 --> 15:19.680]  Instead of saying, hey, do you use PGP?
[15:19.680 --> 15:20.980]  Like, no.
[15:21.460 --> 15:22.040]  Okay.
[15:22.260 --> 15:23.380]  That was a nice answer.
[15:24.040 --> 15:27.820]  And then it gets awkward and there's awkward silences and everyone's uncomfortable.
[15:28.180 --> 15:39.660]  So asking open questions sort of allows that participant to go on and on and you get that really great data and you're writing down and everything that they're saying.
[15:43.600 --> 15:56.780]  And specifically in this context of cryptography tools and surveillance or convention technology, threat modeling is a really great way to sort of start framing these questions.
[15:57.400 --> 16:01.500]  And that way you also get to have sort of that second level of screening.
[16:01.860 --> 16:06.420]  You understand more if they're actually someone who's going to be using a product like yours.
[16:08.220 --> 16:12.540]  And also just a little side note that asking leading questions sucks.
[16:13.320 --> 16:15.820]  You're going to sabotage your research if you do that.
[16:16.080 --> 16:24.500]  So it's basically putting words into your users' mouths instead of saying something like, do you have trouble using PGP?
[16:24.780 --> 16:29.520]  It's asking something more like, tell me about your experiences with PGP.
[16:31.360 --> 16:36.020]  So that, again, you're getting... you're opening the door for...
[16:36.020 --> 16:37.260]  Oh, that's nice.
[16:38.960 --> 16:39.740]  Hold, please.
[16:39.920 --> 16:40.360]  Hold, please.
[16:40.600 --> 16:41.240]  There we go.
[16:41.700 --> 16:56.780]  So you're opening the door for a more in-depth answer from them rather than just yes or no or sort of they feel like they should be saying yes because you're a tech person and they don't want to look dumb, that type of thing.
[16:56.780 --> 17:03.100]  So leading questions really are terrible and should be avoided at all costs.
[17:03.300 --> 17:04.740]  They'll also skew your data.
[17:05.260 --> 17:09.920]  It'll totally f*ck up your data set and that's not good for anyone.
[17:10.560 --> 17:14.660]  So you also want to be writing task-based prompts.
[17:14.660 --> 17:18.380]  This I find to be very important.
[17:18.680 --> 17:19.780]  So when you're...
[17:19.780 --> 17:26.730]  Let's say you have an iOS app and you're going in with your own iPhone or a test iPhone.
[17:27.800 --> 17:41.460]  I find that asking these task-based prompts to the participants gives you a chance to step back and really observe and see where they're fumbling and that type of thing.
[17:42.320 --> 17:49.600]  And an example of this would be, you know, opening, let's say, LinFone and saying make a call.
[17:50.620 --> 17:55.490]  Which I found in the past couple of days doing research here that that's not a thing.
[17:55.660 --> 17:56.710]  No one knows how to do it.
[17:58.300 --> 18:11.440]  So it's really great to have that moment to see their body language and that type of thing and see where their finger is sort of going across the screen trying to find things.
[18:11.440 --> 18:16.510]  And it's really great takeaways from that type of experience.
[18:19.920 --> 18:21.970]  Also practice before you go out.
[18:22.200 --> 18:24.420]  It helps sort of cut down the jitters, obviously.
[18:25.010 --> 18:30.730]  And you can rewrite your prompts and rewrite your questions and get input from people.
[18:30.920 --> 18:35.640]  And that way you know that you have something that's really solid before you go and actually try to get this data.
[18:37.580 --> 18:39.700]  So then you have to find people.
[18:40.160 --> 18:48.600]  I have found that coffee shops are absolute gold mines for people who want to look like they're working but actually are procrastinating and trying to find more ways to procrastinate.
[18:49.320 --> 18:52.230]  So that being said, go to coffee shops.
[18:52.230 --> 18:52.940]  It's great.
[18:53.540 --> 19:00.140]  Everyone, most people want to talk to you if you approach them in a way that's gentle and not startling.
[19:01.230 --> 19:18.940]  I've also found that if you are going to a coffee shop or any sort of environment like that, to ask to speak to a manager first and just say, hey, I'm doing something that has nothing to do with a competing coffee shop and here are my questions and are you cool with this.
[19:19.680 --> 19:22.860]  Nine out of ten times, if not more, they'll say yes.
[19:23.210 --> 19:29.580]  And then you have pretty much free reign over everyone in the coffee shop who looks like they are distracted.
[19:32.680 --> 19:51.800]  And mind your dang manners when you're talking to them because here's the thing with a lot of feasibility research is that it's so easy to get frustrated because this thing that you're making has become a part of you and somebody is looking at it and they can't use it and it's frustrating to watch it happen.
[19:52.600 --> 20:01.040]  But practice self-restraint and just make sure that you're not, you know, doing things like telling them where to tap or click.
[20:02.160 --> 20:04.840]  You know, correcting them if they screw something up.
[20:04.990 --> 20:07.080]  Just let them go through it.
[20:07.180 --> 20:08.040]  Let them do their thing.
[20:09.040 --> 20:18.060]  Because that's, again, how you're going to get that really great data of actually looking at the person who used the app and seeing for yourself where they're fumbling.
[20:19.210 --> 20:26.800]  This is, again, this is going back to that qual versus quantitative research.
[20:27.080 --> 20:36.300]  The quantitative stuff, you really can't get that personal experience with that and you can with, especially with user interviews because you're sitting right in front of them.
[20:36.300 --> 20:48.750]  So, as long as you sort of practice good usability research etiquette, you're going to get a lot of really wonderful answers and great data from this type of thing.
[20:51.160 --> 21:04.440]  So, do feel free, though, when you're... I found a lot of times that when people are getting frustrated, they tend to comment on things like the visual design of the app or whatever you're showing them.
[21:05.830 --> 21:12.860]  And then they start to give you suggestions and then they turn into the expert and you're like, that's nice, thank you.
[21:14.200 --> 21:22.820]  So, then you sort of... that is something where you do have to find a nice way to sort of reel things back in and get the conversation back in the right direction.
[21:22.820 --> 21:24.970]  It will happen.
[21:25.200 --> 21:31.730]  It always does where the user is like, you know, I think if this button were blue, I would tap it.
[21:32.640 --> 21:35.040]  Thank you so much for that information.
[21:35.620 --> 21:37.540]  And how about this next step?
[21:38.940 --> 21:49.880]  In that same vein, too, I also find it really great when you ask the user what they're expecting to see in the next step of whatever process they're in.
[21:49.880 --> 21:58.600]  Because then you're understanding what their expectations are and then can compare to what is actually in the app.
[22:01.260 --> 22:08.120]  Seeing those two things side by side, you're going to be able to bridge the gap as you get more and more responses.
[22:10.340 --> 22:22.400]  So, the most important part is going off of all of that is to just sit back, shut up for a little bit, take notes and just watch them as they're going through things.
[22:22.400 --> 22:26.920]  It is really important to start to read their body language.
[22:27.260 --> 22:36.120]  I can't tell you how valuable that is because the more you pay attention to that, the better you'll get at asking follow-up questions.
[22:36.120 --> 22:46.100]  Once you're able to sort of pinpoint when they're uncomfortable, when there's a level of discomfort, when there's surprise or frustration, you can sort of go off on that.
[22:47.360 --> 22:51.200]  It's as simple as saying, you know, oh, I saw your shoulders go up.
[22:51.320 --> 22:53.440]  Is this stressing you out?
[22:54.100 --> 22:54.760]  It is.
[22:54.920 --> 22:56.340]  Okay, tell me more about that.
[22:56.540 --> 23:04.900]  And then you get that whole other level of a response that you wouldn't have gotten otherwise from, you know, a written survey.
[23:06.980 --> 23:08.460]  So, then analyzing.
[23:08.700 --> 23:21.040]  This is where things can get a little bit muddy because qualitative research, obviously, they're hard numbers to go off of and it's pretty cut and dry of how to analyze it.
[23:21.200 --> 23:30.680]  I go about analyzing this user interview research in a handful of different ways.
[23:30.680 --> 23:38.100]  And you have to take a lot of what they're saying with a grain of salt, obviously, but sort of like what we were talking about before with the KJ analysis.
[23:38.500 --> 23:58.280]  If you're finding that a certain phrase or, you know, a certain discomfort is happening at the exact same point in a process, as you're gathering more and more of that and you see that, you know, 15 people said that they didn't understand how to get from step A to step B,
[23:58.280 --> 24:06.100]  then that's when you can really go off of that and write something up that is going to be valuable for the project.
[24:07.800 --> 24:15.300]  So, the ways that I go about this are I tend to do, even though they're a little passe now, I guess, word clouds.
[24:15.300 --> 24:18.680]  I feel are a great way to visualize this.
[24:19.360 --> 24:28.620]  If something like, you know, weird comes up 22 times, it's going to be giant as opposed to other words in that word cloud.
[24:29.560 --> 24:38.820]  And that's, again, how you can pinpoint where, you know, certain, like, if your app is going in the right direction or the wrong direction or if it's somewhat neutral.
[24:39.700 --> 25:00.340]  I always do a research write up where it's just sort of long form just going on and typing about exactly, you know, what the objectives of the research were, what problem we're trying to solve and, you know, what next steps might be in order to solve that problem based on the research that you've gotten directly from your users.
[25:02.040 --> 25:07.880]  And then, of course, I don't... I want to just try to stay away from buzzwords, but I'm going to use one right now anyways.
[25:08.920 --> 25:20.080]  This allows you to iterate on your designs and everything else in the app in a really great way because you're going off of things that have come straight from the user's mouth.
[25:21.280 --> 25:33.420]  And there's oftentimes been situations where I have been out in the field and gotten a lot of feedback on one specific thing and just gone, screw it.
[25:33.480 --> 25:34.580]  I have my laptop with me.
[25:34.680 --> 25:39.720]  Let me just comp this up and see if I can, you know, get a better response out of this.
[25:40.360 --> 25:44.800]  So doing that type of iteration while you're on the field, too, is wonderful.
[25:44.800 --> 25:51.860]  And if that's something you're comfortable with, it can be on a napkin, it can be on your computer, however you want to do it.
[25:53.000 --> 26:01.680]  If that's something you're comfortable with, I would highly recommend doing it because then you're getting live, you know, real-time feedback right there on the spot.
[26:05.300 --> 26:13.260]  That being said, iterating on the design after this process is also obviously a natural thing to do.
[26:13.260 --> 26:26.680]  It makes sense in the process here, once you've analyzed the research, to go back, you know, work on it for another couple of days and then go out again and hammer down on that for another couple of sessions.
[26:27.600 --> 26:39.440]  That way you're really getting just the best data possible and you're constantly testing things that are being developed and redeveloped and getting, you know, going through the cycle.
[26:39.440 --> 26:42.940]  And this is why this is something that anyone can do.
[26:43.280 --> 26:46.740]  You don't have to have, like, amazing visual design skills.
[26:46.840 --> 26:55.080]  You don't have to have any design background to really be doing this type of thing, to be going out with the tool that you have developed and asking users to use it.
[26:58.180 --> 27:12.920]  That's why this is such a good tool to have in your arsenal when you're developing crypto tools, to go out and ask people constantly how it's going so that you can have that data to iterate on the design and get the best possible product out there.
[27:14.860 --> 27:17.060]  So, huzzah, go forth and research.
[27:17.840 --> 27:22.100]  I, there's a couple of things here that I forgot to mention.
[27:22.540 --> 27:34.560]  First of all, if you are going, if you decide to do this type of research at a coffee shop, I've also, there's been some questions about whether or not, you know, incentivizing people is ethical or not.
[27:34.560 --> 27:46.920]  But I think it's not an issue if you are to go into a Starbucks and say, you know, go up to the counter, buy three, five dollar gift cards and then hand them to your users.
[27:47.100 --> 27:59.520]  It's like, hey, if you answer my 15 minute, you know, survey question, then you'll be able to...
[27:59.520 --> 28:01.620]  Sorry, let me rewind there.
[28:02.900 --> 28:10.620]  You're making the coffee shop happy by letting people go and buy more coffee and you're sitting there and they're not worried about you.
[28:10.920 --> 28:19.120]  They know what's going on and then the users are even more likely to say yes when they know that they're getting free stuff out of it.
[28:19.300 --> 28:21.720]  So, I find it to be very valuable.
[28:22.020 --> 28:35.340]  I don't see an ethical, any ethical weirdness in there unless you're specifically developing something for a competing coffee shop, which I don't think there's going to be any sort of crypto tools for caribou coffee coming out anytime soon.
[28:35.620 --> 28:37.320]  So, I think you're fine there.
[28:39.080 --> 28:49.960]  And even still, even if you decide to incentivize like that, it's still such a cheaper process than going and getting a third party usability research agency to come do it for you.
[28:49.960 --> 28:55.040]  Or hiring someone who is specific to that if you don't have the budget.
[28:55.260 --> 29:06.840]  Because I know often times on projects that I've worked on where it's with nonprofits, you get a grant, the grant money is separated into different categories and design is never ever one of those things.
[29:06.840 --> 29:16.960]  So, these are sort of ways you can get around that and sort of take it on yourself and make sure that you're getting the best data possible.
[29:18.100 --> 29:23.480]  So, with that, I figured I'd leave about 15 minutes for any questions.
[29:23.520 --> 29:30.100]  If you have anything that you're working on, you're wondering how to test it, I'm happy to answer those questions.
[29:44.690 --> 29:45.970]  That's a great question.
[29:46.530 --> 29:49.710]  So, I tend... if you're going out...
[29:50.470 --> 29:51.450]  Oh, sorry.
[29:51.650 --> 30:00.890]  The question was, how many people can you... can an average researcher go out and ask questions to?
[30:03.730 --> 30:08.370]  I tend to stick to like five or six in like an average workday.
[30:08.370 --> 30:10.270]  So, in like an eight-hour period.
[30:10.390 --> 30:19.690]  It gives you time to research, sort of go back, do any design tweaks that you feel like doing or doing any sort of like other low-fidelity prototyping.
[30:20.730 --> 30:22.470]  And then going on to your next person.
[30:22.470 --> 30:24.870]  It gives you sort of that gap in between then.
[30:26.510 --> 30:28.050]  It also gives you time to...
[30:28.050 --> 30:36.010]  I always do like a very quick two-minute recording right into my phone of like, here's what happened in that last session.
[30:36.250 --> 30:37.470]  And here's what I learned, blah, blah, blah.
[30:37.750 --> 30:42.510]  That way you don't have to go through all of your notes if you don't have a whole lot of time for it.
[30:42.610 --> 30:43.490]  It's great that way.
[30:43.510 --> 30:45.330]  But I stick to five or six.
[30:46.430 --> 30:49.590]  Even if you're doing two or three, that's great.
[30:49.590 --> 30:58.410]  I mean, this can be done literally any time that anyone's in public and looks like they don't care if you come up and talk to them.
[30:58.930 --> 31:02.450]  So, as much as you can do is great.
[31:02.450 --> 31:08.590]  But if you're getting tired, it's not worth it because then you're just going to start slipping and the data's going to suck and everything else.
[31:08.590 --> 31:09.250]  So, yeah.
[31:10.590 --> 31:12.070]  How do you pick who to approach?
[31:12.510 --> 31:13.150]  What was that?
[31:13.270 --> 31:14.790]  How do you pick who to approach?
[31:15.330 --> 31:17.030]  The question is, how do you pick who to approach?
[31:20.730 --> 31:21.990]  So, there's a couple of things.
[31:22.330 --> 31:23.730]  One, read their body language.
[31:24.690 --> 31:30.670]  If they see you and then turn and put their headphones on and look like they're working, it's like, okay, I get it.
[31:31.250 --> 31:36.690]  Sort of the same phenomenon of walking into a retail store and being like, don't talk to me.
[31:36.770 --> 31:37.650]  I just want to look for something.
[31:39.370 --> 31:40.310]  Don't go for that.
[31:40.850 --> 31:55.410]  I tend to go for people who are doing things like, they don't look like they're typing a whole lot if they're sitting down with their laptop or they're just sort of staring into the middle distance, which there are a lot of people who will do that.
[31:56.870 --> 32:03.690]  So, that type of person, you really have to be there and read their body language and everything.
[32:04.090 --> 32:08.830]  It's pretty clear when somebody doesn't want to talk to you and then you just don't bother.
[32:09.990 --> 32:20.310]  The good news is, if somebody doesn't want to talk to you and they look approachable and you approach them, they'll tell you, no, I don't feel like talking.
[32:20.490 --> 32:21.630]  And then you just move along.
[32:22.390 --> 32:26.870]  There's been instances where I just have gone up to anyone.
[32:27.030 --> 32:27.950]  I don't really care.
[32:28.090 --> 32:29.090]  I'm on a time crunch.
[32:29.270 --> 32:30.170]  Do you want to talk to me?
[32:30.330 --> 32:30.610]  No?
[32:30.850 --> 32:31.250]  Okay, whatever.
[32:31.650 --> 32:32.250]  You just move on.
[32:33.230 --> 32:33.490]  Yeah.
[32:33.710 --> 32:35.530]  Not very scientific, but that's how I do it.
[32:36.950 --> 32:41.210]  Could you list some of the most common applications in your toolbox that you use?
[32:41.490 --> 32:47.470]  And the second part, do you have any advice or experience with working with younger demographics?
[32:48.930 --> 32:49.410]  Hmm.
[32:50.030 --> 32:50.550]  That's it.
[32:50.850 --> 32:54.650]  So, what was the first part of your question?
[32:54.830 --> 32:55.990]  I got two into the second part.
[32:57.290 --> 33:07.470]  So, most common applications I use in my toolbox for researching, a pen and a notepad and perhaps I'll bring my laptop if I feel like it.
[33:07.470 --> 33:10.090]  This is very cheap.
[33:10.290 --> 33:11.630]  You don't need to buy a single thing.
[33:11.730 --> 33:13.850]  If you don't want to, you can go get a napkin.
[33:16.610 --> 33:18.550]  It's really anything that you would like to use.
[33:18.790 --> 33:23.050]  That being said, there are a couple of apps out there that make life a little bit easier.
[33:24.370 --> 33:26.410]  There's a tool called UX Pin.
[33:27.150 --> 33:38.130]  They make these tool kits that are really wonderful that have different Android phones and iOS devices stenciled out onto pages.
[33:38.650 --> 33:43.030]  And then they have these little sort of post-it note sticky things that are all...
[33:43.030 --> 33:47.990]  It's the whole GUI, basically, the whole standard GUI that you can just stick on there.
[33:47.990 --> 33:51.850]  And that way you don't even have to lift up the pencil, which is wonderful.
[33:52.690 --> 33:56.650]  You can just sort of stick things on a page and say, Oh, you didn't like the button there?
[33:56.790 --> 33:57.790]  How about if I move it here?
[33:57.930 --> 33:58.910]  Does that make more sense?
[34:00.070 --> 34:01.270]  That's a really wonderful tool.
[34:01.450 --> 34:04.650]  It's a little bit pricey, but I've found that it's completely worth it.
[34:04.850 --> 34:09.990]  They also, along with that kit, have sort of a questionnaire that's...
[34:10.650 --> 34:14.310]  All the things that you need to know before you're going into your project.
[34:14.310 --> 34:22.490]  It's things like user personas, like, what's your target audience, that type of thing that really helps you think through it before you go out into the field.
[34:22.850 --> 34:23.850]  That's a great tool.
[34:24.750 --> 34:34.710]  There is a handful of wonderful free apps like Pop, which is prototype on paper, where you sketch anything on a piece of paper.
[34:34.950 --> 34:44.130]  You open up your app, you take a picture of it, and then you can draw targets so that it's tappable and it feels more substantial and more realistic.
[34:45.330 --> 34:47.350]  Those are the two that I tend to gravitate to.
[34:47.590 --> 34:54.130]  There's a ton out there, but honestly, the best tools in your arsenal are going to be, you know, a pad of paper and a pen.
[35:14.740 --> 35:22.060]  So, you're asking if there's any other things like KGA analysis to sort of drive the research forward?
[35:22.060 --> 35:26.240]  In the specifics of trying to identify clusters and properties.
[35:27.200 --> 35:28.620]  Oh, okay, I see what you're saying.
[35:30.280 --> 35:34.260]  So, there's a couple of different techniques for that.
[35:35.540 --> 35:52.880]  Card sorting is something that's sort of in the more traditional information architecture toolbox, where, similar to KGA analysis, you're writing things on note cards, you know, user personas, that type of thing, and then categorizing it from there.
[35:53.160 --> 36:11.900]  So, basically, let's say you're trying to figure out what the average threat model is for your user, getting together with your team and sort of going through that exercise yourself, writing these things down, and then this is the sorting part, the card sort,
[36:11.900 --> 36:18.720]  where you're putting it all down on the table, and everyone has input on what's going into which category.
[36:19.060 --> 36:33.680]  That's usually used for more things, more like layouts, and like what the hierarchy of the page could be, but I've used it for things where we're trying to determine exactly who our user is, for example.
[36:34.580 --> 36:35.480]  that type of thing.
[36:37.620 --> 36:43.660]  Okay, so you mentioned that you do a lot of your sampling at, like, coffee shops and stuff like that.
[36:45.360 --> 37:03.140]  But when you are looking at people in coffee shops, and you said you target people who are, like, working, or whatever in there, how do you avoid, like, a locational-based sample biasing with people who are already familiar with working, obviously, working on their laptops or whatever,
[37:03.240 --> 37:09.060]  versus someone who's, say, more blue-collar and doesn't have quite as much experience with a computer or working like that?
[37:09.620 --> 37:11.040]  That's a great point.
[37:11.240 --> 37:15.340]  So the question is, how do you avoid sort of this locational bias?
[37:15.340 --> 37:23.000]  If you need, if you really need to be asking users who are more blue-collar and aren't going to be on their laptop in a coffee shop, where do you go?
[37:24.300 --> 37:25.600]  That is an excellent question.
[37:25.600 --> 37:32.060]  I have, in the past, this also ties into your second question, which I forgot to answer, about other age groups.
[37:32.340 --> 37:38.700]  So if you're testing children or teenagers, that type of thing, it's a little less gorilla at that point.
[37:38.700 --> 37:41.480]  You have to sort of do a little bit more planning.
[37:41.800 --> 37:45.880]  That being said, it's not anything that's too strenuous.
[37:46.200 --> 37:56.200]  I've done things in the case of children, like called up schools in a given area, which is, you know, getting more to your point about locational bias.
[37:56.200 --> 37:59.720]  You can, you have more of a wide range of options in that case.
[38:00.180 --> 38:04.160]  And setting something up that's a little bit more structured.
[38:06.780 --> 38:15.780]  If you're trying to go for more of a, that sort of gorilla thing where you're trying not to plan too much and just need to get out.
[38:15.780 --> 38:32.700]  So finding a place in a location where you know that whatever demographic you're going for is going to be, there's always going to be some sort of gathering place anywhere you go in any community.
[38:33.680 --> 38:46.740]  I've actually, for something that I was testing a couple of years back, set up something with a church and did sort of like a round table discussion type of thing.
[38:46.840 --> 38:50.400]  It ended up being more like a focus group, but still valuable.
[38:51.480 --> 38:52.820]  I've gotten into bars.
[38:54.520 --> 39:01.460]  People are great in bars because they've had something to drink and they're a little bit, they're like more willing to talk to you.
[39:01.460 --> 39:05.460]  And, but then again, you can also get some sort of shitty data.
[39:05.700 --> 39:07.100]  Someone's had a little bit too much.
[39:08.000 --> 39:09.740]  So take that with a grain of salt.
[39:09.900 --> 39:13.640]  But you can, you can do this type of thing anywhere where people are gathering.
[39:13.780 --> 39:25.020]  It just use obvious, obviously use some discretion if it's going to be something where you're going to have to get some level of permission before you just sort of walk in.
[39:26.340 --> 39:44.500]  But that's, that's always something that's possible is, again, most of the time, whether it's a single person that you're asking or it's an organization, if you just explain who you are, what you're trying to achieve, most of the time, overwhelming majority of the time,
[39:44.660 --> 39:46.820]  they're going to say, yeah, come on in, that's fine.
[39:48.880 --> 39:54.960]  If you're working with like a multiple developer team, which developer do you think would be better?
[39:55.180 --> 40:00.360]  Like the one who's more familiar with the exact code you're wanting to test or more project overview?
[40:00.840 --> 40:03.980]  Like what, what do you look for with the exact tester?
[40:06.240 --> 40:13.580]  Well, it's less, well, it depends on how your, how your sort of group is structured, how your working group is structured.
[40:14.820 --> 40:24.980]  In the past, when other people on my team, different teams I have worked on have gone out and done their research, it's because they had a question about something and just wanted to go answer it.
[40:25.060 --> 40:32.100]  So it doesn't, doesn't really matter in that context if they're, you know, one type of developer versus another.
[40:32.100 --> 40:40.140]  If you're choosing somebody specifically to go out onto the field, first, I'd find the person who's the most comfortable with that idea that trumps anything.
[40:42.080 --> 40:47.440]  But again, it depends on the problem that you're trying to solve for that particular research session session.
[40:47.820 --> 41:05.280]  So if it's something that is, you know, having to deal with the flow of something going from point A to point B in a process, you might want someone who's more familiar with the sort of the umbrella overview of the app.
[41:05.840 --> 41:19.840]  Then again, if the sort of the nitpicky more like bug squasher comes along and sits and observes also, that's going to be also very valuable.
[41:19.840 --> 41:31.200]  Because then they're going to understand sort of, start to understand the difference between, you know, the usability bugs and the functionality bugs and how those are playing together, that type of thing.
[41:31.200 --> 41:37.360]  So again, depends, but I'm giving great answers.
[41:39.220 --> 41:49.140]  It does depend, but I would say that it's valuable for anyone involved in making a product to go out and test it.
[41:50.540 --> 42:01.000]  Because then that helps sort of share the expertise of usability testing, usability research to everyone on the team.
[42:01.160 --> 42:07.520]  And then everyone's more familiar with what they need to be doing to make the product more usable for their end user.
[42:08.820 --> 42:11.940]  And it's also great to go out in teams too, so.
[42:14.600 --> 42:15.680]  Anything else?
[42:16.120 --> 42:16.520]  You
[42:19.620 --> 42:20.960]  know, kind of paper kind of thing.
[42:21.120 --> 42:35.580]  But with all the additional technology that we've got now, like Google Glass and stuff like that, where you can actually take feedback back for others to see it, what kind of legal privacy kind of issues do you get with getting permission for recording something?
[42:35.760 --> 42:39.300]  So most of the time when I'm recording something, it's in a much more formal environment.
[42:39.300 --> 42:40.800]  It's like in a research lab.
[42:41.200 --> 42:44.200]  And then there are, you know, there are release forms that you have to sign.
[42:45.800 --> 42:53.560]  There have been times I have gone out with sort of, like, different rigs that'll work, or simply just recording on my laptop.
[42:55.220 --> 42:58.400]  And all you need is just a basic consent form.
[42:58.800 --> 43:05.620]  If they don't want to be, if they don't want to show their face, you can cover up your camera very obviously.
[43:05.620 --> 43:07.540]  You can show them that you're doing it.
[43:08.540 --> 43:13.420]  And if they don't want their voice recorded, then you just deal with it that way.
[43:13.500 --> 43:16.400]  And just take really, really good notes.
[43:17.840 --> 43:21.760]  But a lot of the time, recording is great.
[43:21.760 --> 43:26.760]  But I've also found that it's hard to go back to.
[43:27.100 --> 43:32.360]  Because scrubbing through everything, you don't want to sit through a full user research session a lot of the times.
[43:32.360 --> 43:35.460]  That's why also I do that sort of two-minute recording at the end.
[43:37.080 --> 43:46.340]  But in the case that you do want to sort of see, have a record of where everyone's tapping, that type of thing, you are going to want to get a release form signed.
[43:46.340 --> 43:50.460]  You are going to want to get consent while you're screening them.
[43:51.200 --> 43:53.640]  You know, do you have any problems with your face being shown on camera?
[43:53.840 --> 43:54.300]  Yes or no?
[43:54.300 --> 44:03.880]  Well, going through that type of thing so that you know exactly what they're comfortable with and how to make those changes to make sure that they're the most comfortable when you're doing the test.
[44:19.530 --> 44:25.290]  So the question is challenges with going out with actual hardware that you're testing.
[44:27.910 --> 44:35.610]  Any time... I mean, most of the time, as far as being the researcher, all I'm doing is taking notes and that type of thing.
[44:35.850 --> 44:41.370]  But when you're actually... most of the time you're bringing some other sort of device with you anyway to test on.
[44:42.450 --> 44:50.010]  If it's particularly unwieldy, then I would think that this type of research might be a little bit more difficult.
[44:51.270 --> 45:07.050]  But then again, if there's a certain part of that that you can easily replicate using lighter weight materials, that type of thing, to just have the basic interface on there and have the users go through it like that.
[45:07.210 --> 45:12.170]  It's sort of the same thing as having a paper prototype where it feels dumb, but I swear to God it works.
[45:13.430 --> 45:16.970]  You can still get the same valuable data from that.
[45:17.690 --> 45:18.950]  If it's giant...
[45:20.330 --> 45:21.070]  There's that.
[45:21.250 --> 45:23.690]  If it's giant and unwieldy and that type of thing.
[45:23.850 --> 45:34.230]  If that is the case, then it's best to sort of try to set something up where you're recruiting users and bringing them in to test.
[45:35.850 --> 45:37.490]  So... anything else?
[45:40.610 --> 45:41.210]  All right.
[45:41.970 --> 45:42.870]  Oh, oh, baby.
[45:43.890 --> 45:45.810]  You said that all the paperwork...
[45:45.810 --> 45:47.510]  other people would help you.
[45:48.150 --> 45:51.930]  Like, do they actually do it?
[45:52.090 --> 45:53.910]  Or is it always fall back on you?
[45:55.230 --> 45:56.530]  Can you repeat that?
[45:56.530 --> 46:00.590]  Like, you said that other people in the team can help you to do research.
[46:00.850 --> 46:01.130]  Oh, yeah.
[46:01.330 --> 46:02.750]  Do they actually do it?
[46:02.970 --> 46:03.590]  And if they do...
[46:03.590 --> 46:04.030]  Absolutely.
[46:04.430 --> 46:04.630]  Yeah.
[46:04.630 --> 46:15.050]  Yeah, so the people who are really interested in finding out... doing research and finding out how to do it and becoming more involved in the usability process, they're...
[46:15.050 --> 46:18.010]  most of the time, really, they're fully into it.
[46:19.110 --> 46:26.230]  When you're bringing someone along with you, if you're going in teams of two, I wouldn't go above two, by the way, because then you're going to start to overwhelm people.
[46:26.550 --> 46:37.070]  But if you're going in teams of two, that's also great because the person who's not so sure if they want to be involved or not can have that chance to sort of sit back and be the note taker and see how the process works.
[46:37.070 --> 46:41.250]  And at least then everyone's starting to get more familiar with the process.
[46:44.190 --> 46:44.650]  Great.
[46:44.910 --> 46:45.870]  Well, thank you so much.
[46:46.190 --> 46:46.530]  Thank you.