Prototyping with Sound: Audio & Speech Playback in Adobe XD with Matt Rae

[MUSIC] 0:00 Well, next up, I'd like to introduce Matt Rae. 0:07 Matt is a designer advocate and XD instructor for Adobe based in Canada. 0:11 Coming from a decade and product design across industries including travel 0:17 autonomous vehicles air tech and advertising technology. 0:22 He's now focused on equipping designers with resources to design 0:27 the best experiences using Adobe XD. 0:31 He has a passion for user experience and developing the design community. 0:34 Please welcome Matt. 0:40 >> So I'm gonna jump in and share my screen. 0:42 Today, we're gonna be talking about prototyping with your voice and 0:46 sound in XD. 0:49 And as Toni mentioned, I'm Matt Rae, I'm a design advocate for Adobe XD. 0:51 My job is teaching XD and showing you how you can use it in your design process. 0:56 So if you're learning design, this is a great tool to get started with and 1:00 kind of carry forward in your professional career. 1:04 So if you don't know what XD is, XD is a fast and 1:08 efficient UX design platform. 1:13 And it plugs in with all your other creative Cloud tools. 1:17 So if you've used Photoshop or Illustrator, it just fits in as part of 1:19 your workflow and gives you really powerful tools to design cool experiences. 1:23 And so today we're gonna be talking about voice and sound. 1:28 I only have a little bit of time to show you these things. 1:31 So if you're looking to learn more about how you can use this tool in your design 1:34 workflow, check out adobe.com/learnxd and there's lots of tutorials and guides. 1:39 It's a great way to learn alongside things that you might be learning with Treehouse. 1:46 So with that, why don't we dive right in? 1:50 And so we're gonna start by talking about different types of sounds 1:53 that we can use in our design. 1:58 And I'm just gonna turn off my headphones here so 2:00 that you're gonna hear some sounds ambient in my room going through the microphone. 2:03 So drop in chat if you can't hear those for any reason. 2:08 But we're gonna get started. 2:12 So right here I've kind of really boiled things down, made them simple. 2:15 Here's four different ways that we can use sound and 2:20 audio in our designs to augment them and make them even more accessible, 2:23 provide better feedback, or make them more interesting. 2:28 So the first one is notifications and feedback. 2:32 You've probably heard some of these in the past when you send an email and 2:35 you get a nice whoosh as it sends off into space, somewhere in the cloud. 2:39 So notifications and feedback are a great way to add on top of visual and 2:44 tactile feedback in a mobile device or web. 2:49 So hopefully you heard that, I'll play it one more time. 2:54 That's a little ding. 2:58 This could be used for a success message, something like that. 3:00 And then with accessibility, this is becoming even more and 3:05 more important every day. 3:09 Accessibility gives us different tools and 3:12 ways to interact with a website, or a mobile app, or a desktop app. 3:15 And you can think of using sound as like how can we emulate a screen reader in 3:21 our prototypes or accessible interactions like that? 3:26 So using voice and sound features that I'm gonna show you here in a minute, 3:31 we can actually build some of those into our prototype so that people who can't see 3:35 the prototype or have to interact with it visually or audibly, they can do so. 3:40 So here's an example of that. 3:45 >> Play button, press Enter to plan Treehouse teams. 3:47 >> So there you go. 3:51 Then you can press Enter and you're navigating with the keyboard. 3:51 And then of course voice assistants, which are all over the place now, 3:55 everything seems to have a voice assistant. 3:58 And this is a great way of just having a dialogue between a device or 4:01 a virtual assistant. 4:06 And so we can do that as well. 4:09 And this is all built in XD where we're designing our websites, designing our 4:12 mobile apps, everything from emails to games, and we can build it all in here. 4:16 And then of course, we can add some fun. 4:20 So if we're designing a music player like Spotify or SoundCloud, 4:22 we can actually prototype with real music. 4:26 [MUSIC] 4:28 And have a little bit fun, add some fidelity to the design. 4:32 [MUSIC] 4:35 So with that, let's dive in and 4:37 show you how we can integrate this into some of our designs. 4:40 So hopefully, that sound is coming through all right, 4:44 it looks like there's no red flags or anything. 4:47 That's great. 4:50 So what I have here is I have a web design, a very simple flow that I've 4:52 built for booking an adventure travel, adventure trip somewhere in Colorado. 4:57 This isn't the real place. 5:04 I mean, it kind of is but this is my own website design. 5:05 And you'll see I have some different screens here. 5:09 It's kind of talking about the booking process. 5:12 So let's just start by showing you what this is like without any sound. 5:14 So when we play it we have two book buttons. 5:19 I can scroll through the website. 5:23 And then when I click on Book Now, it's gonna pull up in a modal. 5:25 And this is really just to confirm that the selected dates 5:29 are what I want to book. 5:31 Of course, this is a very simplified flow and so I can hit Yes, continue and 5:33 it's gonna go into this loading process. 5:38 And I mean all of this is happening on the web, some cool graphics, 5:41 we added some confetti, that's great. 5:44 And then we can go ahead, print our receipt or add to calendar. 5:47 Of course, I'm missing a full booking flow here, I never gave them a credit, 5:50 that's all right, we're gonna glaze over that. 5:53 So that was okay. 5:57 There were some cool elements there. 5:58 But I really only had visual feedback. 6:00 I didn't have anything tactile because it is the web. 6:03 So I'm not getting any haptics on my device, 6:05 and I'm not getting any sound either. 6:08 So why don't we take that same flow and prototype in some sound for feedbacks? 6:12 And just to add to the experience, you're booking a cool adventure trip, 6:18 you should be able to get excited about it, right? 6:22 So let's start down here. 6:25 And you'll notice XD as a whole is pretty lean. 6:28 There's some simple controls here. 6:31 I can access libraries of stock imagery, 6:33 plugins down here, and then the layers of my design. 6:36 So this is a very similar view to many tools that you may have used. 6:41 I can move layers around, group them, lock them, and 6:45 then I can go from design prototype and share all in one spot up top. 6:49 So I've just designed this website in design mode. 6:53 I've made use of all the powerful grids and guides in here. 6:56 And now I want to prototype, I don't have to export any of this, 7:02 I can just do it in the same tool. 7:05 So I'll just toggle to the prototype mode. 7:07 And you can see I already have some things hooked up. 7:09 Just to save us time, I'll walk you through them. 7:12 But from here, I can just start prototyping. 7:15 And this means I can prototype really at any stage of my design and get a feel for 7:18 what it might actually feel like as I experience it, 7:22 versus just kind of going through slides. 7:25 So here we have our Book Now buttons. 7:29 You can see I've connected these with a simple tap, so this is a mouse click and 7:31 it's gonna open it overlay which I have designed right here. 7:36 This is the same one we saw. 7:40 And I just wanna add a little bit of feedback to this when I click Book Now. 7:42 So on this Book Now button, I have a tap. 7:48 But you can see down here under my action, I'm also going to add a ding. 7:51 So I'm just gonna pull that in, and I'll make sure this one has it too and it does. 7:56 Okay, great. 8:02 We now have positioned the overlay, that's what this green box is. 8:05 So this screen is now gonna show up here. 8:09 And when we click on this, you can now hear the ding, it opens the modal. 8:12 That might not be the right sound for this application, but you get the point. 8:19 So now that we're in this modal, we can add in things like voice 8:24 playback that will actually read the contents of the modal to us. 8:28 So, for instance, if we're designing something to try and 8:33 simulate a screen reader experience, we can do that here. 8:36 And so to do that, we can click on our prototype here. 8:40 You can see that I have one tap back going to the previous art board. 8:45 And when I play that, all that's gonna do is take me back to the original screen 8:50 that I brought the overlay up on. 8:54 And so I can actually click plus here add multiple 8:57 interactions into the art board so that when I land here, 9:01 it's actually going to go ahead and play it. 9:06 So I'm just gonna hit plus and add a time delay. 9:10 And we'll just set it to 0.2 seconds so that there's a chance for 9:15 the page to load or people to start reading it. 9:19 And then in my actions, I have all these different action types, 9:23 really nice animations, simple dissolves, the overlays like we are using here. 9:27 And then I have two options, audio and speech playback. 9:32 Audio playback, I can load up a music file, a voice-over file, 9:36 any kind of audio file I want to play. 9:41 I can use that with the audio playback. 9:44 And then with speech playback, I can actually enter in text and 9:47 then have one of these different personalities read it out. 9:50 And they all have slightly different sounds depending on the vibe you're trying 9:53 to portray. 9:58 So we're gonna stick with Joanna for this. 10:00 And what I can do is I can just simply copy the text that I want Joanna to read. 10:02 And then come back to my time trigger, paste that in. 10:09 And then, the text is in there, and she's gonna read it like that, but 10:14 let's add a little bit more context. 10:18 And let's say, please confirm the dates 10:24 you have selected for your trip. 10:29 September 1st, 2021, and this box is going to basically read it how you write it. 10:35 So if you write it all jumbled up into a run-on sentence, 10:42 the voice is gonna read it that way too. 10:45 So be sure to kind of pace things out how you want the virtual voice to read it. 10:47 But that's why I'm adding these spaces or paragraphs in between each one. 10:55 So that it reads it with a pause. 11:00 And you wanna kind of match that natural cadence so that it's easy to understand. 11:02 We're used to hearing the way we speak, right? 11:06 So is this correct? 11:08 And so now when we play this, if we come back to that home screen, 11:10 hot only are we gonna get a ding when it moves over, but 11:15 we're also gonna get this voice reading out the action to us. 11:18 So when I hit Book Now. 11:23 >> [SOUND] Okay, let's get you booked. 11:24 Please confirm the dates you have selected for your trip. 11:27 September 1st 2021, October 1st 2021, is this correct? 11:30 >> And so you can see here, as an answer to that question, I have, Yes, Continue. 11:36 I could modify this edit button say, no, edit my selection. 11:41 Whatever I need to make that CTA really clear. 11:45 And if I was to click on this button, right now I have it wired up to then 11:49 progress into the next section of the flow. 11:54 But just because the system can talk to us, we can actually talk back to it and 11:57 control our prototype entirely with our voice. 12:02 So I'm not gonna just settle for a simple tap, I'm gonna add in, 12:06 I'm gonna add in a voice command. 12:13 And this voice command is going to transition, just do a simple dissolve. 12:16 And it's gonna move over to our Aspen Adventures confirmation screen. 12:21 That's this one right here. 12:29 And so using a dissolve, 12:30 but now you'll see this command field up under the voice trigger. 12:31 And this is where I can enter in a command. 12:35 And I can stack many of these out to capture different things that people 12:38 will say depending on which region of the world you're in. 12:41 They might have a different way of saying it. 12:45 So you can capture those things with multiple, different commands. 12:47 And so let's just enter, Yes, Continue, to match the, call to action of the button. 12:53 Now this is gonna take us over here. 13:01 And then this is gonna actually enter into a new flow. 13:03 And I already have this wired up. 13:06 So I'm just gonna walk you through it before we play the whole thing. 13:09 And I apologize, I have something in my eye, and it's driving me nuts. 13:14 So once we arrive at this screen, 13:18 we basically have this loading indicator that is going to appear. 13:20 And then it's gonna come down here, and all I've done is I've used auto animate, 13:26 which is great for 13:31 creating nice micro interactions to animate this loading spinner. 13:32 So simple, it's not using any sound. 13:36 Our tap is, though. 13:40 So if we go back to our tap, 13:42 you'll see I have this action that's actually gonna play back audio. 13:43 So that was a nice way to add it into the tap gesture. 13:47 The voice gesture doesn't, but you're gonna get the point. 13:52 And then from here, it's gonna come in and 13:56 start another loading sequence with confetti. 13:58 So what's more exciting than confetti exploding on the page? 14:00 I don't know. 14:07 But we're gonna go into confetti. 14:08 And this is all just a simple little auto animate interaction. 14:10 And at the end, 14:13 we're gonna pull in another type of audio just to kind of seal the deal. 14:14 So if we come back, we know that, Book Now, it's gonna read out. 14:20 >> Please confirm the dates you have selected for your trip. 14:27 September 1st 2021, October 1st 2021. 14:30 Is this correct? 14:35 >> And so to reply with my voice, 14:36 I don't have to do anything other than hold the spacebar. 14:38 That is the trigger in Adobe XD to listen for your voice, so 14:41 it's not listening all the time. 14:45 You do have to prompt it and say okay, listen to me now. 14:47 And I'll say, Yes, Continue. 14:51 And then it's gonna progress into this loading flow. 14:56 It's gonna go to that empty screen and then show confetti. 14:59 And now. 15:02 [MUSIC] 15:03 So just like that, we could add in this whole other element with a little bit of 15:08 fiesta music, celebrating that you've now booked your trip. 15:12 And it gets people excited, right? 15:16 It adds a whole other element that just the confetti didn't really accomplish. 15:17 So that's just a really quick look at some of the ways that we can apply this. 15:22 And you can go as far as you want and 15:27 build out full flows using just voice and sound playbacks. 15:30 I've seen situations where people build full presentations recording their 15:34 voice and just adding in those narration samples throughout the design. 15:39 So you can click through it like a slide show or a lesson. 15:44 And, of course, when you're on mobile or doing voice assistance, 15:50 you can use sound design in a way that really helps educate the user, 15:55 gives them proper feedback. 16:00 So, just as an example, I'm gonna pull open some sounds. 16:03 And this is from an Adobe designer, Sam Anderson. 16:07 He put together this nice sound kit with all these different sounds for 16:11 different purposes. 16:16 So you can see sometimes, [SOUND], you'll have a nice positive ding. 16:18 You might have a low power indicator. 16:23 [SOUND] And that's just aligns to, your phone is getting low on battery, 16:24 you might wanna plug it in. 16:29 And then you can see these falling and rising indicators. 16:30 So, usually when it's rising, it's gonna be positive. 16:33 [SOUND] Like a nice, happy ding. 16:36 And then you can get your sad dings. 16:39 So sound design can actually play a really important role in 16:42 giving that auditory feedback to a user using your app or device. 16:46 And you'll notice, depending on the platform, there's a lot of consistency. 16:51 And it can make a pretty big difference in your experience. 16:56 Yeah, I could swipe through these all day. 17:01 [SOUND] That's one of my favorites, the swoosh. 17:03 [SOUND] Right, so you can play with sound all day. 17:05 If you're looking for any kind of music, 17:10 there's also things like the YouTube Audio Library, where I pulled that song from. 17:13 And things like Adobe Stock have free audio samples, too. 17:18 So there's lots of different sources that you can pull from to experiment with this. 17:22 So with that, just to make sure we have time for questions, 17:27 I'm gonna switch over to questions now. 17:31 And so feel free to go into the Q&A pod, 17:34 if you do have questions. 17:39 And I can answer those, but in the meantime, 17:43 I will show you some more things. 17:49 So, as I mentioned, auto animate is really handy for 17:56 creating micro interactions, and you can also create components. 17:59 So just like in this button, I'm gonna copy this button out, 18:04 And paste it over here. 18:13 You can actually create reusable elements called components here that 18:15 allow you to reuse this element and build all the interactions in it. 18:21 So if you want to associate a ding or something to an element, 18:26 you can build them into these components. 18:29 And all you have to do is take that element, right-click, hit Make Component. 18:32 And then you can add in different states. 18:36 So hover interactions, toggle states, you can design these out and style 18:39 them in a way that basically saves them as a state or a variation of that component. 18:44 And then once you're into prototype mode, 18:51 you can actually go between the states and use those same commands or 18:54 triggers to add interactions that are baked into the element. 18:58 So then every time you grab that button, you can set up the command to say, 19:03 Book Now, and it's gonna take you to the booking page. 19:07 So when you have global destinations in your interactions like that, 19:10 it's a really great way of connecting it all together. 19:14 Okay, here's a couple questions. 19:20 So can you speak more about user testing sounds, 19:27 particularly when emulating a screen reader? 19:30 How have you user tested screen reader performance? 19:33 So just to be clear, we're not necessarily designing a screen reader. 19:35 We're providing feedback in the apps that kind of simulate what that 19:41 experience might be like. 19:45 So when you share out these prototypes, you share them as a web link, they can 19:47 open them in the web and then hear the sounds like they would in this preview. 19:51 So, there's different ways to achieve it, right? 19:57 One of the other triggers is keyboard interactions. 19:58 So, let's say I just select this logo, I can select keys in game pad. 20:02 So, I can actually map certain keys on my keyboard to the prototype. 20:09 So, one way you could simulate a screen reader is using certain keys to activate, 20:15 and this can be other game pads, right? 20:20 I have a PS5 controller here that I can map to the design as well. 20:22 So, if you have a Bluetooth keyboard with braille, you could connect that and 20:27 then map certain keys to certain elements in the design. 20:31 And when you do that, you could then trigger a speech playback or 20:34 voice command. 20:38 So, there's different ways to approach it, depending on the depth you wanna go. 20:40 But it's not going to play exactly like a screen reader, 20:43 you do have to kind of build that up. 20:47 But you can make the entire design talk back to you as you're going, 20:49 which is nice. 20:54 And then, of course, share it as a share link. 20:57 Can it generate baseline code that can then be used to build the real 21:00 site on top of it? 21:04 Yeah, so XE will provide you some CSS snippets. 21:06 When you go to share a prototype, you can share it out as development mode. 21:10 And then when you create that link, you'll still have the preview mode. 21:15 But then you can go into inspect mode and see each element, 21:20 see all the parameters of it, character styles, colors, get a CSS snippet for it. 21:24 It's not gonna export a full HTML page, but 21:29 it's going to give you all the information you need to actually build it. 21:32 And it's also geared to work well with design systems. 21:38 And then the audio piece of that, of course, 21:41 it'll tell you the information you need, 21:45 and there's more to come on that front in the future. 21:49 Okay, I'm not sure, Anonymous, on the question about web development 21:57 libraries or tools that offer this type of accessibility. 22:01 That would really depend on probably a lot of factors. 22:05 I don't know too much about that, but somebody else does and 22:09 wants to answer that in chat. 22:12 Looks like Daniel already did. 22:15 Yeah, so Anima is definitely another plugin that 22:19 you can use to export HTML code. 22:23 It's not always clean like Daniel says, so depending on your development workflow, 22:27 you might need to tweak it and adjust it for your needs. 22:32 Okay, so it looks like that's all of the questions. 22:38 I'll flash my contact info back up. 22:46 If you do have questions about XD, 22:51 wants to learn more about prototyping with voice and sound. 22:54 Of course, we only have about 20 minutes, so 22:59 feel free to reach out to me right here. 23:02 Find me on LinkedIn or Twitter, and that's gonna be your fastest response, probably. 23:04 Yeah, thank you for listening and, yeah, enjoy the rest. 23:11 Pull eyelid to help with lash. 23:18 Thank you, thank you that's great advice to end on. 23:20

You need to sign up for Treehouse in order to set up Workspace

Sign up