Podcast Transcript
Vira: 0:41
Hi everyone, and welcome back to Email Einstein, the podcast by Flowium. We are your hosts.
Andriy: 0:47
Hello, hello.
Vira: 0:48
Vira Sadlak and Andriy Boychuk. Andriy, say hi.
Andriy: 0:51
Hi, everybody.
Vira: 0:53
Yeah, we’re happy to have you back. Welcome back. By the way, if you’re still listening to our podcast and if you didn’t know that we have YouTube videos now as well, definitely check them out because right now we have podcasts on both audio platforms and YouTube videos. And very often we share some cool visual references that you can only see on our YouTube videos as well. So definitely go and check them out, especially today.
Andriy: 1:24
Just one caveat, Vira. Sometimes we post YouTube videos on YouTube live. I mean, making live video like with delays. However, if you’re listening under this podcast, there’s, there’s link to our website and you will see the video which will be live when this podcast episode is live because we have special schedule on for— we have Thursday schedule for YouTube and Tuesday schedule for podcast. So that’s why sometimes it’s like YouTube video comes with some delay.
Vira: 2:00
Yes. And today’s episode is actually a special one. We just wanted to do something light and something fun around the busiest time of the year. So today we will be talking about the A/B test, but we will not just be talking about the A/B test. We’ll test Andriy’s marketing intuition and we’ll see whether or not Andriy can pick the winning variation for the A/B test that we’ve done for our clients. Andriy, how do you feel about your marketing intuition? Yeah.
Andriy: 2:32
So when, when, when you said, when you said, so when you sent me this idea last week, I was like, yeah, 100%, let’s do it. But now I’m kind of a little bit afraid or afraid about my intuition because with years, listen, it all depends on the brands. It all depends on personality and what I love to say. Even like in our marketing department, when we work in our internal marketing, I said like, what I think does not matter, data matters. So like I might say, Like I like this thumbnail for YouTube better versus this. But if the data shows that that thumbnail that I like does not perform as close to the previous one, no matter what I hear, like, like we need to pick the winning version for your audience. And in our marketing department, we do things according to who our listeners are, who our viewers are. So same thing here. I’ll do my best to guess and let’s take take apart those A/B tests. But again, I will not have all the details about the brand. So my guess might be a little bit off.
Vira: 3:46
Totally. Yeah, I’m like curious to see what you, what your answers will be. Full disclosure, you guys, this little game was prepared by our amazing coworker Amanda. Amanda, big shout out to you. She like gathered all of this information. She presented all of this like results on our weekly calls. And to be honest, Andriy, this was very humbling to me because I pretty much like lost in majority of like A/B tests, my guesses were like off on like a lot of them because I was trying to, I mean, I was probably like basing it on my own preferences. And just like you said, your own preferences, they are not necessarily the same as your audience preferences. So that’s why running this like A/B tests are so, so important. So let’s do this, Andriy. Number 1.
Andriy: 4:34
Let’s start.
Vira: 4:35
Number one. So we have, for those of you guys who are just like listening to us, we have this like beautiful brand where we test it. It’s an apparel fashion brand where we tested having discount code information in the hero image versus not having discount code information in the hero image. So we’ve tested just like a lifestyle image and more like an informational, like lifestyle plus informational image. Andrey, what’s your take What do you think about these designs, first of all? And what do you think will be the winner here?
Andriy: 5:11
I need to understand what is the goal here.
Vira: 5:14
Yeah. So the goal was to improve the click-through rates here. And obviously the revenue is also like a nice bonus to improve, but the click-through rates were the main thing. What’s your take? Option A, discount in the hero image. Version B, no discount in the hero image. So, Andriy, what’s your take?
Andriy: 5:36
I like how— I love how you did this, or Amanda did this.
Vira: 5:39
It’s not me, it’s Amanda. All our amazing girl Amanda. Yeah, we just need to have this music.
Andriy: 5:44
How is the game called?
Vira: 5:45
It’s like Who Wants to Be a Millionaire or something. I don’t know how it’s called in English because like I grew up watching it in Ukrainian and it was like a hit growing up. But yeah.
Andriy: 5:55
Yeah, I don’t think we can use that music because they will suspend us. because probably have this like licensing, right? Personally, in terms of click-through rate, if we are talking about click-through rate, I think the images or emails without revealing the discount in the top part will have higher click-through rate. And my logic there is it has some kind of curiosity or it’s like does not stand out to you. Mm-hmm.
Vira: 6:27
Yeah.
Andriy: 6:27
So this is my opinion.
Vira: 6:28
Let’s see, Andriy, if you are right. Yes. Let’s see. No discount in the hero is a winner. So Andriy, you were right. Actually, the no discount is a winner in both variations. And it’s not just in the term, in terms of like click-through rates, but also obviously in terms of like placed order rates. So good job, Andriy. One point to you. Okay, cool.
Andriy: 6:54
Thank you.
Vira: 6:55
The next one, and obviously the big, the big lesson here is to know your audience because obviously the results in one account could be like completely different in another account. That’s why it’s not a one-size-fits-all. Definitely test it with your account, but we will give you a few ideas on what can be tested. Okay, Andriy, in this example, I want you to pick the winner. The goal here was also like the improvement of the click-through rates. And for those of you guys who are listening right now, we are looking at 2 different, at 2 different opt-in forms. One has one CTA, just like a get 50% off CTA. The other one has claim 50% off CTA, and it has additional CTA, which is decline offer. The rest of the form is identical. The fonts, the colors, everything is identical. The only difference is that second form has additional CTA. Andriy, what’s your What are your thoughts here?
Andriy: 7:56
So yeah, I’ll vote for number B where it says decline offer. And one additional thing that I would change, probably remove that X sign because there’s 2 options that actually each— this form has 3 call to action. The X in the top right corner also as a call to action kind of. But sorry, set aside that, I would vote for number 2 because decline offer, I forgot the term. in the psychology. But basically, you’re doing something kind of negative. Like, I don’t want to decline. Like, why are you telling me to decline the offer? You know, like, and you kind of want to accept it. So it’s like counterintuitive. But my vote is for option B.
Vira: 8:38
Okay, let’s see. Honestly, that’s my favorite too. But let’s see. So option B, adding a decline offer is your choice. And Andriy, you’re again right. Although the difference was not statistically significant. I mean, the difference was like pretty evident in the revenue or in the submitted form numbers, but it’s like they are pretty close to each other. So we’ll need to gather more information to have the definite answer. But yeah, you were right. Probably the don’t tell me what to do psychology works here. So that was interesting.
Andriy: 9:17
It’s not only don’t tell me what to do, but one when you say to somebody, like, do you want, like, do you want something? Like, they have option yes or no. And they’re like, when you kind of taking things away, they immediately want this. It’s like kind of reverse psychology or whatever it’s called.
Vira: 9:38
Mm-hmm.
Andriy: 9:38
So that’s why, because I know those forms, I read the case, few case studies where they have options. Yes, or no, those, those kind of forms perform much better versus just get, get it now or give it to me or something like that.
Vira: 9:57
Probably like creates the illusion of a choice that someone is like giving you a choice and you’re like in control of choosing whether or not you want this thing. It’s pretty cool. It’s pretty cool idea, you guys. If you haven’t tried that, definitely try this with your accounts. It might be a fun little experiment to do. Okay, Andriy, one more opt-in form A/B test. So for those of you guys who are not watching us right now, but listening to us, we have 2 very similar forms. On one form, Guy is pointing out just like to the signs above his head. On other form, he’s pointing out to the CTA where it says take survey. Andriy, for the click-through rates, which one do you think performed better? The first one or the one where he points out to the CTA?
Andriy: 10:47
So I think it’s when he points out to CTA because in the first one there’s no CTA at He just helped us improve, but there’s no explanation of what, what are you asking. However, this email might have not dramatic difference, but I will vote for B option.
Vira: 11:07
So you’re voting for option B. And actually, the option B, the one where the guy is pointing out to the survey link, actually is the winning metric. And not only in the click-through rates, obviously we do have some difference in the click-through rates, 2.27 versus 2.83.
Andriy: 11:28
Yeah, but you see, I’m right. Hold on. I’m right about the second part as well, because it says that’s not statistically significant, isn’t it?
Vira: 11:37
It’s not statistically significant when it comes to click-through rates, but if you look at the revenue, Andriy, the second form generated pretty much like twice as much revenue. So yeah, while the results were not statistically significant in the click-through rate department, in the placed order rate, and I think we need to look at the combination of both. it did generate more revenue.
Andriy: 11:56
I would disagree. I would politely disagree with that statement because like revenue, again, I don’t know, like, but it’s double revenue. Why is double revenue? Like, were you sending them to the same form?
Vira: 12:11
The same audience, same form. And I think the EMS who works on this account, he like did a few tests like this. So this is just like one of 5 tests that he, that he’d done and he decided to go with the form When were the guys like pointing to the CTA? But yeah, this is just like one of those examples where like little tweaks can help you, can help you improve the overall performance of the form.
Andriy: 12:36
I agree, I agree. But the second, the first email didn’t have clear call to action. So this is kind of my feedback that I would even, I would personally would not even test those, those kind of emails because if you have clear CTA, it’s like, yeah, you can maybe guy pointing to the CTA and guy does not point to CTA. So, right.
Vira: 12:57
I think it actually did have it. It was just like at the bottom of the image. So that’s why it’s not shown here. But yeah, I see, I see what you’re saying. So I think the EMS was testing the placement of the CTA and in the original version it was like below the guy. But yeah, it was like interesting to see. Okay, the next one, Andriy, we were testing long versus short form campaigns and this one can be a hit or miss depending depending on what you are selling. But what’s your take on like long versus short campaign? First of all, which one do you prefer normally?
Andriy: 13:32
For, again, for what? Like, what is our goal here?
Vira: 13:36
So again, we’re testing like click-through rates here, pretty much in like majority of this test, we will be testing the click-through rates and revenue as a nice bonus as well. So what’s your take on these designs?
Andriy: 13:52
Since this is about discount, which are expiring, and it has urgency, honestly, it’s hard to say. I would say longer, let’s— but I might be wrong. So I’m not sure here. So let’s go with long one.
Vira: 14:10
So you think the long one would outperform the short one in terms of click-through rates, right? The variation?
Andriy: 14:16
Yeah, because you know why? Because I see this brand and they have many SKUs and Typically when some like kind of accessory brand or fashion brand, when they have more accessories, it’s better to include more products. So there’s more way, kind of more call to actions because if you don’t like that dress, you might like another one. So there’s more options for you to click. So here’s kind of my logic here, but this one, it’s weird because it has urgency, discount urgency, which is we want to focus on the kind of get the offer now. And also we including other products. So it’s like, honestly, I have a mixed, mixed feeling, but let’s go with the same.
Vira: 15:00
Yeah, I’m like, I’m in between the two because very often I see that when you’re giving the customer like too many options, it can almost be like a distractive. It can almost like distract the audience from the main CTA. And that’s why I’m like very hesitant of like adding a lot of product blocks in those sales campaigns. But you think that the long campaign would outperform the short one, right?
Andriy: 15:21
Yeah, let’s, let’s try this case.
Vira: 15:24
Let’s see. Well, actually, in this case, in this case, the short campaign with 100% winning confidence outperformed the long one in terms of click-through rates. So we are talking like 1.21% compared to 1.41%. Why do you think this is, this is the case here, Andriy?
Andriy: 15:49
What you just said, like there’s not many options, you just click here, that’s it. Like there is no— that’s why text-based emails with one link performs very well because there’s nowhere else to click. And yeah, and also additional thing, maybe it kind of sparks curiosity like, oh, what do they have in their store, in their collections, you know?
Vira: 16:13
Right.
Andriy: 16:13
Because when you’re showing people what you have, it removes the curiosity to find out what you have.
Vira: 16:19
Agree. Cool. Okay. I think we have a similar campaign test. Again, we were testing a long versus short campaign, but this time we were doing it for this like wellness brand that is all about like educating their customers, nurturing their customers through some information that is like relevant to them, educational or entertaining. So what we’ve done here, we’ve tested short form of the email. I still wouldn’t say it was like a short email. We still had the product and basically like the ingredients explained. And in the version A, we actually had like this almost like a paragraph that explains the benefit of each and every ingredient in each and every product. So, Andriy, what’s— what do you think is going to happen here? Shorter or longer email?
Andriy: 17:12
I think the longer one, the longer one is on the right, correct? Because you have more products there.
Vira: 17:18
Actually, no, the variation A, it just like it didn’t fit basically. But the variation A is the longer one. It has more text here. Variation B is shorter one. It’s just have— it just has like a product block and like a literally a one-sentence explanation of what this product is. So I think this email actually You think B would outperform the long one?
Andriy: 17:42
Yeah. So the shorter one. Why? Because it has a short blurb. And if I, by the way, beautiful email, like kudos to our team.
Vira: 17:49
Thank you. Raphael, shout out to you. Yes. Yes. Irina and Raphael, you guys love what you do.
Andriy: 17:56
Yeah, no, like beautiful. I mean, beautiful email, beautiful layout. So I, if you are listening, I strongly recommend to check out the YouTube video to see the, the structure of this email. This is amazing inform— infomercial kind of email layout plus like selling the product. I think the version B. Why? Because it’s— I cannot read it, but it provides a short blurb with a call to action. Short blurb with call to action versus like long explanations. After I read the long explanation, maybe you will answer all my questions. I’m like, I’m not interested versus you might spark curiosity with a short description and I want to click to find out more. So this is my logic.
Vira: 18:41
Let’s, let’s see if your theory is correct. So you think that the shorter campaign would outperform the long one. Let’s check.
Andriy: 18:49
In this case, right? In this case, yeah.
Vira: 18:52
Well, in this case, the longer campaign actually won. And we’ve done this test on like multiple campaigns. So we had a sequence of, I think, 5 campaigns, and in 4 out of 5 campaigns, the longer one won. We were testing the click-through rates. But what we’ve noticed is actually, while click-through rate difference is like not that significant, it’s like 0.7 compared to 0.72. The placed order rate consistently was higher in those emails where we were providing that like extra education to our customers. And I think it’s partially because this brand, it’s basically why they do what they do. They deeply care about their community. That’s how they connect with their audience. And I think that’s like works for this specific audience. And honestly, I was like very surprised with the results from this series of campaigns. But yeah, sometimes the longer one means the more revenue, you guys. Yeah. Are you surprised with this result, Andriy?
Andriy: 19:51
Yes, I am.
Vira: 19:52
Yeah, same. I was shocked to see this, this numbers, but we kind of have like a formula for this brand now. Every time we’re doing the educational campaigns, we are going for the longer form. But every time we have this like promotional emails like Black Friday, Cyber Monday, we literally all we do is discount code information, sometimes even in the hero image, then short text and product blocks. That’s it. So that’s like the formula that you use.
Andriy: 20:14
So here’s a recommendation. Here’s a recommendation for you. If you’re like an e-commerce brand owner or if you work for one, here’s a recommendation. We strongly recommend for you to create a document internally. I believe we call it key findings and document those split, document things that you learn. What Vira just said for this particular brand, the longer educational content email works better. So She will document in that, so in the future you don’t have to reinvent the wheel or retest the things. You can just build on top of it, right?
Vira: 21:20
Yeah, yeah. We do keep track of all of our findings in those documents, and all of our knowledge comes together and shines, especially around Black Friday Cyber Monday.
Andriy: 21:32
So that’s by the way nobody knows this, but let’s let’s talk about some internal secrets right now. We like testing testing a tool which we might. implement for all our clients. We’ll see. So basically it would be kind of internal ChatGPT where you upload the results, you upload your theory, like you explain to the chat what you were trying, what kind of results you got. And in the future, either you or somebody else like asking, it will say, oh no, don’t do that because like 2 years ago we were testing X, Y, and Z and it wasn’t working. So it’s kind of like internal—
Vira: 22:10
That’s cool.
Andriy: 22:11
Kind of AI to like educating with the things and it will spit out the answer or ideas.
Vira: 22:20
That’s cool. Yeah, but it’s still a secret, so don’t tell it to anyone, Andriy. Yeah, yeah.
Andriy: 22:24
Let’s keep it just to you and I.
Vira: 22:25
Just like to hundreds of listeners on this podcast. No, that’s so exciting. That would make our lives easier, honestly. Sometimes you just like forget about all of these things that you’ve tested in the past and it’s It’s cool to be able to build up on what you already know about the brand for sure. Okay, cool. Next test, we were testing darker, aka bold design versus like lighter white background design that was like more like focusing on like brand colors and stuff like that. Also, we were trying to improve the click-through rates. So Andriy, what do you think?
Andriy: 23:02
I think darker. I think darker one, just because black mode, right now is so popular. This is my hypothesis.
Vira: 23:10
Honestly, that was my thinking too. I thought that dark background would be more funny.
Andriy: 23:18
Because you listen, when you started that podcast episode, you said like, my, uh, this is how I was thinking and it was counterintuitive. So you just—
Vira: 23:26
Yeah, that actually was one of the ones that I didn’t get right either. I thought that dark mode would outperform the light one because I think the majority of the users, I don’t remember the number off my head, but it’s something like 60% something of like iPhone users are using dark mode now. So like majority probably of mobile users are doing the dark mode. That’s why I thought that the dark one would outperform, but in 2 out of 3 tests, the white one outperformed statistically significant. So do you have any thoughts why the white background emails performed better here?
Andriy: 24:02
Honestly, no clue. I mean, one hypothesis, maybe it broke the pattern. So, you know, when you’re checking all emails on the black mode, it’s like black, black, black, black, and like white stands out. So like kind of—
Vira: 24:17
Right. Like, yeah, I mean, visually speaking, I do prefer the white one. Just it’s like it’s easier to read. It’s like more— it has more contrast, so it’s like easier on, on the eyes. Just like comparing these 2 designs, I would prefer the white one, but I did think that the black one would outperform. Yeah, so that was, that was an interesting finding. Okay, cool. Let’s do the next one, Andriy. This one we kind of discussed with you on the podcast, but with all of my— possibly with all of my accounts, I like extensively test different types of imagery. So in this case, for this automotive brand, we were testing the lifestyle imagery. Versus the product imagery. So Andriy, what do you think about like these 2 approaches? Which one do you think will be the winning one, the product or the lifestyle?
Andriy: 26:00
Based on what I read and the case studies I saw, the lifestyle will win, I believe. But again, we’ll see.
Vira: 26:10
We’ll see how it goes.
Andriy: 26:12
This brand is so, this brand is so kind of like The product itself is such a kind of dry product. I mean, not dry product, but it’s just like by standalone product, it’s like it doesn’t mean much. But when you provide the picture of how you can use product, how your life would be adventurous, cool, on your car, the shocks. This is what the shocks for the car. I think, again, I believe it’s a lifestyle imagery. Right.
Vira: 26:41
Yes. And this brand, they are doing an amazing job with their content creation. And they do have like a ton of lifestyle content. But also they have like a ton of content that is doing like the reviews for the specific parts of the car, for like specific shock brands and stuff like that. So, Andriy, you think that the lifestyle images would outperform the product images? I was actually surprised with your answer because I know that you’re subscribed to this brand. And lately, we’ve been exclusively doing the product images and not the lifestyle ones because every time we send the product image in the hero image, it outperforms the lifestyle image like times and times again, especially when it comes to, when it comes to placed order rates. So click-through rates, the difference is like not that drastic, like 0.95 compared to 1 point. 0.02, or like 0.76 compared to, well, the same click-through rates. But like when it comes to placed order consistently, product images won in all of our tests. Here we have displayed only like 3, but we’ve been testing it like multiple, multiple times. And this is like the consistent result that we are getting. So, Andriy, what’s your thoughts here?
Andriy: 27:58
So, Yeah, so my take, I still like, I mean, personally, I like lifestyle. However, I have no clue, like, I have no experience, no professional expertise in those products. So for me, lifestyle makes much more sense. However, I think people who are ideal customers for this brand are people like professionals who know, like experts or fanatics about their cars. So they are like, when they see that product close up, they can either spot the perfection or imperfection. It can click, oh yeah, this is what I need. Versus me who is like, I mean, I don’t even know how to—
Vira: 28:44
how this thing works. I just like pretty things.
Andriy: 28:47
Yeah. It’s like, I just like, yeah, yeah. I don’t know. So for me it doesn’t make sense to see those like sticks and, and springs, like It’s just sticks and springs for me. But so for somebody who buys them and it’s, it’s not cheap, it’s like what, like a few hundred or thousand dollars?
Vira: 29:02
A few thousand, a few hundred.
Andriy: 29:03
Yeah. Yeah, yeah, yeah. So like when you spend, it’s like you probably know what you’re buying. You’re not like me shopping on eBay, the cheapest one from like made in China. I don’t care if it’s not branded, just send me the cheapest one and I’ll, I’ll use it for the next 2 months.
Vira: 29:21
Right. Yeah, no, I was like shocked honestly about this shock images. I was like shocked to see the results here because I also like pretty things, but to be honest, I don’t know much about shocks and people who are the target audience of this brand, they come to this brand for education, right? This brand is doing an amazing job of like educating their customers about like certain like brands, certain shocks. They test the shocks on like different vehicles. So So that’s how they bond with their customers, right?
Andriy: 29:56
Yeah. But one thing that I would challenge you to do is like for Roosevelt’s, we used to work— I mean, I used to work a lot on that brand. They have shirts and sometimes it was— they had a collection called Little Things. So they have like little icons or images on their shirts. So like if we show the whole shirts, it will not be visible. So they show the whole shirt and only bubble of of that icons like blown up. So it’s much bigger. So it would be extremely curious to see here where you have lifestyle and maybe the blown out version of the product you kind of promoting. So it’s kind of combination and see if it makes any difference.
Vira: 30:42
What you’re saying is that you think it’s a good idea to see how like the product looks like in the car, like on the car as a part of the car setup. Is that what you are saying?
Andriy: 30:53
So the third, the third task that you did where you had the white Jeep on, like, yeah, that one. So for example, you have the same image, but you will also add the product itself on top of it somehow.
Vira: 31:08
Like, like, so like a combo, like a collage almost.
Andriy: 31:10
Yeah, yeah, yeah. Kind of. Yes.
Vira: 31:13
Oh, that’s interesting. We might do that. Yeah. But actually, this task was like Yeah, this test was so successful that we ended up updating like majority of our like images and majority of our flows. And yeah, and that’s what we’re doing now.
Andriy: 31:29
One more, one more thing that I would kind of strongly recommend and challenge you to do, to do the heatmap and see where exactly people clicking. Are they clicking on call to actions? Because that thing might give you better understanding of what’s going on in this email. Right. Because maybe it’s not a cool idea. Yeah. Because for example, the last test, the last 2 emails, maybe people didn’t click on the lifestyle or product. Maybe they were clicking those 2 white, white products blocks, you know?
Vira: 32:02
Right. Yeah. And I think that’s a possibility. But like what we’ve noticed is that like the revenue is like consistently higher on the product ones compared to the like the lifestyle ones. And I think it’s because the brand is showing themselves as experts in the industry. So they, like, connect with something that is, like, valuable to their customers. That might be the reason. But I really like the idea of creating that, like, heat map and seeing, like, what is happening with an actual email.
Andriy: 32:33
By the way, by the way, if you, like, if you’re not sure what we are talking about, go to YouTube, search for email heat maps and add Flowium, or just email heat maps, and I will be the should be the first. My face should be there. If you don’t know who I am, just type in Florium and you will see. I made a quick tutorial on how to do it. It’s free, nothing extra you have to pay. Just you have to add additional code for each link to see where exactly people are clicking.
Vira: 33:02
Cool. Okay, Andriy, are you ready for the next one? I think it’s going to be actually the last one. Are you ready? Let’s do this. So for this high fashion brand, we were testing 2 different opt-ins. Again, the goal here was improving click-through rates. One brand, one branded opt-in form had this like beautiful model on the image, very like high fashion vibe, very like on brand. And the second one had the founder of the brand. It’s a bit like a more like a warmer image, more personal. So the first one is like a high-end lifestyle image. The second one is a personal CEO, like warmer image. Andriy, what do you think about this one?
Andriy: 33:49
You know, like I have an answer right away, but now you threw me off at the beginning of the podcast. So now I’m like, I have to rethink all of my answers now, but I will still stick with version B because let’s say listeners to our podcast go to our website and if it’s image of me or you there, and some kind of other model or other stock image, I think, or I assume people will probably pick the option with us or you or me because kind of they know us kind of, or they feel that they know us. Again, this is my opinion. Yes. And I think the version B, I don’t even know that they are CEO of this company.
Vira: 34:36
You don’t know either of these people on the image, Andriy. You don’t know even like the girl with long hair.
Andriy: 34:42
No, no, no, I agree. So yeah, yeah. But let me say, but those people looks to me more kind of real, like, like took picture on the iPhone or similar versus the left one. I see she’s posing and she’s kind of more kind of— it looks like she’s like more model. She did it.
Vira: 35:03
Yeah. I mean, it’s a high fashion brand, right? So the first one is like a more like a high fashion model almost. The second one, the reason why I ask whether or not you know any of these girls is because this lady with like a longer hair is actually the famous pop songwriter, pop singer Meghan Trainor. And you’ve heard her songs, Andrey, like gazillion of times. And I think it actually helped as well because you were right, the variation B, the CEO plus Meghan Trainor actually won. And you can see that the form submission rate, the difference is actually very significant, 1.37% compared to 2.12%. So you were right, Andriy. I don’t know which one contributed more to the success of the form, whether it was just like the vibe, overall warmer vibe, or that they had a pop star in their image. But yeah, I love what they’ve done here.
Andriy: 36:05
I’m definitely not their audience, you know, like I’m not their potential customer.
Vira: 36:10
Yeah, I could tell. But I mean, normally, like for this type of brands, I would think that more polished one would work better just because it’s like their brand. But I kind of like this.
Andriy: 36:24
No, I would disagree. I would disagree. Like, look at the Adidas, Nike, all the like recent advertising people are, or Victoria’s Secret or other brands, people trying to kind of make models more real now.
Vira: 36:40
Now, again, right?
Andriy: 36:43
Yeah, maybe. Listen, maybe still 80% are still those like perfect ladies or whatever the perfect is versus, I don’t know, like Nike did the big campaign with people with amputees, like they were— they didn’t have legs. And it was huge, huge success according to different media outlets. It was a huge success for Nike. And I think, again, this is what I see lately, those kind of advertising, they were— those people looks like more like relatable people versus models, right?
Vira: 37:20
Yeah, I can see that trend too. But good job, Andriy. You were right on this one. So these were all of our examples. And the main takeaways here, you guys, and you can probably have, maybe you have some other takeaways as well, but for us, the biggest ones was that always do at least like 3 tests, even better, like 5 for more accurate results. So that’s like the number one. Make sure you know the intention behind your test. It’s super, super important. Like know why you’re doing this test going into the test. Understand your brand and audience because as we could see, like the same test could perform completely different for different audiences, like that longer versus shorter example that we just shared. And, and also sometimes what you think would perform better does not necessarily perform better. It’s not always the case. So that’s, these are the reasons why we test and these are the reasons why you guys should test as well. Andriy, thank you so much for doing this.
Andriy: 38:24
You’re welcome. It was fun. And actually, I would love to do more of this. Or maybe we’ll come up with some different, like, playful thing. And or actually, you listeners or viewers, please send us your split tests. Like, it would be cool to kind of, I don’t know, play with you. Like, if you provide us your version, let us guess. But also what I was trying to search on YouTube Honestly, I’m bad, extremely bad searching my own videos, but I record the video about, and we will include it under this podcast. I recorded how we do split tests. So we always start with like hypothesis and we do take some time to do this. Then we record the data and we make a conclusion and we make the next steps. So I have that video explaining that, but I cannot I don’t know the name, so after I will include it under this under this podcast so we can watch it and learn it from us how we do it internally.
Vira: 39:27
Yeah, totally. That’s something that we always do, you guys. Everything that we mention in this podcast, even like the brands that we are mentioning, the things that we like, things that we don’t like, everything will be listed in the description box. So there you have it. All of this amazing tests that can be done by your brand as well. And I think this like post holiday time, this like January December is. a great time to do all of the A/B tests because it’s kind of like a slower time in the year, and that’s your time to perfect everything that you have to prepare for busier seasons. Thank you so much for everyone who listened, and we hope to have you here next Tuesday. Take care. Bye. Bye.