GameChangerSF uses data science to get creative

Peggy: Hey, hello. Welcome to "Mobile Growth," the podcast series where frontline growth marketing experts share their insights and experiences so you can become a better mobile marketer. That's what it's all about here. I'm your host Peggy Anne Salz from MobileGroove where on my watch in this series, I'll be introducing you to the people who know how to drive growth. It's because they're either UA experts, they have the tech or they have the talent. Either way, they're here on our show and today my guest is Phil Shpilberg. He is president and founder of GameChangerSF. Phil, great to have you first of all here on "Mobile Growth" today directly from San Francisco. That's what the SF is for, right?

Phil: Yeah. Thank you, Peggy. And I'm excited to be talking to you. I've been following you for a long time.

Peggy: Well, thank you. And I think I'll be meeting up with you because you're going to be speaking at MGS San Francisco in February, correct?

Phil: Yes. Absolutely. I'll see you there. I hope to see everybody there.

Peggy: Okay. So, is it going to be a panel or are you going to have an actual talk?

Phil: I'm actually going to be.... yeah, I'm going to be talking about making creative in 2020.

Peggy: So, I couldn't hope for a better segue because that's what we're going to be talking about, it's all about the creative but first, before we get to that Phil, because I want to get to a very interesting new exciting evolution of what's going on with GameChangerSF. What kind of company? I mean, it is a leading app, you know, UA, user acquisition agency, but I think it's more than that now. So, how would you describe your company?

Phil: We really are a data science company at our core. We always have been, before it was trendy, like back when we started in 2012 we always worked with data to get our clients the best results. You know, user acquisition was the way that we utilized that data at first. But now it's really, you know, it's just a lot more than that. Like, you know, developing creative has become a big thing. And using data to augment the way creatives work is really where we're at these days. So, you know, we're doing creative beyond user acquisition, but that's still most of what we do, doing user acquisition and all the activities in the data that support it.

Peggy: Well, I would have to say, you know, user acquisition now is, first of all, it's data-driven, anyway. That's the way it's been built. But, you know, the focus on the creative is incredible. You know, I'm going, I'm hearing, you know, how many creatives do I need? How many versions of every version, not just the size but the different aspects, different color, different background. And that's what you're doing in a new tool that I'm happy that you sort of maybe debuted for me or showed me a little bit more about. I mean, is this the core of what you're doing now at your company?

Phil: Yeah. Exactly. So, we are introducing a product called MultiVid and it's really born out of the need to generate a high number of videos, both to do multivariate testing but also to address multiple platforms, multiple localizations. So, we found that, you know, in a typical campaign we might have to do, you know, three variations on let's say character, three variations on background, three different audio choices. You know, 2 platforms, 7 localizations, 4 sizes, you quickly get 1,500 videos, right? So, you can't really do that with humans. You very quickly get overwhelmed. So, you know, what we started doing was creating a system where we can script all these variables and then output algorithmically all these different videos. And the original intent was really to just help deal with the volume of this stuff. But as we started doing multivariate testing, you know, being able to test all these different videos, we found that so much of the performance we see wasn't actually in the creative concept.

It was actually in the execution of the creative concept, which really started to flip our thinking on creative. Like a lot of what we called good performance and creative turned out to be just sort of noise because there's this spectrum of like how an ad will perform when you're generating a 1000 versions of it. And the difference between the best version and the worst version can be like 300% in terms of, you know, click-through rates, conversion rates. So, where you were on that spectrum in a traditional ad production environment is you made, you know, maybe two or three ads and you tested that versus your other concepts, but you were just seeing just this couple of tiny data points. So basically, we developed this system to... on the front end to generate all these ads. Then to upload them, obviously like you can't upload a 1000 videos manually, but then also to... on the backend to report on the performance and to break apart, you know, the performance you're seeing by the variables. By like did you change gameplay? Did you use a different character? Did you have different texts in the beginning? You know, when you do the multivariate testing, you really need to be able to pick up on what's making a difference and what isn't. And that's really what MultiVid is. It's an integrated system to generate creative, get it to the channels where you need to test it, and then a reporting system to tell you what's actually going on.

Peggy: So, it's probably from end to end. What is it? Is it machine learning, AI sort of driven because something's making the decisions in the background and we'll get to a moment what I'm doing as the human marketer in the front, but is that what is driving this in the background or is it something different?

Phil: Well, I thought, you know, before we started recording you had a really good observation that this is essentially augmenting the marketer. So, it's something in bet... It's really something in between. We're freeing up our creatives to do creative work and then they don't have to worry about the small execution details that actually drive a lot of the performance. So, creatives do creative work. We use machine learning and we piggyback on algorithms that Facebook has, for example, to figure out which creative is performing best. And then we, you know, we report that in a, you know, really easy to understand way like, "Hey, the character's driving, you know, most of the performance variation or, you know, your audio choices aren't." Those kinds of things we get back to our creatives and we keep iterating. You know, there's still human factors in this, like which variables that you choose to iterate on.

Peggy: Oh yeah, absolutely. I mean, we're going to get into that because I love the idea of the augmented marketer. And there's one other point here before I get into that side of it, which is also understanding the number that you need because I have a feeling that the MultiVid can also give me some advice on really the right number and really the right sizes and just all of those details. Because I go into this thinking I know what I need, but I think that it also does some calculation there. Am I correct?

Phil: Yeah. I mean, we're seeing a lot just from being able to do this. Like you learn so much more when you're able to generate this many videos. Like we can see when it's too much and not making any difference or when it's not enough. Because there's a whole bunch of variation going on in this, you know, small number we're producing. I mean, every situation's a little bit different and, you know, how fast do you want to do things matter? Because you can, for example, you can go out and test, you know, three variables and one size on one channel. And then once you get results you can decide to, you know, localize to resize and to, you know, put it on other channels. Or you can decide to do all that at once and go and, you know, run a bunch of tests which will get you faster results, but maybe it will be more expensive. So, there's definitely, you know, it's situational, but we definitely have much better information about that than we used to about how many creatives does it actually take to zero in on something that's performing very well.

Peggy: Almost like how many people does it take to screw in....No, nothing. It's almost a little bit like that joke. I'm going to go down that road for you because I'm going to bite at this one. So, how many creatives, I mean is there any sort of rule of thumb, because all I know is that it is multiplying?

Phil: Right. Well, a lot of it is, what do you think is going to matter? And I find, you know, through our testing we find that there's usually about two or three things that matter a lot. And then some smaller things that matter a lot less. So, you know, you can sort of do the math, right? Like if you're going to have per creative concept, if you're going to have like 2 variables that, you know, you know what those variables are and you're going to make, you know, 3 to 4 iterations, then you're going to end up with, you know, maybe 50 videos. Let's say, you know, in a video... like we can do this with static too, but video is a good example. So, maybe 50 videos per creative concept without resizing it on Facebook is.... is now pretty good number. But let's say you're in beta and you don't know what those factors are, then you want to test, you know, 10 different variables and in an extreme case, well, yeah, then you need like 500 videos. So, it's going to vary along those kind of parameters.

Peggy: So, what are you seeing in some of the trends? Because I think what's great is not just that you have this product and we'll talk a little bit more about that as well, but it gives you some additional insights into like, "Hey, wow, you know, here are some things that I'm learning like the right number to be developing and testing, depending on maybe a certain type of app genre, or you know, a certain type of game genre." You know, there's got to be some rule of thumb that is emerging here. I'd like to push you a little bit more to share that because some people don't realize yet probably what's coming. You know, they don't understand that we're getting into a period in testing iteration optimization, that it's just beyond human capacity to deal with it.

Phil: Yeah. You know what? The biggest realization for me has been like the once in a career kind of like, "Oh my God" moment has been that we've been doing creative all wrong. I mean, that's really it. Like we've been generating creative concepts, you know, testing them and hoping for the best where we really needed to be doing was creating frameworks, you know, putting a bunch of variations through and executing the best possible version of an idea. So, it really completely flips, you know, the way we do creative is completely different now. So, you know, if you're making creative the way we used to, you know, five, six years ago where you're just making more, you know, slightly more versions like the difference between one and five is very small. You know, we're talking about going from 5 to 500 in some cases. That's a much different setup.

Like you have to completely change your process, the way you create this stuff, the way you look at it and the way you optimize it. And there's, you know, there's so much detail that's emerging from that that I can, you know, like rules of thumb. We can talk about all that. But I think the more important point is you really need to rethink how you're doing this process. Like I've been a marketer for 20 years, we've been doing creative and to just sit there like a few months ago looking at the data and being like, "Oh my God, the performance on any given ad has been noise." That's all it's been. It's been noise. And the signal has been completely hidden from us.

Peggy: So, I'm going to make you explain that because some people are going to be saying, "Okay, I heard that, but I have to really understand that." You know, it's a very strong statement. We've been doing creative all wrong, which is probably right from what I can tell. But let's elaborate on that. What are the key mistakes we've been making?

Phil: Well, I think it's just the approach. I mean, it's not really the mistakes, it's just we didn't have the tools to do it. So, I want to clarify. It's not really that we've been doing it wrong. We've been doing it wrong in the way that like making, you know, Nintendo games initially had bad graphics, right? We just didn't have the tools to have high definition graphics at that time, and now we do. It's just... I mean, it's almost like, to get a little "Rick and Morty" on you, it's like there's infinite universes out there where the only difference is your ad works, you know, or works slightly better. And we're just trying to bring that kind of thing. We're trying to bring that to fruition where you could test almost an infinite amount of ads to find the best, you know, the best iteration of it. And so, like when you think like that it kind of flips... it flips the script on how you do it.

Peggy: I get it. Because what you're saying is you're saying, "okay, human marketer, you're augmented by the machine learning." It's going to come up with the multiple, multiple "Rick and Morty" universes where this works out like in the parallel places, but you come up with the concept and leave the multiplication, right? Multiplying... Here we are, we're human, right? We're going to make mistakes. You're going to leave that to the machines and you're going to come in with initial creativity, some serendipity, and then mix it all together and then boom, you've got a 1000 plus different versions of this, thanks to your platform, thanks to your tool.

Phil: Exactly. Yeah. You know, I'm not here to only talk about the tool. I actually... I think it's just...it's bigger than any tool. I really hope to change the way people think about that stuff. That's what I'm going to be out talking about.

Peggy: Well, that is going to be a teller. That's exciting. That means, you know, people listening in, listeners, friends, we don't miss this talk because I want to hear that myself. I'll be there anyway, but I'll make certain I won't miss your talk. We do have to go to break for a moment, Phil, but absolutely coming back because we're going to talk about, okay, we talked about the fun of the creative development. Now we're going to talk about the hard work of like figuring out what matters, what are the results, all that measurement, all that stuff, the reasons that you have to come back after the break.

Male: If you haven't already, be sure to check out the Mobile Growth global events series where thousands of the world's top growth marketing experts come together to discover the latest in growth marketing innovation. We run events all around the world in cities like San Francisco, Los Angeles, New York, Berlin, Montreal, and more. And our candid panel discussions feature industry leaders including Facebook, Google, Uber, EA, and loads of other top tier publishers. So, attend our lineup of intimate workshops and gain the in-demand skills you need to stay on top of your growth marketing game, expand your professional network and build meaningful relationships in a fun, friendly publisher-first environment. When and where can you get in on the action? Well, just visit mobilegrowthsummit.com for a complete list of upcoming shows. And if you should decide to join us, and I hope that you will, then be sure to use our special promo code MGSPODCAST30 for an additional 30% off your ticket order. Once again, that's MGSPOCAST30 all caps, no spaces for 30% off of your order. We hope to see you there. I certainly hope to see you there and please enjoy the rest of our episode.

Peggy: And we're back. I'm your host, Peggy Anne Salz with MobileGroove. We have Phil Shpilberg. He is president and founder of GameChangerSF. Phil, great talking about your tool. We're not going to stay on the tool. We're going to stay on the concept because I found it was just very creative. I really enjoyed the "Rick and Morty" comparison there. It makes me feel creative. That must come from somewhere. Are you just like a die-hard fan or does that have relevance here?

Phil: I am a die-hard fan. My son actually introduced me to it who's 14 and kind of horrifying when you know what the content is. But we actually for a time, we worked with Adult Swim on the Pocket Mortys game. So, that wasn't the biggest engagement our company ever had but that was probably my, you know, as a geek, one of my favorite projects.

Peggy: I could see that. I'm binge-watching currently. Actually I wrote an article for Forbes recently and I had to mention the "Rick and Morty" campaign with McDonald's and I thought, "You know what? This is great to write about it, but I'm not up on my Szechuan sauce." So, I thought, "Okay, that's what Netflix is there for." So, you know what I'm doing on weekends? Not entirely, but, you know, get the idea.

Phil: Yeah. We've got a new season too. So that's exciting.

Peggy: I saw that. So, I have to catch up. I gotta be ready for this stuff. Got to stay on top of it. Speaking of staying on top, you know, creative development, it is amazing. That's where I said before and I'll stick to it. Augmented marketer is what I'm going to call it. Makes a lot of sense.

Phil: I love it. I think you came up with that.

Peggy: I came up with it myself because I was thinking about augmented...

Phil: Yeah. You should patent that.

Peggy: I got it. I'm going to be writing about it soon. It's all mine. You can't use it, Phil.

Phil: Awesome. We got it on record. It's yours.

Peggy: Absolutely. But you know there is discipline to that. Even if we're not disciplined this evening, it's ended Friday here on my day, but there is a discipline to it. There is saying, "Okay, this cuts it, this doesn't, this makes the cut." You know, this performs, you're looking into this, we talked about the creative process, how that's changed, how that's evolved. What about measuring results? How has that been changing and how are you maybe even making that possible?

Phil: That's been one of the difficulties, right? If you don't have the tools and you just look at the reporting that Facebook has, it's going to be very hard to get the bigger trends when you're looking at individual ad performances. Like, I've heard somebody actually who I really respect in the industry say that only 1 in 20 creatives that you produce will beat your best creative. And that's just not an acceptable situation for us, right? We don't want to fail, you know, that high percentage of the time. So, being able to separate out the data, like which variables that you produced actually move the needle and which didn't is really the goal of the multivariate testing. And even in a, you know, creative that "fails" you're going to get something that's going to help you make better decisions. And that's really the goal is to always be moving things forward.

So, separating out, you know, when you create these multivariate tests, right? Let's say you pick five variables, it's really important that you go back and you first of all look at which variables actually mattered and which didn't because a lot don't. Like that's what we're finding is there's usually just a couple of things that really move the needle. And the other question is, did you pick good variables to begin with? That's the human part of this. If you, you know, if you pick audio on Facebook and it's off by default, you don't expect that it's going to make much of a difference. We actually threw that into a test just to show that and it absolutely doesn't make that much of a difference. You know, the character you use like the spokesperson and like literally their hairstyle, it'd make a huge difference. So, there's stuff like that. And it's really important to be disciplined about, you chose the variables for a reason. You have to follow that all the way through. And, you know, if you just choose some stuff, throw some stuff at the wall and just say, "This one ad performed the best," and that's all you've learned, you've probably not learned that much.

Peggy: What about key learning? You know, I've talked to some people and it's like, for example, you think something failed and you say to yourself, "Well, that's a fail." No, I've listened to people say, "No, just put it away somewhere. It's going to come back." So, it's like there's never really ever a black and white here. That was something that I found interesting as a learning, but maybe it is or isn't matching with reality out there remembering that I don't have an app, I just talked to app marketers.

Phil: I've seen some stinkers that just need to be retired. But the biggest thing that I've learned, you know, as a marketer, a performance marketer for 10 years is like that, you know, just the marketer for 20 is that, you know, I can look at creative all day long and I can never predict what's going to really resonate with people. Like it's amazing. Like my ability to stay ignorant I guess, you know, never ceases to amaze me. And so, looking at, you know, looking at data, as long as you learn something, I guess it's not really a failure. The way you can completely fail though is if you... you know, if you don't follow through, if you don't follow through the process, if you don't.... you know, if every time you make new creative, it's completely new to you, you're not coming from a place of having learned something, then that would be what I would consider failure.

Peggy: What about in-housing versus outsourcing? What side are you thinking? You know, I wouldn't say it's, which is going to win, but I know a lot of people who thought it was like, oh well, you know, Marc Pritchard over at P&G says, "This is what we need to be doing." It's like, "Yeah, go ahead and try and do that." But do you have that organization? You know, are you Proctor and Gamble? Do you really want to take this gamble literally?

Phil: You know, it's got so many...we could do a whole, you know, talk about that because it's a fascinating topic. Like I think, you know, both have their place. I completely understand when people want to scale to the point where they're doing in-house as, you know, Mobile Growth is, you know, it's pretty fundamental to the growth of a business. So, I understand that. The tricky part, having built a team, you know, over seven years is it's really difficult to build the team, get it to work well together. Because remember your creatives and your user acquisition managers, they have to work really closely together. They have to have a common language, you know, having creatives understand performance is not a small task. And like this kind of creative development we're talking about is, you know, infusing a lot of data into creative.

And so, creating that coherent team, keeping them interested. Like how many apps do you have? Like we... you know, we work on hundreds of apps. It's always exciting. We could learn things. My people, you know, I like to think they're really engaged and having a good time. You know, back when I was working on two, three apps at... you know, at a game publisher, you know, it got boring at some point. There just wasn't that much going on. So, I think there's some challenges to building in-house teams. Location is actually a big thing. Like if you want people locally and you're someplace other than couple of the big, you know, places where you have people, are you going to find people and when they switch jobs are you going to be able to replace them? Like people leaving is... really has a huge impact on teams. And then there's all this technology that you're constantly having to rebuild. Like that's the other thing, you're never done. Like you come up with an algorithm, you think it's good and then Facebook changes their algorithm and everything's broken. I think in those scenarios you're better off letting agencies, you know, people outside who deal with only that deal with it. But I am a little bias.

Peggy: Yeah. But I would also agree on the point though about the technology because you think you have it all in place and then it's just like, "Oh, had I only had, you know, more of a cloud-based solution, I'd be up to date on everything here rather than having to tear it all apart and put it back together again." So, I can definitely see that. Are there certain types of companies that would go more, you know, not just for your services but overall for outsourcing because, you know, games is vast?

Phil: Well, I mean, we would work with app developers as well. I would say when you're launching a product, you're not sure if it's going to work or, you know, this isn't necessarily your core business. Let's say you have a business and you're launching an app to support it. That's not a good time to go hire a whole team because what happens if it doesn't work out in three months? Like you want to be hiring and firing people? That scenario is a really great one to engage with an agency who, you know, is going to bring a lot of learning in and then you're not having to do all this kind of shuffling. You know, on the flip side, if you're a company that publishes an app a month, you know, then that's a good candidate for having an in-house time because the next thing is coming. You can predict what you need to do. So, you know, handling the ups and downs of launching and scaling things are a pretty good opportunity to work with somebody who does it all the time.

Peggy: And you work with all app genres and categories, or do you specialize on any particular or do you see there's more opportunity in one than the other?

Phil: We really have done that. You know, in the seven years we've done this, we've worked with everybody from, you know, Samsung, you know, "Highlights" magazine to you know, Atari and traditional game companies and everything in between. You know, we've had more experience in gaming only because gaming took off faster, I guess with mobile and my... you know, my background's gaming, so I just had a lot of contacts.

Peggy: But now everything else is coming, you know, and it's interesting because I'm talking, I was talking the other day to, might have been Mastercard. We were talking about gamification, right? So, I mean, everyone isn't on it now. Where are you seeing your clients come from now?

Phil: We're just seeing a lot more traditional businesses, you know, large...you know, like the Microsofts of the world and the Samsungs coming and, you know, working with us because a lot of these things were pioneered with gaming because gaming tends to be on the forefront of this stuff. We're just, you know, it's just the nature of gaming. So, a lot of what we learned from games is now really broadly applicable. You know, even back when I was in game publishing, like we were using social media 10 years ago where people didn't... you know, that wasn't really done in business. So, it's just a good way to test things they are going to work. And then it comes over to kind of more mainstream business.

And then there's, you know, there's the whole category of businesses just born of app, you know, apps like Lyft and Airbnb, you know, we're seeing more of those. But those are actually, those are good candidates for having in-house teams for obvious reasons.

Peggy: When you're talking about business, you know, I'm also thinking about, you know, the video business. I mean, this is the format I'm looking into, ad formats and their performance right now, you know, which are the ones that perform the most, the best. What are your observations? I mean, I'm going to put my hand up for video. You're going to do that, because you have a tool as well. But looking at it as an agency, what do you see as the formats where there's still like a lot of growth and mileage left?

Phil: Well, yeah, so this tool is actually, it can be used for static as well. It's not a great name. I probably...I might change it. You know, if you have ideas though, I'm open.

Peggy: Okay. We'll make that into a contest. We get to name your product. I get to keep augmented. We'll work it out.

Phil: The augmented marketer. Yeah. Absolutely. You know, it's interesting. Yeah. So, video does rule in most cases. Playables, it's interesting. Playables, I really feel like there was a lot of hype for it, but we've just not seen the performance. We've not seen enough impressions out there. I feel like Facebook and Google are moving away from it and it's expensive to produce. Like actually our creative director came from a background of making playables. So, when he came on, you know, a year and a half ago, I thought we'd be making a bunch of playables. I was excited about it. It hasn't really panned out. Like we're not seeing the performance. The place where we see static still working really well is, there are certain categories where I think static works better, like kids' apps interestingly because I think parents just need to see a brand they recognize in a lot of cases. So, if you're Crayola or Highlights or, you know, one of those iconic brands, you don't need to produce like fancy video. Parents are sort of like, "Oh, I need somebody to babysit my kid in the back of the car and I know this, I don't need to watch a video." So, we do see we get a lot of..yeah, there are situations where we get mileage out of static but I would say like 80% of what we're doing is video these days.

Peggy: Well, you know I feel I could keep on going here but we are running out of time. I'm going to have to bring it to an end. I really, really regret that. But, you know, for many of our listeners and friends who are hanging out in San Francisco, they'll get to see you in person at Mobile Growth SF, so MGS SF, MGS20 in February. But in the meantime, how can they stay in touch with you and keep up with what you're doing over there at GameChangerSF?

Phil: Absolutely. Well, can I just do a quick plug for the session because we want to do something fun there?

Peggy: Absolutely.

Phil: So, I'm going to go on record as saying we're going to try in the short time we have to produce a bunch of ads and launch them while I'm talking. So, come see that because there's a high chance of failure with any live thing with internet involved. So, check it out. We're going to start at 8, I don't know.

Peggy: That would be very cool.

Phil: Yeah. We'll try to make 50 videos just to show people how that works. If people want to get in touch, the website, gamechangersf.com is a good place to just see what we're about and what we're doing. I'm on.... you know, or the company's on all the socials, so GameChangerSF on Twitter, on LinkedIn, on Facebook. And then personally the best place to reach me is on Twitter. I'm @Phast, P-H like Phil, Phast P-H-A-S-T.

Peggy: Okay. Yeah. So, we know we're going to see in February, we know a good check out your session where you're going to do this live and hope to have you back again as well, Phil. It was such a delight and I may even come after you and ask you to do a guest post on Mobile Growth on the website because I think you have a lot to offer, some insights around what works and what doesn't in creative, what to automate, what not to automate and somehow and why. Would you be game for that literally?

Phil: Oh yeah, I would love it. Talking to you has been great. Actually, you've helped me crystallize some of the ideas as well, so thank you for that. And I'd love to write a post.

Peggy: Hey, cool. And I won't send you a bill for that, Phil.

Phil: Oh, thank you.

Peggy: My consulting services today are just, you know, free of charge. I'm in a good mood. So, listeners, friends, we're having a great time today. So, thanks for listening to this episode of "Mobile Growth" podcast. A quick reminder to visit mobilegrowthsummit.com for a complete list of upcoming events. And don't forget to use that very special promo code MGSPODCAST30 for 30% off of your offer. We hope to see you there. I hope to see you there. In the meantime, of course, check out this and every episode of our series over at mobilegrowthsummit.com and on SoundCloud and coming to many more channels, providing you many more ways to listen in. So, watch for that. We'll watch for you and we'll see you soon.