Reams of 2019 trends predictions posts are calling out AI as the mega-trend. There’s no doubt AI will power more cost-effective UA and bring other efficiencies, but there’s also a lot of hyperbole around where it will shine, and where it will suck. To cut through the hype our host and Chief Content Officer Peggy Anne Salz catches up with Matt Sadofsky, Senior Director of Growth at TIDAL, a subscription-based music, podcast and video streaming service, for an edgy discussion about the *real* costs and benefits of baking AI into your UA. Matt also draws from his experience, including building an AI system from scratch, to share best practice and debunk myths about how teams will need to work when AI has a seat at the table. Enjoy—and Matt will be back for an encore in a special mini-series where he and Peggy talk shop and grill growth marketers.Peggy: Hey, hello and welcome to "Mobile Growth," the podcast series where frontline growth marketing experts share their insights and experiences so you can become a better mobile marketer. That's what it's all about here and I'm here as well, your host, Peggy Anne Salz from MobileGroove where I help my clients grow their revenues and audience reach their content marketing and on my watch this series will introduce you to the people who know how to drive growth. But I'm really excited because we're not just gonna talk about how to drive growth. We're gonna team up with a growth marketing expert, Matt Sadofsky, senior director of growth marketing at Tidal. Matt, great to have you on "Mobile Growth" and great to have you on the team.
Matt: Yes, thank you for having me aboard. I'm glad to be coming on today's show and doing more with you in the future.
Peggy: Absolutely, because we both know...well, I know because I've known you for quite a while, you were one of the first, probably two years ago, to point out that there is no UA without AI, which I thought was pretty cool to say, and that's what we're gonna be doing. We're gonna be talking about AI, we're gonna be talking about those developments, we'll be talking about other stuff too throughout the year. But, you know, hey, you're gonna be my partner in crime here on a fairly regular basis. So tell us all a little bit about yourself, Matt. What are you doing these days? What are you doing at Tidal?
Matt: Sure. So I joined Tidal about four months ago now to come on and lead growth marketing for them. So for them, that's kinda defined as user acquisition and conversion rate optimization and product marketing. So super excited to have joined the team at Tidal and excited to see, you know, what I'll be able to do there over, you know, the course of 2019.
Peggy: So what is it about the trends in AI? I mean, I'm seeing so much about it, you're seeing so much about it as part of your daily job, and we're really talking about it as being more than a trend. We're talking about it as being, you know, a tactic. First of all, do you feel like you were, you know, one of the pioneers, you were on the money, you feeling a little smug maybe about having figured that out so early on?
Matt: Oh, I think a lot of people knew it was coming but I don't think a lot of people were confident enough to dive into it head on. And maybe I was a bit foolish but I...you know, the first thing I did when I started at Tilting Point, the last company I was at, was literally, I think day one or day two, they had this very rudimentary version of a rule set based optimization system and I said, "Okay, well, we need to take this much further."
And it kinda just snowballed into what ultimately became a full AI. So, you know, kinda it was more of just a passion that I decided, "You know what? Let me take this opportunity and run with it." So I guess you could say I was an early mover in that sense.
Peggy: So where does the confidence come from, Matt? Because, you know, I'm reading tons and always researching something and, yes, so many surveys of marketers where they're frightened that, you know, AI is going to replace them, replace their jobs, you know, wreck their career. I don't know. Just a lot of fear, a lot of anxiety and you're...you know, there you are embracing it, you know, completely. What makes it possible for you to see AI as like a friend and not a foe?
Matt: I think I agree that a lot of marketers are going to lose jobs because of AI. I think a lot of people are gonna lose jobs not just in marketing. So for me, it really came down to do I wanna be the person that's replaced or do I wanna be the person that's doing the replacing. You know, and as cynical as that may sound, you know, it's inevitable that technology is going to decrease the need for workers in the workplace and we're gonna need more strategists than we're gonna need boots on the ground people, people coming up with the ideas rather than people executing the campaigns. And even if marketers like myself aren't coming up with the technology platforms like Google and Facebook, they are just gonna become more automated anyway. So it's kind of inevitable.
Peggy: I mean, you bring that up and makes me think of some work I did with Autodesk a while back actually where they were talking about you're gonna need to co-create with AI, you're gonna need to sort of let AI into the process and it's gonna need a completely different skill set. And on the back of that, you know, we've also seen a number of schools, including MIT most recently, you know, they've completely revamped the curriculum to say, "Okay, you need soft skills, you need to be creative because maybe AI won't be so creative. It'll just be really, really smart." That is backdrop. What are then the skill sets you think that marketers are gonna need to co-create, collaborate with AI in their daily job?
Matt: Yeah, so I think a good marketer in 2019 is going to be someone who understands data very well but also has a creative eye. And that's a hard mix to typically find and it makes hiring for growth marketing pretty difficult sometimes. But I think if we kinda look beyond 2019 and we start looking into the 2020s when AI really starts to take off, I think the necessity to actually have true data skills goes down but the ability to understand software and not necessarily be able to know how to code itself but understand how systems work and how software development works becomes more important and that creative aspect is always going to be there.
Peggy: So what exactly is the skill set that you have, Matt, because definitely data, a passion for it, but I believe your actual formal education is more in psychology, correct?
Matt: Marketing actually.
Peggy: Okay.
Matt: So yeah, so I studied marketing and management and global business. My original thought when I was first going into college was either to do psychology or finance and I didn't know which one I wanted to do so for me marketing kinda felt like the psychology of business. So that's kind of why I decided to go down that route. But I started off my career as a data analyst, a marketing analyst, so that's where I kinda got my data background from.
Peggy: And of course, I think the psychology or the interest in it, you know, initially, that brings a lot to the table as well. Now, we've been interacting for a while because, you know, that's also why I thought it'd be great to team up with you here is because, you know, you've been looking at this but you don't just look at it and talk about it, you actually really do it. And at your earlier company, you built a system called DORA that I wrote about in a Forbes article. I know all about it. I'm really excited but I think you should recount some of that for our listeners because it was really, you know...it wasn't just unique and just wasn't your handiwork but it also drove some pretty incredible results.
Matt: Yeah, it was kind of a game changer for Tilting Point. We always had this vision of us being a small game publisher that was gonna never be more than a 100 people but we wanted to find a way to support a large number of games and we were gonna do that through technology. And I ended up having a seven-person marketing team that was supporting 25 games and we had a hundred-something-million-dollar marketing fund that we were spending across those games and bringing on more every month. And the reason we were able to do that with such a small team is because we built data system called DORA which used machine learning based lifetime value projections in conjunction with integrations with the APIs of different networks like ironSource, Facebook, Google, Apple Search Ads to fully automate all of our bid and budget adjustments based off of real-time learnings. So our UI managers were still looking at the numbers, looking at return on ad spend projections and moving budget between sources but...and working with our creative team on developing the creative but they weren't necessarily doing a lot of the heavy lifting that, you know, some of these other companies had typically drove hundreds of bodies at...to be able to manage a Facebook campaign.
Peggy: You make it sound fairly straight forward. I mean, is it really that straight forward? I mean, how much work went into, I guess, architecting it before you were actually building it? Because I'm hoping that maybe, you know, some of our listeners might understand what goes into this, you know. First, it's great to have a strategy but how actually do you execute on it? And there are some specific steps in that process.
Matt: Yeah, it's not as, you know, terrifying I think as many people would think, you know, in terms of building out an AI-based automation system sounds intense but we had one really solid data scientist at Tilting Point and he was kinda the, you know, brains behind the algorithm and it took...and it's interesting for us too. We originally started off with a rule set based system. It was, you know, pretty rudimentary and kind of old school and over time, it just slowly evolved into, "Okay, well, let's start doing projections." And then, "Okay, well, let's start building out machine learning capabilities to make these projections better."
So that whole process from start to finish probably took about eight months but the actual AI development part of it was, you know, maybe a month or two of work for one person. So you know, if you have a good data scientist and if you do the math for a data scientist and a technology that they're gonna need to build this out and research that they're gonna have to do, if you have to send them to conferences, whatever it may be, let's say it's a $200,000 investment. If you're spending $10 million a year, it's a relatively small percentage investment for building out something that's gonna save you more time and money in the long term.
Peggy: Right now, listeners, we do have to get a break so don't go away. We'll be right back.
Man: If you haven't already, be sure to check out the Mobile Growth Global Event Series where thousands of the world's top growth marketing experts come together to discover the latest in growth marketing innovation. We run events all around the world in cities like San Francisco, Los Angeles, New York, Berlin, Montreal and more. And our candid panel discussions feature industry leaders including Facebook, Google, Uber, EA and loads of other top tier publishers. So attend our lineup of internet workshops and gain the in-demand skills you need to stay on top of your growth marketing game. Expand your professional network and build meaningful relationships in a fun, friendly, publisher-first environment. When and where can you get in on the action? Well, just visit mobilegrowthsummit.com for a complete list of upcoming shows and if you should decide to join us, and I hope that you will, then be sure to use our special promo code MGSPODCAST30 for an additional 30% off your ticket order. Once again, that's MGSPODCAST30 for 30% off of your order. We hope to see you there. I certainly hope to see you there and please enjoy the rest of our episode.
Peggy: And we are back with Matt Sadofsky, Senior Director of Growth Marketing at Tidal. And Matt, right before the break, we were talking about, you know, basically how you built the system, the timeline for it, the thought process that went into architecting it. Of course, you make it sound, you know, pretty simple but a lot of us are gonna have, yeah, trial and error, particularly if they take it...you know, listeners take it as their goal in 2019. "Yes, I am going to delve into AI. I'm gonna take the months of effort it takes to think it through and build it." As you say, I believe, three months to build it, was that it for one data scientist?
Matt: Two to three months.
Peggy: Two to three months, okay. So it's an investment of time. If they're lucky, I guess they can wrap up this year, right, in 2019?
Matt: Yeah.
Peggy: Fully possible?
Matt: Absolutely.
Peggy: Okay, so that is backdrop. Talk about your personal achievements and also personal failures in bringing AI to UA.
Matt: Yeah. Absolutely. I think...let's start on the positive side. I think on the achievement front, I think it was how quickly we were able to iterate once we got the first version out. You know, it went from us taking two months to build out the first version of it to, you know, if we were adding a new game to our system that had a different way of monetization and we needed to build out a new algorithm for i, we were able to pump out a new algorithm in two weeks rather than two months. So I think kind of automating the automation was kind of nice as well. That is just because our data scientist kinda got used to building out these algorithms and kinda have that central knowledge, which was great for kind of flexibility in the long term. But I think that was also our biggest challenge was sometimes we moved a little bit too fast. You know, we'd hook up the systems, we plugged them in, we'd start letting it optimize off of, you know, maybe not as much data as we would've liked and it didn't always work out, right. Sometimes the projections were off and we'd look back six months later and say, "Okay. well, for most of our games these were right but these were actually a little wrong here so what did we do wrong?" And we'd have to go back and do that analysis and I think that's just a challenge of looking at long term machine learning based projections is you'll never know if they're accurate until, you know, that projection date actually hits, right. A 12-month projection takes a year to validate.
Peggy: That was my next question actually is signs of success or failure. It's a little bit like UA overall, you know. If we don't...if we can't wait for that timeline, we can't wait necessarily for a year to validate then we look at, you know, indications of success. At least in events you say, "Oh, this is an event that indicates high LTV" and you go with it. You know, what are the events or what are the signs or signposts that you can be looking at if you can't wait a year?
Matt: Yeah. So for us, we typically benchmark things at the first day after acquisition. We looked at about 7 days in, 30 days in, 60, 90 and every 30 days after that to kinda just keep an eye on how things were trending. Sometimes the projections would be a little bit too ambitious early on, sometimes it would be not ambitious enough, and we would be able to see those trends start to appear, you know, at the 30-day mark and at the 60-day mark. And then that could let us know, "Okay, well, maybe our projections aren't right here." But the beautiful thing about AI is we don't need to proactively tell the system that it's wrong. It will start to pick up that the original projection is wrong and update their projections. So kind of a blessing and a curse that it's fully automated sometimes.
Peggy: What about the actual teaching, the learning with machine learning? I've been reading a lot of articles about this because there's this great section actually of the MIT Review that talks just about AI because of course they are building a billion-dollar school about it so of course they're gonna be delving into it just proving what a hot topic it is, and it's all about breaking things down into their component parts so you can actually teach machine learning, you know. They have to learn with you. Is that what you had to do? I mean, and do you have any thoughts on how you have to break things down to that level?
Matt: Yeah, I think our original version of this system was a more simplistic version of AI which is great for someone who just wants to dive into it. We used a nearest neighbor approach. So it would take a look at what the day zero, day one, day two lifetime value curve was of a specific cohort and compare that against similar cohorts and use that to project out. So it didn't really need anything else other than the daily cohorted revenue. But as we kinda dove deeper into it and we wanted to become a little bit more sophisticated, we would start using deep learning models that would take a look at 13 to 15 different factors per user. You know, what was their day 0 session length time, did they watch an ad on their first day, did they open the app more than 4 times in the first 24 hours? All these different metrics would kinda be fed into this deep learning model and that model would then project out what the different waiting of each one of those events was on the likelihood of that person eventually becoming a paying user.
So, you know, you can go from very basic version where it's just like looking at revenue to you're taking a look at a large set of data with very minute factors and using that to make a projection.
Peggy: What is...I mean, it's a lot of work. There's no escaping it. I mean, AI is here, machine learning is here. We're talking about it. It's definitely arrived full force. What's the uplift or what can I look at to make it worth the effort? Because it's gonna be time, it's definitely gonna be some money, I might have to hire some people on to do this. But, you know, we don't talk too much about the rewards. We talk about more effectiveness and we talk about, you know, stretching the ad spend. Maybe you can tell me from your experience what you saw as being sort of like the main achievements in making this worthwhile.
Matt: So I think the average agency in New York typically charges around a 10% fee of spend. The average tech company that you can use to automate your spend, even ones that have an AI based background, typically charge about 5% of spend. So my recommendation is, depending on what your spend level is, if you can build out an internal team that can make something that's custom to you and works well and it's gonna cost less than 5% of your projected spend for that year then it makes sense to try to build this out yourself. If you have a relatively low spend, let's say you're only spending $500,000 next year, it doesn't make sense to build out a $200,000 technology to spend $500,000. So that's where you could probably go with an off the shelf solution like Hidalgo.
Peggy: What about the range of apps? I mean, you were at a gaming app. Now you're at music. Is there a rule on the types of app marketers, the types of apps that should be looking at AI more seriously than others maybe in the year ahead?
Matt: I don't think so. And I think a lot of people have this misconception that, "Oh, well, I need to accumulate a lot of data before I can make good projections." But being now in...and originally in the dating app space, then in the gaming space and now in the music space, there is so much data that we get from users within their first session that a good machine learning model can start to digest and start making projections. And it's better to make some projection than no projection.
So, you know, I think no matter what space you're in, your users are telling you something from the moment that they sign up for your service and start figuring out what they're trying to tell you and use that to make informed decisions.
Peggy: And what is it about AI? You know, we talked about the challenges, we talked about the opportunities but AI in 2019 is gonna be different than it was in 2018 and in 2017 when you were a pioneer in this direction. I mean, if you could make recommendations for how to approach it this year differently than all the others or how to build it this year differently from all the others because maybe the tools and tech make it possible. What would that blueprint look like?
Matt: I think there's two factors that are super important here. I think when I originally kind of was working on the first version of DORA, it was a more basic version of AI. It was projections based off of just revenue data. And I think in 2018 going to 2019 and 2020, the revenue data is obviously still important for making lifetime value projections but user behavior data is so much more important because you might not have revenue or much revenue on day zero but you know things like how many sessions people had, how many songs did they listen to. You know all this data, right, for your app. But integrity of data is super important because if that data is not accurate, if you're missing 50% of the streaming data, if you're missing...you know, if your session length times are off then your projections are going to be wrong. So I think as we get into more complex models that are using more minute user behaviors rather than high-level indicators, having a really strong data engineering team, data warehouse and clean data is going to become more important than ever.
Peggy: So let's...you know, we're just entering into the new year but if you had to give app marketers a scorecard for how well they did in 2018 or maybe some idea of the, you know, the mistakes they're making, or we can be optimistic, we could say, you know, the achievements they have to their credit, just give me an idea of how you rate growth marketers, app marketers last year. That'll give us an idea of where they are going into this year.
Matt: I think app marketers in 2018 were probably at a B collectively.
Peggy: Okay.
Matt: And...
Peggy: That's not so bad though, Matt. It's not.
Matt: It's not so bad. I think that they...what they did really well and I think I can give them maybe a B plus or an A minus on was I think data. I think we made big leaps and bounds with understanding that growth marketing is not marketing. It's data-driven marketing. And I think we really saw people step into that in 2018. I think the shortfalls, or maybe the subscore would maybe be a C or a C plus, as I don't think people have figured out really yet how to make smart decisions when it comes to ad creative. I think that is something where technology can help us get to eventually but it's still that elusive thing that AI will never be able to fully automate for us and I think a lot of marketers are now so obsessed with data that they actually lost a little bit of the vision that they can have for coming up with new and exciting creative rather than just doing rips off of other people.
So I hope in 2019 I can move that up to a B plus or maybe an A minus for the growth marketing community. Keep that data-driven, you know, strive but make sure that you have a really good art team behind you as well because as everything becomes more automated, the biggest differentiator that we'll have against each other is ad creative.
Peggy: I mean, that is definitely the words to end this on and also a great...hopefully, what we'll be doing in other segments, you know. We're gonna have segments where we talk about creative and we're gonna eventually transition to what I'm looking forward to, it's gonna be having growth marketers as guests for us both and grilling the growth marketer is what I'm calling it. So in the meantime, Matt, you know, give us a better idea of where to find you because I know you're active in a lot of things.
Matt: Yeah. I've been stepping up my content game a little bit so I'm gonna be writing some new blog posts going into the new year. I'm gonna be moderating a growth marketing focused Slack channel with Liftoff Mobile Heroes and I'll be speaking at Mobile Growth Summit in San Francisco, MAU in Las Vegas. And I also just rolled out a new website for myself at matthewsadofsky.com where I post my marketing stuff but...and some blog posts but I also do voice acting, and I also post my favorite chocolate spots in New York City. So, some fun stuff...
Peggy: I like the voice acting and chocolate spots. See, that makes you very well rounded indeed and of course, you know, we'll be connecting at Mobile Growth Summit and great just to have you here of course on "Mobile Growth" to begin with and to continue for the rest of 2019. And listeners, thanks for tuning in to this episode of the "Mobile Growth" podcast. A quick reminder to visit mobilegrowthsummit.com for a complete list of our upcoming events and don't forget to use the very special promo code, MGSPODCAST30, for 30% off of your offer. We hope to see you there and we also encourage you to check out this and every episode of our series by going to mobilegrowthsummit.com and also on Sound Cloud and episodes coming soon to more channels, providing you more ways to listen in. So watch for that. We'll watch for you. I'll watch for you and we'll see you soon.
