Machine Learning in Product: Swayable

As part of the series Machine Learning in Product, Swayable CTO and co-founder, Valerie Coffman, discusses the business wins and challenges of using machine learning in customer-facing products. Swayable is a data science platform focused on understanding public opinion. Their core product is pre-testing content for its persuasive potential.

Below is the transcript of the conversation:

Valerie. Hello.

Andrew. Hello. How's it going?

Valerie. Good. How are you?

Andrew. Doing well. I wanted to start kind of by first learning a little bit more about Swayable so we have a little bit of context there and then just kind of dive into maybe some interesting ways that you all are using machine learning and you know interesting ways

Valerie. Yeah.

Andrew. In the future. So I guess let's start with that just maybe tell me a little bit about swayable and what you're working on now.

Valerie. Yeah, so swayable is a data science platform focused on understanding public opinion and our core product is content testing, pre-testing of content for its persuasive potential.

Andrew. Sorry I was just gonna and for the end goal both of campaigns and consumer you know products is that right

Valerie. That's right and some B2B

Andrew. Yeah very neat. And so what kind of content are you testing and sort of what's your mechanism for testing that content.

Valerie. Yeah, so we test mostly video messages and advertisements. They can also be static images and we've done a few audio, audio-only, and our mechanism is to do a randomized control trial survey experiment. Yeah.

Andrew. How did you come to focus in this area like what's your, what drove you to become passionate about public opinion and consumer insights?

Valerie. Yeah, well it started, we started coming up with ideas, the year was 2017, and the broad problem we were looking to solve is why do people believe things that are not true and not believe things that are true. You know like climate change is a problem that was sort of one of the original topics we worked on that was the first topic we worked on was climate change. And so we explored lots of ideas on what we converged on was testing ads just like how drugs are tested really it's a treatment.

Andrew. And have you found, I mean I'm sure you found a lot of interesting things campaign by the campaign you know both for consumer products as well as for big kind of public policy issues like the environment but any sort of big, after now working on it for a few years, any sort of I don't know big takeaways about what it what you know what to expect when trying to apply a sort of how you might test drugs methodology to how we test opinion.

Valerie. Yeah, so one thing we found is that a certain proportion roughly 25% and depends on what time of what time you are where you are in the timeline related to an election but roughly 25% of advocacy-related ads can cause backlash actually which is why one of the reasons it's very important to test. I haven't I haven't really seen anyone who's good at guessing what the outcomes are people might think they're good at guessing it there's a whole series called MadMen where they stand around and argue over it right but when you test it rigorously sometimes the results can be surprising there can be backlash. Closer to the election it gets harder and harder to change people's opinions. We can see that in our data and it's consistent with external research.

Andrew. Very interesting. You know obviously, this is a very data-driven approach. Can you talk a little bit about how machine learning has played a role in the offering that your company has made to your clients?

Valerie. Yeah, absolutely. So for every metric that we're measuring, so every sort of KPI and a series of supporting metrics, a test can have anywhere from three to thirty metrics or even more in some cases. We train a model in the cloud on the fly that's custom to that test, so if you're testing, you know a few pieces of content and you have, you know, 10 metrics we're training 10 models - one per metric - the inputs to those models are you know all the questions that are pre-treatment so demographics, psychographic, things that we pull in based on your zip code, all go into that model and then we use that to make the most, get the most statistical power possible out of the data we collect through the survey.

Andrew. And what does the end result for the user look like? What do they what are they presented with?

Valerie. Yeah, so right now that the end result is our results page where you can see you can explore all of the output of this test and the statistics that are calculated from the model. I also should mention we have we also have some NLP models that we're using because we also ask qualitative questions usually after the test sometimes it's before the test it depends on the goal but we're doing some clustering to discover the themes that are in the free-form text answers people leave as well as tracking sort of sentiment and overall sort of weightings of the most statistically improbable phrases associated with each piece of content in each segment.

Andrew. Very cool I've heard a lot of people talk recently in the area of machine learning and AI about how you communicate confidence how you communicate doubt, how do you all, you know to handle that on that result's page. Are you kind of giving people some sort of spectrum or are you kind of giving people a confidence interval or how are you kind of contextualizing whatever it is that they're reading.

Valerie. Yeah, it's a great question. So both James, CEO, and co-founder, and I are our scientists that's our background. We know each other from physics grad school and so we decided very early on that you need confidence intervals even if you think you don't, you're getting confidence intervals. So every result every numerical result has a confidence interval and we also have confidence levels that can be chosen right now the default is 80 but we can adjust that to you know 90 or 95. We also, if a result, if an RCT result, where we're comparing the baseline to the test group, some metrics we don't show that but if we're comparing a baseline to a test group, you know we're really doing a hypothesis test and if the confidence is below that threshold we gray it out in the dashboard so it's colored in if it's if the if it's a confident result and it's grayed out if it's not.  And we always show those bars as well.

Andrew. And what sort of challenges I mean you and your co-founder obviously as you mentioned are scientists and so come at this with, I'm sure a lot of preparation and insight that many business leaders wish they had when they started to implement things like machine learning but nevertheless what challenges have you faced in trying to make the most use out of machine learning within your product and for your customers.

Valerie. Yeah, that's a great question. So building a system, a very flexible system that trains the model in the cloud, on the fly takes bringing together a lot of, I call it the science and the engineering, right, there's a lot of people who might know the science and then there are people who know the engineering, not many people really can you know do both at the same time so it was truly a collaborative effort to you know to write this robust flexible code that can take this data set, train all of these models in the cloud, on the fly. In parallel, you know we have this distributed parallel computing system. Now setting that up has been and keeping it running as we're doing, as our business is growing, and the tests are getting more complicated has been it's an exciting challenge.

Andrew. And I'm curious to hear more about how you've approached that challenge, so has it been that you have had to develop and more engineering competency in your data scientists or more data science competency in your engineers? Is it that you've just found ways to like you know almost project manage that gap and you know let people stay more or less in their own lanes or some other approach? How are you, because I've heard, I've heard challenges like that in the past.

Valerie. We, I mean, really to build out the machine learning component we had a series of working sessions. I mean I think that's the best way to do it you can't communicate the stuff in you know pivotal tracker tickets or Jira tickets, you really need to just get together, a lot of it was done over zoom really so or in person if possible, that hasn't been possible for the last you know year and a half but just having a regular session where we go through the next chunk of code that we're going to write and we make sure that we bring together that the architecture, architectural thinking, software engineering thinking, with the data science, so that we can meet both challenges at the same time.

Andrew. And it sounds like you mentioned this is something that is ongoing which makes sense because the business continues to grow. I'm sure the challenge continues to expand and so just something that I'm sure you continue to have to kind of invest in.

Valerie. Yeah, absolutely. We're doing a lot of pair programming and group mob programming.

Andrew. Very neat. And from your customer's perspective maybe even going back a bit to the confidence interval question. Any challenges there or you know in making sure that they kind of understand the what you're providing them. I mean to do, how I guess are they interpreting what you're providing them and do they see it as machine learning is that's an attractive sort of offering that is part of you know your value proposition that you pitch to them or is that kind of just background technology that you know kind of they're not super clued into.

Valerie. No, they definitely are experiencing some benefits of it so we had a really interesting use case a few months ago with a major brand that has, they have both cut consumer-facing and B2B facing products, and they were interested in testing an ad focused on the B2B. Now it's tricky to target that audience, it's small, so business decision-makers but with our model, we were able to survey a general audience and then create a filter for the audience they're interested in so a filter is really just a segment that's cross-tabbed with all the other breakdowns that we're looking at how it was a small sample in that filter but the advantage of the machine learning approach is that you're maximizing the statistical power for every sub-segment borrowing statistical power from the whole sample and what we were able to do is on the side in our staging environment we reanalyzed it as if we weren't using the model using a frequencies approach and we found that the confidence intervals were 10x smaller using the model so it was a really interesting case, so this is a test that would be outrageously expensive if you weren't using machine learning, outrageously expensive to you know acquire data from this very very tiny segment.

Andrew. Very interesting and I'm curious from like an almost market demand side of things do you envision then customers in the future kind of seeking out like hey I need a machine learning solution to my market testing because I'm trying to you know because I can't you know there's no other way to do it economically like do you invent because that sounds to me so attractive that I imagine that it's we're at the cutting edge of it but it feels like that's the kind of thing that will be sought after pretty explicitly in the not just future if it's not already.

Valerie. Absolutely I mean if you're not using a model to analyze this kind of test survey, the traps that you fall into are, when you start looking at small segments you end up with you know if you're just taking an average and you've got 10 people you know one person can throw the whole thing off right so you see you end up with a lot of red herrings a lot of results that aren't real so the model corrects for that. The other is that teeny tiny imbalances even if you have robust treatment group assignment randomization you'll get just the slightest imbalance you know a few more conservative people were in that treatment group will impact the results and the model corrects for that as well.

Andrew. That's very cool and I imagine I imagine we'll just continue to hear a lot more about this because it's I think no secret to anyone because it's both in sort of it's like in everyday newspapers as well as all sorts of specific verticals and industries that you know at least you hear it's it seems like traditional methods of trying to gather public opinion are just getting more and more difficult and so it's very interesting to hear that machine learning of all things is helping to kind of combat maybe some of that.

Valerie. Yeah, yeah absolutely.

Andrew. Last question here for you, is just about the future you know I'm so curious to hear what you're most excited about for Swayable like what what do you see as maybe the next frontier when it comes to leveraging machine learning to improve the growth of the business and the customers that you serve.

Valerie. Yeah, so some of the future, I think is continuing to iterate on which questions we ask on the survey. Some questions are more predictive than others and the more predictive they are and the better your results can be, so building a feedback loop where we can look at every test and see okay what was the, what was the best question what were the most helpful questions for this test and how can we use that information to continue to improve results in the future.

Andrew. Very neat, and I guess maybe a brief follow-up to that so you all as you're continuing to you know to develop these new tests these new surveys you're able to draw on sort of the performance and the predictive power of those questions, for you know previous iterations of that question even in different contexts so I guess that's also a pretty interesting component. Very cool.

Well, hey I really appreciate you taking the time it's been super interesting to learn a little bit more about swayable and the way that you all are using data and machine learning clearly very ahead of the curve and I'm very excited to stay in touch and see how things go in the future thank you.

Valerie. thanks for inviting me to to to share our story 

Andrew. Absolutely, well good luck and have a great rest of your day.

Valerie.  Thank you you too.

Andrew. All right take care bye