how to: evaluate a listening vendor

With dozens of listening/social monitoring/engagement management vendors out there, finding the one that's right for you can be quite daunting.  Every vendor looks the same on the surface, at wildly varying prices and most claim the others have significant flaws that they do not.  Many also claim to have anything you ask for "coming soon" (but that's the same answer you heard last year!).  With this in mind, I thought I would put together an overview guide, outlining a few of the key components you should look for when conducting your own reviews of the space.

Who should conduct my vendor review?

Whoever is going to be leading the project activation should lead the review.  That said, serious input needs to come from your Product Leads, PR/Corp Comms, Marketing, Research and Analytics teams, not to mention your agencies and any relevant consultants.  Notice that I just listed pretty much everyone.  Whoever this poor person is leading this team is going to need to work closely with and across multiple teams.  Listening will not be effective in a silo.  This is real business.  If you don't have someone that is highly knowledgeable and fairly experienced, consider enlisting a consultant to assist.  If you would like a referral, please feel free to email me.

You may find that there are disparate needs across your teams that cannot be met by one vendor, in which case you will have to decide if you are willing to accept a two or three vendor solution.  Multiple vendors will make your teams more agile but will generally cost more in both vendor fees, not to mention the headache that is back-end integration across overlapping efforts - all of which will occur on your own time and dime.

Three Types of Listening Solutions

1. Self Service Tools - give you the tools to do all your own work.  These firms may offer consultative services along with the initial setup.
2. Full Reporting Services - provide regular reporting based on your needs.  Some of these vendors may not "own" their own tools, relying on "white labeling" Self Service platforms.
3. Reporting and Strategy - bring solutions to the table. These firms may also provide activation services.

Hint: Technology will not get you to the promised land.  Knowing your own business culture, if staffing or learning on-the-fly is going to be an issue, don't go with a Self Service tool.  Most Full Reporting and Reporting and Strategy vendors will have a live interface for your real-time needs should you desire a deeper dive.

Five Types of Capabilities

1. Real Time Data - often with minimal analytic depth, real time data provides the flexibility you need to stay current and in the now.  This will generally be most relevant for conversational engagement and crisis management.
2. Advanced Data Manipulation Tools - these tools include author analysis (age, sex, location), social graph analysis (who is connected to whom, engages with whom, how and what does this mean), meme flow analysis (how did the story spread), influencer identification and more.
3. Insight Generation Services - not everyone is great at conducting primary research that nets an actionable insight or brand platform.  But crunching all those numbers and overlaying human insights take quite a bit of time, investment and skill.
4. Conversation Management - between twitter, blogs, forums, facebook and more, there are millions of conversations taking place everyday.  Getting the right people responding to and participating in the right conversations at the right time requires both a well designed business infrastructure and the tools of the trade to manage the team and track progress.
5. Consultative Services - having access to data and even insights is not the same as knowing how to drive your business with this data and insights at hand.  Consultative services often include social business design, social communications training and management and social policy consulting.

Data Gathering

Everyone claims to gather "millions of sources" yet when you need that one community or site it doesn't seem to be in their database. 

Data sources typically include Twitter, public Facebook pages, profiles and updates, all public blogs and blog comments, all public forums and responses, a selection of video sites and the comments on those videos,  a selection of Q&A sites (like Yahoo Answers), a selection of consumer review sites (like Amazon and Yelp) and coverage of News media.  No vendor covers everything perfectly.  Also, if you require access to password protected sites, this may require a different set of technologies or services to gather and synthesize.  Some vendors are beginning to offer integration with Customer Contact Centers (call centers, email support transcripts etc), as well as integration with traditional web metrics packages for more traditional web metrics overlays.  Additionally, the more reporting-focused vendors offer some nice insights based on non-traditional tools such as Google Search Trends and Facebook's Media buying platform. 

*Updated* Global scalability will be key to any global business.  Global scalability not only requires that local sources (such as boards in China) are properly gathered, but that data scoring and analysis are available in the particular dialect you are gathering.  Translation engines just aren't accurate enough, yet.  Additionally, as conversational norms and social interactions will differ by culture, it's is imperative that analysts have a deep familiarity with the local language and culture.  Kudos to Mike from Visible Tech for pointing this out.  Just slipped my mind when writing this post originally.

Note that meaningful "big picture" analysis or benchmark analysis will require historical perspective.  While some vendors only begin to gather data after the contract has been signed, some vendors claim to have 2-3 years of historical data available, often for an additional fee.

Hint: Ask the vendor how they gather their data.  Many vendors license the same handful of feeds.  Most of these feeds are incomplete or not as comprehensive as they would appear to be.  Few vendors have access to the Twitter "firehose" which allows unlimited data access, so few will have full twitter coverage.  That said, no vendor will be perfect.  Find the compromise that works best for you.

Data Types, Coverage and Accuracy

There is no shortage of social metrics or analysis approaches.   Focus on what you believe will be most meaningful and actionable with an eye towards testing and learning as you go.  Don't forget to test and learn!

• Sentiment: Sentiment is the ability to determine if if the conversation is positive or negative, often represented on a five point scale.  Sentiment generally comes in two flavors: gross sentiment and topic level sentiment.  Gross sentiment is the total tone of the post or posts in aggregate.  Topic level sentiment is the sentiment around a specific item.  Topic level sentiment is a must have for any meaningful analysis or trending.  Unfortunately, while the technology behind affordable automated sentiment is often highly hyped, most solutions fall flat in the real world.  Also, a strong sentiment solution for an automotive client may not be as accurate when it comes to consumer packaged goods clients.  Look for a sentiment solution that includes a layer of machine learning based on human inputs, that understands grammar and syntax, and has a process for continued learning and optimization.  Most importantly, RUN YOUR OWN SENTIMENT BENCHMARKS FOR EACH VERTICAL.
• Trending: Pie charts are nice.  Histograms are pretty.  Challenge the vendor to explain how each of their trending tools enable insight generation.  Make them show you real examples.  Everything is groovy in theory, but the second you identify that this one chart isn't clickable, or that the only view of the data is a laundry list of millions of posts, you'll regret you choice in vendor.
• Meme Analysis: A meme is a social theme.  As many tools rely on keyword lists to create a theme, this analysis can become quite tedious.  There are a number of vendors offering automated theme identification, as well as theme trending and association.  These tools can make your life a lot easier further down the road should they prove accurate and reliable.  A popular usage of Meme Analysis would be Message Flow Analysis, tracking how a story spreads across the web.
• Search/Reputation Analysis: This analysis typically looks at high ranking items in your brand's search results, categorizes these results and provides trending data as it relates to these results.
• Author Analysis/Influencer ID: This is a fun one.  Influencers come in three flavors: General Influencer, On Topic Influencer and Network Influencer.  General Influencers demonstrate influence overall.  On Topic Influencers demonstrate influence as it relates to your specific topic.  Network Influencers demonstrate influence across their node within a network, but not necessarily across a broader network due to strong internal overlap within the particular network.  Typical influencer indicators/metrics include inbound links (total, recent, on topic), comments/responses (total, recent, on topic), followers/subscribers/readers/hits/network reach, and shares/likes/embeds/retweets/pass-along.  Some influencer tools will give you more flexibility and control than others, but the technology won't land the ship without some strong human interaction.  A few vendors have introduced systems that aim to deliver "single ID" systems whereby an influencer's multiple social accounts across different platforms are aggregated into a single ID.

Other types of author analysis include age/sex/location data.  This data is rarely available at scale.  Most vendors offering this data rely on weak solutions, such as IP address tracking (this is too accurate because most conversational media is hosted by a third party).  Other vendors will approach the a/s/l data set by starting with a defined audience based on third party data or site metrics.  This approach has it flaws as well.  At the end of the day this one will come down to (a) how important this data is to you, and (b) how accurate you really need it to be.  Manual tagging is always an option, but it will rarely scale.  Manual identification is general reserved for Focused and Diary analysis (see below).

• Integrated Analysis:  Your brand already invests in research.  You already have more than a few KPIs, brand health and Marketing Mix Models in your back pocket.  A good study starts by looking at what you already know and ends by integrating with your business solutions and metrics.
• 30K Foot Analysis: This approach looks at your data in aggregate, highlighting broad trends.  For this analysis, it's all about scale.
• Focused Analysis: This approach doesn't look to analyze every single post.  Rather, Focused Analysis dives very deep on a select group, delivering far more insight but generally at a lesser scale.
• Diary Analysis: This approach highlights a smaller group of individuals and tracks the individuals rather than the brand.  Insights are often related to their lifestyles and emotional drives rather than brand conversations.
• Data Ownership:  You may think this is a minor point, but when you need to transition to a new vendor and suddenly lose access to the past year's work and scored data, trust me, you will wish you had this sorted out in advance.  Make sure you own all your own data and that all of your data is exportable as a CSV.

Flexibility/Data Interface

• Interface Usability and Flexibility:  You will be spending a lot of time with the tool.  Data presentation and interface usability will save or cost you hours of frustration.  Get to know the tools, where they overlap and where various systems may not yet come together.  Spend a good deal of time with it.  Learn it's quirks before you sign on with the vendor.
• Data Flexibility: Is ALL of the data exportable?  Is there a meaningful/usable/accessible API?  What strategic partnerships does the vendor offer?  How do these partnerships align with your existing relationships?  Consider the likelihood of your company to invest in building cross-platforms integrations before falling in love with vendor APIs.  Another point to consider would be real-time alerts.  Alerts can be emails, text messages, etc.  While many alerts solutions simply send a notification every time your brand is mentioned, some solutions are beginning to explore smarter alert solutions.  In theory, these solutions would only alert you when a certain threshhold has been passed, such as "A 300% increase in conversation about a given topic over a given period of time with strong sentiment displayed."  This will be a great tool to have in your toolkit.
• Pricing Flexibility:  How easy is it to add or detract services?  To try new services?  To scale up or down your scope?  It's hard to predict what the next year will bring, but it would be a shame to shut the door to Listening because of a lack of foresight.

Dirty Laundry

Everyone has ghosts in their closet.  No vendor is going to do everything you could possibly need perfectly.  Keep a list of the various trade-offs and weaknesses.  Keep a list of features "coming soon" and ask for hard dates for delivery that can be baked into a contract.  Ask about what will be "beta" and how reliable/stable they expect this beta to be.  Feel free to bake these conversations into a contract as well.  This will help you provide the appropriate caveats down the road as you scale your engagement. 

Conversation Management/CRM

There are times when you will want to yell at the screen because you're just bursting with the urge to correct a misperception or help a wronged customer.  Eventually, listening often leads to conversational participation.  Between marketing communications, corporate communications, customer service, events marketing, promotions, community managers, Twitter and Facebook engagement, blog commenting and forum participation, keep tabs on who is handling what and where you stand with each conversation can be a pain.  Conversation Management platforms simplify this process by allowing teams to collaborate, assign posts to one another, keep a track record of interactions around a topic or with a user, and more.  While CRM and Influencer Outreach Databases can also be managed from within a Conversation Management platform or in concert with a Conversation Management platform, stand-alone solutions are also available.

Strategic Consulting

As I mentioned above, Strategic Consulting is nothing to shrug off.  Social challenges the internal and external communications dynamics and there are often far too many chefs in the kitchen to stay productive.  Many people will claim to be experts.  Few really are.  Ask them for references.  Ask them for proven success.  Challenge them.  Do not accept a book published or a massive twitter audience as a criteria for proving success.  Personal success in talking about business is not a qualification when it comes to making business happen.

What metrics will matter to me?

There are more social scores than you can shake a stick at.  Focus on your business.  What does your business need today and where do you forecast they will be tomorrow?  Challenge vendors to demonstrate how they meet your needs.  Do not accept oversimplification or over-complication. 

How much time will this take?

Once you have a bit of experience, you can generally feel out the vendor in 30-45 minutes.  However, really digging deep into their solutions will take time.  Once you have narrowed down the field to a couple of finalists, get into the tools yourself for at least 3-4 hours. Get your teams into the tools. 

Closing Thoughts

Rarely will a single solution meet your every need at a level of coverage and accuracy you desire.  Lay your trade-offs on the table and set caveats appropriately.  It's generally better to make an informed decision than a hasty one.  If you have an urgent need, look for someone with a good deal of experience in the space to advise. 

Lastly, focus on timely actionable insights.  Benchmarks are nice, but social is more than a number.  Get to know the people you are studying, get to know their conversations and lives and you will find unbelievable worlds of insight.