Hi again, it has been way to long since I updated this blog and has been for good reason. As a few of you know I have been heads down developing a system to data mine keyword and social media data. Yes, I know the subject name sucks – any suggestions? What seemed like and easy thing to do had turned into a total beast and development feat. I am here to day that we have accomplished most of the phase 1 specification and are now ready to realize as a commercial product. We we will add features in 3 phases which I will talk about in more details later.
More information is at http://www.back-azimuth.com – if your interested in it ping me and we can do a demo and maybe make you a beta client.
For now the application does the following:
Aggregates all of your keyword related data into a single repository stored on the cloud. Then allows you to quickly identify critical issues and missed opportunities.
Why did I do this? Over my years in search marketing one of the biggest challenges have been to manage keyword data. At IBM I built a keyword de-duping application just to find and manage duplicates between 30 business units doing paid search.
Only until we had this tool did we know that 21 business units were all bidding on the same keyword phrases. Last year working with a large multinational in Europe I asked them for their keywords and it took 8 days and I received 21 different Excel worksheets from the Search manager that represented their words. In another case, a multinational with thousands of products only had selected 25
keywords for any optimization efforts meaning tens of thousands were going without any efforts. All of these led to my desire to create a tool that would eliminate problems.
Over the past few months I have conducted a few survey’s of companies and from this research I have identified a number of keyword management challenges.
Keyword Management Challenges:
- Companies struggle to manage keywords in Excel (99.8% of companies surveyed)
- No lens into collaboration of Paid and Organic data (99.87% of companies surveyed)
- No way to see where keywords are under-performing against KPI’s (98.75% of companies surveyed)
- No way to monitor if the “right†page is ranking (99.25% of companies surveyed)
- No way to understand keyword performance by category or buy cycle (99.98% of companies surveyed)
- No way to understand performance based on the Searcher’s Intent (99.98% of companies surveyed)
- No way to leverage searcher interest data to prioritize content in the organization (100% of companies surveyed)
If you want to add to the survey simply go and add your responses to the Keyword Management Current State Survey.
What can we do today?
1. Aggregate all of your keyword data into a single searchable repository with role-based login access by different roles in the company.
2. Conduct paid and organic co-optimization analysis – are they cannibalizing or complimenting each other.
3. Preferred Landing Page Analysis – is the page you “want to rank” the one that is ranking?
4. Rank Analysis – same as everyone else but we allow you to sort by priority words, line of business and any keyword cluster or classification
5. HitWise Integration – if you have a HitWise account we can pull in the API feed and compare HitWise trends to your actual data
6. Data mine and Report on any of 55 different keyword variables
7. Develop Searcher Intent and Persona Segmentation – using any of 50 performance or segmentation factors cluster keywords into logical segments
8. Store, sort and report on data across business units, lines of business or countries
9. Understand performance by keyword length, position and paid and organic assists
10. ROI Modeling based on multiple variables
What has been the outcome?
In my first generation I used Microsoft Access and Excel Pivot Tables to test the theory then moved into MVP “Minimum Viable Product” mode to quickly develop functions that allowed us to scale data resulting in the following success stories:
Success Story 1: UK travel site matched keywords to top ranking pages found less than optimal pages ranking – fixing just 5 pages resulted in $60k incremental revenue in 25 days.
Success Story 2: PC Maker found significant searches but no traffic for “End of Life†products that they had no web page representation for – They added new content and PPC campaigns generating $400k
Success Story 3: Fortune 50 company realigned keyword ownership and budgets based on segmentation analysis for maximum opportunity resulting in a decrease in PPC spend of 12% but 300% increase in sales
Success Story 4: Travel site identified 50 keywords in top 5 position with less than 5% share of clicks – optimized snippets increasing click rates from 5% to 15% resulting in 85% increase in revenue
Where are we at today?
This is not a mainstream consumer site product. It is designed for a site with a large base of keywords typically more than 500,000 of them that want to get more out of the product. We are looking to develop a mainstream version of the application but seems those customers want something cheap, that does a lot of automated analysis and does not require them to think.
We are not quite thee yet and when we are we will roll that version out.
What is is not?
While the tool does a lot, it does not or will not do any of the following:
1. A bid management tool – there are plenty of them that will work great for you
2. A SEO Automation tool – there are plenty of them that will work great for you
3. A Search Analytic tool – sort of but does not replace Omniture, Google or Web Trends
4. Keyword Research tool – sort of since we can mine data but we typically are not looking for new words. Great tools like Keyword Discovery, Wordstream exist for this function.
The challenges:
Where do I start with this one – this project has taken every ounce of patience and sanity I could muster to not abort it along the way. Fortunately, I have been working with a great team at OC4 on managing this in the cloud. We have nailed most of the big issues and are just fine tuning.
Data Integration – The biggest challenge was integrating and managing the data. Readers, there is some messed up data out there. I found that there are a lot of agencies and people that should be fired for incompetence if not fraud. This was the biggest challenge of integrating the data. There are large volumes of it and we needed to suck them in and align them.
Data Clean Up – people use some crazy words to find your products and there is a lot of bot activity. We has to write a data import and cleaner tool just to parse log traffic data. Omniture does a decent job of sorting these into “Small Elements” but other tools do not. We had to develop a routine to process 286 different types of data contamination before we could import CSV files into the system. We had whole paragraphs coming in, multiple commas, and ton of scrapping strings looking for pricing or other elements that all had t be cleaned out. Not to mention beginning and trailing white space and other issues just to normalize the data. In one case there were 854 different misspellings of the company name which we leave in the tool but don’t necessary want to gather other data for.
Product Naming – Another big challenge is a name for it – My original concept was to call it “VOCDMS” – Voice of the Consumer Data Management System – but clearly that does not roll of the tongue so we are working on a better naming. We have thought to keep it aligned to Back Azimuth since that is still the foundation of what we believe – helping you get back to your consumer.
Development Teams – I am now on the 3rd iteration of a development team. The current team from Exadel are awesome. They have fixed many of the bugs and problems from the previous India teams. I have tried local developers but they were way to expensive, too distracted or wanted a large share of the company while only offering mediocre skills.
Market Interest – This one has been strange – when I have told people about the tool they are skeptical assuming there is already one like it then realize or ask their teams to find they are like the rest of the companies using Excel or maybe a in-house database to manage it.
Whats next?
We have a few clients and pilots going now and will start to market the product to a wider audience by the end of the year. If you have an wish list items send them my way.