Andrew Cohen, General Manager at Bit.ly, told me today that the company watches the huge number of links passed through its service and drills down to index their text content. Using the Apache Lucene-based technology developed at LinkedIn called Zoie, Bit.ly determines what the most relevant parts of every page are, excluding footers, spammy comments and other cruft. The company then checks those parts of the page for customer-entered keywords and determines the relevance of the article shared.
That’s non-trivial work and in at least one case I found a URL deemed relevant to a keyword when that keyword didn’t show up in the text – it was in the HTML keywords for the page only. I thought that was a stretch, myself.
Once a page is deemed relevant to a customer keyword then its likeliness to go big is assesed.
Below: Reputation monitoring for Michele Bachmann, click to view full size.
Cohen told me about 3 factors taken into consideration in making that evaluation of likeliness for content to spread far and fast.
The first is the pace of sharing. If a link is being shared faster and faster as time proceeds, that’s significant.
The second factor is whether it is being shared outside of groups of people who already share things online together on a regular basis. Cohen said that if a link was passed around the Bit.ly office, that would register as different from something that was shared by the same number of people but in different places and on different social networks.
The third factor is whether the link is being shared by key influencers online.
It’s pretty remarkable to think that when you share a Bit.ly link online, the company knows who you are, where you are and what your relationship with other people sharing it has been. It’s nothing to fear – it’s interesting – but it is notable.
Hot links of the future don’t really come up very often. Cohen says it’s like a smoke alarm – it’s rare that it goes off but you don’t want to miss it when it does.
Indeed, the links shown in one reputation tracking results page for one huge brand I looked at (Barack Obama – which was linked to on Twitter by John Borthwick of Betaworks) had only 2 out of 10 links listed that were published within the last 48 hours. I would be curious to see how often the service predicts spikes in attention for smaller brands.
The reputation monitor also tries to perform some sentiment analysis, using third party technology from Lexalytics. Sentiment analysis is very hard to do though and at least one example I looked at was clearly wrong. Many people believe that sentiment analysis is most valuable when performed on aggregate content, over time. That’s not how the Bit.ly interface uses sentiment, though.
The arrows up and down that appear to signify the sentiment of a particular article remind me of the semantic keyword extraction that Bit.ly used, also via a 3rd party technology, when the link shortener launched. That feature proved ineffective and inefficient and has since been dropped from the public interface. The company has been interested from the start in building a platform for software on top of traffic data plus content analysis.
Bit.ly’s new reputation tracking service is being rolled out to Enterprise customers starting now. It is the first of a number of products the company will build on top of its indexing and tracking of content shared around the social web.
Article source: RRW http://feedproxy.google.com/~r/readwriteweb/~3/I6vOW8_iVUk/how_bitly_now_predicts_the_future.php