Since 1996, Alexa has been crawling and storing the Web at millions of pages per day. Alexa has also been building out the infrastructure to store and analyze the data and serve it to toolbars, browsers, and websites worldwide. Now, all of that infrastructure is yours to use via the Alexa Web Search Platform:
- Three online web snapshots of up to 100 terabytes each
- Powerful tools to sift through the content to create your own data set
- Upload, compile and run your own programs on a processing cluster across the data set
- Store your output on a storage cluster
- Integrate your data into a search index
- Access your new search via Amazon Web Services
The Alexa Web Search Platform will unleash the best minds in search everywhere.
Imagine: you have the next great idea for search... What do you need to get off the ground? The first and perhaps most difficult and expensive piece of the puzzle is a good Web crawl. Your crawl will need to access a high-speed internet connection, it will need to pull down thousands of pages per second. It will need to manage thousands of connections and process pages to extract links. Then you will need to store hundreds of terabytes of data. And that doesn't even begin to tackle the rest of the equation: processing the documents, indexing, storing the index, serving it, and keeping it updated. You are going to need millions of dollars in technology and staff and at least a year to get things rolling.
But what if all that infrastructure was publicly available and all you needed to do was get access and start running your code?
Just imagine all the talented entrepreneurs who have been stymied by a lack of web-scale tools and data. Now, for less than the cost of an iPod, they can get into the search field and begin inventing and creating. You as a consumer can begin using these new search services, and all of us can begin reaping the benefits of an expanded search space with hundreds of wholly new search services being created by anybody with an idea and a credit card .
We're looking forward to seeing what exciting applications developers surprise us with. Link.
