Having written a number of simple scripts that aggregate information that can be found on various Internet sites I have a feeling that I am doing some repetitive, useless and low-level work. It is like writing a classic “Hello, world!” program in machine codes. Clearly there is something missing here, something that could have helped me be more productive in formulating my questions and getting results.
I think this is the work that should be done by search engines and the work that they do not yet do well. Search engines should be able to aggregate, filter and enrich data to produce the information that I am currently looking for and present it to me in the most convenient form possible. They should not just be indexing content which is available on the Internet but they should process this content so that I can get to the information I need. The value that Internet provides lies in the useful information being transmitted, not in the variety of documents, formats, contradicting facts, false statements, etc. If it is possible the search engine should sift through the information and return only the information that I should be interested in (based on my browsing history, personalized settings, location) generating the needed representation of the information if it is not readily available on the Internet in the convenient form I need it in. Of course here comes into play the problem of the ownership of information and derivative works that are allowed without violating the rights of copyright holder.
How great would it be if I type “software development skills in demand” and in response I get the chart I am interested in and probably even some advertisements that suggest that I learn the most popular technology right now, for example, Android?