Include search results from trusted sites

[AdiChat]

I'm submitting a:

• bug report
• feature request

Description:

Include results from other trusted sites for a search term in addition to our current results.

To begin with, we may consider the following trusted sites:

• Wikipedia
• Tutorialspoint
• Stackoverflow

To fetch results, we may use create a crawler to crawl trusted sites or scrape the data from a search engine.

You may fetch a few of the top results from a search engine like Google. For a specific site, use search terms like linear search site:wikipedia.org. The number of results should vary for various sites. For instance, we can provide 1 Wikipedia result, 2 Tutorialspoint result and 4 StackOverflow results. This approach will have limitations so, we can cache results to avoid unnecessary scraping.

The ideal cached results may include the url and the metadata to enable further searches. Implementing this will include a scraping code to present results to users during searching.

Caching can be taken care of in a separate feature. For caching, a separate script can be used to collect results for a set of keywords and store it in a database. Additionally, searching should include our cached results as well.

This feature will make Cosmos-Search more usable in general and help us provide a complete learning experience. 👍

[Shweta4321]

@AdiChat I would like to work on this.

[AdiChat]

Sure @Shweta4321 Go for it 👍

Keep us updated on your progress.

[sarvanideekshitula]

@AdiChat Please review the Pull Request

[soc221b]

@AdiChat In my opinion, we need to implement this with Ajax and lazy loading tech or we could code with client side (i.e., use JS), because:

1. Efficiency. Engine are always plan that give precision results which users want, so we should just load recommendations from trust sites as top as need.
2. Slow down a bit. In some scenario, Users aren't always need all links, and he/she might possibly type wrong string for query.

[AdiChat]

Yes, this will be nice. 👍

It will reduce the load on the server and serve the initial results faster and thus, improving the user experience. 👍