Tech Radar PI, the Web Crawler for Poste Italiane
Our Web Crawler to scan the news in the IT world
Identifying news in the field of Information Technology by developing a Web Crawler. With the Tech Radar project, Spindox Labs met Poste Italiane’s need to explore the Web in search of up-to-date information on technologies in the IT world. We proceeded with the analysis of online editorial content through Augmented Intelligence tools. The indexing of materials was done by means of classification techniques developed in the NL. The information, presented by means of graphical interfaces, reflected a broad coverage of data sources.
A Web Crawler for personalised, information-analysis systems in the IT area
Project name:
Tech Radar PI
Duration:
11 month
Years:
2019-2020
Technology:
Augmented Intelligence
Our solution
Two solutions were identified to reorganise and present information:
- A news page, indexed by their thematic area and filtered by their interest score, based on the engagement rate recorder on social media. Just like a specialised newspaper, the news page offers titles and summaries, selected from the search criteria set by the user.
- A technological radar for mapping, where the different slices represent the thematic areas (NL, IOT, Big Data, Augmented Intelligence) and the coloured pattern refer to the development of the different technologies (library, framework, language). The distance from the centre of the radar indicates the maturity level of the technologies. information is validated by an analyst operator which, thr
Advantages
- In order to develop our Web Crawler, multiple data sources were selected to adhere to multiple standards of reliability, accessibility and completeness through a unified solution.
- StackOverflow is unanimously relied on by developers for the extraction of information concerning technologies and their adoption trend in the IT panorama.
- StackShare is a search engine to explore technological stacks in the major global actors.
- Feedly is a news collector that allows the subscription to personalised feeds according to specific thematical areas. This gives referenced feedback on the interest levels regarding every news.
