LOOQME (LookSMI)
Online and offline media monitoring system
Country: UkraineTechnologies: Django, Python, PostgreSQL, Mongodb, Angular, Solr, SassTime together: 5 years Team: PM, BA, 2 developers, QAOVERVIEW
LOOQME (LookSMI) is an advanced automated monitoring and analytics system. On a 24/7 basis, it provides users with up-to-date information about their public reputation and the overall situation on the market. With LOOQME (LookSMI), companies can assess the quality of their PR campaigns, quickly discover the negative or adverse information (for example, during a crisis), run competitor intelligence, and follow the latest industry news. This system accommodates information from both traditional (newspapers, magazines, TV, radio) and digital (the Internet and social networks) media.
Apart from data collection and monitoring, the main LOOQME (LookSMI) gimmick is a convenient content visualization and analysis provided for its users.
At the time when we worked with the company, it was called LookSMI. Later, the company conducted greater reorganization and rebranding and continued its work on the market under the new name LOOQME. And it has become the best media monitoring software in its market.
PURPOSE AND NEED
The problem with media monitoring these days is the colossal volume of data originating from a huge amount of diverse media sources. Before, one had to manually collect different data streams, compile them using different media monitoring software, and then send them to the client one or two times a day. However, today information disseminating happens very fast, and if you care about your reputation, you must keep track of what's going on in real time.
Thus, LOOQME turned to Gearheart, setting the task to create a universal offline and online media monitoring platform to collect and analyze information from all available open sources. Our task was:
- to develop a system that would be capable of unifying all media types in one format;
- arrange information uploading from over 10,000 various media sources into one database;
- create a straightforward and user-friendly interface for the clients;
- automate report compiling.
The client's requirement was to accomplish these tasks and launch the system prototype as soon as within one month.
SOLUTION
To speed up the system’s release to the market, Gearheart offered a solution that prioritized creating the interface for the user's end. Back-end development and system fine-tuning were attributed to the second phase.
Under these circumstances, LOOQME would at first fill the system manually, but the information on the user's side would be presented via a convenient interface. This approach would allow the company to put its idea to the test right away and see how it worked out.
TECHNOLOGY
We describe in detail the technical side further on certain features. In the meantime, we will list the technologies used: Angular, Django, Python, MongoDB, PostgreSQL, Solr, and Sass.
STRUCTURE AND FEATURES
DESIGN
To save time and budget, we offered LOOQME several ready-to-use design templates with a broad selection of ready-made visual modules. The client embraced the idea and chose the template that fully complied with his likings and requirements. Several visual themes are available to choose from.
For data visualization, we used ECharts.
Of course, today the interface of the system is optimized for mobile.
TECHNICAL SIDE
At first, we tested Elastic Search to deal with search engine indexes. But we quickly saw that its query language was not broad enough for the task. So we moved to Solr, which has been used in the system ever since.
The main technical problem of the project was the search index load, as the LOOQME system is all about simultaneously reading and writing the vast volumes of data. It's simple enough to create a system that doesn't load servers too much by focusing either on writing or on reading. But creating a system that wouldn't bring servers down while performing intensive concurrent writing and reading operations is a much more complicated task.
Nevertheless, after bringing Solr into the system, thoroughly exploring the problem, and fine-tuning the algorithms, we managed to find the right balance and walk the thin line between reading and writing, speed, and load. Thanks to this, now LOOQME knows exactly when they can get along by optimizing configurations and when they need to add servers.
We finished up the first development stage by creating the user interface – a dynamic single-page application on Angular. It requires sending less data and thus reduces server load.
In the second stage, we automated the mechanism of filling the system with data by developing a unique programmatic solution. Our software constantly processes thousands of websites and captures new articles within just 15 minutes after they are published.
Eventually, to make the system work quickly and consistently, we created two databases:
- The first database was meant to store all data from all media. We used MongoDB for that; it's ideal when you need to store huge volumes of data without doing something with it frequently.
- The second database was intended to work with the users' portals and let users edit data. For that, we used PostgreSQL. The data that the client needs are imported to this database from Mongo – and then the user can work with it. He can also download the data/reports in Excel, PDF, HTML, or ZIP.
Finally, we created a custom on-site search engine based on Scrapy framework. It can handle complex search queries, taking into consideration multiple sophisticated rules.
As a result, we developed a stable, secure, and fast system that can process huge volumes of data for many concurrent users. In total, creating the working prototype that could be sold to clients took one month – as was requested by LOOQME.
During the support period, we made four system versions with redesigns and dozens of new features.
Today, LOOQME is the leader of the monitoring market in Ukraine. The system is used by the majority of large and medium-sized companies from absolutely different business areas. Now, the internal team of LOOQME developers successfully continues to work on supporting, improving, and developing the system.
RESULTS
- We created real time media monitoring system that can process data from both online and offline sources. The system worked with over 10,000 sources with a daily increment of around 300,000 inputs.
- The system’s stability, speed, and integrity allowed LOOQME to triple their clients and sales every year since 2012/2013. Among LOOQME clients one can find McDonald’s, UNIQA, Henkel, Yves Rocher, Philip Morris, OLX, Polpharma, etc.
- The system’s algorithms reduced the report preparation period by four times.
- Thanks to the minimalistic design and implemented technical solutions, LOOQME is very simple to navigate for users.
The main takeaway from Gearheart's experience with LOOQME is that even one month can be enough to create a complex and comprehensive online media monitoring software. The key to success here is choosing the right technology and knowing how to fine-tune it to perfection.
We're working with Gearheart for over 3 years, and we're completely satisfied. One of the best things about them is that they don't just do what the client says them to do but get to the roots of the idea/problem and advise on how it can be done better conceptually or technically. They do the job with a view to the future – peak loads, new features/functions, etc. Hence, when you need to change or upgrade some part of the project, you don't have to change the basics.
We are delighted that Gearheart's views on project development and design were very close to ours.
These are some of our perfect matches
We are burning not only with cool projects, but also with the people behind them. And a perfect team match truly matters. Our clients are our partners, co-authors, and friends at all project stages. Here are some examples of times when Gearheart became a best-fit teammate for our customers.
