Back
Oct 22, 2016

Solr Sharding

Rostyslav Stekh
Rostyslav Stekh

When dealing with one of our projects (LookSMI media monitoring platform) we have to handle the huge volume of data – and its quantity is constantly growing. At the same time, we must run quick searches with smart rules. To make this work, the whole index should be placed into RAM.

LookSMI Case: Using Shards for the News Portal

It is obvious that when millions of records are being added to the index regularly, RAM size would never be enough. Eventually, you will have to divide index into several parts in order to run search on several machines simultaneously.

LookSMI is dealing with the news, which means that the portal is heavily working with the new records while almost not using the older ones. LookSMI utilizes Solr as a full-text search engine. To ensure fast search within recent records, we decided to shard LookSMI's index.

Sharding is a type of database partitioning that separates very large databases the into smaller and faster, parts called data shards. The word shard itself means a 'small part of a whole'. Technically, sharding means horizontal partitioning but in practice, the term is often used to refer to any database partitioning that is meant to make a very large database more manageable and easier to search through.

Accordingly, we partitioned LookSMI's search index into shards where each shard corresponds with one month. A limited number of shards are set to be active concurrently – just a few last months. Thus, all data a user needs is housed in RAM.

Whenever there is a need, older shards can be activated to accomplish necessary request – and then deactivated. Moreover, when filtering the search by predefined date, we can engage just a part of active shards so that the search is performed only on those servers where the data for the specified period is located.

Steb-by-step Guide to Arranging Solr Sharding

There are several ways to arrange sharding in Solr. The easiest one is to divide index into a few cores. Hence, when carrying out a search query, Solr should be commanded to perform search throughout several cores simultaneously.

First, you should create cores with the identical structure. This can be easily done with the help of cores’ pattern.

For starters, copy configuration files into the configsets folder:

cp sorl/data//conf solr/data/configsets/conf

Then specify the folder’s name when creating the core:

mkdir solr/data/configsets/
cp solr/data/configsets/conf    solr/data/configsets//conf

We have automated the process of creating the cores so that new cores are proactively set up every month:

http:///solr/admin/cores?action=CREATE&name=&instanceDir=path/to/instance&configSet=path/to/instance/`

When the cores are created, revise the code so that data would be added to a specific core while performing a concurrent search over the other cores:

http:///solr/select?q=*:*&shards=http:///solr/,http:///solr/`

Then, revise the code so that data would be recorded to a corresponding core:

http:///solr//update -H

Cores can be located on different servers. Furthermore, they can be arranged not only by time periods but also by categories.

Don’t be afraid to create multiple cores as resource overheads under these conditions are quite small. On the other hand, sharding makes database systems smoothly scalable and helps to deal with the problem of slower response times for growing indexes.

More thoughts

May 10, 2018Technology
How to Build a Cloud-Based Leads Management System for Universities

Lead management is an important part of the marketing strategy of every company of any size. Besides automating various business processes, privately-held organizations should consider implementing an IT solution that would help them manage their leads. So, how should you make a web-based leads management system for a University in order to significantly increase sales?

Vladimir Sidorenko
Vladimir Sidorenko
Feb 28, 2010Technology
Composing multiple views in Django

In UNIX way, each view should solve single task. This is good idea, but sometimes we need to mix logic of different views on same page. Filter, sort, paginate, or, for example, add comment on product page. In this article I'll show how we can mix such multiple views.

Vladimir Sidorenko
Vladimir Sidorenko
Jun 25, 2011Technology
Ajax blocks in Django

Quite often we have to write paginated or filtered blocks of information on page. I created a decorator that would automate this process.

Vladimir Sidorenko
Vladimir Sidorenko
Dec 11, 2016Technology
Auto WebSocket Reconnection with RxJS (with Example)

In this RxJS tutorial article, we will focus on restoring the websocket connection when using RxJS library.

Igor Tokarev
Igor Tokarev
Sep 23, 2010Technology
OR and AND without django.db.models.Q

I just found out that __or__ and __and__ are defined for QuerySet. This means that to do queries union or intersection, you can do:User.objects.filter(...) | User.objects.filter(...)User.objects.filter(...) & User.objects.filter(...)

Vladimir Sidorenko
Vladimir Sidorenko
Sep 21, 2020Technology
How to Optimize Django ORM Queries

Django ORM is a very abstract and flexible API. But if you do not know exactly how it works, you will likely end up with slow and heavy views, if you have not already. So, this article provides practical solutions to N+1 and high loading time issues. For clarity, I will create a simple view that demonstrates common ORM query problems and shows frequently used practices.

Alexey Demianenko
Alexey Demianenko