In this article we will write a small library for creating a preview for a site. The description of how the preview works in slack you can check here.
4 Data sources will be used:
In this exact order we will try to retrieve the data.
For parsing we will use:
oEmbed is an open format designed to simplify the embedding of the contents of one web page to another. In a role of the content there may be photos, videos, links or other types of data. You can find specification and description of the format here.
The basic idea of how this format works: a site provides specific endpoint on which you can make a GET request. This request allows passing the URL of the page you want to extract data for, in the parameters. Here’s the example:
A popular social network for sharing and adding ratings to photos and short videos Instagram supports oEmbed.
Endpoint for receiving data:
The final URL for receiving information about a page (url has to be encoded):
The main problem is that we need to know if the site supports the oEmbed format or not. A full list of providers does not actually exist, so we need to add those providers we know. For this, I decided to use a separate JSON file to which we will add providers.
The format is as follows (borrowed from the specification site, a small list can be found here):
If the reference falls under one of the masks mentioned in the
endpoint specified in the
url will be used.
Let’s write support functions to download a list of providers
Then the parser itself:
Example of using:
if the provider has been added to the file, we'll get a link to the oembed endpoint from which we will be able to receive data. The
get_oembed_url_from_html method will be described below.
Open Graph protocol is a special set of meta tags, which are integrated into the page html-code, the format was created in Facebook. Description of the protocol can be found here. Open graph data is embedded directly into the page and has a simple structure, so it is quite simple to remove it.
Open Graph parser code:
This parser supports retrieval of data from the first level of Open Graph tags and supports arrays, but does not support nested structures, i.e. it can extract data from these tags:
but not from the nested ones:
This format is very similar to the Open Graph, special meta tags are added into the page HTML that contain information about the page. Tags description.
We retrieve all the data from the meta tags with the
name attribute, which starts with
We will receive data by means of an asynchronous HTTP client
The main two functions are:
fetch_all- getting all data from all sources
get_preview_data- getting basic information title, description, image to create a preview
Both functions take the same parameters (only the first two parameters are mandatory):
url- URL, which we want to get the data for
loop- asyncio event loop object
oembed_providers- a list of oEmbed providers
oembed_params- parameters for the oEmbed query (maxwidth and maxheight)
Regarding the session object transfer, this is done in order to create a single session on the application, from the aiohttp documentation:
Don’t create a session per request. Most likely you need a session per application which performs all requests altogether. A session contains a connection pool inside, connection reusage and keep-alives (both are on by default) may speed up total performance.
Use as a library in your application.
For the validation of the data I use a small library marshmallow.
python srv.py, then you can make a GET request to the address
http://127.0.0.1:8080/extract?url=your_link to get all site data or
http://127.0.0.1:8080/preview?url=your_link to get the data which is necessary for the preview.
I also added an example in aiounfurl example repository with GUI, examples:
I added a docker image with the example in http://hub.docker.com/ to run the sample as a separate independent service.
Running in the background:
then you can open our example http://127.0.0.1:8080/.
Using the list of oEmbed providers (a json file with a list of providers
/path_to_file/providers.json has to be preliminarily created):
Django ORM is a very abstract and flexible API. But if you do not know exactly how it works, you will likely end up with slow and heavy views, if you have not already. So, this article provides practical solutions to N+1 and high loading time issues. For clarity, I will create a simple view that demonstrates common ORM query problems and shows frequently used practices.
SQL is a fairly complicated language with a steep learning curve. For a large number of people who make use of SQL, learning to apply it efficiently takes lots of trials and errors. Here are some tips on how you can make your SELECT queries better. The majority of tips should be applicable to any relational database management system, but the terminology and exact namings will be taken from PostgreSQL.