Creating a site preview like in slack (using aiohttp)

If the reference falls under one of the masks mentioned in the schemes, the endpoint specified in the url will be used.

Let’s write support functions to download a list of providers

Then the parser itself:

Example of using:

if the provider has been added to the file, we'll get a link to the oembed endpoint from which we will be able to receive data. The get_oembed_url_from_html method will be described below.

Open Graph

Open Graph protocol is a special set of meta tags, which are integrated into the page html-code, the format was created in Facebook. Description of the protocol can be found here. Open graph data is embedded directly into the page and has a simple structure, so it is quite simple to remove it.

Open Graph parser code:

This parser supports retrieval of data from the first level of Open Graph tags and supports arrays, but does not support nested structures, i.e. it can extract data from these tags:

but not from the nested ones:

Twitter Cards

This format is very similar to the Open Graph, special meta tags are added into the page HTML that contain information about the page. Tags description.

Parser code:

We retrieve all the data from the meta tags with the name attribute, which starts with twitter:

Meta tags

Parser:

Retrieving data

We will receive data by means of an asynchronous HTTP client aiohttp.

The main two functions are:

• fetch_all - getting all data from all sources
• get_preview_data - getting basic information title, description, image to create a preview

Both functions take the same parameters (only the first two parameters are mandatory):

• session - aiohttp.ClientSession
• url - URL, which we want to get the data for
• loop - asyncio event loop object
• oembed_providers - a list of oEmbed providers
• oembed_params - parameters for the oEmbed query (maxwidth and maxheight)

Regarding the session object transfer, this is done in order to create a single session on the application, from the aiohttp documentation:

Don’t create a session per request. Most likely you need a session per application which performs all requests altogether. A session contains a connection pool inside, connection reusage and keep-alives (both are on by default) may speed up total performance.

Example of using

Use as a library in your application.

For the validation of the data I use a small library marshmallow.

srv.py

requirements.txt

Run python srv.py, then you can make a GET request to the address http://127.0.0.1:8080/extract?url=your_link to get all site data or http://127.0.0.1:8080/preview?url=your_link to get the data which is necessary for the preview.

I also added an example in aiounfurl example repository with GUI, examples:

Running the example in Docker.

I added a docker image with the example in http://hub.docker.com/ to run the sample as a separate independent service.

Running in the background:

then you can open our example http://127.0.0.1:8080/.

Using the list of oEmbed providers (a json file with a list of providers /path_to_file/providers.json has to be preliminarily created):

The application on github.