Back
Nov 21, 2016

Crawling FTP server with Scrapy

I won’t describe how to install the library and use it, but here is the link to the documentation where it is described in detail.

At first glance, it is a trivial task. Scrapy can work with FTP and handle files.

A simple example:

import scrapy
from scrapy.http import Request


class FtpSpider(scrapy.Spider):
    name = "mozilla"
    allowed_domains = ["ftp.mozilla.org"]
    handle_httpstatus_list = [404]

    def start_requests(self):
        yield Request('ftp://ftp.mozilla.org/pub/firefox/releases/9.0b4/contrib/solaris_pkgadd/README.txt',
                      meta={'ftp_user': '', 'ftp_password': ''}) 

    def parse(self, response):
        print response.body

As you can see, we simply put link with ftp:// and add data for authorization. Scrapy understands that it deals with the FTP server and uses FTPDownloadHandler that is able connect and download files. The difficulty here is that Scrapy can download a file using a specific link to it, but it can’t download a list of files from the directory and walk the directory tree.

In my case it is the FTP server with a list of files and I need to get a list of links and deal with each link separately. Default FTP handler in Scrapy can’t work with the file list.

After googling a couple of articles and libraries, I came across ftptree. After analyzing the code, it became clear that the default FTP handler is replaced by their own one, which can download a list of files, but can not handle the files separately,

import json
from twisted.protocols.ftp import FTPFileListProtocol
from scrapy.http import Response
from scrapy.core.downloader.handlers.ftp import FTPDownloadHandler


class FtpListingHandler(FTPDownloadHandler):
    def gotClient(self, client, request, filepath):
        self.client = client
        protocol = FTPFileListProtocol()
        return client.list(filepath, protocol).addCallbacks(
            callback=self._build_response,
            callbackArgs=(request, protocol),
            errback=self._failed,
            errbackArgs=(request,))

    def _build_response(self, result, request, protocol):
        self.result = result
        body = json.dumps(protocol.files)
        return Response(url=request.url, status=200, body=body)

To replace the default scrapy handler, you need to write your handler in the parser settings:

DOWNLOAD_HANDLERS = {'ftp': '.FtpListingHandler'}

While writing the parser (spider) we decided to combine the two approaches as follows. When the link was pointed not to the file, but to the server itself, the code from ftptree was used, which retrieved and returned a list of all links to the file. Then this list was transmitted and each link from it was handled by the default handler in Scrapy.

Everything worked as it should, all links to files were received and each file was handled, and articles were received. But each time while running it, all the files were received and handled, even those that had already been handled, so we had to do something with that. Meta-data was transmitted with each file, for example: owner, creation date, etc. We decided to define new files by Date Modified indicator. While handling, I saved the most recent date when the file was created, and used it to filter files in the next operation. As a result, only the newest files were handled. This is an example of how a list of links to all files was received:

import scrapy


class FtpMetaRequest(scrapy.http.Request):
    # add user with password to ftp meta request
    user_meta = {'ftp_user': 'username', 'ftp_password': ''}

    def init(self, args, **kwargs):
        super(FtpMetaRequest, self).init(args, **kwargs)
        self.meta.update(self.user_meta)


class FileFtpRequest(FtpMetaRequest):
    pass


class ListFtpRequest(FtpMetaRequest):
    pass


class MedisumSpider(scrapy.Spider):
    name = "articlespider"

    def start_requests(self):
        # start request to get all files
           yield ListFtpRequest("ftp:///")

    def parse(self, response):
        # get response with all files
        files = json.loads(response.body)

        # filter files
        with open("article_max_date.txt", "r") as outfile:
            date = outfile.read()
        if date:
            scrp_time = parser.parse(date)
            files = filter(
                lambda i: parser.parse(i['date']) >= scrp_time, files)

        # get max date
        date_max = max([parser.parse(fl['date']) for fl in files])
        with open("article_max_date.txt", "w") as outfile:
             outfile.write(date_max.isoformat())

        # get data from each file
        for f in files:
            path = os.path.join(response.url, f['filename'])
            request = FileFtpRequest(path, callback=self.parse_item)
            yield request

    def parse_item(self, response):
         # do some actions
         pass

Let's analyze the code:

  • FtpMetaRequest - adds user and password for requests to the server to meta
  • FileFtpRequest, ListFtpRequest - with the help of these classes our FtpListingHandler detects when it is necessary to get a single file, and when to get a list. This detection is also possible by adding your own flag to the request.meta, but we prefer individual classes.
  • MedisumSpider - parser (spider) itself

And here is the code of the handler itself:

import json
from twisted.protocols.ftp import FTPFileListProtocol
from scrapy.http import Response
from scrapy.core.downloader.handlers.ftp import FTPDownloadHandler


class FtpListingHandler(FTPDownloadHandler):
    # get files list or one file
    def gotClient(self, client, request, filepath):
        # check what class sent a request
        if isinstance(request, 'FileFtpRequest'):
            return super(FtpListingHandler, self).gotClient(
                client, request, filepath)

        protocol = FTPFileListProtocol()
        return client.list(filepath, protocol).addCallbacks(
            callback=self._build_response,
            callbackArgs=(request, protocol),
            errback=self._failed,
            errbackArgs=(request,))


    def _build_response(self, result, request, protocol):
        # get files list or one file
        # check what class sent a request
        if request.class.name == 'FtpMetaRequest':
            return super(FtpListingHandler, self)._build_response(
                result, request, protocol)

        self.result = result
        body = json.dumps(protocol.files)
        return Response(url=request.url, status=200, body=body)

Let’s analyze the handler code. As we can see, the FtpListingHandler itself is inherited from the default Scrapy FTP handler and it overrides two methods - gotClient and _build_response. When FileFtpRequest is received, it falls back to base handler behaior to process single file, in other cases it uses custom methods to work with list of files.

This example shows that we can add and change logic of handlers, which are available in Scrapy, in the way we need. Yet, the code remains understandable, concise and it is easy to maintain or extend it further if necessary. But Scrapy has more than enough standard solutions, which cover 99% of needs when writing a parser (spider).

Subscribe for the news and updates

More thoughts
Apr 18, 2023Technology
TDD guide for busy people, with example

This quick introduction to TDD is meant to show the implementation of this practice with a real-life example.

Nov 29, 2022Technology
React Performance Testing with Jest

One of the key requirements for modern UI is being performant. No matter how beautiful your app looks and what killer features it offers, it will frustrate your users if it clangs.

Jul 27, 2022Technology
Forge Viewer: Our Experience with an Unusual Project

Once we received an interesting task from a client. They needed to allow their users to upload a 3D model of the building and show it in a timelapse video from the construction site.

Apr 27, 2022TechnologyBusiness
How to Choose the Best Javascript Framework: Comparison of the Top Javascript Frameworks

In our article, you will find the best JavaScript framework comparison so that you know for sure how to choose the right one for your project.

Feb 12, 2020Technology
5 Best Payment Gateways For 2020

We reviewed the best payment gateways in 2020. Here’s our comparison of their features, advantages, and disadvantages.

Jul 1, 2010Technology
Overriding QuerySet in Django

As you know, model managers can be overriden in Django. It's convenient to add custom filtration method.