Crontab

Start

If you want to run some tasks periodically or run tasks at the absolute time in the future, you can use these parameters in self.crawl():

crawl_at:

The absolute time to start the crawl. It must be a timestamp.

crawl_later:

Starts the crawl after crawl_later seconds.

crawl_period:

Schedules the request to be called periodically. The crawl is called every crawl_period seconds.

For example:

from fulmar.base_spider import BaseSpider

class Handler(BaseSpider):

    base_url = 'http://doc.scrapy.org/en/latest/'

    def on_start(self):
        self.crawl(Handler.base_url, callback=self.save, crawl_period=60*60)

    def save(self, response):
        return {
            'content': response.content,
            'title': response.page_lxml.xpath('//title/text()')[0]
        }

Now we save it to a file called scrapy_spider and run command:

fulmar start_project scrapy_spider.py

We satrtted a project called scrapy_spider and it will run every one hour.

It’s convenient to see it by:

fulmar crontab

–help

You can get help, just run:

fulmar crontab --help

You will see:

Usage: fulmar crontab [OPTIONS]

  Crontab infos and operations.

Options:
  -d, --delete TEXT  Delete a cron task. Here use taskid, e.g, -d taskid
  -v, --verbose      Verbose mode. Show more information about this crontab.
  --help             Show this message and exit.

–delete/-d

Delete a cron task. Here use taskid to represents a cron task. You can delete a task which you put just now:

fulmar crontab --delete=ab99d6167abe14aa2a8dc1a17dcdf1a9e38f96ca

Now if you run fulmar crontab, you will see:

The task has been deleted successfully.

–vorbose/-v

Verbose mode. Show more information about this crontab.