Scrapy – Link Extractors

In this article, we are going to learn about Link Extractors in scrapy. “LinkExtractor” is a class provided by scrapy to extract links from the response we get while fetching a website. They are very easy to use which we’ll see in the below post. 

Scrapy – Link Extractors

Basically using the “LinkExtractor” class of scrapy we can find out all the links which are present on a webpage and fetch them in a very easy way. We need to install the scrapy module (if not installed yet) by running the following command in the terminal:

pip install scrapy

Link Extractor class of Scrapy 

So, scrapy have the class “scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor” for extracting the links from a response object. For convenience scrapy also provides us with “scrapy.linkextractors.LinkExtractor“.

Firstly we need to import the LinkExtractor. There are quite a few ways to import and use the LinkExtractor class, but one of them is to import it in the following way:

from scrapy.linkextractors import LinkExtractor

Also, Using the LinkExtractor class. To use the “LinkExtractor” class you need to create the object as given below :

link_ext = LinkExtractor(arguments) 

We can also fetch the links. Now that we have created an object, to fetch links we will use the “extract_links” method of the LinkExtractor class. For that run below code : 

links = link_ext.extract_links(response)

The links fetched are in list format and of the type “scrapy.link.Link” . The parameters of the link object are:

1. url : url of the fetched link.
2. text : the text used in the anchor tag of the link.
3. fragment : the part of the url after the hash (#) symbol.
4. no-follow : tells whether the value of “rel” attribute of the anchor tag is “nofollow” or not.

Stepwise Implementation

Step 1: Creating a spider

A spider is basically a class in scrapy which is used to fetch requests and get a response from a particular website. The code for creating a spider is as follows:

So, here we first imported the scrapy module along with specifically importing the “LinkExtractor” class. Then we created a class named “MySpider” and inherited it from “scrapy.Spider” class.

Then we created a few class variables namely “name” and “start_urls“.

• name : whatever name you want to give to the class
• start_urls: all the URLs which need to be fetched are given here. 

Then those “start_urls” are fetched and the “parse“ function is run on the response obtained from each of them one by one. This is done automatically by scrapy.

Step 2: Creating the LinkExtractor object and Yielding  results

You can create an instance of the “LinkExtractor” class anywhere you want.

In this let us create an instance of the class in the “parse” method itself.

Finally, the full code is :

Step 3 : Running the code

Now we can run the “spider” and fetch the desired result in “json” file (or any other formats supported by scrapy) .

scrapy runspider <python-file> -o <output-file-name> 

Link Extractors using Scrapy

Example 1 :

Let us fetch all the links from the webpage https://quotes.toscrape.com/ and store the output in a JSON file named “quotes.json” :

To run the above code we run the following command :

scrapy runspider scrapy_link_extractor.py -o quotes.json

Output:

scrapy link extractor example 1

Example 2 :

Let us this time fetch all the links from the website https://www.geeksforgeeks.org/email-id-extractor-project-from-sites-in-scrapy-python/  .

This time let us create the instance of the “LinkExtractor” class in the constructor of our Spider and also yield the “nofollow” parameter of the link object. Also let us set the “unique” parameter of “LinkExtractor” to “True” so that we fetch unique results only.

To run the above code, we run the following command in the terminal :

scrapy runspider scrapy_link_extractor.py -o geeksforgeeks.json

Output:

output of scrapy link extractor example 2