A Complete Tutorial on Web Scraping Using Python: Full Codes Included

Now let us look at how dynamically loaded content is scraped from a website – IMDb in this case.

Go to the Editorial Lists section. If you implement inspect on the page, you will encounter an element with having the parameter data-tested anchored as firstListCardGroup-editorial. However, if you try to locate it in the page source, you will not find it anywhere. That is because the website dynamically loads this entire section onto the browser.

The following lines of code are used to scrape the editorial list and combine the output with the previously scraped statically loaded content. Several more package imports are included in the code to speed up the process.

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
 
option = webdriver.ChromeOptions()
# The codes are for a window subsystem linux. 
option.add_argument('--headless')
option.add_argument('--no-sandbox')
option.add_argument('--disable-dev-sh-usage')
# Replace YOUR-PATH-TO-CHROMEDRIVER with your chromedriver location
driver = webdriver.Chrome('YOUR-PATH-TO-CHROMEDRIVER', options=option)
 
page = driver.get('https://www.imdb.com/chart/top/') # Calling the HTML through request
soup = BeautifulSoup(driver.page_source, 'html.parser') # Parsing the content using the library
 
totalScrapedInfo = [] # This list contains and saves all the information we scrape
links = soup.select("table tbody tr td.titleColumn a") # Selecting all the titled anchors
first10 = links[:10] # Holding onto the first 10 anchors only
for anchor in first10:
    driver.get('https://www.imdb.com/' + anchor['href']) # Accessing the movie’s page 
    infolist = driver.find_elements_by_css_selector('.ipc-inline-list')[0] # Finding the first element with class ‘ipc-inline-list’
    informations = infolist.find_elements_by_css_selector("[role='presentation']") # Finding all elements with role=’presentation’ from the first element with class ‘ipc-inline-list’
    scrapedInfo = {
        "title": anchor.text,
        "year": informations[0].text,
        "duration": informations[2].text,
    } # Saving all the scraped information in a dictionary
    WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "[data-testid='firstListCardGroup-editorial']")))  # Waiting 5 seconds for our element with the attribute data-testid set as `firstListCardGroup-editorial`
    listElements = driver.find_elements_by_css_selector("[data-testid='firstListCardGroup-editorial'] .listName") # Extracting the editorial lists elements
    listNames = [] # Creating an empty list and then appending only the elements texts
    for el in listElements:
        listNames.append(el.text)
    scrapedInfo['editorial-list'] = listNames # Adding the editorial list names to our scrapedInfo dictionary
    totalScrapedInfo.append(scrapedInfo) # Appending the dictionary to the totalScrapedInformation list
    
print(totalScrapedInfo) # Displaying the list with all the information we scraped