Tag: selenium

selenium python

Selenium is a web automation framework that can be used to automate website testing. Because Selenium starts a webbrowser, it can do any task you would normally do on the web.

If you are new to selenium and browser automation, I recommend the course below.

Related course
Browser Automation with Python Selenium

Web Driver

To start a web browser, the Selenium module needs a web driver. Python interacts with the selenium web driver and the web driver interacts with the browser.

Supported browsers are:

• Chrome
• Firefox
• Internet Explorer
• Safari
• Opera
• PhantomJS (invisible)

To start a browser, you will need to corresponding web driver. The driver “ChromeDriver” is needed to start Chrome, “FirefoxDriver” for Firefox.

All drivers can be downloaded from: https://docs.seleniumhq.org/download/

Example code
Python will start and control the chromium browser using the code below:

from selenium import webdriver
import time

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://python.org')

You can change the browser by creating a different instance:

# Firefox 
driver = webdriver.Firefox()

# Google Chrome 
driver = webdriver.Chrome()

# iPhone
driver = webdriver.Remote(browser_name="iphone", command_executor='http://172.24.101.36:3001/hub')

# Android
driver = webdriver.Remote(browser_name="android", command_executor='http://127.0.0.1:8080/hub')

The first two lines in the above code will open the browser on the same computer, the others lines open the browser remotely: on a phone.

Then open a webpage using the get() method:

driver.get('https://python.org')

Download Selenium Examples

python selenium get html

Selenium is a web automation module that can be used to get a webpages html code. In this article we will show how to achieve that.

You can use the web drivers attribute .page_source to grab the html code of any webpage.
If you are new to selenium, I recommend the course below.

Related course
Browser Automation with Python Selenium

Install selenium

If you haven’t done so, install the selenium module (pip), the web browser and the web driver.

pip install selenium

For this example, you may need to set the path to chromium:

export PATH=$PATH:/usr/lib/chromium/

Get html source

You can import thet webdriver from the selenium module. A webdriver object is created (chromium) and we can optionally specify if we want to ignore certificate errors.

Of course any web browser can be used, but for this example I’ve used chromium.

Once the web browser started we navigate it to a webpage URL using the get() module. Then we get the page source.

from selenium import webdriver
import time

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://python.org')

html = driver.page_source
print(html)

It will output the webpage source, which is stored in the variable html.

Download Selenium Examples

selenium get link

Related course:
Browser Automation with Python Selenium

Selenium
Selenium automates browsers. The selenium module can make the browser do anything you want including automated testing, automating web tasks and data extraction. In this article we’ll use it for data mining, extracting the links from a web page.

Install it using:

pip install selenium

To use the module, you need a selenium web driver. All the popular browsers are supported including Chrome and Firefox.

After installing the web driver, you may need to add it to path:

export PATH=$PATH:/usr/lib/chromium/

Starting Selenium
Test if selenium is installed correctly using:

from selenium import webdriver
import time

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)

driver.get('https://www.w3.org/')
time.sleep(3)
driver.quit()

Depending on your setup, you can start it without parameters:

webdriver.Chrome()

Extract links
To get links from webpage, use the code below:

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)

driver.get('https://www.w3.org/')
for a in driver.find_elements_by_xpath('.//a'):
    print(a.get_attribute('href'))

Download Selenium Examples

selenium click button

Selenium can automatically click on buttons that appear on a webpage. In this example we will open a site and click on a radio button and submit button.

Related course
Browser Automation with Python Selenium

Selenium button click
Start by importing the selenium module and creating a web driver object. We then use the method:

drivers.find_elements_by_xpath(path)

to find the html element. To get the path, we can use chrome development tools (press F12). We take the pointer in devtools and select the html button we are interested in. The path will then be shown, as the example screenshot:

Find element by xpath, using chrome dev tools
After we have the html object, we use the click() method to make the final click.
Full code:

from selenium import webdriver
import time

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)
driver.get('http://codepad.org')

# click radio button
python_button = driver.find_elements_by_xpath("//input[@name='lang' and @value='Python']")[0]
python_button.click()

# type text
text_area = driver.find_element_by_id('textarea')
text_area.send_keys("print('Hello World')")

# click submit button
submit_button = driver.find_elements_by_xpath('//*[@id="editor"]/table/tbody/tr[3]/td/table/tbody/tr/td/div/table/tbody/tr/td[3]/input')[0]
submit_button.click()

Download Selenium Examples

selenium screenshot python

Selenium is a web automation framework that can be used for automated testing, web scraping and anything you can do with a web browser. We can use Selenium to take automated screenshots of a webpage.

Related course:
Browser Automation with Python Selenium

Take screenshot Selenium

The way this works is that Python uses the selenium driver to open a module, then selenium will start the defined web browser and open the page url. It will then take a screenshot and save it to the local hard disk.

We start a web driver (Chromium) and open the webpage python.org.

from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://python.org')

Then we call the method:

driver.save_screenshot("screenshot.png")

The screenshot will be saved in the same directory as the program: the program path.

The full code is shown below. Now because I’ve tested with the chromium browser, it contains the ChromeOptions as parameter.

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)

driver.get('https://python.org')
driver.save_screenshot("screenshot.png")

driver.close()

Remember to call driver.close() otherwise the browser stays open after the program finishes.

So you could use a shorter version, if you use another web browser like

from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://pythonspot.com');
driver.save_screenshot("screenshot.png");

Download Selenium Examples

Take screenshot of html element

You can take a screenshot of a html element. The way this works is that you first take a screenshot of the whole page and then crop it to its html element size.

from selenium import webdriver
from PIL import Image

# take screenshot
driver = webdriver.Chrome();
driver.get('https://www.google.com');
element = driver.find_element_by_id("hplogo");
location = element.location;
size = element.size;
driver.save_screenshot("pageImage.png");

# crop image
x = location['x'];
y = location['y'];
width = location['x']+size['width'];
height = location['y']+size['height'];
im = Image.open('pageImage.png')
im = im.crop((int(x), int(y), int(width), int(height)))
im.save('element.png')

driver.quit()

Download Selenium Examples

selenium download image

Selenium,a web automation framework, can be used to get the all of the image links from a webpage.In this article we’ll given an example of that.

Related course
Browser Automation with Python Selenium

Get image links
At first we import the selenium module and start the web driver object.

driver = webdriver.Chrome(chrome_options=options)

The webpage consists of html code, defined by tags. To show an image a webpage has a code:

<img src=".../image.jpg">

Then we find all the elements using the img tag (from html):

images = driver.find_elements_by_tag_name('img')

Finally we print the link of each image using:

for image in images:
    print(image.get_attribute('src'))

Full code:

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)

driver.get('https://imgur.com/')

images = driver.find_elements_by_tag_name('img')
for image in images:
    print(image.get_attribute('src'))

driver.close()

That will return all the image urls on the webpage.
To download import urllib and use the line:

urllib.urlretrieve(src, "filename.png")

You could also use wget, by using os.system();

Download Selenium Examples