python logo


Category: Selenium

selenium python tutorial w3schools

Selenium is a web automation framework. It starts a web browser and any task that can be done typically on the web, Selenium+Python can do for you.

selenium

Related course

Articles

Note
To use Selenium you need either Python version:


Python 2.6, 2.7
Python 3.3+

To check your version use:

python --version

Installation of a web driver is mandatory, without installing one the browser wont start even though one is installed on your computer.

pip install selenium

Selenium is a web automation tool. A web browser can be controlled using Python code, any task you would normally do on the web can be done using the selenium module.

To use use selenium, you need both the selenium module and the web driver installed. This can be quite tricky to get right, if you are new to selenium I recommend the course below.

Related course
Browser Automation with Python Selenium - Novice to Ninja

Install selenium

To get started, first you should setup a virtual environment. Once that’s setup and activated, you want to install the selenium module inside it. You can do that by typing the command:


pip install -U selenium

This will install the selenium module, but that’s not all yet. You need to install the driver.

Then get the web driver from https://docs.seleniumhq.org/projects/webdriver. There are all kind of webdrivers including:

  • ChromeDriver
  • FirefoxDriver
  • RemoteWebDriver
  • EdgeDriver
  • IEDriver
  • SafariDriver
  • OperaDriver

Selenium Example

Depending on which driver you install, you can load a different browser. If you use Chrome, you could do this:

from selenium.webdriver import Chrome
driver = Chrome()

For the Firefox driver, initialize like this:

from selenium.webdriver import Firefox
driver = Firefox()

This also works for Edge

from selenium.webdriver import Edge
driver = Edge()

After installation of the web driver, we can make Python start the browser using the code below:


from selenium import webdriver
import time

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://python.org')

Save the program as example.py and run it with

python3 example.py

If everything went right, it will start the Chromium browser and open the python site.

selenium chromium Selenium will start the chromium browser automatically

Download Selenium Examples

selenium python

Selenium is a web automation framework that can be used to automate website testing. Because Selenium starts a webbrowser, it can do any task you would normally do on the web.

If you are new to selenium and browser automation, I recommend the course below.

Related course
Browser Automation with Python Selenium

Web Driver

To start a web browser, the Selenium module needs a web driver. Python interacts with the selenium web driver and the web driver interacts with the browser.

Supported browsers are:

  • Chrome
  • Firefox
  • Internet Explorer
  • Safari
  • Opera
  • PhantomJS (invisible)

To start a browser, you will need to corresponding web driver. The driver “ChromeDriver” is needed to start Chrome, “FirefoxDriver” for Firefox.

All drivers can be downloaded from: https://docs.seleniumhq.org/download/

Example code
Python will start and control the chromium browser using the code below:


from selenium import webdriver
import time

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://python.org')

You can change the browser by creating a different instance:


# Firefox 
driver = webdriver.Firefox()

# Google Chrome 
driver = webdriver.Chrome()

# iPhone
driver = webdriver.Remote(browser_name="iphone", command_executor='http://172.24.101.36:3001/hub')

# Android
driver = webdriver.Remote(browser_name="android", command_executor='http://127.0.0.1:8080/hub')

The first two lines in the above code will open the browser on the same computer, the others lines open the browser remotely: on a phone.

Then open a webpage using the get() method:

driver.get('https://python.org')

Download Selenium Examples

python selenium get html

Selenium is a web automation module that can be used to get a webpages html code. In this article we will show how to achieve that.

You can use the web drivers attribute .page_source to grab the html code of any webpage.
If you are new to selenium, I recommend the course below.

Related course
Browser Automation with Python Selenium

Install selenium

If you haven’t done so, install the selenium module (pip), the web browser and the web driver.


pip install selenium

For this example, you may need to set the path to chromium:


export PATH=$PATH:/usr/lib/chromium/

Get html source

You can import thet webdriver from the selenium module. A webdriver object is created (chromium) and we can optionally specify if we want to ignore certificate errors.

Of course any web browser can be used, but for this example I’ve used chromium.

Once the web browser started we navigate it to a webpage URL using the get() module. Then we get the page source.


from selenium import webdriver
import time

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://python.org')

html = driver.page_source
print(html)

It will output the webpage source, which is stored in the variable html.

selenium chromium Selenium will start the chromium browser automatically

Download Selenium Examples

selenium input text python

Given a webpage containing a text area or text field, text can be automatically written using Python code.

Python can start a web browser, open the page and write code in it.
This is done with the the selenium framework.

Related course
Browser Automation with Python Selenium

Setup selenium

In this article we will demonstrate this automation using a small code snippet.

Start by importing the selenium module. This module is not installed by default, so if you don’t have it install it with the Python package manager (pip).

pip install selenium

The driver needs installed aswell as the web browser.

Selenium add textbox test

Lets take a look at the code. First you need the webdriver to create the web browser. You can open the target website using the method below


driver.get(url)

where the parameter specfies the website url.

We select the html element using the method find_element_by_id. The html element must have an id for this to work, otherwise you need xpath.


driver.find_element_by_id(id)

Then the browser writes text in the textbox using the method send_keys, as parameter it takes a string:


send_keys(str)

The complete code example:


from selenium import webdriver
import time

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)
driver.get('http://codepad.org')

text_area = driver.find_element_by_id('textarea')
text_area.send_keys("This text is send using Python code.")

The browser will start automatically and starts adding the text to the textbox:

selenium textbox send_keys method writes in textbox

Download Selenium Examples

selenium get link

Related course:
Browser Automation with Python Selenium

Selenium
Selenium automates browsers. The selenium module can make the browser do anything you want including automated testing, automating web tasks and data extraction. In this article we’ll use it for data mining, extracting the links from a web page.

Install it using:

pip install selenium

To use the module, you need a selenium web driver. All the popular browsers are supported including Chrome and Firefox.

After installing the web driver, you may need to add it to path: 


export PATH=$PATH:/usr/lib/chromium/

Starting Selenium
Test if selenium is installed correctly using:


from selenium import webdriver
import time

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)

driver.get('https://www.w3.org/')
time.sleep(3)
driver.quit()

Depending on your setup, you can start it without parameters:


webdriver.Chrome()

Extract links
To get links from webpage, use the code below:


from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)

driver.get('https://www.w3.org/')
for a in driver.find_elements_by_xpath('.//a'):
    print(a.get_attribute('href'))

Download Selenium Examples

selenium click button

Selenium can automatically click on buttons that appear on a webpage. In this example we will open a site and click on a radio button and submit button.

Related course
Browser Automation with Python Selenium

Selenium button click
Start by importing the selenium module and creating a web driver object. We then use the method:


drivers.find_elements_by_xpath(path)

to find the html element. To get the path, we can use chrome development tools (press F12). We take the pointer in devtools and select the html button we are interested in. The path will then be shown, as the example screenshot:

find_element_by_xpath Find element by xpath, using chrome dev tools
After we have the html object, we use the click() method to make the final click.
Full code:


from selenium import webdriver
import time

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)
driver.get('http://codepad.org')

# click radio button
python_button = driver.find_elements_by_xpath("//input[@name='lang' and @value='Python']")[0]
python_button.click()

# type text
text_area = driver.find_element_by_id('textarea')
text_area.send_keys("print('Hello World')")

# click submit button
submit_button = driver.find_elements_by_xpath('//*[@id="editor"]/table/tbody/tr[3]/td/table/tbody/tr/td/div/table/tbody/tr/td[3]/input')[0]
submit_button.click()

Download Selenium Examples

phantomjs python

PhantomJS is a headless browser that can be used with the Selenium web automation module. Unlike the FirefoxDriver or ChromeDriver, the browser stays totally invisible during the process.

It behaves just like the other browsers. To design the process, you can change the webdriver to ChromeDriver or FirefoxDriver and change it to PhantomJS once its working.

Related course
Browser Automation with Python Selenium

Install PhantomJS

Like the other web drivers, the PhantomJS driver needs to be installed.
On Apple Mac you can install PhantomJS with the line:

brew install phantomjs

On Microsoft Windows or Linux, get PhantomJS from the official site.

Selenium PhantomJS

To use the PhantomJS webdriver, all you need to do is change it to PhantomJS().
Example PhantomJS code:

from selenium import webdriver
import time

driver = webdriver.PhantomJS()
driver.get('https://python.org')

html = driver.page_source
print(html)

After importing the selenium module, we create the webdriver object using 


driver = webdriver.PhantomJS()

Then will This will work with both Python 2.7 and Python 3.

If you are on a Windows machine you can specify the path to the phantomjs executable:


driver = webdriver.PhantomJS("C://phantomjs.exe")
driver.get("https://python.org/")

On Windows, the path should be changed to the location of your phantomjs installation.

On Mac or Linux, simply create the driver using:


driver = webdriver.PhantomJS()
driver.get("https://python.org/")

Download Selenium Examples

selenium screenshot python

Selenium is a web automation framework that can be used for automated testing, web scraping and anything you can do with a web browser. We can use Selenium to take automated screenshots of a webpage.

Related course:
Browser Automation with Python Selenium

Take screenshot Selenium

The way this works is that Python uses the selenium driver to open a module, then selenium will start the defined web browser and open the page url. It will then take a screenshot and save it to the local hard disk.

take screenshot using python code take screenshot using python code

We start a web driver (Chromium) and open the webpage python.org.

from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://python.org')

Then we call the method:


driver.save_screenshot("screenshot.png")

The screenshot will be saved in the same directory as the program: the program path.

The full code is shown below. Now because I’ve tested with the chromium browser, it contains the ChromeOptions as parameter. 


from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)

driver.get('https://python.org')
driver.save_screenshot("screenshot.png")

driver.close()

Remember to call driver.close() otherwise the browser stays open after the program finishes.

So you could use a shorter version, if you use another web browser like

from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://pythonspot.com');
driver.save_screenshot("screenshot.png");

Download Selenium Examples

Take screenshot of html element

You can take a screenshot of a html element. The way this works is that you first take a screenshot of the whole page and then crop it to its html element size.

from selenium import webdriver
from PIL import Image

# take screenshot
driver = webdriver.Chrome();
driver.get('https://www.google.com');
element = driver.find_element_by_id("hplogo");
location = element.location;
size = element.size;
driver.save_screenshot("pageImage.png");

# crop image
x = location['x'];
y = location['y'];
width = location['x']+size['width'];
height = location['y']+size['height'];
im = Image.open('pageImage.png')
im = im.crop((int(x), int(y), int(width), int(height)))
im.save('element.png')

driver.quit()

Download Selenium Examples

selenium download image

Selenium,a web automation framework, can be used to get the all of the image links from a webpage.In this article we’ll given an example of that.

Related course
Browser Automation with Python Selenium

Get image links
At first we import the selenium module and start the web driver object. 


driver = webdriver.Chrome(chrome_options=options)

The webpage consists of html code, defined by tags. To show an image a webpage has a code:


<img src=".../image.jpg">

Then we find all the elements using the img tag (from html):


images = driver.find_elements_by_tag_name('img')

Finally we print the link of each image using:


for image in images:
    print(image.get_attribute('src'))

Full code:


from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument("--test-type")
options.binary_location = "/usr/bin/chromium"
driver = webdriver.Chrome(chrome_options=options)

driver.get('https://imgur.com/')

images = driver.find_elements_by_tag_name('img')
for image in images:
    print(image.get_attribute('src'))

driver.close()

That will return all the image urls on the webpage.
To download import urllib and use the line:


urllib.urlretrieve(src, "filename.png")

You could also use wget, by using os.system();

Download Selenium Examples