Return to Answer

welcome to StackOverflow! If you aren't getting an exception and don't know how to troubleshoot, try line-by-line debugging. It is easy to use pdb to step through your code, or learn to use your IDE's debugging tools. You can even simply open a Python REPL and paste in your commands one by one. Learning to step through your code will benefit you greatly.

In this case it's pretty simple--since your program isn't printing anything, there are only three possibilities: is_allowed_domain() is never returning True, ad_elements are not being found, or search_results are not being found. The issue is the line ad_elements = result.find_elements(By.XPATH, ".//div[@data-text-ad='1']") -- the search returns nothing and so ad_elements is an empty list. This is because you are searching from a result element but ads are not children of search result elements, they are search results. (Assuming we are talking about the same thing--I am defining an "ad" as a Google result marked with "Sponsored").

To correct this, search for ads then iterate through them:

search_box.send_keys(search_keyword) search_box.submit() WebDriverWait(driver, 20).until(EC.visibility_of_any_elements_located((By.CSS_SELECTOR, ".g"))) ad_elements = driver.find_elements(By.XPATH, ".//div[@data-text-ad='1']") for ad_element in ad_elements: # note: there may be multiple links in the same ad element ad_links = ad_element.find_elements(By.CSS_SELECTOR, "a") ad_link = ad_links[0] ad_url = ad_link.get_attribute("href") if is_allowed_domain(ad_url, allowed_domains): # finding the title is not as simple as locating h3 children ad_title = ad_element.find_elements(...) ... 

BTW, I get an ElementClickInterceptedException when trying to directly click() an <a> element on the search results page--I would recommend using driver.switch_to.new_window(), driver.get(ad_url) instead.

As a side note, the line except (NoSuchElementException, Exception) as e: makes no sense, because NoSuchElementException is a subclass of Exception and would be caught by using Exception alone. If you want to have different behavior for NoSuchElementException and other Exception types (which would be a good idea), use separate excepts:

except NoSuchElementException as e: # do something except Exception as e: # do something else 

If you aren't getting an exception and don't know how to troubleshoot, try line-by-line debugging. It is easy to use pdb to step through your code, or learn to use your IDE's debugging tools. You can even simply open a Python REPL and paste in your commands one by one. Learning to step through your code will benefit you greatly.

In this case it's pretty simple--since your program isn't printing anything, there are only three possibilities: is_allowed_domain() is never returning True, ad_elements are not being found, or search_results are not being found. The issue is the line ad_elements = result.find_elements(By.XPATH, ".//div[@data-text-ad='1']") -- the search returns nothing and so ad_elements is an empty list. This is because you are searching from a result element but ads are not children of search result elements, they are search results. (Assuming we are talking about the same thing--I am defining an "ad" as a Google result marked with "Sponsored").

To correct this, search for ads then iterate through them:

search_box.send_keys(search_keyword) search_box.submit() WebDriverWait(driver, 20).until(EC.visibility_of_any_elements_located((By.CSS_SELECTOR, ".g"))) ad_elements = driver.find_elements(By.XPATH, ".//div[@data-text-ad='1']") for ad_element in ad_elements: # note: there may be multiple links in the same ad element ad_links = ad_element.find_elements(By.CSS_SELECTOR, "a") ad_link = ad_links[0] ad_url = ad_link.get_attribute("href") if is_allowed_domain(ad_url, allowed_domains): # finding the title is not as simple as locating h3 children ad_title = ad_element.find_elements(...) ... 

BTW, I get an ElementClickInterceptedException when trying to directly click() an <a> element on the search results page--I would recommend using driver.switch_to.new_window(), driver.get(ad_url) instead.

As a side note, the line except (NoSuchElementException, Exception) as e: makes no sense, because NoSuchElementException is a subclass of Exception and would be caught by using Exception alone. If you want to have different behavior for NoSuchElementException and other Exception types (which would be a good idea), use separate excepts:

except NoSuchElementException as e: # do something except Exception as e: # do something else 

welcome to StackOverflow! If you aren't getting an exception and don't know how to troubleshoot, try line-by-line debugging. It is easy to use pdb to step through your code, or learn to use your IDE's debugging tools. You can even simply open a Python REPL and paste in your commands one by one. Learning to step through your code will benefit you greatly.

In this case it's pretty simple--since your program isn't printing anything, there are only three possibilities: is_allowed_domain() is never returning True, ad_elements are not being found, or search_results are not being found. The issue is the line ad_elements = result.find_elements(By.XPATH, ".//div[@data-text-ad='1']") -- the search returns nothing and so ad_elements is an empty list. This is because ads are not children of search result elements, they are search results (assuming we are talking about the same thing--I am defining an "ad" as a Google result marked with "Sponsored").

To correct this, search for ads then iterate through them:

search_box.send_keys(search_keyword) search_box.submit() WebDriverWait(driver, 20).until(EC.visibility_of_any_elements_located((By.CSS_SELECTOR, ".g"))) ad_elements = driver.find_elements(By.XPATH, ".//div[@data-text-ad='1']") for ad_element in ad_elements: # note: there may be multiple links in the same ad element ad_links = ad_element.find_elements(By.CSS_SELECTOR, "a") ad_link = ad_links[0] ad_url = ad_link.get_attribute("href") if is_allowed_domain(ad_url, allowed_domains): # finding the title is not as simple as locating h3 ad_title = ad_element.find_elements(...) ... 

BTW, I get an ElementClickInterceptedException when trying to directly click() an <a> element from the search results page--I would recommend using driver.switch_to.new_window() driver.get(ad_url) instead.

As a side note, the line except (NoSuchElementException, Exception) as e: makes no sense, because NoSuchElementException is a subclass of Exception and would be caught by using Exception alone. If you want to have different behavior for NoSuchElementException and other Exception types (which would be a good idea), use separate excepts:

except NoSuchElementException as e: # do something except Exception as e: # do something else 

welcome to StackOverflow! If you aren't getting an exception and don't know how to troubleshoot, try line-by-line debugging. It is easy to use pdb to step through your code, or learn to use your IDE's debugging tools. You can even simply open a Python REPL and paste in your commands one by one. Learning to step through your code will benefit you greatly.

In this case it's pretty simple--since your program isn't printing anything, there are only three possibilities: is_allowed_domain() is never returning True, ad_elements are not being found, or search_results are not being found. The issue is the line ad_elements = result.find_elements(By.XPATH, ".//div[@data-text-ad='1']") -- the search returns nothing and so ad_elements is an empty list. This is because you are searching from a result element but ads are not children of search result elements, they are search results. (Assuming we are talking about the same thing--I am defining an "ad" as a Google result marked with "Sponsored").

To correct this, search for ads then iterate through them:

search_box.send_keys(search_keyword) search_box.submit() WebDriverWait(driver, 20).until(EC.visibility_of_any_elements_located((By.CSS_SELECTOR, ".g"))) ad_elements = driver.find_elements(By.XPATH, ".//div[@data-text-ad='1']") for ad_element in ad_elements: # note: there may be multiple links in the same ad element ad_links = ad_element.find_elements(By.CSS_SELECTOR, "a") ad_link = ad_links[0] ad_url = ad_link.get_attribute("href") if is_allowed_domain(ad_url, allowed_domains): # finding the title is not as simple as locating h3 children ad_title = ad_element.find_elements(...) ... 

BTW, I get an ElementClickInterceptedException when trying to directly click() an <a> element on the search results page--I would recommend using driver.switch_to.new_window(), driver.get(ad_url) instead.

As a side note, the line except (NoSuchElementException, Exception) as e: makes no sense, because NoSuchElementException is a subclass of Exception and would be caught by using Exception alone. If you want to have different behavior for NoSuchElementException and other Exception types (which would be a good idea), use separate excepts:

except NoSuchElementException as e: # do something except Exception as e: # do something else 

welcome to StackOverflow! If you aren't getting an exception and don't know how to troubleshoot your code, try some basic debugging. It is easy to use pdb to step through your code, or you can even simply open a Python REPL and paste in your commands one by one. Furthermore, all IDEs come with debugging tools allowing line-by-line execution, and it will benefit you greatly to learn how to do that.

In this case it's pretty simple--since your program isn't printing anything, there are only three possibilities: is_allowed_domain() is never returning True, ad_elements are not being found, or search_results are not being found. The issue is the line ad_elements = result.find_elements(By.XPATH, ".//div[@data-text-ad='1']") -- the search returns nothing and so ad_elements is an empty list. This is because ads are not children of search result elements, they are search results (assuming we are talking about the same thing--I am defining an "ad" as a Google result marked with "Sponsored").

To correct this, search for ads then iterate through them:

search_box.send_keys(search_keyword) search_box.submit() WebDriverWait(driver, 20).until(EC.visibility_of_any_elements_located((By.CSS_SELECTOR, ".g"))) ad_elements = driver.find_elements(By.XPATH, ".//div[@data-text-ad='1']") for ad_element in ad_elements: # note: there may be multiple links in the same ad element ad_links = ad_element.find_elements(By.CSS_SELECTOR, "a") ad_link = ad_links[0] ad_url = ad_link.get_attribute("href") if is_allowed_domain(ad_url, allowed_domains): # finding the title is not as simple as locating h3 ad_title = ad_element.find_elements(...) ... 

BTW, I get an ElementClickInterceptedException when trying to directly click() an <a> element from the search results page--I would recommend using driver.switch_to.new_window() driver.get(ad_url) instead.

As a side note, the line except (NoSuchElementException, Exception) as e: makes no sense, because NoSuchElementException is a subclass of Exception and would be caught by using Exception alone. If you want to have different behavior for NoSuchElementException and other Exception types (which would be a good idea), use separate excepts:

except NoSuchElementException as e: # do something except Exception as e: # do something else 

welcome to StackOverflow! If you aren't getting an exception and don't know how to troubleshoot, try line-by-line debugging. It is easy to use pdb to step through your code, or learn to use your IDE's debugging tools. You can even simply open a Python REPL and paste in your commands one by one. Learning to step through your code will benefit you greatly.

In this case it's pretty simple--since your program isn't printing anything, there are only three possibilities: is_allowed_domain() is never returning True, ad_elements are not being found, or search_results are not being found. The issue is the line ad_elements = result.find_elements(By.XPATH, ".//div[@data-text-ad='1']") -- the search returns nothing and so ad_elements is an empty list. This is because ads are not children of search result elements, they are search results (assuming we are talking about the same thing--I am defining an "ad" as a Google result marked with "Sponsored").

To correct this, search for ads then iterate through them:

search_box.send_keys(search_keyword) search_box.submit() WebDriverWait(driver, 20).until(EC.visibility_of_any_elements_located((By.CSS_SELECTOR, ".g"))) ad_elements = driver.find_elements(By.XPATH, ".//div[@data-text-ad='1']") for ad_element in ad_elements: # note: there may be multiple links in the same ad element ad_links = ad_element.find_elements(By.CSS_SELECTOR, "a") ad_link = ad_links[0] ad_url = ad_link.get_attribute("href") if is_allowed_domain(ad_url, allowed_domains): # finding the title is not as simple as locating h3 ad_title = ad_element.find_elements(...) ... 

BTW, I get an ElementClickInterceptedException when trying to directly click() an <a> element from the search results page--I would recommend using driver.switch_to.new_window() driver.get(ad_url) instead.

As a side note, the line except (NoSuchElementException, Exception) as e: makes no sense, because NoSuchElementException is a subclass of Exception and would be caught by using Exception alone. If you want to have different behavior for NoSuchElementException and other Exception types (which would be a good idea), use separate excepts:

except NoSuchElementException as e: # do something except Exception as e: # do something else