Forums

I am using this code:

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")

driver = webdriver.Chrome(options=chrome_options)
print("IT RAN>>>>>>>>>>>")

driver.get("https://www.upwork.com/ab/account-security/login")
print('1')
wait_scrape = WebDriverWait(driver, 120)
print('2')
email_field = wait_scrape.until(EC.element_to_be_clickable((By.ID, "login_username")))
email_field.send_keys(default_email)
email_field.send_keys(Keys.RETURN)
print('22')
driver.maximize_window()
time.sleep(4)
close_button = wait_scrape.until(EC.element_to_be_clickable((By.ID,"onetrust-accept-btn-handler")))
close_button.click()
print('3')
time.sleep(2)
password_field = wait_scrape.until(EC.element_to_be_clickable((By.ID,"login_password")))
password_field.send_keys(default_password)
password_field.send_keys(Keys.RETURN)
print('4')
time.sleep(5)

The same code is working fine in my local

it prints 1, 2 but then does not move forward. then after 120 seconds, it says:

2023-05-29 13:28:12,303: Exception on /run_scraping [POST] Traceback (most recent call last):   File "/home/TahaAnwar01/.virtualenvs/venv/lib/python3.10/site-packages/flask/app.py", line 2190, in wsgi_app response = self.full_dispatch_request()   File "/home/TahaAnwar01/.virtualenvs/venv/lib/python3.10/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)   File "/home/TahaAnwar01/.virtualenvs/venv/lib/python3.10/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()   File "/home/TahaAnwar01/.virtualenvs/venv/lib/python3.10/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)   File "/home/TahaAnwar01/mysite/app.py", line 64, in run_scraping
    run_rss_feed(selected_case_study, select_rss_url, email, password)   File "/home/TahaAnwar01/mysite/bot_functionalities/scraping_bot.py", line 34, in run_rss_feed
    email_field = wait_scrape.until(EC.element_to_be_clickable((By.ID, "login_username")))   File "/home/TahaAnwar01/.virtualenvs/venv/lib/python3.10/site-packages/selenium/webdriver/support/wait.py", line 95, in until
    raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message:

It's possible that the site that you are accessing is displaying something different when it is accessed from a cloud computing environment to what it would if you visited it from your normal home IP address. That's quite common, especially with login endpoints, which is what it looks like you're trying to access.

A good debugging tip is to take a screenshot of the page just before you wait for the login_username element to be visible; you can use the driver.get_screenshot_as_file(filename) function to do that.

I solved that issue It is now logging in and I have confirmed that with the screenshot as well .But now something weird is happening. For example, when i am running this:

try:
    description = driver.find_element(By.XPATH, '//div[@data-test="description"]').text
except:
    description = "Not available"

On my local pc it is fetching the details. But for same page here I am getting "Not available". Same thing is happening with other elements too. Now how can i solve this? Please help

That is because the page that you're accessing does not have the element that you're looking for. You will need to inspect what you're getting back to see why that element is not there.