Forums

okay so , using headless chrome browser through selenium , headless chrome was not able to locate the element afterwards i compared the html of the same page which i got in local machine and in console of python anyhwere and on comparing them there were many difference but the key difference i found was that the element i was trying to locate was not mentioned in the html page i got from python anywhere . The html part of the element from local machine and python anywhere are given below : local machine " '<!----></div> <span class="user-id">PK7845</span></a> ' '<!----></div></div></div></div></div> <div class="container wrapper"><div ' 'class="container-left"><div class="marketwatch-sidebar ' 'marketwatch-wrap"><div class="omnisearch"><div class="search"><div ' 'class="su-input-group su-has-icon search-input-field"><!----> <input ' 'type="text" placeholder="Search eg: infy bse, nifty fut, nifty weekly, gold ' 'mcx" autocorrect="off" icon="search" autocomplete="off" label="" rules="" ' 'dynamicwidthsize="8"> <span class="icon icon-search"></span> <!----> ' '<!----></div> <span class="counts">\n' '\t\t\t5 / 50\n' '\t\t</span></div> <!----></div> <div class="instruments"><!----> <!----> ' '<!----> <div class="vddl-list list-flat"><div class="vddl-draggable '"

and html page of the particular element in python anyhwere is " <!----></div> <span class="user-id">PK7845</span></a> ' '<!----></div></div></div></div></div> <div class="container wrapper"><!----> '' " SO there was difference of many lines and in place of the whole code it was replaced by this '<!---->' and the element which i want to locate lies in this section so is there any way to locate the element in python anywhere.

It sounds like the page that you are scraping is returning different results based on where the request comes from. For example, a site might want to present a different appearance to people in different geographical locations, or present one set of pages to requests coming from cloud compute environments but a different one to requests coming from regular ISPs.

The easiest way to see what is coming back for your selenium requests is to uses selenium's screenshot function -- that can save an image file that shows exactly what is displayed on the rendered page that came back in response to your request, which is normally much easier to read than the raw HTML. The function will look like this (assuming that your webdriver object is called browser:

browser.get_screenshot_as_file(filename)