python - How do I simulate Google robots in scrapy spider? -


i want crawl website robots.txt :

user-agent: mediapartners-google*  user-agent:googlebot   disallow:

i have added line settings.py :

user_agent = 'googlebot'

but doesn't work, site still refuse spider's connection.

any ?


Comments

Post a Comment

Popular posts from this blog

assembly - 8086 TASM: Illegal Indexing Mode -

this question has answer here: illegal use of register in indirect addressing 1 answer i writing 8086 assembly program needs compile through tasm v3.1. running error can not seem fix. my data segment has following set purposes of keyboard input: parao label byte maxo db 5 acto db ? datao db 5 dup('$') what trying first character entered, first byte of datao: lea dx, datao mov bl, [dx] however, when attempt compile, error: **error** h5.asm(152) illegal indexing mode line 152 "mov bl, [dx]" any appreciated. if matters, running tasm through dosbox (my laptop running 64-bit win7) google hasn't come helpful answers. can post entirety of code if necessary. pretty sure reason can't use dx register pointer. try instead using [si], [di] or [bx]: lea bx, data0 mov al, [bx]
Read more

Java, LWJGL, OpenGL 1.1, decoding BufferedImage to Bytebuffer and binding to OpenGL across classes -

i've been attempting solve problem without luck. have 4 classes, main class, object class, sprite class , image loader class. i'm attempting load png using image loader , using method found here: http://www.java-gaming.org/topics/bufferedimage-to-lwjgl-texture/25516/msg/220280/view.html#msg220280 convert bytebuffer, , bind opengl. the image in it's own seperate resource folder. should being drawn this: ( http://i.imgur.com/j8gbu4c.png ) (32x32), i'm seeing white box has dimensions of image, not actual texture. if has idea of go fix i'd appreciate it. i'm new opengl , looking avoid using external library in order learn how actual code works. thanks. updated! to following, implemented suggestion provided vallentin, quad taking color of first pixel in image. i've attempted implementing image loader provided c.s here . i'm receiving error: javax.imageio.iioexception: error reading png image data @ com.sun.imageio.plugins.png.pngimagereader.r...
Read more

javascript - addthis share facebook and google+ url -

i attempting use addthis create simple url shares whichever current page user on when click share button. of right now, have twitter working. when user clicks, able see url populated in tweet box. when user clicks facebook or google+, url doesn't seem show in share box. <ul class="addthis_toolbox addthis_default_style "> <li><a class="addthis_button_facebook"><img src="/img/facebook-social.png" width="20" height="20" border="0" alt="share" /></a></li> <li><a class="addthis_button_twitter"><img src="/img/twitter-social.png" width="20" height="20" border="0" alt="share" /></a></li> <li><a class="addthis_button_google_plusone_share"><img src="/img/google-social.png" width="20" height="20" border="0" alt="s...
Read more