[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

Using a proxy with urllib2

Stubby

1/10/2008 4:46:00 AM

I'm trying to use a proxy server with urllib2.
So I have managed to get it to work by setting the environment
variable:
export HTTP_PROXY=127.0.0.1:8081

But I wanted to set it from the code. However, this does not set the proxy:
httpproxy = '127.0.0.1:3129'
proxy_support = urllib2.ProxyHandler({"http":"http://" + httpproxy})
opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
urllib2.install_opener(opener)
I'm using it from a web.py URL handler file, not sure if it matters.

I have another question though. It seems that using either of the
methods above, the proxy will be global. What if I want to use
a proxy with one site, but not with another site? Or even use a
proxy for some URLs but not others? The proxy having to be global
is really not convenient. Is there any way to do per-fetch proxy?


6 Answers

Rob Wolfe

1/10/2008 6:06:00 PM

0

"Jack" <nospam@invalid.com> writes:

> I'm trying to use a proxy server with urllib2.
> So I have managed to get it to work by setting the environment
> variable:
> export HTTP_PROXY=127.0.0.1:8081
>
> But I wanted to set it from the code. However, this does not set the proxy:
> httpproxy = '127.0.0.1:3129'
> proxy_support = urllib2.ProxyHandler({"http":"http://" + httpproxy})
> opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
> urllib2.install_opener(opener)

Works for me.
How do you know that the proxy is not set?

> I'm using it from a web.py URL handler file, not sure if it matters.

I don't think so.

> I have another question though. It seems that using either of the
> methods above, the proxy will be global. What if I want to use
> a proxy with one site, but not with another site? Or even use a
> proxy for some URLs but not others? The proxy having to be global
> is really not convenient. Is there any way to do per-fetch proxy?

Try this:

<code>
import urllib2

def getopener(proxy=None):
opener = urllib2.build_opener(urllib2.HTTPHandler)
if proxy:
proxy_support = urllib2.ProxyHandler({"http": "http://" + proxy})
opener.add_handler(proxy_support)
return opener

def fetchurl(url, opener):
f = opener.open(url)
data = f.read()
f.close()
return data

print fetchurl('http://www.pytho..., getopener('127.0.0.1:8081'))
</code>

HTH,
Rob

Stubby

1/11/2008 5:39:00 PM

0


> Works for me.
> How do you know that the proxy is not set?

The proxy drops some URLs and the URLs were not being dropped when I did
this :)

> Try this:

Thank you. I'll give it a try.


Stubby

1/11/2008 5:57:00 PM

0

Rob,

I tried your code snippet and it worked great. I'm just wondering if
getopener( ) call
is lightweight so I can just call it in every call to fetchurl( )? Or I
should try to share
the opener object among fetchurl( ) calls?

Thanks,
Jack


"Rob Wolfe" <rw@smsnet.pl> wrote in message
news:87ir21o8sj.fsf@merkury.smsnet.pl...
> Try this:
>
> <code>
> import urllib2
>
> def getopener(proxy=None):
> opener = urllib2.build_opener(urllib2.HTTPHandler)
> if proxy:
> proxy_support = urllib2.ProxyHandler({"http": "http://" + proxy})
> opener.add_handler(proxy_support)
> return opener
>
> def fetchurl(url, opener):
> f = opener.open(url)
> data = f.read()
> f.close()
> return data
>
> print fetchurl('http://www.pytho..., getopener('127.0.0.1:8081'))
> </code>
>
> HTH,
> Rob


Rob Wolfe

1/11/2008 6:28:00 PM

0

"Jack" <nospam@invalid.com> writes:

> Rob,
>
> I tried your code snippet and it worked great. I'm just wondering if
> getopener( ) call
> is lightweight so I can just call it in every call to fetchurl( )? Or I
> should try to share
> the opener object among fetchurl( ) calls?

Creating an opener for every url is rather not reasonable way to go.
Sharing is the better approach. In your case you might
create e.g. two instances: simple_opener and proxy_opener.

Regards,
Rob

Stubby

1/12/2008 6:07:00 AM

0


>> I'm trying to use a proxy server with urllib2.
>> So I have managed to get it to work by setting the environment
>> variable:
>> export HTTP_PROXY=127.0.0.1:8081
>>
>> But I wanted to set it from the code. However, this does not set the
>> proxy:
>> httpproxy = '127.0.0.1:3129'
>> proxy_support = urllib2.ProxyHandler({"http":"http://" + httpproxy})
>> opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
>> urllib2.install_opener(opener)

I find out why it doesn't work in my code but I don't have a solution -
somewhere
else in the code calls these two lines:

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)

and they override the proxy opener. Could anyone tell me how to use both
openers?

Rob Wolfe

1/12/2008 12:16:00 PM

0

"Jack" <nospam@invalid.com> writes:

>>> I'm trying to use a proxy server with urllib2.
>>> So I have managed to get it to work by setting the environment
>>> variable:
>>> export HTTP_PROXY=127.0.0.1:8081
>>>
>>> But I wanted to set it from the code. However, this does not set the
>>> proxy:
>>> httpproxy = '127.0.0.1:3129'
>>> proxy_support = urllib2.ProxyHandler({"http":"http://" + httpproxy})
>>> opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
>>> urllib2.install_opener(opener)
>
> I find out why it doesn't work in my code but I don't have a solution -
> somewhere
> else in the code calls these two lines:
>
> opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
> urllib2.install_opener(opener)
>
> and they override the proxy opener. Could anyone tell me how to use both
> openers?
>

You don't have to create another opener if you only want to add
some handler. You can use `add_handler` method, e.g.:
opener.add_handler(urllib2.HTTPCookieProcessor(cj))

HTH,
Rob