Asp Forum - subprocess and & (ampersand

Steven Bethard

1/23/2008 5:53:00 AM

I'm having trouble using the subprocess module on Windows when my
command line includes special characters like "&" (ampersand)::

>>> command = 'lynx.bat', '-dump', 'http://www.example.com/?x=1&am...
>>> kwargs = dict(stdin=subprocess.PIPE,
.... stdout=subprocess.PIPE,
.... stderr=subprocess.PIPE)
>>> proc = subprocess.Popen(command, **kwargs)
>>> proc.stderr.read()
"'y' is not recognized as an internal or external command,\r\noperable
program or batch file.\r\n"

As you can see, Windows is interpreting that "&" as separating two
commands, instead of being part of the single argument as I intend it to
be above. Is there any workaround for this? How do I get "&" treated
like a regular character using the subprocess module?

Thanks,

STeVe

9 Answers

Steven D'Aprano

1/23/2008 6:39:00 AM

On Tue, 22 Jan 2008 22:53:20 -0700, Steven Bethard wrote:

> I'm having trouble using the subprocess module on Windows when my
> command line includes special characters like "&" (ampersand)::
>
> >>> command = 'lynx.bat', '-dump', 'http://www.example.com/?x=1&am...
> >>> kwargs = dict(stdin=subprocess.PIPE,
> ... stdout=subprocess.PIPE, ...
> stderr=subprocess.PIPE)
> >>> proc = subprocess.Popen(command, **kwargs) proc.stderr.read()
> "'y' is not recognized as an internal or external command,\r\noperable
> program or batch file.\r\n"
>
> As you can see, Windows is interpreting that "&" as separating two
> commands, instead of being part of the single argument as I intend it to
> be above. Is there any workaround for this? How do I get "&" treated
> like a regular character using the subprocess module?

That's nothing to do with the subprocess module. As you say, it is
Windows interpreting the ampersand as a special character, so you need to
escape the character to the Windows shell.

Under Windows, the escape character is ^, or you can put the string in
double quotes:

# untested
command = 'lynx.bat -dump http://www.example.com/?x=1^&am...
command = 'lynx.bat -dump "http://www.example.com/?x=1&y=2&...

In Linux land, you would use a backslash or quotes.

To find the answer to this question, I googled for "windows how to escape
special characters shell" and found these two pages:

http://www.microsoft.com/technet/archive/winntas/deploy/prodspecs/she...

http://technet2.microsoft.com/WindowsServer/en/library/44500063-fdaf-4e4f-8dac-476c497a166...

Hope this helps,

--
Steven

Steven Bethard

1/23/2008 7:04:00 AM

Steven D'Aprano wrote:
> On Tue, 22 Jan 2008 22:53:20 -0700, Steven Bethard wrote:
>
>> I'm having trouble using the subprocess module on Windows when my
>> command line includes special characters like "&" (ampersand)::
>>
>> >>> command = 'lynx.bat', '-dump', 'http://www.example.com/?x=1&am...
>> >>> kwargs = dict(stdin=subprocess.PIPE,
>> ... stdout=subprocess.PIPE, ...
>> stderr=subprocess.PIPE)
>> >>> proc = subprocess.Popen(command, **kwargs) proc.stderr.read()
>> "'y' is not recognized as an internal or external command,\r\noperable
>> program or batch file.\r\n"
>>
>> As you can see, Windows is interpreting that "&" as separating two
>> commands, instead of being part of the single argument as I intend it to
>> be above. Is there any workaround for this? How do I get "&" treated
>> like a regular character using the subprocess module?
>
>
> That's nothing to do with the subprocess module. As you say, it is
> Windows interpreting the ampersand as a special character, so you need to
> escape the character to the Windows shell.
>
> Under Windows, the escape character is ^, or you can put the string in
> double quotes:
>
> # untested
> command = 'lynx.bat -dump http://www.example.com/?x=1^&am...
> command = 'lynx.bat -dump "http://www.example.com/?x=1&y=2&...

Sorry, I should have mentioned that I already tried that. You get the
same result::

>>> command = 'lynx.bat', '-dump', 'http://www.example.com/?x=1^&am...
>>> proc = subprocess.Popen(command,
... stdin=subprocess.PIPE,
... stdout=subprocess.PIPE,
... stderr=subprocess.PIPE)
>>> proc.stderr.read()
"'y' is not recognized as an internal or external command,\r\noperable
program or batch file.\r\n"

In fact, the "^" doesn't seem to work at the command line either::

>lynx.bat -dump http://www.example.com/?x=...

Can't Access `file://localhost/C:/PROGRA~1/lynx/1'
Alert!: Unable to access document.

lynx: Can't access startfile
'y' is not recognized as an internal or external command,
operable program or batch file.

Using quotes does work at the command line::

C:\PROGRA~1\lynx>lynx.bat -dump "http://www.example.com/?x=1&...
You have reached this web page by typing "example.com",
"example.net", or "example.org" into your web browser.

These domain names are reserved for use in documentation and are
not available for registration. See [1]RFC 2606, Section 3.

References

1. http://www.rfc-editor.org/rfc/r...

But I get no output at all when using quotes with subprocess::

>>> command= 'lynx.bat', '-dump', '"http://www.example.com/?x=..."'
>>> proc = subprocess.Popen(command,
... stdin=subprocess.PIPE,
... stdout=subprocess.PIPE,
... stderr=subprocess.PIPE)
>>> proc.stderr.read()
''

Any other ideas?

STeVe

Tim Golden

1/23/2008 11:07:00 AM

Steven Bethard wrote:
> I'm having trouble using the subprocess module on Windows when my
> command line includes special characters like "&" (ampersand)::
>
> >>> command = 'lynx.bat', '-dump', 'http://www.example.com/?x=1&am...
> >>> kwargs = dict(stdin=subprocess.PIPE,
> ... stdout=subprocess.PIPE,
> ... stderr=subprocess.PIPE)
> >>> proc = subprocess.Popen(command, **kwargs)
> >>> proc.stderr.read()
> "'y' is not recognized as an internal or external command,\r\noperable
> program or batch file.\r\n"
>
> As you can see, Windows is interpreting that "&" as separating two
> commands, instead of being part of the single argument as I intend it to
> be above. Is there any workaround for this? How do I get "&" treated
> like a regular character using the subprocess module?

A little experimentation suggests that the problem's somehow
tied up with the .bat file. ie this works for me (doubly
complicated because of the long firefox path:

<code>
import subprocess

cmd = [
r"c:\Program Files\Mozilla Firefox\firefox.exe",
"http://local.goodtoread.org/search?word=tim&cach...
]
subprocess.Popen (cmd)

</code>

but this doesn't:

<c:/temp/firefox.bat>
"c:\Program Files\Mozilla Firefox\firefox.exe" "%*"
</c:/temp/firefox.bat>

<code>
import subprocess

cmd = [
r"c:\temp\firefox.bat",
"http://local.goodtoread.org/search?word=tim&cach...
]
subprocess.Popen (cmd)

</code>

although, interestingly, it seems to cut off at the
"=" before the "&". Not sure how significant that is.

So, even assuming we're looking at the same situation,
I suppose one solution for you is to break out the
..bat file. But that may not be a possibility.

TJG

Tim Golden

1/23/2008 11:13:00 AM

Steven Bethard wrote:
> I'm having trouble using the subprocess module on Windows when my
> command line includes special characters like "&" (ampersand)::
>
> >>> command = 'lynx.bat', '-dump', 'http://www.example.com/?x=1&am...
> >>> kwargs = dict(stdin=subprocess.PIPE,
> ... stdout=subprocess.PIPE,
> ... stderr=subprocess.PIPE)
> >>> proc = subprocess.Popen(command, **kwargs)
> >>> proc.stderr.read()
> "'y' is not recognized as an internal or external command,\r\noperable
> program or batch file.\r\n"
>
> As you can see, Windows is interpreting that "&" as separating two
> commands, instead of being part of the single argument as I intend it to
> be above. Is there any workaround for this? How do I get "&" treated
> like a regular character using the subprocess module?

Although I'm sure you'll have looked into this already, the
subprocess module on Windows is using CreateProcess pretty
straightforwardly:

http://msdn2.microsoft.com/en-us/librar...(VS.85).aspx

The docs there say that, to call a batch file, you need to
specify the command interpreter with /c and pass the batch
file, but as far as I can see it makes no difference! (Probably
means there's a special-caser behind the scenes of CreateProcess).

TJG

Ross Ridge

1/23/2008 2:58:00 PM

Tim Golden

1/23/2008 3:10:00 PM

Ross Ridge

1/23/2008 3:57:00 PM

Tim Golden <mail@timgolden.me.uk> wrote:
> but this doesn't:
>
> <c:/temp/firefox.bat>
> "c:\Program Files\Mozilla Firefox\firefox.exe" "%*"
> </c:/temp/firefox.bat>
>
> <code>
> import subprocess
>
> cmd = [
> r"c:\temp\firefox.bat",
> "http://local.goodtoread.org/search?word=tim&cach...
> ]
> subprocess.Popen (cmd)
>
> </code>

Ross Ridge wrote:
> You need to use double quotes both in the .BAT file and in the string
> you pass to subprocess.Popen().

Tim Golden <mail@timgolden.me.uk> wrote:
>... If you simply requote the second element in the cmd list
>('"http:/....."') then the internal quotes are escaped by some part of
>the mechanism and it still doesn't work.

Hmm... I guess things are much more messy than that. CMD doesn't do
standard quote processing of it's arguments or .BAT file arguments, and
so is incompatible with how subprocess quotes args lists. It looks like
you need use a string instead of list with subprocess.Popen and not use
quotes in the batch file. So something like:

firefox.bat:
"c:\Program Files\Mozilla Firefox\firefox.exe" %1

code:
subprocess.Popen(r'c:\temp\firefox.bat "http://local.goodtoread.org/search?word=tim&cach...')

Ross Ridge

--
l/ // Ross Ridge -- The Great HTMU
[oo][oo] rridge@csclub.uwaterloo.ca
-()-/()/ http://www.csclub.uwaterloo.c...
db //

Steven Bethard

1/23/2008 4:11:00 PM

Ross Ridge wrote:
> Tim Golden <mail@timgolden.me.uk> wrote:
>> but this doesn't:
>>
>> <c:/temp/firefox.bat>
>> "c:\Program Files\Mozilla Firefox\firefox.exe" "%*"
>> </c:/temp/firefox.bat>
>>
>> <code>
>> import subprocess
>>
>> cmd = [
>> r"c:\temp\firefox.bat",
>> "http://local.goodtoread.org/search?word=tim&cach...
>> ]
>> subprocess.Popen (cmd)
>>
>> </code>
>
> Ross Ridge wrote:
>> You need to use double quotes both in the .BAT file and in the string
>> you pass to subprocess.Popen().
>
> Tim Golden <mail@timgolden.me.uk> wrote:
>> ... If you simply requote the second element in the cmd list
>> ('"http:/....."') then the internal quotes are escaped by some part of
>> the mechanism and it still doesn't work.
>
> Hmm... I guess things are much more messy than that. CMD doesn't do
> standard quote processing of it's arguments or .BAT file arguments, and
> so is incompatible with how subprocess quotes args lists. It looks like
> you need use a string instead of list with subprocess.Popen and not use
> quotes in the batch file. So something like:
>
> firefox.bat:
> "c:\Program Files\Mozilla Firefox\firefox.exe" %1
>
> code:
> subprocess.Popen(r'c:\temp\firefox.bat "http://local.goodtoread.org/search?word=tim&cach...')

This works. The other thing that works is to put extra quotes around
the URL when you pass in the list::

>>> command= 'lynx.bat', '-dump', '"http://www.example.com/?x=1^&y=2&...
>>> proc = subprocess.Popen(command,
... stdin=subprocess.PIPE,
... stdout=subprocess.PIPE,
... stderr=subprocess.PIPE)
>>> proc.stdout.read()
' You have reached this web page by typing "example.com",
"example.net",\n or "example.org" into your web browser.\n\n These
domain names are reserved for use in documentation and are not\n
available for registration. See [1]RFC 2606, Section
3.\n\nReferences\n\n 1. http://www.rfc-editor.org/rfc/r...\n'

I tried this before, and posted it, but I had mistakenly read from
stderr instead of stdout. Thanks for making me double-check that code!

STeVe

Albert van der Horst

2/1/2008 7:33:00 PM

In article <mailman.963.1201086420.896.python-list@python.org>,
Tim Golden <mail@timgolden.me.uk> wrote:
<SNIP>
>
>So, even assuming we're looking at the same situation,
>I suppose one solution for you is to break out the
>.bat file. But that may not be a possibility.

Unless you need compatibility with Windows 3.1., I suggest you
stay away from .bat files. Better call them .cmd, such that they
are executed by cmd.exe (32 bits).
It could be part of your problem.

>
>TJG

Groetjes Albert

--
--
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- like all pyramid schemes -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van...

comp.lang.python

subprocess and & (ampersand

Steven Bethard

Steven D'Aprano

Steven Bethard

Tim Golden

Tim Golden

Ross Ridge

Tim Golden

Ross Ridge

Steven Bethard

Albert van der Horst

x Login to ForumsZone