Asp Forum - how to lex javascript for an assert_js system?

Phlip

12/30/2006 6:22:00 AM

Ruboids:

Someone recently posted this:

> o There's a difference between syntax checking and verification of
> functional correctness

That is indeed why a test case that spot-checks your syntax is less useful
than a test case that understands your needs. All unit testing
starts at the former and aims at the latter. Here's an example of
testing Javascript's syntax:

ondblclick = div.attributes['ondblclick']
assert_match /^new Ajax.Updater\("hammy_id"/, ondblclick

It trivially asserts that a DIV (somewhere) contains an ondblclick
handler, and that this has a Script.aculo.us Ajax.Updater in it. The
assertion naturally cannot test that the Updater will indeed update a DIV.

To get closer to the problem, we might decide to get closer to the
Javascript. We may need a mock-Javascript system, to evaluate that string.
It could return the list of nuances commonly called a "Log String Test".
Here's an example, using a slightly more verbose language:

public void testPaintGraphicsintint() {
Mock mockGraphics = new Mock(Graphics.class);
mockGraphics.expects(once()).method("setColor").with(eq(Color.decode("0x6491EE")));
mockGraphics.expects(once()).method("setColor").with(same(Color.black));
mockGraphics.expects(once()).method("drawPolygon");
mockGraphics.expects(once()).method("drawPolygon");
hex.paint((Graphics) mockGraphics.proxy());
mockGraphics.verify();
}

From the top, that mocks your graphics display driver, and retains its
non-retained graphics commands. Then the mockGraphics object
verifies a certain series of calls, with such-and-so parameters.

(That is a Log String Test because it's the equivalent of writing commands
like "setColor" and "drawPolygon" into a log file, and then reading this
to assert things.)

That test case indeed fits the ideal of moving away from testing raw
syntax, and closer to testing semantics. Such a test, for example, could
more easily ignore extraneous calls, and then check that two dynamic
polygons did not overlap.

Now suppose I envision this testage:

def ondblclick(ypath)
%(new Ajax.Updater("node",
"/ctrl/act",
{ asynchronous:true,
evalScripts:true,
method:"get",
parameters:"i_b_a=Parameter" })
).gsub("\n", '').squeeze(' ')
end

def test_some_js
js = ondblclick()
parse = assert_js(js)
statement = parse.first
assert_equal 'new Ajax.Updater', statement.get_method
assert_equal '"node"', statement.get_param(0)
assert_equal '"/ctrl/act"', statement.get_param(1)
json = statement.get_param(2)
assert_equal true, json['evalScripts']
end

The goal is the target JS can flex easily - can reorder its Json, or
change fuzzy details, or add new features - without breaking the tests.
Ideally, only changes that break project requirements will break tests.

Now suppose I want to write that assert_js() using less than seven billion
lines of code.

The first shortcut is to only parse code we expect. I'm aware that's
generally against the general philosophy of parsing, but I'm trying to
sell an application, not a JS parser. That's a private detail. I can
accept, for example, only parsing the JS emitted by Rails's standard
gizmos.

So before getting down to some actual questions, here's the code my
exquisite parsing skills have thrashed out so far:

def test_assert_js
source = 'new Ajax.Updater('+
'"node", '+
'"/controller/action", '+
'{ asynchronous:true, '+
'evalScripts:true, '+
'method:"get", '+
'parameters:"i_b_a=Parameter" })'

js = assert_js(source)
assert_equal 'new Ajax.Updater', js.keys.first
parameters = js.values.first['()']
assert_equal '"node"', parameters[0]
assert_equal '"/controller/action"', parameters[1]
json = parameters[2]['{}']
assert_equal 'true', json['evalScripts']
assert_equal '"get"', json['method']
assert_equal '"i_b_a=Parameter"', json['parameters']
end

Now that's good enough for government work, and I could probably upgrade
the interface to look more like my idealized example...

....but the implementation is a mish-mash of redundant
Regexps and run-on methods:

Qstr = /^(["](?:(?:\\["])|(?:[^\\"]+))*?["]),?\s*/

def assert_json(source)
js = {}
identifier = /([[:alnum:]_]+):/

while m = source.match(identifier)
source = m.post_match
n = source.match(/^([[:alnum:]_]+),?\s*/)
n = source.match(Qstr) unless n
break unless n
js[m.captures[0]] = n.captures[0]
source = n.post_match
end

return { '{}' => js }
end

def assert_js(source)
js = {}
qstr = /^(["](?:(?:\\["])|(?:[^\\"]+))*?["]),?\s*/
json = /^(\{.*\}),?\s*/

if source =~ /^([^$"]+)(.*)$/
js[$1] = assert_js($2)
elsif source =~ /^\((.*)$$/
js['()'] = assert_js($1)
else
index = 0

while (m = source.match(qstr)) or
(m = source.match(json))
break if m.size < 1

if source =~ /^\{/
js[index] = assert_json(m.captures[0])
else
js[index] = m.captures[0]
end

source = m.post_match
index += 1
end
end

return js
end

Now the questions. Is there some...

...way to severely beautify that implementation?
...lexing library I could _easily_ throw in?
...robust JS Lexer library already out there?
...assert_js already out there?

--
Phlip
http://c2.com/cgi/wik... <-- NOT a blog!!

15 Answers

Luke Graham

1/2/2007 12:03:00 PM

On 12/30/06, Phlip <phlip2005@nogmailspam.com> wrote:
> Now the questions. Is there some...
>
> ...way to severely beautify that implementation?
> ...lexing library I could _easily_ throw in?
> ...robust JS Lexer library already out there?
> ...assert_js already out there?

Parsing with regexps makes baby Jesus cry. Javascript itself can be
quite flexible, so it may be possible to do enough with a standard
interpreter's run-time. Alternatively, have a look at the Mozilla
projects repository for a real interpreter you could hack on.

Phlip

1/2/2007 12:38:00 PM

spooq wrote:

> Parsing with regexps makes baby Jesus cry. Javascript itself can be
> quite flexible, so it may be possible to do enough with a standard
> interpreter's run-time. Alternatively, have a look at the Mozilla
> projects repository for a real interpreter you could hack on.

This question is for an academic paper, so it has even more ridiculous
constraints on the amount of fun I can have. (And why hasn't the
industry invented a Lex in a Bottle, using Regexp-like strings and a
BNF notation?)

Can I add a parser to the Syntax library? It only does Ruby, XML, and
YAML so far...

And, yes, JavaScript was designed to be parsed, unlike some other
languages...

--
Phlip

Luke Graham

1/2/2007 1:23:00 PM

On 1/2/07, Phlip <phlip2005@gmail.com> wrote:
> spooq wrote:
>
> > Parsing with regexps makes baby Jesus cry. Javascript itself can be
> > quite flexible, so it may be possible to do enough with a standard
> > interpreter's run-time. Alternatively, have a look at the Mozilla
> > projects repository for a real interpreter you could hack on.
>
> This question is for an academic paper, so it has even more ridiculous
> constraints on the amount of fun I can have. (And why hasn't the
> industry invented a Lex in a Bottle, using Regexp-like strings and a
> BNF notation?)

Not sure exactly how you want to improve on lex?

> Can I add a parser to the Syntax library? It only does Ruby, XML, and
> YAML so far...

I don't see how that's better than grabbing the Javascript grammar off
the web in BNF :
http://www.mozilla.org/js/language/es4/formal/lexer-gr...
http://www.antlr.org/grammar/1153976512034/ecma...
etc.

> And, yes, JavaScript was designed to be parsed, unlike some other
> languages...

No names need be mentioned... ;)

http://corion.net/perl-dev/Javascript-Pur... does javascript to
xml, which seems the best/quickest solution that I've seen in my 2
minutes of googling. Just write enough perl to put that into a file
somewhere, and get into a nicer language ASAP ;)

Luke Graham

1/2/2007 1:58:00 PM

http://lxr.mozilla.org/mozilla/source/j...

Have a look around line 2315.

Doing some investigation into aspect-oriented programming in
javascript may also be worthwhile.

I'll stop now :)

Phlip

1/2/2007 3:47:00 PM

spooq wrote:

I might go with Javascript-Pure-Perl - see below. The following is just
wrap-ups.

> http://lxr.mozilla.org/mozilla/source/j...

> Have a look around line 2315.

Interesting, but I don't get It. The IT object ... exists at debug
time, and traces all its calls?

> Not sure exactly how you want to improve on lex?

Regexp is itself also a "little language". However, to get to the
language, we don't need to write a .regex file, compile it with special
compilers, produce a .c file, compile this, link into it, bind to it,
yack yack yack, and so on just to use it.

So, I envision Ruby lines like Lex.new.e(' LetterE -> E | e'). Instead
of externally compiling the little language, we just host it.

That is not important for the current project...

> > Can I add a parser to the Syntax library? It only does Ruby, XML, and
> > YAML so far...
>
> I don't see how that's better than grabbing the Javascript grammar off
> the web in BNF :

In theory, I only need a dirt-simple way to spot-check the source; I'm
not writing a JavaScript interpreter. But it could be better if it
doesn't force me to externally compile the lexer.

> http://corion.net/perl-dev/Javascript-Pur... does javascript to
> xml

Righteous! The project already uses XML (and unit tests with
assert_xpath), so that will fit right in!

Thanks! I honestly would never have thought to try Perl...

--
Phlip

Luke Graham

1/2/2007 4:56:00 PM

On 1/2/07, Phlip <phlip2005@gmail.com> wrote:
> spooq wrote:
>
> I might go with Javascript-Pure-Perl - see below. The following is just
> wrap-ups.
>
> > http://lxr.mozilla.org/mozilla/source/j...
>
> > Have a look around line 2315.
>
> Interesting, but I don't get It. The IT object ... exists at debug
> time, and traces all its calls?

It just makes a pre-defined object that you can poke at when you run
scripts in that interpreter. The implication was that you could
recreate the Ajax.* methods and use them to log.

> > Not sure exactly how you want to improve on lex?
>
> Regexp is itself also a "little language". However, to get to the
> language, we don't need to write a .regex file, compile it with special
> compilers, produce a .c file, compile this, link into it, bind to it,
> yack yack yack, and so on just to use it.

Lex generates C and lives by the rules of that coding universe. Doing
stuff at run-time can be difficult and wierd there. Much easier to
transform to a familiar language and compile and link with exactly the
same tools you use for the rest of your project. Yack (yacc) is an
entirely different project ;)

> So, I envision Ruby lines like Lex.new.e(' LetterE -> E | e'). Instead
> of externally compiling the little language, we just host it.

Which would be living by the rules and expectations of the Ruby
universe. Not that theres anything wrong with that; I happen to quite
like living there myself. It's just useful to remember there's more
than one way of doing things.

> That is not important for the current project...

Agreed.

> > > Can I add a parser to the Syntax library? It only does Ruby, XML, and
> > > YAML so far...
> >
> > I don't see how that's better than grabbing the Javascript grammar off
> > the web in BNF :
>
> In theory, I only need a dirt-simple way to spot-check the source; I'm
> not writing a JavaScript interpreter. But it could be better if it
> doesn't force me to externally compile the lexer.
>
> > http://corion.net/perl-dev/Javascript-Pur... does javascript to
> > xml
>
> Righteous! The project already uses XML (and unit tests with
> assert_xpath), so that will fit right in!

Not sure what assert_path is, guess it's a function from some kind of
test harness.

> Thanks! I honestly would never have thought to try Perl...

It's not exactly my first choice either, but any port will do in a
storm. At least you found one acceptable suggestion in my ramblings :)

Giles Bowkett

1/2/2007 5:53:00 PM

Sorry, why do you want to do this in the first place? The original
post mentioned unit testing, if you want to unit test JavaScript,
there are much easier ways.

--
Giles Bowkett
http://www.gilesg...
http://gilesbowkett.bl...
http://gilesgoatboy.bl...

Phlip

1/2/2007 6:06:00 PM

To spooq:

Consider this snip of C++, via Boost/Spirit:

rule<> LetterE = chr_p('e') | chr_p('E');

The bad news, of course, is all the excessive chr_p stuff. The good
news is that's raw C++, not even a string, and it all compiles at
compile time.

Giles Bowkett wrote:

> Sorry, why do you want to do this in the first place? The original
> post mentioned unit testing, if you want to unit test JavaScript,
> there are much easier ways.

It's a secret. You'l see!...

--
Phlip

Luke Graham

1/3/2007 10:24:00 AM

On 1/2/07, Giles Bowkett <gilesb@gmail.com> wrote:
> Sorry, why do you want to do this in the first place? The original
> post mentioned unit testing, if you want to unit test JavaScript,
> there are much easier ways.

No doubt, I said as much in my first reply, but I elaborated on the
parsing because that interests me.

Luke Graham

1/3/2007 10:38:00 AM

On 1/2/07, Phlip <phlip2005@gmail.com> wrote:
> To spooq:
>
> Consider this snip of C++, via Boost/Spirit:
>
> rule<> LetterE = chr_p('e') | chr_p('E');
>
> The bad news, of course, is all the excessive chr_p stuff. The good
> news is that's raw C++, not even a string, and it all compiles at
> compile time.

Boost is indeed cool, they push the boundaries of C++ further than
anyone. I really like their XML parser, much better than that horrid
Xerces port. C++ != C though, especially when lex was written. :)

Could you let me know how the perl script is going, either here or off-list?

comp.lang.ruby

how to lex javascript for an assert_js system?

Phlip

Luke Graham

Phlip

Luke Graham

Luke Graham

Phlip

Luke Graham

Giles Bowkett

Phlip

Luke Graham

Luke Graham

x Login to ForumsZone