[lnkForumImage]
TotalShareware - Download Free Software

Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

Looking for library to estimate likeness of two strings

agenkin@gmail.com

2/6/2008 10:02:00 PM

Are there any Python libraries implementing measurement of similarity
of two strings of Latin characters?

I'm writing a script to guess-merge two tables based on people's
names, which are not necessarily spelled the same way in both tables
(especially the given names). I would like some function that would
help me make the best guess.

Many thanks in advance!
14 Answers

Tim Chase

2/6/2008 10:29:00 PM

0

> Are there any Python libraries implementing measurement of similarity
> of two strings of Latin characters?

It sounds like you're interested in calculating the Levenshtein
distance:

http://en.wikipedia.org/wiki/Levenshtei...

which gives you a measure of how different they are. A measure
of "0" is that the inputs are the same. The more different the
two strings are, the greater the resulting output of the function.

Unfortunately, it's an O(MN) algorithm (where M=len(word1) and
N=len(word2)) from my understanding of the code I've seen.
However it really is the best approximation I've seen of a "how
similar are these two strings" function. Googling for

python levenshtein distance

brings up oodles of hits.

-tkc



Jeff Schwab

2/6/2008 10:57:00 PM

0

Tim Chase wrote:
>> Are there any Python libraries implementing measurement of similarity
>> of two strings of Latin characters?
>
> It sounds like you're interested in calculating the Levenshtein distance:
>
> http://en.wikipedia.org/wiki/Levenshtei...
>
> which gives you a measure of how different they are. A measure of "0"
> is that the inputs are the same. The more different the two strings
> are, the greater the resulting output of the function.
>
> Unfortunately, it's an O(MN) algorithm (where M=len(word1) and
> N=len(word2)) from my understanding of the code I've seen. However it
> really is the best approximation I've seen of a "how similar are these
> two strings" function. Googling for
>
> python levenshtein distance
>
> brings up oodles of hits.

If the strings happen to be the same length, the Levenshtein distance is
equivalent to the Hamming distance. The Wikipedia article gives the
following Python implementation:

# http://en.wikipedia.org/wiki/Hammin...
def hamdist(s1, s2):
assert len(s1) == len(s2)
return sum(ch1 != ch2 for ch1, ch2 in zip(s1, s2))

Jeff Schwab

2/6/2008 10:57:00 PM

0

Tim Chase wrote:
>> Are there any Python libraries implementing measurement of similarity
>> of two strings of Latin characters?
>
> It sounds like you're interested in calculating the Levenshtein distance:
>
> http://en.wikipedia.org/wiki/Levenshtei...
>
> which gives you a measure of how different they are. A measure of "0"
> is that the inputs are the same. The more different the two strings
> are, the greater the resulting output of the function.
>
> Unfortunately, it's an O(MN) algorithm (where M=len(word1) and
> N=len(word2)) from my understanding of the code I've seen. However it
> really is the best approximation I've seen of a "how similar are these
> two strings" function. Googling for
>
> python levenshtein distance
>
> brings up oodles of hits.

If the strings happen to be the same length, the Levenshtein distance is
equivalent to the Hamming distance. The Wikipedia article gives the
following Python implementation:

# http://en.wikipedia.org/wiki/Hammin...
def hamdist(s1, s2):
assert len(s1) == len(s2)
return sum(ch1 != ch2 for ch1, ch2 in zip(s1, s2))

Robert Kern

2/6/2008 11:33:00 PM

0

Jeff Schwab wrote:
> Tim Chase wrote:
>>> Are there any Python libraries implementing measurement of similarity
>>> of two strings of Latin characters?
>> It sounds like you're interested in calculating the Levenshtein distance:
>>
>> http://en.wikipedia.org/wiki/Levenshtei...
>>
>> which gives you a measure of how different they are. A measure of "0"
>> is that the inputs are the same. The more different the two strings
>> are, the greater the resulting output of the function.
>>
>> Unfortunately, it's an O(MN) algorithm (where M=len(word1) and
>> N=len(word2)) from my understanding of the code I've seen. However it
>> really is the best approximation I've seen of a "how similar are these
>> two strings" function. Googling for
>>
>> python levenshtein distance
>>
>> brings up oodles of hits.
>
> If the strings happen to be the same length, the Levenshtein distance is
> equivalent to the Hamming distance. The Wikipedia article gives the
> following Python implementation:
>
> # http://en.wikipedia.org/wiki/Hammin...
> def hamdist(s1, s2):
> assert len(s1) == len(s2)
> return sum(ch1 != ch2 for ch1, ch2 in zip(s1, s2))

I'm afraid that it isn't. Using Magnus Lie Hetland's implementation:

http://hetland.org/coding/python/leve...

In [1]: %cpaste
Pasting code; enter '--' alone on the line to stop.
:def levenshtein(a,b):
: "Calculates the Levenshtein distance between a and b."
: n, m = len(a), len(b)
: if n > m:
: # Make sure n <= m, to use O(min(n,m)) space
: a,b = b,a
: n,m = m,n
:
: current = range(n+1)
: for i in range(1,m+1):
: previous, current = current, [i]+[0]*n
: for j in range(1,n+1):
: add, delete = previous[j]+1, current[j-1]+1
: change = previous[j-1]
: if a[j-1] != b[i-1]:
: change = change + 1
: current[j] = min(add, delete, change)
:
: return current[n]
:--

In [2]:

In [3]: def hamdist(s1, s2):
...: assert len(s1) == len(s2)
...: return sum(ch1 != ch2 for ch1, ch2 in zip(s1, s2))
...:

In [4]: hamdist('abcdef', 'fabcde')
Out[4]: 6

In [5]: levenshtein('abcdef', 'fabcde')
Out[5]: 2


--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

Steven D'Aprano

2/7/2008 12:36:00 AM

0

On Wed, 06 Feb 2008 17:32:53 -0600, Robert Kern wrote:

> Jeff Schwab wrote:
....
>> If the strings happen to be the same length, the Levenshtein distance
>> is equivalent to the Hamming distance.
....
> I'm afraid that it isn't. Using Magnus Lie Hetland's implementation:
....
> In [4]: hamdist('abcdef', 'fabcde')
> Out[4]: 6
>
> In [5]: levenshtein('abcdef', 'fabcde')
> Out[5]: 2


I can confirm Robert's results, using a different implementation of the
Levenshtein edit distance. It isn't true that Levenshtein simplifies to
Hamming when the strings are equal length.


For what it's worth, here's my implementation, which I wrote some years
ago. I doubt it's optimal.





def levenshtein(s1, s2):
"""Return the Levenshtein edit distance between two strings.

s1 and s2 should be two arbitrary length strings. Returns an
integer count of the edit distance between the strings.
"""
m, n = len(s1), len(s2)
if m == 0: return n
elif n == 0: return m
# Create an array with rows 0..m and columns 0..n.
array = [None]*(m+1)
for i in range(m+1):
array[i] = [None]*(n+1)
# Initialise the first row to 0..n.
for i in range(n+1):
array[0][i] = i
# Initialize the first column to 0..m.
for i in range(m+1):
array[i][0] = i
# Measure the differences.
for i in range(1, m+1):
c1 = s1[i-1]
for j in range(1, n+1):
c2 = s2[j-1]
cost = int(c1 != c2)
x = array[i][j-1] + 1 # Cell immediately above plus one.
y = array[i-1][j] + 1 # Cell immediately left plus one.
z = array[i-1][j-1] + cost # Cell diagonally above and
# to the left, plus the cost.
array[i][j] = min(x, y, z)
# When done, the bottom-right cell contains the Levenshtein distance.
return array[m][n]




--
Steven

Jeff Schwab

2/7/2008 3:52:00 AM

0

Steven D'Aprano wrote:
> On Wed, 06 Feb 2008 17:32:53 -0600, Robert Kern wrote:
>
>> Jeff Schwab wrote:
> ...
>>> If the strings happen to be the same length, the Levenshtein distance
>>> is equivalent to the Hamming distance.
> ...
>> I'm afraid that it isn't. Using Magnus Lie Hetland's implementation:
> ...

> I can confirm Robert's results, using a different implementation of the
> Levenshtein edit distance. It isn't true that Levenshtein simplifies to
> Hamming when the strings are equal length.

Whoops! My mistake.

Daniel Fetchinson

2/7/2008 7:38:00 AM

0

> Are there any Python libraries implementing measurement of similarity
> of two strings of Latin characters?
>
> I'm writing a script to guess-merge two tables based on people's
> names, which are not necessarily spelled the same way in both tables
> (especially the given names). I would like some function that would
> help me make the best guess.
>
> Many thanks in advance!


Hi folks, just went through this thread and a related one from 2006
and I was wondering what the best solution is for using these string
metrics in a database search. If I want to query the database for a
string or something that is close to it (close being defined by one of
the string metrics discussed above) it seems I have to select each and
every word from the database and compare it with the query word which
is very ineffective.

Very concretely if I have an sqlite db with SQLObject as the ORM what
would be the most effective way of doing such a query?

Cheers,
Daniel

Lee Capps

2/7/2008 1:12:00 PM

0

At 14:01 Wed 06 Feb 2008, agenkin@gmail.com wrote:
>Are there any Python libraries implementing measurement of similarity
>of two strings of Latin characters?
>
>I'm writing a script to guess-merge two tables based on people's
>names, which are not necessarily spelled the same way in both tables
>(especially the given names). I would like some function that would
>help me make the best guess.
>
>Many thanks in advance!

I used difflib.get_close_matches for something similar:

http://docs.python.org/lib/module-di...

HTH.

--
Lee Capps
Technology Specialist
CTE Resource Center


Steve Holden

2/7/2008 1:38:00 PM

0

Matthew_WARREN@bnpparibas.com wrote:
>
>
>
>
>
>
>> On Wed, 06 Feb 2008 17:32:53 -0600, Robert Kern wrote:
>>
>>> Jeff Schwab wrote:
>> ...
>>>> If the strings happen to be the same length, the Levenshtein distance
>>>> is equivalent to the Hamming distance.
>
> Is this really what the OP was asking for. If I understand it correctly,
> Levenshtein distance works out the number of edits required to transform
> the string to the target string. The smaller the more equivalent, but with
> the OP's problem I would expect
>
>
> table1 table2
> brian briam
> erian
>
>
> I think the OP would like to guess at 'briam' rather than 'erian', but
> Levenstein would rate them equally good guesses?
>
> I know this is pushing it more toward phonetic alaysis of the words or
> something similar, and thats orders of magnitude more complex.
>
> just in case,
>
> http://www.linguistlist.org/sp/Softwa...
>
> might be a good place to start looking into it, along with the NLTK
> libraries here
>
> http://nltk.sourceforge.net/index.php/Doc...
>
You could perhaps use soundex to try to choose between different
possibilities with the same Levenshtein distance from the sample.
Soundex by itself is horrible, but it might work as a prioritizer.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.hold...

JKPeck

2/7/2008 2:57:00 PM

0

On Feb 7, 6:11 am, Lee Capps <lca...@cteresource.org> wrote:
> At 14:01 Wed 06 Feb 2008, agen...@gmail.com wrote:
>
> >Are there any Python libraries implementing measurement of similarity
> >of two strings of Latin characters?
>
> >I'm writing a script to guess-merge two tables based on people's
> >names, which are not necessarily spelled the same way in both tables
> >(especially the given names). I would like some function that would
> >help me make the best guess.
>
> >Many thanks in advance!
>
> I used difflib.get_close_matches for something similar:
>
> http://docs.python.org/lib/module-di...
>
> HTH.
>
> --
> Lee Capps
> Technology Specialist
> CTE Resource Center

Algorithms typically used for name comparisons include soundex,
nysiis, and levenshtein distance. The last is more general and
variations are used in spell checkers. You can probably Google for
Python versions. You can find implementations of these comparison
functions at
www.spss.com/devcentral in the extendedTransforms.py module.
(Requires a login but free).

HTH,
Jon Peck