[lnkForumImage]
TotalShareware - Download Free Software

Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

Strange problem: MySQL and python logging using two separate cursors

Frank Aune

1/9/2008 9:11:00 AM

Hi,

Explaining this problem is quite tricky, but please bear with me.

Ive got an application which uses MySQL (InnoDB) for two things:

1. Store and retrieve application data (including viewing the application log)
2. Store the application log produced by the python logging module

The connection and cursor for the two tasks are completely separate, and they
connect, query, execute and commit on their own. But they do use the same SQL
database.

Task 1 typically consist of alot of SQL queries.

For task 2 I'm using the logging_test14.py example from the logging module as
a template. (The only thing I've added basically is an escape_string() call
to properly escape any SQL queries from task 1 logged by task 2 to the
database.)

From a MySQL shell I can view the logging updates as they are commited, eg.
the log continues to grow when using the application.

But if I try to do the same from inside the application, the cursor in task 1
will only return between 50 and 60 logentries, even though more updates
exists (I can still view them as they grow from the MySQL shell). If I try to
re-run the same query, the same 50-60 logentries are returned. No error, no
message - nothing.

To recap: task 2 writes all log messages to the database, and task 1 reads
these same log messages based on user input and present them in a GUI.

I don't know if this is explained well enough, but its kind of tricky
explaining such weird behaviour.

The only clue I have so far, is that the cursor in task 1 seems to be unable
to "register" any new entries in the log table produced by task 2 as soon as
task 1 perform an SQL query of some kind.

Im contemplating using the same cursor for task 1 and 2, but I think keeping
them separate is a better design - if it only worked :)

Any input on this "nutcracker"?

Thanks,
Frank
5 Answers

Dennis Lee Bieber

1/9/2008 5:52:00 PM

0

On Wed, 9 Jan 2008 10:11:09 +0100, Frank Aune <Frank.Aune@broadpark.no>
declaimed the following in comp.lang.python:

> The only clue I have so far, is that the cursor in task 1 seems to be unable
> to "register" any new entries in the log table produced by task 2 as soon as
> task 1 perform an SQL query of some kind.
>
How often do you issue a commit? For some DB-API adapters (I forget
which database -- think it was SQLite) a select query does not complete
until the last data has been fetched from it -- meaning the transaction
(the DB-API spec is that auto-commit is OFF) is still open and "other
transaction changes" will not be seen. {I do find it perplexing that
transactions are started by cursor actions, but committed by the
connection!}

> Im contemplating using the same cursor for task 1 and 2, but I think keeping
> them separate is a better design - if it only worked :)
>
I'd probably suggest using a separate connection and cursor -- with
liberal usage of conn.commit() to ensure that transaction "views" are
flushed/refreshed.
--
Wulfraed Dennis Lee Bieber KD6MOG
wlfraed@ix.netcom.com wulfraed@bestiaria.com
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: web-asst@bestiaria.com)
HTTP://www.bestiaria.com/

Frank Aune

1/11/2008 10:01:00 AM

0

On Wednesday 09 January 2008 18:52:02 Dennis Lee Bieber wrote:
> On Wed, 9 Jan 2008 10:11:09 +0100, Frank Aune <Frank.Aune@broadpark.no>
>
> declaimed the following in comp.lang.python:
> > The only clue I have so far, is that the cursor in task 1 seems to be
> > unable to "register" any new entries in the log table produced by task 2
> > as soon as task 1 perform an SQL query of some kind.
>
> How often do you issue a commit?

I experience the behaviour for task 1 even if the select query only reads out
data and no commit is needed.

Do I really need to perform commits on a handler even though it only reads out
data? From a MySQL shell I can see the changes from the other handler without
the commits, but afaics that shouldnt be the case if the above were true.

Thanks,
Frank

Dennis Lee Bieber

1/11/2008 6:49:00 PM

0

On Fri, 11 Jan 2008 11:00:54 +0100, Frank Aune <Frank.Aune@broadpark.no>
declaimed the following in comp.lang.python:

> I experience the behaviour for task 1 even if the select query only reads out
> data and no commit is needed.
>
DB-API spec is that auto-commit is OFF... Even a read-only (select)
operation begins a transaction, and any decent DBMS will "freeze" the
view of the tables specified until a commit or rollback.

> Do I really need to perform commits on a handler even though it only reads out
> data? From a MySQL shell I can see the changes from the other handler without
> the commits, but afaics that shouldnt be the case if the above were true.
>
That shell is probably acting in an autocommit mode (I believe that
is MySQL default behavior).
--
Wulfraed Dennis Lee Bieber KD6MOG
wlfraed@ix.netcom.com wulfraed@bestiaria.com
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: web-asst@bestiaria.com)
HTTP://www.bestiaria.com/

Chris Mellon

1/11/2008 7:06:00 PM

0

On Jan 9, 2008 11:52 AM, Dennis Lee Bieber <wlfraed@ix.netcom.com> wrote:
> On Wed, 9 Jan 2008 10:11:09 +0100, Frank Aune <Frank.Aune@broadpark.no>
> declaimed the following in comp.lang.python:
>
> > The only clue I have so far, is that the cursor in task 1 seems to be unable
> > to "register" any new entries in the log table produced by task 2 as soon as
> > task 1 perform an SQL query of some kind.
> >
> How often do you issue a commit? For some DB-API adapters (I forget
> which database -- think it was SQLite) a select query does not complete
> until the last data has been fetched from it -- meaning the transaction
> (the DB-API spec is that auto-commit is OFF) is still open and "other
> transaction changes" will not be seen. {I do find it perplexing that
> transactions are started by cursor actions, but committed by the
> connection!}
>
> > Im contemplating using the same cursor for task 1 and 2, but I think keeping
> > them separate is a better design - if it only worked :)
> >
> I'd probably suggest using a separate connection and cursor -- with
> liberal usage of conn.commit() to ensure that transaction "views" are
> flushed/refreshed.

The MySql api doesn't have a concept of a cursor, only connections. If
you want truly separate cursors in MySql you need to use individual
connections.

Ed from AZ

2/3/2010 2:10:00 PM

0

I just realized one of the major reasons I included Word in this mix.
I am pulling the data into an Excel sheet so I can add to it. I then
need to write the modified data back into the XML file.

Not being familiar with XML or XPath, my thought train was:
-- Read XML file via FSO and paste into Word
-- Find the major groups and sub-groups and set bookmarks
-- Pull the data into Excel by parsing the tags and text
-- Add new data and modify existing data as required
-- Create an array of each sub-group and group, wrapping the data in
the proper tags
-- Replace the contents of the associated bookmark range with the new
data
-- Write the whole kaboodle back into the XML file

So the hiccup in my thinking was how to get the data from Excel back
into the XML file. All I could think of was using the bookmarks and
ranges in Word.

Is there a better way?
Ed


On Jan 28, 12:26 pm, Chip Pearson <c...@cpearson.com> wrote:
> You can read the XML file directly using the MSXML library.  In VBA,
> go to the Tools menu, choose References, and scroll down to "Microsoft
> XML, v6.0" and check that item.  The code uses some relatively simple
> XPath to select the nodes, so I assume that you (1) already know
> XPath, or (2) can figure it out, or (3) can ask for assistance with
> XPath.
>
> Without know the actual layout of your XML, I just guessed and made up
> the following XML file:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <MAIN_CATEGORY>
>         <DATA_PAGE num="001">
>                 <GRID_LIST>
>                         <LINE num="001">
>                                 <name>Chip Pearson</name>
>                                 <city>Kansas City</city>
>                         </LINE>
>                         <LINE num="002">
>                                 <name>Jen Filson</name>
>                                 <city>Mission Hills</city>
>                         </LINE>
>                 </GRID_LIST>
>         </DATA_PAGE>
>         <DATA_PAGE num="002">
>                 <GRID_LIST>
>                         <LINE num="003">
>                                 <name>Janice Downey</name>
>                                 <city>Westwood Hills</city>
>                         </LINE>
>                         <LINE num="04">
>                                 <name>Mike Stitt</name>
>                                 <city>Charlotte</city>
>                         </LINE>
>                 </GRID_LIST>
>         </DATA_PAGE>
> </MAIN_CATEGORY>
>
> Then, you can use code like
>
> Sub AAA()
> ' Required reference:
> '   Name: MSXML2
> '   Description: Microsoft XML, v6.0
> '   Typical location: C:\Windows\System32\msxml6.dll
> '   GUID: {F5078F18-C551-11D3-89B9-0000F81FE221}
> '   Major: 6    Minor: 0
>
> Dim DOM As MSXML2.DOMDocument60
> Dim FName As Variant
> Dim DataPageList As MSXML2.IXMLDOMNodeList
> Dim DataPage As MSXML2.IXMLDOMNodeList
> Dim GridList As MSXML2.IXMLDOMNodeList
> Dim Grid As MSXML2.IXMLDOMNodeList
> Dim LinesList As MSXML2.IXMLDOMNodeList
> Dim LineX As MSXML2.IXMLDOMNode
> Dim Arr() As String
> Dim N As Long
> Dim M As Long
> Dim P As Long
> Dim K As Long
> Dim R As Range
> Dim J As Long
>
> Set R = Range("C3") '<<<<< OUTPUT ON WORKSHEET
> FName = Application.GetOpenFilename("XML Files (*.xml),*.xml")
> If FName = False Then
>     Exit Sub
> End If
> Set DOM = New MSXML2.DOMDocument60
> DOM.Load CStr(FName)
> DOM.setProperty "SelectionLanguage", "XPath"
>
> ' get all data pages
> Set DataPageList = DOM.SelectNodes("/MAIN_CATEGORY/DATA_PAGE")
> For N = 0 To DataPageList.Length - 1
>     Set GridList = DataPageList.Item(N).SelectNodes("./GRID_LIST")
>     For M = 0 To GridList.Length - 1
>         Set LinesList = GridList.Item(M).SelectNodes("./LINE")
>         ReDim Arr(0 To LinesList.Length - 1)
>         K = 0
>         For P = 0 To LinesList.Length - 1
>             Set LineX = LinesList.Item(P)
>             Arr(K) = LineX.ChildNodes(0).Text & vbCrLf & _
>                 LineX.ChildNodes(1).Text
>             K = K + 1
>         Next P
>         '<< write to worksheet
>         For J = 0 To UBound(Arr)
>             R.Offset(0, J).Value = Arr(J)
>         Next J
>         Set R = R.Offset(1, 0)
>     Next M
> Next N
> End Sub
>
> This lists the contents between the <LINE> and </LINE> tags out to the
> worksheet.  You'll have to modfy the code a bit to conform with your
> actual data structure, but the code will be very muich like that shown
> above.
>
> Cordially,
> Chip Pearson
> Microsoft MVP 1998 - 2010
> Pearson Software Consulting, LLCwww.cpearson.com
> [email on web site]
>
> On Mon, 25 Jan 2010 14:21:20 -0800 (PST), Ed from AZ
>
>
>
> <prof_ofw...@yahoo.com> wrote:
> >This is posted to both the Word and Excel VBA NG because so far that's
> >the only way I can figure out to do this.  But wait - I throw in FSO
> >to make it even messier!  Using Word and Excel 2007.
>
> >We have a proprietary program that saves its working files as XML.
> >When a user needs to add a list of data, he's presented with either a
> >grid format that's a pain to work with, or a single-form-per-item page
> >that's easier to use but wearying after 35+ items.  What I want to do
> >is get the part of the XML to be revised into an Excel worksheet so
> >the user can work easier.  Because this program is proprietary, the
> >schema is not available to me, and since I'm not familiar with XML
> >except as text separated by tags, I wouldn't know how to use it.
>
> >So my bright idea so far is to open the file for reading in FSO, then
> >paste that into a new Word doc.  This gives me a text file with XML
> >tags, but without Word's interpretation of the XML.  Direct import
> >into Excel causes Excel to interpret the tags and buries the
> >information I want to get.  Parsing the text in Word, I can set ranges
> >using the different tags and work my way down to the info.  In my
> >current plan, I'm thinking of driving this from Excel.
>
> >The data is set up like so:
> ><MAIN_CATEGORY>
> ><DATA_PAGE num="0001">
> >-- page info --
> ><GRID_LIST>
> ><LINE num="0001">
> >-- line info --
> ><LINE num="0002">
> >-- line info --
> >etc.
>
> >The data I want is between the <LINE> tags.
>
> >I could get all the data I need by looping through and reading each
> >data point from <LINE> to </LINE> into an array, then writing that
> >back into Excel.  Or I could simply capture each <GRID_LIST> into a
> >range, save it out as a separate temp XML file, then import that into
> >Excel.
>
> >Is this really the best way to do this??  Or is there a much easier
> >way?  Any recommendations?
> >Ed-