Bill D
10/29/2008 1:43:00 PM
This could add a whole new level of complication to your system, but we
handled a similiar priority requirement using a messaging system. In out case
we used SQL Server's Service Broker, but I'm sure what we did could be
replicated with anyone. We have 2 queues, one is a low priority queue and has
one reader (think thread) at the end of it. The other is a high priority
queue and has several readers at the end so that it can process more data.
The application must determine which messages are high and low priority, but
after that most of the multi threading aspect is handled by the messaging
system. This also allows us to dynamically increase out processing
capabilities when the need arises by increasing the number of readers on the
queue. Another benefit of this is that the messages are garaunteed to be
delivered once they are in the queue.
On the other hand in an environment that does not already use messaging
this introduces allot of new pieces that can and will break and will need to
be monitored.
Bill
"Peter Duniho" wrote:
> On Tue, 28 Oct 2008 10:58:18 -0700, David
> <david.colliver.NEWS@revilloc.removethis.com> wrote:
>
> > [...]
> > Once the file has been saved, I need to then work on it. The process can
> > become quite intensive. Also, each file can come from a different
> > customer
> > and each customer will have a 'priority' preference, for example, some
> > files
> > have to be acted upon immediately (due to them being time critical) where
> > others that are not so critical can wait.
> >
> > My thoughts to handle this is a windows service with a filewatcher to
> > watch
> > the webservice receiving folder. Then, depending on the file (customer),
> > react to it...
> >
> > I guess I should be doing threading here (I have not really done any
> > threading before...) but I do have to bear in mind priority files. Also,
> > if
> > there are for example, 3 files from one phone, they have to be handled
> > chronologically.
> >
> > [...]
> > Ideas on how I should approach this would be very much appreciated.
>
> The question is very broad, and this is probably not the best forum for a
> deep discussion about your options.
>
> That said, some thoughts about what you've posted:
>
> -- Don't use FileSystemWatcher unless you really have to. It's a
> useful class in certain scenarios, but it isn't 100% reliable. If file
> system changes happen too rapidly for your own code to keep up,
> notifications may be discarded. It's extra work to make it reliable --
> basically you wind up having to do your own scans to double-check that you
> haven't missed anything -- and if you do that, then the apparent
> simplicity of using FileSystemWatcher winds up negated.
>
> Since you appear to have control over the code that's receiving the files,
> you will be much better off just managing your own queue of work based on
> what files you know you've received, rather than relying on the file
> system as your mediator between code modules.
>
> -- The prioritization feature has the potential for really causing
> problems. It's not really clear from your description what prioritization
> would be used, but hopefully it's something relatively simple. If it's
> okay for starvation to occur, then a simple ordered queue is probably a
> good bet. One thing you're vague about is whether priorities are for
> manging files for a specific user only, or whether users themselves have
> individual priorities that affect which user's files get processed first.
>
> A possible fallback to you implementing all the prioritization logic
> yourself would be to dedicate individual threads to each priority class
> you have, setting the thread priorities appropriately. I almost always
> recommend _against_ setting thread priorities, but in this case, as long
> as you can categorize your priorities in a reasonably few number of
> groups, it might actually work in your favor to set the thread priorities
> and then let Windows do the heavy lifting for you. Its thread scheduler
> has lots of little details in it to ensure that prioritization happens
> correctly without starving work and while maintaining a consistent flow.
>
> Other than that, I'd say that without a much more detailed description of
> your architecture,it would be difficult or impossible to offer any really
> useful design advice.
>
> Pete
>