Hi,

In our production system we have a directory that keeps files. The
average file size is about 90kb and we currently have about 800,000
files in that single directory. There are no subdirectories. This
will be split into subdirectories soon but in the meantime I have to
deal with this huge directory.

The directory lives on a Windows file server and getting console or
direct disk access to this drive is not an option so I have to deal
with it either as a UNC or as a mapped drive. I would like to run a
process every 5 minutes (or so) that copies changed content onto a
local disk for backing up but the problem that I have is that every
time I want to get a list of files off the share (either using the DIR
command or using ROBOCOPY, XCOPY etc) it can take up to 10 minutes to
generate this list. Depending on how the file server feels at the time
this action can actually DENY any other access to the directory.
Obviously in a production system this is unacceptable.

As I understand it, once a directory listing has been read into memory
it will remain there for quick retrieval until the directory contents
changes. This would help, but the contents of the directory is being
updated almost continuously.

Other than suggesting design changes, can anyone offer any pearls of
wisdom with regard to tackling this problem?

Thanks,

Neville.

<neville.whyte@gmail.com> wrote in message
news:1136942141.341012.276820@g49g2000cwa.googlegr oups.com...
You're probably asking for the impossible. Running a process
once every 5 minutes in a dynamic environment is fine except
when the process takes 10 minutes just to compile a directory
listing.

It really makes no sense to scan a folder with 800,000 entries
for two or three changes that might have taken place in the last
five minutes.

I think you have to grab this one straight away and restructure
the process so that the amount of uselessly repetitive scanning
is reduced to an absolute minimum. Perhaps a shadowing
process would be most effective.