I want to run a program that continually processes > 9 GBytes of data
located in binary files. Now, the program reads in the data as needed.

Is there any sort of "External RAM" that can be loaded, and treated from
within a program as if it is a hard drive (i.e using C statements like
fopen() and fread() ) using a regular 32 bit PC?

I don't care that much about increasing speed. I don't like having to
continuously access the hard drive non-stop for days at a time.

KRK wrote:

If you have the main memory, use something called RAM drive or
Ramdisk. There are lots of products.


Martin

--
Quidquid latine dictum sit, altum viditur.

On Tue, 27 Sep 2005 15:33:52 UTC, "KRK" <karlk@eng.umd.edu> wrote:

RAM Drives and Virtual Memory can make this task easier, but they both
hold a portion of the data in memory and the rest on your normal external
storage -- the disks. You may be able to speed your processing up and
reduce the activity on your hard drives by partitioning the problem such
that the normal RAM is used more than the drives.

Another method is prefetching data. If you have to repeatedly scan
9GB of data and you are only reading this in small chuncks, that is a
poor use of your resourses. Read in much larger quantities of data
(perhaps a 1GB into RAM) and perform as much work on that data at once.

Adding a little multithreading might also optimize the process a bit.
One or more of the threads processes data in memory while another
thread gets the next batch of data from the hard drive.

In general, adding more RAM to your system, and making proper use of
it, will greatly increase your performance with the least cost. A
little extra thought and a better processing method might take that
multi-day process and reduce it to an hour or less. I guess it all
depends on what you are trying to do.

David

Martin Eisenberg wrote
(in article <1127839821.553969@ostenberg.wh.uni-dortmund.de>):

Let me know when you locate a ram drive for a 32-bit box that
supports 9GB ramdisks.




--
Randy Howard (2reply remove FOOBAR)

KRK wrote
(in article <dhboou$b6d$2@grapevine.wam.umd.edu>):

If you only read the data in as needed, then the total amount
isn't an issue, provided you free up that you're no longer
using.

The file system has quite sophisticated buffer cache already,
trying to reinvent that wheel (on top of the existing one) is
unlikely to be of much benefit.

Unless you can reorder your disk operations to benefit the
cache? For example, if you are reading and writing data from
random locations in the binary files, can you do it sequentially
instead?

Try describing the file operations you are performing (nevermind
the calculations on the data). If the drive is churning
constantly, then your application is I/O bound, and you need
faster drives (or perhaps a RAID 0 working set across multiple
drives).



--
Randy Howard (2reply remove FOOBAR)

Randy Howard <randyhoward@FOOverizonBAR.net> writes:

If your 32-bit box supports 9 GB RAM, then there's little reason
to believe it wouldn't support 9 GB RAM disks. Why not?
--
Ben Pfaff
email: blp@cs.stanford.edu
web: http://benpfaff.org

Ben Pfaff wrote
(in article <878xxhc27s.fsf@benpfaff.org>):

Then it should be easy for you to name one. :-)





--
Randy Howard (2reply remove FOOBAR)

"KRK" <karlk@eng.umd.edu> wrote in message
news:dhboou$b6d$2@grapevine.wam.umd.edu...
Yes, eg Gigabyte i-RAM.

There are several things you may be able to do to reduce access and/or make
the access pattern more "drive friendly":
- Fit more RAM.
- Change the algorithm.
- Change the implementation, eg use memory mapped files.
- Use multiple drives (independent, not a RAID setup).

Alex

Randy Howard <randyhoward@FOOverizonBAR.net> writes:

A brief scan through the Linux 2.6 ramdisk driver doesn't show
any arbitrary limits. It's based out of the page cache, which is
itself not limited to 4 GB on 32-bit boxes.
--
Ben Pfaff
email: blp@cs.stanford.edu
web: http://benpfaff.org

Ben Pfaff wrote
(in article <87wtl19swi.fsf@benpfaff.org>):

http://www.ussg.iu.edu/hypermail/lin...07.0/0586.html


--
Randy Howard (2reply remove FOOBAR)

Randy Howard <randyhoward@FOOverizonBAR.net> writes:

Hmm. Well, that's disappointing, but there are two approaches to
dodging the problem mentioned in that thread: use ramfs instead
of a ramdisk, or concatenate multiple 1 GB ram disks using a raid
driver. Either one would allow the effect of a 9 GB ram disk.
--
"The sound of peacocks being shredded can't possibly be
any worse than the sound of peacocks not being shredded."
Tanuki the Raccoon-dog in the Monastery

Ben Pfaff wrote
(in article <87oe6d9re5.fsf@benpfaff.org>):


But about what I expected when I went googling. 32-bit
platforms rarely deal with anything above 4GB and sometimes 2GB
in a consistent way. If you want to make effective use of 10GB
of RAM, you run a 64-bit OS. To do otherwise may be a fun
afternoon diversion, but apart from that it's just silly.

Something other than a ramdisk, and have you tried it? I don't
have a spare 10GB system laying around to play with.

Nevermind that the Linux fs buffer cache achieves most of this
on its own, especially in a machine with lots of free ram, for
no extra cost in terms of work or overhead.

The folks in Windows land aren't so lucky, but that has always
been the case, and is likely to remain so.

Actually, this one is likely to have some measurable overhead,
if it does work. At the end of the day, no great answer is
available on 32-bit boxes.




--
Randy Howard (2reply remove FOOBAR)

Randy Howard <randyhoward@FOOverizonBAR.net> writes:

It's certainly true that the best choice would be to spend $300
for an AMD64 box.
--
"I admire him, I frankly confess it; and when his time comes
I shall buy a piece of the rope for a keepsake."
--Mark Twain

Ben Pfaff wrote
(in article <87oe6dhy51.fsf@benpfaff.org>):

I didn't realize there were many $300 solutions that came with
9GB ++ of RAM installed. If you know of one, I'd appreciate a
link to order one. :-)

Nevermind finding a $300 box with enough slots to even populate
that much memory, provided you had the memory already available
to you some other way.


--
Randy Howard (2reply remove FOOBAR)