From: bernecky Newsgroups: comp.parallel Subject: Re: Reading and Processing large files. Date: 30 May 1999 23:53:40 GMT Organization: Snake Island Research Inc Approved: bigrigg@cs.cmu.edu Message-Id: <7isj24$r6p$1@goldenapple.srv.cs.cmu.edu> Originator: bigrigg@ux6.sp.cs.cmu.edu Xref: ukc comp.parallel:15633 It looks like you're doing something along the lines of a simple database query. Here are a few possible approaches: a. Two gigs of dram costs (if memory, umm, serves) less than $2000USD. Buy an alpha or SGI 64-bit box and do it in memory. b. Or, consider transposing the array so that you have N array variables, one for each of the N fields. Deal with them one-by-one. c. Use memory-mapped I/O to do the accesses. All of these approaches allow your program to consider the problem, rather than the technology underlying its solution. That is, you should not have to rewrite your program to deal with arrays that are slightly too big for memory. (As opposed to apps in which the working parts of datasets are thousands of times larger than memory.) You might want to look at the work being done in the J and K languages using memory-mapped I/O on arrays. See www.jsoftware.com and www.kx.com. Gbrdman wrote: > I am working on an app now that needs to read a file that is close to > 2 gigs in size. The file is composed of records that are 3570 bytes -- Robert Bernecky bernecky@acm.org Snake Island Research Inc 18 Fifth Street, Ward's Island Toronto, Ontario M5J 2B9 Canada -- Articles to bigrigg+parallel@cs.cmu.edu (Admin: bigrigg@cs.cmu.edu) Archive: http://www.hensa.ac.uk/parallel/internet/usenet/comp.parallel