www.infinidb.org

Hi,
I am loading a file containing 14 million records using the load data local infile syntax
around 40 mins have passed by the query is still executing. Is this normal behaviour for infinidb. The same process happens much faster in the mysql MyISAM engine.

Any help would be welcome
Regards
Sumit

Sumit,

First, thanks for checking out InfiniDB.

Using the cpimport bulk load utility will be your best option to load large quantities of data into InfiniDB. Using batch inserts or load data infile will be the next fastest option. Load rates will be dependent on your data definition and hardware, but in general cpimport will load 10s to 100s of thousands or rows per second. Batch insert and load data infile will typically load 100s to a few thousand rows per second, while individual inserts will typically load 10s of rows per second.

The Performance and Tuning Guide has a section on data load rates that discusses this in further detail. It is available here - www.infinidb.org/downloads/cat_view/71-a...d-user-documentation .

Thanks,
Walt Weeks

Hi Walt,
Thanks for reply to my post. The reason I am using Load Data is because i do set transformations during load time. I am not sure if cpimport will allow me to do that.
Eg: One column is a timestamp. During insert times, i want to populate another column with just the yyyymmdd value.
I can do that using load data not sure if cpimport allows realtime transformations

I had one more question.
I have loaded only 3 gigs worth of data. But the data1 directory of calpont shows it occupying around 26G . Could you help me on that
Sumit

The cpimport program does not currently support such transformations. It will probably be worth the extra effort to pre populate the import file with the transformed data for cpimport to use. The load rates will be significantly faster. I've logged this into our system as a potential future enhancement for cpimport.

On the size of the data files, we currently pre allocate space for 8M rows of data when a table is created. We've received several posts on that topic and we are doing some work to address it in our 1.1.0 alpha release that will be out the middle of this month. We will be reducing the initial footprint for new tables such that smaller tables will not utilize as much disk space. We will also be releasing that functionality in our 1.0.4 maintenance release due in early April.

We appreciate your feedback.

Thanks,
Walt

Thanks Walt for your quick reply.I am having other issues during inserts
and have opened a different thread on that. I am getting BRM copy errors