www.infinidb.org

Hi,
While inserting rows into the table. I am getting the following errors

1. DBD::mysql::db do failed: CAL0001: Insert Failed: CAL0001: Insert failed on the impressions_dfp table. It was due to: a BRM Begin copy error.


2. DBD::mysql::db do failed: IDB-2008: The version buffer overflowed. Increase VersionBufferFileSize or limit the rows to be deleted.

Can someone help me with this?
Regards
Sumit

Sumit,

There was a bug that we fixed in 1.0.3 that is likely the cause for the BRM error -
bugs.launchpad.net/infinidb/+bug/524157 . If a prior load data infile was done on the table with 1.0.2 or an earlier version, this error can occur. To remedy, the table will need to be dropped and reloaded using the 1.0.3 release.

There is a configurable available to increase the size of the version buffer. The default setting is 4 files sized 1 GB each for a total of 4 GB. To increase, edit the VersionBufferFileSize setting in /usr/local/Calpont/etc/Calpont.xml and restart InfiniDB. You may also consider keeping the setting as is and breaking up your data into multiple loads.

I know from our discussion last week that you are using load data infile rather than the bulk load utility because of it's support for data transformations. You may want to try an experiment doing a cpimport without the transformation to get an idea of the performance benefit. It doesn't have the overhead of versioning and is multi-threaded and it will be significantly faster.

Thanks for your feedback and please let us know if we can be of further assistance.

Walt

Hi Walt,
Thanks for the reply. I will upgrade to the 1.0.3 version and see if this happens again
Also, I understand the value of the bulk import. But then I would have to do the transformations from an update statement so I wont get the performance benefit overall.
Do you have any statistics for updating say 14 million records?
Sumit

Here are some results from early this morning from some of our tests that run every night. This test updated 16 million rows for two ints and a varchar column and finished in just under a minute.
--------------
update orders set o_comment='whatever', o_custkey=44, o_shippriority=8 where o_orderkey <= 64000000
--------------
Query OK, 16000000 rows affected (59.03 sec)
Rows matched: 0 Changed: 0 Warnings: 0

However, we currently only support col=value updates in 1.0. We are adding support for col=col updates and col=virtual col updates for the 1.1 release. That feature should be available for evaluation in the 1.1.2 alpha release due out in mid May. Another possibility may be to do the tranformation prior to the import so that the import file contains the transformed column.

Here's a link to our roadmap - www.infinidb.org/resources/tech-articles...nity-edition-roadmap .

If it would be possible for you to post the load data statement with the transformation or the update statement that you would potentially use that would be helpful. Thanks.

Hi Walt,
This is really awesome stat. I will move over to the cbimport bulk load method. This will really take away some of the query load hit i am facing.
Also, I upgraded to the 1.0.3 version
now when i am trying to drop/create tables the script is just stuck
Sumit

Sumit,

Start by stopping and restarting the infinidb service and making sure things are up okay. Please then run the statements individually and if you still see the issue please post the statement along with what you are seeing and we'll go from there.

On the prior post, I want to make sure the message got across that we do not currently support the update transformation that it sounds like you'll need. Support for updating a column based on another column will be coming in 1.1. If you could post the update statement that would be helpful. I want to make sure the function that you would be using is one we currently support.

Thanks.