We are pleased to announce the availability of the 2.2.6 version of Calpont InfiniDB Community.  This is our latest maintenance release and it includes a number of bug fixes that you can see in the release notes and at http://bugs.launchpad.net/infinidb.  You can download the latest InfiniDB binaries, source code, and updated documentation at http://infinidb.org/downloads.  For those interested in scale-out and physical compression, please download InfiniDB Enterprise 2.2.6 at http://www.calpont.com/products/tryinfinidb.

As always, thank you for your support of Calpont InfiniDB and we welcome your feedback.

Andy Hayler released his 2011 Information Difference Data Warehouse Landscape Survey on Feb 10. This is the second year Calpont has been included in the survey and we’ve made big strides over 2010, moving up the Technology axis of the landscape, surpassing both Infobright and Greenplum.

Data Warehouse Landscape courtesy and copyright of Information Difference

http://www.informationdifference.com/dw-landscape.html

The recognition of InfiniDB’s progress in the larger data warehouse market is significant considering that the commercial product has been in the market less than two years. The real testament here, however, is the response Information Difference received from Calpont customers during the survey, which resulted in Calpont’s customer satisfaction characterization as follows:

“Based on this survey, the data warehouse vendor with the happiest customers in 2011 was Teradata, followed by Calpont, then IBM, followed by Kognitio and Kalido.”

What can we say to our customers, but a huge “Thanks!” The feedback on InfiniDB in all circles has been tremendous, and we look forward to increasing our position on the Information Difference Data Warehouse Landscape in 2012 by continuing to evolve our product set and making customer satisfaction job one.

In a recent Wall Street Journal opinion article, Big Data and the wireless Communications revolution are cited as two of the three breakthroughs poised to transform businesses this century. The combination underscores the battle that’s playing out between Web and Communications companies for the consumer. Most service providers are inundated with an explosion of data traffic in their network, undermining profit as traffic grows. What’s more, most of this data and usage traffic are not correlated in a way that can be utilized to provide improved services, customer experience or lower network costs. To solve this problem, service providers are embracing real-time analytics. Telecom analytics will continue to play a major driver in understanding the increasingly mobile customer base. Understanding customer behavior helps a company provide services that are more personalized, thus increasing customer loyalty. As the market evolves and new products are launched, analysis becomes critical to understand the tariffs, product and service migrations, customer profitability and loyalty.

The proliferation of online data intensive applications are driving hundreds-of-billions in advertising and digital content revenue; over 7.5 billion smart devices accessing the Internet by 2015; and smartphone data network traffic increasing to 56% of total traffic in 2015, driving hundreds of petabytes in traffic per month. The telecommunications industry is at the forefront of the ‘big data’ wave. Online interactive marketing spend is growing a whopping 17% CAGR through 2014 to 21% of the total advertising budget. There are tremendous opportunities for Operators to convert their capital intensive businesses in to new and exciting sources of revenue, offsetting the rapid deterioration in network profits with the staggering increase in data and traffic. Telecom operators worldwide have spent billions of dollars on business intelligence (BI) software, services and systems integration over the past few years, but have a long way to go. Communications is the third-fastest growing industry sector in BI and analytics with Gartner predicting a 9% annual growth rate over the next five years.

Analytics guru Dan Baker of the Technology Research Institute puts the coming changes in communications in context with the last two decades: “we’re now in the decade of analytics, typified by software used to analyze, audit, and assure operations in greater detail”. The progression we’ve seen over the previous two decades has been extraordinary, but it’s only a fraction of what’s coming. Let’s take a look: Voice CDRs were relatively easy to track and store. Then when SMS came on board, we went from say 20 CDRs per user, per day, to 100 mixed xDRs per day. Now as you add mobile data to the mix, we’re talking about another 10-100x increase, in terms of the number of events that are going through each device. And with video content, we’re talking about millions of network IP protocol events that make up a single customer facing event from a mobile TV sporting event. A smartphone, by its very nature, is on all the time. And depending on a carrier’s business model, you need to analyze a majority of that subscriber’s activity just to put it all into perspective.

Telecom data is complex. Companies work hard to make sense of data from multiple sources, including billing systems, customer service applications, network activity and thousands of product and activity codes. Without consistent data and analytics, companies find it nearly impossible to produce an accurate picture of the business. John Myers, founding principal and senior strategist for Blue Buffalo Group, a Colorado telecom-focused business intelligence (BI) consultancy, put it in context: “while we in the telecom industry certainly have made a lot of progress in terms of business analytics relating to Revenue Assurance and Fraud Management systems in the past few years, we still have a long ways to go in terms of adding true business intelligence analytics to those practices”.

The rapid pace of services growth sprouting from the Internet has forced operators to rethink their business models and role of online analytics. Operators need to prepare for the huge event record volumes that are here now, and will only continue to grow with mobile broadband. Extending current data management and legacy warehousing environments is not a feasible solution when considering massive and ever-increasing volumes of data that must be analyzed. The future of the telecommunications industry is on the precipice of a new vision for the new, online economy players with a strategy based on an ‘application aware analytics platform’. Operators can offer online businesses and marketing professionals a predictable and cost-effective way to reach and engage their online consumers in a “personalized” way, no matter the medium. To support this, the data infrastructure needs to include a specialized database that can support the data volumes, concurrency, load rates and query complexities involved.

Increasingly, companies in the telecommunications industry such as Guavus, 1&1 Internet, Bandwidth.com and JiWire, are turning to Calpont’s InfiniDB analytics database to handle the growth in data volume in a high performance manner. InfiniDB is a purpose-built, highly reliable database with Columnar and MPP technologies that are essential for massive, big data on this scale. It’s infinitely scalable, highly flexible architecture can deliver exceptional query performance at any data scale in an economically sensible way. InfiniDB allows Communications providers to load and analyze complex BI queries against billions of CDRs in real-time at a fraction of the cost of legacy systems.

If you missed it, one of the biggest 2011 events surrounding Big Data was the NoSQL Now Conference, which took place August 23-25 in San Jose. William McKnight, President of the McKnight Consulting Group, presented on columnar database technologies for Big Data analytics. (He also authored a white paper on the Best Practices for Columnar Databases, which you can find here).

As William discussed, the columnar approach dramatically speeds up the performance of analytic queries by minimizing I/O to disk. Though not new (column-oriented DBs have been in play since the 1970’s), the emergence of large data and the need for “real time” analytics to become a reality has spurred the development of columnar and row-column hybrid data engines. The columnar paradigm has proven to be the most efficient in addressing Big Data analytics. In fact, columnar databases have become so popular that they are on the cusp of hitting critical mass within the market, rounding out the last phase in visibility and maturity on the Gartner hype cycle.

2011 Gartner Hype Cycle for Data Management

Columnar databases benefit an enterprise in a multitude of ways, making it clear that they’ve pushed past the “slope of enlightenment” and are into mainstream adoption. Query performance for large data sets, the obvious value add, has been so much greater than that of OLTP and row OLAP dbs, hardly a comparison is warranted.

BlackBox-BI
Wow @Calpont @InfiniDB is 85 times faster than @Oracle on a 1B record table. The perfect start to our p.o.c.
25 Aug via web

Traditional RDBMS have also struggled with data load performance, also picked up nicely by columnar databases. These two in combination have enabled an expansion of dimensional analyses within the same data warehouse scheme (i.e., without parsing into separate federated marts) while performing as needed, at scale. They’ve also enabled improved segmentation of data for analysis (i.e. aggregations).

Earlier this year, I sponsored a primary research study in the use of analytic databases for telco and online media organizations. Over 95% of my respondents were familiar with columnar databases, with a good number of those planning to evaluate for use within their organizations for dimensional business analytics and non dimensional predictive/data mining analytics, either as an augmentation or replacement for legacy database systems. (In fact, why build MOLAP cubes on top of columnar-based relational star schemas if the stars will run faster than the cubes whether sparsely or densely populated? Keep the semantic layers simple.)

So now that you know what columnar databases are and why they are important, what do you know about InfiniDB and why should you be interested? You may be surprised to know that InfiniDB is not only a MPP columnar analytic database, but also converts SQL (MySQL to be exact) statements into map and reduce operations to execute queries of massive size. So, not only does it provide the benefit of a 100% columnar design (which is, by the way, more efficient than row columnar hybrids from an I/O perspective), it also enables SQL for Big Data (i.e., NoSQL = Not only SQL, providing the best of both worlds).

Although not a Hadoop implementation, InfiniDB’s map reduce style execution provides the best of query performance and scale in one environment - fully distributed, and parallel, and also right “out-of-the-box”. Add automatic partitioning, compression, partition drop, and db tuning, then InfiniDB is one of the easiest high performance data solutions on the market to own and use.

With the inevitable rise in Hadoop use, we continue to look for ways to enhance our product and anticipate the needs of data administrators, developers, and users looking to leverage other data environments and tools. This is why Calpont created the InfiniDB-Hadoop data connector, which transfers data to/from InfiniDB and the Hadoop Cluster. It removes the heavy lifting needed to leverage Hadoop data for low latency analytics of Big Data, providing a perfect complement to a heterogeneous Hadoop environment.

As was made clear by William’s presentation and other industry talk at the NoSQL event, Big Data continues to grow bigger and bigger, with need for the right data infrastructure to tackle the needs of the enterprise. A “one size fits all” approach to data processing and analytics just won’t work with today’s varied workloads. In fact, Forrester research indicates that the term we’ve all become familiar with, “Big Data,” will soon likely be replaced by “Any Data”, as ALL data is becoming of huge importance for enterprises the world over. With InfiniDB, we’ve got Any Data analytics covered, now and in the future.

This post is the third in a three-part series covering InfiniDB's hallmark characteristics: I/O efficiency, parallelism and ease-of-use.

When we set out to design InfiniDB, we wanted to build a database that was extremely easy-to-use. Historically, companies have required DBAs and enterprise IT to be closely engaged to enable analytics within their organizations.

We wanted analysts and data scientists to be able to use InfiniDB without a significant burden on precious DBA resources. As such, we designed InfiniDB with the hugely popular MySQL interface, index-free, and with features that extend MySQL for analytics. Let’s examine each of these.

Of the many database solutions on the market, InfiniDB is the only MPP columnar database tightly integrated with MySQL. This means that, for tens of thousands of MySQL applications that have large-scale data, InfiniDB is the tool-of-choice for structured Big Data analytics. No need to learn a new language or rewrite code: just install InfiniDB and you can be executing complex commands on massive datasets in minutes.

InfiniDB is designed to minimize performance tuning. Gone are the days when DBAs have to create indexes or other materialized views to get blazing-fast performance. Through its unique Extent Map architecture, paired with automatic horizontal and vertical partitioning, InfiniDB scales effortlessly without technical tweaking.

In fact, we’ve even added features that make MySQL easier to use for analytics. For example, InfiniDB fully supports online DDL (i.e. one session can be adding columns to a table while another session is querying that table) using calonlinealter. This is a feature that’s currently not supported in standard MySQL.

With enhanced I/O, record parallelism and unparalleled ease-of-use, InfiniDB is the top choice for helping take control of your data for deep analytics.

Coming soon is InfiniDB 3, which will feature greater storage architecture flexibility, enabling it for massive cloud deployments. 2012 will be a tremendous year for Big Data analytics and InfiniDB!

As part of a second post in our three-part series, we'd like to discuss the importance of parallelism and how InfiniDB uses parallelism to achieve breath taking performance.

InfiniDB is architected for effortless scalability.

The InfiniDB database is comprised of two types of modules: User Modules and Performance Modules. User Modules interpret MySQL commands and convert them into parallelized code for execution.

Performance Modules execute the parallelized code and return the result to the User Modules. Performance Modules are architected to be lightweight and to execute in a MapReduce-fashion, similar to Apache Hadoop. (To be clear, though, Performance Modules do not use Hadoop code. In fact, we started designing this architecture well before Hadoop/HDFS’ adoption.)

Our customers typically have one or a few User Modules, although customers may use any number of User Modules to provide high-availability and workload balancing. In a distributed environment, one or more User Modules may be interacting with any number of Performance Modules.

Due to InfiniDB's Map-Reduce-like scale-out architecture, our Performance Modules execute requests extremely effectively.   Each thread within the distributed architecture operates independently, avoiding thread-to-thread or node-to-node communication that can cripple scaling.

What this means for our customers is that InfiniDB scales linearly. In internal tests, we've scaled InfiniDB on Amazon EC2 to 1024 cores without noticeable loss in performance. In fact, we believe that we can scale to 1 Million cores (although our customers have not yet asked us for deployments of this size).

The impact of this is groundbreaking. Large deployments, such as companies delivering analytics over the Cloud, require extremely efficient databases to enable their customers’ needs. Lightweight chips, such as mobile processors and GPUs, also need efficient databases due to hardware constraints. Companies with massive amounts of data (such as Petabyte-size data volumes) need highly-efficient technology to enable analytics their data. InfiniDB's architecture remains performant under all of these scenarios.

Stay tuned for the third post in this three part series, where we discuss the unparalleled ease-of-use with which our customers use InfiniDB.

Since the launch of InfiniDB last year, we've been seeing InfiniDB enabling tremendous customer successes. We've seen hundreds of customers use InfiniDB to power their most impactful analytics projects.

We’d like to describe InfiniDB's architecture and explain what makes it so scalable, fast and simple. In a three-part blog series, we'll be covering how InfiniDB differs from other database systems.

Compared to relational databases, InfiniDB has three key benefits: I/O, parallelism and ease-of-use.

First, let's start with I/O. Traditionally, the bottleneck in database processing has been I/O. When you're moving large data volumes, small differences in I/O start to add up. You can imagine the phrase "being bitten to death by ducks"; a growing series of small I/O penalties make traditional relational database systems unusable for analytics on large datasets (i.e. typically data volumes over 500 GB or more, often described as "big data").

Traditionally, database technologies have found ways to alleviate -- but not fix -- the pain. For example, Netezza (an IBM company) allows for scaling the scan rate to overcome the I/O bottleneck. However, such solutions remain costly as they require large investments in proprietary hardware.

We set out to change all of that with InfiniDB.

One of the most impactful ways to alleviate the I/O bottleneck is to align the way that data is stored with the way that it's used. For analytics, this suggests 'columnar databases' or column-stores. Unlike a traditional row-based database, which store data in rows ("First Name", "Last Name", "Age") columnar databases store data in columns (i.e. for the column "Age": '32', '44', '65').

For analytics, specific columns tend to be pulled frequently (i.e. Average of Ages '32', '44', and '65') which makes columnar databases typically the tool-of-choice. And, since columnar databases like InfiniDB can be deployed on commodity hardware, this solution tends to be much less expensive as well.

Once installed, our customers often can't believe the performance that they gain with InfiniDB. They're even more excited when they learn that InfiniDB is priced by the core and doesn't tax their growing data volumes.

What happens when your data volumes increase further and you need to scale out? Read our next post on Parallelism where we describe InfiniDB's industry-leading scale-out functionality.

We are pleased to announce the availability of the 2.2.4 version of Calpont InfiniDB Community.  This is our latest maintenance release and it includes a number of bug fixes that you can see in the release notes and at http://bugs.launchpad.net/infinidb.  You can download the latest InfiniDB binaries, source code, and updated documentation at http://infinidb.org/downloads.  For those interested in scale-out and physical compression, please download InfiniDB Enterprise 2.2.4 at http://www.calpont.com/products/tryinfinidb.

As always, thank you for your support of Calpont InfiniDB and we welcome your feedback.

We are pleased to announce the availability of the 2.2 version of Calpont InfiniDB Community. This release includes a number of great new features including: support for group_, bit_and, bit_or, and bit_xor aggregate functions, enhanced cpimport support allowing more flexibility on import files, support for truncate table, support for 45 additional distributed functions, improved performance for queries with dictionary column comparisons, and reduced memory utilization for queries with wide result sets. The release also includes a number of bug fixes that you can see at http://bugs.launchpad.net/infinidb.

You can download the latest InfiniDB binaries, source code, and updated documentation at http://infinidb.org/downloads.  If you are interested in taking a look at the 2.2 release of InfiniDB Enterprise, be sure to check out http://calpont.com.  As always, thank you for your support of Calpont InfiniDB and let us know what you think!

Calpont's InfiniDB has hit a new milestone, loading over 1 trillion rows with our columnar analytics DBMS with the actual value being 1039909436172 rows.  As expected, the load rate is stable over the entire duration, loading better than 1.1 million rows per second. 


The size of source files was 20195.94GB, size on disk was 5657.52GB.  Special thanks to Chris Wolf, Auburn '14 for execution of this benchmark.  Additional details to follow including query performance and a breakdown on compression. 
Let us help you put your data to work.  - Jim Tommaney CTO, Calpont. 

With the 2.0.1 Community maintenance release, the 64-bit and 32-bit Windows Community release is now available as well.

This release includes a number of bug fixes that you can see at http://bugs.launchpad.net/infinidb.

You can download the latest InfiniDB binaries, source code, and updated documentation at: http://infinidb.org/downloads.

We welcome your feedback and we appreciate your continued support of InfiniDB.

We are pleased to announce the availability of the 2.0.1 release of Calpont InfiniDB Community.  This is our first maintenance release for 2.0.

This release includes a number of bug fixes that you can see at http://bugs.launchpad.net/infinidb.

You can download the latest InfiniDB binaries, source code, and updated documentation at: http://infinidb.org/downloads.

We welcome your feedback and we appreciate your continued support of InfiniDB.

I am very excited to announce that the 2.0 version of Calpont InfiniDB Community is now available.  This release includes a large performance improvement for distinct and group by queries with large result sets, improvements in memory management, and support for several new distributed functions.

This release also includes a number of bug fixes that you can see at http://bugs.launchpad.net/infinidb.

You can download the latest InfiniDB binaries, source code, and updated documentation at: http://infinidb.org/downloads.

We also made 2.0 Calpont InfiniDB Enterprise available today which includes additional new functionality including column compression, drop partition, and user-defined functions.  If you're interested in taking a look at the Enterprise side, be sure to check out http://calpont.com.

As always, thanks for your support of Calpont InfiniDB and let us know what you think!

We are pleased to announce the availability of the 1.5.4 release of InfiniDB Community Edition.  This is our second maintenance release for 1.5.

This release includes a number of bug fixes that you can see at http://bugs.launchpad.net/infinidb.

You can download the latest InfiniDB binaries, source code, and updated documentation at: http://infinidb.org/downloads.

We welcome your feedback and we appreciate your continued support of InfiniDB.

Is your MySQL deployment reaching its performance and data limits? Calpont InfiniDB is a massive scaling, high performance analytic database built on MySQL. InfiniDB provides ultra fast query response against small, large and extremely large data.

Unlike other MySQL solutions, only Calpont provides a multi-threaded, scale out software architecture that maintains query performance as your data and user requirements grow. Since you know MySQL, it's also easy to install, learn and maintain.

Learn How Calpont InfiniDB:

• Architecture and capabilities make it the right choice for scaling your MySQL environment.
• Can easily leverage your existing BI and analytic applications.
• Loads and execute queries fast, even with extremely large data.

Learn how Calpont InfiniDB achieves these compelling scale and performance benefits, and if it's right for you.

Wednesday, August 18, 2010
9am PDT / 12pm EDT / 6pm GMT

To learn more and discuss your requirements, you can contact us at 214-618-9500, or at info@calpont.com.

Calpont Corporation
3011 Internet Blvd, Suite 100
Frisco, TX 75034

Wednesday, August 18, 2010
9am PDT / 12pm EDT / 6pm GMT

"InfiniDB 1.5 delivers fantastic price versus performance for our MySQL data intensive application. With our recent FCC partnership, we are faced with a significant data collection task in analysing end user broadband experience. InfiniDB 1.5 takes away the scalability concerns we had with our previous vanilla MySQL database, whilst at the same time requiring little or no redevelopment effort."

Sam Crawford
Founder, SamKnows