Have you not used compression or have used a wrong compression algorithm? Not a problem, you can alter the table and next time when major compaction takes place, new compression will be applied to the table. number of nodes, memory, CPU etc) remained unchanged.įollowing were the readings obtained after 7 distinct execution of each query. Query 2 - select DayofMonth ,sum(deptime) from airline2 where DayofMonth in (12,18) and month in (1,4,6,9) group by DayofMonth Īll other parameters of the system (i.e.Query 1 - select dayofmonth, avg(deptime) from airline1 group by dayofmonth.We executed following queries on the data: An external table in impala was created on top of the HBase table and a set of queries were executed on both the tables. One was created without compression and second table was created with Snappy compression. About 5 million records from this dataset were loaded in Hbase table. We performed a benchmarking of compression performance using Snappy compression. Snappy or LZO are a better choice for hot data, which is accessed frequently.

GZIP is often a good choice for cold data, which is accessed infrequently. GZIP compression uses more CPU resources than Snappy or LZO, but provides a higher compression ratio. There are 3 compression algorithms that can be applied on HBase data: LZO, GZIP and Snappy. On the other hand with compression comes the need to un-compress the data after reading and this results in increase in CPU load. This leads to increased performance of reads. Thus the process of reading data has to take place on smaller, compressed data size. By using compression, the data occupies less space on disk. This is because Hadoop and Hbase jobs are data intensive, thus making data reads a bottleneck in overall application. In this blog, we will take a look at two important aspects of Hbase performance.ĭisk performance is almost always bottleneck in Hadoop clusters. Visualization tools also make use of Impala to access HBase data.

We are using Impala to read the data being streamed into HBase. This is a series of blogs in which we are sharing experience of optimization of HBase design for managing streaming, time series data. In my previous blog about Hbase performance tuning, I shared the importance of correct usage of rowkey in Hbase performance.

Snappy compression vs others