The need is clear
We all know that data warehouse (DW) applications typically:
- Support complex analytical query activities using very large data sets.
- Fetch millions of records from the database for processing and take a long time to complete queries.
As data warehouses grow, high performance continues to be a critical need for most of the organizations. High performance is about achieving both speed and scale while dealing with increasing complexity and concurrency. Microsoft realized this and developed a new feature called the columnstore index in SQL Server 2012 that accelerates DW query performance significantly.
A new perspective
Columnstore index groups and stores column data and then joins all the columns to complete the index. This differs from traditional indexes which group and store row data. You might wonder how a simple change in the storage format of the table data can improve performance drastically. With some research, I found the below reasons.
- It allows SQL Server to retrieve only the necessary columns from storage disk which reduces the amount of data retrieved from storage.
- It uses a compression algorithm that eliminates redundancy. Because columnar data has a higher chance of being redundant, compression helps store less data in the SQL Server buffer pool.
- It stores frequently accessed columns in the memory which reduces I/O to the storage.
For details, read the MSDN Library article “Columnstore Indexes”.
What’s in it for Database Admins
Columnstore Index offers query performance and space saving benefits for SQL Server data warehouse database applications.
- Space Savings
- Query Performance ( Index compression, Optimized memory structures)
- Parallelism
There can be both row store index and column store index on the same table. The query optimizer will decide which index is the most appropriate to use.
Gotchas-Need to know
Some key limitations of columnstore index listed by Microsoft are:
- Creating a columnstore index makes the table read-only. Columnstore index might not be the ideal choice for selective queries, which touch only one (or a few) rows or queries that lookup a single row or a small range of rows.Ex: OLTP workloads
- Cannot be clustered. Only non-clustered columnstore indexes are available.
- Cannot act as a primary key or a foreign key.
- A columnstore index on a partitioned table must be partition-aligned.
- A column store cannot be combined with
- page or row compression
- Replication
- Change tracking
- Change data capture
- File stream
It’s in the proof
To make sure columnstore really had an impact, I ran tests to evaluate CSI by implementing it on the two largest tables in a TPC-H database. These tests proved that using columnstore index on large tables improved the query response times significantly for TPC-H queries. Using columnstore index along with table partition makes use of the table partition benefits (such as DW loading and maintenance) as well as the columnstore index benefits of improved query execution times.
To get a better understanding of the test results and details on columnstore index performance tests and other DW tests, refer to our recent publication “Best practices for Decision Support Systems with Microsoft’s SQL server 2012 using Dell EqualLogic PS series storage arrays"