Friday, March 20, 2015

CXPacket Wait in SQL Server



CXPACKET
Occurs with parallel query plans when trying to synchronize the query processor exchange iterator. If waiting is excessive and cannot be reduced by tuning the query (such as adding indexes), consider adjusting the cost threshold for parallelism or lowering the degree of parallelism.

CXPacket wait occurs when we use multiple processors/thread to execute a query.
When a server has multiple CPU, we can configure setting to use multiple cpu/thread, if query is taking
Excess time compare to approximate time of a query.

We can define this setting using “cost threshold for parallelism”, having default value 5 seconds, we can change values to something else based on our requirement
Suppose we have mentioned 100 seconds and if any query is taking more than 100 seconds then it will start parallel processing based on the configuration values of server max degree of parallelism.
According to below screenshot our server has two CPU’s, CPU0 and CPU1 and let us
Change the server configuration values “cost threshold for parallelism” to 100 and max degree of parallelism to 2.

Using this configuration query will start using two cpu/thread, if query takes more than 100 seconds.
According to below Figures A and B, we have two cpu, defined “cost threshold for parallelism” to 100 and max degree of parallelism to 2, highlighted in color yellow and black in figure A and Figure B respectively
Figure A





Figure B






We can change these setting using system defined stored procedure sp_configure.
As per below setting, if any query taking more than 100 seconds will qualify for parallel processing and SQL server will create and execute parallel plans using two thread(CPU0 & CPU1)

EXEC sys.sp_configure N'cost threshold for parallelism', N'100'
GO
EXEC sys.sp_configure N'max degree of parallelism', N'2'
GO
RECONFIGURE WITH OVERRIDE
GO






We can also set parallel thread setting using OPTION (MAXDOP 2) at query level like below
Let us create table CXPacketInfo in tempdb database and insert 100000 rows

USE TEMPDB

CREATE TABLE CXPacketInfo (CX_id INT, CXO_id INT, CX_amount INT, CX_description CHAR(2000))



BEGIN TRAN
DECLARE @i INT
SET @i = 1

WHILE (@i <= 100000 ) /*>*/
BEGIN

INSERT INTO CXPacketInfo VALUES (@i % 1, @i, RAND() * 100000, REPLICATE('a', 2000))
SET @i = @i + 1

END

COMMIT TRAN


SET STATISTICS time ON
GO

/*Execute query with single thread & Clear the wait stats entry in dmv sys.dm_os_wait_stats */


DBCC sqlperf('sys.dm_os_wait_stats', clear)

DECLARE @order_amount INT
SELECT @order_amount= Max(CX_amount)
FROM CXPacketInfo o
OPTION (maxdop 1) /* Execute query with single thread */


SELECT * FROM sys.dm_os_wait_stats
WHERE wait_type = 'CXPACKET'



Output:Output of dmv is showing 0 waiting_tasks_count and 0 wait_time_ms

wait_type waiting_tasks_count wait_time_ms max_wait_time_ms signal_wait_time_ms
CXPACKET 0 0 0 0


Execution Plan: In execution plan we can see query is only using single thread and this single thread is processing 100000 Rows without parallelism.


Execution Plan A:





/*Execute query with single thread Clear the wait stats entry in dmv sys.dm_os_wait_stats */

DBCC sqlperf('sys.dm_os_wait_stats', clear)

DECLARE @order_amount INT

SELECT @order_amount= Max(CX_amount)
FROM CXPacketInfo o

OPTION (maxdop 2) /* Execute query with two threads */

SELECT * FROM sys.dm_os_wait_stats
WHERE wait_type = 'CXPACKET'

Output: Output of dmv is showing 4 waiting_tasks_count and 65 wait_time_ms


wait_type waiting_tasks_count wait_time_ms max_wait_time_ms signal_wait_time_ms
CXPACKET 4 65 65 0


Execution Plan: In execution plan we can see query is using two threads and these two threads are processing 100000 Rows.
In Right hand rectangle box following Threads are used



Thread Rows Comments
Thread 0 0 Organizer Thread
Thread 1 55280 First Thread
Thread 2 44720 Second Thread

Thread 0 is organizer thread, when processing threads (Thread 1 & Thread 2) have time differences for processing the rows then organizer thread (Thread 0) has to wait for lacking behind thread, this wait we called CXPacket wait.
In plan we can also see parallelism is used in below execution plan marked in rectangle.




Execution Plan B:



Here's a list of additional considerations:
Make certain statistics are up-to-date; the optimizer thrives—or dies—based on up-to-date statistics.
• Fragmentation of internal structures & external disk must be kept low.
• In no case should max degree of parallelism be set to a value higher than the number of physical cores.
• For most servers max degree of parallelism should be set to no more than eight, even if more than eight cores are available.
• For NUMA-enabled servers, max degree of parallelism should not exceed the number of cores assigned to each NUMA node.
• Hyper-Threading often (not always) compromises SQL Server performance. My recommendation: In the absence of unequivocal supporting evidence that H-T enhances performance in your environment, H-T should be disabled.
• OLTP – parallelism can have a negative impact on performance.
• OLAP/reporting benefits from parallelism.
• Look at your plan cache, which of your queries use parallelism
• Check query plans – do you have missing indexes or out of date statistics
• What other waits are occurring that might be causing CXPACKET
• Consider using MAXDOP to fine tune specific queries.
• Set max degree of parallelism and cost threshold for parallelism to best suit your server’s function.
• Do not adjust max degree of parallelism without proper analysis of your servers workload/queries

Friday, January 2, 2015

In-Memory OLTP Engine (High Level Architecture)

In-Memory OLTP Engine (High Level Architecture)
In-Memory OLTP Engine (High Level Achitecture)
In Memory OLTP Engine   (High Level Architecture)

Thursday, December 11, 2014

New and Enhanced tSQL Functions in SQL Server 2012

Enhanced and New  tSQL Functions in SQL Server 2012

1. Try_Convert 
/*Try_Convert and Try_Parse Functions to handle errors by returning NULL values*/

SELECT TRY_CONVERT(INT, 100) AS CorrrectConvert, TRY_CONVERT(INT, 'abc') AS ErrorConvert;

Result:
CorrrectConvert ErrorConvert
100 NULL


SELECT PARSE('7/17/2011' AS DATE USING 'en-US') AS USDate, PARSE('2011/7/17' AS DATE USING 'ja-JP') AS JapanDate;
Result:
USDate JapanDate
7/17/2011 7/17/2011


2. TRY_PARSE
/*If we try to parse Invalid values it will show below error like japan support 2011 but if we pass only 11
it will show error*/
SELECT PARSE('7/17/11' AS DATE USING 'ja-JP') AS dt;

Error:
--Msg 9819, Level 16, State 1, Line 1
-- Error converting string value '7/17/11' into data type date using culture 'ja-JP'.

SELECT TRY_PARSE('7/17/11' AS DATE USING 'ja-JP') AS dt;



3. Date and Time Functions
SELECT
DATEFROMPARTS(2014, 07, 22)    AS DATE_FROMPARTS,
DATETIME2FROMPARTS(2014, 07, 22, 14, 31, 7, 2, 3) AS DATETIME2_FROMPARTS,
DATETIMEFROMPARTS(2014, 07, 22, 14, 31, 7, 997) AS DATETIME_FROMPARTS,
DATETIMEOFFSETFROMPARTS(2014, 07, 22, 14, 31, 5, 1, -5, 0, 7) AS DATETIMEOFFSET_FROMPARTS,
SMALLDATETIMEFROMPARTS(2014, 07, 22, 14, 31) AS SMALLDATETIME_FROMPARTS,
TIMEFROMPARTS(15, 33, 6, 1, 7) AS TIME_FROMPARTS;



4. Logical/Conditional Functions
SELECT CHOOSE(3, 43466, 'Delhi', 'Agra') AS '1stPosition', CHOOSE(5, 'MUM', 'pune', 'Bangalore','allahabad','KNP') AS '5thPosition',CHOOSE(3,'a','b') 'NoPosition' ;

SELECT IIF(4 = 5, 'Ram', 'Shyam') AS iif_result;
SELECT IIF(4 = 4, 'Ram', 'Shyam') AS iif_result;




5. Concate and Format
DECLARE   @Passenger TABLE
  (ID INT,
  PassengerID VARCHAR(10),
       CityName   VARCHAR(10),
       StartDate  DATE
    )

INSERT INTO @Passenger VALUES (1,'P00001', 'Delhi', '1-Sep-2014')
INSERT INTO @Passenger VALUES (2,'P00001', 'Mathura', '1-Oct-2014')
INSERT INTO @Passenger VALUES (3,'P00001', 'Etah', '10-Oct-2014')

Query1.
SELECT *,CONCAT(PassengerID, ', ' + 'Name&Type_Desc', ', ' + CityName) AS ConcatinatedInfo
FROM  @Passenger
Result:
ID PassengerID CityName StartDate ConcatinatedInfo
1 P00001 Delhi 9/1/2014 P00001, PsgrID&CtyName, Delhi
2 P00001 Mathura 10/1/2014 P00001, PsgrID&CtyName, Mathura
3 P00001 Etah 10/10/2014 P00001, PsgrID&CtyName, Etah

Query2.
SELECT *,CONCAT(PassengerID, ', ' + 'PsgrID&CtyName', ', ' + CityName) AS ConcatinatedInfo
FROM  @Passenger

Result:
USDateFormat FormatWithZero
12/11/2014     00000001
12/11/2014     00000002
12/11/2014     00000003
Query3.
SELECT FORMAT(GETDATE(), 'd', 'en-us') AS USCulture, FORMAT(GETDATE(), 'd', 'ja-JP') AS JAPANECULTURE,FORMAT(GETDATE(), 'd', 'de-DE') AS GERMANCULTURE

Result:
USCulture JAPANECULTURE GERMANCULTURE
12/11/2014 12/11/2014 11.12.2014




6. LOG
Query4.
SELECT LOG(256, 2);
Result.
LOG
8



7. EOMONTH

Query5.
SELECT EOMONTH('12/15/2014')Endofmonth ,EOMONTH(SYSDATETIME()) AS Endofmonth1,EOMONTH(getdate()) AS endofmonth2;

Result:
Endofmonth Endofmonth1 endofmonth2
12/31/2014 12/31/2014 12/31/2014

Wednesday, December 10, 2014

Pagination in SQL Server 2012

Pagination in SQL Server 2012

/*Queries to Generate Data for pagination*/

DECLARE   @Passenger TABLE
  (
   ID INT,
       PassengerID VARCHAR(10),
       CityName   VARCHAR(10),
       StartDate  DATE
    )

INSERT INTO @Passenger VALUES (1,'P00001', 'Delhi', '1-Sep-2014')
INSERT INTO @Passenger VALUES (2,'P00001', 'Mathura', '1-Oct-2014')
INSERT INTO @Passenger VALUES (3,'P00001', 'Etah', '10-Oct-2014')
INSERT INTO @Passenger VALUES (4,'P00001', 'Kanpur', '25-Oct-2014')
INSERT INTO @Passenger VALUES (5,'P00001', 'Lucknow', '27-Oct-2014')
INSERT INTO @Passenger VALUES (6,'P00002', 'Allahabad', '1-Oct-2014')
INSERT INTO @Passenger VALUES (7,'P00002', 'Banaras', '1-Nov-2014')
INSERT INTO @Passenger VALUES (8,'P00002', 'Mumbai', '10-Nov-2014')
INSERT INTO @Passenger VALUES (9,'P00002', 'Banaras', '15-Nov-2014')
INSERT INTO @Passenger VALUES (10,'P00002', 'Pune', '25-Nov-2014')
INSERT INTO @Passenger VALUES (11,'P00002', 'Hyd', '28-Nov-2014')
INSERT INTO @Passenger VALUES (12,'P00002', 'Chennai', '30-Nov-2014')
INSERT INTO @Passenger VALUES (13,'P00003', 'Bangalore', '1-Dec-2014')


Query 1:
SELECT   * 
FROM     @Passenger 
ORDER BY Id
OFFSET 3 ROWS
FETCH NEXT 6 ROWS ONLY
Result:
IDPassengerIDCityNameStartDate
4P00001Kanpur25-Oct-2014
5P00001Lucknow27-Oct-2014
6P00002Allahabad1-Oct-2014
7P00002Banaras1-Nov-2014
8P00002Mumbai10-Nov-2014
9P00002Banaras15-Nov-2014
Query 2:
SELECT   * 
FROM     @Passenger 
ORDER BY  Id
OFFSET 9 ROWS
FETCH NEXT 3 ROWS ONLY


Result:
IDPassengerIDCityNameStartDate
10P00002Pune25-Nov-2014
11P00002Hyd28-Nov-2014
12P00002Chennai30-Nov-2014

Note: OFFSETand Next Rows count never be Negative.

SQL Server Logical/Physical Reads

SQL Server Logical/Physical Reads

Summary Info:

Logical Reads : Reading Data pages from Cache
Physical Reads : Reading Data pages from Hard Disk
Buffer Cache Hit Ratio:  (logical reads – physical reads)/logical read * 100%



Detailed Info:

Logical Reads:
Logical read indicates total number of data pages needed to be accessed from data cache to process query. It is very possible that logical read will access same data pages many times, so count of logical read value may be higher than actual number of pages in a table. Usually the best way to reduce logical read is to apply correct index or to rewrite the query.

Physical Reads 
Physical read indicates total number of data pages that are read from disk. In case no data in data cache, the physical read will be equal to number of logical read. And usually it happens for first query request. And for subsequent same query request the number will be substantially decreased because the data pages have been in data cache.

Buffer Cache Hit Ratio:
Buffer hit ratio will be calculated based on these two kinds of read as the following formula: (logical reads – physical reads)/logical read * 100%. The high buffer hit ratio (if possible to near 100%) indicates good database performance on SQL Server level. So use information from physical read and buffer hit ratio to measure performance in server level and logical read to measure individual query level 


Excess of the Logical Reads tends high memory Usage, there are some ways by which we can Reduce Logical Reads:
1. Improper/Useless/Insufficient Indexes: Indexes should be build on the basis of data access or retrieval process if any of the indexes is build on the columns which are not used in a query will leads to High Logical reads and will degrade the performance while reads and writing the data....

2.Poor Fill Factor/Page Density: Page use should should not be very less. otherwise large number of page will be used for small amount of data which will also leads to High Logical Reads....

3.Wide Indexes: Indexing on the large number of columns will leads to high logical reads....

4. Index scanning: if query is leads to index scanning on the table then logical reads will be high...


Logical Reads count can be get by using following ways

Below are the ways to check logical Reads:
1. set statistics io on 

2. sys.dm_exec_query_Stats
by executing the below statement we can find detailed info about reads/writes
select * from sys.dm_exec_query_Stats

3. SQL Profiler: by executing the sql profiler on that database we can find out logical reads..

There are some other SQL DMV/F which also help us to get logical reads...

also visit:
http://www.sqlservercentral.com/Forums/Topic1250154-391-1.aspx

LEAD And LAG Analytic Functions in SQL Server 2012

LEAD And LAG Functions in SQL Server 2012

LEAD And LAG provides access to a row at a given physical offset respective to current row.
Use these analytic function in a SELECT statement to compare values in the current row with values in a Previous and following row according to given offset.
Note: Offset cannot be negative.

Let us take a passengers example: 

suppose some passengers are travelling from one city to another, they are passing
through many cities, now if we want to find out which was the previous
& city which will be next city we can achieve this using Common table expression and ranking
functions as follow

-- Let us code this in SQL Server
-- Declare @Passenger Table as below.
DECLARE   @Passenger TABLE
  (
       PassengerID VARCHAR(10),
       CityName   VARCHAR(10),
       StartDate  DATE
    )
/* Insert record in Passenger table, suppose passengers are passing different cities 
on different dates as below*/

INSERT INTO @Passenger VALUES ('P00001', 'Delhi', '1-Sep-2014')
INSERT INTO @Passenger VALUES ('P00001', 'Mathura', '1-Oct-2014')
INSERT INTO @Passenger VALUES ('P00001', 'Etah', '10-Oct-2014')
INSERT INTO @Passenger VALUES ('P00001', 'Kanpur', '25-Oct-2014')
INSERT INTO @Passenger VALUES ('P00001', 'Lucknow', '27-Oct-2014')
INSERT INTO @Passenger VALUES ('P00002', 'Allahabad', '1-Oct-2014')
INSERT INTO @Passenger VALUES ('P00002', 'Banaras', '1-Nov-2014')
INSERT INTO @Passenger VALUES ('P00002', 'Mumbai', '10-Nov-2014')
INSERT INTO @Passenger VALUES ('P00002', 'Banaras', '15-Nov-2014')
INSERT INTO @Passenger VALUES ('P00002', 'Pune', '25-Nov-2014')
INSERT INTO @Passenger VALUES ('P00002', 'Hyd', '28-Nov-2014')
INSERT INTO @Passenger VALUES ('P00002', 'Chennai', '30-Nov-2014')
INSERT INTO @Passenger VALUES ('P00003', 'Bangalore', '1-Dec-2014')

/*Now if we want to find out where passengers were on next day and previous day we need to right below queries*/
--- Query 1
;
WITH   CTE as
(
        SELECT RN = ROW_NUMBER() OVER (PARTITION BY   PassengerID ORDER BY   StartDate ASC),   *
        FROM @Passenger
)
SELECT
        [Current Row].*,
        ISNULL(DATEADD(DAY, -1, [Next Row].StartDate), '31-Dec-2099') AS EndDate

,ISNULL([Previous Row].CityName,'NoPrv_Cty') PrvCity
,ISNULL([Next Row].CityName,'NoNxt_Cty') NextCity

FROM   CTE AS  [Current Row]
        LEFT JOIN CTE AS [Next Row] ON [Current Row].PassengerID   = [Next Row].PassengerID   AND  [Next Row].RN   = [Current Row].RN   + 1
LEFT JOIN CTE AS [Previous Row] ON [Current Row].PassengerID   = [Previous Row].PassengerID   AND  [Previous Row].RN = [Current Row].RN-1
ORDER BY [Current Row].PassengerID, [Current Row].RN;

/*But using LEAD and LAG Function we can achieve using below query*/
--- Query 2
SELECT *,
DATEADD(DAY, -1, LEAD(StartDate, 1,'01-Jan-2100')
        OVER (PARTITION BY   PassengerID ORDER BY   StartDate ASC))   AS EndDate,
LAG(Cityname, 1,'NoPrv_Cty')
OVER (PARTITION BY   PassengerID ORDER BY   StartDate ASC)   AS PreviousCity,
LEAD(Cityname, 1,'NoNxt_Cty')
OVER (PARTITION BY   PassengerID ORDER BY   StartDate ASC)   AS NextCity
FROM @Passenger

-- Queries will give result like below
PassengerID CityName StartDate EndDate PreviousCity NextCity
P00001 Delhi 1-Sep-2014 30-Sep-2014 NoPrv_Cty Mathura
P00001 Mathura 1-Oct-2014 9-Oct-2014 Delhi Etah
P00001 Etah 10-Oct-2014 24-Oct-2014 Mathura Kanpur
P00001 Kanpur 25-Oct-2014 26-Oct-2014 Etah Lucknow
P00001 Lucknow 27-Oct-2014 31-Dec-2099 Kanpur NoNxt_Cty
P00002 Allahabad 1-Oct-2014 31-Oct-2014 NoPrv_Cty Banaras
P00002 Banaras 1-Nov-2014 9-Nov-2014 Allahabad Mumbai
P00002 Mumbai 10-Nov-2014 14-Nov-2014 Banaras Banaras
P00002 Banaras 15-Nov-2014 24-Nov-2014 Mumbai Pune
P00002 Pune 25-Nov-2014 27-Nov-2014 Banaras Hyd
P00002 Hyd 28-Nov-2014 29-Nov-2014 Pune Chennai
P00002 Chennai 30-Nov-2014 31-Dec-2099 Hyd NoNxt_Cty
P00003 Bangalore 1-Dec-2014 31-Dec-2099 NoPrv_Cty NoNxt_Cty

/*Now if we want to find out which was the Previous to previous City and Which is Next to Next city
we need to modify our queries but in case of LEAD and LAG we just need to change off set Value
as below*/

SELECT *, 
DATEADD(DAY, -1, LEAD(StartDate, 1,'01-Jan-2100') 
        OVER (PARTITION BY   PassengerID ORDER BY   StartDate ASC))   AS EndDate,
LAG(Cityname, 2,'NoPrv_Cty') 
OVER (PARTITION BY   PassengerID ORDER BY   StartDate ASC)   AS PreviousCity,
LEAD(Cityname, 2,'NoNxt_Cty') 
OVER (PARTITION BY   PassengerID ORDER BY   StartDate ASC)   AS NextCity
FROM @Passenger

PassengerID CityName StartDate EndDate PreviousCity NextCity
P00001 Delhi 1-Sep-2014 30-Sep-2014 NoPrv_Cty Etah
P00001 Mathura 1-Oct-2014 9-Oct-2014 NoPrv_Cty Kanpur
P00001 Etah 10-Oct-2014 24-Oct-2014 Delhi Lucknow
P00001 Kanpur 25-Oct-2014 26-Oct-2014 Mathura NoNxt_Cty
P00001 Lucknow 27-Oct-2014 31-Dec-2099 Etah NoNxt_Cty
P00002 Allahabad 1-Oct-2014 31-Oct-2014 NoPrv_Cty Mumbai
P00002 Banaras 1-Nov-2014 9-Nov-2014 NoPrv_Cty Banaras
P00002 Mumbai 10-Nov-2014 14-Nov-2014 Allahabad Pune
P00002 Banaras 15-Nov-2014 24-Nov-2014 Banaras Hyd
P00002 Pune 25-Nov-2014 27-Nov-2014 Mumbai Chennai
P00002 Hyd 28-Nov-2014 29-Nov-2014 Banaras NoNxt_Cty
P00002 Chennai 30-Nov-2014 31-Dec-2099 Pune NoNxt_Cty
P00003 Bangalore 1-Dec-2014 31-Dec-2099 NoPrv_Cty NoNxt_Cty