Wednesday, August 14, 2013

Test a Percentage Of Rows

There are times when working with a large data set that you may not have space or time to test every row you have processed. I use a limiting table that contains a sleect set of key values you will inner join your test query to.This throttles the test to just that set of key values, as a percentage of the total data.

DECLARE
--set the percent of the rows, as a whole number, you want to run these tests against
    @PercentRows INT = 10,
    @RowsToProcess INT,
    @SQL NVARCHAR(MAX)

DECLARE @LoanTbl TABLE (Loan_ID INT)

SET @RowsToProcess = (SELECT COUNT (DISTINCT Loan_ID)FROM TableOne (NOLOCK))*(@PercentRows*.01)
SET @SQL = 'SELECT DISTINCT TOP '+CONVERT(VARCHAR(20),@RowsToProcess)+' Loan_ID FROM TableOne (NOLOCK) ORDER BY Loan_ID DESC';

INSERT INTO @LoanTbl
EXECUTE sp_executesql @SQL

Monday, August 12, 2013

Find A Date Range Gap In A Daily Loaded Table

:SETVAR DatabaseName "NAME"
:SETVAR SourceTable "dbo.TABLE"
:SETVAR ControlColumn "etl_date"

USE
[$(DatabaseName)]

DECLARE
@Tbl1 TABLE (ID INT IDENTITY(1,1) , Dater1 DateTime)DECLARE @Tbl2 TABLE (ID INT IDENTITY(1,1) , Dater2 DateTime)DECLARE @DateFilter DATETIME = '2013-01-05 00:00:00.000'
INSERT
INTO @Tbl1SELECT DISTINCT eff_start_date FROM $(SourceTable)WHERE $(ControlColumn) >= @DateFilterORDER BY eff_start_date
INSERT
INTO @Tbl2SELECT DISTINCT eff_start_date FROM $(SourceTable)WHERE $(ControlColumn) >= @DateFilterORDER BY eff_start_date
SELECT
'$(SourceTable)' SourceTable,t.Dater2,CASE
WHEN (CAST(o.Dater1 AS DATE) <> CAST(DATEADD(D,-1,t.Dater2) AS DATE)) THEN 'FAIL'
ELSE 'PASS' END AS 'Skipped Day',CASEWHEN (DATEPART(weekday,t.Dater2) = 2) THEN 'Monday'WHEN (DATEPART(weekday,t.Dater2) = 3) THEN 'Tuesday'
WHEN (DATEPART(weekday,t.Dater2) = 4) THEN 'Wednesday'
WHEN (DATEPART(weekday,t.Dater2) = 5) THEN 'Thurday'WHEN (DATEPART(weekday,t.Dater2) = 6) THEN 'Friday'WHEN (DATEPART(weekday,t.Dater2) = 7) THEN 'Saturday'WHEN (DATEPART(weekday,t.Dater2) = 1) THEN 'Sunday'ELSE 'Nomads!' END DayOfWeeker FROM @Tbl1 o JOIN @Tbl2 t ON o.ID = (t.ID-1)WHERE o.ID <> 1ORDER BY Dater1 DESC

Tuesday, August 6, 2013

Sample Regex Patterns

Used for test data generation

US Phone [0-9]{3}(-)[0-9]{3}(-)[0-9]{4}
US Phone Optional dash [0-9]{3}(-|)[0-9]{3}(-|)[0-9]{4}
US Phone Extension [X][ ][0-9]{3,4}
US SSN [0-9]{3}-[0-9]{2}-[0-9]{4}
US SSN no dash [0-9]{3}[0-9]{2}[0-9]{4}
Basic Email [a-z]{4,8}([0-9]{0,3}|\.[a-z]{4,10})@([a-z]{4,9}\.)?([a-z-]{6})\.(org|com|net|biz|cx)

Company Name
(Re|Ad|Par|Tru|Thru|In|Bar|Cip|Dop|End|Em|Fro|Gro|Hap|Kli|Lom|Mon|Qwi|Rap|Sup|Sur|Tip|Tup|Un|Up|Var|Win|Zee)(ban|cad|dud|dim|er|frop|glib|hup|jub|kil|mun|nip|peb|pick|quest|rob|sap|sip|tan|tin|tum|ven|wer|werp|zap)(il|ic|im|in|up|ad|ack|am|on|ep|ed|ef|eg|aqu|ef|edg|op|oll|omm|ew|an|ex|pl)?(icator|or|ar|ax|an|ex|istor|entor|antor|in|over|ower|azz)(([ -]?Pro|Duplex|Multi){0}) (Direct|WorldWide|Holdings|International||) (|Inc|Company|Group|Corp.)?

Tuesday, July 30, 2013

Clear Log Files In SQL Server 2008 and Beyond

Sometimes your recovery mode prevents you from clearing your log files. Change the recovery mode and then do SHRINKFILE


:SETVAR DB Datamart
:SETVAR __IsSqlCmdEnabled "True"
GO
IF
N'$(__IsSqlCmdEnabled)' NOT LIKE N'True'
BEGIN
PRINT
N'SQLCMD mode must be enabled to successfully execute this script.';SET NOEXEC ON;END
GO
USE
$(DB)
ALTER DATABASE $(DB) SET RECOVERY SIMPLE WITH NO_WAIT--SELECT name ,size/128.0 - CAST(FILEPROPERTY($(DB), 'SpaceUsed') AS int)/128.0 AS AvailableSpaceInMB
--FROM sys.database_files;
DBCC SHRINKFILE ($(DB)_log,TRUNCATEONLY)--SELECT name ,size/128.0 - CAST(FILEPROPERTY([$(DB)], 'SpaceUsed') AS int)/128.0 AS AvailableSpaceInMB
--FROM sys.database_files;
ALTER DATABASE [$(DB)] SET RECOVERY FULL WITH NO_WAIT

Friday, June 14, 2013

Dynmic SQL - List Space Allocation Across Dbs


DECLARE @Tbl Table
(ID INT IDENTITY(1,1),name    VARCHAR(100))


INSERT INTO @Tbl
(name)
VALUES
('model'),
('master')

DECLARE
@StartCnt INT = 1,
@MaxCnt INT = (SELECT COUNT (1) FROM @tbl),
@Db nVarchar(100),
@sql nVarchar(MAX)

WHILE @StartCnt <= @MaxCnt

BEGIN
SET @Db = (SELECT Name FROM @Tbl WHERE ID = @StartCnt)
PRINT @db
SET @sql = '
USE '
+@Db
+' SELECT '''
+ @DB
+''' DB, SUM(Size)/128 Used_MB, SUM(maxsize)/128 Allocated_MB
FROM sysfiles'


  exec sp_executesql @sql OUTPUT
  SET @StartCnt = @StartCnt+1
  END

Saturday, May 18, 2013

SQL Tips - Cross Apply vs Cross Join

I have a query at work with a CROSS APPLY that I found hard to read and incorrectly assumed was under performing. I tried rewriting it using an inner join to a table variable and realized I had decreased the performance four fold. My miscalculation came from my confusing CROSS APPLY with CROSS JOIN. I found a nice definition of the two provided by Pandian Sathappan.

CROSS JOIN
1.A cross join that does not have a WHERE clause produces the Cartesian product of the tables involved in the join.

2.The size of a Cartesian product result set is the number of rows in the first table multiplied by the number of rows in the second table (N x M)

CROSS APPLY
1.The APPLY operator allows you to invoke a table-valued function for each row returned by an outer table expression of a query.

2.The table-valued function acts as the right input and the outer table expression acts as the left input.

3.The right input is evaluated for each row from the left input and the rows produced are combined for the final output.

Reference:
Pandian Sathappan
www.SQLServerbuddy.blogspot.com  

Thursday, May 16, 2013

Some Ruby Rapid Prototyping Notes

How to articles
http://www.akitaonrails.com/2009/1/13/the-best-environment-for-rails-on-windows/#.UZW4q8ogacs
http://rubyonrails.org/download

IDE
http://www.aptana.com/downloads/start

Language Installer
http://rubyforge.org/projects/rubyinstaller/


Package Manager.
This is ruby gems. It seems to be included in ruby 2. You can install from rubygems.org/pages/download

A Text Editor
ftp://ftp.vim.org/pub/vim/pc/gvim72.exe

Rails
Using the command line package manager, issue $> gem install rails

MVC for Ruby, built into rails 3.
http://rubyonrails.org/merb

Add on management is done through Git. You need a tool to get through git. Use the IDE above.
Git Book
http://git-scm.com/book

ORM STUFF
https://www.ruby-toolbox.com/categories/orm
http://stackoverflow.com/questions/3609482/activerecord-talk-to-two-databases

Active Record may not be the best, but it is the best supported and widely used. Sequel may be the best functionally, with Datamapper being in the top three. Below is a basic note from the Datamapper dude on when yo use SQL vs ORM to code modeling.


accepted
I'm the DataMapper maintainer, and I think for complex reporting you should use SQL.
While I do think someday we'll have a DSL that provides the power and conciseness of SQL, everything I've seen so far requires you to write more Ruby code than SQL for complex queries. I would much rather maintain a 5 line SQL query than 10-15 lines of Ruby code to describe the same complex operation.
Please note I say complex.. if you have something simple, use the ORM's build-in finders. However, I do believe there is a line you can cross where SQL becomes simpler. Now, most apps aren't just reporting. You may have alot of CRUD type operations, for which an ORM is perfectly suited and far better than doing those things by hand.
One thing that an ORM will usually provide is some sort of organization to your application logic. You can group code based around each model in the same file. It's usually there that I'll put the complex SQL query, rather than embedding it in the controller, eg:
class User
  include DataMapper::Resource

  property :id,   Serial
  property :name, String,  :length => 1..100, :required => true
  property :age,  Integer, :min => 1, :max => 130

  def self.some_complex_query
    repository.adapter.select <<-SQL
      SELECT ...
        FROM ...
       WHERE ...
       ... more complex stuff here ...
    SQL
  endend
Then I can just generate the report using User.some_complex_query. You could also push the SQL query into a view if you wanted to further cleanup this code.
EDIT: By "view" in the above sentence I meant RDBMS view, rather than view in the MVC context. Just wanted to clear up any potential confusion.

TERMS

  • ruby - The interpreter itself. Run Ruby scripts or statements.
  • gem - Ruby Package Manager. Great for automatically downloading or updating small Ruby modules like XML libraries, web servers, or even whole Ruby programs.
  • irb - Interactive Ruby Prompt. This is an entire Ruby shell that will let you execute any Ruby code you want. You can load libraries, test code directly, anything you can do with Ruby you can do in this shell. Believe me, there is quite a lot that you can do with it to improve your Ruby development workflow [1].
  • ri - Quick shell access to Ruby documentation. You can find the RDoc information on nearly any Ruby Class or method. The same kind of documentation that you would find on the online ruby-docs.
  • erb - Evaluates embedded Ruby in Ruby Templated documents. Embedded Ruby is just like embedding php into a document, and this is an interpreter for that kind of document. This is really more for the rails crowd. An alternative would be haml.
  • rdoc - Generate the standard Ruby documentation for one of your Ruby classes. Its like Javadocs. It parses the Ruby source files and generates the standard documentation from special comments.
  • testrb and rake. I'm not familiar enough with these. I'd love it if someone could fill these in!

Other DB stuff
http://guides.rubyonrails.org/migrations.html
http://lostechies.com/rayhouston/2008/05/03/connecting-activerecord-to-sql-server/

https://github.com/rails-sqlserver/activerecord-sqlserver-adapter/wiki/Using-TinyTds
gem install activerecord
gem install activerecord-sqlserver-adapter