Running ETL Code in Powershell Workflow

I needed a test harness to run multiple concurrent versions of the same Pentaho job. I wanted to test that the pid file feature I added prevented subsequent executions while the job was already running.

Create a file named ConcurrencyTest.ps1 with the following content

param(
 [Parameter(Position=0,
      Mandatory=$True,
      ValueFromPipeline=$True)]
    [INT]$Attempts=$(throw "You did not provide a value for Attempts parameter.")
    )

function DoStuff
{
    param(
    [Parameter(Position=0,
    Mandatory=$True,
    ValueFromPipeline=$True)]
    [int]$Iter
    )
    $root ="$env:programfiles"
    Set-Location $root\Pentaho\design-tools\data-integration
    cmd /c .\Kitchen.bat /file:C:\Source\Trunk\Transforms\Job_Ods_AggregateMongo.kjb /Level:Detailed | Out-File out.$Iter.log
}

workflow RunStuffParallel
{
    param(
        [Parameter(Position=0,
        Mandatory=$True)]
        [int]$MaxIter
    )

    $ExecutionAttempts=@(1..$MaxIter)
  
    ForEach -Parallel ($Attempt in $ExecutionAttempts)
    {
        DoStuff -Iter $Attempt
    }
}

RunStuffParallel -MaxIter $Attempts

Execute the test using .\ConcurrencyTest.ps1-Attempts 5

Microsoft Web API Custom Model Binding

Web API is great. It is easier than WCF service creation and costs 100% less than Service Stack (service Stack is more feature rich, FWIW). There is a ton of documentation on line for Web API and I like it better than the Service Stack documentation available outside of Pluralsight (my opinion, not a huge deal).

One topic I found annoyingly less well documented we how to parse the query of URI query string, validate the values, and build a conditional filter (or where clause) for the query string in the repository class. The link below has a really good article showing some of this.

using System;
using DemoService.Common;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Web.Http.Controllers;
using System.Web.Http.ModelBinding;
using Newtonsoft.Json;

namespace DemoService.Models
{
    public class ServiceOptionsModelBinder : IModelBinder
    {

       
        public bool BindModel(HttpActionContext actionContext, ModelBindingContext bindingContext)
        {

            var key = bindingContext.ModelName;
            var val = bindingContext.ValueProvider.GetValue(key);
            List<KeyValuePair<string, string>> requestValuePairs;
            var request = actionContext.Request;
            var requestMethod = request.Method;
            var requestHeader = JsonConvert.DeserializeObject<QueryObj>
                  (request.Content.Headers.ToDictionary());
           

            //Check and get source data from uri
            if (!string.IsNullOrEmpty(request.RequestUri.Query))
            {
                //also consider using QueryStringValueProvider
                requestValuePairs = request.GetQueryNameValuePairs().ToList();
            }
            //TODO: when we need to create a POST request, fix the type mismatch below
            //Check and get source data from body
            else if (request.Content.IsFormData())
            {
                var requestBody = request.Content.ReadAsStringAsync().Result;
                requestValuePairs = null;
                //requestValuePairs = Parsers.ConvertToKvp(requestBody);
            }

            else throw new NotSupportedException("Not supported, Aint HTTP compatible");
            bindingContext.Model = requestValuePairs;          

            return true;
        }
    }


}


REFS:
http://www.strathweb.com/2013/04/asp-net-web-api-parameter-binding-part-1-understanding-binding-from-uri/
http://www.codeproject.com/Articles/701182/A-Custom-Model-Binder-for-Passing-Complex-Objects
http://stackoverflow.com/questions/29393442/custom-model-binder-for-a-base-class-in-web-api

Reading List

The Design and Implementation of Probabilistic Programming Languages
Visualizing Algorithms
Probabilistic Models of Cognition
A Dynamic Programming Algorithm for Inference in Recursive Probabilistic Programs
Learning Structured Generative Concepts