Friday, February 22, 2019

Creating A Secure Schema in PostgreSQL



Your SUPERUSER account may be deployed to the database as part of provisioning your instance. This is the case when you use AWS RDS. If so you manage admin rights using a different tooling (Puppet, Terraform, Ansible).
A basic security model in an idempotent parameterized script may look like this 

-- ROLES WE USE FOR SCHEMA ACCESS GRANTS

--Role
DO $$
BEGIN
CREATE ROLE sisense_or_somthing_read_role;    
EXCEPTION   
WHEN duplicate_object THEN
RAISE NOTICE 'Role already exists. Ignoring...';    
END$$;

--Role
DO $$
BEGIN
CREATE ROLE sisense_or_somthing_readwrite_role IN ROLE sisense_or_somthing_read_role;    
EXCEPTION   
WHEN duplicate_object THEN
RAISE NOTICE 'Role already exists. Ignoring...';    
END$$;

--Role
DO $$
BEGIN
CREATE ROLE sisense_or_somthing_exececutor IN ROLE sisense_or_somthing_readwrite_role;
EXCEPTION   
WHEN duplicate_object THEN
RAISE NOTICE 'Role already exists. Ignoring...';    
END$$;

--Role
DO $$
BEGIN
    CREATE USER sisense_or_somthing_reader WITH PASSWORD :var_pwd1 IN ROLE sisense_or_somthing_read_role;
EXCEPTION   
WHEN duplicate_object THEN
RAISE NOTICE 'Role already exists. Ignoring...';    
END$$;

--Role
DO $$
BEGIN
CREATE USER sisense_or_somthing_readerwriter WITH PASSWORD :var_pwd2 IN ROLE sisense_or_somthing_readwrite_role;
EXCEPTION   
WHEN duplicate_object THEN
RAISE NOTICE 'Role already exists. Ignoring...';    
END$$;

--Role
DO $$
BEGIN
CREATE USER sisense_or_somthing_maintenance WITH PASSWORD :var_pwd3 IN ROLE sisense_or_somthing_exececutor;
EXCEPTION   
WHEN duplicate_object THEN
RAISE NOTICE 'Role already exists. Ignoring...';    
END$$;

CREATE SCHEMA IF NOT EXISTS sisense_or_somthing;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO sisense_or_somthing_read_role;
GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA public TO sisense_or_somthing_exececutor;
GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA sisense_or_somthing TO sisense_or_somthing_exececutor;
GRANT SELECT ON ALL TABLES IN SCHEMA sisense_or_somthing TO sisense_or_somthing_read_role;
GRANT INSERT ON ALL TABLES IN SCHEMA sisense_or_somthing TO sisense_or_somthing_readwrite_role;
GRANT UPDATE ON ALL TABLES IN SCHEMA sisense_or_somthing TO sisense_or_somthing_readwrite_role;
GRANT DELETE ON ALL TABLES IN SCHEMA sisense_or_somthing TO sisense_or_somthing_readwrite_role;
GRANT TRUNCATE ON ALL TABLES IN SCHEMA sisense_or_somthing TO sisense_or_somthing_readwrite_role;


A validation query or two can help
   SELECT 
   pr.oid role_oid, pr.rolname role_name, pr.rolcanlogin, pr.rolinherit, pr_p.rolname parent_role, 
   m.roleid role_member_oid, m.grantor, m.admin_option,
   pr_g.rolname grantor_name, pr_g.rolcanlogin, pr_g.rolinherit
   FROM 
   pg_roles pr
   left join pg_auth_members m ON m.member = pr.oid
   left join pg_roles pr_g on pr_g.oid = m.grantor
   left join pg_roles pr_p on pr_p.oid = m.roleid
   WHERE pr.rolname in (
'kuras_or_something_reader', 'kuras_or_something_readerwriter', 
'kuras_or_something_writer', 'kuras_or_something_wrtie_role', 'kuras_or_something_exececutor', 
'kuras_or_something_maintenance')

SELECT grantee, string_agg(privilege_type, ', ') AS privileges
FROM information_schema.role_table_grants 
WHERE table_name=‘model_results_4’
GROUP BY grantee;

And the revert migration script would be something like this. 

DROP SCHEMA IF EXISTS sisense_or_somthing;
DROP USER IF EXISTS sisense_or_somthing_admin;
DROP USER IF EXISTS sisense_or_somthing_reader;
DROP USER IF EXISTS sisense_or_somthing_readerwriter;
DROP USER IF EXISTS sisense_or_somthing_maintenance;
DROP ROLE IF EXISTS sisense_or_somthing_read_role;
DROP ROLE IF EXISTS sisense_or_somthing_readwrite_role;
DROP ROLE IF EXISTS sisense_or_somthing_exececutor;

Wednesday, June 6, 2018

Data Architecture - Document Model Design

Relational modeling has drawbacks to design implementation and to access speed. One differentiator is our common language around column based design patterns. Codd's normal forms still provide some guidance for us when it comes to document model design principals. A fundamental design guideline is ease of access is sometimes aided by denormalization, but more often aided by atomicity.

Below assume a document attribute is the same as a column and that a row is synonymous with a document.

NOTE: This is consistent across most engines except PostgreSQL, which has some designs that externalize highly read JSONB document elements into individual columns in the same row as the document for old school indexing purposes pre-GIN indexing.

1NF

Each column (or attribute) has no more than one value, each row (or document) is unique.
Can identify a unique candidate key.

2NF

Each column (or attribute) has no more than one value, each row (or document) is unique.
Can identify a unique candidate key.
All attributes or columns are dependent on the candidate key.

3NF

Each column (or attribute) has no more than one value, each row (or document) is unique.
Can identify a unique candidate key.
All attributes or columns are dependent on the candidate key.
No document or row creates a transitive association
  • A -> B -> C where A !-> C  
  • Driver -> Car -> Wheel Count where Driver has no Wheels. 
In columns this is easy. If one property should have its own unique id in the table, it moves to a new table. This is much more complicated in a document model. Below are different ways to model this.The first three create transitive associations. I can select driver and wheels where id = 1 and it will return the transitive value “Bob has four wheels.” The fourth creates a child document, but embeds it. 
  1. {id: “1”, [“Bob”, “car”, 4]}
  2. {id: “1”, “driver”:”Bob”, “vehicle”: “car”, “wheels”: “4”}
  3. {id: “1”, [“driver”:”Bob”, “vehicle”: “car”, “wheels”: “4”]}
  4. {id: “1”, “driver”:”Bob”, “car”:{id: “1”, "type“: "wheels”, "count": 4}}
  5. {id: “1”, “driver”:”Bob”, “car”:[{id: “1”, "type“: "wheels”, "count": 4}, {id: “2”, "type“: "hubs”, "count": 4},{id: “3”, "type“: "lug nuts”, "count": 16}]}

    Examples four and five are different forms of the same child document embedding. This form of normalization is faster to access and has less data access code to write. Well, it is faster as long as the sub-document relationship is bounded in its many-ness (somewhere between 1:1 and 1:5).

    Areas of Concern in Embedding


    When the set of possible children can balloon out the size of the document beyond reasonable access speed, it no longer represents a well defined single entity. It defines an entity relationship, one that requires multiple entities to allow for atomicity. Large heterogeneous documents are really poorly designed entity relationships hiding in an elephant costume.

    Many to many relationships also require externalization of child documents. 

    Data with different volatility requirements (write, update, delete operations) should be considered for externalized sub documnet storage.

    Sunday, November 12, 2017

    NodeJs Bash Build Script

    #!/bin/bash
    # Each lambda is technically its own project with its own dependencies
    function npmTask() {
      startingDir=$(pwd)
      npmCommand=$1
      for nodeDir in $startingDir/lambdas/*/;
        do
          echo "start node dir: $nodeDir from $startingDir"
          if [ -d $nodeDir  ] && [ $nodeDir != "$startingDir/lambdas/spec/" ];
          then
            echo "start node dir: $nodeDir from $startingDir"
            cd $nodeDir
            npm $npmCommand
            cd $startingDir
          else
            echo "no node modules found"
          fi
        done
    }
    npmTask "install"
    npmTask "build"
    npmTask "test"

    Sunday, October 1, 2017

    Python Map Reduce on Tuple List

    I was recently asked to read some pseudocode for calculating a class average of weighted scores. I wanted to work this out in Python to demonstrate how the design decisions around data structures impact implementation. Two independent lists assume order is maintained properly in two places. However it makes the code easier to explain.

    flatScores = [75,95,85,65]
    weights = [0.2,0.35,0.15,0.3]

    def mult(x,y): return x * y
    # map(mult,flatScores,weights)
    # [15.0, 33.25, 12.75, 19.5]
    reduce((lambda sum,score: (sum + score) ),list(map(mult,flatScores,weights)))/len(flatScores)
    #20.125

    A single list of tuples allows members to include all relevant data in one place. 

    scores = [[75,0.2],[95,0.35],[85,0.15],[65,0.3]]
    def mapper(scoreWeights): return map( lambda scoreWeight: scoreWeight[0] * scoreWeight[1] , scoreWeights )
    # mapper(scores)
    # [15.0, 33.25, 12.75, 19.5]
    reduce(lambda sum, cur: sum + cur, mapper(scores))/len(scores)
    #20.125

    Thursday, May 4, 2017

    Bash Function To Create Desktop Shortcuts

    Shortcuts are located in three places
    /usr/share/applications/intellij.desktop for all users
    a folder in the home dir for a given user
    ~/Desktop to see the shortcut on the desktop

    Format is
     [Desktop Entry]
     Version=13.0
     Type=Application
     Terminal=false
     Icon[en_US]=/home/rob/.intellij-13/bin/idea.png
     Name[en_US]=IntelliJ
     Exec=/home/rob/.intellij-13/bin/idea.sh
     Name=IntelliJ
     Icon=/home/rob/.intellij-13/bin/idea.png

    The function looks like this:

    function add_a_shortcut() {
      APP=$1
      ICON=$2
      EXEFILE=$3
      EXEC="bash -ic \"$EXEFILE &\""
      SHORTCUT="/usr/share/applications/$APP.desktop"
      USERSHORTCUT="/home/vagrant/Desktop/$APP.desktop"
      if [ -e "$EXEFILE" ]; then
        sudo touch $SHORTCUT $USERSHORTCUT
        sudo chmod 777 $SHORTCUT $USERSHORTCUT
        echo "[Desktop Entry]" | tee $SHORTCUT $USERSHORTCUT
        echo "Encoding=UTF-8" | tee -a $SHORTCUT $USERSHORTCUT
        echo "Comment=Launch $1" | tee -a $SHORTCUT $USERSHORTCUT
        echo "Type=Application" | tee -a $SHORTCUT $USERSHORTCUT
        echo "Terminal=false" | tee -a $SHORTCUT $USERSHORTCUT
        echo "Exec=$EXEC" | tee -a $SHORTCUT $USERSHORTCUT
        echo "Name=$APP" | tee -a $SHORTCUT $USERSHORTCUT
        echo "Icon=$ICON" | tee -a $SHORTCUT $USERSHORTCUT
       
        sudo chmod 644 $SHORTCUT
        sudo chown root:root $SHORTCUT
        sudo chown vagrant:vagrant $USERSHORTCUT
        echo "INFO: Created $SHORTCUT $USERSHORTCUT"
      else
        echo "ERROR: Failed to create $SHORTCUT $USERSHORTCUT"
      fi
    }
    EXAMPLE USAGE
    add_a_shortcut aggregation-designer /opt/pentaho/design-tools/aggregation-designer/aggregation-designer.app/Contents/Resources/pad.icns /opt/pentaho/design-tools/aggregation-designer/startaggregationdesigner.sh

    Thursday, March 16, 2017

    Why is it so hard to install VirtualBox Guest Additions on Centos

     yum update
     yum install gcc make kernel-devel bzip2
     mkdir -p /media/cdrom
     mount /dev/sr0 /media/cdrom
     sh /media/cdrom/VBoxLinuxAdditions.run

    Wednesday, December 14, 2016

    Notes on Slick 3

    Most examples are using await and that is blocking. Why use Slick if you are going to block? Look at onComplete, onSuccess, onFailure, and promise.

    ** Chaining transactions looks like this in one case
     
    db.run{ (for {
        customer <- customerTable.save();
        address <- a
    ddressTable.save()
       } yield ()).transactionally

    } 
     
    ** Returning a list of ids from a select looks like this
     (but you can also use returning keyword) 
     
    db = Database.forConfig("myactors.database")
    
    session = db createSession;session.conn.setCatalog("mydb")
    session.conn.setSchema("myschema")
    
    val query = Tables.rawStringTest.to[List].map(_.id)
    val resultSet = db.run(query.result)
     
    **this prints the list
    resultSet.onComplete(res =>   println(s"string returns $res") ) 
    **this waits for the future to return and uses the success/failure traits
    
    resultSet.onComplete{
      case Success(res) => res.foreach(x => db.run(Tables.rawStringTest.filter(_.id === x).delete))
      case Failure(ex) =>  println(s"error $ex")
    }
    **Testing is a challenge for aysync operations. Using an eventually retry block or a continuation both work.  This is with eventually.  
    it("ids are generated when None is passed in table calls"){
      val pk = None
      val validation = Tables.rawStringMessages.map(_.message_data)
    
      sut.insertStringMsg(None, "this is a fancy message", mockSuccessFunction(waiter), mockFailureFunction(waiter))
      sut.insertStringMsg(None, "this is another fancy message", mockSuccessFunction(waiter), mockFailureFunction(waiter))
    
      eventually(timeout(duration)) {
      val resultSet = Await.result(db.run(validation.result), duration)
    resultSet.onComplete{
     case Success(res) => resultSet.head.equals(99) shouldBe true 
     case Failure(ex) => fail(s"this failed $ex) 
    } 
    
    
     
    **Some ways to insert
    table returning table forceInsert JsonMessage(receiptId, messageData, None)
    table returning table insertOrUpdate JsonMessage(receiptId, messageData, None)
    table returning table += JsonMessageInsert(receiptId, messageData)
     
    **And any of these can return nothing
    table += JsonMessageInsert(receiptId, messageData)
    table insertOrUpdate JsonMessage(receiptId, messageData, None)
    
     
     
    /** * `Await` is what is used to ensure proper handling of blocking for `Awaitable` instances. * * While occasionally useful, e.g. for testing, it is recommended that you avoid Await * when possible in favor of callbacks and combinators like onComplete and use in * for comprehensions. Await will block the thread on which it runs, and could cause * performance and deadlock issues. */ 
    Clean up is a chore as well for integration tests. In this case I did clean up in the AfterAll block  to avoid cleaning data during concurrent test runs. 
    
    
    override def afterAll() {
      db = Database.forConfig("rtisystem.database")
    
      session = db createSession;  session.conn.setCatalog("rti")
      session.conn.setSchema("rti")
    
      val query = Tables.rawStringMessages.to[List].map(_.id)
      val resultSet = db.run(query.result)
    
      resultSet.onComplete{
        case Success(res) => res.foreach(x => db.run(Tables.rawStringMessages.filter(_.id === x).delete))
        case Failure(ex) =>  println(s"error $ex")
      }
    
      val jquery = Tables.jsonMessages.to[List].map(_.id)
      val jresultSet = db.run(jquery.result)
    
      jresultSet.onComplete{
        case Success(res) => res.foreach(x => db.run(Tables.jsonMessages.filter(_.id === x).delete))
        case Failure(ex) =>  println(s"error $ex")
      }
    
    
    
    
    This is an test example where I insert two records through a DAO extension method and compare them in a select query. 
    
    it("generates ids when None is passed instead of a long in the table call"){
      val pk = None
      val msgList: List[String] = List("this is a fancy message", "this is a fancy message also")
      val read = Tables.rawStringMessages.filter(row => row.message_data inSet msgList).map(_.id)
    
      sut.insertStringMsg(None, msgList.head,      mockSuccessFunction(waiter), mockFailureFunction(waiter))
      sut.insertStringMsg(None, msgList.tail.head, mockSuccessFunction(waiter), mockFailureFunction(waiter))
    
      eventually(timeout(duration)) {
        val resultSet = db.run(read.result)
        resultSet.onComplete {
          case Success(res) => assert(res.head != res.tail.head && res.head > 100000 && res.tail.head > 100000)
          case Failure(ex) => println(s"error $ex")
        }
      }
    } 
    
    
    
    
    
    
    
    
    val visitorId: Int = // whatever
    val locationCodes = List("loc1","loc2","loc3"...)
    // your query, with bind params.
    val q = for {
        v <- Visits 
        if v.visitor is visitorId.bind
        if v.location_code inSetBind locationCodes
      } yield v
    // have a look at the generated query.
    println(q.selectStatement)
    // run the query
    q.list
     
     
    *** To use a db default ***
    O.AutoInc 
    column_name.?
    Column Option[Timestamp]
     
    table += Class(id, data, Some(timestamp))
    
    
    Or a detailed version
    
    
    import slick.driver.PostgresDriver.api._
    import slick.lifted._
    import java.sql.{Date, Timestamp}
    
    /** A representation of the message decorated for Slick persistence
      * created_date should always be null on insert operations.
      * It is set at the database level to ensure time syncronicity
      * Id is the Twitter snowflake id. All columns NotNull unless declared as Option
      * */
    class RawMessages(tag: Tag) extends Table[(String, Option[String], Timestamp)](tag, Some("rti"), "RawMessages") {
      def id = column[String]("id", O.PrimaryKey)
      def MessageString = column[Option[String]]("MessageString")
      def CreatedDate = column[Timestamp]("CreatedDate", O.SqlType("timestamp default now()"))
      def * = (id, MessageString, CreatedDate)
    }
    

    Monday, November 21, 2016

    Notes on Log4Net

    1. Add log4net.config to project
    2. Add keys to app.config pointing to log4net config file and setting watch attribute.
    Watch dictates whether log4net polls for changes in the log4net config
        <add key="log4net.Config" value="log4net.config"/>
        <add key="log4net.Config.Watch" value="True"/>
    3. Add the configSsection to the app.config
      <configSections>
        <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net"/>
      </configSections>
    4. Add any appenders. Set log rollover and locking model for file appenders. Each appender must have a unique name. Include a layout

          <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="%date [%thread] %-5level %logger [%property{NDC}] - %message%newline - %type cow" />
          </layout>

    5. Add at least one appender name to the root logger. More than one appender can also be added. Note that you can filter log message types using <level value="All"/>
        <root>
          <level value="All"/>
          <appender-ref ref="ManagedColoredConsoleAppender"/>
        </root>


    If logging to Mongo you want to consider using a capped collection. Capped collections are disabled by default. A capped collection allows for a FIFO buffer like behavior for self regulating log storage.

    If logging to Rabbit (requires custom appender https://github.com/haf/log4net.RabbitMQ/blob/master/src/log4net.RabbitMQ/ExchangeBinding.cs) consider separate security and whether vhost should be the same as the app.

    Smashing Cassandra 4.6 Into Pentaho BA 6.1

    I am working on a POC with the Cassandra wide column store and the Pentaho reporting suite. At the time of this POC Pentaho support is behind the DataStax Cassandra release cycle, which is going to 5.0 in a few weeks. The thing I like about Pentaho is that it is designed to allow a huge amount of flexibility. So much it can destroy your mind with the possibilities.

    First off I had to change my ETL in Pentaho DI and report in Pentaho BA to use a generic database connector. The URL was the connection string URI to my Cassandra cluster

     and a generic table output step. I changed my report to use the same generic database connection. I had no idea if this would work and at first it failed utterly. But the error indicated a missing JDBC driver. So I started plugging in different JDBC drivers (BigSql, DBSchema, and Datastax). I could get each to connect to my version of Cassandra but they all threw cryptic errors like:
        Codec not found for requested operation: [int <-> java.lang.Long Dbschema]
        Codec not found for requested operation: [timestamp <-> com.datastax.driver.core.LocalDate]

          public Timestamp convertToDatabaseColumn(LocalDate ld) {
            return Timestamp.valueOf(ld);
        }

            @Override
    public timestamp format(LocalDate value) {
      if (value == null) {
      return "NULL";
      } 
      else {
      return convertToDatabaseColumn(value.getMillisSinceEpoch()).toDateTime();
      }
        }

    If you want to make a generic database connection with a JDBC connector you would use
    Build Command Required: & C:\tools\apache-maven-3.3.3\bin\mvn clean package site install
    URL Structure: jdbc:cassandra://host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[keyspace][?options]]
    URL Example: jdbc:cassandra://CassandraContactNode1,CassandraContactNode2,CassandraContactNode3/rti?consistency=LOCAL_QUORUM 
    Java Driver Class: com.datastax.driver.core.Connection

    To find the connection class you need you would open folders for the class path until you found a connection class that allowed you to pass in multiple nodes for your cluster and a consistency level. For a recent Apache update to Cassandra I downloaded the tarball and navigated to 

    D:\downloads\apache-cassandra-3.0.7-src.tar.gz\apache-cassandra-3.0.7-src.tar\apache-cassandra-3.0.7-src\src\java\org\apache\cassandra\transport\

    In there are a SimpleClient class that accepts a channel and version, a Client class that accepts a host, port, version, and ssl settings,  and a Connection class that accepts a more complex netty channel and the CQL version. 

    REFS:
    http://opensourceconnections.com/blog/2015/12/22/exploring-custom-typecodecs-in-the-cassandra-java-driver/


    PostgreSQL Create and Modified Triggers


    /* Use BEFORE UPDATE triggers for this as AFTER UPDATEcreates an infinite loop with the new comparison */CREATE OR REPLACE FUNCTION update_modified_column()
    RETURNS TRIGGER AS $$BEGIN   IF row(NEW.*) IS DISTINCT FROM row(OLD.*) THEN      NEW.moded = now();      RETURN NEW;   ELSE      RETURN OLD;   END IF;END;$$ language 'plpgsql';
     
    DROP TABLE IF EXISTS temper;
    CREATE TEMP TABLE IF NOT EXISTS temper(
      id INT,
      word VARCHAR,
      creatd TIMESTAMP WITH TIME ZONE DEFAULT NOW()::TIMESTAMP WITH TIME ZONE,
      moded TIMESTAMP WITH TIME ZONE);
    
    DROP TRIGGER IF EXISTS temper_update_date ON temper;
    
    CREATE TRIGGER temper_update_date BEFORE UPDATE ON temper FOR EACH ROW EXECUTE PROCEDURE update_modified_column();
    
    INSERT INTO temper (id, word) VALUES  (1, 'one'),
      (2, 'two');
    
    SELECT * FROM temper;
    
    UPDATE temper SET word = 'tttoooo' WHERE id = 1;
     
    
    
    SELECT * FROM temper;
     

     

    Thursday, November 10, 2016

    slick

    For a durable elastic scaling system Akka is hard to beat. However some of the advanced configuration is not easy to find end to end examples of.

    Things change with versions. This is relevant to
    Akka 2.4
    SBT 0.13.12
    Scala 2.10.6

    One decision is what to configure in code and what to configure in the application.conf file.

    Do you have a muti-actorsystem build process? If so you need each actor system to have a namespace in the config.

    logging-system {akka{}...}
    ai-system{akka{}...}

    /** An instance of the entire Config object loaded 
      * with the application.conf data  * @see src/main/resources/application.conf */
    val config = ConfigFactory.load()
    /** it is important to get a specifc system config namespace
      * if you have a multi akka system project  
    * for our single system setup we could have akka as a root node in the config*/
    val akkaConfig = config.getConfig("ai-system").withFallback(config)
    
    /** An akka actor system setup for the ai system application */
    val actorSystem = ActorSystem(name = "ai-system", config = akkaConfig)

    If you have a single system there is no need for the root namespace
    akka{}...

    And Akka will find the akka{} section
    /** An instance of the entire Config object loaded 
      * with the application.conf data  * @see src/main/resources/application.conf */
    val config = ConfigFactory.load()/
    ** An akka actor system setup for the ai system application */
    val actorSystem = ActorSystem(name = "ai-system", config = config)

    Sample App Object
    package com.company.aisystem
    
    import akka.actor.SupervisorStrategy.{Decider, Escalate, Restart, Resume, Stop}
    import akka.actor.{ActorInitializationException, ActorKilledException, 
    ActorSystem, Inbox, OneForOneStrategy, Props, SupervisorStrategy}
    import akka.routing._
    import com.company.aisystem.actors.{OrchestratorActor, PersistorActor, 
    ValidationActor}
    import com.typesafe.config.ConfigFactory
    
    import scala.concurrent.duration._
    
    /** Acts as the "main" class for the application to start  *  
    * It extends the scala: "App" trait  */
    object aiSystemApp extends App {
      /** An instance of the entire Config object loaded with the application.conf data    
    * @see src/main/resources/application.conf */  
    val config = ConfigFactory.load()
      /** it is important to get a specifc system config namespace 
    if you have a multi akka system project
        * for our single system setup we could have akka as a root node in the config*/ 
     val aisystemAkkaConfig = config.getConfig("aisystem").withFallback(config)
    
      /** An akka actor system setup for the ai system application */ 
     val actorSystem = ActorSystem(name = "aisystem", 
    config = aisystemAkkaConfig)
    
      /** A one for one supervisor strategy to be used in the router */ 
     val superviseOneForOne =
        OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 60.seconds){
          case _: ArithmeticException      => Resume
          case _: NullPointerException     => Restart
          case _: IllegalArgumentException => Stop
          case _: Exception                => Escalate
        }
    
      /** A default akka inbox for the ai system actor system. not sure 
    we need to declare this */  
    val inbox = Inbox.create(actorSystem)
    
      /** An akka actor ref of the persistor actor */  val persistorActor = 
    actorSystem.actorOf(Props[PersistorActor]
    .withDispatcher("aisystem.akka.fork-join-dispatcher"), 
    config.getString("aisystem.persistor-actor-name"))
    
      /** An akka actor ref of the validation actor *
    ///TODO why are we passing in another actor ref as the whole props?  
    val validationActor = actorSystem.actorOf(ValidationActor
    .props(persistorActor)
    .withDispatcher("aisystem.akka.ai-thread-pool-dispatcher"), 
    config.getString("aisystem.validation-actor-name"))
    
      final val defaultStrategy: SupervisorStrategy = {
        def defaultDecider: Decider = {
          case _: ActorInitializationException ⇒ Stop
          case _: ActorKilledException         ⇒ Stop
          case _: Exception                    ⇒ Restart
        }
        OneForOneStrategy()(defaultDecider)
      }
    
      /** An akka actor ref of the orchestrator actor */   
     //todo: setup resizer = Option[DefaultOptimalSizeExploringResizer]  
    val orchestratorActor = actorSystem.actorOf(OrchestratorActor
          .props(validationActor)
          .withRouter(SmallestMailboxPool(
            nrOfInstances = 5,        supervisorStrategy = superviseOneForOne,      
      routerDispatcher = "aisystem.akka.ai-thread-pool-dispatcher",        
    usePoolDispatcher = false)),     
     config.getString("aisystem.orchestrator-actor-name"))
    
      ...other app init code
    Sample Config
    aisystem {
    
    queueBroker{}
    
    database{}
    akka {
        # expose actor system level debug logging for local    
    loglevel = DEBUG //options include DEBUG and off    
    # stdout-loglevel is only in effect during system startup and shutdown   
     stdout-loglevel = DEBUG 
    #options include DEBUG and off  
      log-config-on-start = on
        actor {
          debug {
            # enable DEBUG logging of all AutoReceiveMessages (Kill, PoisonPill et.c.)  
            autoreceive = on
            # enable DEBUG logging of subscription changes on the eventStream
            event-stream = on
            # enable DEBUG logging of all LoggingFSMs for events, transitions and timers  
            fsm = on
            # enable DEBUG logging of actor lifecycle changes
            lifecycle = on
            # enable DEBUG logging of message receive events
            receive = on
            # enable DEBUG logging of unhandled messages
            unhandled = on
          }
          # Timeout for actors
          timeout = 30s
          # Timeout for ActorSystem.actorOf
          creation-timeout = 30s
          serialize-messages = off #generally only enable for testing with serialzers  
      }
        fork-join-dispatcher {
          # Dispatcher is the name of the event-based dispatcher
          type = Dispatcher
          # What kind of ExecutionService to use
          executor = "fork-join-executor"  
        # Configuration for the fork join pool      fork-join-executor {
            # Min number of threads to cap factor-based parallelism number to
            parallelism-min = 2 
           # Parallelism (threads) ... ceil(available processors * factor)
            parallelism-factor = 2.0
            # Max number of threads to cap factor-based parallelism number to
            parallelism-max = 10      }
          # Throughput defines the maximum number of messages to be 
         # processed per actor before the thread jumps to the next actor. 
         # Set to 1 for as fair as possible.
          throughput = 100
          throughput-deadline-time = 10ms
          supervisorStrategy = "Restart"    }
        ai-thread-pool-dispatcher {
          # Dispatcher is the name of the event-based dispatcher 
         type = Dispatcher
          # What kind of ExecutionService to use
          executor = "thread-pool-executor"
          # Configuration for the thread pool
          thread-pool-executor {
            # minimum number of threads to cap factor-based core number to 
           core-pool-size-min = 2 
           # No of core threads ... ceil(available processors * factor)
            core-pool-size-factor = 2.0 
           # maximum number of threads to cap factor-based number to
            core-pool-size-max = 10      }
          # Throughput defines the maximum number of messages to be
          # processed per actor before the thread jumps to the next actor. 
         # Set to 1 for as fair as possible. 
         throughput = 1      supervisorStrategy = "Restart"    }
      }
      akka.actor.deployment {
        //TODO move this from App object to deployment    Orchestrator {
          router = smallest-mailbox-pool
          nr-of-instances = 5      dispatcher = aisystem.akka.aii-dispatcher
        }
        Persistor {
          router = smallest-mailbox-pool
          nr-of-instances = 5      dispatcher = aisystem.akka.ai-dispatcher
        }
        Validator {
          router = smallest-mailbox-pool
          nr-of-instances = 5    }
    }
    

    *REFS*
    http://doc.akka.io/docs/akka/2.0.2/intro/getting-started-first-scala.html
    Supervisor Strategy
    Fault Tolerance Primer
    http://danielwestheide.com/blog/2013/03/20/the-neophytes-guide-to-scala-part-15-dealing-with-failure-in-actor-systems.html
    routers-in-akka-with-parameterized-actors
    Router Choices

    Wednesday, November 2, 2016

    Docker on Deb Cheatsheet

    getting help
    $ docker --help

    what images are running right now. related to docker run
    $ docker stats

    what containers are running. related to docker start
    $ docker ps

    Clean up exited containers
    $ docker rm -v $(docker ps -a -q -f status=exited)
    All
    $ docker stop $(docker ps -a -q);docker rm $(docker ps -a -q)

    make sure docker service starts (two ways to do it, or use a daemon)
    $ sudo service docker start
    $ sudo chkconfig docker on

    what images can I start right now
    $ docker images

    Repo of <none> is an intermediate image that can optionally be discarded with a runtime flag.

    is the last run container still up
    docker inspect --format '{{.State.Running}}' $(docker ps -lq)
    

    Clean up images
    $ docker rmi  651505724f29

    Start a container from an image
    docker start <image>

    start an instance of a container by name instead of guid
    this will immediately terminate if there is not a foreground process running
    this will keep a terminal running

    $ docker run <REPO>:<TAG>

    start an instance of a container as a daemon and keep a running fg process
    the tail of dev null keeps docker alive
    docker run -d 09c51b079466 tail -f /dev/null

    start an instance of a container with a term running for commands
    docker run -t 09c51b079466 bash

    using tail -f /dev/null with the daemon option is a work around if you do not have a script  initializing your application as your initial starting point that runs its own  make your container continuously running is point to a shell file in docker which will keep your application running. You can try with a start.sh file

    Eg: docker run -d centos sh /yourlocation/start.sh

    This start.sh should point to a never ending application.
    In case if you dont want any application to be running,
    you can install monit which will keep your docker container running.


    Access a running container
    this uses the cute name. you can also trap the container id at run 
    docker exec infallible_newton ls -l|grep u*

    this uses the id from variable
    docker exec $RABCID ls -l|grep u*


    details on a container
    $ docker inspect 09c51b079466

    find your ip
    $ CIP=$(docker inspect 09c51b079466 | grep IPAddress| grep 172| head -1)
    $ $CIP

    Capture your container id as a variable
    RABCID=$(docker run -d rabbitmq:3-management); docker inspect $RABCID  | grep IPAddress| grep 172| head -1


    networking
    docker network ls ; ifconfig
    docker inpsect rtinet

    Below shows two ways to join a network. First adds the network at run
    #!/usr/bin/env bash
    docker rm rti-rabbit -f
    docker network create rtinet
    docker run -d --network rtinet --hostname rabbit1 --name rti-rabbit -p 15672:15672 -p 4369:4369 -p 5671:5671 -p 5672:5672 rabbitmq:3-management

    Second connects after run
    DMDCID=$(docker run -d dynamicmessagediscovery:0.1-SNAPSHOT tail -f /dev/null);
    DMDIP=$(docker inspect $DMDCID  | grep IPAddress| grep 172| head -1)
    docker network connect rtinet $DMDCID


    view logs
    $ docker logs f69d53c188af

    Is an instance of my container running
    $ docker ps -a | grep -i 09c51b079466

    Sample Startup Script
    docker stop $(docker ps -a -q) 
    docker rm rti-rabbit -f
    docker rm dmdhost -f
    docker rm pghost -f
    docker network rm rtinet
    docker network create rtinet
    RABCID=$(docker run -d --network rtinet --hostname rabbit1 --name rti-rabbit -p 15672:15672 -p 4369:4369 -p 5671:5671 -p 5672:5672 rabbitmq:3-management)
    DMDCID=$(docker run -d --network rtinet --hostname dmdhost --name dmdhost -p 15200:15672 -p 15201:4369 -p 15202:5671 -p 15203:5672 dynamicmessagediscovery:0.1-SNAPSHOT tail -f /dev/null);
    PGCID=$(docker run -d --network rtinet --hostname pghost --name pghost -p 5432:5432 postgres)

    DMDIP=$(docker inspect $DMDCID | grep IPAddress| grep 172| head -1)
    RABIP=$(docker inspect $RABCID | grep IPAddress| grep 172| head -1)
    PGIP=$(docker inspect  $PGCID  | grep IPAddress| grep 172| head -1)

    firefox -new-tab "http://$RABIP:15672/#/queues/"


    sbt test stage deploy and start
    $ sbt test stage docker:publishLocal; CID=$(docker run -d dynamicmessagediscovery:0.1-SNAPSHOT tail -f /dev/null);docker inspect $CID | grep IPAddress | grep 127|head -1

    <> run runs an image
    <> start starts a container.
    <> exec accesses a running container

    To make a daemon (bg process)
    /usr/local/bin/confd -interval=30 -backend etcd -node $CONFIG_CENTER &






    1. A daemon is a background, non-interactive program. It is detached from the keyboard and display of any interactive user. The word daemon for denoting a background program is from the Unix culture; it is not universal.
    2. A server is a program which responds to requests from other programs over some inter-process communication mechanism (usually over a network). A service is what a server provides. For example, the NFS port mapping service is provided as a separate portmap server, which is implemented as the portmapd daemon.
      A server doesn't have to be a daemon, but usually is. A user application with a GUI could have a server built into it: for instance, a file-sharing application. Another example is the X Window server, which is anything but in the background: it takes over your screen, keyboard and pointing device. It is a server because it responds to requests from applications (to create and manipulate windows, et cetera), which can even be elsewhere on the network. But the X server also responds to your every keystroke and mouse movement.
    3. A process is one or more threads of execution together with their shared set of resources, the most important of which are the address space and open file descriptors. A process creates an environment for these threads of execution which looks like they have an entire machine all to themselves: it is a virtual machine.
      Inside a process, the resources of other processes, and of the kernel, are invisible and not directly accessible (at least not to a thread which is executing user-space code). For example, there is no way to refer to the open files of another process, or their memory space; it is as if those things do not even exist.
      The process, and its relation to the kernel and other processes, perhaps constitutes the most important abstraction in Unix-like operating systems. The resources of the system are compartmentalized into processes, and nearly everything is understood as happening inside one process or another.


    https://github.com/wsargent/docker-cheat-sheet

    Monday, September 26, 2016

    Ubuntu On Windows


    Enable Hyper V
    Start  - Control Panel - Programs - Turn Windows Features On - enable all - Reboot



    Enable Networking
    Open up Hyper-V Manager.
    Right-click on the name of the Hyper-V host and select Virtual Switch Manager...
    Under ‘Virtual Switches’, select New virtual network switch.
    Under ‘What type of virtual switch do you want to create?’, select External.
    Select the Create Virtual Switch button.


    Under ‘Virtual Switch Properties’, give the new switch a name such as VM Virtual Switch.
    Under ‘Connection Type’, ensure that External Network has been selected.
    Select the physical network card to be paired with the new virtual switch. This is the network card that is physically connected to the network. I use my wireless to prevent disruptions in my work.



    Create a New VM
    New Virtual Machine