Sunday, October 1, 2017

Python Map Reduce on Tuple List

I was recently asked to read some pseudocode for calculating a class average of weighted scores. I wanted to work this out in Python to demonstrate how the design decisions around data structures impact implementation. Two independent lists assume order is maintained properly in two places. However it makes the code easier to explain.

flatScores = [75,95,85,65]
weights = [0.2,0.35,0.15,0.3]

def mult(x,y): return x * y
# map(mult,flatScores,weights)
# [15.0, 33.25, 12.75, 19.5]
reduce((lambda sum,score: (sum + score) ),list(map(mult,flatScores,weights)))/len(flatScores)
#20.125

A single list of tuples allows members to include all relevant data in one place. 

scores = [[75,0.2],[95,0.35],[85,0.15],[65,0.3]]
def mapper(scoreWeights): return map( lambda scoreWeight: scoreWeight[0] * scoreWeight[1] , scoreWeights )
# mapper(scores)
# [15.0, 33.25, 12.75, 19.5]
reduce(lambda sum, cur: sum + cur, mapper(scores))/len(scores)
#20.125

Thursday, May 4, 2017

Bash Function To Create Desktop Shortcuts

Shortcuts are located in three places
/usr/share/applications/intellij.desktop for all users
a folder in the home dir for a given user
~/Desktop to see the shortcut on the desktop

Format is
 [Desktop Entry]
 Version=13.0
 Type=Application
 Terminal=false
 Icon[en_US]=/home/rob/.intellij-13/bin/idea.png
 Name[en_US]=IntelliJ
 Exec=/home/rob/.intellij-13/bin/idea.sh
 Name=IntelliJ
 Icon=/home/rob/.intellij-13/bin/idea.png

The function looks like this:

function add_a_shortcut() {
  APP=$1
  ICON=$2
  EXEFILE=$3
  EXEC="bash -ic \"$EXEFILE &\""
  SHORTCUT="/usr/share/applications/$APP.desktop"
  USERSHORTCUT="/home/vagrant/Desktop/$APP.desktop"
  if [ -e "$EXEFILE" ]; then
    sudo touch $SHORTCUT $USERSHORTCUT
    sudo chmod 777 $SHORTCUT $USERSHORTCUT
    echo "[Desktop Entry]" | tee $SHORTCUT $USERSHORTCUT
    echo "Encoding=UTF-8" | tee -a $SHORTCUT $USERSHORTCUT
    echo "Comment=Launch $1" | tee -a $SHORTCUT $USERSHORTCUT
    echo "Type=Application" | tee -a $SHORTCUT $USERSHORTCUT
    echo "Terminal=false" | tee -a $SHORTCUT $USERSHORTCUT
    echo "Exec=$EXEC" | tee -a $SHORTCUT $USERSHORTCUT
    echo "Name=$APP" | tee -a $SHORTCUT $USERSHORTCUT
    echo "Icon=$ICON" | tee -a $SHORTCUT $USERSHORTCUT
   
    sudo chmod 644 $SHORTCUT
    sudo chown root:root $SHORTCUT
    sudo chown vagrant:vagrant $USERSHORTCUT
    echo "INFO: Created $SHORTCUT $USERSHORTCUT"
  else
    echo "ERROR: Failed to create $SHORTCUT $USERSHORTCUT"
  fi
}
EXAMPLE USAGE
add_a_shortcut aggregation-designer /opt/pentaho/design-tools/aggregation-designer/aggregation-designer.app/Contents/Resources/pad.icns /opt/pentaho/design-tools/aggregation-designer/startaggregationdesigner.sh

Thursday, March 16, 2017

Why is it so hard to install VirtualBox Guest Additions on Centos

 yum update
 yum install gcc make kernel-devel bzip2
 mkdir -p /media/cdrom
 mount /dev/sr0 /media/cdrom
 sh /media/cdrom/VBoxLinuxAdditions.run

Wednesday, December 14, 2016

Notes on Slick 3

Most examples are using await and that is blocking. Why use Slick if you are going to block? Look at onComplete, onSuccess, onFailure, and promise.

** Chaining transactions looks like this in one case
 
db.run{ (for {
    customer <- customerTable.save();
    address <- a
ddressTable.save()
   } yield ()).transactionally

 
** Returning a list of ids from a select looks like this
 (but you can also use returning keyword) 
 
db = Database.forConfig("myactors.database")

session = db createSession;session.conn.setCatalog("mydb")
session.conn.setSchema("myschema")

val query = Tables.rawStringTest.to[List].map(_.id)
val resultSet = db.run(query.result)
 
**this prints the list
resultSet.onComplete(res =>   println(s"string returns $res") ) 
**this waits for the future to return and uses the success/failure traits
resultSet.onComplete{
  case Success(res) => res.foreach(x => db.run(Tables.rawStringTest.filter(_.id === x).delete))
  case Failure(ex) =>  println(s"error $ex")
}
**Testing is a challenge for aysync operations. Using an eventually retry block or a continuation both work.  This is with eventually.  
it("ids are generated when None is passed in table calls"){
  val pk = None
  val validation = Tables.rawStringMessages.map(_.message_data)

  sut.insertStringMsg(None, "this is a fancy message", mockSuccessFunction(waiter), mockFailureFunction(waiter))
  sut.insertStringMsg(None, "this is another fancy message", mockSuccessFunction(waiter), mockFailureFunction(waiter))

  eventually(timeout(duration)) {
  val resultSet = Await.result(db.run(validation.result), duration)
resultSet.onComplete{
 case Success(res) => resultSet.head.equals(99) shouldBe true 
 case Failure(ex) => fail(s"this failed $ex) 
} 

 
**Some ways to insert
table returning table forceInsert JsonMessage(receiptId, messageData, None)
table returning table insertOrUpdate JsonMessage(receiptId, messageData, None)
table returning table += JsonMessageInsert(receiptId, messageData)
 
**And any of these can return nothing
table += JsonMessageInsert(receiptId, messageData)
table insertOrUpdate JsonMessage(receiptId, messageData, None)
 
 
/** * `Await` is what is used to ensure proper handling of blocking for `Awaitable` instances. * * While occasionally useful, e.g. for testing, it is recommended that you avoid Await * when possible in favor of callbacks and combinators like onComplete and use in * for comprehensions. Await will block the thread on which it runs, and could cause * performance and deadlock issues. */ 
Clean up is a chore as well for integration tests. In this case I did clean up in the AfterAll block  to avoid cleaning data during concurrent test runs. 

override def afterAll() {
  db = Database.forConfig("rtisystem.database")

  session = db createSession;  session.conn.setCatalog("rti")
  session.conn.setSchema("rti")

  val query = Tables.rawStringMessages.to[List].map(_.id)
  val resultSet = db.run(query.result)

  resultSet.onComplete{
    case Success(res) => res.foreach(x => db.run(Tables.rawStringMessages.filter(_.id === x).delete))
    case Failure(ex) =>  println(s"error $ex")
  }

  val jquery = Tables.jsonMessages.to[List].map(_.id)
  val jresultSet = db.run(jquery.result)

  jresultSet.onComplete{
    case Success(res) => res.foreach(x => db.run(Tables.jsonMessages.filter(_.id === x).delete))
    case Failure(ex) =>  println(s"error $ex")
  }


This is an test example where I insert two records through a DAO extension method and compare them in a select query. 

it("generates ids when None is passed instead of a long in the table call"){
  val pk = None
  val msgList: List[String] = List("this is a fancy message", "this is a fancy message also")
  val read = Tables.rawStringMessages.filter(row => row.message_data inSet msgList).map(_.id)

  sut.insertStringMsg(None, msgList.head,      mockSuccessFunction(waiter), mockFailureFunction(waiter))
  sut.insertStringMsg(None, msgList.tail.head, mockSuccessFunction(waiter), mockFailureFunction(waiter))

  eventually(timeout(duration)) {
    val resultSet = db.run(read.result)
    resultSet.onComplete {
      case Success(res) => assert(res.head != res.tail.head && res.head > 100000 && res.tail.head > 100000)
      case Failure(ex) => println(s"error $ex")
    }
  }
} 




val visitorId: Int = // whatever
val locationCodes = List("loc1","loc2","loc3"...)
// your query, with bind params.
val q = for {
    v <- Visits 
    if v.visitor is visitorId.bind
    if v.location_code inSetBind locationCodes
  } yield v
// have a look at the generated query.
println(q.selectStatement)
// run the query
q.list
 
 
*** To use a db default ***
O.AutoInc 
column_name.?
Column Option[Timestamp]
 
table += Class(id, data, Some(timestamp))

Or a detailed version

import slick.driver.PostgresDriver.api._
import slick.lifted._
import java.sql.{Date, Timestamp}

/** A representation of the message decorated for Slick persistence
  * created_date should always be null on insert operations.
  * It is set at the database level to ensure time syncronicity
  * Id is the Twitter snowflake id. All columns NotNull unless declared as Option
  * */
class RawMessages(tag: Tag) extends Table[(String, Option[String], Timestamp)](tag, Some("rti"), "RawMessages") {
  def id = column[String]("id", O.PrimaryKey)
  def MessageString = column[Option[String]]("MessageString")
  def CreatedDate = column[Timestamp]("CreatedDate", O.SqlType("timestamp default now()"))
  def * = (id, MessageString, CreatedDate)
}

Monday, November 21, 2016

Notes on Log4Net

1. Add log4net.config to project
2. Add keys to app.config pointing to log4net config file and setting watch attribute.
Watch dictates whether log4net polls for changes in the log4net config
    <add key="log4net.Config" value="log4net.config"/>
    <add key="log4net.Config.Watch" value="True"/>
3. Add the configSsection to the app.config
  <configSections>
    <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net"/>
  </configSections>
4. Add any appenders. Set log rollover and locking model for file appenders. Each appender must have a unique name. Include a layout

      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%date [%thread] %-5level %logger [%property{NDC}] - %message%newline - %type cow" />
      </layout>

5. Add at least one appender name to the root logger. More than one appender can also be added. Note that you can filter log message types using <level value="All"/>
    <root>
      <level value="All"/>
      <appender-ref ref="ManagedColoredConsoleAppender"/>
    </root>


If logging to Mongo you want to consider using a capped collection. Capped collections are disabled by default. A capped collection allows for a FIFO buffer like behavior for self regulating log storage.

If logging to Rabbit (requires custom appender https://github.com/haf/log4net.RabbitMQ/blob/master/src/log4net.RabbitMQ/ExchangeBinding.cs) consider separate security and whether vhost should be the same as the app.

Smashing Cassandra 4.6 Into Pentaho BA 6.1

I am working on a POC with the Cassandra wide column store and the Pentaho reporting suite. At the time of this POC Pentaho support is behind the DataStax Cassandra release cycle, which is going to 5.0 in a few weeks. The thing I like about Pentaho is that it is designed to allow a huge amount of flexibility. So much it can destroy your mind with the possibilities.

First off I had to change my ETL in Pentaho DI and report in Pentaho BA to use a generic database connector. The URL was the connection string URI to my Cassandra cluster

 and a generic table output step. I changed my report to use the same generic database connection. I had no idea if this would work and at first it failed utterly. But the error indicated a missing JDBC driver. So I started plugging in different JDBC drivers (BigSql, DBSchema, and Datastax). I could get each to connect to my version of Cassandra but they all threw cryptic errors like:
    Codec not found for requested operation: [int <-> java.lang.Long Dbschema]
    Codec not found for requested operation: [timestamp <-> com.datastax.driver.core.LocalDate]

      public Timestamp convertToDatabaseColumn(LocalDate ld) {
        return Timestamp.valueOf(ld);
    }

        @Override
public timestamp format(LocalDate value) {
  if (value == null) {
  return "NULL";
 
  else {
  return convertToDatabaseColumn(value.getMillisSinceEpoch()).toDateTime();
  }
    }

If you want to make a generic database connection with a JDBC connector you would use
Build Command Required: & C:\tools\apache-maven-3.3.3\bin\mvn clean package site install
URL Structure: jdbc:cassandra://host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[keyspace][?options]]
URL Example: jdbc:cassandra://CassandraContactNode1,CassandraContactNode2,CassandraContactNode3/rti?consistency=LOCAL_QUORUM 
Java Driver Class: com.datastax.driver.core.Connection

To find the connection class you need you would open folders for the class path until you found a connection class that allowed you to pass in multiple nodes for your cluster and a consistency level. For a recent Apache update to Cassandra I downloaded the tarball and navigated to 

D:\downloads\apache-cassandra-3.0.7-src.tar.gz\apache-cassandra-3.0.7-src.tar\apache-cassandra-3.0.7-src\src\java\org\apache\cassandra\transport\

In there are a SimpleClient class that accepts a channel and version, a Client class that accepts a host, port, version, and ssl settings,  and a Connection class that accepts a more complex netty channel and the CQL version. 

REFS:
http://opensourceconnections.com/blog/2015/12/22/exploring-custom-typecodecs-in-the-cassandra-java-driver/


PostgreSQL Create and Modified Triggers


/* Use BEFORE UPDATE triggers for this as AFTER UPDATEcreates an infinite loop with the new comparison */CREATE OR REPLACE FUNCTION update_modified_column()
RETURNS TRIGGER AS $$BEGIN   IF row(NEW.*) IS DISTINCT FROM row(OLD.*) THEN      NEW.moded = now();      RETURN NEW;   ELSE      RETURN OLD;   END IF;END;$$ language 'plpgsql';
 
DROP TABLE IF EXISTS temper;
CREATE TEMP TABLE IF NOT EXISTS temper(
  id INT,
  word VARCHAR,
  creatd TIMESTAMP WITH TIME ZONE DEFAULT NOW()::TIMESTAMP WITH TIME ZONE,
  moded TIMESTAMP WITH TIME ZONE);

DROP TRIGGER IF EXISTS temper_update_date ON temper;

CREATE TRIGGER temper_update_date BEFORE UPDATE ON temper FOR EACH ROW EXECUTE PROCEDURE update_modified_column();

INSERT INTO temper (id, word) VALUES  (1, 'one'),
  (2, 'two');

SELECT * FROM temper;

UPDATE temper SET word = 'tttoooo' WHERE id = 1;
 

SELECT * FROM temper;