Sisters

“I believe that children are our future. Teach them well, and let them lead the way…” – Whitney Houston

After an evening of library time, Daddy’s Crossfit, and a Lacrosse pickup, my two girls, Zoe (the oldest) and Lydia were tired and hungry. During a quick evening meal of leftover lasagna, I popped open the laptop to work on a quick kata, when my oldest asked “Whatcha doin’?”

Zoe’s interest in my work is rare, so I happily showed her some of my F# test code. She looked around. Even asked a few questions.

So I popped open the FSI and showed her how stuff works.

Microsoft (R) F# Interactive version 10.2.3 for F# 4.5
Copyright (c) Microsoft Corporation. All Rights Reserved.

For help type #help;;

> let zoe = "awesome";;
val zoe : string = "awesome"

> zoe = "not awesome";;
val it : bool = false

>

She was pretty happy that Zoe != “not awesome”.

I gave her her own FSI to try out, and I happily present to you, the very first code in what I’m sure is a very long lucrative functional programming career.

Microsoft (R) F# Interactive version 10.2.3 for F# 4.5
Copyright (c) Microsoft Corporation. All Rights Reserved.

For help type #help;;

> let lydia = "annoying";;
val lydia : string = "annoying"

>

I guess sisters will be sisters.

What do you mean it has to be done in a half hour?

Sometimes, you can see problems coming.

A vendor of ours provides security information monthly in a series of text files. Typically, those text files are in the 10s of megabytes large. Those files get viewed  cleaned up, and imported into our system by portfolio managers monthly.

The system to do the loading is homegrown, not ETL based (someone thought users should ‘upload’ the files) into an ASP.NET MVC app, and then render UIs with the data. :eyeroll:

Naturally, as our data requirements got larger the files got bigger. And bigger.

This last month, the system eventually exploded. A 1 GB text file proved to be too much for our cute little “not-quite-an-ETL” tool. The portfolio managers needed a solution, and in order to quickly get ’em one, I present (with relevant bits redacted) a 30 minute “lets play with F#” script to solve the problem.

Nothing too fancy. The function simply takes a file path, creates a couple of functions to help it along, and then loads up the file and splits it up into distinct new files. Whole thing took 30 minutes to do. Yes, the complexity is O(n-squared), but when you’ve got panicked users, and all of a half hour to hit it, getting it ‘working’ first is the best way to go.

Greensleeves

Yesterday afternoon, I had a bit of time on my hands, and a report that someone desperately needed redone. The original author was a support engineer who was trying to learn Ruby and didn’t realize that Ruby hash-maps are case sensitive. In the interest of time, I corrected the issue by downcasing all the keys, and reran the script (which ran successfully, but inelegantly), but I looked at the problem, and once again, found a perfectly lovely use case for the language I love so much, F#.

First off, some of the basics.

open System

// these may not be strictly necessary, but they do help describe
// precisely what data I'm dealing with
module UpperCaseString = 
    type UpperCaseString = UpperCaseString of string
    let private upper (a:string) =
        a.ToUpper()

    let create (a:string) = 
        if String.IsNullOrEmpty(a) then None
        else Some (UpperCaseString (upper a))
    let value (UpperCaseString s) =
        s
open UpperCaseString

module FloatingDecimalBetweenZeroAndOne = 
    type FloatingDecimalBetweenZeroAndOne = FloatingDecimalBetweenZeroAndOne of decimal

    let create (a:decimal) = 
        if a > Decimal.One || a < Decimal.Zero then              None          else              Some (FloatingDecimalBetweenZeroAndOne a)          let value (FloatingDecimalBetweenZeroAndOne f) =               f     let make a = (create a |> Option.get)    
   
open FloatingDecimalBetweenZeroAndOne 

Generally, I’ll do this for most business application coding nowadays. Creating a simple type that encapsulates correctness from the get-go (including basic validation) makes it so I don’t have to try to figure out weird results. I know the sorts of things I’m dealing with, and I deal with them from the beginning. Also, given the fundamental problem of the original script I had (the case sensitive hash-maps), ensuring I had a type that enforced upper-casing seemed worthwhile.

Next, the domain.

type Model = Model of UpperCaseString
    with static member Value (Model (UpperCaseString s)) = s
type Ticker = Ticker of UpperCaseString
    with static member Value (Ticker (UpperCaseString s)) = s

type Holding = { Ticker : Ticker; Allocation : FloatingDecimalBetweenZeroAndOne }
type ModelAllocations = { Model : Model; Holdings : Holding list }
type ModelWeight = { Model : Model; Weight : FloatingDecimalBetweenZeroAndOne }

type SleevedHolding = { Ticker : Ticker; Model : Model; Allocation : FloatingDecimalBetweenZeroAndOne } 
    with override x.ToString () = sprintf "%s,%s,%5M" (Ticker.Value x.Ticker) (Model.Value x.Model) (FloatingDecimalBetweenZeroAndOne.value x.Allocation) 


// This is the simplest description of what we're doing.
// taking a list of holdings, a list of models and their holdings
// a list of weights to apply to the model, and returning a set of 'sleeved' holdings.
type SleeveProcess = Holding list -> ModelAllocations list -> ModelWeight list -> SleevedHolding list

The problem the original script was attempting to solve was to assign an appropriate ‘Model’ to a holding. A Model is, for sake of brevity, simply an identity (representing the “name” of a standard financial benchmark (like the S&P 500), and a Ticker an identity representing a stock ticker.

A Holding represents an individual stock in a of a portfolio, with the Allocation representing the percentage of the portfolio. ModelAllocations represent a model, and the list of holdings it has (what stocks are in the S&P 500, etc.) ModelWeight represents the approximate weight a portfolio is associated to a given model or benchmark.

E.G. S&P 500 -> 50% , and Russell 3000 -> 50%.

let makeModel str =
    UpperCaseString.create str |> Option.get |> Model  

let makeTicker str =
    UpperCaseString.create str |> Option.get |> Ticker

let holding str all =
    { Ticker = makeTicker str; Allocation = make all }

let makeSleeve t m a = { Ticker = t; Model = m; Allocation = a }
let modelWeight str a = { Model = makeModel str; Weight = make a }
let modelAllocation str h = { Model = makeModel str; Holdings = h }

One of the problems of heavy use of custom types is that you’ll typically want ‘shorthand’ functions for creating the types as necessary. It can be a pain, but difficult to explain results after the fact is worse.

Finally, the meat of it.

let sleeveHoldings holdings models weights =
    let findHolding t = 
        models |> 
             List.collect (fun n -> n.Holdings 
                                        |> List.filter (fun n -> n.Ticker = t) 
                                        |> List.map (fun x -> (n.Model, x.Allocation)))
    let sleeve h = 
        let t = h.Ticker
        let m = findHolding t // a list of models that hold the security
        match m with 
        | [] -> 
            weights |> 
                List.map (fun w -> 
                    let weightTimesAllocation = (value w.Weight * value h.Allocation)
                    makeSleeve t w.Model (make weightTimesAllocation))
        | [(x,_)] -> [makeSleeve t x h.Allocation]
        | xs -> 
            let weightsMap = weights |> List.map (fun n -> (n.Model, value n.Weight)) |> Map.ofList
            let total = xs |> List.sumBy (fun (m, w) -> weightsMap.[m] * value (w))
            xs |> List.map (fun (m, w) -> 
                                let weightTimesAllocationOverTotal = (value w * weightsMap.[m]) / total
                                makeSleeve t m (make (weightTimesAllocationOverTotal * value h.Allocation)))
        

    holdings |> List.collect sleeve   

First, we define a function to find the models with a given holding, and return that model and the allocation. Then we define a function to sleeve an individual holding. Finally, we call that function over the list of holdings passed in.

A simple set of tests:

let testHoldings = [
        holding  "A" 0.20m;
        holding  "B" 0.20m;
        holding  "C" 0.20m;
        holding  "D" 0.20m; 
        holding  "E" 0.20m; ]
        
let testWeights =  [
        modelWeight "MODEL1" 0.5m; 
        modelWeight "MODEL2" 0.3m;
        modelWeight "MODEL3" 0.2m;
    ]

let testModelAllocations = [
    modelAllocation "MODEL1" [
            holding  "A" 1.0m;
    ];
    modelAllocation "MODEL2" [
            holding  "A" 0.5m;
            holding  "B" 0.3m;
            holding  "D" 0.2m;
    ];
    modelAllocation "MODEL3" [
            holding  "B" 0.5m;
            holding  "C" 0.3m;
            holding  "D" 0.2m;
    ]
]

sleeveHoldings testHoldings testModelAllocations testWeights |> List.map (fun m -> m.ToString ())

val it : string list =
  ["A,MODEL1,0.1538461538461538461538461538";
   "A,MODEL2,0.0461538461538461538461538462";
   "B,MODEL2,0.0947368421052631578947368421";
   "B,MODEL3,0.1052631578947368421052631579"; "C,MODEL3, 0.20";
   "D,MODEL2,0.120"; "D,MODEL3,0.080"; "E,MODEL1,0.100"; "E,MODEL2,0.060";
   "E,MODEL3,0.040"]

Explaining the image: At the financial services firm I work for, the concept of attributing holdings is called ‘sleeving.’ The photo is from an original 19th century (1872) painting by Dante Rossetti, of “Lady Greensleeves.”

A Fishful of Dollars

First of all, allow me to apologize for a solid month of failed updates. The world of coaching my son’s football team and bugbashing caused me to want to spend my time at home, asleep, rather than writing about my various misadventures in F#.

To update on the application rewrite, upcoming tax legislation has my firm in quite a tizzy, and is requiring an all-hands on deck approach for the next few weeks. Assuming the bill passes, we may be changing a lot of software, quite quickly, and absolutely none of that has anything to do with an old poorly-written report. The upcoming business need will be drastic, and preparing for that change is important to do.

That said, let’s play around a bit in F#!

I was watching an old episode of Futurama, when a question popped into my mind. If Fry left a bank account with $0.93 in it, at 2.25% interest, would he actually be sitting on a cool $4.3 billion after being frozen for 1000 years? Time to find out!

val fry : float = 4283508450.0

Looks like the math works… but man oh man does it take a while to get there. Using FSharp Charting and iterating over each year…

let fryOverTime = [ 1.0 .. 1000.0 ] 
                    |> List.map (fun t -> compound { Compounding = TimesPerYear 1.0
                                                     Rate = 0.0225
                                                     Principal = 0.93
                                                     TermInYears = t }  )

Chart.Line(fryOverTime
            , Name="Balance over Years"
            , XTitle = "Years"
            , YTitle = "Balance in Dollars")
       .WithXAxis(Min=0.0, Max=1000.0)
// because it's almost impossible to see any change until year 650 or so...
Chart.Line(fryOverTime
            , Name="Balance over Years"
            , XTitle = "Years"
            , YTitle = "Balance in Dollars")
       .WithXAxis(Min=0.0, Max=1000.0)
       .WithYAxis(Log = true)
Fry's Balance
Fry's BalanceIt looks like Fry’s bank account doesn’t get interesting until about 300 years or so. Still, compound interest is a wonder, despite whether or not Albert Einstein said so.

Coming Soon – Application Redesign in F#

We have a reporting application at {Redacted}, one I’ve spent more than a few hours maintaining. A colleague of mine and I (the junior developer I spoke of in a prior post) met up with the business owner of the reporting application, to talk about rebuilding the application in more functional way.

This application is not a terribly complicated one. In short, it takes data from multiple data sources, and re-aggregates it into a series of reports. It has however, had multiple developers come and “peek in”, drop code into it, and leave.  In the current C# implementation, it’s 5 distinct assemblies. There are 8 test assemblies to go with it.  It has a lot of business value riding on it being robust and correct, so the tests do seem warranted.

A word to the wise though… simple things that are more complicated than they should be AND that have a ton of tests are REFACTORING GOLD. Nothing feels better than taking something complex and unwinding it to it’s core essentials, and nothing feels better than doing it in F#!

We took a page out of For Fun and Profit’s DDD page and focused for nearly an hour on the objects and processes involved in this report. Naturally, the process seemed extremely solvable in a functional way.

Unfortunately, due to time constraints on the rest of the day we were only able to start with some very high level type definitions, but those type definitions described our problem in such a way that the business owner was able to see and understand.

We did the whole thing with an instance of VS Code and Ionide, and had types describing the objects and functions involved, all with just a simple “domain” setup in the process. Did we implement anything particularly? Well, kinda yeah, this is perfectly compile-able F# code, which as opposed to Gherkin or some other spec-based “code”, does not need a secondary interpreter.

That’s the crux of this application and function. I think we may be able to get this down to a few pages of code… as opposed to the novella you’d call it now. #FeelingHopeful

Fractions Cont’d, Factoring a Number

As we last spoke about Fractions, we had a simple task after getting a Fraction object. We wanted to reduce them. I was taught to rit educe a fraction by factorizing the numerator and denominator, and then crossing out the common factors. Then, multiplying the numbers together to get the reduced value. E.G.

 60        2 * 2 * 3 * 5        2 * 2 * 3 * 5       2 
----  =  -----------------  =  ---------------  =  ---
 90        2 * 3 * 3 * 5        2 * 3 * 3 * 5       3

The example above is a simple one.

So the question is, how do we best factor a number?

First, we define the mechanisms we require.  We need a way to define that a number is a factor of another.

let isFactor i pf =
    i % pf = 0I

What does this method buy us? First, it gives us a simple True/False flag defining whether or not a number is a factor of another. Namely, if the module of i and pf is 0I  (specifically, zero, as a System.Numerics.BigInteger), then we know that pf is a factor of i.

Next, we define a recursive function that takes two parameters, start and i. The idea of this function is to return a sequence of numbers from start to i, counting by twos. The start value is an option type, so if not present, it assumes 2 is a valid option, and starts the sequence there, then goes with every odd number. Note: I’m accepting as a given that this will do some checks against numbers it’s unnecessary to do that against.

Finally, I define my internal recursive loop to accumulate the resulting found factors. An interesting thing we’ve got here is some issues of partial application for function calls. The “findFunc” function is a partially applied isFactor call, with the i value already passed in. The reason I did this was to make the Seq.tryFind call easy to use, because the Seq.tryFind call is shaped like this:

('a -> bool) -> seq 'a -> Option 'a

But to get to my “bool”, I needed more than just a single ‘a parameter.  That’s why partial application was so valuable here! Instead of changing the function signature to meet my needs, I simply made a quick, easy to reference function by supplying some arguments ahead of time.

The rest of the function is fairly apparent. The loop recurses over the sequence, finding factors and then appending them to as the head of the accumulator (f::acc), until the Seq.tryFind call eventually returns None, and the accumulated list of factors is returned.

The last bits of the function just wrap up return values. In this case, we check for negative numbers, or a 0I being passed in.  Negative numbers are fairly easy to factor as they only require normal factorization of the inverse with -1I appended, and 0I simply returns an empty list.

With that, we get our results (mapped to ints, for ease of readability.)

getFactors 64000I |> List.map int;;
val it : int list = [5; 5; 5; 2; 2; 2; 2; 2; 2; 2; 2; 2]

getFactors 6423453000I |> List.map int;;
val it : int list = [8599; 83; 5; 5; 5; 3; 3; 2; 2; 2]

getFactors 9536923853063I |> List.map int64;;
val it : int64 list = [354041L; 178393L; 151L]

getFactors 953692385306332I |> List.map int64;;
// this one's still chugging.

The last thing to do here is to figure out how to avoid unnecessary iterations but that’s a topic for another blog.

Fun Friday (Monday version) – Fractions

This will be a slightly longer series of posts. I thought about treating fractions as a distinct problem, and wanted to find a way to code reducing fractions in a logical and meaningful way.

The first problem was describing the type. Naturally, you can do it a as simple pair:

type Fraction = int * int
let a = (-1, -2 : Fraction)  // negatives?

But that design describes some issues I’m not super keen on handling, namely negative denominators. Mathematically, negative denominators aren’t all that complicated to deal with, but in my head, it seemed unnecessary to solve them, so that led to the following:

type Fraction = int * uint32
let a = (-1, 2u : Fraction) // ugh, not as clean as I'd like

Now we’ve dealt with the negative denominator problem, but that leaves us describing fractions in a sort of “planar” way. Great for talking to a professor, but I’ve always been something of a “make it clear” personality. Record types work well for this.

type Fraction = { Numerator : int; Denominator : uint32 }
{ Numerator = 1 ; Denominator = 2u; }

Nice and pretty. Except 0 is a valid uint32 value. Shit.

type BiggerThanZero = private BiggerThanZero of uint32

module BiggerThanZero =

     let create uintValue =
          if (uintValue > 0u ) then BiggerThanZero uintValue
          else failwith "No zeros"

     let value (BiggerThanZero u) =
          u

type Fraction = { Numerator: int; Denominator: BiggerThanZero }

Alright. Something is breaking my “F# is less verbose” than other languages spidey-sense here, in C#.

using System;

public class Fraction
{
    public struct BiggerThanZero
    {
        public BiggerThanZero(UInt32 u)
        {
            if(u == 0u) then 
                throw new ArgumentException("No zeros");
            Value = u;
        }

        public UInt32 Value 
        {
            get;
        }
    }

    public Fraction(int numerator, BiggerThanZero denominator)
    {
        Numerator = numerator;
        Denominator = denominator;
    }
    public int Numerator { get; }
    public BiggerThanZero Denominator { get; }
}

Nope, spidey-sense was off. C# is still more code. Phew… thought I was going to have to go back to OO land. 🙂

OK… so now we have a domain object. We cannot represent an object in the domain that is “invalid” in any way, so fundamentally, our functions should be easy to reason about.

So first thing… decimals to my new “Fraction” type.

let getFraction decimalValue = 
    let rec fract dm =
        let a = (d * m)
        let r = a % 1.0m
        match r with
            | 0.0m -> { Numerator = (int a) ; Denominator = BiggerThanZero.create (uint32 m)}
            | _ -> fract d (m * 10.0m)
    fract decimalValue 10.0m

This function goes back to our old friend, the recursive inner function. We take our input decimal value, and multiplying it by 10, and calling the result ‘a’ (shorthand for ‘amount’). Then we take ‘a’, and get the decimal part of that value by applying the modulo function to it and 1.0, and naming that value ‘r’ (shorthand for remainder). Assuming that ‘r’ is non-zero, we recurse into the loop, updating the multiplier to another factor of 10 greater than what we had before. Otherwise, we simply return a Fraction object, with the Numerator set to ‘a’, and the Denominator set to the multiple. E.G.

 

getFraction 0.4m;;
val it : Fraction = { Numerator = 4; Denominator = BiggerThanZero 10u }

getFraction 0.542m;;
val it : Fraction = { Numerator = 542; Denominator = BiggerThanZero 1000u }
 
getFraction 0.8675421m;;
val it : Fraction = { Numerator = 8675421; Denominator = BiggerThanZero 10000000u }

This is the start of our Fractions work, and although it’s correct, it’s certainly got some potential error cases. Our int and uint32 bases for values could be overflowed. That’s fixed in the following.

Next time, we’ll deal with reducing our fractions.

 

Modeling the Domain : Short Codes

Anyone who’s read For Fun and Profit’s domain modeling exercise may see some similarities in today’s code. Mainly, I wanted to point out the value of quickly being able to create types that describe my functions here.  For context, I’m in the middle of rewriting a sticky report that consumes data from a ton of different places… this is an example (slightly modified, to avoid spilling too many of {Redacted}’s beans.)

In the above, we’re dealing with a thing called ‘Short Codes.’ At redacted, we have a shortened string which represents many of our more common domain objects, called a ‘Short Code’ which makes domain objects easily identifiable when viewed in spreadsheets.

In C# code, we’ll typically treat these objects as simple strings, or you deal with of domain types as espoused in Vladir Khorikov’s Pluralsight course, “Applying Functional Principles in C#”.

But in F#, you get drastically simpler code, that gives you similar benefits.

  1. Any change to ShortCodes can be done once, and all references using it get the change. That’s as DRY as it comes.
  2. ShortCodes can be equal to each other, but AccountShortCodes cannot be equal to ProductShortCodes cannot be (try it in FSI, you get compiler errors!)
  3. The modules allow us to retain the business logic, so we avoid the annoying issues of duplicating validation code everywhere. If I reference an AccountShortCode, it’s implied that I created one successfully in the first place!

The code to do stuff becomes quite easy:

type ConsumingRecord = { ShortCode : AccountShortCode
                         ImportantValue : decimal }

let m = AccountShortCode.create "APPLE";;
// m is an AccountShortCode option, because of the 
// validation logic there, so we need it from the option, 
// before we push it into our consuming type.

let n = { ShortCode = (Option.get m)
          ImportantValue = 5.0m}

The subtle thing here is that we have to actually deal with the fact that it’s an option. We CAN fail to get an AccountShortCode here, depending on what we pass in, but once we have a “ConsumingRecord” object, the validity of the ShortCode is guaranteed. This only works, however, because F# doesn’t do nulls. Once your language does nulls, it throws this stuff right out the window.

But I’m sure eventually C# will get that, too. You’ll just have to use an attribute to make your class less C-sharpy. 😉

Not Much F# Today

You ever have one of those days when everything goes wrong and it’s all a C# developer’s fault? I kid, of course, but it made for a difficult day with almost no sitting at my own desk to actually get anything done.

My mission is to update this blog every day, regardless of what I actually accomplish for the day. Today, I saved the world, and solved problems that were a little more tactical in nature. I did enjoy a welcome to the firm lunch with a new QA in the office, but beyond that, very little coding.