The Power Of Information

Evaluating Change

2011-11-29T23:33:00.001-05:00

When you are considering changing the way you operate your business apply the same critical assessment of the change you are considering to the status quo and you should quickly conclude that change is the correct course.

Incomplete improvements to the current process are always better than no improvement at all. Achieving all possible improvements is perfection and this is not possible.

Overcome the fear of change by lowering your expectation of perfection. I believe expectation of perfection is a natural human psychological barrier we put up to avoid change and stepping into what we perceive to be the unknown.

I experience this resistance from most of the clients and companies I work with when they are considering adding analytics to core operational processes. It is quite simple and true that better information leads to better decisions ... period.

Perfection does not exist so move on and step into change ... you can experience incredible business results in as little as a few weeks. Your willingness to change is the only barrier.

Money, Money, Money

2010-03-09T20:32:00.002-05:00

Is $1 worth $1 dollar in all situations... is money linear?

I believe that money is highly non-linear and our 1000's of years old concept of money needs an overhaul.

Some questions ....

If I am a millionaire how do I value an incremental dollar offered to me versus a homeless person?

If I am a millionaire and I need to get out of a parking garage and my bill is $10 but I only have $9 in cash, what is $1 worth?

If I want to spend $100 million on televisions will the cost per TV be the same as if I spent $3,000 on TV's?

If I love books and hate music would spending $1000 on books be the same as spending $1000 on CDs to paying a $1000 tax bill?

Is earning $1000 at work equivalent to winning a $1000 lottery?

Are 100,000 loyalty program points which I can buy a $1000 TV with worth $1000 in cash? Are 100,000 loyalty program points which I can buy a $1000 plane ticket with worth a different amount of cash than the 100,000 points I can buy the TV with?

Is a $10 million offer to buy TVs from a $10 billion dollar retailer worth the same as a $10 million offer to buy the same TVs from the $100 million dollar manufacturer of the TVs?

Why do rich celebrities get a $40,000 gift bag at the academy awards for free? Are those gifts actually worth $40,000 to the giver(s) or the receiver?

Why is there an adage which says "It is much hard to make your first million than your second"?

Why is it that the more money you have the more you get for free?

Is paying $100,000 for products/services to a large corporation with cultural beliefs about salary, compensation, worth the same as paying a different consulting firm with different beliefs $100,000 for the same products/services?

The value of money is a function many things... incremental utility of the buyer, incremental utility of the seller, the marketplace, time, the situational utility, quality, belief systems, what the economic value of your currency is plus many other variables...

I don't believe that supply, demand, price or traditional macro/micro economic concepts can accurately account for the situations described above.

Big questions ... how much inefficiency exists in our economic, political and social systems because we have such a simple concept of money? Can we create new value out of this inefficiency that would increase GDP, reduce unemployment etc.? Can we change macroeconomic properties by optimizing the micro-utility of trillions of transactions.

Can we create a new concept of money that allows us to maximize our utility better than we currently do? or can we broaden the definition of a transaction that recognizes more than just our current system of money that will impact our utility and on aggregate will impact macroeconomic properties like GDP.

Yes, Yes, Yes ... we can use information technology and analytics and the willingness to change to do so.

Better use of information can create value out of seemingly nothing. It is there for the taking by any individual, corporation or organization.

Change is Everything

2010-02-18T01:25:00.003-05:00

Technology implementations, statistical analysis, discovering actionable insights, changing business processes, developing information use cases, measuring financial results, modeling business performance... All easy things to accomplish. I have been involved in dozens of projects where we have delivered some or all of these things. Clients have been on the precipice of dramatic business gains... yet many analytic projects do not deliver as promised.

Getting organizations and people to change is very difficult and is the real challenge when delivering strategic analytic solutions. By strategic analytic solutions, I mean solutions which could double a company's bottom-line performance.

Overlooking change management issues will doom projects to the dustbin.

The types of analytic solutions in the marketplace, fraud detection for insurance and banking, pricing optimization and assortment planning for retail, forecasting for retail and manufacturing, retail customer loyalty require organizations to be able to create, assess and act upon the output of sophisticated analytics and reporting and to be able to measure the impact of your actions, learn from those actions and adjust your operating plan.

There are two types of change required: surgical change to business processes that are impacted by information and strategic change which requires an organization to be change its management style and mindset to become analytically driven and willing to experiment.

Surgical change while narrow in scope can be painful. Employees who execute business processes may not have the skills to assess, review and take action based on analytic insight and business intelligence reporting. Employees who were otherwise well regarded and successful in their positions may find themselves floundering to adapt to the changes. With information based business execution comes transparency and accountability, further compounding employees willingness to accept change.

Changing the management mindset in an organization is even more difficult. When decision making has been largely based on intuition, or "how we've always done it" and the impact of decisions has not been measured, it becomes a monumental task to get employees, middle managers and executives to change the way they make decisions and learn how to make better decisions. I don't know many companies who track their decision performance!

Recognizing these issues before embarking on strategic analytic initiatives will improve the likelihood for success. Setting appropriate expectations for time to results should be set. Implementing a change management program and plan is mandatory. Crawl, walk and run is highly recommended. Expect the change management issues to long outlive the technology implementations.

Seek outside help as change from within is very difficult. Select a change management method that accomodates both surgical change and organization paradigm shift.

Analytic technology implementations take 2 to 4 months. If you can overcome the change issues, I think every organization is 12 to 18 months away from doubling and more their bottom-line profits. Start now...

Comments welcome

Living in the Moment

2009-10-20T08:13:00.003-04:00

Short blog this morning .... something a little different

I have had a life long love affair with the game of tennis, having played competitively and recreationally. After many years of practice I have finally begun to understand the game. Over the last 5 years my game has improved significantly, and even out of shape as I am now, I am playing better than I ever have at my competitive peak. The key learning is to live in the moment, and forget about outcomes. When I was younger, I focused so much on the score, worrying about what people would think if I won or lost. What I learned was that you cannot achieve an outcome if you don't love executing the moments and actions you need to get there.

I have learned to not worry about scores and outcomes. I focus on executing each shot, enjoying a well placed forehand, a good volley, a backhand slice. I can now reach a level of calmness, not having external thoughts and worries, when I play that allows me to perform to my potential which then achieves the outcomes I have so desired.

The learning also applies to the work world. Focusing on your day to day activities, learning to love those and doing the best you can in the moment, is the sure fire way to success. I have been working hard to bring lessons learned from recreation to my day to day business life. I'll keep you posted.

Appliance as a Business Solution

2009-10-14T23:35:00.002-04:00

We've been working on developing applianced based business solutions.... (keyword business)

what is an appliance according to me (the first list is technology focused, if you're a business person read-on beyond this list...)

Hardware
Database Software
ETL Software
Reporting Software
Advanced Statistical Analysis Software
Systems Integration Assets (Data Models, ETL Code, Standard Reports, Automated Statistical Modeling)
Business Process Workflow
Financial Model
Remote Operational Management

What does the business appliance do ... according to me

It improves a specific business process (i.e. increase revenue or reduces cost) by using advanced analytics.
the output of the appliance is integrated into a business process with defined user workflows that generate operational business activity
The output of the appliance also generates a P&L and Balance Sheet actuals & forecast which resulted or will result from the operational business activity executed by using the appliance

What type of business processes are amenable to an appliance model ... according to me again

Processes where a small set of decision metrics are obvious to act upon
Processes that generate operational data

Some Examples

fraud detection... output metric is a scored claim where the score value indicates whether or not the claim should be investigated. The result of claims investigation is less claims paid out. 10 to 15% of claims are typically fraudulent, eliminating 50% of fraud will more than double bottom-line results
Price optimization for retail ... output is a recommended price on a product ... when executed price optimization can deliver up to 5% sales and margin increase which for a typical retailer will double bottom line profits...
Forecasting for retail and manufacturing ... output is recommended inventory levels ... when executed for rertail will reduce stockouts and markdowns ... when executed for manufacturing will increase inventory turns or capital tied up in inventory... both can double bottom-line results.
Customer Loyalty ... output is recommended marketing campaigns to specific customer sets ... when executed will increase frequency, transaction revenue, transaction gross margin, and customer lifetime which can add 5% to top line or gross margin which again would double a typical retailers bottom line results...
A huge list of other business opportunities which achieve dramatic results

from a technology perspective what distinguishes a business appliance from software which could deliver the above 4 examples ... according to me

The appliance is pre-built and avoids a 12-18 months systems integration project. Most large integration projects fail
The appliance has business workflow which integrates output of the appliance into existing operational business process which ensures that output is acted upon.
The appliance has financial model output which integrates into P&L and Balance Sheet tracking and forecasting to ensure the top-line and bottom-line results are achieved.
The appliance is a black-box to customers ... it has a data interface to receive data and output windows (workflow, reporting and financial output) requiring no administration except a power and network cable.
the appliance's day to day function is remotely administered to ensure data is loaded properly, data is backed up, statistic models are tuned, financial results are achieved...
The appliance can be delivered as a hosted or SaaS model...
The appliance can be virtualized...
The appliance can be deployed in 20-30 days from order
The client will achieve business results in 21-31 days from order

Why would a company consider an appliance .... according to me

Avoids risky system integration projects
Delivers results quickly
Is business focused with clear integration point to business process and shareholder/public financial reporting
The appliance deliverable is a business result ... not a technology system ...the appliance is a means to an end.
Corporate Information Technology departments are not good at delivering strategic results... The appliance requires minimal IT involvement
Corporate IT departments can learn to "appliance-ize" solutions
Technology vendors (hardware and software) do not deliver on the sales proposition... the appliance does.

My company plans to roll-out many appliance solutions in the coming months ....

What do you think? Sound interesting? Feedback welcome ....

Five thoughts ... one blog

2009-07-06T09:25:00.004-04:00

Long time no blog ... been very busy, which is great!

Just wanted to put down some short statements that I always use with clients, colleagues, friends related to the power of information.

1) You can't sell pink basketballs... The fundamental driver of business growth is the quality of your company's value proposition and how authentically (see http://www.ted.com/index.php/talks/joseph_pine_on_what_consumers_want.html) you execute it. You can build great management methods, execute your operational processes to perfection, use advanced analytics in all parts of your business and do it all for nothing if your customers don't like your value proposition. Great companies deliver great value to their customers. Facing the facts on this one is a hard thing to do.

2) If you want to increase revenue/profit ... start right now... in the words if Nike Corporation "Just Do It". Integrate advanced analytics and your operational processes... you could be a matter of weeks away from top-line/bottom-line growth. Change is scary but the results are worth it. You have a 100% chance of success applying analytic insight. Read the books "Freakonomics, Supercrunchers, Competing on Analytics, Smart Enough systems,....".

3) Companies don't have relationships with their customers. I don't know anyone at the bank, telco, insurance company, retailers I do business with. We don't trade family pictures, vacation... Getting hung up on the terminology of relationship I believe is dangerous. Again, customers seek value propositions. Offering an authentic value proposition is the secret sauce, understanding what customers value and informing them of the value propositions you offer is key. Doing this in a profitable manner is critical. Customer insight ... yes! Creating a great value proposition(s)... yes! Executing brilliantly ... yes! Marketing, promoting and operationally executing your value propositions ... yes! Track the results of your operational processes ... yes! Analyzing your processes to create insight and execute better ... yes! If all these are the definition of a "relationship" then yes! .... but not the dictionary definition of relationship.

4) Stop seeking the silver bullet. Execute your business model. It's boring... you have to work harder... if you're a bank, be a bank (not some high risk investment house), I find it hard to imagine that you could ever lose money being a bank or insurance company. I think executives seek the silver bullet and end up straying from the core business and lose the ability to execute the core business model well. The hamster on the treadmill wins. Always has, always will ...

5) Test and learn ... conduct experiments every day ... measure the results of those experiments and improve your operational processes. The current business paradigm makes operational changes once every decade and rides the process until it fails. Continuous tweaking is the right way... approve 1000 loans randomly, underwrite 1000 policies randomly and see what happens. Build a new loan approval or underwriting model from the results and see if it significantly different than your current model.

That's it for now. Would love your comments...

A rant

2009-03-24T20:51:00.003-04:00

Something completely different...

Does anyone else think it is a huge conflict of interest to allow police to use the proceeds of applying the traffic act, that is they get to keep the monies collected from handing out traffic tickets? Isn't this the violation of some basic legal prinicple?

Police are incented to hand out more tickets so they can stay employed and hire more policemen to hand out tickets. It is human nature to want to stay employed and delude yourself into thinking you are serving the greater good when you're not necessarily doing so ...

For example, in my neighbourhood their is a school where the speed limit is 50km/h. The police hand out tickets like crazy all weekend long. They are not there when the kids are in school or just before or after school and at lunch. How is this a benefit to society?

Police should hand out traffice tickets. The money should be collected by the government and put into the big pot. The government should propose levels of policing and funding for police as their platform based on the consensus of policy makers and citizens and we should elect the government which represents our beliefs...

Police using the proceeds of the Traffic Act is undemocratic, a conflict of interest and does not necessarily lead to safer roads (if that is the objective of traffic laws).

I'll probably get ticketed like crazy from here on in ....

Multi-Dimensional Currency

2009-03-23T20:55:00.006-04:00

Another idea that I have been thinking about for a long time... this current recession motivates this blog.

Currency is currently one dimensional, that is, it has one value and all currencies are just a variation of the same thing apart by only a factor. My idea, which I haven't completely worked out, is that currency should be multi-dimensional, that is, it has different value dimension depending on the usage domain.

In Canada, we might consider healthcare a form of alternate currency for citizens. Healthcare is universally free for the most part, we contribute via employee benefits to additional health coverages like dentists, drug and eyecare etc. The value of the services received for some individuals who use the system is much more than what it would cost them individually and for some it is less. We could have a bank account that we tap into which has monies in it placed by the government which is the equal balance each citizen receives annually for our tax contributions. This account might have atleast $160 billion (total cost of health care)/30 million (approx population of Canada) or just over $5,000. We could see the money come and go as we use the healthcare system but we cannot use it for anything else. Availability of healthcare is independent of whether there is a recession or not.

Another example of alternate currencies are loyalty rewards that we accumulate. Again they have value but are only usable with certain rules attached to their usage.

My first thought is to attach all your loyalty points from all the programs you participate in, your healthcare dollars and the actual cash you have to one account or institution, maybe the bank of your choice or a central bank. If this happened you could see the multiple dimensions of the currency that you have and you could have one card that you could use to access this value in the appropriate situations.

Another thought. Like healthcare can we decouple certain rights from the fluctuations of the economy. Shouldn't we all have the right to food, shelther, clothing in addition to healthcare if we are productive members of society. I don't mean this in a communist sense, that everyone has exactly the same but we all get a minimum based on the fact that we go to work everyday or contribute in some other important way like raising children etc. Some of has who have the fortune to earn more can use the additional earnings to purchase additional food, shelther, clothing, leisure etc. with the additional income earned. I don't advocate innefficent government running with these additional programs but I don't have an alternative at this moment. These additional accounts would have to be funded from the productivity of all participants in the economy.

We could add these currency dimensions to our one currency account, which now contains, healthcare dollars, loyalty points, food account, shelter account, clothing account, your cash and credit. Only your cash and credit are exposed to market fluctuations.

We could also add tax credits or government loyalty points as another dimension to our one currency account (see my previous post on zero sum currencies). Now we would have all these different dimensions of value available in one place and maybe the actual real money value of all these dimensions accessible in that one place also accessible by one smart card in our wallet.

What would happen if we allowed individuals to move value from one dimension to another, if their behaviour led to excess in some dimensions and shortfalls in another. The rules for moving from one dimension to another to be defined maybe by government programs and their incentives to promote behaviour that leads to government budget efficiencies or by loyalty programs or corporations. This would be like turning some portion of government and corporate spending to individuals and allowing individuals to arbitrage government programs, loyalty points, cash etc. in unique and unimaginable ways that would drive orders of efficiency not otherwise predictable at an aggregate level. The government portion is only be possible first by aggregating the value of all of the productivity of the economies participants and then distributing it back to the individuals with programs and incentives for behaviour which allow all the value to be utilized. Loyalty programs do essentially the same thing, aggregate consumer spending and use the aggregate buying power to create rewards not individually achievable and then distributing the buying power back to individuals. The loyalty program seems to create incredible value out of nothing. (It creates value out of the inefficiency of retail). We can similarly get much more value out of government programs because guess what ... they're inefficient too.

The benefits,

1) Basic human needs decoupled from business fluctuations. Recessions hurt dumb businesses not the individuals who show up to work everyday and have very little say in what businesses do.

2) Opportunities to create significant economic efficiencies by creating one multi-dimensional currency accessible (or at least viewable) in one place attached to one smart card which allow individuals to move value across dimensions and allow individuals to seek the personalized efficiencies. Let's call this economic micro optimization ...

3) a minor benefit would be to eliminate all the cards (loyalty, credit, debit, ...) and replace them with one.

I haven't worked out all the details but hopefully presented enough of an idea or ideas that if some smart individual or economist hopped on this, they could either rip it apart or run with it.

I'll continue to think about it and work out the details to the best of my ability ... and maybe launch another information based business.

The power of information....

Zero Sum Currencies

2009-03-20T12:48:00.003-04:00

An idea I have thought about for a long time...

First, what is a zero sum currency. One example I know of, but there many, is the futures market. The total return on commodities futures is zero but the market performs an important function, it lowers the risk for individuals or business who make a living digging commodities out of the ground or raising livestock or growing food. For example, a farmer needs to achieve a minimum sale price for his/her goods to make a profit. They would sell a futures contract for an amount of their goods (quality, lot size, delivery, etc.) all specified by the standardized futures contracts sold on the various markets that would guarantee them a price on a specified date.

Buyers of the contract would either be a business who uses the farmer's goods to make their product and would like to guarantee a specific purchase price that fits their business model or a speculator who makes money by predicting a price that will be higher than the farmers guarantee and can sell the contract for a profit before taking delivery. The commodity buyers and sellers use the instrument to reduce the impact of the commodity price on their business and view it as a cost of doing business, it is like insurance. Total wins and losses across all contracts sum to zero but importantly the risk for businesses depending on the commodity price is reduced.

Other examples are the Kyoto protocol or Carbon taxes which are types of zero sum currencies where the polluters pay the non-polluters with the overall objective of reducing greenhouse gases or carbon emissions.

The government should do this for its citizens for all the programs it plans to execute. For example, if the government's healthcare program is designed to improve healthcare quality and reduce costs it should publish metrics about the current state and what it wants to accomplish. Maybe the metrics would be number of visits, cost per visit and number of deaths by age band. The objective being to reduce number of visits, reduce cost per visit and reduce number of deaths by age band. (I am sure you could come up with better metrics but for arguments sake lets consider these)

The government knows how many visits to the healthcare system there are because here in Canada we show our healthcard everytime we get medical attention, and they know how much each visit costs because the healthcare provider bills the government for that visit. The government would also know deaths and ages of the deceased.

Now the goverment healthcare experts come up with their brilliant ideas to incent citizens to behave in a way to achieve the metrics. For example, the government might pay for gym memberships because being healthy is proactive and you will go to doctor less. How will they pay? They should create a zero sum currency, in the form of tax credits. Everytime you go to the gym you present your healthcard which the government can track. Perhaps the government also thinks that everyone should see the doctor once per year for a physical, something they can already track, because that would allow early detection or doctor recommended preventative activities to reduce your dependence on healthcare. The overall result being improved health and longer lifespans over time.

The citizens who believe in the governments plan will take advantage of the opportunity and earn tax credits, the citizens who don't believe or care will be taxed more to offset the cost of the tax credits. Lets say everyone climbs on the bandwagon and the tax increases don't offset the tax credits. In this case, the monies will then be taken from the existing healthcare budgets we are already taxed for reducing the overall cost of healthcare and putting money back in our pockets. If no one participates , i believe a less likely scenario, then we will all pay more tax which the government can use to put towards the current healthcare model. If the theory of gym visits, GP visits works we should see the death rate and cost of healthcare go down. The tax credit is a zero sum currency (tax credits offset by tax increases and reduction in healthcare funding equals zero) and healthcare gets cheaper and better. Obviously an overly simplified example but you get the idea.

Side benefits are transparency, we all know what the governments programs are and how they translate into results and what is expected of us. Transparency brings accountability and allows us to hold our politicians accountable (see previous posts on transparency). You can't improve what you don't measure (see previous posts on this topic)

This is kind of government loyalty program. Instead of using hard dollars you could use a different currency or points which the government translates into dollars at the end of the tax year based on how much tax revenue they have collected. Repeat this concept for all government programs.

Business should also follow. Some businesses already have loyalty programs but haven't set them up as zero sum currencies. Most companies cannot determine what loyalty is doing for them partly because there are no concrete objectives or they have difficulty measuring the results or they don't have clearly marketed customer incentives except to buy more spend more. Setting up a zero sum currency for loyalty is critical. I think the next generation of loyalty programs should align social responsiblity, corporate values and consumer behaviour to create more interesting and authentic experiences for consumers.

Check this ted.com talk out http://www.ted.com/index.php/talks/joseph_pine_on_what_consumers_want.html

I'll post more on loyalty programs later ...

My bottom line, zero sum currencies everywhere set up to improve the world in measurable, transparent and accountable ways. Analytics used to discover the levers that create the incentives and penalties.

Disruptive Innovation versus Continuous Improvement

2009-03-18T15:59:00.007-04:00

I was at a seminar last week and one of the speakers shared a few great quotes which stimulated some thought and led to this. The two quotes were "Edison’s electric light did not come about from the continuous improvement of the candle…"-Oren Harari and "If I had asked people what they wanted, they’d have told me ‘a faster horse’!"-Henry Ford.

These quotes led me to consider what today's business leader's are looking for. Are they seeking the next great innovation that will change their business or are they trying to improve their existing core operational processes and get more efficient. I think the answer is both but I believe that continuous improvement is a lower strategic priority. I'll tell you what leads me to this belief and what its implications are for analytics professionals.

I have been working as a data mining/management consultant for 20 years helping large corporations get efficient by improving marketing efficiency, improve location selection, make less risky loans, eliminate fraud, improve manufacturing throughput - in a nutshell, trying to make analytics a strategic priority by marrying analytics and core operational process. Everytime a client has acted on the analytic recommendations (perhaps 1 out of 10 times) the results have been staggering in terms of return on investment. The other 9 out of 10 times, a great business case, clear findings and definitive recommendations have led to inaction. I have struggled to understand why many clients are not willing to act. In previous posts, I talked about transparency and willingness to act and these are some of the reasons. My current realization is that companies seek disruptive innovations because

1) A new innovation comes in an form easily recognizable to the experiences of a business executive, it's a new business model (internet channels) , a new product ( a light bulb, a car ), a new value proposition (combo meals at quick service) ... you get the idea

2) Innovations are exciting and understood by an organization's customers.

3) Continuous improvement requires analytic capabilities (not in abundance), sophisticated processes (measurement, tracking, forecasting), is not exciting (most retailers would rather have new iPod that sells like crazy than predictive forecasting models to improve merhcandise buying), not easy to understand, and invisible (Aside definition of invisible: bad direct marketing starts with a list of names randomly chosen, after building a data warehouse, hiring statisticians, building mathematical models, spending significant time and effort improving the process you end up with a list of names. The new and improved list looks no different than the original list. To prove it is better you have to set up and execute design of experiments and track business execution and generate sophisticated reporting to show on paper which list was better. Not obvious to say the least. The output of most analytic based continuous improvement activities leads to output which on the surface looks no different than what you started with).

4) In my experience people look for a silver bullet (disruptive innovation). Continuous improvement is not easy ... it is continuing to execute and be the "hamster on a treadmill"

My conclusion here is that analytics will always be a second class citizen to disruptive innovation unless we educate a new generation of business leaders and create new business management paradigms based on analytics. The implication for analytics professionals is that we will not get the strategic executive support required for significant corporate initiatives, we will continue to have to seek those few management individuals who understand the power of information and that the business intelligence/analytics industry will not achieve that status and promise that it should.

My belief is that continuous improvement through analytics will generate double digit bottom line gains (see my original Power of Information post) and that year over year continuous improvement will generate more return than disruptive innovation which happens on the order of a decade in most industries.

Companies who have focus on continuous improvement and use analytics to understand every operational facet of their business are those companies that will not only survive this recession but grow and gain the marketshare lost by their peers who have been looking for that silver bullet and hence have less insight into operations and less ability to make critical decisions about survival.

My recommendation is that companies should focus more on continuous improvement while keeping an eye out for disruptive innovation. If your organization is not analyzing your operations, start now. The business return from analytics can happen in weeks. It could be the difference between survival or failure.

The Recession

2009-03-06T14:26:00.001-05:00

I just experienced a very enlightening lunch event that I was invited to. Refreshing first because I am usually the one taking clients and prospects to lunch, sporting events and the like but this time I was the prospect and was invited to hear the Honorable Minister of Finance, Jim Flaherty address the Toronto Board of Trade... a pleasant change.

I really had no expectations for the speech I was about to hear but was keenly interested in what one of our leaders at the centre of Canada's efforts to minimize the recession had to say.

I learned several things which I thought I would share. First some pertinent facts, according to Mr. Flaherty

1) Canada entered the recession with no deficit, contrary to many other countries in the G20
2) Canada's current national debt to GDP ratio is 29.1%, contrary to the United States in the mid 40's and Japan much higher (somewhere near or above 100% according to wikipedia)
3) Mr. Flaherty by running deficits for the next 3 to 4 years will increase the debt to GDP ratio to 32.2% and then immediately bring it back down in the years coming out of the recession
4) Canada is running a higher deficit than most G20 countries who all agreed to run a minimum of 2.0% to stabilize global economies. Canada has decided to run a 3.5% deficit. The United States will run a much higher deficit due to the dramatically worse situation there.
5) The planned deficits are not long term programs but short term funds that have a "use it or lose it clause" which will ensure that we do not end of up with long term debt from entrenched or structured programs.

All facts which I find very heartening and postive.

The Minister also shared some personal anecdotes and did some "cool" name dropping. Saying how in his conversations with fellow finance ministers, the US Secretary of the Treasury, the UK's Chancellor of the Exchequer that all of these folks expressed praise for how Canada's financial system and government were well run, stable and able to deal with the current situation. Minister Flaherty is also co-chairing a committee to deliver a report to the leaders of the G8 countries in early April. Again very positive to know that Canada is playing a significant role in sharing our views and progress with world leaders about this current situation.

Finally on a personal note he said [I'll quote him as best as I can] "I plan to take the Island Ferry to the airport to catch a flight on Porter air later. It reminds me of many of our parents and grandparents who came to this country by ship from Europe and other parts of the world. Their experiences and challenges in building a life in this country were much more difficult and trying than what we are currently experiencing. We are Canadians and we will persevere and come out of this recession strong and sooner than most" or something to that effect. I certainly echo that perspective. Although the current times are difficult most of us are still very fortunate to lead a great life here in Canada.

I found Mr. Flaherty to be very passionate and sincere about his desire to help Canadians and he conveyed a sense of his diligince and urgency to work with our local, provincial and federal leaders as well as his international counterparts to "cushion" Canadian families and businesses from the effects of the recession. This gave me great confidence.

I continue as ever to work hard to build my business and provide employment opportunities to our staff and contribute in a small way to our economy and country.

Data Preparation

2009-02-17T12:15:00.000-05:00

My view is that feature extraction or variable creation or data preparation is the largest single activity which determines the quality of data mining results. I believe that feature extraction has an order of magnitude larger impact on the final result of a data mining exercise than does the algorithm utilized.

I have several motivations for data preparation

1) Current data mining tools cannot directly ingest the normalized data found in OLTP and data warehouse systems. They require the data to be "flattened" or "de-normalized", creating attributes related to the entities of interest (i.e. customers, locations, transactions, suppliers etc.) Aside - Algorithms which can mine the data in normalized format would be fabulous. Something I've thought about for years but haven't figured out yet.

2) The creation of variables creates context or relates an atomic transaction to the complete set of information about a particular entity. I.e. the fact that I had a credit card transaction at 2:47pm on February 16, 2009 for $10.11 at Starbucks is of little relevance or use in this form. relating this transaction to my entire transaction behaviour is the critical leap i.e. This transaction represents 4% of my week's spend, is 10% more than I spent at starbuck's last week, is 25% of my quarterly coffee purchases etc.

3) The selection of which features or variables to be created is guided my engineering background. Let us develop features which describe all the properties of a system. In engineering systems like air flow around and aircraft we measure temperature, pressure, density, 3 dimensional velocity, viscosity, etc. In a retail system, for example, if we are modeling customer behaviour we should ask what are the properties which describe customer shopping behaviour over a period of time. Recency, frequency, montary value, product mix, channel mix, consistency, trends, counters of/from major events, time series metrics etc. are many of the feature categories we can derive from a customers transaction series. We can create millions of features from a customers transaction series using this approach. ie. days since recent transaction, number of transactions by month, montary value of transactions by month. If we calculate this for the last 3 years by month, for 20 product categories we have 3x12X20x3 variables. If we further calculate month over month changes, year over year changeswe can further double the variables. Depending on the granularuity of time and product you can easily see how we can derive millions of variables. The largest number of variables I created for a data mining project was on the order of 25,000. I typically create ~1,000 attributes.

The exercise in step 3 results in a flattened data set where we have thousands of variables for each entity that we are modelling. This extensive set of variables provides a very rich description of an entities behaviour over a period of time. It provides the context about an entity which allow us to make intelligent conclusions, predictions or assumptions about that entity which are useful to improved decision making.

I apply the same variable creation method to all problems in all industries. The categories recency, frequency, monetary value, product mix, channel mix, trends, stochastics, counters etc. all have analogies in every problem I have countered. I prefer to describe a system and let the algorithms and feature selection approaches determine which features are important to the problem at hand, as opposed to hypothesizing a small set of features based on my knowledge.

In my experience, I can generate better data mining results through data preparation or feature extraction by building a rich description of a system's entities over time relating atomic transactions or events to the entity behaviour as a whole than by focusing on improving an algorithm's performance.

Comments welcome.

Transparency, Change and Equality

2008-05-19T17:17:00.000-04:00

I spent the last month or so meticulously planning my next bajillion blogs in a very structured manner and along the way realized that blogging should be a little more spontaneous otherwise I should just write a book. So I will be a little more random in the order of topics that I write about but they will still relate to my original premise. The power of information.

Many years ago a large fraction of people were paid or evaluated by piece work. You may have spent the day picking apples in an orchard and at the end of the day you received your pay based on the number of apples you picked. The harder you worked, the more you achieved. Our corporate world lost the ability to see into the labour of the individuals executing its business processes. In recent years the introducion of technology and workflow systems creates visbility.

If your objective is to improve your business processes then you must be able to measure the process at a granular level. You cannot improve what you cannot measure. If people are involved in the execution of a process then you need to be able to measure their productivity. Creating information systems that promote the execution of a particular workflow, track it at all levels and measure the result are key. Many such workflow systems exist, ERP systems, Sales Forces automation, etc. Management should know which sales person is making more cold calls than the next, they should know which programmer generates less lines of code than the average ... you get the idea. This is transparency, the ability to see a business process at work at all levels.

Whenever I have worked with clients to try to improve their business processes by providing intelligence based on careful analysis of the data created by the processes I have run into significant change management issues. We know that change is not easy, there is a whole branch 0f consulting dedicated to change management. Transparency, however, creates its own change issues. Creating information about a process, creates transparency and therefore creates accountability (i.e. if every one knew you did half the work of the person next to you, you might be held accountable in some fashion. Your peers might ostracize you, management might dock your pay or at least pass you over for raises etc. )People do not like to be measured in a transparent fashion because it forces them to be accountable on a daily basis for results or contribution delivered. In my experience, the biggest barrier to leveraging the power of information is the willingness to act. If a company wants to improve a business process it needs to "Just do it". There is no barrier to change, never has been, never will be. The fear of transparency is, I believe, a major barrier to information based business process improvement.

One way to reduce the fear associated with transparency and hence reduce the barriers to change is to level the playing field or create equality around certain information. We know that in a transaction or negotiation the party who has better information has a distinct advantage. Corporate systems are typically set up so that management has access to information that employees do not. When it comes to measuring the effectiveness of an employee's contribution to a business process we should include the contribution of all levels of management to that process. Perhaps if all levels of the corporation have equal visibility into a process and are therefore held equally accountable to the execution of a process then we may lower some of the barriers related to information based change.

So in summary, you can't change what you can't measure, creating transparency into a process allows you to measures its results, transparency creates acountability, equality in transparency for all stakeholders of a process reduces the fear associated with accountability and can remove one of the key barriers to change ... willingness to act.

These same concepts can be extended to political systems, social systems etc. not just the corporate world.

Look forward to your comments...

My Premise

2008-04-03T22:08:00.000-04:00

I believe that information can be used to dramatically transform the world we live in today. Information can be created from systems that generate data. We can collect data about business, political, social, natural and other systems. Business leaders, politicians, environmentalists, scientists etc. try to glean information about the systems they work with so they can make intelligent decisions.

The vast majority of organizations and business today do not create good information and hence make suboptimal decisions. Information is almost solely used to support tactical decision making. How many companies would dramatically change their core profit generating operational processes based on data analysis on a regular basis? How often do governments make radical policy changes to our fundamental social programs based on data analysis? I have never in my lifetime witnessed evolutionary change to the fundamental systems that impact our lives. Decision makers are risk averse and resistant to change. The fundamental systems which impact our world have massive inertia and might change a few times in a human lifespan.

I believe the missing link to aid in decision making is a feedback loop which would allow organizations to create decision making intelligence, that is to understand which decisions work best under different system conditions or states. We need to capture every decision that is made in every organization and the information that was available at the instant the decision was made. A new class of operational systems developed on top of todays transaction processing systems will permit us to capture the human and organizational intelligence displayed by our best decision makers and committees to create knowledge.

I am not aware of a single business or organization that has evolved beyond using transaction processing and data collection. If our decision makers had access to every decision made in their organization, if they understood what the decision outcome was and had statistical analysis of the likelihood of success for all known decisions under all known system states would they not be far less risk averse? Would humanity progress faster if we were willing to make dramatic changes to our world with higher confidence?

I believe that every business can increase their bottom-line profits by 100+% within 12 months if they created good information out of their data today. I believe governments could deliver social, economic and environmental programs with a significantly smaller tax base if they created good information out of their data today. I believe we could have significant impact on our natural world if we created good infomation out of the data we collect today. This is the promise of the Information Age. We are today at the earliest stages of this evolution. If every organization and business maximized the value of their data today, poverty would decline, production would increase, decision making knowledge could cross generational boundaries, life would radically change.

Evidence of the Power of Information is growing. Regulatory changes like Basel II and Sarbanes-Oxley are proof that excellent information is not a nice to have it is a must have. Academics publishing books like Freakonomics and Super Crunchers present numerous examples of how intelligence can be created from the data our systems generate.

I hope with this blog to put forth my ideas related to this premise. I hope to discuss and debate these ideas with interested parties and in doing so advance my understanding about this topic and hopefully influence others to promote this evolutionary view of the Power of Information.