I want to measure everything. I want it all. And I want it now.

The most-heard sentence from Managers: we must measure everything. However, this is far from being true. Not everything must be measured.

Lets imagine you own a business. An e-business. Maybe successful, maybe not (yet). You are in the moment in which you consider implementing a tracking tool (Google Analytics, Omniture, MixPanel, etc.). Then the fateful sentence comes: we must measure everything. Please track every single click, every single action, every field entered in a form, etc. EVERYTHING! Usually when a Manager is asked why everything should be measured he answers: because everything can be optimized with data.

At this point two sentences crosses the Manager's mind:

"In you can't measure it, you can't improve it".

"If you didn't measure it, it didn't happen".

The first sentence is absolutely true in almost all situations. The second one would need some elaboration. Indeed, is true: if you didn't measure it, it didn't happen. My question, as an analyst, is: "Yes, we did not measure it. Yes, indeed, it did not happen. So?". I will state it very clearly here and now: it's not mandatory to track everything. Why? Simply because you can't optimize everything at the same time, or simply because the benefit of optimizing some features is insignificant.

When we measure everything we overcomplicate the implementation. It becomes a pain, it becomes never-ending. The analyst is always thinking what to measure instead of actionating the data that arises from the current implementation. The tool's interface turns into a nightmare: sampling, unclean data, impractical volumes of data to process, etc. The answer to this is: keep it simple.

How do we keep simple a tracking tool implementation? The key is to design Measurement Plans. Avinash Kaushik expresses this in a majestic way. A Measurement Plan consists of five steps:

- Goals: identify the business objectives (sell more, get more leads, increase CLV, decrease returns, improve margins, etc.). According to Kaushik's framework, the Goals must be Doable, Understandable, Manageable, and Beneficial).

- Strategies: for each objective identify crisp goals. They must be specific in the sense that they will be used to accomplish the goals (increase repurchase ratio, increase new users, decrease budget for some marketing campaigns, etc.)

- KPIs: no need to say what a Key Performance Indicator is. Of course, there are metrics that will tell you how are we doing with respect the established strategies.

- Targets: not mandatory but very useful. They are used to establish an end-point for our KPIs.

- Segments: The most important part of the plan. We take segments of users or behaviors that we'll analyze in order to understand where the fail or the success is (new users, paid campaign users, mobile users, users that land in the homepage, etc.). This is the hardest point within the Measurement Plan, and this is really where actionability arises.

The measurement plan is devoted to actionability. Whatever it brings no action is not a valid Goal or Strategy. A KPI or a segment that is not devoted to fulfil a strategy is not useful. At the end is very simple. If your Measurement Plans does not consider that a given button should be tracked, don't track it! If your Measurement Plan does not consider that the fields of a form should be tracked, don't track them! If your Measurement Plan does not consider that scrolling should be tracked, don't track it! In this way you will have a clean tracking tool interface full of data that can be transformed into actionable information.

Ideally the Measurement Plans are build by the team of analysts. They (should) know the business and they (should) know the technology. They are able to talk to the Business stakeholders and talk to the developers. They can gather business requirements across the company, think about business opportunities, and transform them into technical specifications. They are also able to gather all the data, build the KPIs, compare them to the targets, and recommend actions. In other words, they must take ownership of the Measurement Plans, from its conceptualization to its implementation.

At this point, I want to recall Kaushik's "Three Layers of So What" Test. It's a very simple test that will help you to decide whether a KPI (or a metric) is useful or not. Against every metric you want to report ask the question "So What?" three times. Each question provides an answer that will raise another question. If at the third time you don't get a clear action that must be taken, then you just face a nice-to-have metric and not an actionable one. Non-actionable metrics keep the focus off what is really important. Non-actionable metrics and, hence, non-actionable tracking, is like having Diogenes syndrome for data: you collect, collect, and collect data without extracting any useful information from it.

Last but not least, don't fell guilty for leaving features of your site without tracking. Feel proud for the recommendations to actions taken from the tracked items. And, again, keep it simple.

Read More

Using data effectively. I bet you don't!

Every company has data. Every company understands (or should understand) the value of data. But, are you using data in an effective way?

According to FORTUNE, in a post early this year only 27% of C-level executives think their company makes "highly effective" use of data. Now, the question is: is your company making "highly effective" use of data? Or even: is your company making "effective" use of data? Or even more: is your company using data?

Every company has data, and working with data is simple to start: just start collecting and processing some data. You will find very soon that making a (highly) effective use of data is much harder than desired. In any case, I can focus for the rest of the post in the case in which your company uses data.

The key (and very complex) exercise is to define what (highly) effective use of data mean. Let's do it in the opposite way: let's define what an ineffective use of data mean by pointing out some situations that we've found over the last years. Will you be able to pass these tests? And, please, be honest!

Test #1: Which three indicators you check every morning when sitting at your desk? If you can't answer this question with a clear set of KPIs or with a clear set of dashboards, then your company is making an ineffective use of data.

Test #2: Do you have any doubts about the data you retrieve? Is it reliable? Is it clean? Is it readable? If one single answer is "no", then your company is making an ineffective use of data.

Test #3: Does data take ages to be retrieved? Then your company is making an ineffective use of data.

Test #4: Can you retrieve joined data from different sources? If not, or if you need to manually join it, or some sources are not accessible, then your company is making an ineffective use of data.

Test #5: Can you read and relate large amounts of data? If Excel is not enough and you use no other tool to do so, then your company is making an ineffective use of data.

Test #6: Can you retrieve simple data by yourself? If you need permanently to ask for help (either because the systems are too complex, or because you simply don't want to do it by yourself), then your company is making an ineffective use of data.

Test #7: Are the insights properly communicated and understood (not necessarily agreed) by everybody? If data is misunderstood, or poorly communicated, then your company is making an ineffective use of data.

Test #8: Are you having tons of bureaucracy that keeps relevant information from reaching the decision makers who need to see it? Then your company is making an ineffective use of data.

Test #9: Do you figure out the specific question you need to answer, and then determine whether the right information exists and where it's located in the organization? If not, then your company is making an ineffective use of data.

Test #10: Do you take actions out of the data and insights? If not, then you only face nice-to-have data. Hence, your company is making an ineffective use of data.

Your company will achieve a truly data-driven culture if and only if none of these 10 situation take place. So, how do we solve them?

Solution for #1: Define KPIs, organize them, and conceptualize dashboards. Start with a napkin, then draw them in a piece of paper. Have somebody generating a pdf file with them and make sure they are in your inbox every morning. Then evolve with a BI tool.

Solution for #2: If data is not reliable then you must investigate if the source is shaky, the retrieval processes have flows, or the consolidation and calculation rules are buggy. The first case would probably require data-freezing processes and rules. The second and third cases would probably require new data manipulation rules.

Solution for #3: If data takes a long time to be retrieved, you must investigate the cause. It can be that the sources are slow to access and retrieve. It can also be that the transformation and manipulation processes are buggy. It can also be that the reporting tool is not optimized. Last but not least, it can also be that your BI department is flooded with requests.

Solution for #4: If you have many sources that need to be accessed and joined, then you must define ETL processes (Extract, Transform, Load). If the volume and number of sources is really big, then a data warehouse is a good solution.

Solution for #5: First you need to wonder whether you actually need such amount of data. Data pre-processing and calculation at the server side are good ideas as well. If not of these apply (despite I bet they do), then you must find a tool able to read such amount of data.

Solution for #6: Empowerment is a must-have in any data-driven company. Start by saying no to silly data requests. Train your people. Make it simple: implement a reporting tool and teach people how to use it! You can start with fact tables that can't be modified. Then (much sooner than you think) users will start asking for the edition capabilities!

Solution for #7: Check where the bottlenecks are. Make sure your analysts develop soft skills such as communication techniques. Apply Barbara Minto's Pyramidal Principle to your communication techniques. Avoid presentations with 1000 slides. Focus, focus, and focus.

Solution for #8: Again, check where the bottlenecks are. Make sure decision makers can read data (and make sure they use it!). Improve transparency at the BI or Analysts department, and make sure to have proper feedback loops.

Solution for #9: Please avoid the Diogenes Syndrome for data. Don't store ALL data waiting for a miracle to occur and insights appear out of them. Know your business and identify the pain points. Then, and only then, figure out which data is needed. If the data is there, use it. If not, start recording it now!

Solution for #10: Avoid having too many operations and strategical dashboards. Kill the non-essential indicators. A good hint here is the Three Layers of So What test. Ask every indicator or analysis or insight the question "so what" three times. Each question provides an answer that in return raises another question (a "so what" again). If at the third "so what" you don't get a recommendation for an action you should take, then you have the wrong information. Kill it.

In any case, if this sounds complicated or unachievable, reach us: info@ducks-in-a-row.es.

As a summary, data is devoted to actionability. For this to happen it must be accessed, relied, and properly communicated. Then and only then your company will be making a highly effective use of data.

Read More

The broken pyramid. Or the hungry hungry hippos game.

I already heard this story too many times: why do I need to hire analysts if we can build awesome dashboards with this amazing-and-expensive tool? With this mindset, your company will start to play the hungry hungry hippos game.

Ask not, what your data can do for you. Ask what, you can do for your data. When managers delegate the responsibility of analyzing and interpreting data to each department or stakeholder, each one of those transforms into a hungry hungry hippo, trying to catch for themselves all the possible amount of data without contemplating the full landscape, or even fighting against other departments for the insights, data, and information. Furthermore, when managers consider that a Business Intelligence department does not need to go beyond reports and dashboards, they are automatically delegating the responsibility of data interpretation on possibly non-expert eyes or heavily-biased eyes.

In other words, they are breaking the following pyramid into two pieces:

Furthermore, by thinking that having a possibly-very-expensive reporting or BI tool (Qlikview, Tableau, Pentaho, Business Objects, etc.) is enough, they just ignore the real power of data. Who is going to make unbiased predictions? Who is going to play the devil's advocate role? Nobody. Data analysis is much more than gathering, charting, and reporting data. Is about asking the right questions and try to foresee the answers by means of data. Tools are necessary. And expert eyes heavily trained on read them and challenging them is also a must-have.

The analyst role is not only about try to find what data can tell. It's about being continuously challenging the status quo and try to make things to work different. And this sentence applies in the two most-common scenarios: either to turn around a dangerous situation, or either to leverage and boost business opportunities. For this, the analyst should be able to speak business language. Reports, charts, and data sets don't speak it. Predictions and actions, together, do.

Don't let your company become a hungry hungry hippo. Make predictions. Test. Innovate. Don't think analysis is just reporting. Invest on people and tools. But always have in mind that tools are just a small part of the road. Analyst will walk the rest.

Read More

Bed & Breakfast Analytics Foundation #1: Burn the silos!

We are surrounded by skilled people. We are also surrounded by unskilled people. How do we manage such diversity? How can we be sure that our Data setup is the one you need? Discover my thoughts throughout this post, being the first one of a series of posts explaining the foundations of this blog. Want to read it? Avant!

Already as an experienced worker, in my first job as a BI and Analysis "manager" (at that moment I had no clue what a manager was because I never had good managers), I was first required to make an evaluation of their set up on the Data and Analysis department, although they felt very proud of their Business Intelligence efforts. The first question I asked, ironically, was: "Ah, do we have such department"? I was presented a Data Scientist, a Campaign Manager, a guy that was creating some dashboards in Excel, and a self-denominated UX expert. That was the marvellous team. After talking individually with them for a couple of days I came back to my Manager and said to him: "Sorry, you have no team. You have a set of skills but no cohesion between them. You built your Data strategy based on silos!". "What's your recommendation?", he said. The answer was the easiest one I've given in my life: "Burn the silos!".

Is my Data and Analysis setup the right one? Sorry, I would say: NO. I can bet my next month's salary that there are important flaws in the way you've organized your Data, BI, and/or Analysis team.
Before starting, let's make very clear what is probably the most important topic when building your Data team: there is no unique and/or magical solution. Every Data team should be built according to the Business needs, the company culture, the skills available at that moment, the available budget, and a long etc. I'm going to give you the classical consultant answer to the question of how to organize the Data team: "Depends". If you came here wanting to seek a magical solution, this is definitively not your place. Further more, this is not your best job. Building up the best team possible is a tough task, and, for sure, not an easy one. It will require a lot of trial and error and a lot of adjustments.
The basic outcome: what do you expect from your Data team?

I was told many times that the Data and Analysis team should give added value to the company. If you think that a Data or Analysis team should bring knowledge then I must say you're partially wrong. Knowledge is the path, not the mean. The ultimate end of tracking, analyzing, extracting knowledge, etc., is to create action! A feature in the website should not be tracked or analzyed if these actions bring no action to the business. The team should be able to trigger a change in the business when delivering knowledge based on an analysis. If there is no action out of these efforts, then the whole Data strategy is totally worthless.
Simplifying the lifecycle of a Data team, we could say that there are four main steps in an analysis strategy:

In first place we have the business needs. The team should be able to gather them, understand them, and forecast what they could look like, in order to deliver Actions proactively. This implies clear and crystal communication skills with the business owners. Once the need is clear, we proceed to evaluate the availability of the necessary data for the potential analysis (in a separate post I'll come back to this point). Then we analyze, out of which we obtain knowledge. Afterwards we come out with actions: we need to make sure that every single analysis done, every single report delivered, any single data extracted is actionable, in the sense of provoking some change, some further thinking on the management level, or, in the last-but-not-least case, a change of operations and/or strategical decisions. If an outcome is not actionable then, simply, should not take a single minute of the team's time. This is where many companies and set ups fail, and actually creates the sensation that there should be no further investments on the Data and Analysis team. As well, I'll come with further insights in following posts.

Now that, in a high-level point of view, we understand how the lifecycle of Data works it's time to understand that this process, as it's exposed right now, it's based on silos. Each step is traditionally done in independent ways: the web analyst specifies and validates tracking, the data scientist probably determines that a regression analysis is the most suitable to mine the data, the campaign manager states that some campaigns should be stopped, the reporting manager tries to gather-and-show all this information (after it's being generated), etc. All this process, again, has been built over a silo basis. With the process implemented this way is virtually impossible to extract real action out of it.

Here and now is where the Manager needs to step in, in order to wrap up these processes into a single one. Then, and only then, we could start talking about having a successful set up of our Data and Analysis team. When doing this for the first time, is very common to fall into one of the two following mistakes:

Excessive focus on Tracking+Analysis:

This is the most common mistake. Everything needs to be tracked and stored, said the CEO. There's not enough budget for that, said the CFO. There are not enough resources, said IT. There are not enough skills, said the analyst. There's not need for all that, should be saying the BI Manager. If you decide to track, store, and analyze absolutely everything, then I must say you have the data-version of the Diogenes Syndrome. You gather a lot of knowledge. You gather a lot data (which generates a cost that should not be underestimated). However, if there is no clear connection with the Business, we'll never be able to derive action out of this massive amount of knowledge. In other words, all this data and knowledge is totally unorganized and unstructured. The Manager should be able to gather concrete business needs, or, even better, identify clear business opportunities, and then (and only then) make sure the data is available and that a proper analysis is conducted. Actions will appear automatically after this process is properly followed.

Disconnection form the Business:

This is a very dangerous mistake. This set up is what I call the corpses set up. Deriving actions out of a massive tracking and analyze phase is like walking around by killing everybody you find in your path. If there is no connection with the Business (the motivation, the core of Data existence), the proposed actions will result in a worthless effort. Usually this set up ends with a generalized burnout of the Data and Analysis team. If the team (or its Manager) doesn't understand why are we tracking and analyzing, then the actions that we'll derive out of it will not answer any question. Let me depict this with a concrete example. As an analyst in a flash-sales online vendor I was asked by the COO: "For how long we should be running Promotional Sales campaigns? Please, tell me a number and I'll make sure that we'll implement such length". Indeed, was a very simple request, and it took me less than one day to come with an answer (actually, it was 4 days). As promised, the COO implemented by force such number, although nobody understood where that number came from. Even more, at the Data and Analysis team we ignored at all why we were running such campaigns. That is, we did not understand the essence of the request. In other words, the COO was assuming that we should have such campaigns "in order to generate unique buyers". A simple analysis afterwards revealed that such campaigns weren't generating more unique buyers, or that such buyers were spending more after a given time. At the end, by understanding what was the Business need (are Promotional campaigns worthwhile?) we were able to collect the necessary data and to conduct specific analysis to come with a clear action: kill Promotional campaigns. This was only possible by means of wrapping up all four silos in a unique and crystal process: from the Business need to the action.

A new way to understand the wrap-up.

The previous example showed that, indeed, we should consider all four stages as part of a single process. However, there is a way to further pressure the process: instead of considering it a linear process, consider it a cycle:

This is nothing more than having a proper follow-up of the implementation of the proposed actions. Ideally, such implementations will lead on more business needs, that, at the same time, will need more data and analysis in order to be optimal. Part of the Manager's skills we should count for proper feedback loops, implementation follow-ups, etc. This implies that very strong communication skills and technical and architecture knowledge are a must-have.

This cycle will automatically derive in chasing new business opportunities. In other words, when the team, and its manager, perform the proper follow-ups, and closes the cycle, then they will start to do analysis by themselves. Why? Because they will understand the business and check whether the implementation were a success or not.

Now this needs to come to an end. You can have the best business analysts, web analysts, campaign managers, data scientists, etc. etc. etc. But you will only succeed if their manager is able to wrap-up these silos, from the business needs to the implementations and assessment of the proposed solutions. It well worths a try. Guaranteed!

Read More

Bed & Breakfast Analytics: The 10 Motivations and the 10 Foundations

We are surrounded by data. What can we do with that? For what? How? What can we expect and what not? What are the common errors? Size matters? Open source or commercial tools? Here you should find some tips to discover your own journey on the data realm. Bon voyage!

I still remember a nearly-hilarious situation I faced in my first job as data analyst. The CEO came to me and asked me for tons of data very important for a strategic decision. Wow! Panic! Just graduated from College. Just landed in this job. No clue about the business. No clue about data structure. No clue about KPI names. No clue about anything. After some minutes of panic, I deeply breathed and tried to deliver what I've been requested. I promised I did the best I could: gathered data from different departments (no Data Warehouse, no unique source of truth) and different people, in very different formats, I used some advanced and fancy stuff in Excel, and, after 10 hours of intense job, I delivered a kind-of-report. I really had no idea what I was doing. I had no idea what data I was delivering. Some days after I went back to my CEO and asked him how useful my data was. His answer was: "Which data? Ah, that report. Well, we did not use it. We took the decision XXX based on a market research the CMO found in a blog". I suppose I should say thanks.

Sorry. Probably your data-setting is not correct. You should consider start reading this.

After years of experience, probably you've heard these stories many times. The Marketing Manager (random Manager example) requires some data. Let's depict some standard scenarios. The requested data corresponds to...

1. ... clicks, sessions, bounces, etc. This one should be easy. The Web Analytics Manager easily performs this task (is part of its basic skill set), probably by applying some complex advanced segments to the data (easy does not necessarily mean simple). Nowadays, the implementations of the web analytics tools trend to be very complex, mainly because they need to cover a lot of business cases. Simple, right? Well, now imagine that for some unfortunate reason, the Web Analytics Manager is on vacation. Panic! Then, the request is given to, let's say, a Campaign Manager. Of course, he/she has access to the web analytics tool, and hence tries to retrieve the requested data. A bit of panic appears, as the data seems not coherent (of course not, he's not applying those complex advanced segments he should be applying). He then tries to search for some documentation regarding this topics and...  surprise! he/she finds no such documentation. Finally he delivers some numbers, but they all know those numbers might not be totally reliable. At the end, and as the requests get more complex, the process to retrieve such data gets more complex as well. If the process is not clear enough for all stakeholders, the result is a lack of trust on the delivered data, leading to a lack on trust on the data strategy (if such thing exists in the company).

Here I already find my first three motivations:

• Motivation 1: there is a lack of proper documentation. Information transference is virtually inexistent in many e-commerce companies, specially for data-related topics.

• Motivation 2: business complexity translates directly into data complexity. Not every stakeholder understands this implication.

• Motivation 3: wrong data strategy leads to lack of trust and, even worse, to wrong decisions. 

2. ... revenues, sales, etc. This one gets a bit trickier. The Marketing Manager pings somebody by BI, or by Finance. Traditionally the request is not complete or it's poorly written: time frames missing, before/after refunds, etc. Normally, such simple requests, requires 2-3 iterations, leading, again, to a lack of trust in the provided data. In some cases, a variation of this scenario takes place: reports and data is built by manually joining data retrieved from different data sources, as the full data map is not clear for everyone.

Again, two more motivations appear:

• Motivation 4: it's very hard to write clear requests.

• Motivation 5: outside our comfort area, finding data could be a challenge. Even when having a data warehouse, or a nice-and-expensive-but-totally-useless BI tool.

• Motivation 6: it's easier to request than to retrieve, and it's easier to retrieve than to process.

3. ... data that has been already requested any time (many times?) before. This one is a quite disappointing. There is nothing more frustrating, in both directions, that performing a recurrent request, and being requested for the same time after time. Assume for a second that, indeed, such data is available. Why is data recurrently needed not easily available? Even worse, what if we have (as I mentioned in the previous paragraph) a very nice BI tool? Why some users are reluctant to use self-service data platforms? Now, assume that the data isn't available. Tough times are about to come: it's time to reach IT in order to start gathering this data. Normally, from a BI/Data department is very hard to write clear specifications for IT to start gathering some data, due to several reasons: lack of knowledge on the platform, lack of database architecture knowledge, etc.

With this, two further motivations appear:

• Motivation 7: having a BI tool does not ensure self-service. Having a self-service platform does not ensure data availability.

• Motivation 8: communication between BI and IT could be a struggle.

4. ... data, or analysis that we don't know whether it can be accomplished or not, or data which is not clear how is going to be used upon delivery. The first challenge when receiving a request, or when performing it, is to determine whether it can be done or not (assuming whether it makes sense or not). Many of the analysts work directly with data, without designing a plan for such analysis or request. That is, both requesters and analysts work without an analysis framework, even when it's clear that the analysis will require some time to be finished, probably due to its complexity. A different case appears when the request is coming from the CEO. We have to admit that is very hard to say no to our CEO. However, the CEO does not know everything, and he's not always right. Even CEO's requests need to be challenged, understood, and accepted.

With this, we find my two final motivations:

• Motivation 9: working with analysis framework is a must-have.

• Motivation 10: determine whether a request (for data or for an analysis) makes sense. Find the way to challenge every single request.

The Decalogue: the 10 Foundations of Bed & Breakfast Analytics

With my thoughts over the desk, and the motivations I find out of them, I'm ready to state my Decalogue.

1. Burn the silos! Managing data requires transversality and deepness on each vertical. Skill silos are not suitable any more.

2. Complexity matters! Understand how business complexity affects data complexity.

3. Better alone than... No data is better than wrong data.

4. Write, write, and write. Documentation is a must-have. Learn how to document and learn how to request.

5. Going beyond your comfort area. You should consider expanding your comfort area. Even more, you should consider not having any comfort area at all.

6. Bring order to chaos. Narrow your analysis: understand the need, design and framework, and only then, retrieve data.

7. Communication is the key. Your CEO does not care about regression models, decision trees, or how fast your database engine is. He wants a way to keep a sustainable and profitable business.

8. Choose wisely. The right tool for the right set-up. Self-service is not always the best solution.

9. Don't rush! Data is a path with some mandatory steps. Cheating leads to frustration and lack of trust.

10. So, how are you doing? Integrate data. Move aways from data silos. Design KPIs, reports, and dashboards based on integrated data.

So, what's next?

With all these, I want to share with you how do I measure, why do I measure and how do I analyze, with the hope that you will join me walking through the learning curve of the data-related world. In a Bed and Breakfast hotel you share your experiences with many others, and you obtain a clean and cheap way for sightseeing. This is exactly what I pretend to do here: every two or three weeks I will be sharing my thoughts, tips, tools, and techniques. Everything what I know will be shared. I'm willing to do so!

Hope you find this interesting, and welcome onboard!

Read More