In this huddle we discussed the current status of privacy. The discussion, perhaps a little unexpectedly, drifted to how different brands perceive privacy legislation and the role of education as a driver for privacy advocacy.

The discussion clearly highlighted the complexity of the topic and why most of us still are not entirely clear about next steps. But it was a thoroughly enjoyable exploration of the topic with all participants agreeing that:
(a) It was a valuable discussion and (b) there is so much more to discuss

The main points covered included:

• A brief discussion about the impending Canadian privacy legislation (as two of the delegates were Canadian)
  • The expectancy is that the Canadian legislation will be stricter than current US legislation
  • Prompted a discussion about how to handle varying country legislation
    • Do you treat visitors based on their local legislation or do you apply the strictest in each case?
    • Still open for debate given the evolving nature, particularly of the Canadian legislation
• Brands will treat privacy differently
  • GDPR is designed to protect the European consumer irrespective of the website geo-location
    • Five years on, there still seems to be confusion or uncertainty about some of the practical aspects of GDPR
    • Some North America lawyers have interpreted GDPR differently, consequently privacy settings' implementations differ from brand to brand
  • Brands approach to privacy generally falls into one of three:
    1. Some are taking a very stringent approach to privacy
    2. Some are ignoring it altogether ("wait and see")
    3. Some are developing a "consent acquisition strategy":
       • E.g. privacy settings will depend on the geo-locality so if the user is US-based they will automatically be opted in; if in EU automatically opted out
  • Europe appears more serious about building privacy into everything
  • US government agencies are similar but many brands are still in a bit of wait and see (who get's a fine!)
• Customer attitude to privacy
  • We generally agreed that customers do not care about privacy so much because they do not understand what all/most consent requests are really about
    • They click on 'no' without necessarily thinking about it - "just get that annoying overlay of my screen" attitude
  • Education
    • In UK, Internet safety and privacy are part of the obligatory curriculum in public schools
      • While teaching methods may vary by school it is perceived that the next generation of kids will be much more privacy savvy
    • But in some contradiction, one US-based delegate shared his experience growing up in 1990s and being taught to be careful sharing any PII online; but now younger adults in US are sharing a lot of PII online publicly, which is a concern
    • Who is going to advocate for privacy? Most market actors don't seem to have the incentive to do so
      • Brands certainly do not have an incentive
      • Governments should but are lobbied by brands/enterprise to minimize legislation
      • Individuals as a whole do not seem to understand it and therefore either ignore it or are almost apathetic to it
      • Yet privacy legislation is moving forward globally so must be championed by someone. Privacy interest groups?
• Offense vs defence
  • Generally seems like companies prefer taking defensive strategies
  • But there re some brands that are centered around privacy which are offensive (e.g. Hey email service launched by Basecamp in June 2020)
• Need to differentiate between licensing and privacy regulation
  • Global brands (e.g. Netflix, Spotify, Apple+ etc.) will have territory-specific content rights and licensing agreements
  • The general consensus is that licensing is based on the user's current geo-location but their privacy settings will be governed by where their account was set up or currently domiciled
    • E.g. A US-based customer may consume content abroad when travelling and might be limited in terms of content accessibility but their privacy will be regulated by US regulation irrespective of where they consumed the content
    • This too could differ from one brand to another as currently it seems to be open to interpretation

  • The conclusion was thinking about how brands treat content licensing could help in our approach to how we think about some aspects of data privacy

    • Specifically how we handle different laws in different countries and people changing locations when they access sites

The pandemic brought many organizations’ hiring and promotion efforts to a screeching halt. Less face time with peers and leaders can make it even more challenging to demonstrate capabilities for growth. In this session, Amber Zaharchuk of Walt Disney DTC & Int. will lead a discussion about strategies for building technical and soft skills remotely, keeping our careers trending in the right direction. We will discuss the following questions:

• What are you doing differently to ensure your work is valued and understood?
• Have your career goals shifted moving from a physical to virtual office? How so?
• How are you maintaining relationships with leaders and peers?
• Has your org adjusted employee evaluations or education/training opportunities?

While the pandemic is temporary (we hope), virtual working is likely to increase in the future, making this an increasingly relevant topic and huddle.

Here are some highlights from both the original (abbreviated) huddle and the re-run a couple of weeks later:   

• How is remote work different today vs. prior to COVID?
  • People are more accepting and more human, where now it’s ok for dogs to bark and kids to run in
  • Now everyone is in the same situation, so it levels the playing field and makes everyone feel included
  • Expecting some of it to remain after the pandemic, but we need to avoid leaving people out if only some members of the team return to the office
    • The hybrid model may be more effective if entire departments go back or remain remote, not individuals within the same department
  • Extroverts miss working together
  • We discovered trust and learned that people do have a strong desire to do the work, and if they have to deal with personal challenges during the work day, they will work later in the day
    • Some companies previously didn’t allow WFH on Mondays or Fridays due to the perception that employees were really enjoying a long weekend. Still to be seen whether this rule will be removed, but the hope is that companies will see beyond that, given the trust built over the past year
  • We need to avoid a situation where we take the "bad" aspects of the office and adopt them to WFH
• How do you keep yourself visible to management?
  • Control mechanisms provide transparency to management, and that builds trust
  • In agile environments, contributions should be (by the nature of the methodology) fairly visible via daily scrums and moving Kanban cards
  • Kanban board and project management tools could definitely help
    • But not a good idea to add them just for the sake of visibility
    • Must be part of the work process already
  • Jira, Salesforce, Microsoft Teams were some examples of tools being used to make work visible
  • One delegate who manages employees mentioned how they always used time sheets, but now it plays a much more important role
    • Not for controlling people, but rather as a supportive tool to understand how employees are splitting their time between all essential tasks
    • Now uses time sheets to ensure employees are happy about their time allocation between the various tasks
    • Became an additional employee satisfaction optimization tool
  • Now that everyone is virtual, remote workers can take more ownership of communicating results
    • Making yourself accountable and making yourself known
    • Leaders must help their team members to become better at communicating results and making themselves more visible
  • One delegate targeted 25 people that he values and asked for their advice on how to make himself more visible (but without making it too much about himself; show off without showing off, because if you do not toot your own horn, there is no music!)
    • Created weekly sessions on non-work related topics and invited senior leaders to connect with people on a personal level, and to build trust and familiarity with key stakeholders
    • Targeted some leaders based on topics that would be of interest to them (e.g., sneakers, wine, cooking, dogs, books, happy hour, etc.)
    • Created a shared doc with all the possible topics and suggested to people that they could lead the discussions
    • Ended up running all of the sessions, as it does require a few extroverted people to make it work, but was great success
  • Book recommendation for how to overcome barriers to getting conversations going: Never Eat Alone
  • Continue growth programs for your team
    • Do StrengthsFinder, find out who we are, then who we want to be, then set goals to figure out how we want to get there
  • Use donut intros on Slack to meet new people and build your internal network
• Have you changed your career interest or path?
  • No one in our group indicated that they were looking to change career path
  • One delegate shared advice she got recently to interview for a position, internally or externally, at least once a year even if you do not intend to move
    • Does not need to be about moving up in your career or more money, but to keep you evaluating your career and keep your resume relevant
• What are you doing differently to ensure your work is valued and understood?
  • One delegate shared that she now works much more on strategic initiatives
  • Another delegate pointed out that because we are all remote, less work is generated by others for us, so employees might find they have more time to be creative and think of the bigger picture
    • That might tie with people finding more time to focus on creative and possibly strategic initiatives
  • Do things outside of work that you love, because you will work better when your mind is fresh
• How have you adjusted feedback or employee evaluations?
  • It is really hard to give difficult feedback on Zoom
  • Work on being very explicit with instructions and feedback
  • Some people need more time to adjust to WFH, and that could be slightly frustrating for those that have been WFH for a while
  • Everyone is dealing with different challenges and deals with things in different ways, so we have to be patient and empathetic
  • "Be ruthless in business but nice to people" -> one delegate shared that she had to spend a lot more time speaking to employees about general issues and handling lockdown
  • Book recommendation: Radical Candor
  • Apply your skills to volunteerism through orgs such as Common Impact

Increasingly analysts they are working more closely with data scientists and data engineers who have entered the scene along with large and diverse data sets. Undoubtedly, a welcomed development for analysts but it also presents some challenges.

• How does this shift change the way that analysts frame their skills and deliverables?
• How do we go about sharing analysis across different toolsets?
• Is there a common tool/language all can share?
• How do we ensure reproducibility and cross coverage (tools and techniques)?

Jose Maldonado of Verizon will lead this huddle aimed at team managers and analysts looking to improve team and personal productivity respectively.

This huddle ran twice – once during the main DA Hub event (truncated session) and as a re-run a couple of weeks later. Given the delegate groups were different and the original truncated huddle still produced a very good discussion, we have chosen to split the summary into two:

Original Huddle Summary:

• Tools
  • Can be easier to collaborate when being prescriptive on the teams’ choice of tools/programming language. Time saving element to converging on tools
    • This can be difficult with members that are stuck in their ways (R users refusing to code in Python or vice versa)
  • Avoid excluding potential candidates based on tool experience
    • Hire for potential and base skills
  • There are challenges in some orgs when it comes to frameworks for analysis and governance
    • The enterprise is sometimes too focused on data governance rather than the process
• Soft Skills & Experience
  • How do we optimize for different mindsets; BI / Analyst / Data Scientist?
    • How do we get folks on common ground
      • Book Clubs to unite teams
      • Assign a team member to serve as "connector" across teams
        • Sometimes natural connectors emerge
          • Likely a personality trait (could help with finding connectors)
        • Connectors serve as the "cross function glue"
        • If a role is created to serve this purpose it may not be as impactful as a natural analyst/team member (credibility of connectors)
        • Creating connectors based on greater exposure to other skills/teams
      • Rotational programs can play a part by providing more context and org ecosystems
  • Challenge of team members acquiring skills to become "Data Scientist"
  • Allure of becoming a data scientist can create shortage for the other required roles in the org such as Analyst, Data Engineers, etc.
  • Importance of showing the value other roles add

Re-Run Huddle Summary

• Team knowledge management
  • One delegate shared his experience with a banking client where they had multiple analytics teams so they set up internships in the various teams to cross-pollinate
  • Silos could be challenging not only in big orgs but small ones too
  • Even if there are common data applications
    • Comes to politics and must be done top to bottom; simply won’t work without senior management buy-in and active support
  • Building relationships are critical to bringing together different analytics professionals and enhancing practitioners' skills
  • A delegate managing a team of analysts and experimentation practitioners initially had the philosophy of trying to keep up the skills and knowledge of all team members
    • However, with time realized it might be better to have different team members specialize in different areas (both technical and domain expertise)
    • Given the growing demand for data and the data tech it is much harder for any one person to be an expert in multiple disciplines
    • So need to actively manage your team's skills based on the organizational needs and the individual's areas of interest
  • Now might be the right time to develop diverse skilled teams given we're not working in a shared physical space
    • Leverage synchronous work to do more pair projects and cross pollinate skills
  • One delegate shared a technique called Mob Coding Sessions (used in Agile development)
    • https://www.lisihocke.com/2017/04/our-teams-first-mobbing-session.html
    • Analysts work together on a challenge with the keyboard control moving from one to the other
    • Can also do it virtually
    • The idea is to get everyone involved and inputting and at the same time improve skills across the team
• One delegate shared how Airbnb use R and data science to scale up skills
  • https://medium.com/airbnb-engineering/using-r-packages-and-education-to-scale-data-science-at-airbnb-906faa58e12d
• Need to keep ambitious employee engaged
  • But could be a challenge in an org that does not have growth potential for individual contributors
  • A delegate shared the importance to start with employee nurturing as early as they join; much harder to do a year in
    • Uses a 16-types personality assessment (Myers Briggs) during the recruitment process
      • https://www.16personalities.com/personality-types
      • Helps identify which interview questions to ask, what skills they can bring, potential strengths, how they would fit into the team etc.
      • Found it extremely valuable and helps save a lot of time
• We also discussed the importance of good documentation in a diverse team and in general to improve productivity and cross coverage as members take time off or move on to other opportunities

With decreasing attention spans (in this case by our business stakeholders) and increasing need to paint a full picture of customer experience partnering quantitative analytics and qualitative consumer research is no longer an option but a must.

In this huddle, Laura Guerin of Wayfair will lead a conversation on marrying quantitative with qualitative data. We will answer these challenges:

• Use cases leveraging qualitative data sources
• How to map qualitative data to align with customer journey insights?
• Presentation tips and suggestions (relating specifically to combined qual/quant analysis)
• Strategies for building a cross-functional team in absence of a dedicated function
• What are the organizational challenges and how to work towards resolving them?

You will come away from this huddle with practical examples of how to improve your customer analysis which in turn will help your org provide customers with better solutions.

So Many Tools:

• Use the right tool for the situation
  • Quant to understand “what” and be statistically rigorous
  • Qual to understand “why” and get directional insight with low lift
  • usertesting.com for general consumer feedback
  • Surveys, focus groups, and reviews for your customer’s feedback
  • Session replay to validate data-driven hypotheses

Qual then quant:

• Use research to get in front of customers fast
• Then test the new customer experience for rigor before rollout (poison cookie test)

Quant then qual:

• Do the rigorous analysis with data
• Then emphasize your point using a customer quote to generate internal buy-in

Surveys:

• When curating a customer survey, the first question should get the customer engaged
• Make it about them. Be targeted by surveying a specific customer segment and/or specific part of the customer journey

Actionability:

• Have a clearly defined problem statement and focus on actionability - always

Diplomacy:

• Be diplomatic when working across teams. Leverage the synergies of quant and qual teams to make an overall bigger impact

Many, particularly non-analysts, might consider growth analytics and product analytics as one and the same. Both disciplines examine customer behaviours and share common metrics (e.g. revenue generated, purchase and churn rates, repeat visits and retention rates) but serve different purposes. The main purpose of growth analytics is to analyse the health of the business whereas product analytics is particularly concerned with measuring engagement with products online, informing product development and improving customer product experience.

Ole Bahlmann has vast experience using both disciplines. In this session, we will discuss and debate how to get the most out of product analytics including:

• Use cases for improving product retention, customer experience and reducing churn
• What are the essential metrics and KPIs for product analytics?
• How best to manage the relationship with product team/s?
• What should your product analytics tool stack look like? Do you need a dedicated product analytics platform?

We welcome practitioners using GA/Adobe Analytics/other for product analysis as well as those using (or looking to use) a dedicated product analytics tool such as Amplitude, Mixpanel or Heap. Expect to come away with a fresh perspective on product analytics.

In this huddle we aimed to answer the following three questions:

1. Which features/functionality would you change in GA/Adobe/your tool?
   • Challenges with closed source tools
     • Can act as a black box and can then be hard to get data out (through official or non-official methods)
     • Often marketed that you can just "flick a switch" when it rarely is
     • Work well for simple use-cases
     • Challenging pulling the data out and moving it somewhere else
     • Orgs with diverse technical use-cases or complex technical set ups can struggle
     • Do not own the data; often do not have access to the raw data
     • Moving data around multiple different systems
     • Non-retroactive data is painful
     • Often marketing people set up the GA tracking but then data engineers cannot get at the retroactive data
     • Data engineers often do not find GA very intuitive
     • At the end more configuration required
     • You can make your business/product into GA, using arbitrary decision
       • This works so long as the translating documentation is accurate and stays up to date
   • The role of data engineers
     • Engineers are great if they know what they are tracking
     • Does not work as well when engineers are shoe-horning data into a restrictive data model such as GA
     • Giving clarity to those implementing who understand how the product works
     • Should this be addressed by people and process, rather than the tool?
     • This does create extra process, but does make data collection more reliable in the long term
     • Has anyone ever use a home-built solution? Very rare
2. What direction are these big players taking their analytics/digital marketing products?
   • The big players are trying to tie customers into their ecosystem
   • Adobe has acquire Magento and Marketo but these tools still do not integrate well with the rest of their tech stack
   • Should allow users more access to raw data like GA4 and BigQuery
   • GA4 changing from sessions to events
   • GA3 was a hook to get customers spending on ads; GA4 is to hook users to GCP
   • Big players are likely to embark on new acquisitions
     • Tools such as Indicitive, Amplitude are performing GTM strategies together
     • Their strategy is to claim you can do a lot of this with a small group of tools
     • Similar to Looker, Snowflake, dbt and Fivetran
       • Looker and Snowflake no longer as pally as they used to be since the Google acquisition
     • Some significant privacy concerns rising from such integrations/acquisitions
   • CDPs
     • CDPs probably are not super privacy safe; can be a privacy nightmare
     • Salesforce, Oracle and Adobe are all launching CDPs
     • 1^st  party data in CDPs will become increasingly more important
     • Market maturity – shifting from "should we get a CDP" to "what CDP should we get?"
     • One of the participants shared a use case in healthcare where all data had to be anonymised
     • But this is still potentially not enough and data can still be de-anonymised
   • Some delegates indicated they would like to see more self-service from packaged tools
     • Not quite low-code, but customisable across the data stack
     • All tools are quited silo'd
     • Should be easier to configure across the whole stack
   • SaaS tools benefitted when customers did not know how to handle big data, and tools were focused on a single use-case
     • Data mesh? Only a few companies do this currently fewer do it well
     • Large companies acquire a lot tools
     • Need vendor fabric, then data fabric and tech fabric
   • Is transparency and observability going to become more important in web analytics?
     • OS is growing in popularity
     • The open framework is definitely growing in popularity in digital analytics
     • This is very parallel to the tech open stack (VMware, openstack) separating hardware/os/vendors
   • People often forget that a product with no licence fee is rarely free
3. What is the next big innovation in digital analytics?
   • Is ML the answer?
     • Google are pushing ML into a lot of their products now
     • There are often much simpler, non-ML solutions that will drive as much value
     • It is not silver bullet
   • What does it mean for analytics talent?
     • Do orgs know if they should hire in data scientists or ML engineers or train internally?
     • Therefore, there is more pressure for analysts to be more horizontal and have a wider range of skills
     • More about the problems you are trying to solve
     • A simple solution may well be the best answer
     • Depending on which stakeholder you aim to provide an answer, you will get multiple different answers to whatever problem you are trying to solve
   • Does the proliferation of ML in tools, mean that data scientists are no long so important???
     • There is still the "Black Box" – there need to be explainable models
     • For some industries, interpretability is a must – finance and health
     • Still a long time before we have human data analysts
     • Always will be important to be able to have a conversation with data scientists
     • In the near future environment, architects roles could be in more demand than fully skilled analyst or scientist roles

In this huddle, Mindy provided a short intro presentation into the topic and how Hudl is tackling this challenge. A roundtable discussion followed the presentation. The key takeaways were:

• How do you keep a pulse on the data talent market?
  • oKeeping your networking alive
    • Meet/catch up with managers in similar positions to you on a regular basis (at least quarterly)
    • Staying networked with local recruiters
  • Reviewing nationwide salary surveys for data roles
  • Learning from offers that team members get
• How do you cultivate a talent pipeline and diversify your data team?
  • Managers are not the only ones to cultivate a talent pipeline
  • Team member referrals are a valuable resource for backfilling
    • The upside is referrals are already screened for fit within the org
    • The downside is creating an echo chamber of like hiring like and creating future risk of attrition in groups
• What is the role of engagement v. money and how do you keep engagement on your data team high?
  • Engagement v. money is specific to the employee
  • It also changes throughout one’s career
  • Managers should create together with the employee a spider diagram of the attributes that currently matter to them
    • Tailor their personal development plan accordingly
  • Managers could also create a spider cumulative spider diagram for their entire team to get an overview at the team level
• What is your compensation philosophy and retention plan? How do you get organizational alignment around that plan?
  • Budget challenges are always there
  • Other ways to compensate employees without breaking salary bands
    • Professional development / education budgets for training
    • Attending events either as speakers or delegates
    • Granting employees time to work on their own projects or charity projects
  • Organizational alignment depends on decision makers’ understanding of the value of data
    • Need to educate them on the state of the market for data talent
  • There will always be companies willing to pay more for talent

In this huddle we set out to answer the following three questions:

1. How do you choose which data to collect and which metrics to measure?
   • Start by defining the goal or working with teams to understand their goals
   • Define the questions that need to be answered
   • Teams often are not aware of what data is being collected or what the limitations and considerations are for collecting data
   • It is helpful to educate internal teams on data collection requirements
   • It is important to think about whether the juice is worth the squeeze with measurement
   • Ask “why are we doing this?” before making big investments in data collection and analysis
2. What role do attribution models play in understanding what works and does not work?
   • Many companies are interested in explorating algorithmic attribution models
   • Many analysts would like to improve their companies attribution model but struggle to get buy-in
   • It is important to show an example of how improving attribution could improve the marketing team’s effectiveness
   • Often trying out different attribution models show similar results because teams have tailored their tactics to the current attribution model
   • The true benefit of more sophisticated attribution models comes after teams emboldened to start adapting their tactics
   • Many analysts would like to expand attribution beyond acquisition channel and think more about how to attribute impact to different components of their websites
3. What tactics do you use to go a layer deeper to find truly actionable insights?
   • Controlled experiments are a great way to go beyond attribution and uncover what is truly driving incremental growth
   • Controlled experiments also work well to understand which parts of the website contribute the most benefit
   • When it comes to understanding what works offsite, geo experiments can be a good option. Methodologies include Google’s:
     • CausalImpact framework
     • Geo framework
   • Segmenting data is another very effective way to go a layer deeper. We discussed some examples
   • Sometimes teams need help defining segments. We discussed how to do that
     • While there is usually a standard way of segmenting, exploring ad hoc segments can be illuminating as well
   • Look for 80/20 opportunities where one channel, part of the website, or other segment accounts for a disproportionate amount of revenue or engagement

Most actual analytics is about proving/disproving hypotheses not data exploration. Sometimes, those hypotheses get dumped in your lap by a curious business partner – but what if they do not?

In this huddle, led by Gary Angel, we will explore techniques for hypothesis generation considering specific types of research (VoC, Behavioral, Ecometric, etc.), hypothesis generating analytic techniques, creating formal research programs, organizational crowdsourcing techniques and even cognitive tips/tricks.

This huddle aims to help participants develop a more conversant approach to hypothesis generation which in turn would serve to educate business stakeholder on how to pose better hypotheses back to the analytics team.

• The huddle focused on the when in the analysis process hypotheses are generated, techniques for generating them, and the extent to which analytics and data science can be usefully compared to science and can/do borrow from the Scientific Method.

1. When are hypotheses generated
   • Participants saw numerous cases of hypotheses being generated at both start of analysis and during data exploration. Neither method was clearly preferable.
   • Many analysis projects start with a business question or embody an initial business hypothesis
   • Other times, a data anomaly is detected or a broad exploratory analysis reveals potential issues that then serve to form the basis of hypothesis generation
2. Techniques for Hypothesis Generation
   • When driven by the business, it was often the case that hypotheses were generated by “subject matter experts” – gut feel. In this instance, there appeared to be no significant reason to discount that type of hypothesis
   • A number of use-cases explored hypothesis generation from VoC and operational reporting or alerting
     • Some companies allow field/call-center agents to surface potential issues or hypotheses about customer problems that then become analytic problems
     • Similar use was made of VoC – particularly open-ended questions from which summary categorizations could be derived
3. Science and Method
   • Some companies focus heavily on pre-generation of a hypothesis tree – a grouping of all interesting hypothesis for research
     • This is generally part of an integrated experimentation program
     • By pre building the tree, it’s possible to target appropriate resolutions to hypotheses – so experimentation, VoC or analytics might each be used to find a resolution/answer
   • One of the drawbacks to using the scientific method is that it relies on a research program where integrated hypotheses can be tested together and verified or refuted
     • Most business analytics problems don’t allow for as careful experimentation and don’t allow for replicability
     • This often creates either garden of many paths situations or strong confirmation bias situations
       • Both are likely unavoidable in many situations
     • The idea of grouping hypotheses into a “tree” provides a way to potentially come closer to a true scientific method since it avoids some of the problems of trying to establish or refute a single hypothesis
4. Using more formal hypothesis generation
   • Participants felt that more attention to training analysts would yield two kinds of benefits:
     • Young analysts need to be trained to think about hypothesis – to question the data not simply report it. Adding in formal hypothesis generation steps might be useful
     • Getting more experienced analysts to focus on rejecting the null hypothesis as a part of the analytic process

In this huddle we set out to answer the following three questions:

1. What challenges do you face when standardizing digital analytics across your organization?
   • Receiving buy-in from marketing partners outside of your department and external digital marketing agencies can be challenging
   • When the initiative comes from leadership, it becomes more valuable
   • There can be push back from employees who have invested time developing an analytics methodology that is going away
   • Some have developed ways to gamify the process – making it a competition from site owners/marketers to adopt the new standards
   • Proper communication to marketing partners and external agencies along with integrating them into the new ways of working can help with adoption
2. What digital tools have you implemented to solve for discrepancies in digital data collection?
   • While most attendees use Google Analytics and Adobe Analytics for implementation, there are several tools that are used to support the process. These included:
     • ObservePoint for monitoring tags and tagging governance
     • Internally developed tools for integrating of tags documentation to updates to the websites
     • Documentation is key to ensuring process is maintained and can be shared across org and with partner agencies
     • Confluence to document all analytics request and integrate into the tagging process seamlessly
     • Cypress.io for documentation linked to tagging implementation
     • One organization used an internal wiki to create a data dictionary and implementation detail repository:
       • Supported by an initiative to gamify the process among employees to regularly keep the wiki up to date
3. What roles or team members do you currently have in place to implement a broad digital analytics practice?
   • Some attendees rely solely on internal employees to train, implement and analyze the data
   • Other employees rely on agency support to implement while an internal team focuses on strategy and socialization
   • It was agreed that proper communication and documentation were key for a successful analytics implementation strategy

Top takeaways from this huddle included:

• How are CX teams different that digital analytics or optimization teams?
  • At their core, they are very much the same, but CX teams add experiential multichannel human centered design and the inclusion of brand impression (i.e. trustworthy, credible, thought leader, etc.) goals to the analytics and optimization KPIs that are more typically conversion, completion, etc..
  • “Activation Team” is an emerging title for those working in the analytics to CX activation realm
• The struggle to pivot from using data to produce purely “nouns” analyses, dashboards, segments to activating data as a “verb” has challenges at several stages in the live cycle:
  • Data Capture:
    • CX data capture relies heavily on “who and the what, so visitor identification, “events”, and meta data about content and product interactions (i.e. merchandising your digital content) is the primary currency
  • Assembly and relationships:
    • CX requires the bringing together of data from multiple sources (web, in-store, etc.) and orients the data to the customer first, then the activity (who-what), a slightly different north star then traditional activity centered analytics (i.e. checkout  process)(what-who)
    • It also requires, particularly in B2B environments, a healthy solution to “relationships” that customers belong to
  • Analysis and Opportunity:
    • Translating the data into what stage of the journey this customer is in and what the opportunity to help them move forward is, is a significant task requiring very different (and often AI/ML based) solutions to solve
  • Content:
    • A strategy involving four customer segments x a four–journey stages journey is 16 unique pieces of content and business rules for orchestration
    • Make that 8 segments and 8 stages and you now have 32 pieces of content and business rules to create, deliver, manage, and monitor
    • The cost of ownership is high here, important for business to be aware
  • Delivery:
    • Real time integration into your omni-channel solution to experience delivery engine is where all the rubber hits the road
• Atomization of teams into functional silos (Analytics Team, Content Team, Marketing Team, Merchandising Team, etc.) is a significant barrier to activation
  • Pods, Labs, and Agile Teams help tear down the walls between stages of the process, reduce resource and prioritization overhead, and bring practical utility to everyone’s work by seeing the full, end to end, value chain of idea to delivery
• Voice of Customer and session replay tools are critical in understanding and evaluating the customer’s experience in CX
• Interesting debate... as IoT continues to add devices to the CX mix, is all of that data “digital analytics” and the role of the digital analyst to analyze?
  • Is that where we are headed?
  • Most participants thought so and highlighted BI as customer master record work and processing and crafting of the interactions and behaviors across channels as the work of the digital analysts!

We have discussed a variety of relevant subjects and topics during this huddle.
Below are some of the practical outcomes shared:

• Ensure you have flexible staffing models.  Vacancies will happen – and should happen if we are helping our folks grow.  We need to be prepared.  For in-house teams this includes:
  • Partnering with an agency who can help when you have immediate needs
  • Over hiring, not waiting till you have a full person’s work load before bringing some one new in
  • Training up – find ways to grow talent internally, before you need them
• When you are recruiting, consider people who have taken non-traditional routes into the field including:
  • People transitioning from Direct Mail or other non-digital but data heavy marketing roles
  • People who have augmented their education through certifications and online training
  • Look for interest, talent and commitment
• How do you look for interest, talent & commitment?  Here are some interview questions that may help:
  • Tell me about the last time you looked up and several hours had gone by?
  • What is your favorite data set?
  • What analysis are you most proud of?
  • Tell me the story behind this chart (draw a random chart)?
  • What is your favorite game?

We have discussed a variety of relevant subjects and topics during this huddle.
Below are some of the practical outcomes shared:

• Quarantine has made mentoring much more difficult
  • Connecting with staff has become much harder
  • Development often gets pushed to the side
  • Replicating the camaraderie of the office takes extra work and different tools
• It is helpful to divide work check-ins and development into separate meetings
  • Virtual walk and talks have become a useful tool
• Soft skills are tougher skills to mentor, and even harder in a virtual sense
  • Advocating, managing up, and providing feedback were some of the most difficult but most important
• Patience is becoming an issue for younger staff
  • They are often frustrated when they do not see change happen fast enough
  • Helping staff learn and manage patience will be important in their career development
• Helping analysts get out of their chairs and better understand the business and its problems is another important area of mentorship
  • We discussed pairing the analyst with business stakeholders and subject matter experts to learn both broad and deep contexts
• Take the time to both reflect on progress and celebrate accomplishments. Both build confidence in the analyst
• Resources: The Coaching Habit, Situational Leadership

It is a bit cliché to say that 2020 was unprecedented. As we have all grappled with the impacts of the global pandemic there are shifts happening in eCommerce that are worth noticing. Useful and effective paradigms for understanding customer acquisition and conversion were already undergoing a shift due to the changing regulatory landscape and browser privacy settings.

In this huddle, Michael Helbling will lead a discussion around how we are adapting to the unexpected shifts in channels, how customer preference shifts and the closing of most retail locations impacted strategies, and how the regulatory and browser changes have necessitated shifts in tactics. We will also examine how the pandemic has impacted predictability and how we are managing analytics through those challenges.

Besides being able to just vent a little bit about what this past year has looked like in eCommerce, delegates will gain shared knowledge on tactics and tricks that will be immediately applicable heading into what will definitely be an interesting holiday shopping season.

“Every day, people and organisations produce approximately 2.5bn GB of data.” It is nearly impossible for any data team to keep up with the pace of catering to the data hungry and the insights starved product owners, marketers and executives among others. How can we promote data democratization without feeling like we are giving away our jobs?

In this huddle, led by Asif Rahman of AccuWeather, we will answer the following questions:

• What makes a successful self-service analytics program?
• How can we reduce data bottlenecks?
• How does your self-service stack look like?

Come share your analytics self-service experiences and get to hear from other leading practitioners theirs. Together we will come away with practical ideas on improving self-serve analytics.

We had a constructive discussion about the challenges of data democratization and self-service in analytics. Delegates shared both frustrations and use cases of successful self-service.

Some key points included:

• Before we can start to think about sharing our data widely, we have to ensure we (the analysts) trust the data 100% and its validated
• Democratizing access to “certain” data allows analysts to focus on what they “want to do” rather than answering ad-hoc questions all day (“what they have to do”)
  • Democratizing data should not come off as a threat to the analysts; leaders have to ensure they are communicating the value so there is wider adoption
• How to enable self-service
  • Short training videos hosted on an intranet or similar platform; on how to pull some canned reports (this versus a static FAQ or document) can have a great impact
  • Need to have a well-documented repository of data / reports / dashboards we can easily share with others
• Without the narrative from analysts, allowing end users to pull data for themselves can lead to different interpretations of the data
  • To limit this, there must be some “helper” texts to guide users

Accessible, accurate and up to date documentation plays a critical role in building stakeholder’s trust in your data. The challenge is becoming more complex in an evolving technology landscape with an increasing number of data sources.

Do you have a data documentation strategy? This session, moderated by Sonia Mehta, Director of Analytics at Elevate Labs, the company behind the brain training app Elevate, will focus on how to improve data quality using metadata tools, data documentation processes and naming conventions.

The huddle group will look to answer:

• How do data teams keep documentation up to date?
• What tools do you use for documentation?
• Should non-technical stakeholders have access to data documentation?
• How do you handle event documentation?

If you strive to build trust in an elaborate data environment – this huddle is for you!

• Data documentation can fall into two larger domains
  • Internal data team documentation
  • External stakeholder facing documentation
  • For the former, traceability is key; for the latter, relevance and curation is key. (Our discussion primarily focused on the latter)
• Calendars are a great tool to use to highlight major changes and/or significant events that could impact the data
• Slack is not a great way to communicate data disruptions, data logic changes, or dashboard changes
• We do collect tech debt and having dedicated time to prune/Marie Kondo metrics, remove stale or irrelevant information is critical. Unfortunately, this is usually one of the first tasks to cut when workloads increase but let us fight for the tech debt time!
• Documentation or Wiki ownership is ideally shared across data teams (think: rotational process vs seeking volunteers), though it is important to recognize that some folks may not be strong at this
• Templates can help standardize data documentation across many hands
• Documentation should be intentional
  • It is not always about creating, but rather curating. We should ensure we are surfacing what is actually relevant for the stakeholder
• Dynamic event tracking has helped standardize naming conventions within companies
• A question that came up towards the end of the conversation, what is the MVP of documentation?

Cloud-based technologies have changed the face and pace of business. Fading away are the deep, customized integrations. Self-serve applications are flourishing, and enterprise level SaaS has gained great momentum simply because its flexibility creates the opportunity to build better technology than traditional methodologies. Yet integration challenges have not disappeared completely.

In this huddle, Jason Harris of Fivetran, will engage participants in a discussion around these key questions:

• What are your current key pain points with respect to data integration?
• How are you solving for these challenges?
• What are the use cases for automated vs manual integrations?

Whether you have conquered all your integration demons or not this is the huddle for you. You will emerge from this conversation with new ideas and use cases to boost your data stack efficacy and value to your org.

The digital analytics community is small, but tightknit. How can we use the power of this community to foster inclusivity amongst ourselves and across our organizations? This open discussion, led by ESPN’s Amber Zaharchuk, will be a space to share your experiences and find steps we can all take to be more aware of our own behaviors.

• How can we make the analytics community more inclusive, and extend that to our organizations and other social groups?
• How can members of the analytics community better support each other?
• What have you learned about your own behaviors when it comes to being inclusive? Have you taken steps to change your behaviors?

Companies that act, even in small steps, toward being better and more equitable workplaces will thrive in the long run. This is a rare opportunity to share openly and exchange views on this imperative challenge we all face.

• Diversity, inclusion, and belonging are broad terms, and we need to consider all types of inclusion, race, gender, accessibility, neurodiversity, diversity of thought
• View a diverse workplace as a competitive advantage, because it is
• Build a pipeline of diverse candidates, so when the time comes for you to hire someone or make a recommendation, you already have a pool of talent and you can do what is right, not what is fastest:
  • Ensure not only diverse candidates, but also diverse interviewers
  • Take unconscious bias training
  • Think about non-traditional interview methods
• This industry needs to reconsider job descriptions; they are terribly exclusive:
  • Remove industry-specific knowledge
  • Break down tasks to traits, like intellectual curiosity or growth mindset
  • Make requirements more vague instead of more specific
  • Years of experience are mental shortcuts that we take to exclude people that are perfectly talented
  • Applicants should prepare a skills map in addition to a resume
  • Add statements to encourage applicants, like this one from SEER Interactive:
    • “Sometimes the best opportunities are hidden by self-doubt. We disqualify ourselves before we have the opportunity to be considered. Regardless of where you came from, how you identify, or the path that led you here-- you are welcome. If you read this job description with a belly full of excitement, we are just as excited about you. You’ve gotta apply though :)”
  • Provide a way for potential talent to submit general applications and then match them to positions that best suit them, like the US Digital Service application: https://www.usds.gov/apply
• Make analytics more accessible:
  • Allow fonts in your deliverables to be re-sized
  • Red/yellow/green is the worst combination for color blind readers! Add up/down/flat arrows so that colors are not as critical
  • Use color blind friendly palettes in your visualizations
• Once people are part of your team, provide ways for each person to feel like they belong, tailored to their individual needs:
  • Flatten your orgs (for those that have the power to do that)
  • Set up a buddy system for new hires so that they always feel they have an ally
  • Create networking and mentoring opportunities for senior leaders to meet with under-represented team members
  • Support rotational programs for talented people to expand their skills and understanding of the business
  • Discuss game plans before meetings to help prep people that may need more support
• Be even more aware of diversity of learning styles and accessibility needs when WFH:
  • Provide allowances for accessible WFH setups
  • Create policies on work hours to accommodate family or other personal situations
  • Provide structure with flexibility

The change in the digital world has rapidly accelerated in the last year. Whether your target audience is a customer, patient, employee, citizen, or any other audience, their expectations are different today than they were 12 months ago. Personalization is more important than ever to provide the type of experience that your audience expects.

In this huddle, Roopa Carpenter, Sr Director, Optimization and Josh Morrow, Sr Conversion Optimization Consultant, both at Blast Analytics, will discuss the digital personalization trends that they see for 2021 and how it applies to the companies they work with. These include:

• The increase adoption of AI/ML helping to scale personalization
• Prioritizing customer loyalty to create/maintain a competitive advantage
• Digital personalization strategy as a staple of corporate success

Come share your personalization success stories and challenges and leave better enabled to make 2021 a year of exceeding customer expectations.

In this huddle we looked to tackle the question of how will the increasing adoption of AI and ML help scale personalization?

We first discussed the challenges with leveraging AI:

• Platform infrastructure and not being able to use it
• Ability to scale content (e.g. need legal to review all content which slows down the process)
• Concern for ensuring that predictive analysis does not result in “creepy” personalization
• For teams that are just getting started with personalization, some platforms offer the ability to “dip toe” in the water with AI/ML and can help provide a proof of concept to others in the organization
  • Example - Adobe Target’s activities, such as Auto-Target and Auto-Personalization

Privacy changes – how are we going to prepare for that?
We discussed privacy considerations in the context of personalization

• One option - encourage users to authenticate higher up the funnel
• Consider using a Customer Data Platform (CDP) to stitch user behavior
  • CDP best practices
    • Adopt a crawl, walk, run approach
    • Strategize on initial use cases before getting data into the CDP
    • Understand that not all CDPs are the same. Need to see which one suits your organization’s tech stack

How can we prioritize customer loyalty to maintain or create a competitive advantage?

• We discussed how customer loyalty is the focus for personalization efforts in 2021
  • Pandemic has forced organizations to think beyond the “transactional” and more about the relationship
  • We briefly touched upon how to design and develop habit-based products that can maximize LTV
• Customer Loyalty
  • How can customer loyalty be quantified?
    • Potential option - Cohorts can be created in Google Analytics to see LTV
• Voice of the Customer (VoC)
  • Golden opportunity for organizations to tie VoC back to personalization efforts
  • How can the impact of long-term personalization efforts be measured?
    • Potential option - Net Promoter Score
    • Asking users about a certain feature of the website (customer satisfaction with a specific tool or feature)

How to develop a digital personalization strategy to build success?

•  We discussed key aspects for successful personalization:
  • Buy-in from key stakeholders
  • Break down silos - organization and data silos
• One of the biggest challenges to doing personalization is inertia
  • It takes time, money, resources -- start with quick wins before selling the bigger scenarios/use cases. It requires a balance between too small to have an impact and too large a level of effort
  • When deciding on a proof of concept listen to stakeholders to understand key priorities and metrics
  • Leverage data to identify valuable audience segments (does not always have to be the biggest audience)
  • Opportunity to leverage other technology (Full Story, QuantumMetric) to help inform personalization strategy

• Two types of surges: planned (campaigns) and unplanned (events or news driven)
• Use previous high water marks to prepare for future surges
  • For example, previous daily page view high for CDC was 15 million.  Operationally prepared for a doubling of that to 30 million
  • Was able to successfully serve over 65 million on a single day in 2020
• Use daily metrics monitoring and trend forecasting to monitor traffic and react earlier
• Use and monitor Google Trends and internal search terms for gaps for content that may need to be created or optimized
• Account for past surges when doing year over year or seasonality analyses
• If surges last long enough (e.g. year long pandemic) then traffic will show surges within surges, each triggered by different causes and needing uncovering for the root cause

• COVID has clearly transformed the workplace, but it has also had an impact on hiring practices
  • Remote work is likely here to stay in some form, whether it be full time WFH or a hybrid model, and this means workers are less likely to be constrained by geography
• Hiring for diversity to benefit from varied perspectives is more possible in a virtual process, as there is an opportunity to cast a wider net
  • Tests such as Kolbe and Caliper can be used to assess for style and fit for remote work
  • Data and analytics activities can help provide insight to how the candidate performs when given an assignment
  • Intentional inclusion of underrepresented groups can provide opportunities to diversify teams
• Supporting remote analytics workers requires a personal connection that is typically found in face-to-face environments
  • Opportunities for social interaction need to be built into work meetings to address the need to interact on a human level
  • Can be done by reserving time at the beginning of each meeting, dedicating half of 1-on-1 updates to non-work related topics, or even organizing specific online events to bring people together socially
  • A token of appreciation or "pleasant surprise" delivered directly to someone's home is also a way to show sincere appreciation remotely.
• Innovation and risk-taking can be promoted post-COVID by modeling the desired behaviour in online discussions and communications
  • Online tools enable collaboration and easily building on team members' ideas
  • Periodic retrospectives provide opportunities to reflect on accomplishments as well as ignite improvement initiatives based on what did not work
  • Encouraging and building in time for innovation projects can lead to significant leaps for the organization

Running this huddle was motivated by Caitlin and Alexis’ articles on the subject:

• Linear & Circular Projects Part 1: https://locallyoptimistic.com/post/linear-and-circular-projects-part-1/
• Linear & Circular Projects Part 2: https://locallyoptimistic.com/post/linear-and-circular-projects-part-2/

How can you help manage circular projects?

• Break things down very small - make them “locally linear”
• Set expectations up front - understand timeline, urgency, what metric matters most. Commit to iterative deliverable.
• Just using the phrase enables other teams to understand the issue. It’s so visual - can be very helpful in setting expectations
• When trying to answer an open ended question, outline your hypotheses up front. Identify your first path, try it, if no, come back and come up with next best hypothesis. Follow that trail instead.
• Avoid shiny object syndrome and focus on impact.
• Prioritize regularly and make sure there are consistent check ins. Put some light structure around it with sprint planning.  
  • Having an epic dedicated to open ended work can be helpful. Consistently dedicate time (but limited time) to exploratory work

Identifying up front what is linear and what is circular

• Identification up front is one of the biggest challenges
  • Analysts are eternal optimists. Data is not as clean as I thought, etc.
  • Circularity is sometimes an unexpected detour
    • Assume by default that every data project will be circular if it is the first time you have done it
    • Product and program managers could make a huge difference here. Understanding scope, requirements, etc. Not a core skillset for analysts. However, it can be difficult for a non-data PM to write scopes because they do not have enough context
    • Unknowns are what blows up scope – break out a research phase before you even try to scope. https://erikbern.com/2019/04/15/why-software-projects-take-longer-than-you-think-a-statistical-model.html

Linear tasks are easier to streamline

• Automating those tasks and making time for circular tasks is another good approach
• Make playbooks, documentation and tools so that non-data team members can go down the basic paths themselves. Confluence, drill down dashboards to answer common KPI questions, and pre-built notebooks can be helpful. Keeps data team from becoming “data monkeys” and allows them to focus on new research

What’s good about circular projects?

• Just more fun! This is the interesting part of the job
• Research and exploration can be very fruitful. The goal just has to be going down rabbit holes and coming back with a rabbit - can be very valuable
• A circular path is not a static circle, it is a spiral. You cover more ground than a linear path. Here are some relevant references from HBR:
  • https://hbr.org/2020/11/to-recognize-risks-earlier-invest-in-analytics
  • https://hbr.org/2020/03/whats-the-best-approach-to-data-analytics

Analysis is like an escape room

• You are solving mysteries and puzzles but you have a limited amount of time
• When do you ask the clue master (stakeholder) a question? There are a limited number of times you can ask

How do you balance open ended exploration and needing to meet the tactical, linear needs of the business?

• Exploration can be controversial. Some folks think it’s surfing / waste of time. But it’s creativity! Analysts are often managed by check lists, which limits their ability to explore and their impact.
• We don’t want our teams to be “reporting monkeys,” but the business needs reporting and we need to deliver.
• Find a balance between empowering teams and letting them be creative, and delivering
  • Framework for deciding how to use limited discovery + innovation time based on potential impact: https://towardsdatascience.com/prioritizing-data-science-work-936b3765fd45
  • Framework for balancing open ended work / keeping the lights on: https://locallyoptimistic.com/post/proficiency_v_creativity/
• Periodic Hack Days are helpful to give a creative outlet.
  • Another flavor of hack days: https://betterwithdata.substack.com/p/deepworkfordatateams

How can you deal with high level projects (e.g. OKRs) that are circular?     

• Break out research v. build KRs
  • Sometimes all you can commit to is having a recommendation for the next step – if you do not know it is possible, try not to commit to (and have other projects be dependent on) a build KR
• When the business sets a goal, it forces you into a linear modality
  • Break down projects into bite sized pieces, set timelines on research, cut scope
  •  “You can tell me what to do or you can tell me how long it will take, but you can’t do both.” (Say it with a smile :-))

In this huddle, Jorge provided a short intro presentation into the topic and how Best Buy is tackling this challenge. A roundtable discussion followed the presentation. Some of the key points discussed included:

Principal learnings on building a data enabled culture from Best Buy

• Ensure data/knowledge/information is easy to access for everyone
  • Best Buy have their own self-service video platform, Data University, for any employee to learn about our analytics tools
• Create avenues for people to get support and nurture their analytics knowledge
  • Best Buy created office hours, Analytics Jam, were anyone in the company can get their work reviewed by an analytics professional
• Get your analytics team to show the way on what is possible
  • Best Buy run regular meetings, called Site Club, where stakeholders debate strategic initiatives
    • Key rule -> everything has to be supported by data or it does not count
      • A method to show business units how data should drive decision making

Principal learnings from everyone in the huddle

• Do not force people to learn
  • Cultures where people are forced to learn analytics tend to fail
  • Instead create positive incentives around why they should focus on it: career growth, business impact, etc.
• Make business teams accountable for knowing their KPIs
  • Work with leaders in your org to keep teams accountable for knowing the data of their business units
    • E.g. have business review meetings were leaders ask about results and how decisions were made
• Find influencers/champions in business teams to be examples, not necessarily executives
  • Others will copy this behavior and will start using data to inform their decisions or will emulate their learning habits
• Standardize tools across the organization (as much as possible)
  • Having some teams using Google Analytics and others Adobe Analytics will decrease trust in data as the base for decision making
  • Have one source of truth that everyone can rely on
• Connect business results to business initiatives to get support from Execs
  • Training sessions/courses can be time consuming for your analytics team and business users -> document wins to prove the ROI of this work
    • E.g. if the marketing team designed an A/B test as a results of your training/influence, document its financial impact and broadcast this win

Examples of what great companies are currently doing to create data enabled cultures

• Learning and Development teams focused on teaching analytics across the org
  • In most companies, it is up to analysts to drive a data culture, most of the time as a side project by the side of their desk
  • Having specialized teams focus on analytics training, allows analysts to focus on what they do best, without sacrificing the benefits of having a data driven culture
• Analytics training program for executives.
  • At PBS they built specialized programs targeting executives.
  • Most training initiatives focus on the average employee
  • If you aim to transform your culture, you need to start from the top and get everyone else to follow

Questions for a follow-up huddle

• What is the best way to accurately measure the value and effectiveness of your training initiatives?
  • We know that data enabled cultures drive more business value but how do we know if a specific learning program is better than another?
  • How do you know if your programs are getting better?
  • Traditionally, teams have used surveys and anecdotal evidence, but should we use more robust measuring methods?
  • Similarly to how we A/B test on our sites, should we experiment with our training methods?
    • E.g. give one cohort a specific type of training and compare performance over time

The huddle set the stage with three examples where a lack of intuition and specific statistical skills led to a misinterpretation of data:

• The interpretation of A/B test results: marketers are tempted to see the observed lift between two treatment groups as being an absolute truth that can be extrapolated and annualized, rather than understanding that the observed result falls within a range of variability
• Correlations on non-stationary data: correlating two metrics that are trending in opposite directions and seeing a high correlation rather than converting the trending (non-stationary) data to be stationary metrics using a technique like first difference (and seeing that there is, in actuality, no correlation)
• Causal inference and counterfactuals: recognizing that, even in the absence of an experiment (a test), counterfactuals still exist and can be estimated. So, rather than assessing campaign impact by totaling up all of the revenue that can be linked to the campaign, comparing the total observed revenue to the estimated potential revenue (counterfactual) in the absence of a campaign using techniques such as diff-in-diff, 2-stage least squares, or regression discontinuity

Should marketers increase their statistical intuition? There was no consensus:

• Some delegates felt that this would be trying to "teach statistics to marketers," and that is simply an impossible task. Instead, the marketers should just trust their analysts
• Others felt that analysts themselves actually lack the degree of statistical intuition and skills that they should have (a view the huddle leader shared, but that there was far from universal agreement on), and that needed to be addressed first
• Some delegates felt that this was an area that both analysts and marketers did need to grow, but recognized that there is a massive challenge, in that marketers have been conditioned to think that the data will provide them with an objective truth, and productively addressing that misperception is not a trivial undertaking
• When it came to the discussion of how to provide both analysts and marketers a deeper understanding of statistically-oriented concepts, metaphors ruled the day, and various delegates shared some powerful ones:
• Framing uncertainty in terms of thinking about a weather forecast: the "probability of rain" level at which you do or do not take an umbrella with you when you leave the house varies by person AND by the importance of staying dry during the specific outing
• (If the environment is such that it is okay) Pointing to the predictions from the 2016 Presidential election compared to the results
• Framing the analyst's role as helping the marketer to draw a map and then helping them decide how to get from now to the business outcome they are trying to achieve...but there are multiple routes and the map has missing and smudged pieces on it
• Thinking about confidence intervals as some vomit on a table that has been covered up with varying-sized stickers (the vibe of the huddle was that, while this spontaneously produced metaphor was both illustrative and entertaining, it might not be effective in other environments)

Some final takeaways were:

• Using an external consultant (this was NOT suggested by an external consultant) come in as an authority to do some light education, if well-targeted and positioned, could be one way to take the burden off the in-house analysts to try to educate their stakeholders to shift their ways of thinking regarding data
• "Critical thinking" is something that really needs to be fostered in both analysts and their stakeholders
• Managers of analytics teams need to fight to protect the analysts' time—to ensure there is time to do that critical thinking, to apply the appropriate statistical techniques and methods, and to prepare deliverables to communicate the underlying complexity of the data effectively (without losing their stakeholders)