Agenda

Please note this is a live event; all times are Eastern Standard Time.
We do not record the discussions to ensure delegates are comfortable to share openly and honestly.

Each delegate participates in four huddles (roundtable discussions), two per day. Huddles are 80 minutes long with 24+ huddles to pick from, split into four time slots (round #1 to #4).

Thursday, March 4th – Day 1

All times are EASTERN STANDARD TIME

10:30 am: Pre-Conference Interactive Workshop

Thu, Mar 4
10:30 12:30 am EST

Krissy Tripp
Evolytics

Getting your test read and making a rollout decision was the relatively easy part. How do you ensure you are spending your testing resources on the best experiments that are going to derive meaningful insights? Can you multi-purpose your experiments with machine learning and data science analysis?  
 
In this interactive workshop, led by Krissy Tripp, Director of Decision Science at Evolytics, we will look at methods to further analyze experiments through applied statistical and machine learning methods that drive results.
 
Krissy will cover:

  • Using Evolytics’ Data Science Decision Tree to determine which advanced statistical methods will best serve your post-test analysis
  • How to balance the likelihood of actionable insights with data analytic resources
  • Introduction to using data science for experimentation programs

 
Krissy will walk you through a structured learning approach to experiment takeaways. You will come away with an experimental learning roadmap with prioritized experiments and quick wins for the data science team.

12:30 pm: Networking Lounge
  Meet old colleagues & new peers for spontaneous 1:1 & 1:many conversations in our visual networking lounge.

1 pm: Opening Welcome

1:10 pm: Platinum Keynote

Join Rob Clarke, VP of Product for ObservePoint, and discover how to:

  • Validate the deployment and accuracy of all martech collecting data on your site
  • Optimize user experience across your digital channels
  • Ensure you’re meeting digital privacy regulations for customer data
  • Automate campaign tracking and measurement across the customer journey

1:25 pm: Huddle Discussions Round #1

Each delegate selects and participates in one huddle per round

Round #1
Thu, Mar 4
1:25 - 2:45 pm EST

Chris O'Neill
ObservePoint

The “new normal” is a term that has been flying around social and news channels since the beginning of 2020, but for digital analysts and marketers, a different sense of “new normal” has been pervading the industry for the past few years: privacy compliance.

Web privacy regulations like GDPR, CCPA, and others destined to come along stem from a growing public awareness and concern over their data privacy and protection. Your company needs to take proactive steps to protect your customer’s data and your reputation.

In this huddle, Chris O’Neill, Solutions Architect at ObservePoint, will lead a discussion on privacy best practices and how to make it easier to stay compliant, while still creating fantastic experiences for your customers. We will cover:

  • Using automated testing to discover non-compliant areas of your site
  • Implementing flexible processes and solutions for long-term compliance
  • Adhering to privacy demands while still providing personalized experiences

Delegates will share their own use cases and issues and together we will emerge with practical solutions to several of your biggest privacy challenges.

In this huddle we discussed the current status of privacy. The discussion, perhaps a little unexpectedly, drifted to how different brands perceive privacy legislation and the role of education as a driver for privacy advocacy.

The discussion clearly highlighted the complexity of the topic and why most of us still are not entirely clear about next steps. But it was a thoroughly enjoyable exploration of the topic with all participants agreeing that:
(a) It was a valuable discussion and (b) there is so much more to discuss

The main points covered included:

  • A brief discussion about the impending Canadian privacy legislation (as two of the delegates were Canadian)
    • The expectancy is that the Canadian legislation will be stricter than current US legislation
    • Prompted a discussion about how to handle varying country legislation
      • Do you treat visitors based on their local legislation or do you apply the strictest in each case?
      • Still open for debate given the evolving nature, particularly of the Canadian legislation
  • Brands will treat privacy differently
    • GDPR is designed to protect the European consumer irrespective of the website geo-location
      • Five years on, there still seems to be confusion or uncertainty about some of the practical aspects of GDPR
      • Some North America lawyers have interpreted GDPR differently, consequently privacy settings' implementations differ from brand to brand
    • Brands approach to privacy generally falls into one of three:
      1. Some are taking a very stringent approach to privacy
      2. Some are ignoring it altogether ("wait and see")
      3. Some are developing a "consent acquisition strategy":
        • E.g. privacy settings will depend on the geo-locality so if the user is US-based they will automatically be opted in; if in EU automatically opted out
    • Europe appears more serious about building privacy into everything
    • US government agencies are similar but many brands are still in a bit of wait and see (who get's a fine!)
  • Customer attitude to privacy
    • We generally agreed that customers do not care about privacy so much because they do not understand what all/most consent requests are really about
      • They click on 'no' without necessarily thinking about it - "just get that annoying overlay of my screen" attitude
    • Education
      • In UK, Internet safety and privacy are part of the obligatory curriculum in public schools
        • While teaching methods may vary by school it is perceived that the next generation of kids will be much more privacy savvy
      • But in some contradiction, one US-based delegate shared his experience growing up in 1990s and being taught to be careful sharing any PII online; but now younger adults in US are sharing a lot of PII online publicly, which is a concern
      • Who is going to advocate for privacy? Most market actors don't seem to have the incentive to do so
        • Brands certainly do not have an incentive
        • Governments should but are lobbied by brands/enterprise to minimize legislation
        • Individuals as a whole do not seem to understand it and therefore either ignore it or are almost apathetic to it
        • Yet privacy legislation is moving forward globally so must be championed by someone. Privacy interest groups?
  • Offense vs defence
    • Generally seems like companies prefer taking defensive strategies
    • But there re some brands that are centered around privacy which are offensive (e.g. Hey email service launched by Basecamp in June 2020)
  • Need to differentiate between licensing and privacy regulation
    • Global brands (e.g. Netflix, Spotify, Apple+ etc.) will have territory-specific content rights and licensing agreements
    • The general consensus is that licensing is based on the user's current geo-location but their privacy settings will be governed by where their account was set up or currently domiciled
      • E.g. A US-based customer may consume content abroad when travelling and might be limited in terms of content accessibility but their privacy will be regulated by US regulation irrespective of where they consumed the content
      • This too could differ from one brand to another as currently it seems to be open to interpretation
    • The conclusion was thinking about how brands treat content licensing could help in our approach to how we think about some aspects of data privacy

      • Specifically how we handle different laws in different countries and people changing locations when they access sites

Amber Zaharchuk
Walt Disney

The pandemic brought many organizations’ hiring and promotion efforts to a screeching halt. Less face time with peers and leaders can make it even more challenging to demonstrate capabilities for growth. In this session, Amber Zaharchuk of Walt Disney DTC & Int. will lead a discussion about strategies for building technical and soft skills remotely, keeping our careers trending in the right direction. We will discuss the following questions:

  • What are you doing differently to ensure your work is valued and understood?
  • Have your career goals shifted moving from a physical to virtual office? How so?
  • How are you maintaining relationships with leaders and peers?
  • Has your org adjusted employee evaluations or education/training opportunities?

While the pandemic is temporary (we hope), virtual working is likely to increase in the future, making this an increasingly relevant topic and huddle.

Here are some highlights from both the original (abbreviated) huddle and the re-run a couple of weeks later:   

  • How is remote work different today vs. prior to COVID?
    • People are more accepting and more human, where now it’s ok for dogs to bark and kids to run in
    • Now everyone is in the same situation, so it levels the playing field and makes everyone feel included
    • Expecting some of it to remain after the pandemic, but we need to avoid leaving people out if only some members of the team return to the office
      • The hybrid model may be more effective if entire departments go back or remain remote, not individuals within the same department
    • Extroverts miss working together
    • We discovered trust and learned that people do have a strong desire to do the work, and if they have to deal with personal challenges during the work day, they will work later in the day
      • Some companies previously didn’t allow WFH on Mondays or Fridays due to the perception that employees were really enjoying a long weekend. Still to be seen whether this rule will be removed, but the hope is that companies will see beyond that, given the trust built over the past year
    • We need to avoid a situation where we take the "bad" aspects of the office and adopt them to WFH
  • How do you keep yourself visible to management?
    • Control mechanisms provide transparency to management, and that builds trust
    • In agile environments, contributions should be (by the nature of the methodology) fairly visible via daily scrums and moving Kanban cards
    • Kanban board and project management tools could definitely help
      • But not a good idea to add them just for the sake of visibility
      • Must be part of the work process already
    • Jira, Salesforce, Microsoft Teams were some examples of tools being used to make work visible
    • One delegate who manages employees mentioned how they always used time sheets, but now it plays a much more important role
      • Not for controlling people, but rather as a supportive tool to understand how employees are splitting their time between all essential tasks
      • Now uses time sheets to ensure employees are happy about their time allocation between the various tasks
      • Became an additional employee satisfaction optimization tool
    • Now that everyone is virtual, remote workers can take more ownership of communicating results
      • Making yourself accountable and making yourself known
      • Leaders must help their team members to become better at communicating results and making themselves more visible
    • One delegate targeted 25 people that he values and asked for their advice on how to make himself more visible (but without making it too much about himself; show off without showing off, because if you do not toot your own horn, there is no music!)
      • Created weekly sessions on non-work related topics and invited senior leaders to connect with people on a personal level, and to build trust and familiarity with key stakeholders
      • Targeted some leaders based on topics that would be of interest to them (e.g., sneakers, wine, cooking, dogs, books, happy hour, etc.)
      • Created a shared doc with all the possible topics and suggested to people that they could lead the discussions
      • Ended up running all of the sessions, as it does require a few extroverted people to make it work, but was great success
    • Book recommendation for how to overcome barriers to getting conversations going: Never Eat Alone
    • Continue growth programs for your team
      • Do StrengthsFinder, find out who we are, then who we want to be, then set goals to figure out how we want to get there
    • Use donut intros on Slack to meet new people and build your internal network
  • Have you changed your career interest or path?
    • No one in our group indicated that they were looking to change career path
    • One delegate shared advice she got recently to interview for a position, internally or externally, at least once a year even if you do not intend to move
      • Does not need to be about moving up in your career or more money, but to keep you evaluating your career and keep your resume relevant
  • What are you doing differently to ensure your work is valued and understood?
    • One delegate shared that she now works much more on strategic initiatives
    • Another delegate pointed out that because we are all remote, less work is generated by others for us, so employees might find they have more time to be creative and think of the bigger picture
      • That might tie with people finding more time to focus on creative and possibly strategic initiatives
    • Do things outside of work that you love, because you will work better when your mind is fresh
  • How have you adjusted feedback or employee evaluations?
    • It is really hard to give difficult feedback on Zoom
    • Work on being very explicit with instructions and feedback
    • Some people need more time to adjust to WFH, and that could be slightly frustrating for those that have been WFH for a while
    • Everyone is dealing with different challenges and deals with things in different ways, so we have to be patient and empathetic
    • "Be ruthless in business but nice to people" -> one delegate shared that she had to spend a lot more time speaking to employees about general issues and handling lockdown
    • Book recommendation: Radical Candor
    • Apply your skills to volunteerism through orgs such as Common Impact

Increasingly analysts they are working more closely with data scientists and data engineers who have entered the scene along with large and diverse data sets. Undoubtedly, a welcomed development for analysts but it also presents some challenges.

  • How does this shift change the way that analysts frame their skills and deliverables?
  • How do we go about sharing analysis across different toolsets?
  • Is there a common tool/language all can share?
  • How do we ensure reproducibility and cross coverage (tools and techniques)?

Jose Maldonado of Verizon will lead this huddle aimed at team managers and analysts looking to improve team and personal productivity respectively.

This huddle ran twice – once during the main DA Hub event (truncated session) and as a re-run a couple of weeks later. Given the delegate groups were different and the original truncated huddle still produced a very good discussion, we have chosen to split the summary into two:

Original Huddle Summary:

  • Tools
    • Can be easier to collaborate when being prescriptive on the teams’ choice of tools/programming language. Time saving element to converging on tools
      • This can be difficult with members that are stuck in their ways (R users refusing to code in Python or vice versa)
    • Avoid excluding potential candidates based on tool experience
      • Hire for potential and base skills
    • There are challenges in some orgs when it comes to frameworks for analysis and governance
      • The enterprise is sometimes too focused on data governance rather than the process
  • Soft Skills & Experience
    • How do we optimize for different mindsets; BI / Analyst / Data Scientist?
      • How do we get folks on common ground
        • Book Clubs to unite teams
        • Assign a team member to serve as "connector" across teams
          • Sometimes natural connectors emerge
            • Likely a personality trait (could help with finding connectors)
          • Connectors serve as the "cross function glue"
          • If a role is created to serve this purpose it may not be as impactful as a natural analyst/team member (credibility of connectors)
          • Creating connectors based on greater exposure to other skills/teams
        • Rotational programs can play a part by providing more context and org ecosystems
    • Challenge of team members acquiring skills to become "Data Scientist"
    • Allure of becoming a data scientist can create shortage for the other required roles in the org such as Analyst, Data Engineers, etc.
    • Importance of showing the value other roles add


Re-Run Huddle Summary

  • Team knowledge management
    • One delegate shared his experience with a banking client where they had multiple analytics teams so they set up internships in the various teams to cross-pollinate
    • Silos could be challenging not only in big orgs but small ones too
    • Even if there are common data applications
      • Comes to politics and must be done top to bottom; simply won’t work without senior management buy-in and active support
    • Building relationships are critical to bringing together different analytics professionals and enhancing practitioners' skills
    • A delegate managing a team of analysts and experimentation practitioners initially had the philosophy of trying to keep up the skills and knowledge of all team members
      • However, with time realized it might be better to have different team members specialize in different areas (both technical and domain expertise)
      • Given the growing demand for data and the data tech it is much harder for any one person to be an expert in multiple disciplines
      • So need to actively manage your team's skills based on the organizational needs and the individual's areas of interest
    • Now might be the right time to develop diverse skilled teams given we're not working in a shared physical space
      • Leverage synchronous work to do more pair projects and cross pollinate skills
    • One delegate shared a technique called Mob Coding Sessions (used in Agile development)
  • One delegate shared how Airbnb use R and data science to scale up skills
  • Need to keep ambitious employee engaged
    • But could be a challenge in an org that does not have growth potential for individual contributors
    • A delegate shared the importance to start with employee nurturing as early as they join; much harder to do a year in
      • Uses a 16-types personality assessment (Myers Briggs) during the recruitment process
  • We also discussed the importance of good documentation in a diverse team and in general to improve productivity and cross coverage as members take time off or move on to other opportunities

June Dershewitz
Amazon Music

Super Huddle

Build vs. buy in analytics is no longer a new debate. However, the historical paradigm where build was favored by start-ups and buy by established corporations is changing. The incentives to build are greater today than ever before, yet the challenges are still present and significant.

In this super huddle, four expert practitioners with experience of both build and buy will share their experiences and knowledge on the pros and cons of both routes.
June (Amazon), Pubudu (UpWork), Yael Farkas (Douglas), Scott (Brooklyn Data Co.) and Joe (PBS) will discuss and answer the following questions:

  • What are the business use cases for building an in-house stack?
  • What are the core competencies required for build and for buy?
  • What are the potential pitfalls to consider before embarking on either journey?
  • How do you continue to grow your solution, build or buy, to address changing conditions?

While the four will lead the conversation, delegates will get an equal opportunity to share their views and experiences, ask questions, and hear from others.

Whether you are at the consideration stage, already building your own solution, or in post rollout, this huddle will help shape your thoughts around your next steps.

Laura Guerin
Wayfair

With decreasing attention spans (in this case by our business stakeholders) and increasing need to paint a full picture of customer experience partnering quantitative analytics and qualitative consumer research is no longer an option but a must.

In this huddle, Laura Guerin of Wayfair will lead a conversation on marrying quantitative with qualitative data. We will answer these challenges:

  • Use cases leveraging qualitative data sources
  • How to map qualitative data to align with customer journey insights?
  • Presentation tips and suggestions (relating specifically to combined qual/quant analysis)
  • Strategies for building a cross-functional team in absence of a dedicated function
  • What are the organizational challenges and how to work towards resolving them?

You will come away from this huddle with practical examples of how to improve your customer analysis which in turn will help your org provide customers with better solutions.

So Many Tools:

  • Use the right tool for the situation
    • Quant to understand “what” and be statistically rigorous
    • Qual to understand “why” and get directional insight with low lift
    • usertesting.com for general consumer feedback
    • Surveys, focus groups, and reviews for your customer’s feedback
    • Session replay to validate data-driven hypotheses

Qual then quant:

  • Use research to get in front of customers fast
  • Then test the new customer experience for rigor before rollout (poison cookie test)

Quant then qual:

  • Do the rigorous analysis with data
  • Then emphasize your point using a customer quote to generate internal buy-in

Surveys:

  • When curating a customer survey, the first question should get the customer engaged
  • Make it about them. Be targeted by surveying a specific customer segment and/or specific part of the customer journey

Actionability:

  • Have a clearly defined problem statement and focus on actionability - always

Diplomacy:

  • Be diplomatic when working across teams. Leverage the synergies of quant and qual teams to make an overall bigger impact

Let’s face it, we’ve all been there, staring at senior management across the table with our graphs and charts, stacks of data, finely tuned models and slide rule handy (okay, maybe not the slide rule…). Discussing the numbers and the analysis is relatively easy, but then the discussion goes down a rabbit hole of ambiguous digital metrics.

What does 220 million potential reach for Twitter mean?  What then exactly is a page view?  Are single page visits and a short time on page a good thing or a bad thing?  When is a technical explanation too technical?  How do we communicate upward and downward trends in digital metrics and distinguish those factors out of our organization’s control and those factors in our organization’s control?

In this huddle, run by Fred Smith, Technology Team Lead at CDC, we’ll discuss metrics reporting and communication beyond self-serve dashboards and automated reports and get into the realm of analysis and communication.  We might even touch on aspects of Crisis and Emergency Risk Communication and Digital Storytelling and how they apply to our daily jobs.

  • How do you communicate this to senior executives, in ways that result in accountability and action?
  • How can you establish a culture of learning at the executive level, to continually strengthen this causal understanding?

Expect to come away with a set of ideas for engaging senior executives in the science of cause and effect.

Ole Bahlmann
JustWatch

Many, particularly non-analysts, might consider growth analytics and product analytics as one and the same. Both disciplines examine customer behaviours and share common metrics (e.g. revenue generated, purchase and churn rates, repeat visits and retention rates) but serve different purposes. The main purpose of growth analytics is to analyse the health of the business whereas product analytics is particularly concerned with measuring engagement with products online, informing product development and improving customer product experience.

Ole Bahlmann has vast experience using both disciplines. In this session, we will discuss and debate how to get the most out of product analytics including:

  • Use cases for improving product retention, customer experience and reducing churn
  • What are the essential metrics and KPIs for product analytics?
  • How best to manage the relationship with product team/s?
  • What should your product analytics tool stack look like? Do you need a dedicated product analytics platform?

We welcome practitioners using GA/Adobe Analytics/other for product analysis as well as those using (or looking to use) a dedicated product analytics tool such as Amplitude, Mixpanel or Heap. Expect to come away with a fresh perspective on product analytics.

2:45 pm: Break & Networking
  Meet old colleagues & new peers for spontaneous 1:1 & 1:many conversations in our visual networking lounge.

3:50 pm: Platinum Keynote

Join Blast Analytics for this keynote about analytics maturity and walk away with the knowledge of why analytics maturity matters, five key factors of analytics maturity, and access to a free online assessment.

4:05 pm: Huddle Discussions Round #2

Each delegate selects and participates in one huddle per round

Round #2
Thu, Mar 4
4:05 - 5:25 pm EST

Jordan Peck
Snowplow Analytics

How far are you willing to leap from the comforts of the standard to the benefits (and challenges) of the bespoke? Is an out-of-the-box analytics stack still a viable possibility in today’s demanding analytics environment? Don’t we all operate some hybrid tech stack?

In this huddle, led by Jordan Peck, Solutions Architect as Snowplow, we will share insights and use cases relating to the following questions:

  • Which features/functionality would you change in GA/Adobe/your tool?
  • What are the limitations of packaged solutions? How can these be overcome?
  • What direction are these big players taking their analytics/digital marketing products?
  • What’s the next big innovation in digital analytics?

Whether you at the consideration stage, already looking to build or buy or in post roll out, this discussion will help shape your thoughts around your next steps. Grab some post lunch treats on us (we will send you an Uber Eats voucher prior to the huddle) and together we will walk away with practical ideas on how we can help our own organizations stay ahead in this rapidly changing world.

In this huddle we aimed to answer the following three questions:

  1. Which features/functionality would you change in GA/Adobe/your tool?
    • Challenges with closed source tools
      • Can act as a black box and can then be hard to get data out (through official or non-official methods)
      • Often marketed that you can just "flick a switch" when it rarely is
      • Work well for simple use-cases
      • Challenging pulling the data out and moving it somewhere else
      • Orgs with diverse technical use-cases or complex technical set ups can struggle
      • Do not own the data; often do not have access to the raw data
      • Moving data around multiple different systems
      • Non-retroactive data is painful
      • Often marketing people set up the GA tracking but then data engineers cannot get at the retroactive data
      • Data engineers often do not find GA very intuitive
      • At the end more configuration required
      • You can make your business/product into GA, using arbitrary decision
        • This works so long as the translating documentation is accurate and stays up to date
    • The role of data engineers
      • Engineers are great if they know what they are tracking
      • Does not work as well when engineers are shoe-horning data into a restrictive data model such as GA
      • Giving clarity to those implementing who understand how the product works
      • Should this be addressed by people and process, rather than the tool?
      • This does create extra process, but does make data collection more reliable in the long term
      • Has anyone ever use a home-built solution? Very rare
  2. What direction are these big players taking their analytics/digital marketing products?
    • The big players are trying to tie customers into their ecosystem
    • Adobe has acquire Magento and Marketo but these tools still do not integrate well with the rest of their tech stack
    • Should allow users more access to raw data like GA4 and BigQuery
    • GA4 changing from sessions to events
    • GA3 was a hook to get customers spending on ads; GA4 is to hook users to GCP
    • Big players are likely to embark on new acquisitions
      • Tools such as Indicitive, Amplitude are performing GTM strategies together
      • Their strategy is to claim you can do a lot of this with a small group of tools
      • Similar to Looker, Snowflake, dbt and Fivetran
        • Looker and Snowflake no longer as pally as they used to be since the Google acquisition
      • Some significant privacy concerns rising from such integrations/acquisitions
    • CDPs
      • CDPs probably are not super privacy safe; can be a privacy nightmare
      • Salesforce, Oracle and Adobe are all launching CDPs
      • 1st  party data in CDPs will become increasingly more important
      • Market maturity – shifting from "should we get a CDP" to "what CDP should we get?"
      • One of the participants shared a use case in healthcare where all data had to be anonymised
      • But this is still potentially not enough and data can still be de-anonymised
    • Some delegates indicated they would like to see more self-service from packaged tools
      • Not quite low-code, but customisable across the data stack
      • All tools are quited silo'd
      • Should be easier to configure across the whole stack
    • SaaS tools benefitted when customers did not know how to handle big data, and tools were focused on a single use-case
      • Data mesh? Only a few companies do this currently fewer do it well
      • Large companies acquire a lot tools
      • Need vendor fabric, then data fabric and tech fabric
    • Is transparency and observability going to become more important in web analytics?
      • OS is growing in popularity
      • The open framework is definitely growing in popularity in digital analytics
      • This is very parallel to the tech open stack (VMware, openstack) separating hardware/os/vendors
    • People often forget that a product with no licence fee is rarely free
  3. What is the next big innovation in digital analytics?
    • Is ML the answer?
      • Google are pushing ML into a lot of their products now
      • There are often much simpler, non-ML solutions that will drive as much value
      • It is not silver bullet
    • What does it mean for analytics talent?
      • Do orgs know if they should hire in data scientists or ML engineers or train internally?
      • Therefore, there is more pressure for analysts to be more horizontal and have a wider range of skills
      • More about the problems you are trying to solve
      • A simple solution may well be the best answer
      • Depending on which stakeholder you aim to provide an answer, you will get multiple different answers to whatever problem you are trying to solve
    • Does the proliferation of ML in tools, mean that data scientists are no long so important???
      • There is still the "Black Box" – there need to be explainable models
      • For some industries, interpretability is a must – finance and health
      • Still a long time before we have human data analysts
      • Always will be important to be able to have a conversation with data scientists
      • In the near future environment, architects roles could be in more demand than fully skilled analyst or scientist roles

Aimee Bos
Blast Analytics

The CDP is a technology that continues to gain traction within today’s marketing technology stack of enterprises. With the objective of centralizing customer data and acting on it, the CDP has the potential to make a significant impact. It also has the potential to be a challenge because of its complexity and power.

This huddle is intended for a more intermediate to advanced audience where specific challenges and opportunities will be discussed and explored.

The discussion will be led by Aimee Bos, Sr Director, Analytics Strategy and Jer Tippetts, Sr Analytics Implementation Consultant, both with Blast Analytics. Aimee and Jer have worked with a variety of CDP technologies for companies in multiple industries. They will share their observations of the potential and the challenges they have seen in CDP technologies.

This will be a lively discussion where we will cover these topics:

  • The data readiness required to achieve the full potential of a CDP
  • How to accomplish identity resolution within your CDP
  • The top use cases to drive value out of your CDP

Come to this huddle to learn and share with your peers more advanced topics to take advantage of your CDP.

Presentation Huddle

A sizable shift is happening in the market rates for data talent as companies get more comfortable with remote workers. Coastal companies are actively recruiting data talent and offering coastal market rates for fully remote workers in the Midwest and elsewhere. In response to this market pressure, local companies are also offering substantially higher salaries. The most attractive recruiting targets are data teams who use the same data stack as Silicon Valley technology companies. How is this challenge impacting you?

In this huddle, Mindy Chen, Director of Decision Science at sports analytics platform Hudl, will paint the picture on how the market for data talent is changing. She will encourage delegates to share their views and experiences on how to address the risk of losing team members that are otherwise happy, but who are being actively recruited and offered substantial pay increases that they cannot turn down. We will explore the following questions:

  • How do you keep a pulse on the data talent market?
  • How do you cultivate a talent pipeline and diversify your data team?
  • What is the role of engagement v. money and how do you keep engagement on your data team high?
  • What is your compensation philosophy and retention plan?  How do you get organizational alignment around that plan?

You will leave this huddle better equipped to address this challenge which is bound to impact managers irrespective of the geo location.

In this huddle, Mindy provided a short intro presentation into the topic and how Hudl is tackling this challenge. A roundtable discussion followed the presentation. The key takeaways were:

  • How do you keep a pulse on the data talent market?
    • oKeeping your networking alive
      • Meet/catch up with managers in similar positions to you on a regular basis (at least quarterly)
      • Staying networked with local recruiters
    • Reviewing nationwide salary surveys for data roles
    • Learning from offers that team members get
  • How do you cultivate a talent pipeline and diversify your data team?
    • Managers are not the only ones to cultivate a talent pipeline
    • Team member referrals are a valuable resource for backfilling
      • The upside is referrals are already screened for fit within the org
      • The downside is creating an echo chamber of like hiring like and creating future risk of attrition in groups
  • What is the role of engagement v. money and how do you keep engagement on your data team high?
    • Engagement v. money is specific to the employee
    • It also changes throughout one’s career
    • Managers should create together with the employee a spider diagram of the attributes that currently matter to them
      • Tailor their personal development plan accordingly
    • Managers could also create a spider cumulative spider diagram for their entire team to get an overview at the team level
  • What is your compensation philosophy and retention plan? How do you get organizational alignment around that plan?
    • Budget challenges are always there
    • Other ways to compensate employees without breaking salary bands
      • Professional development / education budgets for training
      • Attending events either as speakers or delegates
      • Granting employees time to work on their own projects or charity projects
    • Organizational alignment depends on decision makers’ understanding of the value of data
      • Need to educate them on the state of the market for data talent
    • There will always be companies willing to pay more for talent

Understanding what works is a marketing challenge as old as marketing itself. Even though we have more data than even with the rise of digital, measuring the effectiveness of various marketing tactics and strategies remains an ongoing challenge.

Cassandra Campbell has faced this challenge many times in her work helping growth teams at Shopify measure and optimize their work.

In this huddle, Casandra will start us off with a discussion about the importance of actionable approaches to measurement. We will then ask:

  • How do you choose which data to collect and which metrics to measure?
  • What role do attribution models play in understanding what works and doesn’t work?
  • What tactics do you use to go a layer deeper to find truly actionable insights?

Join us for a lively discussion about the best approaches to measuring marketing and understanding what works, and walk away with actionable ideas and tactics to help level up your own organization’s effectiveness.

In this huddle we set out to answer the following three questions:

  1. How do you choose which data to collect and which metrics to measure?
    • Start by defining the goal or working with teams to understand their goals
    • Define the questions that need to be answered
    • Teams often are not aware of what data is being collected or what the limitations and considerations are for collecting data
    • It is helpful to educate internal teams on data collection requirements
    • It is important to think about whether the juice is worth the squeeze with measurement
    • Ask “why are we doing this?” before making big investments in data collection and analysis
  2. What role do attribution models play in understanding what works and does not work?
    • Many companies are interested in explorating algorithmic attribution models
    • Many analysts would like to improve their companies attribution model but struggle to get buy-in
    • It is important to show an example of how improving attribution could improve the marketing team’s effectiveness
    • Often trying out different attribution models show similar results because teams have tailored their tactics to the current attribution model
    • The true benefit of more sophisticated attribution models comes after teams emboldened to start adapting their tactics
    • Many analysts would like to expand attribution beyond acquisition channel and think more about how to attribute impact to different components of their websites
  3. What tactics do you use to go a layer deeper to find truly actionable insights?
    • Controlled experiments are a great way to go beyond attribution and uncover what is truly driving incremental growth
    • Controlled experiments also work well to understand which parts of the website contribute the most benefit
    • When it comes to understanding what works offsite, geo experiments can be a good option. Methodologies include Google’s:
    • Segmenting data is another very effective way to go a layer deeper. We discussed some examples
    • Sometimes teams need help defining segments. We discussed how to do that
      • While there is usually a standard way of segmenting, exploring ad hoc segments can be illuminating as well
    • Look for 80/20 opportunities where one channel, part of the website, or other segment accounts for a disproportionate amount of revenue or engagement

Gary Angel
Digital Mortar

Most actual analytics is about proving/disproving hypotheses not data exploration. Sometimes, those hypotheses get dumped in your lap by a curious business partner – but what if they do not?

In this huddle, led by Gary Angel, we will explore techniques for hypothesis generation considering specific types of research (VoC, Behavioral, Ecometric, etc.), hypothesis generating analytic techniques, creating formal research programs, organizational crowdsourcing techniques and even cognitive tips/tricks.

This huddle aims to help participants develop a more conversant approach to hypothesis generation which in turn would serve to educate business stakeholder on how to pose better hypotheses back to the analytics team.

  • The huddle focused on the when in the analysis process hypotheses are generated, techniques for generating them, and the extent to which analytics and data science can be usefully compared to science and can/do borrow from the Scientific Method.
  1. When are hypotheses generated
    • Participants saw numerous cases of hypotheses being generated at both start of analysis and during data exploration. Neither method was clearly preferable.
    • Many analysis projects start with a business question or embody an initial business hypothesis
    • Other times, a data anomaly is detected or a broad exploratory analysis reveals potential issues that then serve to form the basis of hypothesis generation
  2. Techniques for Hypothesis Generation
    • When driven by the business, it was often the case that hypotheses were generated by “subject matter experts” – gut feel. In this instance, there appeared to be no significant reason to discount that type of hypothesis
    • A number of use-cases explored hypothesis generation from VoC and operational reporting or alerting
      • Some companies allow field/call-center agents to surface potential issues or hypotheses about customer problems that then become analytic problems
      • Similar use was made of VoC – particularly open-ended questions from which summary categorizations could be derived
  3. Science and Method
    • Some companies focus heavily on pre-generation of a hypothesis tree – a grouping of all interesting hypothesis for research
      • This is generally part of an integrated experimentation program
      • By pre building the tree, it’s possible to target appropriate resolutions to hypotheses – so experimentation, VoC or analytics might each be used to find a resolution/answer
    • One of the drawbacks to using the scientific method is that it relies on a research program where integrated hypotheses can be tested together and verified or refuted
      • Most business analytics problems don’t allow for as careful experimentation and don’t allow for replicability
      • This often creates either garden of many paths situations or strong confirmation bias situations
        • Both are likely unavoidable in many situations
      • The idea of grouping hypotheses into a “tree” provides a way to potentially come closer to a true scientific method since it avoids some of the problems of trying to establish or refute a single hypothesis
  4. Using more formal hypothesis generation
    • Participants felt that more attention to training analysts would yield two kinds of benefits:
      • Young analysts need to be trained to think about hypothesis – to question the data not simply report it. Adding in formal hypothesis generation steps might be useful
      • Getting more experienced analysts to focus on rejecting the null hypothesis as a part of the analytic process

Guido X Jansen
CRO.CAFE

Super Huddle

Culture is critical to business success. At the micro level experimentation is focused on incremental improvement; at the macro level it can serve as a mechanism for sweeping organizational change. Consequently, the success of an experimentation program often hinges on creating/having a constructive culture around it.

In this huddle, moderated by Guido Jansen, host of the CRO.CAFE podcast and joined by experimentation experts Ton Wesseling, Kelly Wortham and Lukas Vermeer, we will discuss what makes for a great experimentation culture and how you can achieve it.

While the four will lead the conversation, delegates will get an equal opportunity to share their views and experiences, ask questions, and hear from others.

Guido will open the huddle with a survey of suggested topics to cover and we will proceed based on delegates’ interest as you voted for it.

Expect to come away with tangible examples of what to do (and what not to do) from companies that have developed a great experimentation culture.

Joy Wendelken
Johnson & Johnson

Standardized data is the key to scaling analytics. But standardization is challenging, particularly in a large organization (think prioritization, multiple analytics teams and potentially independent business units).

In this huddle, Joy Wendelken, Sr. Manager Omnichannel Operations at The Janssen Pharmaceutical Companies of Johnson & Johnson, will share her experiences of standardizing digital analytics implementations across the organization. She will encourage delegates to discuss and answer the following questions:

  • What challenges do you face when standardizing digital analytics across your organization?
  • What digital tools have you implemented to solve for discrepancies in digital data collection?
  • What roles or team members do you currently have in place to implement a broad digital analytics practice?

Come share your analytics standardization examples and learn valuable lessons from other leading organizations on how to make the most from their implementations.

In this huddle we set out to answer the following three questions:

  1. What challenges do you face when standardizing digital analytics across your organization?
    • Receiving buy-in from marketing partners outside of your department and external digital marketing agencies can be challenging
    • When the initiative comes from leadership, it becomes more valuable
    • There can be push back from employees who have invested time developing an analytics methodology that is going away
    • Some have developed ways to gamify the process – making it a competition from site owners/marketers to adopt the new standards
    • Proper communication to marketing partners and external agencies along with integrating them into the new ways of working can help with adoption
  2. What digital tools have you implemented to solve for discrepancies in digital data collection?
    • While most attendees use Google Analytics and Adobe Analytics for implementation, there are several tools that are used to support the process. These included:
      • ObservePoint for monitoring tags and tagging governance
      • Internally developed tools for integrating of tags documentation to updates to the websites
      • Documentation is key to ensuring process is maintained and can be shared across org and with partner agencies
      • Confluence to document all analytics request and integrate into the tagging process seamlessly
      • Cypress.io for documentation linked to tagging implementation
      • One organization used an internal wiki to create a data dictionary and implementation detail repository:
        • Supported by an initiative to gamify the process among employees to regularly keep the wiki up to date
  3. What roles or team members do you currently have in place to implement a broad digital analytics practice?
    • Some attendees rely solely on internal employees to train, implement and analyze the data
    • Other employees rely on agency support to implement while an internal team focuses on strategy and socialization
    • It was agreed that proper communication and documentation were key for a successful analytics implementation strategy

Digital analytics for performance reporting is well-tread territory. However, collecting and analyzing digital data for the purposes of triggering and powering the next best experience for your customers and prospects is a completely different use case that presents its own set of unique challenges.

In this huddle, led by GSK’s Head of Customer Experience Solutions, Rusty Rahmer, we will share practices and insights re:

  • Overcoming the organizational challenges of this new data orientation
  • Techniques for customer journey mapping, defining high value actions, scoring visitor engagement, content analysis, manual and AI-based approaches to modeling
  • The potential differences in digital analytics talent and competencies for achieving success

You will acquire a host of fresh ideas to help you drive real improvements in customer experience.

Top takeaways from this huddle included:

  • How are CX teams different that digital analytics or optimization teams?
    • At their core, they are very much the same, but CX teams add experiential multichannel human centered design and the inclusion of brand impression (i.e. trustworthy, credible, thought leader, etc.) goals to the analytics and optimization KPIs that are more typically conversion, completion, etc..
    • “Activation Team” is an emerging title for those working in the analytics to CX activation realm
  • The struggle to pivot from using data to produce purely “nouns” analyses, dashboards, segments to activating data as a “verb” has challenges at several stages in the live cycle:
    • Data Capture:
      • CX data capture relies heavily on “who and the what, so visitor identification, “events”, and meta data about content and product interactions (i.e. merchandising your digital content) is the primary currency
    • Assembly and relationships:
      • CX requires the bringing together of data from multiple sources (web, in-store, etc.) and orients the data to the customer first, then the activity (who-what), a slightly different north star then traditional activity centered analytics (i.e. checkout  process)(what-who)
      • It also requires, particularly in B2B environments, a healthy solution to “relationships” that customers belong to
    • Analysis and Opportunity:
      • Translating the data into what stage of the journey this customer is in and what the opportunity to help them move forward is, is a significant task requiring very different (and often AI/ML based) solutions to solve
    • Content:
      • A strategy involving four customer segments x a four–journey stages journey is 16 unique pieces of content and business rules for orchestration
      • Make that 8 segments and 8 stages and you now have 32 pieces of content and business rules to create, deliver, manage, and monitor
      • The cost of ownership is high here, important for business to be aware
    • Delivery:
      • Real time integration into your omni-channel solution to experience delivery engine is where all the rubber hits the road
  • Atomization of teams into functional silos (Analytics Team, Content Team, Marketing Team, Merchandising Team, etc.) is a significant barrier to activation
    • Pods, Labs, and Agile Teams help tear down the walls between stages of the process, reduce resource and prioritization overhead, and bring practical utility to everyone’s work by seeing the full, end to end, value chain of idea to delivery
  • Voice of Customer and session replay tools are critical in understanding and evaluating the customer’s experience in CX
  • Interesting debate... as IoT continues to add devices to the CX mix, is all of that data “digital analytics” and the role of the digital analyst to analyze?
    • Is that where we are headed?
    • Most participants thought so and highlighted BI as customer master record work and processing and crafting of the interactions and behaviors across channels as the work of the digital analysts!

5:25 pm: Summary & Prizes

5:30 pm: Happy Hour & Networking

  Meet old colleagues & new peers for spontaneous 1:1 & 1:many conversations in our visual networking lounge.

Friday, March 5th – Day 2

All times are EASTERN STANDARD TIME

10:30 am: Pre-Conference Interactive Workshop

Fri, Mar 5
10:30 – 12:30 am EST

Chase Porter
Evolytics

Choosing a marketing or analytics tool can be an agonizing decision. The potential cost and risk of choosing the wrong tool can be paralyzing. The potential benefits and return on investment of choosing the right tool can be exponential.

There are dozens of options to consider.  Every tool claims to do everything that every other tool does, except more and better! How does a marketer or analyst successfully navigate vendor jargon and marketing fluff to choose the tool that best meets stakeholder needs and best integrates with the existing technology stack?

In this interactive workshop, Chase Porter, Director of Data Operations at Evolytics will use a client case study to demonstrate how to successfully apply an evaluation framework and scorecard to assess features and choose the best-fit option.

Delegates will learn how to:

  • Narrow down dozens of technology options to a few
  • Identify and prioritize key decision criteria
  • Objectively assess technologies for best fit using an evaluation framework and scorecard
  • Minimize stakeholder bias and reach consensus on technology selection
  • Choose technologies that meet stakeholder needs and best integrate into the existing technology stack

This workshop will provide you a structured framework approach to vendor selection leading to higher ROI on your tech stack.

12:30 pm: Networking Lounge
  Meet old colleagues & new peers for spontaneous 1:1 & 1:many conversations in our visual networking lounge.

1 pm: Opening Welcome

1:10 pm: Platinum Keynote

Countless solutions have been touted Self-service Analytics and Business Intelligence capabilities for years, but self-service for whom and what exactly? And if your self-service tool is so great, then why are you still pulling data out of it and into spreadsheets?

The BI process of the past 15 years can no longer support the demands of the always-on, data-driven world we now live in. The insatiable hunger for real time data from every corner of an organization is just too great. Expectations for data-driven decision-making, the sheer increase in data volume, and the advent of cloud data warehouses have all paved the way for a new need: access to granular, row-level data. It is at this level of detail that businesses find competitive advantage today.

In this keynote from Sigma Computing Evangelist Jon Avrach, he will discuss the inflection point at which BI teams now find themselves and why it’s time to redefine Self-service Analytics.

1:25 pm: Huddle Discussions - Round #3

Each delegate selects and participates in one discussion per round

Round #3
Fri, Mar 5
1:25 - 2:45 pm EST

Jon Avrach
Sigma Computing

While we have yet to discover all of the ways that COVID-19 has impacted organizations and will continue to affect them for years to come, there are two things we know for certain: it dramatically accelerated cloud technology adoption and put a spotlight on all of the BI dashboard’s shortcomings. Dashboards that merely display a predefined set of metrics are no longer enough. From decision-makers in the C-suite to marketers to accounts payable managers, everyone in the organization needs the ability to explore data, uncover critical answers, and make decisions on demand.

In this roundtable led by Sigma Computing Evangelist Jon Avrach, we will discuss:

  • The failure of self-service BI and why we are still analyzing data in spreadsheets
  • The decline of the dashboard and why empowering business teams to explore data benefits the entire business
  • Transitioning from a “read-only” BI mentality to interacting with data in a way that reveals untapped opportunities and potential outcomes

Come ready to share ideas for developing a future-proof data strategy that empowers non-technical teams and increases the value analysts bring to organizations.

Presentation Huddle

Retaining analytics talent is a challenge. Churn happens, people move on, new projects come up. With demand for analytics practitioners at an all-time high, hiring occupies a significant part of the analytics management role.

So how do you build a talent acquisition strategy that supports the everchanging needs of the business? Without paying for bored talent?  How do you nurture internal talent to help support your organization’s analytics needs – as they grow?

In this huddle, Melissa Shusterman, Global Marketing Measurement Lead at Vanguard, will share Vanguard’s approach to talent acquisition and development strategies, and open the huddle for delegates to comment and ask as well as share the strategies that have worked for them.

Get practical advice from peer managers to help you improve your analytics talent pipeline.

We have discussed a variety of relevant subjects and topics during this huddle.
Below are some of the practical outcomes shared:

  • Ensure you have flexible staffing models.  Vacancies will happen – and should happen if we are helping our folks grow.  We need to be prepared.  For in-house teams this includes:
    • Partnering with an agency who can help when you have immediate needs
    • Over hiring, not waiting till you have a full person’s work load before bringing some one new in
    • Training up – find ways to grow talent internally, before you need them
  • When you are recruiting, consider people who have taken non-traditional routes into the field including:
    • People transitioning from Direct Mail or other non-digital but data heavy marketing roles
    • People who have augmented their education through certifications and online training
    • Look for interest, talent and commitment
  • How do you look for interest, talent & commitment?  Here are some interview questions that may help:
    • Tell me about the last time you looked up and several hours had gone by?
    • What is your favorite data set?
    • What analysis are you most proud of?
    • Tell me the story behind this chart (draw a random chart)?
    • What is your favorite game?

Aurélie Pols
Aurélie Pols & Associates

Super Huddle

From "let's collect everything" to risk abiding (digital) data management – the sentiment on the digital market is changing. What are the implications of this change on marketing analytics? How are you adapting to the new normal?

In this huddle, led by privacy expert Aurélie Pols, Andrea Mestriner, Global Head of Analytics at luxury etailor Yoox Net a Porter and Gary George, Product Manager for Privacy & Data Governance at Indeed, we will look to bring clarity on the moving pieces within regulatory obligations as well as the stances taken by actors such as Apple, Google, Facebook and the IAB (impacting our ability to track the customer journey).

We aim to populate the Privacy Risk Assessment Matrix below to help you determine the impact of recent privacy changes for your organization and to help you communicate the challenge and potential solutions clearly to your internal stakeholders.

We will first answer the following questions for Apple, Google, Facebook and IAB (and others if identified as key to the discussion):

  • Who is impacted?
  • What are the risks?
  • Who are they catering to?

Once we understand how and why those actors made certain choices, we will be able to answer:

  • What are the risks and consequences to your business?
  • What mitigate measures are available to you?
  • What is the estimated data impact to your business?

Delegates should walk away with a complete view of the challenge and the ability to rationalizing the threats and possible solutions internally in your business.

Super Huddle

Background (if you get a chance to read):
European data protection legislation has always required a lawful basis to process personal data. The GDPR broadens the scope of application, moving way beyond the US concept of PII now also embraced within the CCPA/CPRA. When data is written and/or read on a device, not only does the GDPR apply for EU residents but ePrivacy as well. Yet ePrivacy, unlike the GDPR, only includes one legal basis for processing personal data, which is consent. Valid consent is then defined within the GDPR and is increasingly specified through rulings such as Planet49.

How and where personal data is, therefore, processed has implications on the chosen lawful basis. Indeed, according to French supervisory authority CNIL, server-to-server data flows might not fall under ePrivacy and therefore potentially add legitimate interest as well as contract to the lawful basis for data processing.

Additionally reading:

  1. An American's guide to the GDPR https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3620198 (a bit legal, sorry)
  2. Market actors:

Super Huddle

Growing and developing analyst talent continues to be an important agenda item on every analytics leader’s agenda. Training and resources for the tools are abound. But how do you turn a junior analyst into a rock star senior analyst?

Amy (PBS), Ashish (LPL Finance), Sharon (BMO) and Rusty (GSK) have led analytics teams through continuous changes. In this super huddle, we are going to compare approaches and experiences for developing the skills and savvy we need from our analysts. We will ask:

  • What are the key skills to develop in analysts and data engineers?
  • What mentoring techniques are being used to help them?
  • How to we become better mentors to our team members?
  • How do we encourage analysts to produce more meaningful and actionable insights, improve storytelling and data visualization, and develop soft skills such as persuasion, influence, professional presence?

Join this huddle to find out how other businesses help analysts hone their skills, increase their motivation and keep them motivated.

Note: As in any huddle, you will get to share your views and experiences, ask questions, and hear from your peers.

We have discussed a variety of relevant subjects and topics during this huddle.
Below are some of the practical outcomes shared:

  • Quarantine has made mentoring much more difficult
    • Connecting with staff has become much harder
    • Development often gets pushed to the side
    • Replicating the camaraderie of the office takes extra work and different tools
  • It is helpful to divide work check-ins and development into separate meetings
    • Virtual walk and talks have become a useful tool
  • Soft skills are tougher skills to mentor, and even harder in a virtual sense
    • Advocating, managing up, and providing feedback were some of the most difficult but most important
  • Patience is becoming an issue for younger staff
    • They are often frustrated when they do not see change happen fast enough
    • Helping staff learn and manage patience will be important in their career development
  • Helping analysts get out of their chairs and better understand the business and its problems is another important area of mentorship
    • We discussed pairing the analyst with business stakeholders and subject matter experts to learn both broad and deep contexts
  • Take the time to both reflect on progress and celebrate accomplishments. Both build confidence in the analyst
  • Resources: The Coaching Habit, Situational Leadership

Michael Helbling
Stacked Analytics

It is a bit cliché to say that 2020 was unprecedented. As we have all grappled with the impacts of the global pandemic there are shifts happening in eCommerce that are worth noticing. Useful and effective paradigms for understanding customer acquisition and conversion were already undergoing a shift due to the changing regulatory landscape and browser privacy settings.

In this huddle, Michael Helbling will lead a discussion around how we are adapting to the unexpected shifts in channels, how customer preference shifts and the closing of most retail locations impacted strategies, and how the regulatory and browser changes have necessitated shifts in tactics. We will also examine how the pandemic has impacted predictability and how we are managing analytics through those challenges.

Besides being able to just vent a little bit about what this past year has looked like in eCommerce, delegates will gain shared knowledge on tactics and tricks that will be immediately applicable heading into what will definitely be an interesting holiday shopping season.

Asif Rahman
Accuweather

“Every day, people and organisations produce approximately 2.5bn GB of data.” It is nearly impossible for any data team to keep up with the pace of catering to the data hungry and the insights starved product owners, marketers and executives among others. How can we promote data democratization without feeling like we are giving away our jobs?

In this huddle, led by Asif Rahman of AccuWeather, we will answer the following questions:

  • What makes a successful self-service analytics program?
  • How can we reduce data bottlenecks?
  • How does your self-service stack look like?

Come share your analytics self-service experiences and get to hear from other leading practitioners theirs. Together we will come away with practical ideas on improving self-serve analytics.

We had a constructive discussion about the challenges of data democratization and self-service in analytics. Delegates shared both frustrations and use cases of successful self-service.

Some key points included:

  • Before we can start to think about sharing our data widely, we have to ensure we (the analysts) trust the data 100% and its validated
  • Democratizing access to “certain” data allows analysts to focus on what they “want to do” rather than answering ad-hoc questions all day (“what they have to do”)
    • Democratizing data should not come off as a threat to the analysts; leaders have to ensure they are communicating the value so there is wider adoption
  • How to enable self-service
    • Short training videos hosted on an intranet or similar platform; on how to pull some canned reports (this versus a static FAQ or document) can have a great impact
    • Need to have a well-documented repository of data / reports / dashboards we can easily share with others
  • Without the narrative from analysts, allowing end users to pull data for themselves can lead to different interpretations of the data
    • To limit this, there must be some “helper” texts to guide users

Sonia Mehta
Elevate Labs

Accessible, accurate and up to date documentation plays a critical role in building stakeholder’s trust in your data. The challenge is becoming more complex in an evolving technology landscape with an increasing number of data sources.

Do you have a data documentation strategy? This session, moderated by Sonia Mehta, Director of Analytics at Elevate Labs, the company behind the brain training app Elevate, will focus on how to improve data quality using metadata tools, data documentation processes and naming conventions.

The huddle group will look to answer:

  • How do data teams keep documentation up to date?
  • What tools do you use for documentation?
  • Should non-technical stakeholders have access to data documentation?
  • How do you handle event documentation?

If you strive to build trust in an elaborate data environment – this huddle is for you!

  • Data documentation can fall into two larger domains
    • Internal data team documentation
    • External stakeholder facing documentation
    • For the former, traceability is key; for the latter, relevance and curation is key. (Our discussion primarily focused on the latter)
  • Calendars are a great tool to use to highlight major changes and/or significant events that could impact the data
  • Slack is not a great way to communicate data disruptions, data logic changes, or dashboard changes
  • We do collect tech debt and having dedicated time to prune/Marie Kondo metrics, remove stale or irrelevant information is critical. Unfortunately, this is usually one of the first tasks to cut when workloads increase but let us fight for the tech debt time!
  • Documentation or Wiki ownership is ideally shared across data teams (think: rotational process vs seeking volunteers), though it is important to recognize that some folks may not be strong at this
  • Templates can help standardize data documentation across many hands
  • Documentation should be intentional
    • It is not always about creating, but rather curating. We should ensure we are surfacing what is actually relevant for the stakeholder
  • Dynamic event tracking has helped standardize naming conventions within companies
  • A question that came up towards the end of the conversation, what is the MVP of documentation?
2:45 pm: Break & Networking
  Meet old colleagues & new peers for spontaneous 1:1 & 1:many conversations in our visual networking lounge.
3:50 pm: Platinum Keynote

Anyone who has ever managed or tried to measure a marketing campaign knows the frustrations of marrying all your data sources together to see a complete picture of your successes and failures.

Add in multiple channels, questions from various stakeholders, and any sense of urgency, and you quickly realize just how outdated and, frankly, dreadful such analysis can be.

But there is a better way.

Join Cat Origitano, PhD, Director of Product Marketing at Fivetran as she talks about how to evolve your marketing analytics bringing out of individual platforms or excel sheets - and the stone age - and into the 21st century.

4:05 pm: Huddle Discussions - Round #4

Each delegate selects and participates in one discussion per round

Round #4
Fri, Mar 5
4:05 - 5:25 pm EST

Jason Harris
Fivetran

Cloud-based technologies have changed the face and pace of business. Fading away are the deep, customized integrations. Self-serve applications are flourishing, and enterprise level SaaS has gained great momentum simply because its flexibility creates the opportunity to build better technology than traditional methodologies. Yet integration challenges have not disappeared completely.

In this huddle, Jason Harris of Fivetran, will engage participants in a discussion around these key questions:

  • What are your current key pain points with respect to data integration?
  • How are you solving for these challenges?
  • What are the use cases for automated vs manual integrations?

Whether you have conquered all your integration demons or not this is the huddle for you. You will emerge from this conversation with new ideas and use cases to boost your data stack efficacy and value to your org.

Amber Zaharchuk
Walt Disney

The digital analytics community is small, but tightknit. How can we use the power of this community to foster inclusivity amongst ourselves and across our organizations? This open discussion, led by ESPN’s Amber Zaharchuk, will be a space to share your experiences and find steps we can all take to be more aware of our own behaviors.

  • How can we make the analytics community more inclusive, and extend that to our organizations and other social groups?
  • How can members of the analytics community better support each other?
  • What have you learned about your own behaviors when it comes to being inclusive? Have you taken steps to change your behaviors?

Companies that act, even in small steps, toward being better and more equitable workplaces will thrive in the long run. This is a rare opportunity to share openly and exchange views on this imperative challenge we all face.

  • Diversity, inclusion, and belonging are broad terms, and we need to consider all types of inclusion, race, gender, accessibility, neurodiversity, diversity of thought
  • View a diverse workplace as a competitive advantage, because it is
  • Build a pipeline of diverse candidates, so when the time comes for you to hire someone or make a recommendation, you already have a pool of talent and you can do what is right, not what is fastest:
    • Ensure not only diverse candidates, but also diverse interviewers
    • Take unconscious bias training
    • Think about non-traditional interview methods
  • This industry needs to reconsider job descriptions; they are terribly exclusive:
    • Remove industry-specific knowledge
    • Break down tasks to traits, like intellectual curiosity or growth mindset
    • Make requirements more vague instead of more specific
    • Years of experience are mental shortcuts that we take to exclude people that are perfectly talented
    • Applicants should prepare a skills map in addition to a resume
    • Add statements to encourage applicants, like this one from SEER Interactive:
      • “Sometimes the best opportunities are hidden by self-doubt. We disqualify ourselves before we have the opportunity to be considered. Regardless of where you came from, how you identify, or the path that led you here-- you are welcome. If you read this job description with a belly full of excitement, we are just as excited about you. You’ve gotta apply though :)”
    • Provide a way for potential talent to submit general applications and then match them to positions that best suit them, like the US Digital Service application: https://www.usds.gov/apply
  • Make analytics more accessible:
    • Allow fonts in your deliverables to be re-sized
    • Red/yellow/green is the worst combination for color blind readers! Add up/down/flat arrows so that colors are not as critical
    • Use color blind friendly palettes in your visualizations
  • Once people are part of your team, provide ways for each person to feel like they belong, tailored to their individual needs:
    • Flatten your orgs (for those that have the power to do that)
    • Set up a buddy system for new hires so that they always feel they have an ally
    • Create networking and mentoring opportunities for senior leaders to meet with under-represented team members
    • Support rotational programs for talented people to expand their skills and understanding of the business
    • Discuss game plans before meetings to help prep people that may need more support
  • Be even more aware of diversity of learning styles and accessibility needs when WFH:
    • Provide allowances for accessible WFH setups
    • Create policies on work hours to accommodate family or other personal situations
    • Provide structure with flexibility

Roopa Carpenter
Blast Analytics

The change in the digital world has rapidly accelerated in the last year. Whether your target audience is a customer, patient, employee, citizen, or any other audience, their expectations are different today than they were 12 months ago. Personalization is more important than ever to provide the type of experience that your audience expects.

In this huddle, Roopa Carpenter, Sr Director, Optimization and Josh Morrow, Sr Conversion Optimization Consultant, both at Blast Analytics, will discuss the digital personalization trends that they see for 2021 and how it applies to the companies they work with. These include:

  • The increase adoption of AI/ML helping to scale personalization
  • Prioritizing customer loyalty to create/maintain a competitive advantage
  • Digital personalization strategy as a staple of corporate success

Come share your personalization success stories and challenges and leave better enabled to make 2021 a year of exceeding customer expectations.

In this huddle we looked to tackle the question of how will the increasing adoption of AI and ML help scale personalization?


We first discussed the challenges with leveraging AI:

  • Platform infrastructure and not being able to use it
  • Ability to scale content (e.g. need legal to review all content which slows down the process)
  • Concern for ensuring that predictive analysis does not result in “creepy” personalization
  • For teams that are just getting started with personalization, some platforms offer the ability to “dip toe” in the water with AI/ML and can help provide a proof of concept to others in the organization
    • Example - Adobe Target’s activities, such as Auto-Target and Auto-Personalization


Privacy changes – how are we going to prepare for that?
We discussed privacy considerations in the context of personalization

  • One option - encourage users to authenticate higher up the funnel
  • Consider using a Customer Data Platform (CDP) to stitch user behavior
    • CDP best practices
      • Adopt a crawl, walk, run approach
      • Strategize on initial use cases before getting data into the CDP
      • Understand that not all CDPs are the same. Need to see which one suits your organization’s tech stack


How can we prioritize customer loyalty to maintain or create a competitive advantage?

  • We discussed how customer loyalty is the focus for personalization efforts in 2021
    • Pandemic has forced organizations to think beyond the “transactional” and more about the relationship
    • We briefly touched upon how to design and develop habit-based products that can maximize LTV
  • Customer Loyalty
    • How can customer loyalty be quantified?
      • Potential option - Cohorts can be created in Google Analytics to see LTV
  • Voice of the Customer (VoC)
    • Golden opportunity for organizations to tie VoC back to personalization efforts
    • How can the impact of long-term personalization efforts be measured?
      • Potential option - Net Promoter Score
      • Asking users about a certain feature of the website (customer satisfaction with a specific tool or feature)


How to develop a digital personalization strategy to build success?

  •  We discussed key aspects for successful personalization:
    • Buy-in from key stakeholders
    • Break down silos - organization and data silos
  • One of the biggest challenges to doing personalization is inertia
    • It takes time, money, resources -- start with quick wins before selling the bigger scenarios/use cases. It requires a balance between too small to have an impact and too large a level of effort
    • When deciding on a proof of concept listen to stakeholders to understand key priorities and metrics
    • Leverage data to identify valuable audience segments (does not always have to be the biggest audience)
    • Opportunity to leverage other technology (Full Story, QuantumMetric) to help inform personalization strategy

To be fair, not every organization is like the US Centers for Disease Control & Prevention (CDC) and routinely go through Ebola, Zika, H1N1 or COVID-19 level surges in digital traffic.  However, most organizations do experience surges in digital traffic whether from unplanned events and news or from successful marketing and outreach campaigns.

Metrics help us prepare for planned and unplanned surges and they also can be used to improve responses to surges.  Operations, customer experience improvements, content creation and recommendations and other aspects of responses to surges depend on data.

Join the surge to this huddle to discuss how metrics can be used before and during surges, discuss ways that visitor behaviour changes during a planned or unplanned surge and how to capitalize on the increased traffic to gain more repeat visitors. Discuss how CDC handled a year’s worth of traffic in a single month and the many impacts it had on infrastructure, content, visitor experience, contracts and resources, and coordination with partners.

As an added bonus, we’ll talk about surges within surges and what they mean, how they’re different and how metrics help in your organization’s response.

Virgil Strong
Precision Nutrition

COVID has completely altered our lives and the way we conduct business. While the pandemic is (hopefully) temporary, some of the changes forced upon us are likely to persist for much longer (e.g. greater remote working, less business travel). How will the new normal look for analytics leaders and practitioners? And what are you doing to prepare for it?

In this huddle, led by Virgil Strong, CDAO at Precision Nutrition, will discuss the following questions and what changes are expected in each as we come out of the pandemic.
How do we:

  • Build a high-performance culture?
  • Foster diversity and inclusiveness?
  • Promote innovation and risk-taking?
  • Support remote analytics workers?

Join us for an open and honest conversation about the evolving state of analytics management and how you could keep you analytics team on a path to success.

  • COVID has clearly transformed the workplace, but it has also had an impact on hiring practices
    • Remote work is likely here to stay in some form, whether it be full time WFH or a hybrid model, and this means workers are less likely to be constrained by geography
  • Hiring for diversity to benefit from varied perspectives is more possible in a virtual process, as there is an opportunity to cast a wider net
    • Tests such as Kolbe and Caliper can be used to assess for style and fit for remote work
    • Data and analytics activities can help provide insight to how the candidate performs when given an assignment
    • Intentional inclusion of underrepresented groups can provide opportunities to diversify teams
  • Supporting remote analytics workers requires a personal connection that is typically found in face-to-face environments
    • Opportunities for social interaction need to be built into work meetings to address the need to interact on a human level
    • Can be done by reserving time at the beginning of each meeting, dedicating half of 1-on-1 updates to non-work related topics, or even organizing specific online events to bring people together socially
    • A token of appreciation or "pleasant surprise" delivered directly to someone's home is also a way to show sincere appreciation remotely.
  • Innovation and risk-taking can be promoted post-COVID by modeling the desired behaviour in online discussions and communications
    • Online tools enable collaboration and easily building on team members' ideas
    • Periodic retrospectives provide opportunities to reflect on accomplishments as well as ignite improvement initiatives based on what did not work
    • Encouraging and building in time for innovation projects can lead to significant leaps for the organization

Caitlin Moorman
Trove Recommerce

Super Huddle

Some data projects, like building a dashboard from a well-understood data set, are comprised of sequential, linear tasks and have a very high likelihood of success. This makes them relatively easy to scope and manage. Other data projects, like investigating the root cause of a shifting trend in the business, are exploratory and iterative, often follow a circular path, and have a real risk of failure. These circular data projects are often unique within an organization where most teams work via pretty straightforward task completion processes.

The fundamental differences between linear and circular projects should be communicated and managed quite differently to set reasonable expectations and maximize data team impact. As both stakeholders’ reliance on data for decision making and the speed in which those decisions must be taken increases, effectively managing these two types of data projects becomes even more critical to both project success and quality of stakeholder relationships.

Join Caitlin, Alexis, and Celina in this super huddle to answer the following questions:

  • What is the distinction between linear and circular projects in analytics?
  • How can we leverage this distinction to effectively set expectations with stakeholders?
  • How can we manage circular projects to maximize impact and minimize spin?
  • How do we deal with the possibility and reality of failure in analytics projects?

If planning data projects and managing stakeholder expectations are challenges in your organization then this huddle will offer you some practical solutions used by leading analytics teams.

Running this huddle was motivated by Caitlin and Alexis’ articles on the subject:


How can you help manage circular projects?

  • Break things down very small - make them “locally linear”
  • Set expectations up front - understand timeline, urgency, what metric matters most. Commit to iterative deliverable.
  • Just using the phrase enables other teams to understand the issue. It’s so visual - can be very helpful in setting expectations
  • When trying to answer an open ended question, outline your hypotheses up front. Identify your first path, try it, if no, come back and come up with next best hypothesis. Follow that trail instead.
  • Avoid shiny object syndrome and focus on impact.
  • Prioritize regularly and make sure there are consistent check ins. Put some light structure around it with sprint planning.  
    • Having an epic dedicated to open ended work can be helpful. Consistently dedicate time (but limited time) to exploratory work


Identifying up front what is linear and what is circular

  • Identification up front is one of the biggest challenges
    • Analysts are eternal optimists. Data is not as clean as I thought, etc.
    • Circularity is sometimes an unexpected detour
      • Assume by default that every data project will be circular if it is the first time you have done it
      • Product and program managers could make a huge difference here. Understanding scope, requirements, etc. Not a core skillset for analysts. However, it can be difficult for a non-data PM to write scopes because they do not have enough context
      • Unknowns are what blows up scope – break out a research phase before you even try to scope. https://erikbern.com/2019/04/15/why-software-projects-take-longer-than-you-think-a-statistical-model.html


Linear tasks are easier to streamline

  • Automating those tasks and making time for circular tasks is another good approach
  • Make playbooks, documentation and tools so that non-data team members can go down the basic paths themselves. Confluence, drill down dashboards to answer common KPI questions, and pre-built notebooks can be helpful. Keeps data team from becoming “data monkeys” and allows them to focus on new research


What’s good about circular projects?


Analysis is like an escape room

  • You are solving mysteries and puzzles but you have a limited amount of time
  • When do you ask the clue master (stakeholder) a question? There are a limited number of times you can ask


How do you balance open ended exploration and needing to meet the tactical, linear needs of the business?


How can you deal with high level projects (e.g. OKRs) that are circular?     

  • Break out research v. build KRs
    • Sometimes all you can commit to is having a recommendation for the next step – if you do not know it is possible, try not to commit to (and have other projects be dependent on) a build KR
  • When the business sets a goal, it forces you into a linear modality
    • Break down projects into bite sized pieces, set timelines on research, cut scope
    •  “You can tell me what to do or you can tell me how long it will take, but you can’t do both.” (Say it with a smile :-))

Jorge Vasquez
Best Buy Canada

Presentation Huddle

Are you solving the need for analyst resources, talent, and tools but finding your organization is still struggling to achieve its full analytic potential? Has the greater opportunity shifted from the analytics team, to the non-analyst, marketers, leaders, and internal partners that surround them?

In this huddle, Jorge Vasquez, Practice Lead, Digital Analytics at Best Buy Canada, will spend 10 minutes presenting the initiatives and tools put in place and the successes and challenges faced in the process including:

  • Data University – a self-service videos and exercises platform, open to all in the company, educating about analytics
  • Analytics Jam – weekly analytics teach out and office hours
  • Site Club – discussions on strategic initiatives with specific rules, like Fight Club, to guarantee that the discussion revolves around data and not titles in the room

We will follow Jorge’s presentation with a discussion about such initiatives. Delegates will get a chance to share their experiences of how they are getting their orgs to become more data enabled and learn new tips and tricks from the group.

In this huddle, Jorge provided a short intro presentation into the topic and how Best Buy is tackling this challenge. A roundtable discussion followed the presentation. Some of the key points discussed included:

Principal learnings on building a data enabled culture from Best Buy

  • Ensure data/knowledge/information is easy to access for everyone
    • Best Buy have their own self-service video platform, Data University, for any employee to learn about our analytics tools
  • Create avenues for people to get support and nurture their analytics knowledge
    • Best Buy created office hours, Analytics Jam, were anyone in the company can get their work reviewed by an analytics professional
  • Get your analytics team to show the way on what is possible
    • Best Buy run regular meetings, called Site Club, where stakeholders debate strategic initiatives
      • Key rule -> everything has to be supported by data or it does not count
        • A method to show business units how data should drive decision making

Principal learnings from everyone in the huddle

  • Do not force people to learn
    • Cultures where people are forced to learn analytics tend to fail
    • Instead create positive incentives around why they should focus on it: career growth, business impact, etc.
  • Make business teams accountable for knowing their KPIs
    • Work with leaders in your org to keep teams accountable for knowing the data of their business units
      • E.g. have business review meetings were leaders ask about results and how decisions were made
  • Find influencers/champions in business teams to be examples, not necessarily executives
    • Others will copy this behavior and will start using data to inform their decisions or will emulate their learning habits
  • Standardize tools across the organization (as much as possible)
    • Having some teams using Google Analytics and others Adobe Analytics will decrease trust in data as the base for decision making
    • Have one source of truth that everyone can rely on
  • Connect business results to business initiatives to get support from Execs
    • Training sessions/courses can be time consuming for your analytics team and business users -> document wins to prove the ROI of this work
      • E.g. if the marketing team designed an A/B test as a results of your training/influence, document its financial impact and broadcast this win

Examples of what great companies are currently doing to create data enabled cultures

  • Learning and Development teams focused on teaching analytics across the org
    • In most companies, it is up to analysts to drive a data culture, most of the time as a side project by the side of their desk
    • Having specialized teams focus on analytics training, allows analysts to focus on what they do best, without sacrificing the benefits of having a data driven culture
  • Analytics training program for executives.
    • At PBS they built specialized programs targeting executives.
    • Most training initiatives focus on the average employee
    • If you aim to transform your culture, you need to start from the top and get everyone else to follow

Questions for a follow-up huddle

  • What is the best way to accurately measure the value and effectiveness of your training initiatives?
    • We know that data enabled cultures drive more business value but how do we know if a specific learning program is better than another?
    • How do you know if your programs are getting better?
    • Traditionally, teams have used surveys and anecdotal evidence, but should we use more robust measuring methods?
    • Similarly to how we A/B test on our sites, should we experiment with our training methods?
      • E.g. give one cohort a specific type of training and compare performance over time

Tim Wilson
Search Discovery

Presentation Huddle

Whether working in digital analytics or digital optimization (or both!), one of the biggest challenges the analyst faces is interpreting their data correctly and ensuring that their stakeholders do so as well. The increasing messiness of digital data (privacy!), the ever-evolving sophistication of test designs, and the introduction of machine learning to organizations' analytical toolboxes all mean that pitfalls of potential data misinterpretation abound!
 
In this huddle, Tim Wilson of Search Discovery, will share some insights, followed by a discussion and debate of a range of questions around this topic:

  • How important is it for the analysts in an organization to have a strong grasp of core statistical concepts? What about the stakeholders those analysts support?
  • What are the core statistical concepts or techniques are most important for analysts and/or stakeholders to know? Confidence levels? Linear regression? p-values? Causal inference?
  • How can the most important concepts and techniques be taught to analysts and stakeholders' whose time (and interest!) may be in short supply?

Expect to come away with practical ideas of how to increase your and your team's statistical intuition.

The huddle set the stage with three examples where a lack of intuition and specific statistical skills led to a misinterpretation of data:

  • The interpretation of A/B test results: marketers are tempted to see the observed lift between two treatment groups as being an absolute truth that can be extrapolated and annualized, rather than understanding that the observed result falls within a range of variability
  • Correlations on non-stationary data: correlating two metrics that are trending in opposite directions and seeing a high correlation rather than converting the trending (non-stationary) data to be stationary metrics using a technique like first difference (and seeing that there is, in actuality, no correlation)
  • Causal inference and counterfactuals: recognizing that, even in the absence of an experiment (a test), counterfactuals still exist and can be estimated. So, rather than assessing campaign impact by totaling up all of the revenue that can be linked to the campaign, comparing the total observed revenue to the estimated potential revenue (counterfactual) in the absence of a campaign using techniques such as diff-in-diff, 2-stage least squares, or regression discontinuity

Should marketers increase their statistical intuition? There was no consensus:

  • Some delegates felt that this would be trying to "teach statistics to marketers," and that is simply an impossible task. Instead, the marketers should just trust their analysts
  • Others felt that analysts themselves actually lack the degree of statistical intuition and skills that they should have (a view the huddle leader shared, but that there was far from universal agreement on), and that needed to be addressed first
  • Some delegates felt that this was an area that both analysts and marketers did need to grow, but recognized that there is a massive challenge, in that marketers have been conditioned to think that the data will provide them with an objective truth, and productively addressing that misperception is not a trivial undertaking
  • When it came to the discussion of how to provide both analysts and marketers a deeper understanding of statistically-oriented concepts, metaphors ruled the day, and various delegates shared some powerful ones:
  • Framing uncertainty in terms of thinking about a weather forecast: the "probability of rain" level at which you do or do not take an umbrella with you when you leave the house varies by person AND by the importance of staying dry during the specific outing
  • (If the environment is such that it is okay) Pointing to the predictions from the 2016 Presidential election compared to the results
  • Framing the analyst's role as helping the marketer to draw a map and then helping them decide how to get from now to the business outcome they are trying to achieve...but there are multiple routes and the map has missing and smudged pieces on it
  • Thinking about confidence intervals as some vomit on a table that has been covered up with varying-sized stickers (the vibe of the huddle was that, while this spontaneously produced metaphor was both illustrative and entertaining, it might not be effective in other environments)

Some final takeaways were:

  • Using an external consultant (this was NOT suggested by an external consultant) come in as an authority to do some light education, if well-targeted and positioned, could be one way to take the burden off the in-house analysts to try to educate their stakeholders to shift their ways of thinking regarding data
  • "Critical thinking" is something that really needs to be fostered in both analysts and their stakeholders
  • Managers of analytics teams need to fight to protect the analysts' time—to ensure there is time to do that critical thinking, to apply the appropriate statistical techniques and methods, and to prepare deliverables to communicate the underlying complexity of the data effectively (without losing their stakeholders)
5:25 pm: Summary & Prizes

5:30 pm: Happy Hour, Virtual Chocolate Tasting & Networking

  Meet old colleagues & new peers for spontaneous 1:1 & 1:many conversations in our visual networking lounge.