Vasu Sattenapalli of RightData discusses how today’s chief data officers (CDOs) are responsible for managing the organization’s data and analytics operations—including architecture, user requirements, software development, machine learning integration, and more.
The biggest challenge they report is data silos. In fact, Forrester Consulting found that 80% of people in IT say reducing silosOpens a new window is a top priority for their organization. Does this sound familiar? If so, it might be time to take a step back and reconsider your approach.
Data Silos Are a Reality for All
To be clear, it’s understandable that data silos are top of mind. Right now, regardless of size, industry, age, or technical maturity, companies are trying to become more data-driven, which requires democratizing data and making it available to as wide of an audience in the business as possible. However, most organizations also rely on disparate data sources and platforms, as their data architecture was built over time.
That means data is scattered across multicloud and on-prem platforms, which, first and foremost, makes it difficult to know what data is even available for use—let alone whether the data is correct, up-to-date, or appropriate. Second, data is prone to duplication because it is distributed in such a way. For instance, customer data from one platform is duplicated and used elsewhere by the marketing, finance, or customer service teams.
At one point, the solution to these problems was thought to be data warehouses, physically eliminating silos and implementing tighter controls around data, but that approach overlooks two key details: that duplication will always be a reality, even when tighter controls are added, and that a data warehouse mindset ignores the expense and effort that was put into developing the current architecture.
See More: How DevOps Teams Can Manage Telemetry Data Complexity
Duplication and Context-based Data Are Necessary
Consolidating to a single source of truth was the preferred way to handle silos in the past, making it easier to see what data is available and where duplicates exist so the dataset can be streamlined. However, data duplication is incredibly valuable because it allows individual teams to contextualize data and make it more actionable for their specific use case. For example, creating a single “customer data” dataset makes it harder for sales teams to find the unique data points they’re looking for to close new deals or for marketing teams to segment data into existing and prospective customers to build effective marketing campaigns.
Data silos were born out of necessity, so rather than eliminating the ability to duplicate data, we need to reframe our thinking. For most companies, I recommend building a framework that gives transparency and visibility into why different duplicate versions exist, where that data lives, what the business context is, and what trust metrics can be attributed to it. Doing so better serves data users at large, enabling anyone in the company to find what they’re looking for and put it to use for the right purpose.
Virtualization and Persistence Tactics Are Both Valid
There are two schools of thought when it comes to company data: virtualization prefers to leave data in place, whereas persistence advocates for moving all data to one location in a data lake or warehouse. And though persistence tactics were the most commonly used for years, today, it isn’t necessary to dismantle the architecture you already have in place and start from scratch. Distributed governance and platforms are a natural part of doing business, especially as business needs changed and new data tools and platforms are released.
That said, every company is unique, so a persistence mindset and full data transfer might be the right call depending on your organization’s specific needs. Regardless of which strategy you employ, the important thing is to maintain flexibility and prioritize establishing a framework that addresses the underlying issues of lack of trust, accuracy, and findability—instead of focusing on the data’s physical location.
Data Products Grant Data Flexibility While Solving Silo Challenges
Rather than continuing to struggle with finding the right or most accurate data or investing in moving to a data warehouse, a data product can take advantage of your existing architecture and democratize your data to make it more accessible immediately. Bolting on a data product makes disparate data sources searchable by adding in metadata, data quality parameters, and a cataloging system. These can also be flexible; for instance, if a company does want to move forward in a data warehousing direction, they can still use a data product to help solve their silo and data access problems. But if a company is choosing the persistence route, the data product can search their existing data and make it easier to find and validate the right data with ease and know that it’s the correct dataset for their needs and for a given application.
Testing Out a Data Product To Move Beyond Silo Challenges
For companies interested in exploring a data product, I recommend attaching this project to an initiative or modernization effort that may already exist so you build in budget, support, and structure. I also recommend starting by identifying a particular domain, then starting small with a team within that domain. Look for a high-impact area with a low barrier to entry; I often see the greatest impact in those business units that use numerous data platforms and usually conduct ad hoc analysis because you’ll see immediate benefits. Steer clear of teams that rely on a handful of IT-created dashboards or canned reports. Instead, you’ll see a huge improvement if, for example, you have a dataset of sales leads and a sales team of 500 professionals who are eager to be able to access and effectively analyze that data so they can close deals.
Get Back on the Path To Trusting Your Data
The single source of truth isn’t the only answer. Context-based data and data duplication are necessary parts of how we all do business, and we need frameworks, processes, and technology that help us handle and harness them. By adding the right data product into your tech stack, you can begin to immediately address key obstacles to data democratization and get your organization on the path to better business decision-making.
How are you tackling your data silos without losing sight of other data goals for your business? Share with us on FacebookOpens a new window , TwitterOpens a new window , and LinkedInOpens a new window . We’d love to hear from you!
Image Source: Shutterstock
