How to clean your data when you don’t have a data team

Your ERP implementation has started. Then the vendor mentions data migration, and explains what they’ll need from you to move data out of your existing systems and into the new one. Suddenly it feels like too much. Where do you even start?

If that sounds familiar, you’re not alone. It was the focus of our Lunch & Learn “How to Clean Your Data When You Don’t Have a Data Team”. Here’s what we covered, plus some great insights from the discussion.

Why should we care?

Three reasons data migration deserves attention.

  • Data is the blood of your organisation. A blood test gives leading and lagging indicators of your health. Your data does the same for your organisation. If the data is corrupt or poor quality, the indicators are wrong and misleading, and the organisation is likely to be unwell.
  • Data migration is complex, expensive and painful. There’s often more than one source system, and moving everything cleanly into a new ERP is hard work.
  • Data migration is highly dull, yet equally important. “Clean the customer data” sounds simple. But what does clean mean? How do you know you’re improving it and not making it worse? When is it done? For the people doing the work, it’s repetitive and mind-numbing: find the discrepancies, fix them, upload them, seek approval, repeat.

Because it’s complex and dull, most organisations put it aside and deal with it later in the project. That’s where failure starts to creep in.

The easy ways (if you can afford them)

Three valid options: a dedicated team, an agency, or a fixed price with the ERP vendor.

There are three straightforward ways to handle data migration:

  • Deploy a dedicated data migration team. If you have people dedicated to it, they’ll get the job done.
  • Outsource it to an agency. They cleanse and migrate the data on your behalf.
  • Negotiate a fixed price with the ERP vendor. Do this during selection. The vendor has likely done similar work for other customers.

All three are valid and worth serious consideration if you have the money or can see a return on the investment. But they all cost money, and many organisations don’t have that luxury. The rest of this article is for them: you have a set team, and you need to make the most of it while the ERP implementation is under way.

The fundamentals

The fundamentals of data migration.

These are the foundations everything else builds on:

  • Not all data elements are equally important. Customers, suppliers, chart of accounts and parts don’t all matter equally at every point. Early in an ERP project, the chart of accounts comes first.
  • Data dependencies are real. Data has to go into a new ERP in a set sequence, which the vendor will usually give you. You can’t load parts or customers at random.
  • Not all data needs to be migrated. Ten years of historical transactions can sit in a database you access when needed. History is the obvious one, but there are often other things you can consciously leave behind.
  • Data can be cleaned and migrated after go-live. Asset data is often cleansed and migrated after go-live, as long as it isn’t close to financial year end.
  • Not all data needs to be cleaned to the same level. The chart of accounts must be spotless and signed off. For parts, you might clean only those used in the last five years. Customer data might need to be complete, while for suppliers you might focus on the ones you’ve dealt with recently.
  • Data elements are broad, deep and sometimes scary. Customer data isn’t just a list of customers. It can include parent customers, subsidiaries, bill-to and ship-to addresses, and multiple contacts for each. Be patient with the people doing this work.
  • Without scope, we’re all lost. Draw clear boundaries around what’s in and what’s out, so the team can focus on what matters.
  • It’s refinement, not perfection. If someone tells you they’ll have the complete, clean list in three months, something’s wrong. Work in iterations instead: one this month, another next month, a final one after that, improving each time. The sooner you start, the better.

Six keywords

The themes that come out of the fundamentals.

Those fundamentals boil down to six keywords: priority, dependency, scope, ownership, plan, and MVP, or minimum viable product. That last one means being clear on what “done” looks like. When you say customers must be cleansed, what does cleansed mean? When you say suppliers must be migrated, is that the last five years and no more? Define the boundaries and quality expectations up front.

Put together, this means you need a strategy, a schedule, clear ownership, the right tools and governance. Too often these either don’t exist, or they were written once and are now collecting digital dust on a shared drive.

Strategy

A data migration strategy answers five simple questions.

A data migration strategy sounds daunting, but it doesn’t need to be a long document. It answers five questions:

  • Why? The purpose and success criteria. For example, 100% of in-scope data migrated, at least 99% validated accuracy, financial reconciliation with zero variance, and formal business sign-off.
  • What? The scope, plus the source and destination systems. For example, no historical transactions; chart of accounts and balances, customers, suppliers and parts migrated from legacy systems and spreadsheets.
  • How? The methods and tools, such as an iterative extract, cleanse, map, load and validate approach using templates, scripts and reconciliation controls.
  • When? Timing, based on the project schedule. Chart of accounts first, then master data, with mock migrations before cutover.
  • Who? Roles and responsibilities. A lean team might have a data lead for governance, subject matter experts for rules and validation, a technical person for execution, and the project manager for coordination.

Schedule

A simple data migration schedule shows who is doing what, and when.

From the strategy comes a schedule. In a simple example, an accountant finalises the chart of accounts in month one. An accounts receivable administrator spends 10% of their time cleansing customer data for three months, maps it to the new ERP in month two, runs a first migration of a subset in month three, a second iteration in month four, and the final production load in month six.

Your estimates might be off, and that’s fine. You’ll adjust as you go. But without a schedule you’re running blind.

Ownership

A RACI matrix makes ownership of each data domain clear.

A simple RACI matrix (responsible, accountable, consulted, informed) works just as well for data migration as for the rest of the project. In a council, for example, the director of corporate services might be accountable for signing off the chart of accounts, customers and suppliers. Different people are responsible for doing the work. The project manager and vendor are kept informed, and other directors are consulted where relevant.

Governance

With a baseline in place, progress and risk can be reported honestly.

With a strategy, schedule and RACI in place, governance becomes real. You can report that the chart of accounts is 50% complete and on track, that customers are behind with minor risk, and that suppliers are at high risk and need escalating to the steering committee.

Without a baseline, governance is just imagination. Everything is on track or off track depending on how much people are panicking that week.

From the discussion

Treat it as a sub-project. An experienced client-side ERP project manager said they wished data migration had been tackled earlier on several projects, instead of waiting for the vendor. They suggested running it as a sub-project with its own risks, actions and decisions register. They also flagged three things that are easy to miss: the change management impact of new account and job structures on staff who raise purchases; the operational reports that depend on the data (which may be the very reason you’re replacing the ERP); and making sure data coming from integrated systems reconciles.

Make the mapping accessible. The same attendee had seen a simple but valuable idea done well: publishing the old-to-new chart of accounts mapping on the intranet, so staff who only touch the project late, during training, can easily find what they need.

Don’t jump straight to the tools. A data and analytics consultant said many people go too quickly into the technical side. If you haven’t done the planning, you lose sight of things like dependencies and change management. For example, if you decide not to migrate historical data, what happens to the reports that show two or three years of history? You end up doing iterations anyway, but chasing your tail instead of following a plan. AI-based cleansing tools have come a long way in the last few years and can add real value, but only once the foundations are in place.

Scope, scope, scope. Another attendee who works with clients on their data said defining scope is the most fundamental step, and that scope starts with discovery: understanding what your data actually does before you start cleaning it. They pointed to large public sector programs that ran over by millions because this wasn’t done, and to one government that spent a full year understanding its data before cleansing began. They also warned against assuming your data is right just because things seem to work. They described an organisation that believed its HR data was perfect because everyone got paid, only to find payroll stopped when the new system went live, because the data was wrong.

Don’t forget the job and project structure. One attendee noted that the structure sitting behind the chart of accounts, including how you split capital from operating costs, is almost always forgotten and underestimated. That knowledge sits inside the client organisation, not with the vendor. They also suggested that a templated, repeatable approach to data migration would be a real help for small organisations with very limited resources. It’s a point we’ve taken on board.

Closing thoughts

Closing thoughts on data migration.

  • Data migration is a discipline, not a task. It needs a strategy, a plan, ownership and governance, and it calls on technical, business and coordination skills.
  • It starts before the project. Too often it starts mid-project or late, and delays everything. If you can’t start before the project, start at the very beginning.
  • The most efficient way is to have a plan. With limited resources, how well you use them decides whether you succeed. A clear plan, scope and ownership get you the most from what you have.

Recent Blogs

Share Your Details – Let’s Make It Happen