Simplifying a Data Sync: Lessons from Integrating netFORUM and Luminate

Effective and efficient data management is a shared struggle for many nonprofits. Most have multiple systems with different processes, and lots of manual work to make the data sync… if that’s even possible.

The Challenge

You probably track your fundraising through System A but email your supporters through System B. You dream of a way to pull all of the data together for easier reporting, but even if you do find a possible solution, will it really help, or will it just make your data management even more complicated?

The National Trust for Historic Places knew this struggle well, but when they connected with the Abila netFORUM team, they saw an opportunity to link their online communications and advocacy platform, Luminate, with their membership and fundraising database on netFORUM.

They knew they wanted to use netFORUM as their authoritative donor database, with Luminate continuing to serve as their online communications database, but how could they connect the two platforms?

That’s where we came in.

Cornershop Creative undertook with enthusiasm what we knew would be a challenging data sync project and far exceeded expectations. Whether your project is a small design refresh or a major technical undertaking, I can’t recommend Cornershop Creative enough. – Alison Hinchman, National Trust for Historic Places

The Process

The Trust contracted with Cornershop to build a data sync between Luminate and netFORUM. This was a very long process, but here’s a summary of our approach:

  1. Requirements. First, we worked with netFORUM and the Trust to create a Functional Roadmap, outlining the project goals, user stories, and technical recommendations for the sync. We later developed more specific Constituent Use Cases to describe how and when records would be updated, and on which platform.
  2. Data Mapping. We tracked every single field from each platform in a Master Data Map. For each type of data, we listed all available fields and how those fields should match across platforms.
  3. Order of Operations. Next, we then used the Data Map to identify how the data should sync or update. For example, if a constituent record is updated on Luminate, what should happen to it when that record is synced to netFORUM? For each field, we needed to know which platform had the authoritative data and how or when corresponding record data should change or stay the same.
  4. Development and Testing. Once we established how the sync should work, we built and tested scripts to actually do the sync. We first tackled individual scripts by data type, testing constituent records, then donation records, email history, and interactions.
  5. Load Testing. Once we confirmed that our sync worked as expected, we tested the scripts against progressively larger amounts of data, reviewing any errors on particular types of data and confirming that the sync could handle higher volumes of data between the two platforms.
  6. User Interface. Finally, we documented the sync and built a simple user interface so that the Trust’s team can see when the sync was last run, identify and resolve data issues, and monitor results.

The Results

By tackling this very complex project through smaller, more specific tasks, we developed an easy-to-maintain, reliable data sync.

Thinking about taking on your own complex data project? Here are some things we learned will ensure your project smoothly:

  • Make sure decisions are documented and shared with all necessary team members. For complicated projects, this is especially important, since you may need to refer to past history to address new issues.
  • Identify unknowns and stay flexible. Things didn’t always go perfectly smoothly: Sometimes we discovered that a data field was formatted differently than we realized, or we realized a test process needed to be rethought due to system limitations. Being up-front about our assumptions and certainties from the very beginning made it easier to adjust and adapt to new discoveries as needed.
  • Ask questions, even if you worry about sounding stupid. The simplest questions led to in-depth conversations about how, for example, a constituent key, a constituent ID, and a member ID might not actually all be the same thing!

And of course, if you could use a hand with developing your own ideal data sync, please contact us!