The Hidden Complexity of Sync, Building Reliable Integrations in a Distributed World
I have lost count of how many times someone has asked me if we can “just sync the data both ways.” The request usually arrives with the same casual tone someone might use to ask a barista to heat the milk a little longer. There is a quiet assumption baked into it, the belief that data is a polite guest who will follow instructions and arrive where we expect. The reality is closer to babysitting a handful of toddlers who all missed their nap and are now sprinting in different directions with sticky hands.
Once you have lived inside the architecture of two-way syncs, you stop romanticizing the idea of a single unified truth. You start seeing sync as an ongoing discipline. Distributed systems theory in the morning, practical API design by the afternoon, incident reviews for dessert. That is the real job. Not moving data, but preventing the slow decay of integrity through drift, race conditions, retries, ambiguous deletes, and the small cracks that widen with every scale event.
I learned this the hard way while building GTM Engine. The deeper we went, the more I realized every integration is a distributed system in disguise. The sooner you accept that, the better your architecture becomes.
The Illusion of Symmetry in Two Way Sync
The fantasy of a clean two way sync is powerful. It suggests a world where reading and writing are mirrored actions, where System A and System B remain in perfect alignment like a pair of synchronized swimmers. Anyone who has worked on these systems knows how fast that symmetry collapses under real conditions.
Temporal Drift Is Inevitable
Two systems never see changes at the same moment. One receives an update at 11.04.36, the other at 11.04.41. A five second gap is nothing to a human, but it is eternity to an integration pipeline. During that eternity, a user edits a field. A webhook fires late. A retry triggers. Now the sync logic must guess which version represents the intent of the user.
I do not trust guesses in distributed systems. They eventually break your heart.
Order of Operations Refuses to Behave
You might think operations always arrive in the order they were created. That assumption melts as soon as you hit a network hiccup. Updates arrive late. Out of order. Duplicated. Missing. It forces your pipeline to carry a kind of quiet humility. You no longer assume operations are linear. You treat them like puzzle pieces that might never show up in the correct sequence.
The Salesforce to HubSpot Example Everyone Underestimates
Here is the thing about syncing Salesforce with HubSpot. You never sync two identical objects. You sync two worlds with:
- Different schemas
- Different rate limits
- Different identity models
- Different assumptions about what “success” means
Salesforce might accept the update but return a partial success. HubSpot might reject the payload because the field exists but cannot be updated through the API. Conflict resolution becomes an entire chapter in the playbook. Sometimes you use last write wins. Sometimes you need authoritative sources. Sometimes you build a reconciliation job that sweeps for inconsistencies like a quiet custodian who shows up after midnight.
Nothing about this is symmetrical. It only looks symmetrical to the untrained eye.
The Data Integrity Challenges Nobody Mentions in the Pitch Deck
Uniqueness feels simple until you try to enforce it across multiple systems that do not agree on how identity works. One system believes email is the key. Another uses contact ID. Another does a soft match. Suddenly the same human becomes three records. Your sync job sees them as separate entities. Your dedupe logic tries to merge them. Your analytics pipeline starts screaming.
Retries create duplicates. Mismatched keys create orphans. Simultaneous updates create race conditions that sneak through your tests because tests run in a controlled world. Production is anything but controlled.
The thing about sync engineering is that bugs do not hide in your logic. They hide in everything around your logic.
Why “Exactly Once” Is the Most Misleading Promise in Distributed Systems
There is a reason engineers keep repeating the same old mantra. Exactly once delivery cannot be guaranteed. Not at scale, not across networks, not across systems with different latencies and retry policies. The phrase sounds pessimistic. It is actually liberating, because accepting this truth forces you into better patterns.
The Problem with At Least Once Delivery
Retries look helpful until the moment they create duplicates that mutate your downstream state. A webhook times out. The sender retries. Now the consumer receives the same event twice. If your logic is not idempotent, everything unravels.
Even AWS services, the gold standard for reliable delivery, occasionally replay messages during outages. Kafka engineers famously say the only real guarantee is at least once. It is not that exactly once is impossible, but that the cost to simulate it is so high you trade reliability for fantasy.
Engineering Toward Idempotency
Idempotency keys are quiet heroes. They let your pipeline treat duplicates like harmless echoes. The transactional outbox pattern gives your system a clean boundary where state and events cannot drift. Deduplication tables and event hashes turn chaos into something predictable.
These techniques do not create perfection. They create stability, which is far more useful.
Lessons From Real Incidents
Every meaningful sync architecture carries scars. I still remember reviewing logs during an AWS SNS event replay. Messages that should have been delivered once were delivered dozens of times. Some services handled it gracefully. Others buckled. The difference came down to one thing, whether the receiving logic assumed the world was perfect.
Assumptions are the most dangerous things you can store in production.
The Messy Truth About Deletes and Associations
Deletes expose the soul of an API. You can tell a lot about a system by the way it removes things. Some APIs return a flag. Some return nothing. Some return a null that might mean deleted, or might mean omitted for performance. This is not a small issue. It is the origin of data drift.
Missing Fields Carry Ambiguous Meaning
When HubSpot omits an association from a payload, is it gone or just not included in that endpoint version. When Salesforce returns a record without a related object, is it deleted or hidden behind a permission setting. Without clear semantics, sync logic becomes guesswork again.
Guesswork is expensive.
Building Robust Delete Logic
Over time I learned that safe delete handling usually involves more data, not less.
- Use tombstone markers
- Use soft delete flags
- Maintain shadow tables for state tracking
- Rebuild associations through reconciliation jobs
Deletion is not an event. It is a negotiation between systems, each with its own view of what “gone” means.
Designing for Schema Evolution
APIs evolve. Fields appear. Fields disappear. Associations shift. The integrations that break are the ones built with the belief that the schema is permanent. The resilient ones treat schema like a living creature. They version contracts. They handle missing fields gracefully. They fail closed instead of open.
That mindset separates scalable architectures from brittle ones.
All These Problems Are Connected
Two way sync. Delivery guarantees. Deletes. They all express the same truth. There is no single source of truth. There are only multiple systems trying to represent reality in parallel, each with imperfect timing and incomplete information.
The network is unreliable. Latency is a fact of life. Divergence is inevitable. The job is not to prevent it. The job is to detect it, reconcile it, and minimize the cost of repair.
When you design with that mindset, you stop building integrations and start building distributed systems. You create consistent event contracts. You maintain sync state independently from business logic. You accept eventual consistency and design toward reconciliation. Idempotency becomes a habit rather than a feature. Observability becomes a requirement rather than an afterthought.
This is what we built inside GTM Engine. Not just pipelines, but an architecture that respects the messiness of reality. HubSpot becomes the sender. GTM Engine becomes the brain that enriches, interprets, and orchestrates the right operations. Sync stops being a fire drill. It becomes infrastructure.
The Discipline Behind Integration Engineering
There is a quiet satisfaction in building systems that tolerate randomness without losing their shape. It requires a certain humility. You stop expecting perfection. You start designing for ambiguity. You learn to see beauty in reconciliation jobs and dedupe keys and delete markers. It becomes a craft rather than a chore.
The best engineers I know treat sync logic with the same respect others reserve for core product features. They understand that the sharpest bugs do not hide in code. They hide in the moments between systems. The gaps. The silences. The omitted fields. The retries that come half a second too late.
If you want clean integrations at scale, you invest early in idempotency, observability, sync contracts, and cross system reconciliation. You build for the world as it is, not the world you wish it were.
That is how you keep the toddlers from running in completely opposite directions. You will never achieve perfect order. You do not need to. You only need architecture that remains trustworthy under pressure.
That is the craft. That is the discipline. That is the real work of integration engineering.
About the Author
Greg Lee is a seasoned technologist and startup CTO with a passion for building AI platforms that make complex work feel simple. From early-stage ventures to nonprofit tech, Greg brings deep experience across engineering, product leadership, and organizational enablement. He’s led teams at the frontier of AI productivity—most recently as CTO at a stealth AI startup redefining how business logic gets automated.
Previously, Greg co-founded Rowsie AI, an intelligent Excel-native assistant that helps professionals analyze data, build models, and drive decisions without writing code. The idea was simple: meet users where they already work, and turn spreadsheets into superpowers. His approach combines pragmatic engineering with visionary execution—always focused on delivering tools that actually get used.
Even outside of startups, Greg has brought that ethos to volunteer projects like American Whitewater, where he helped modernize critical tools for river enthusiasts across mobile platforms. Whether building for scale or for good, Greg’s mission remains the same: craft technology that empowers people to do their best work.
