01 / Perimeter
Cluster: data automation
Data integration middleware for your business systems
When there are more than two systems to connect, or the same data arrives in different formats from different suppliers, point-to-point links become unmanageable. You need a layer in between that normalises, validates and routes — and that notices its own mistakes, rather than leaving you to find out three days later.
Formats and channels I handle
- CSV
- XML
- JSON
- EDI
- Spreadsheets
- REST APIs
- SFTP
- Mailboxes
- Webhooks
The principle is always the same: whatever arrives is translated into one internal shape. From that point on, adding a source does not mean rewriting the flow — only writing its translator.
- What I deliver
- The middleware layer with the normalised internal shape, the validation rules, the scheduling and the readable logs.
- When it is the right work
- From three systems upwards, or when the same data arrives in different formats from different suppliers.
- Indicative time
- An estimate, not a contractual commitment: two to six weeks for the first flow. Every additional feed costs far less than the first.
- What I don't do
- I don't resell subscription automation platforms: I build the piece you are missing, and it stays yours — code, credentials and documentation.
02 / Diagnosis
Why point-to-point links don't scale
With three systems there are three direct links. With six there are fifteen.
Every pairwise link is a thing to maintain, to update when a format changes and to check when something doesn't add up. The cost doesn't grow with the systems: it grows with the pairs.
Repetitive work should not be organised better. It should be taken away.
Opening the supplier's CSV, correcting two rows by eye, saving and importing is work nobody ever decided to assign to anybody. Yet it occupies a person every week, and that person cannot be ill on the wrong day.
Every supplier has their own format, and changes it when it suits them
Columns in a different order, different separators, dates written three ways, implicit units of measure. If the reading logic is scattered across five scripts, a format change breaks the flow at a point nobody remembers the location of.
Without validation, bad data gets in and spreads
A product code that doesn't exist or a negative quantity entering the ERP does not stay the ERP's problem: it becomes a wrong order, a false stock figure and a reconciliation at the end of the month.
Without logs, there is no truth about what went through
When data is missing, the reconstruction starts from people's memories. With a log per run, the question «did it arrive or not?» has an answer in thirty seconds instead of a meeting.
Before and after, on the same flows
These are not results measured on clients: they are the process changes a middleware layer produces, described for what they are.
| Flow | Before | After |
|---|---|---|
| Supplier price list | File opened by hand, columns reordered, prices pasted into the ERP. | Read from the source, normalised onto the internal shape and loaded with the rejects flagged. |
| Orders from several channels | Three different ways of receiving orders and three different manual procedures. | Three translators onto the same internal shape, then a single flow downstream. |
| Supplier invoices | Data retyped by hand from the document into the ERP. | Data read already structured and reconciled, with the doubtful cases queued for a person. |
| A malformed row | It blocks the import, or it gets in wrong and surfaces later. | It is rejected, logged with the reason and flagged to whoever can correct it. |
| A new supplier | A new manual procedure, explained by word of mouth to whoever will run it. | One more translator onto the internal shape: the rest of the flow is untouched. |
| Month-end check | Spreadsheets are compared and everyone hopes the totals match. | The log is read: rows through, rows rejected, the reason for each. |
03 / Solution
One internal shape, then translators
A layer in between that speaks a single language
Middleware is not one more piece of software to manage: it is the place where you stop managing five different procedures.
The idea is simple and as old as business computing: instead of connecting every source to every destination, you define an internal shape that represents the data the way your company means it. From that point on, adding a supplier does not mean touching the logic — it means writing one more translation.
Normalisation
Every source is translated into one internal shape: codes aligned, units of measure converted, dates in a single format, decimals consistent. It is the choice that lets you add a supplier without reopening everything else.
Validation
Rules declared on every field: whether it is required, its format, the ranges allowed, whether the codes exist in the destination system. Data that fails does not get in, and the reason is known.
Scheduling
Every flow has a declared window and frequency. Dependencies are explicit: if the product load has failed, the price list load does not start against nothing.
Observability
For every run: time, duration, rows read, rows written, rejects with their reason. When a flow is skipped, the alert reaches a person with the useful information already in it.
Retries and idempotency
A network error should not turn into human work. The flow retries with a growing wait, and every record carries a stable key so a second pass updates instead of duplicating.
Reconciliation
At the end of every cycle the counts at source and destination are compared. If they don't match, the system says so: nobody discovers the drift at the end of the quarter.
Typical automations it is used for
- Supplier CSV, XML and EDI feeds normalised and loaded into the ERP with no human step
- Price lists and product records updated automatically when the supplier publishes a new version
- Orders arriving from different channels brought onto a single internal shape before they enter the ERP
- Supplier invoice data entry taken off the desk, with the data arriving already structured and reconciled
- Stock levels and warehouse movements kept aligned between systems on a frequency you decide
- Scheduled exports to analysis tools, with the same figures for everyone and no parallel spreadsheet
The route the data takes
04 / Specifications
What goes in, what comes out, what happens when it is wrong
The specifications of the layer in between
Middleware is judged on three things: what it can read, what it does when the data is wrong, and what trace it leaves behind. The rest is implementation detail.
Input
- Formats
- CSV, XML, JSON, EDI, spreadsheets, responses from third-party APIs. Everything is brought back to one internal shape.
- Channels
- SFTP, network folders, dedicated mailboxes, APIs polled on a schedule, inbound webhooks.
- Dirty formats
- Inconsistent separators, columns in variable order, missing headers, different encodings: handled in the translator, not by hand.
- A new source
- Costs one more translator. The logic downstream and the destinations are untouched.
Processing
- Validation
- Rules per field: whether it is required, its format, ranges, whether the code exists at the destination.
- Transformations
- Unit and currency conversions, rounding declared explicitly, lookup tables between your codes and the supplier's.
- Scheduling
- A frequency and a window for every flow, with explicit dependencies between flows that assume one another.
- Load
- Incremental and batched: after the first alignment only the changed records go through.
Errors and traces
- Rejects
- A bad row does not stop the flow: it is isolated, logged with the reason and made available for correction.
- Retries
- A growing wait on temporary errors, and a threshold beyond which the flow stops and warns a person.
- Logs
- One line per run with time, duration, counts and rejects. Readable by you, not only by me.
- Alerts
- A notification to a person when a flow is skipped, exceeds the reject threshold or does not start at all.
Running it, and owning it
- Where it runs
- On your own infrastructure or on a cloud service in your name, pinned to an EU region or another region if your rules require it. The credentials are yours from day one.
- Personal data
- Only the data the flow needs is processed, kept for the minimum useful to reconstruct a run. It is a GDPR requirement and the simplest way to keep the system small.
- Code
- Source and documentation handed over. No encrypted components, no subscription keeping the service alive.
- Maintenance
- Scope and duration agreed before we start. A supplier changing their format is a piece of work, not an emergency.
Included in a middleware project
- Analysis of the real sources, using the real files rather than the sample ones
- Definition of the internal shape and of the lookup tables between codes
- Development of the translators, the validation rules and the scheduling
- A log per run, handling of rejects and alerts that reach a person
- Proving on real data in a separate environment, before production
- Technical documentation, handover of code and credentials, and an agreed post-release support period
Not included, and I say so up front
- Subscriptions to third-party automation platforms, if we decide to use one
- Hosting and infrastructure, which stay in your name
- Correcting the supplier's dirty data upstream: bad rows are rejected and flagged, not guessed at
- Accounting reconciliation and tax filings, which stay with whoever does them today
- Development on the supplier's side when it is they who must expose an access route or change a format
- Any promise of a financial result: it cannot be guaranteed and I don't make it
05 / Route
From the first feed to a flow at full speed
How it gets built, one flow at a time
We always start from the flow that costs the most manual work, not the one that is easiest to write. The durations are estimates to help you plan, not contractual commitments.
- 01
Inventory of the sources
I collect the real files, not the examples: that is where the missing columns, the odd encodings and the cases nobody remembered turn up.
- Cosa ricevi
- A list of the sources with format, channel, frequency and recurring anomalies.
- Durata
- 1 week (estimate)
- 02
The internal shape
I define the single representation of the data and the lookups between your codes and your suppliers'. It is the decision that determines what the sixth source will cost.
- Cosa ricevi
- A specification of the internal shape, the lookup tables and the validation rules.
- Durata
- 1–2 weeks (estimate)
- 03
The first flow, running
One flow is taken to full speed, complete with validation, logs and alerts. It exists to prove the arrangement on a real case before it is replicated.
- Cosa ricevi
- One flow in production with logs running and rejects handled.
- Durata
- 2–6 weeks (estimate)
- 04
Extension and handover
The remaining feeds are added onto the arrangement already proved. Every line passes through my code review before production.
- Cosa ricevi
- The remaining flows in production, documentation and credentials handed over.
- Durata
- Varies with the number of feeds
One thing to plan for that has nothing to do with code: the sample files. A first flow cannot be sized on the specification a supplier publishes, only on what they actually send — headers that move, encodings that change mid-year, the one row a quarter that breaks everything. Collecting a few weeks of real feeds, rejects included, is usually the longest lead time in phase one, and it is work only your side can do.
06 / Frequently asked
The same ones I get on the phone
Questions about middleware and data automation
Here are the answers to what nearly everyone asks in the first half hour. If yours is missing, write to me.
01What is the difference between middleware and a direct integration?
A direct integration connects two systems, and it is perfectly fine when the systems are two and stay two. Middleware makes sense from three upwards, or when the same data arrives in different formats: instead of N point-to-point links there are N translators onto one internal shape. The practical advantage shows at the fourth source, when adding it costs a translation rather than a rewrite.
02Isn't a subscription automation tool enough?
Sometimes it is, and when it is I say so. Automation platforms work well on standard cases between widely used cloud services. They become fragile and expensive when you need non-trivial transformations, validation against the codes in your own ERP, handling of rejects or meaningful volumes. The substantive difference is ownership: what I build stays yours and keeps working even if you stop working with me.
03What happens to rows that fail validation?
They are isolated, logged with the reason for the reject and made available to whoever can correct them at source. They do not block the rest of the flow: nine hundred and ninety good rows go in, ten stay out and are visible. When the share of rejects passes an agreed threshold the system warns, because at that point the problem is not the individual row — it is the feed.
04What does it cost, and how long does it take?
I have no price list. The first flow is the one that costs: inside it sits the definition of the internal shape, which every other flow then uses. As an estimate and not a commitment, the first flow takes two to six weeks from the analysis; the later ones cost far less because they reuse the same arrangement. After the analysis you get a fixed figure for the project, not an open hourly rate.
05Where does the middleware run? Do I have to buy a server?
It runs where it makes sense for your situation: on your own infrastructure if the systems are on site and the data must not leave, or on a cloud service in your name. Either way the credentials are yours from day one and the hosting is billed to you, not to me with a delegation. The cost of the infrastructure does not pass through me: I don't put a margin on it.
06Our data crosses borders. What does that mean for GDPR?
Two practical things, both settled in the architecture document rather than in a clause nobody reads. First, where it runs: the infrastructure is in your name and can be pinned to an EU region, or another region if your own rules require it. Second, what it carries: a flow is designed to move the minimum data it needs, so a feed that only reconciles product codes never sees a personal detail in the first place. On the paperwork I am a processor working on your systems, I am happy to sign your own data processing agreement, and what is logged, for how long and who can read it is written down before the first flow goes live.
07How do I know it worked last night?
By looking at the log, which is designed to be read by you and not only by me: time, duration, rows read, rows written, rejects with their reason. On top of that, alerts reach a person when a flow is skipped, doesn't start or passes the reject threshold. The stated goal is that nobody should have to check every morning: you get told when it matters.
08If a supplier changes the file format, does everything break?
That translator breaks, and only that one: the rejects make it obvious immediately, with the reason. The rest of the flows keep running. Adapting a translator to a changed format is a short, contained piece of work, and it is exactly why the reading logic sits in one place instead of being scattered across five procedures.
07 / Related
Where to go next
Pages connected to this one
Middleware is the base layer of custom software. The other two specialisations sit on top of it.
08 / Contact
Half an hour, no automated quotation
Send me a feed and I'll tell you whether it can be automated
The quickest way to know whether a flow can be automated is to look at a real file. We spend half an hour: you show me how the data arrives today and who handles it by hand, I tell you what can be taken away and what can't. If the spend makes no sense against the time you save, I say so straight away.
Or write to me with a sample feed attached: I answer, usually within one working day.
All the contact detailsHow the call works
- Length
- Thirty minutes, video call or phone, in English.
- Who is there
- Me. No salesperson, no handover afterwards.
- What I need
- To know where the data comes from, in what format, and who handles it by hand today.
- What you get
- An honest view on what can be automated, in what order and with what order of magnitude of spend.