When the global product experience was first implemented, using a mocked dataset from Lomadee made perfect sense.

The goal at that point wasn't to build a complete product management system. We needed products to make the experience work, test the spinning mechanics, validate the UI, and start thinking about how catalogs would interact with tenants.

A small transformation layer was enough.

The application loaded a Lomadee JSON file, extracted the fields we needed, calculated things such as discount and tier, and exposed the resulting array to the catalog experience.

It was simple.

And, at the time, simple was exactly what we needed.

But the project has evolved since then.

We now have something much more valuable than a mocked product dataset: an ingestion pipeline capable of turning different product representations into a validated internal product structure.

That changes how the global experience should work.

The Current Situation

At the moment, the global catalog still ultimately depends on a local Lomadee dataset.

The catalog resolver effectively has two possibilities:

Tenant has products
       ↓
Use tenant catalog

Tenant has no products
       ↓
Use Lomadee products

No tenant
       ↓
Use Lomadee products

The abstraction itself is useful.

The problem is the source.

The global experience shouldn't fundamentally depend on Lomadee.

Lomadee was useful as an initial source of data, but our application now has its own product model and its own rules for determining whether a product is valid.

That means we can move the responsibility upstream.

Instead of transforming an external mock dataset every time the application starts, we can ingest products once, validate them, persist them, and let the global experience consume the same canonical product structure used elsewhere in the system.

The Ingestion Pipeline Already Exists

This is one of the reasons this transition is relatively straightforward.

We've already been working toward separating the different responsibilities involved in product ingestion.

A CSV currently goes through a process roughly equivalent to:

CSV
 ↓
csvRowToProduct
 ↓
enrichProduct
 ↓
validateProduct
 ↓
verified product
 ↓
persistence

JSON follows the same general idea:

JSON
 ↓
jsonProductToProduct
 ↓
enrichProduct
 ↓
validateProduct
 ↓
verified product
 ↓
persistence

The adapters deal with differences in external representations.

The enrichment layer normalizes and fills in values.

The validation layer decides whether the resulting product satisfies our application requirements.

That separation is important.

The catalog itself shouldn't care whether a product originally came from a CSV file, JSON document, affiliate provider, or another source.

Once ingestion has completed, it should simply be a product.

One Product Structure, Multiple Sources

This is the architectural direction we want to preserve.

Imagine that products arrive from several sources:

Shopee CSV
     │
     ▼
CSV Adapter
     │
     ├──────────────┐
                    │
Partner JSON       │
     │             │
     ▼             │
JSON Adapter       │
     │             │
     └──────┬──────┘
            ▼
       Enrichment
            │
            ▼
        Validation
            │
            ▼
     Canonical Product

At the end of this process, the origin becomes secondary.

The application receives a consistent product representation containing things such as:

• name
• image
• URL
• offer URL
• price
• commission
• commission rate
• store
• category
• affiliate
• description
• tier
• priority
• stock

The important architectural principle is that global products and tenant products should not require different product schemas simply because they belong to different catalogs.

They are still products.

What changes is where they are stored and how they are selected.

Introducing the Global Product Pool

The next step is to introduce a persistent global product collection.

Conceptually:

tenants/
  tenantA/
    products/
  tenantB/
    products/

globalProducts/
  productA
  productB
  productC

Tenant products remain tenant-specific.

Global products become a shared product pool that can be used by the global experience and, when appropriate, as the fallback catalog for tenants.

This gives us a much cleaner model:

                 Product Ingestion
                        │
               ┌────────┴────────┐
               │                 │
            Tenant             Global
               │                 │
               ▼                 ▼
        Tenant Products    Global Products

The ingestion process remains the same.

Only the persistence target changes.

The Catalog Resolver Doesn't Need to Change Much

One of the encouraging things about the current architecture is that the consumer already has a useful abstraction.

The application currently asks for:

const { products: catalog, source: catalogSource } =
  await getCatalogForTenant(tenant);

The global experience doesn't need to know where those products came from.

That contract can remain.

The difference is what happens behind it.

The future flow becomes:

Global experience
       │
       ▼
getCatalogForTenant(null)
       │
       ▼
Global Product Pool

For a tenant:

Tenant experience
       │
       ▼
getCatalogForTenant(tenant)
       │
       ├── Tenant products available
       │          ↓
       └──  Tenant catalog
  
        

This is a small change with a fairly significant architectural consequence.

The global catalog becomes a real domain resource rather than a development fixture.

Moving Lomadee Back to Where It Belongs

The current Lomadee transformation code was useful because it adapted an external representation into the application's expected shape.

But now we have a more general ingestion system.

Instead of keeping special catalog logic like:

Lomadee JSON
 ↓
mapLomadeeProduct()
 ↓
global catalog

we can eventually treat Lomadee as just another possible source:

Lomadee
   ↓
Lomadee Adapter
   ↓
Canonical Product

The same could eventually happen with other providers.

That gives us a much more scalable model:

                 ┌── CSV
                 │
                 ├── JSON
                 │
External sources ├── Lomadee
                 │
                 ├── Shopee
                 │
                 └── Future providers
                         │
                         ▼
                   Ingestion Layer
                         │
                         ▼
                  Canonical Product
                         │
                  ┌──────┴──────┐
                  ▼             ▼
             Tenant Pool   Global Pool

The experience layer doesn't need to change when a new source is introduced.

An Administrative Import Flow

This architecture also opens the door to something we've intentionally avoided until the underlying pipeline was ready: managing the global catalog through an administrative interface.

The intended flow is roughly:

Admin
 │
 │ upload CSV / JSON
 ▼
Authentication
 │
 ▼
Verification
 │
 ├── valid products
 ├── warnings
 └── invalid products
 │
 ▼
Preview
 │
 ▼
Commit
 │
 ▼
Global Product Pool

This is where the existing verification functionality becomes particularly useful.

An administrator shouldn't have to blindly upload a file and hope that the data works.

The system can first report:

Total products:     1,000
Valid:                963
Warnings:              21
Invalid:               37

The valid products can then be previewed before anything is committed to the global catalog.

Verification Before Persistence

This separation is important enough to be considered part of the architecture rather than simply a UI feature.

The desired lifecycle is:

Raw Data
   ↓
Parse
   ↓
Normalize
   ↓
Enrich
   ↓
Verify
   ↓
Preview
   ↓
Commit

Not:

Upload
   ↓
Write everything to production

This gives us an opportunity to reject malformed products before they become part of the experience.

It also means the same verification rules are applied regardless of whether the data comes from CSV or JSON.

Global Products Should Be Source-Independent

Perhaps the most important consequence of this change is that the global experience becomes independent from the original provider.

Today, the mental model is still somewhat:

Global products = Lomadee products

The goal is to change that to:

Global products = validated products available to the global catalog

Those are very different concepts.

Lomadee becomes a source.

CSV becomes a transport format.

JSON becomes a transport format.

The global catalog becomes a domain concept of its own.

Keeping a Temporary Fallback

There is also no reason to make this migration unnecessarily risky.

During the transition, the existing Lomadee dataset can remain as a legacy fallback.

The resolution logic can effectively become:

Try global product pool
        │
        ├── products available
        │       ↓
        │   global catalog
        │
        └── empty
                ↓
        legacy Lomadee fallback

This gives us a safe migration path.

We can upload a real validated dataset, test the global experience, monitor it, and only remove the legacy source once we're confident that the new catalog is behaving correctly.

The Architecture We Are Moving Toward

The complete picture starts looking like this:

                       ADMIN SYSTEM
                            │
                    Authenticated Upload
                            │
                     CSV / JSON files
                            │
                            ▼
                    ┌───────────────┐
                    │   Adapters    │
                    └───────┬───────┘
                            │
                            ▼
                    ┌───────────────┐
                    │  Enrichment   │
                    └───────┬───────┘
                            │
                            ▼
                    ┌───────────────┐
                    │  Verification │
                    └───────┬───────┘
                            │
                            ▼
                    Canonical Products
                            │             
                            │
                            ▼
                       Global Pool
                            │
                            │
                            ▼
                        Global UX

And the catalog service remains the boundary between all of that complexity and the actual experience.

What Comes Next

The next implementation step isn't to redesign the ingestion pipeline.

It is to reuse it.

The main work will be around making the persistence layer understand one destination:

global\

Then we'll introduce an authenticated administrative endpoint capable of sending CSV and JSON data through the existing verification pipeline.

The first version doesn't need to solve every future catalog problem.

It simply needs to establish this relationship:

Validated Product
       ↓
Global Product Pool
       ↓
Global Experience

Once that exists, we have a much stronger foundation for everything that comes after it.

We can later think about catalog composition, multiple affiliate providers, seasonal catalogs, prioritization, deduplication, product lifecycle management, and more sophisticated global selection rules.

But those become extensions to a real product catalog rather than workarounds around a mocked dataset.

And that's probably the most important architectural shift here.

The global experience is no longer going to consume a mock source that happens to contain products. It will consume our own validated product catalog.