Skip to main content

18.7 Object Storage & Data Lakes

1. Object storage is not a database​

Object storage provides durable files and objects.

Typical platforms include:

  • Amazon S3,
  • Azure Data Lake Storage,
  • other cloud object stores.

The architecture must define how files become governed analytical data.

2. Typical flow​

Application / source
        |
   object storage
        |
 file ingestion
        |
 Datasphere / lake processing
        |
 curated table
        |
 semantic model

3. File-oriented versus table-oriented architecture​

Files are useful for:

  • raw landing,
  • exchange,
  • archival,
  • large-scale ingestion.

Tables are useful for:

  • governed analytics,
  • semantic modeling,
  • reusable consumption.

Do not confuse a landing zone with a semantic layer.

4. Schema evolution​

Define:

  • required columns,
  • optional columns,
  • datatype compatibility,
  • versioning,
  • malformed-file handling,
  • late-arriving data.

5. Large-scale architecture​

For very large lake workloads, decide whether Datasphere should:

  • consume selected curated datasets,
  • provide semantic access,
  • replicate a subset,
  • or remain outside the heavy raw-data processing layer.

Datasphere does not need to become the raw lake for every enterprise dataset.

6. Exercise​

A data lake contains 50 TB of clickstream data.

Design a pattern where Datasphere exposes customer and product analytics without copying the entire lake.