Big Data Analytics Services for Actionable Business Intelligence

A data platform is the same eight layers whichever products sit underneath it, and a quote only becomes comparable when both vendors say which of those layers they are quoting for. This section names the layers, defines the vocabulary a proposal will use at you, lists only the technologies this site already claims, and sets out what lands on your side of the table.

The data and analytics services Sthenos sells

Data work is bought in pieces, and which piece you need depends on where the problem actually sits. Sthenos groups it seven ways below. Each group links to the page on this site that sells the surrounding capability, so a shortlist can be assembled from the site rather than from a discovery call.

Data strategy and governance

Deciding what the organisation is allowed to hold, who may read which rows, how long each dataset is kept, and who owns the definition of a metric. This is the layer that makes analytics auditable rather than merely operational, and it is one of the four strategy commitments already set out above on this page. It is also the work that stops a growing platform becoming a second shadow estate nobody can account for. Related: IT consulting and cybersecurity.

Data engineering and pipelines, batch and streaming, ELT and ETL

Getting data out of the systems that produce it and into somewhere it can be modelled: scheduled batch loads for most sources, streaming where a decision cannot wait for the next load, and the transformation logic in between. The ETL practice named above is the same work in its classic order, and ELT is the same job with the transformation run after the load, inside the warehouse. Related: big data development and DevOps.

Warehouse and lakehouse

The storage layer, which this page already describes as scalable data warehouses and data lake architectures across cloud and hybrid environments. In practice that means a raw landing area, a modelled serving area, and a decision about which of the two a given question should be answered from. Related: cloud computing, cloud migration and optimization and the platform pages for AWS, Azure and Google Cloud.

Business intelligence and dashboards

Turning modelled data into reports a decision maker opens, which usually means fewer dashboards rather than more, each answering a question somebody named actually asks. This page already sells descriptive and exploratory analytics and forecasting; the sibling page data analytics services carries the buyer-facing version of the same work, and Microsoft platform services covers the reporting stack most enterprises already own.

Advanced analytics and machine learning

Predictive and prescriptive work on top of a governed platform: feature engineering, model development, validation and deployment, which the research, analytics and data science practices above already describe. Related: data science, AI and machine learning and agentic AI, which is where enterprise data platforms stop being a reporting asset and start being something an agent reads.

Data quality and master data

Freshness, volume and schema drift checks so a broken load is found before a board meeting rather than during it, and one agreed record for the entities every system holds its own version of: customer, product, site, employee. Master data synchronisation and quality observability are both already published work on this site, on the ISV page and on data analytics services respectively.

Data platform operations

Running the thing after it is built: orchestration that alerts when a load fails instead of serving stale numbers silently, cost control on storage and query, access reviews, and an owner for every pipeline. The DataOps practice above is this work named as a discipline. Related: managed services, infrastructure management and the production readiness checklist, which scores the operational half of any platform against seven areas.

The reference architecture of a data platform, layer by layer

A data platform has the same eight layers whichever products sit underneath them, and Sthenos designs against all eight. Naming the layers is what makes a quote comparable between two vendors, because it forces both of them to say which layers they are actually quoting for.

1. Sources

The systems that produce the data: the ERP, the CRM, the billing platform, the operational databases behind your own applications, the third party APIs, and the spreadsheets that nobody will admit are load bearing. Counting them honestly comes first, because source count drives the number.

2. Ingestion

The mechanism that copies data from each source into the platform. Some sources offer a clean API, some allow a database read replica, some export a file overnight and some can only be polled. Ingestion is where the unglamorous engineering lives, and it is the layer that breaks first when a source changes shape.

3. The lake, where raw data lands

Raw data kept in something close to its original form, cheaply, before anybody has decided what it means. The point of keeping it is that a modelling mistake made a year ago can be corrected by reprocessing rather than by asking a source system for history it no longer holds. The risk is a lake that never gets modelled, which is a storage bill with no reporting layer attached to it.

4. The warehouse, where modelled data is served

Structured, modelled tables designed for querying: conformed entities, agreed grain, and columns whose meaning is written down. This is the layer reporting should read from, and the reason a warehouse exists at all is that reporting straight off production systems slows those systems and breaks whenever a source schema changes.

5. Transformation, with tests

The code that turns raw into modelled: joins, deduplication, type handling, slowly changing history and business rules. Treated as code, it is reviewed, versioned and tested, so a change to a definition is visible in a pull request rather than discovered in a meeting. Transformation with tests is already how the sibling data analytics services page describes this layer.

6. The semantic layer

Metric definitions written once, in code, versioned, so that revenue means the same thing in the finance dashboard and the sales dashboard. This is the artefact that stops two reports disagreeing, and where it is absent a technically correct platform is still not trusted.

7. Reporting and consumption

Dashboards, scheduled reports, exports, embedded analytics, notebooks and, increasingly, a natural language interface sitting on top of the same governed definitions. Layer it over governed data rather than instead of it, because a fluent answer is not the same thing as a correct one.

8. Governance, across every layer above

Not a layer at the end but a set of controls running through all seven: who may read which rows, where a number on a dashboard came from, how long data is retained, what a deletion request has to reach, and which changes are audited. Sthenos treats this as design work at the start rather than remediation at the end.

The data vocabulary, defined

These are the terms a data proposal will use at you. None of them is difficult, and knowing what each one means is usually enough to tell a considered design from a shopping list.

Data warehouse, data lake and lakehouse

A warehouse stores structured, modelled data ready for reporting. A lake stores raw data in its original form for exploration and machine learning. A lakehouse is the pattern that puts warehouse style tables, transactions and schema enforcement directly on top of lake storage, so one copy of the data serves both jobs instead of two copies drifting apart. Many organisations end up with a lake and a warehouse whichever name they give it, and the common mistake is building the lake first and never modelling anything.

Star schema and dimensional modelling

Dimensional modelling splits data into facts, which are the measurable events such as an order or a claim, and dimensions, which are the things you slice those events by such as customer, product, date or region. A star schema is one fact table surrounded by its dimensions, which is why it is drawn as a star. It exists because it is fast to query and because a business user can read it, which matters more than elegance when somebody has to confirm that the model matches how the company actually works.

Grain, conformed dimensions and slowly changing dimensions

Grain is what one row of a fact table represents: one order, or one order line, or one day of one product in one store. Getting it wrong is expensive to correct, because every number built on it is quietly wrong. Conformed dimensions are shared dimensions used identically across several fact tables, which is what makes two subject areas comparable. Slowly changing dimensions are the technique for keeping history when an attribute changes, so a sale made last year is still attributed to the region the customer was in at the time.

ELT and ETL

ETL extracts data, transforms it in a separate processing step and then loads the result. ELT extracts, loads it raw into the warehouse first, and transforms it there using the warehouse engine. ELT became the default when cloud warehouses made compute cheap and elastic, and its practical advantage is that the raw data is already in the platform when somebody discovers the transformation was wrong. ETL remains the right order when the data may not land unfiltered for legal reasons, or when the source volume dwarfs what you want to store.

Change data capture

Reading the change log of a source database so the platform receives inserts, updates and deletes as they happen, instead of re-reading whole tables on a schedule. It is the honest answer to two problems at once: load on the source system, and deletes, which a periodic full extract cannot see because the row is simply no longer there.

Streaming and batch

Batch processes data in scheduled groups, typically nightly or hourly. Streaming processes each event as it arrives. Streaming costs more to build and considerably more to operate, because correctness under failure, late arriving events and replay all have to be designed rather than assumed. The question worth asking is not whether the data could be real time but whether a decision actually changes between one load and the next.

The semantic layer

One place where every metric is defined, in code, versioned, and read by every reporting tool rather than reimplemented inside each of them. Without it, active customers is defined four times in four dashboards and all four are defensible, which is how a platform that is technically correct still loses the room.

Data catalog and lineage

A catalog is the inventory: which datasets exist, what each column means, who owns it and how sensitive it is. Lineage is the map from a number on a dashboard back through every transformation to the source column it came from. Lineage is what makes a wrong figure debuggable, and it is the first thing to ask a vendor to show you, because it cannot be faked in a demo.

Row level security and column masking

Row level security filters what a given user can see inside a shared table, so a regional manager querying the same dashboard as head office sees only their own region. Column masking hides or partially obscures a sensitive field for users who do not need it. Both belong in the platform rather than in each report, because a control implemented per dashboard is a control that will be missing from the next dashboard.

Data quality checks and data SLAs

Quality checks are assertions that run with every load: freshness, that the data arrived at all; volume, that roughly the expected number of rows arrived; schema, that the shape has not changed; and business rules, such as no negative quantities. A data SLA is the agreement about what happens when one fails, including who is told, how quickly, and whether the dashboard shows stale data or shows nothing. Serving stale numbers silently is the failure mode worth designing against.

The data technologies this site already works with

Every product named below is named because another page on sthenostechnologies.com already says this company works with it, and that page is linked beside it so you can read the claim in its original context. Where this site publishes no product for a category, the category is described and no product is named. A logo grid is easy to write and proves nothing, so this list is deliberately shorter than the ones you will see elsewhere.

What is deliberately not named here

Streaming platforms, transformation frameworks, the independent cloud data warehouses sold as separate products, and every business intelligence tool other than the two above are described here as categories and named nowhere, because no page on this site claims one. That is a statement about what has been published, not about what an engineer has ever touched. If a specific product decides your shortlist, ask in the first conversation and we will tell you plainly whether it is a strength, a learnable adjacency or a reason to pick somebody else.

What lands on your side of the table

Every item below restates something this company already publishes. Nothing is promised here that is not promised somewhere else on this site.

• The repository and the cloud accounts, in your name from the first commit. No licence back to us and no dependency on Sthenos continuing to exist, exactly as the About page states it.
• A costed roadmap before any build starts. Where the scope is clear we quote a fixed price for a defined outcome.
• The data model, written down. Facts, dimensions, grain and the meaning of each column, so the model can be checked against how the business actually works before code depends on it.
• Pipeline code with its tests, and orchestration that alerts on failure rather than serving stale numbers silently, which is how data analytics services already describes the platform layer.
• Metric definitions in code, versioned. The semantic layer, so two dashboards cannot disagree without somebody having changed a definition in a reviewable place.
• Dashboards people actually open, which usually means fewer of them, each answering a question somebody named actually asks.
• Quality and observability checks for freshness, volume and schema drift, and lineage from source to number, so a wrong figure can be traced rather than argued about.
• Access by role, retention rules, and deletion that reaches every copy. Governance as a built artefact, not a policy document.
• A platform your own engineers can run once we hand it over, which is the standard the Maryland page already sets for our work, and monitoring, maintenance and a roadmap after launch if you would rather we kept running it.

How a data engagement runs, step by step

Sthenos works in two week sprints, so working software arrives every two weeks rather than as a reveal at the end. The stages below are the order the work has to happen in; they overlap in practice, and no stage below carries a duration because a duration without your source count attached to it would be decoration.

• Definitions and source inventoryBefore any pipeline is written, two lists get made: every system that holds data in scope, with how it can be read and who owns it, and every metric the business intends to report, with the person who is allowed to decide what it means. Disagreement surfaces here, which is the cheapest place for it to surface.Produces the source inventory and a first metric dictionary
• Architecture and data modelThe layer decisions from the reference architecture above get made against your actual constraints: which cloud, batch or streaming per source, where raw data lands, what the warehouse grain is, and which controls the governance layer has to enforce.Produces the architecture, the dimensional model and a costed roadmap
• Pipelines, transformation and testsIngestion for each source, transformation as reviewed and versioned code, and the quality assertions that run with every load. The first sprint deliberately takes one unglamorous source end to end, because a thin slice that reaches a real number proves the pattern before ten sources are built on it.Produces working pipelines, their tests and the orchestration around them
• Semantic layer and reportingMetric definitions written once and read by the reporting tool, then the smallest set of dashboards that answers the questions named in stage one. Reports get reviewed by the person who owns the definition, not only by the person who commissioned the dashboard.Produces the semantic layer and the first reporting set
• Handover and operationAccess reviews, lineage documentation, cost controls on storage and query, and a named owner for every pipeline. The platform is built so your own engineers can run it once we hand it over. Where you would rather we ran it, that is a separate ongoing arrangement rather than a hidden dependency in the build.

What moves the cost of a data platform

None of these carries a number here, because a number without your scope attached to it is decoration. The one band this page does publish, for a first warehouse and reporting layer, is answered at the foot of the page.

What to ask any data analytics vendor

These are the questions worth putting to every firm on your shortlist, including this one. The first five are published on our data analytics services page and are repeated here so that a buyer landing on the engineering page gets them too.

• Who decides what a metric means? If the answer is the vendor, you will get numbers nobody in the business trusts.
• Show me a lineage view. From a number on a dashboard back to the source column. If they cannot, nobody can debug a wrong figure later.
• What happens when a source changes shape? Silent breakage is the norm without explicit schema checks.
• Will my team be able to add a metric without you? Ask for the actual pull request that adding one would require.
• What did you refuse to build? Every honest analytics engagement includes talking someone out of a real time dashboard they do not need.
• What documentation is a contractual deliverable? Not what they promise to write. The distinction is already the advice on our software outsourcing page.
• Who is paged when a load fails at 2am? Establish it before you sign, not after the first incident.
• What would make you decline this requirement? A firm that has never declined anything is telling you something about its scoping.

More questions about data platforms and dashboards

What is the difference between a data lake and a lakehouse?

A lake stores raw files cheaply with no guarantees about structure. A lakehouse adds warehouse behaviour on top of that same storage: table definitions, schema enforcement and transactional updates. One copy of the data then serves both exploration and reporting instead of two copies drifting apart.

Where does the data in a typical build actually come from?

An ERP or finance system, a CRM, the operational databases behind your own applications, a billing or payments platform, one or two third party APIs, and at least one spreadsheet somebody maintains by hand. Counting the sources honestly is the first hour of scoping, because source count drives the estimate more than data volume does.

What is a semantic layer, and why does it decide whether two dashboards agree?

It is the place where each metric is defined once, in code and versioned, and read by every reporting tool. Without it, active customers gets defined separately inside each dashboard, every version is defensible, and no amount of correct engineering underneath will make the two reports agree.

Who owns the metric definitions?

Somebody in your business, named, and never the vendor. We write the definitions down, put them in code and make changing one a reviewable event, but the person who decides what revenue means has to be an employee or nobody will accept the number.

Can you build real time dashboards?

Yes, and the more useful conversation is whether you need one. Streaming costs more to build and considerably more to run, because late arriving events, replay and correctness under failure all have to be designed. The test is whether a decision actually changes between one load and the next.

How do you handle data security and compliance?

Access by role, row level filtering inside shared tables, encryption in transit and at rest, retention rules and deletion that reaches every copy, plus lineage so an auditor can trace a number. Where you are bound by HIPAA, SOC 2, GDPR or CCPA we build to those controls and hand over the evidence trail. Sthenos is not SOC 2 attested and holds no attestation of its own.

Do you migrate legacy reports?

Yes. Before rebuilding we inventory which reports are actually opened, which duplicate each other under different names, and whose numbers already disagree, then rebuild those on the modelled layer.

Cloud or on premises for the data platform?

Cloud unless something decides otherwise, and the things that decide otherwise are real: data residency rules, an existing on premises estate that is not moving, or a licence you have already paid for. The cloud is usually the one your team already knows, because migrating skills is slower than migrating workloads.

More on data and analytics

Services: Data analytics services, Big data development, Data science, AI and machine learning, Agentic AI, DevOps.

Platforms: AWS, Microsoft Azure, Google Cloud, Microsoft platform services, Microsoft 365 and SharePoint, Cloud computing.

Before you buy: Production readiness checklist, Production readiness audit, Cybersecurity, How Sthenos works, rates and ownership.

Ready to scope a data platform? Bring the list of systems that hold your data and the metrics you intend to report, and we will tell you which layers you actually need first. Talk to our engineers.