Data sources, structures and data quality
Where business data comes from, the shapes it takes (structured, semi-structured, unstructured; wide vs long), and the dimensions of data quality.
This lesson builds on: How organisations use data — department by department.
- Identify common internal and external data sources
- Distinguish structured, semi-structured and unstructured data
- Recognise tidy (long) versus wide tables and when each is useful
- Assess a dataset against the core data-quality dimensions
Where data comes from
Internal sources are systems the organisation runs: point-of-sale (POS), e-commerce platforms, CRM (customer relationship management), ERP (finance, purchasing, inventory), HR systems, call-centre software, web analytics, spreadsheets maintained by teams.
External sources include market research, government statistics, partner data, social media, weather, exchange rates and purchased datasets.
Each source was built for an operational purpose (take a payment, register an employee), not for analysis. That is why analysts reshape and validate data before using it.
Structures
| Structure | Description | Examples |
|---|---|---|
| Structured | Fixed columns and types in tables | Orders table, payroll, ledger |
| Semi-structured | Self-describing but flexible fields | JSON from an API, event logs, XML |
| Unstructured | No predefined model | Emails, call recordings, PDFs, images |
Analysts mostly work with structured data, increasingly with semi-structured data (Module 6 covers JSON), and occasionally with unstructured text.
Grain: what one row means
The single most important property of a table is its grain — what one row represents. In the Orion Retail database:
orders: one row per orderorder_items: one row per product line in an ordersales_targets: one row per store per month
Summing a column from orders after joining it to order_items repeats each order once per line and inflates totals. Knowing the grain prevents this — you will meet this "fan-out" problem in the SQL joins lessons.
Wide vs long (tidy) tables
A wide table puts values of a variable across columns; a long (tidy) table has one row per observation.
Wide (easy for humans to read):
| store | 2024 | 2025 |
|---|---|---|
| Baku Central | 166k | 178k |
Long (easy for tools to filter, aggregate and chart):
| store | year | revenue |
|---|---|---|
| Baku Central | 2024 | 166k |
| Baku Central | 2025 | 178k |
Databases, Power BI and pandas work best with long data. Reports often present wide layouts. Converting between them is called pivoting and unpivoting.
Data quality dimensions
| Dimension | Question | Orion example |
|---|---|---|
| Completeness | Are required values present? | 69 rows in customers_raw have no email |
| Validity | Do values follow rules and formats? | Signup dates appear as 2024/05/01 and 01.05.2024 |
| Uniqueness | Is each entity recorded once? | 45 duplicate customer rows |
| Consistency | Do related values agree? | City "baku", "BAKU" and " Baku " |
| Accuracy | Does the value reflect reality? | A birth year of 1890 |
| Timeliness | Is the data current enough? | Yesterday's sales not loaded until noon |
You can see these problems in the practice database right now:
SELECT customer_id, email, city, signup_date
FROM customers_raw
ORDER BY customer_id
LIMIT 6;| customer_id | city | signup_date | |
|---|---|---|---|
| 3 | julia.smith3@example.com | DUBAI | 2023-07-22 |
| 10 | liam.novak10@example.com | tbilisi | 2023-01-01 |
| 31 | murad.kaya31@example.com | Dubai | 29.07.2022 |
| 35 | hannah.martin35@example.com | London | 2024-03-22 |
| 52 | rustam.aliyev52@example.com | Ganja | 2023/08/07 |
| 61 | hannah.brown61@example.com | GANJA | 2024-12-19 |
Notice the inconsistent city formatting and date formats. Counting duplicates:
SELECT COUNT(*) AS total_rows,
COUNT(DISTINCT customer_id) AS distinct_customers,
COUNT(*) - COUNT(email) AS missing_emails
FROM customers_raw;| total_rows | distinct_customers | missing_emails |
|---|---|---|
| 645 | 600 | 69 |
You do not need to understand the SQL yet — Module 2 teaches it from scratch. The point: data quality is measurable, and you should measure it before trusting a dataset.
Before any analysis, record row counts, distinct keys, missing values per column and date ranges. It takes ten minutes and prevents most embarrassing errors.
Common mistakes
Operational systems contain test records, duplicates and manual corrections. Validate against an independent total (e.g. Finance's monthly revenue) whenever possible.
Practice
In the Orion Retail database, orders has one row per order and order_items has one row per product line. If you join them and then SUM the discount_pct column of orders, what happens?
A JSON response from a marketing platform API is an example of:
The same customer appears as "BAKU", "baku" and " Baku " in the city column. Which data-quality dimension is affected?
Summary
- Data comes from operational internal systems and external sources; neither was designed for your analysis.
- Know the grain of every table you use.
- Long (tidy) data suits tools; wide data suits human-readable reports.
- Measure completeness, validity, uniqueness, consistency, accuracy and timeliness before trusting data.