SQL
Write correct SQL across PostgreSQL, MySQL and SQLite: joins, window functions, CTEs, upserts, indexes, isolation levels, NULL rules and reusable queries.
On this page
Cheatsheet#
| Task | Snippet |
|---|---|
| Rows in A with a match in B | FROM a JOIN b ON b.a_id = a.id |
| All of A, B where it exists | FROM a LEFT JOIN b ON b.a_id = a.id |
| Rows in A with no match in B | LEFT JOIN b ... WHERE b.id IS NULL or WHERE NOT EXISTS (SELECT 1 FROM b WHERE b.a_id = a.id) |
| Groups meeting a condition | GROUP BY x HAVING count(*) > 1 |
| Running total | sum(amount) OVER (ORDER BY created_at ROWS UNBOUNDED PRECEDING) |
| Rank within a group | row_number() OVER (PARTITION BY customer_id ORDER BY created_at DESC) |
| Previous row’s value | lag(amount) OVER (ORDER BY created_at) |
| Name a subquery | WITH recent AS (SELECT ...) SELECT ... FROM recent |
| Walk a tree | WITH RECURSIVE t AS (... UNION ALL ...) |
| Insert or update (PostgreSQL, SQLite) | INSERT ... ON CONFLICT (id) DO UPDATE SET v = EXCLUDED.v |
| Insert or update (MySQL, MariaDB) | INSERT ... ON DUPLICATE KEY UPDATE v = VALUES(v) |
| Conditional aggregate | sum(CASE WHEN status = 'paid' THEN amount ELSE 0 END) |
| Null-safe default | coalesce(nickname, name, 'anonymous') |
| Null-safe comparison | a IS NOT DISTINCT FROM b (PostgreSQL), a <=> b (MySQL), a IS b (SQLite) |
| Next page without OFFSET | WHERE (created_at, id) < (:last_ts, :last_id) ORDER BY created_at DESC, id DESC LIMIT 50 |
| Delete duplicates keeping the newest | DELETE FROM t WHERE id NOT IN (SELECT max(id) FROM t GROUP BY k) |
| Show the plan | EXPLAIN ANALYZE (PostgreSQL, MySQL 8.0.18+), EXPLAIN QUERY PLAN (SQLite) |
| Bulk conditional update | UPDATE t SET ... FROM src WHERE t.id = src.id (PostgreSQL); UPDATE t JOIN src ON ... SET ... (MySQL) |
| Make a lock explicit | SELECT ... FOR UPDATE |
| Try a write, keep it out of the data | BEGIN; ...; ROLLBACK; |
Examples are written for PostgreSQL 17 and run unchanged on SQLite 3.45+ and MySQL 8.4 or MariaDB 11.4 unless a dialect note says otherwise. Use the PostgreSQL SQL reference as the closest thing to a readable standard; product pages cover PostgreSQL, MySQL and MariaDB and SQLite operations.
How a query executes#
The engine does not run a SELECT top to bottom. Logical order is FROM and joins, WHERE, GROUP BY, aggregates, HAVING, window functions, SELECT expressions, DISTINCT, ORDER BY, LIMIT. That order explains most syntax surprises: a column alias from SELECT is not visible in WHERE (it does not exist yet), WHERE cannot filter on an aggregate (use HAVING), and a window function cannot appear in WHERE or HAVING (wrap it in a subquery or CTE). The physical plan the optimiser chooses may reorder anything as long as the result is the same, which is why EXPLAIN and not intuition decides whether a rewrite helped.
All examples use this schema:
CREATE TABLE customers (id integer PRIMARY KEY, name text NOT NULL, region text);
CREATE TABLE orders (
id integer PRIMARY KEY,
customer_id integer NOT NULL REFERENCES customers(id),
status text NOT NULL, -- 'pending', 'paid', 'cancelled'
amount numeric(12,2) NOT NULL,
created_at timestamp NOT NULL
);
CREATE INDEX orders_customer_created ON orders (customer_id, created_at DESC);Joins#
A join produces one row for every pair of rows that satisfies the condition. Inner drops pairs with no match, outer keeps the unmatched side with NULLs in the other side’s columns.
-- customers with their orders; customers without orders are absent
SELECT c.name, o.id, o.amount
FROM customers c
JOIN orders o ON o.customer_id = c.id;
-- every customer, order columns NULL where none exist
SELECT c.name, o.id, o.amount
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.id;
-- customers with no paid order: the condition on the outer table goes in ON, the NULL test in WHERE
SELECT c.name
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.id AND o.status = 'paid'
WHERE o.id IS NULL;
-- same thing, usually the better plan and immune to duplicate rows
SELECT c.name
FROM customers c
WHERE NOT EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.id AND o.status = 'paid');
-- self join: pairs of orders from the same customer on the same day, each pair once
SELECT a.id, b.id
FROM orders a
JOIN orders b ON b.customer_id = a.customer_id
AND date(b.created_at) = date(a.created_at)
AND b.id > a.id;
-- cross join: every combination, here to build a calendar grid
SELECT r.region, s.status FROM (SELECT DISTINCT region FROM customers) r CROSS JOIN (VALUES ('pending'), ('paid'), ('cancelled')) AS s(status);Putting a filter on the outer table’s column in WHERE instead of ON turns a LEFT JOIN back into an inner join, because the NULL row fails the filter. FULL OUTER JOIN exists in PostgreSQL and SQLite 3.39+, not in MySQL or MariaDB, where it is emulated with LEFT JOIN ... UNION ALL ... RIGHT JOIN ... WHERE a.id IS NULL. USING (customer_id) is shorthand when both columns share a name and it collapses the pair into one output column. A join on a one-to-many relation multiplies rows: summing o.amount after joining a second one-to-many table double counts; aggregate each side in a CTE first.
Aggregation and HAVING#
GROUP BY collapses rows sharing the grouped values into one; every other selected column must be an aggregate or functionally dependent on the group key (PostgreSQL accepts columns of a table whose primary key is grouped; MySQL with ONLY_FULL_GROUP_BY and SQLite are stricter or looser respectively). HAVING filters groups after aggregation; WHERE filters rows before it, which is cheaper when the condition does not need the aggregate.
SELECT c.region,
count(*) AS orders,
count(DISTINCT o.customer_id) AS customers,
sum(o.amount) AS revenue,
avg(o.amount) AS avg_order,
sum(o.amount) FILTER (WHERE o.status = 'paid') AS paid_revenue, -- PostgreSQL and SQLite 3.30+
sum(CASE WHEN o.status = 'paid' THEN o.amount ELSE 0 END) AS paid_revenue_portable
FROM orders o
JOIN customers c ON c.id = o.customer_id
WHERE o.created_at >= date '2026-01-01' -- row filter: cuts work before grouping
GROUP BY c.region
HAVING count(*) >= 10 -- group filter
ORDER BY revenue DESC;count(*) counts rows, count(col) counts non-NULL values, count(DISTINCT col) counts distinct non-NULL values. sum over an empty set is NULL, not 0; wrap it in coalesce(sum(x), 0) when a total feeds arithmetic. GROUP BY ROLLUP (region, status) adds subtotal rows (PostgreSQL, MySQL 8; not SQLite); the extra rows have NULL in the rolled-up column, distinguishable from a real NULL with GROUPING(region). string_agg(name, ', ' ORDER BY name) (PostgreSQL), group_concat(name, ', ') (MySQL, SQLite) and json_agg collapse a group into one value.
Window functions#
A window function computes a value per row from a set of related rows without collapsing them. PARTITION BY splits the rows into groups, ORDER BY orders within the group, and the frame clause picks which ordered rows feed the function. Ranking and offset functions ignore the frame; aggregates such as sum and avg honour it.
SELECT id, customer_id, created_at, amount,
row_number() OVER w AS n, -- 1,2,3 with no ties
rank() OVER w AS rnk, -- 1,1,3 on ties
dense_rank() OVER w AS drnk, -- 1,1,2 on ties
lag(amount) OVER w AS prev_amount, -- NULL for the first row
lead(created_at) OVER w - created_at AS gap_to_next, -- interval in PostgreSQL
sum(amount) OVER (PARTITION BY customer_id ORDER BY created_at
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total,
avg(amount) OVER (PARTITION BY customer_id ORDER BY created_at
ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS moving_avg_7,
sum(amount) OVER (PARTITION BY customer_id) AS customer_total, -- whole partition
amount / sum(amount) OVER (PARTITION BY customer_id) AS share,
first_value(amount) OVER w AS first_amount,
ntile(4) OVER (ORDER BY amount) AS quartile
FROM orders
WINDOW w AS (PARTITION BY customer_id ORDER BY created_at); -- named window; inline the definition on MySQL < 8.0 or omitThe default frame with an ORDER BY is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, which includes every peer row with the same ordering value. For a running total that means rows tied on created_at all get the same total; use ROWS when each row should count individually. RANGE BETWEEN INTERVAL '7 days' PRECEDING AND CURRENT ROW (PostgreSQL 11+, MySQL 8) is a time-based frame. Window functions are supported in PostgreSQL, MySQL 8.0+, MariaDB 10.2+ and SQLite 3.25+.
CTEs and recursive CTEs#
A WITH clause names a subquery so the main query reads top down. In PostgreSQL 12+ a CTE referenced once is inlined into the plan; referenced more than once, or written WITH x AS MATERIALIZED (...), it is computed once and stored, which is an optimisation fence. MySQL 8 and SQLite also inline or materialise at the optimiser’s choice.
WITH paid AS (
SELECT customer_id, sum(amount) AS total
FROM orders
WHERE status = 'paid'
GROUP BY customer_id
), ranked AS (
SELECT c.region, c.name, p.total,
rank() OVER (PARTITION BY c.region ORDER BY p.total DESC) AS r
FROM paid p JOIN customers c ON c.id = p.customer_id
)
SELECT region, name, total FROM ranked WHERE r <= 3 ORDER BY region, r;A recursive CTE has an anchor query, UNION ALL (or UNION to stop on repeated rows), and a recursive query that references the CTE. The engine runs the recursive part on the rows produced by the previous step until a step produces nothing.
CREATE TABLE employees (id integer PRIMARY KEY, name text, manager_id integer REFERENCES employees(id));
WITH RECURSIVE chain AS (
SELECT id, name, manager_id, 1 AS depth, name AS path -- anchor: the root
FROM employees WHERE manager_id IS NULL
UNION ALL
SELECT e.id, e.name, e.manager_id, chain.depth + 1, chain.path || ' > ' || e.name -- CONCAT() on MySQL
FROM employees e
JOIN chain ON e.manager_id = chain.id
WHERE chain.depth < 20 -- guard against a cycle
)
SELECT * FROM chain ORDER BY path;
-- generate a series without a helper function (SQLite, MySQL); PostgreSQL has generate_series()
WITH RECURSIVE days(d) AS (
SELECT date '2026-01-01'
UNION ALL
SELECT d + 1 FROM days WHERE d < date '2026-01-31' -- date_add(d, INTERVAL 1 DAY) on MySQL, date(d, '+1 day') on SQLite
)
SELECT d FROM days;PostgreSQL 14+ adds CYCLE id SET is_cycle USING path to detect cycles without a depth guard. Recursion depth is limited by cte_max_recursion_depth (MySQL, default 1000) and by memory elsewhere; a cycle without a guard runs until it fails.
Subqueries versus joins#
A subquery in WHERE ... IN (...) or EXISTS (...) is a semi-join: it filters rows without multiplying them, which is exactly what a join does not do when the inner side has several matches. A correlated subquery in the SELECT list runs conceptually once per outer row; optimisers rewrite many of them into joins, but a LEFT JOIN on a pre-aggregated CTE is more predictable when the plan shows a nested loop over a large outer set.
-- semi-join: each customer once, however many paid orders they have
SELECT name FROM customers c
WHERE EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.id AND o.status = 'paid');
-- the join version needs DISTINCT to undo the multiplication
SELECT DISTINCT c.name FROM customers c JOIN orders o ON o.customer_id = c.id AND o.status = 'paid';
-- scalar subquery per row: fine for small outer sets
SELECT c.name, (SELECT max(created_at) FROM orders o WHERE o.customer_id = c.id) AS last_order
FROM customers c;
-- LATERAL (PostgreSQL 9.3+, MySQL 8.0.14+) runs a subquery per outer row and can return several columns or rows
SELECT c.name, o.id, o.amount
FROM customers c
CROSS JOIN LATERAL (
SELECT id, amount FROM orders WHERE customer_id = c.id ORDER BY created_at DESC LIMIT 3
) o;NOT IN (subquery) returns no rows at all if the subquery yields a single NULL, because x <> NULL is unknown for every row. NOT EXISTS has no such trap and is the correct anti-join.
UPSERT#
An upsert inserts a row or, when a unique constraint would be violated, updates the existing one atomically. Each engine spells it differently and the conflict target must be a unique index or primary key.
-- PostgreSQL 9.5+ and SQLite 3.24+
INSERT INTO customers (id, name, region)
VALUES (42, 'Acme', 'VIC')
ON CONFLICT (id) DO UPDATE
SET name = EXCLUDED.name, -- EXCLUDED is the row that failed to insert
region = coalesce(EXCLUDED.region, customers.region)
WHERE customers.name IS DISTINCT FROM EXCLUDED.name; -- skip no-op updates (IS NOT on SQLite)
INSERT INTO customers (id, name) VALUES (42, 'Acme') ON CONFLICT DO NOTHING; -- insert-if-absent
INSERT INTO customers (id, name) VALUES (42, 'Acme') ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name RETURNING id, (xmax = 0) AS inserted; -- PostgreSQL: was it an insert-- MySQL 8.0.19+ (row alias); MariaDB and older MySQL use VALUES(name)
INSERT INTO customers (id, name, region)
VALUES (42, 'Acme', 'VIC') AS new
ON DUPLICATE KEY UPDATE name = new.name, region = new.region;
INSERT INTO customers (id, name, region) VALUES (42, 'Acme', 'VIC')
ON DUPLICATE KEY UPDATE name = VALUES(name), region = VALUES(region); -- MariaDB 11.x; deprecated in MySQL 8.0.20+
INSERT IGNORE INTO customers (id, name) VALUES (42, 'Acme'); -- insert-if-absent, but also silences other errors: prefer ON DUPLICATE KEY UPDATE id = idON DUPLICATE KEY UPDATE fires on any unique index, so a table with several unique keys may update a different row than you expect, and with auto_increment it burns an ID on every conflict. REPLACE INTO (MySQL, SQLite) deletes and re-inserts, firing delete triggers and resetting columns you did not supply; use it only when that is the intent. PostgreSQL 15+ and SQL Server have MERGE, which handles insert, update and delete in one statement with WHEN MATCHED / WHEN NOT MATCHED clauses, but is not atomic against concurrent inserts the way ON CONFLICT is.
Indexes and reading EXPLAIN#
A B-tree index is a sorted copy of the indexed columns with pointers to rows. It serves equality and range predicates on a leftmost prefix of its columns, ORDER BY in the same column order, and (when it contains every column the query needs) an index-only scan. The optimiser uses an index when it estimates fewer rows than a sequential scan would cost, which is why the same query flips plans as the table grows or statistics go stale.
CREATE INDEX orders_customer_created ON orders (customer_id, created_at DESC); -- serves WHERE customer_id = ? ORDER BY created_at DESC
CREATE INDEX orders_paid_created ON orders (created_at) WHERE status = 'paid'; -- partial index: PostgreSQL, SQLite
CREATE UNIQUE INDEX customers_lower_name ON customers (lower(name)); -- expression index: PostgreSQL, SQLite; MySQL 8.0.13+ with ((lower(name)))
CREATE INDEX orders_customer_incl ON orders (customer_id) INCLUDE (amount); -- covering columns not in the key: PostgreSQL 11+An index on (a, b) does not help WHERE b = ? alone, and a function or cast on the column (WHERE lower(name) = ?, WHERE created_at::date = ?, WHERE id = '42' against an integer on MySQL) defeats a plain index. Leading-wildcard LIKE '%x' cannot use a B-tree. Every index costs write throughput and space; drop ones the statistics show unused (pg_stat_user_indexes, sys.schema_unused_indexes, PRAGMA index_list plus application knowledge).
EXPLAIN (ANALYZE, BUFFERS) -- PostgreSQL: run it and report real rows, times and I/O
SELECT * FROM orders WHERE customer_id = 42 ORDER BY created_at DESC LIMIT 10;
EXPLAIN ANALYZE -- MySQL 8.0.18+: tree format with actual rows and loops
SELECT * FROM orders WHERE customer_id = 42 ORDER BY created_at DESC LIMIT 10;
EXPLAIN QUERY PLAN -- SQLite: SCAN vs SEARCH ... USING INDEX
SELECT * FROM orders WHERE customer_id = 42 ORDER BY created_at DESC LIMIT 10;Read a plan from the innermost node outwards and look for: a sequential or full scan (Seq Scan, type: ALL, SCAN orders) over a large table where you expected an index; a large gap between estimated and actual rows, which means stale statistics (ANALYZE) or a correlation the planner cannot see; a Sort or Using filesort that an index in the right order would remove; a Nested Loop whose inner side is a scan repeated thousands of times; and Hash Join spilling to disk (Batches > 1). The PostgreSQL page covers plan nodes in depth. EXPLAIN ANALYZE executes the statement, including DELETE and UPDATE; wrap those in a transaction you roll back.
Transactions and isolation levels#
A transaction makes a group of statements atomic and durable. Isolation decides what a transaction sees of others running at the same time. The standard defines four levels by the anomalies they forbid; engines implement them with row versions (MVCC) and locks, and the defaults differ.
| Level | Dirty read | Non-repeatable read | Phantom | Write skew |
|---|---|---|---|---|
READ UNCOMMITTED | possible (PostgreSQL treats it as read committed) | possible | possible | possible |
READ COMMITTED | no | possible: two SELECTs can see different committed data | possible | possible |
REPEATABLE READ | no | no: snapshot taken at first read | PostgreSQL: no; MySQL InnoDB: no for reads, but locking reads see fresh rows | possible |
SERIALIZABLE | no | no | no | no: engine aborts one transaction with a serialisation error |
PostgreSQL defaults to READ COMMITTED, MySQL InnoDB to REPEATABLE READ, SQLite is always SERIALIZABLE because one writer holds the whole database. At READ COMMITTED the read-modify-write pattern SELECT balance; UPDATE balance = balance - 10 loses updates under concurrency; make the update atomic (UPDATE accounts SET balance = balance - 10 WHERE id = 1 AND balance >= 10) or lock the row first.
BEGIN;
SELECT balance FROM accounts WHERE id = 1 FOR UPDATE; -- blocks other FOR UPDATE / UPDATE on this row until COMMIT
UPDATE accounts SET balance = balance - 10 WHERE id = 1;
UPDATE accounts SET balance = balance + 10 WHERE id = 2;
COMMIT;
BEGIN ISOLATION LEVEL SERIALIZABLE; -- PostgreSQL; SET TRANSACTION ISOLATION LEVEL SERIALIZABLE on MySQL before BEGIN
-- ... on ERROR 40001 (PostgreSQL) or 1213/1205 (MySQL), retry the whole transaction
COMMIT;
SELECT * FROM jobs WHERE state = 'queued' ORDER BY id LIMIT 1 FOR UPDATE SKIP LOCKED; -- queue pattern: PostgreSQL 9.5+, MySQL 8
SAVEPOINT before_risky; ... ROLLBACK TO SAVEPOINT before_risky; -- partial rollback inside a transactionLock rows in a consistent order (by primary key) in every transaction to avoid deadlocks; the engine detects a deadlock and aborts one party, so application code must retry. Keep transactions short: an open transaction holds locks, blocks vacuum in PostgreSQL and pins undo in InnoDB. Autocommit means every statement is its own transaction, which is fine for single statements and wrong for a loop of dependent writes.
NULL semantics#
NULL is “unknown”, not a value. Any comparison with NULL yields unknown, WHERE treats unknown as false, and NOT unknown is still unknown.
SELECT NULL = NULL; -- NULL, not true
SELECT * FROM customers WHERE region <> 'VIC'; -- excludes rows where region IS NULL
SELECT * FROM customers WHERE region <> 'VIC' OR region IS NULL;
SELECT * FROM customers WHERE region IS DISTINCT FROM 'VIC'; -- PostgreSQL; SQLite: region IS NOT 'VIC'; MySQL: NOT (region <=> 'VIC')
SELECT coalesce(region, 'unknown'), nullif(region, '') FROM customers; -- nullif turns '' into NULL
SELECT count(*), count(region), sum(amount) FROM ...; -- aggregates skip NULL; sum of nothing is NULL
SELECT ... ORDER BY region NULLS LAST; -- PostgreSQL, SQLite 3.30+; MySQL sorts NULL first ascending, use ORDER BY region IS NULL, region
SELECT 1 FROM t WHERE x NOT IN (SELECT y FROM u); -- empty result if any y is NULLA UNIQUE constraint allows many NULLs in PostgreSQL, MySQL and SQLite by default; PostgreSQL 15+ has UNIQUE NULLS NOT DISTINCT to forbid that. GROUP BY and DISTINCT treat NULLs as equal to each other. String concatenation with || (or CONCAT on MySQL, || needs PIPES_AS_CONCAT) returns NULL if any operand is NULL; concat_ws skips them.
Date and time handling#
Store instants in UTC as timestamptz (PostgreSQL), TIMESTAMP or DATETIME(6) with the application converting (MySQL), or ISO-8601 text or integer epoch (SQLite, which has no date type). Store civil dates (a birthday, a due date) as date. Store local wall-clock times only with the IANA zone name next to them, because offsets change.
-- PostgreSQL
SELECT now(), current_date, now() AT TIME ZONE 'Australia/Melbourne';
SELECT date_trunc('month', created_at) AS month, sum(amount) FROM orders GROUP BY 1 ORDER BY 1;
SELECT created_at + interval '30 days', age(now(), created_at), extract(epoch FROM created_at);
SELECT * FROM orders WHERE created_at >= date '2026-09-01' AND created_at < date '2026-10-01'; -- half-open range uses the index
-- MySQL / MariaDB
SELECT NOW(), CURDATE(), CONVERT_TZ(created_at, 'UTC', 'Australia/Melbourne');
SELECT DATE_FORMAT(created_at, '%Y-%m-01') AS month, SUM(amount) FROM orders GROUP BY month;
SELECT DATE_ADD(created_at, INTERVAL 30 DAY), TIMESTAMPDIFF(DAY, created_at, NOW()), UNIX_TIMESTAMP(created_at);
-- SQLite
SELECT datetime('now'), date('now', 'localtime'), strftime('%Y-%m-01', created_at) AS month;
SELECT datetime(created_at, '+30 days'), julianday('now') - julianday(created_at) AS days_ago, unixepoch(created_at);Filter with half-open ranges (>= start AND < end) rather than BETWEEN, which includes the upper bound and misses or double counts depending on precision. date(created_at) = '2026-09-01' applies a function to the column and defeats the index; created_at >= '2026-09-01' AND created_at < '2026-09-02' does not. CONVERT_TZ on MySQL returns NULL until the time zone tables are loaded (mysql_tzinfo_to_sql).
Pagination#
OFFSET n reads and discards n rows every time, so page 1000 costs a thousand pages of work and rows inserted between requests shift the pages. Keyset pagination remembers the last row’s sort key and asks for what follows, which an index on the sort columns answers directly. The sort must be total: add the primary key as a tiebreaker.
-- first page
SELECT id, created_at, amount FROM orders
WHERE customer_id = 42
ORDER BY created_at DESC, id DESC
LIMIT 50;
-- next page: pass the last row's (created_at, id); row-value comparison works in PostgreSQL, MySQL 8, SQLite
SELECT id, created_at, amount FROM orders
WHERE customer_id = 42
AND (created_at, id) < (:last_created_at, :last_id)
ORDER BY created_at DESC, id DESC
LIMIT 50;
-- portable expansion of the row comparison
WHERE customer_id = 42 AND (created_at < :last_created_at OR (created_at = :last_created_at AND id < :last_id))Offset pagination is fine for a UI with page numbers over a few thousand rows; keyset is the only option for APIs and jobs walking large tables. count(*) for a total page count is a full scan or index scan; estimate it (pg_class.reltuples, information_schema.tables.table_rows) or drop it.
Anti-patterns#
| Pattern | Problem | Instead |
|---|---|---|
SELECT * in application code | Breaks on schema change, drags large columns, blocks index-only scans | Name the columns |
WHERE lower(email) = lower(?) | Function on the column defeats the index | Store normalised, or an expression index |
Implicit type conversion (id = '42' on MySQL varchar vs int) | Index unusable, silent full scan | Match the column type in the parameter |
OFFSET 100000 | Linear cost per page | Keyset pagination |
NOT IN (subquery) | Empty result on a NULL; poor plans | NOT EXISTS |
DISTINCT to hide join duplication | Wrong cardinality treated with a hammer | Semi-join with EXISTS, or aggregate first |
| N+1 queries from an ORM | One round trip per parent row | Join or IN (...) batch, or LATERAL |
| Storing CSV in a column | Cannot index, join or constrain | A child table, or an array/JSON column with an index if the engine supports it |
| Entity-attribute-value tables | Every query is a self-join festival, no types | Real columns, or JSON for genuinely sparse attributes |
| Floats for money | Rounding errors | numeric(12,2) / DECIMAL, or integer cents |
BETWEEN on timestamps | Inclusive upper bound | >= start AND < end |
| Long-running open transaction | Locks held, vacuum blocked, undo growth | Commit early; do not hold a transaction across user interaction |
SELECT ... FOR UPDATE on many rows | Serialises unrelated work | Lock only the rows you change, in key order |
| String-built SQL | Injection and plan cache misses | Parameterised statements |
Reusable queries#
-- duplicates by a key, with the count
SELECT name, region, count(*) FROM customers GROUP BY name, region HAVING count(*) > 1;
-- delete duplicates, keep the lowest id (PostgreSQL, SQLite; on MySQL use a self join or a temp table because a subquery cannot target the same table)
DELETE FROM customers
WHERE id IN (
SELECT id FROM (
SELECT id, row_number() OVER (PARTITION BY name, region ORDER BY id) AS n FROM customers
) d WHERE n > 1
);
-- gaps in a sequence of ids
SELECT id + 1 AS gap_start, next_id - 1 AS gap_end
FROM (SELECT id, lead(id) OVER (ORDER BY id) AS next_id FROM orders) s
WHERE next_id > id + 1;
-- islands: consecutive days with at least one order, grouped into runs
WITH days AS (SELECT DISTINCT date(created_at) AS d FROM orders),
grp AS (SELECT d, d - (row_number() OVER (ORDER BY d))::int AS g FROM days) -- MySQL: DATE_SUB(d, INTERVAL ROW_NUMBER() OVER (...) DAY)
SELECT min(d) AS run_start, max(d) AS run_end, count(*) AS days FROM grp GROUP BY g ORDER BY 1;
-- top 3 orders per customer
SELECT * FROM (
SELECT o.*, row_number() OVER (PARTITION BY customer_id ORDER BY amount DESC, id) AS n FROM orders o
) t WHERE n <= 3;
-- latest row per group (PostgreSQL shortcut)
SELECT DISTINCT ON (customer_id) * FROM orders ORDER BY customer_id, created_at DESC;
-- latest row per group, portable, using an anti-join
SELECT o.* FROM orders o
LEFT JOIN orders newer ON newer.customer_id = o.customer_id AND (newer.created_at, newer.id) > (o.created_at, o.id)
WHERE newer.id IS NULL;
-- pivot: one row per region, one column per status
SELECT c.region,
sum(CASE WHEN o.status = 'pending' THEN o.amount ELSE 0 END) AS pending,
sum(CASE WHEN o.status = 'paid' THEN o.amount ELSE 0 END) AS paid,
sum(CASE WHEN o.status = 'cancelled' THEN o.amount ELSE 0 END) AS cancelled
FROM orders o JOIN customers c ON c.id = o.customer_id
GROUP BY c.region;
-- unpivot: columns back into rows (PostgreSQL; MySQL and SQLite use UNION ALL of one SELECT per column)
SELECT region, v.status, v.amount
FROM region_totals r
CROSS JOIN LATERAL (VALUES ('pending', r.pending), ('paid', r.paid), ('cancelled', r.cancelled)) AS v(status, amount);
-- month-over-month change
WITH m AS (SELECT date_trunc('month', created_at) AS month, sum(amount) AS revenue FROM orders GROUP BY 1)
SELECT month, revenue, revenue - lag(revenue) OVER (ORDER BY month) AS delta,
round(100.0 * (revenue - lag(revenue) OVER (ORDER BY month)) / lag(revenue) OVER (ORDER BY month), 1) AS pct
FROM m ORDER BY month;
-- rows in one table missing from another (both directions)
SELECT id FROM staging EXCEPT SELECT id FROM orders; -- MySQL 8.0.31+ and MariaDB support EXCEPT; older MySQL: NOT EXISTS
SELECT id FROM orders EXCEPT SELECT id FROM staging;
-- percentile
SELECT percentile_cont(0.95) WITHIN GROUP (ORDER BY amount) FROM orders; -- PostgreSQL; MySQL: window PERCENT_RANK or ntile approximation
-- histogram in buckets of 100
SELECT (amount / 100)::int * 100 AS bucket, count(*) FROM orders GROUP BY 1 ORDER BY 1; -- FLOOR(amount / 100) * 100 on MySQL and SQLite
-- cumulative distinct users by day
SELECT d, count(*) OVER (ORDER BY d) AS cumulative_customers
FROM (SELECT customer_id, min(date(created_at)) AS d FROM orders GROUP BY customer_id) f;
-- random sample of about 1% (PostgreSQL); ORDER BY RAND() LIMIT n on MySQL and SQLite is a full sort
SELECT * FROM orders TABLESAMPLE SYSTEM (1);
-- lock a batch for processing, others skip it
UPDATE jobs SET state = 'running', worker = :worker
WHERE id IN (SELECT id FROM jobs WHERE state = 'queued' ORDER BY id LIMIT 10 FOR UPDATE SKIP LOCKED)
RETURNING id; -- MySQL 8: no RETURNING; select the ids first inside the transaction
-- rows updated in the last hour and who changed them, for a table with audit columns
SELECT id, updated_by, updated_at FROM orders WHERE updated_at >= now() - interval '1 hour' ORDER BY updated_at DESC;Troubleshooting#
| Symptom | Cause | Fix |
|---|---|---|
| Totals too large after adding a join | One-to-many join multiplied rows before aggregation | Aggregate each side in a CTE, then join; check count(*) per join step |
LEFT JOIN returns only matched rows | Filter on the right table placed in WHERE | Move it into ON, or add OR right.col IS NULL |
NOT IN returns nothing | NULL in the subquery result | NOT EXISTS, or WHERE y IS NOT NULL in the subquery |
column must appear in the GROUP BY clause | Non-aggregated column not in the group key | Add it to GROUP BY, wrap in an aggregate, or use a window function |
window functions are not allowed in WHERE | Logical order: windows evaluate after WHERE | Compute in a subquery or CTE and filter outside |
| Query fast in one environment, slow in another | Different statistics, data volume or missing index | Compare EXPLAIN ANALYZE; run ANALYZE; check the index exists |
| Index exists but the plan scans | Function or cast on the column, wrong leading column, or low selectivity | Rewrite the predicate to the bare column; reorder or add an index |
deadlock detected / Deadlock found when trying to get lock | Two transactions lock rows in opposite order | Lock in primary-key order; retry on error 40001 / 1213 |
could not serialize access at SERIALIZABLE | Legitimate conflict detected | Retry the transaction; that is the contract of the level |
| Lost update: counter lower than expected | Read-modify-write at READ COMMITTED | Atomic UPDATE ... SET x = x + 1, or SELECT ... FOR UPDATE |
| Duplicate rows despite an application check | Check-then-insert race | Unique constraint plus upsert or handle the violation |
BETWEEN misses the last day’s rows | Inclusive upper bound at midnight | >= start AND < end + 1 day |
| Page contents shift between requests | OFFSET over a changing table | Keyset pagination with a total ordering |
Illegal mix of collations (MySQL) | Comparing columns with different collations | Same CHARACTER SET and COLLATE on both, or COLLATE utf8mb4_0900_ai_ci in the comparison |
| Text sorts or compares unexpectedly | Collation: case or accent insensitivity, or C locale byte order | Check the column collation; use COLLATE explicitly |