How Microcosm builds UK household data

A microsimulation dataset represents households through records of their members, relationships, incomes and other characteristics. Each household has a weight: the number of households it represents in the population. PolicyEngine applies tax and benefit rules to these records and uses their weights to estimate population outcomes. Those rules are modelled in PolicyEngine-UK rather than in this pipeline; the PolicyEngine-UK documentation lists which are covered.

Microcosm starts with the Family Resources Survey (FRS) and adds income, wealth and spending information from other sources. This imputation gives the records their characteristics; calibration, described below, then changes only how many households each record represents. Throughout, Microcosm keeps the links between people, their benefit units and their households. A benefit unit is a single adult or a couple and any dependent children: the family group that means-tested benefits such as Universal Credit assess together.

The completed records then follow one of two routes. The national route calibrates their weights to UK, country and regional statistics. The local route first copies each household and places every copy in a small census area. In Great Britain, each copy then takes the Broad Rental Market Area its area lies in and, if it rents privately, a rent from that market area; in England and Wales, owner-occupied copies also take a home value and council-tax band that fit their area. The local route then calibrates one set of weights against national and local statistics together. A local release can hold either every copy or an exact-size selection of households (150,000 at this revision), whose weights are refitted to the same targets; the selection must pass the same release checks. [2]

PolicyEngine uses both files of a release. Results for the UK and for England, Scotland, Wales and Northern Ireland come from the national file; a country’s results use the households whose region lies in it. Results for a constituency or local authority come from the local-area file, using the copies assigned to that area. The two files start from the same household records and the same pinned statistics but are calibrated separately, so their weights differ: constituency results from the local-area file add up to that file’s national total, which can differ from the national file’s. The local-area file is not the default dataset, but it is certified together with the national file. [1]

• Imputation: giving a household a characteristic the FRS does not record well, drawn from similar households in another source.
• Calibration: adjusting weights so that weighted totals match official statistics, called targets.
• Holdout: a target left out of a fit, to check how well the result matches it.
• Effective sample size: how many equally weighted records an area’s weights are worth; it falls when a few records carry much of the weight.

The FRS supplies the household structure and reported characteristics. Other sources add information that the FRS does not collect in enough detail or represent adequately on its own. The source manifest pins the files and versions a build must use; the vintages below are those of the implementation this page describes, in which 49 ordered stages build the household records that both routes share. [3]

The FRS 2024–25 achieved sample was 16,299 households across the UK (DWP), of which about 1,900 were in Scotland (Scottish Government). Every record in the dataset, including the copies described below, starts from one of these households.

Additional stages use:

• the National Travel Survey for bus use;
• the Effects of Taxes and Benefits dataset for VAT and for the value of education, rail and bus subsidies;
• National Health Service (NHS) activity and cost tables for health-service use;
• published take-up statistics for whether eligible families claim benefits: DWP’s counts of households on Universal Credit by family type and award amount, its Move to Universal Credit statistics for families moved from older benefits, and the Scottish Government’s take-up rates for Scottish Child Payment. No rate is published for carers whom a reform or a later year newly entitles to Carer’s Allowance (Carer Support Payment in Scotland), so their take-up is an assumed rate, informed by the Independent Review of Carer’s Allowance Overpayments and DWP statistics;
• house-price statistics to update property values, and HM Land Registry sale prices for owner-occupiers’ home values in England and Wales;
• HMRC’s Annual Stamp Tax Statistics, for which households buy property during the year and at what price;
• HMRC’s Property Rental Income Statistics, with the Survey of Personal Incomes, for landlords’ rental receipts and mortgage-interest costs;
• published statistics and research for capital gains, pension salary sacrifice and student loans;
• planned for the next release, the Department for Education’s rate of apprentices in learning in England, applied across the UK to flag employees aged 21 to 24 as apprentices, which the FRS does not record;
• planned for the next release, HMRC’s alcohol and tobacco clearances and gambling duty figures, shared out in proportion to each household’s Living Costs and Food Survey spending so that PolicyEngine-UK can calculate these duties, and the Gambling Commission’s Gambling Survey for Great Britain 2025 for how many households gamble; OBR’s forecasts of these duties are measured as checks rather than fitted.

The manifest records each stage’s coverage and assumptions. For example, recent National Travel Survey microdata cover England only, so bus use in Scotland, Wales and Northern Ireland is predicted as if households lived in a stand-in English region. Bus fares in Scotland and Northern Ireland are then priced from their own published statistics, while Welsh fares keep their survey-based values. [3]

Microcosm imputes values conditional on shared characteristics such as age, sex, income, household composition, tenure and region; the predictors differ by stage. Its main method, weighted quantile regression forests, predicts the range of values a characteristic takes among similar households; the imputation working paper explains this choice and compares it with alternatives such as hot-deck matching and linear regression. Most values are drawn at random from that range rather than set to its average, so otherwise similar households still differ. Incomes drawn from the Survey of Personal Incomes are the exception: each copied adult takes the income at the same position among similar taxpayers that the original adult’s own income holds among similar FRS respondents. The draws do not link an FRS household to the same identifiable household in another survey.

The income stage keeps the incomes reported on the original FRS records. It also adds copies of a random sample of FRS households. Adults in those copies who have taxable income, such as pay, self-employment income, a pension or more than £1,000 of investment and other income, take incomes from the Survey of Personal Incomes; other adults keep their FRS values. The copies’ housing costs, and the benefits of adults given new incomes, are then redrawn to match. Because the FRS contains few households with very high incomes, further copies give one adult an income drawn within one of HMRC’s income bands from £200,000 upwards, with starting weights set from HMRC’s published taxpayer counts. These records give calibration enough high-income households to work with. [3]

HMRC’s income statistics count income received over a tax year, while the FRS records each adult’s current income. A further stage therefore describes each adult’s previous 12 months from the FRS’s own questions on jobs, pensions and benefits, for example a job that ended during the year, so that HMRC targets are compared with annual income. Tax, benefit and poverty calculations still use current income. [3]

FRS landlords report rental profit after mortgage interest. A later stage adds back an estimate of that interest, drawn from the Survey of Personal Incomes, and sets their rental receipts from HMRC’s Property Rental Income Statistics.

In England and Wales, owner-occupiers’ home values are set from HM Land Registry sale prices in their region, uprated with the UK House Price Index, at the household’s rank among owners there; the local build later replaces them with values from each copy’s constituency. A further stage draws which households buy property during the year, and at what price, so that purchases in England and Northern Ireland, where Stamp Duty Land Tax applies, match HMRC’s stamp-tax statistics.

Owner-occupied households in the top 5 per cent of home values in their region are split into six records of equal weight, so that the local build can place expensive homes in more areas. Added records take their starting weights from the households they copy or from their region, so the total starting weight is unchanged, and they draw their own childcare take-up rather than sharing their source household’s draw. Wealth and spending stages then draw related variables in sequence.

The FRS identifies a household’s region but does not provide a representative sample for every constituency. In the local build, Microcosm makes 15 copies of each completed household, keeping its people and family relationships together, and splits the household’s weight equally among them. Each copy receives one small census area within its original FRS region, drawn with probability proportional to the area’s census household count. The draws are reproducible: each is tied to the record’s identity, so rebuilding with the same inputs gives the same assignment. [2]

The assignment uses 2021 Census Output Areas in England and Wales, 2022 Census Output Areas in Scotland (the building blocks of Scotland’s own Data Zones) and Northern Ireland’s 2021 Census Data Zones. Published, versioned lookups map each area to one constituency, one local authority and other geographic labels; in Great Britain, an area that straddles a boundary is assigned wholly to one side by the official best-fit lookup. The constituency geography uses the 2024 boundaries; the local-authority geography uses the April 2023 definition. [4]

In England and Wales, each owner-occupied copy then takes a home value from HM Land Registry sale prices in its assigned constituency (the latest sale of each home since 1995, uprated with the UK House Price Index), at the household’s rank among owners in its region, and a council-tax band from the Valuation Office Agency’s band mix for its assigned Lower Layer Super Output Area (LSOA). Renters, and households in Scotland and Northern Ireland, keep the home values and council-tax bands in their household records. The council-tax bill still comes from the source household. [2]

In Great Britain, each copy also takes the Broad Rental Market Area (BRMA) that contains its small area, so the Local Housing Allowance rates PolicyEngine-UK applies match the copy’s location. Each private renter’s rent is then read from the published list of rents for homes of its size in that BRMA (the rent officers’ lists in England and Wales, and Scottish Government rent statistics in Scotland), at the household’s rank among private renters with homes of that size in its region, and rescaled so that each region keeps its FRS average rent. Social renters keep their rents. Northern Ireland has no open BRMA boundaries, so households there keep the BRMA drawn within their region, and their rents. [2]

Each resulting household record has one location and one weight. The same record and weight contribute to its constituency, its local authority and the national total. Constituencies and local authorities overlap, so Microcosm derives both labels from the small area rather than assuming that one sits inside the other.

These assignments create a modelled local population. They do not recover respondents’ actual addresses or establish that a record describes a particular small area accurately. The local calibration described below fits constituency and local-authority statistics; small-area assignment alone does not validate ward or neighbourhood estimates.

For each target statistic, Microcosm calculates every household’s contribution and compares the weighted total with the official value. A target might count people in an age band, sum employment income among taxpayers, or count families receiving Universal Credit. The local build fits national, regional, constituency and local-authority targets in one optimisation. [5]

Published local cells too small to fit reliably are not fitted directly but still count towards their region’s total: council-tax band cells of fewer than 50 dwellings, higher-rate taxpayer cells below 5,500, which HMRC rounds to the nearest thousand, and local-authority cells of fewer than 400 high-value homes. National and regional targets include population, household, income, tax and benefit statistics, along with housing, student-loan, spending and other administrative totals. Targets have defined populations and reference periods; HMRC income statistics, for example, describe taxpayers rather than all residents. Targets refer to 2025, and the build records which targets it fitted, which it excluded, and how it brought source values to that year. Available targets and coverage differ across the UK’s countries. [5]

Scotland enters the national calibration as one of the twelve regional rows: for population by age; for HMRC taxpayers, total income and income tax by income band, scaled so that the twelve regions add up to HMRC’s UK totals; for higher- and additional-rate taxpayers; and for Capital Gains Tax taxpayers and gains. It also enters through Scotland-specific statistics:

• National Records of Scotland’s household estimate, and the Scottish Government’s council-tax stock by band, whose total is set to that estimate;
• Scottish Child Payment spending, and Scottish Government statistics on bus passenger revenue and government support for buses;
• ONS counts of children under 16 and babies under one in Scotland;
• DWP’s count of Scottish Universal Credit households with a child under one, and its State Pension caseloads for Scotland.

OBR’s council-tax receipts are fitted only as one Great Britain total, which includes Scotland’s.

Locally, Scotland’s 32 councils and 57 constituencies take the age, HMRC employment and self-employment income, Universal Credit and census household-count targets. The councils also take tenure, council-tax bands and DESNZ electricity and gas use, and, planned for the next release, the constituencies take electricity and gas use too; there are no gas targets for Orkney and Shetland. Private rents are fitted for Scotland’s 18 Broad Rental Market Areas rather than its councils. There are no Scottish higher-rate taxpayer or high-value-home targets, and some UK benefit totals exclude Scotland; see the limits under Evidence. [5]

Published totals for the same concept can differ across geographic levels because of reference periods, definitions or census disclosure controls. Before fitting weights, both builds reconcile targets that measure the same thing at more than one level:

• where a total is fitted for the UK, Great Britain, England and Wales, a nation or a region, the targets beneath it are rescaled to match it, so regional rows add up to national totals and local rows to the regional ones;
• for household counts, each nation’s own household estimate takes precedence, and counts that split an area’s total, such as tenure, keep their shares of it;
• where there is no higher total, local-authority targets are rescaled to the constituency totals in the same region or nation;
• every adjustment is recorded.

The rule applies only to matching measurements and to relationships declared, with reasons, in one reviewed file; the build stops if a local target has no declared route to a higher total or two overlapping targets are not declared. It does not combine statistics with incompatible populations. [6]

A gradient-based optimiser adjusts the weights to reduce the relative error between weighted totals and their targets; a calibration working paper, in preparation, will compare this approach with classical methods such as raking. Each target’s contribution is capped, so a single badly missed target cannot dominate the fit. In the local build, the objective is shared equally among four groups: targets for the UK, Great Britain or England and Wales; targets for the four nations and the English regions; constituency targets; and local-authority targets. Within each group, every family of statistics, such as age bands or council-tax bands, takes an equal share, and within a family larger counts carry more weight, in proportion to the square root of their size. This keeps the many local targets from overwhelming the national ones. The national build gives each family of statistics an equal share. In both builds, no calibrated weight can exceed ten times its weight at the start of calibration; this is a hard limit applied throughout the fit, with no matching lower limit. The total number of households is fitted as a target rather than held fixed. For an exact-size local release, Microcosm first chooses the households with an L0 penalty, which favours a smaller set of households that can still fit the targets (the L0 working paper describes the method and tests it on US data). The selected households’ weights are then refitted to the same targets under the same limits, with a penalty that keeps the weights close to equal. These constraints can leave residual discrepancies, which the diagnostics report. [7]

For a policy comparison, PolicyEngine calculates baseline and reform outcomes for the same household records. An area’s total income change sums each assigned household’s change multiplied by its weight. With the same dataset and population definition, constituency totals sum to that dataset’s national weighted total; in PolicyEngine, that is the local-area file’s total, not the national file’s. An average divides by the relevant weighted household count; averages and percentages do not add across areas.

Enhanced FRS, built in the policyengine-uk-data repository, was the UK dataset PolicyEngine certified for policy analysis until policyengine.py 6.10.0 replaced it with Microcosm on 21 October 2026; its last certified release was 1.58.1. Like Microcosm, it starts from the FRS 2024–25 and draws on the same families of surveys for income, wealth and spending. It can still be loaded by name, and it remains Microcosm’s benchmark. A national release can be certified only if it fits a shared set of official statistics better than a pinned Enhanced FRS file, and both builds block a release when the weighted total of an input drifts from a pinned Enhanced FRS reference by more than a reviewed tolerance. [1]

For local estimates, Enhanced FRS 1.58.1 fits two separate weight matrices, one for constituencies and one for local authorities. In each, a household carries a separately fitted weight for every area in its country, while its characteristics, and the taxes and benefits calculated from them, stay the same in every area. That release also gives each household one output area within its FRS region, but the local weight matrices are not tied to that location. Since version 6.10.0, policyengine.py no longer loads these matrices; constituency and local-authority results use Microcosm’s local-area file instead. [8]

For example, a household outside London can contribute to a London constituency estimate while keeping the outside-London benefit cap, because PolicyEngine-UK reads the household’s region when calculating the cap. [9] In Microcosm, each copy of a household stays in its original FRS region and carries one weight, so region- and country-based rules match the areas the copy counts towards. Microcosm also sets PolicyEngine-UK’s local-authority input from the assigned area; in Great Britain, the Broad Rental Market Area used for Local Housing Allowance and private renters’ rents; and, for owner-occupied homes in England and Wales, the home value and council-tax band. The council-tax bill still comes from the source household and is not recalculated when the band changes, and in Northern Ireland the Broad Rental Market Area is still drawn within the household’s region.

Microcosm fits national, regional, constituency and local-authority targets with one weight per record in a single optimisation, whereas Enhanced FRS fits its constituency and local-authority matrices separately, each alongside national targets. Microcosm also places 15 copies of each household so that each area has more records to weight, adds census household-count targets, reconciles overlapping targets across levels under a declared order, limits how far any weight can rise, and records local support and release checks. These changes improve geographic consistency and make the assumptions easier to inspect. They do not, on their own, establish that every local estimate is more accurate. A national release must fit shared targets better than Enhanced FRS, but for a local release that comparison of fit is optional, and a release that skips it records that it makes no such comparison. [1]

Calibration diagnostics report discrepancies for individual targets and groups of targets. A headline percentage of targets within a tolerance describes fit to those targets. It does not measure the accuracy of every household characteristic or policy result.

The local pipeline can also run rotated holdouts, which a local release may include but does not have to. A holdout run refits the weights five times, each time leaving out one of five groups of local targets and measuring how well the refit matches them, while national targets stay in every fit; each local target is held out once. For an exact-size release, each rotation also reruns the selection. This checks performance on local statistics that the refit did not directly target. It is not fully independent external validation, because the omitted targets can share sources and information with the fitted targets. A release that does not run the holdouts records that it makes no out-of-sample claim. [10]

Support diagnostics count records, distinct source households, and effective sample size within each area. Effective sample size summarises weight concentration: it falls when a few records carry much of an area’s weight. Copied households do not create additional independent survey observations. Record counts and effective sample size should therefore be read alongside the number of underlying source households. [10]

The build measures the number of distinct source households and the effective sample size for each constituency and local authority. An area below the support floors listed next blocks a local release unless a reviewed exclusion names the area and the floors it misses. Treat estimates for areas with few source households, and for narrowly defined groups within any area, as indicative. The dataset does not provide confidence intervals.

Release checks assess source provenance, imputation stages, household relationships, geography, calibration quality, local support and the exported dataset. In both builds, every national target must be within 25 per cent of its value unless a reviewed exclusion names it. A national release must also fit a shared set of official statistics better than a pinned Enhanced FRS file. A local release is also blocked unless:

• every constituency and local authority has at least 50 records with positive weight, an effective sample size of at least 50 and at least 50 distinct source households, unless a reviewed exclusion names the area;
• every fitted local target is within 25 per cent of its value, unless a reviewed exclusion names it, and at least half of the targets in every family of five or more are within 25 per cent;
• no weight is more than 100 times the median weight, and the effective sample size is at least 1 per cent of the records;
• every copy’s geography is consistent and its Broad Rental Market Area is one PolicyEngine-UK recognises.

Passing software tests or completing an optimisation does not establish that a dataset has passed these checks. Any published fit figures must identify the dataset release, target scope and diagnostic evidence they describe. [1] [10]

• Finite survey support. Rare household types and narrowly defined groups can have little underlying evidence, even after copying and calibration.
• Imputed relationships. Characteristics drawn from different sources depend on the predictors, coverage and assumptions used to connect them.
• Untargeted outcomes. Fitting population, income or Universal Credit counts does not guarantee local accuracy for disability, savings, every benefit or every reform.
• Inputs that do not follow the assigned area. Household characteristics such as income, wealth and spending, whether reported or imputed, come from the source household: imputation runs before areas are assigned, and the area drawn does not depend on them. In Great Britain, copies do take their Broad Rental Market Area and private rents from their area, and in England and Wales owner-occupied copies take a home value and council-tax band from it, but their council-tax bill still comes from the source household.
• Scotland. Adult and Child Disability Payment are read from the FRS and treated like PIP and DLA, and Pension Age Disability Payment and Carer Support Payment arrive under the Attendance Allowance and Carer’s Allowance codes, but no Scottish totals for these payments are targeted. Salary-sacrifice relief targets split by income-tax rate band are fitted using each taxpayer’s Scottish or rest-of-UK bands. Higher-rate taxpayer counts are fitted by constituency and local authority outside Scotland only, and Scottish owner-occupied copies keep the home values and council-tax bands in their household records, although Scottish copies do take their Broad Rental Market Area and private rents from their area. Scottish council-tax bills exclude water and sewerage charges, which are modelled separately, and council-tax income after reduction is checked against Scottish Local Government Finance Statistics but not targeted. Private rents are fitted for Scotland’s Broad Rental Market Areas, not its councils. [5] [10]
• Northern Ireland. There are no local Universal Credit or council-tax targets, household wealth is imputed using the pattern for Wales, and the Broad Rental Market Area, rents and home values come from the household records rather than the assigned area.
• Model and policy assumptions. Local impacts also depend on tax and benefit rules, benefit take-up and the policy comparison. The dataset pipeline does not itself establish behavioural responses or confidence intervals.

This draft covers the FRS 2024–25 implementation at Microcosm revision 90ff43f9575fb5ae5c0c4389ba4790ab4ade5722, which locks PolicyEngine-UK 2.162.8. It does not report measured performance for a named dataset release. A reproducible policy analysis should also identify its dataset release, PolicyEngine-UK version, policy year and reform.

The pipeline code is public under the MIT licence. The survey microdata it uses (FRS, SPI, WAS, LCFS, NTS and ETB) are available to registered users from the UK Data Service under its licence conditions, so rebuilding the dataset requires your own access; publishing the methodology does not make respondent-level data redistributable. The released files are derived from these licensed surveys and are not publicly downloadable. Questions and corrections: open an issue on the Microcosm repository. [3]