The problem: Social assistance benefits are a crucial pillar of the welfare state in many EU countries as they ensure a minimum standard of living to citizens with low income. In December 2022, 394 thousand people received a social assistance benefit in the Netherlands. To design effective welfare policies and simultaneously maintain sound public finances, governments need accurate figures (i.e. Official Statistics) on (1) the number of individuals receiving social assistance benefits, and (2) the number of individuals moving out of social assistance within a certain period (e.g. one year). With these numbers at hand, governments want to ensure that people in need receive welfare benefits, but also that welfare-recipients move quickly to employment so that fiscal expenditure on the welfare state is restrained. Moreover, by achieving quick transitions from welfare to employment, governments want to reduce skill obsolesce and the discouraged-worker effect. Producing official Statistics on social assistance is the task of NSIs, such as CBS1 with the use of register data.
Table 1: Glossary
| National Statistical Institute | NSI |
Statistics Netherlands | CBS |
Employee Insurance Agency | UWV |
| Hidden Markov Model | HMM |
| General Statistical Business Process Model | GSBPM |
| Innovation Exchange Amsterdam | IXA |
Table 2: Inconsistencies in receiving social assistance benefit on 1/1/2021
| Municipal register | UWV register | Total | ||
| Yes | No | |||
| Yes | 99.85% | 0.15% | 100% | |
| No | 24.28% | 75.72% | 100% | |
Currently, policy making on social assistance benefits in the Netherlands is based on official statistics that – although they meet the professional standards of the CBS – still include measurement error. This error is typically the result of administrative delays in registration, differences in conceptual definitions or processing errors2. Measurement error, if not properly dealt with, can lead to a distorted view of the number of people receiving social assistance benefits as well as mobility in- and out of social assistance. In turn, this could lead to suboptimal welfare policies.
The standard manual methods that the CBS applies to improve data quality, such as ‘data validation and data editing’3 have always been time-consuming and expensive. Although the process of automatic data validation that CBS has introduced in the past decades – that resulted in tremendous efficiency gains with respect to time and other resources – and produces the so-called ‘tidy data’ is still imperfect. The reason is that this automatic validation is mainly focused on the so-called visible error4, which is the error that can be detected by combining information from a single dataset with deductive rules (e.g. if in the item receiving social assistance benefit, the information ‘receiving’ is reported, while in the item total personal income, the individual appears to receive no income). In other words, some error remains unremedied with data validation. COMBINE provides a solution for this type of error that can only be revealed by linking data from multiple sources. Such error is called invisible. In the case of social assistance benefits, such invisible measurement error is revealed by linking validated municipal register data – which is currently used to produce official statistics on social assistance – to validated national UWV register data for the same individuals. As illustrated in Table 2, the two registers provide diverging information for the same individuals at the same moment in time. Notably, almost 25% of individuals that, on 1/1/2023, are registered as not receiving social assistance benefits in the municipal register are registered as receiving benefits in the UWV register! This actually means that, at least for these individuals, one or even both data sources contain incorrect information (measurement error).
Measurement error threatens the integrity of official statistics5 and therefore also the effectiveness of policies based on these statistics. Most importantly, even a small amount of measurement error in a variable can lead to large error in transitions/change over time.6 In my previous work concerning the type of the employment contract7, I show that almost half of the observed transitions from temporary to permanent employment in the Netherlands is not real.[1] This means that this error could affect policy-making as the presumption that the labour market is very mobile is incorrect, since, in reality, mobility to permanent employment is shown to be very low. A similar finding for social assistance benefits would lead policy-makers wrongly presume that people do not receive these benefits for long and existing policies are successful in helping social assistance recipients move to employment.
The solution: error-corrected statistics in social assistance benefits
COMBINE tests, applies and evaluates a novel model-based error-detection and correction method on official statistics that concerns one of the most important elements of social welfare polices: social assistance benefits. COMBINE produces error-corrected figures on (1) the number of individuals receiving a social assistance benefit at a given point in time and on (2) the transition rate out of social assistance between two points in time. Furthermore, COMBINE illustrates the potential of applying this method for correcting other official statistics that are published by National Statistical Institutes and used in policy-making throughout Europe.
The error correction method involves the application of HMMs. HMMs are statistical models that approximate categorical variables that are incorrectly measured with observed data. In our case, HMMs estimate the two figures on social assistance benefits that we are interested in. First, HMMs use two observed measures of social assistance benefits to approximate the error-corrected number of individuals who receive social assistance benefits at every moment in time. The two observed measures help to triangulate “true” information and reduce the effect of measurement error that exists in each of them. By doing so, there is no need any more of knowing the exact cause of measurement error. The two measures of social assistance benefits come from the municipal register and the UWV register. Second, HMMs estimate the transition rate from receiving social assistance benefits to other states (e.g. employment) and vice versa using the error-corrected measure that was approximated before. To do so, HMMs require several (at least three) observations at different points in time per individual, which are available in our case in both registers
The origin:
In a nutshell, as illustrated in Figures 1 and 2, COMBINE applies to official statistics the error-correction method that was developed in the ERC CoG project DYNANSE. In DYNANSE, we developed the HMM that can be used for error detection and correction. DYNANSE applied this method on the employment contract type and used data from a survey (Labour Force Survey) and a register (UWV register). COMBINE tests this method further, applies it in the production of official statistics and illustrates its significance for policymaking and society. Specifically:
COMBINE will lead to a breakthrough by introducing two major innovations:
These more reliable and more efficiently produced official statistics will serve as improved input for evidence-based policy making in social welfare. Moreover, once tested in the CBS environment, the method can be further applied to other official statistics produced by NSIs throughout the world and be used in policy making and public debate. Therefore, this innovation could lead to a breakthrough in effectiveness and efficiency both in the area of official statistics (by NSIs), and policy measures (by policy makers). As far as NSIs are concerned, this can encourage a cultural shift as it will boost the confidence of these institutions to produce model-based error-corrected statistics. As the model is applicable to a wide range of topics, in the field of policy, the project contributes to better policy-making.
An alternative (to HMM) method to deal with measurement error – which is currently not used by the CBS specifically for measurement-error correction – is macro integration. Macro integration is a method that aims to achieve consistency when combining different data sources in producing official statistics. Inconsistencies between datasets can be the result of various types of error and not just measurement error (i.e. sampling error, processing error, linkage error). Macro integration is easier to implement than our method as it only uses the tables of aggregated statistics from the different data sources to achieve consistence and does not include a statistical model with all the data as our method. However, macro integration cannot directly deal with systematic measurement error as it assumes that both datasets are free from systematic error. In contrast, HMMs can provide detailed information on both random and systematic measurement error, which can be used to correct the data. Even if HMMs are not fully adopted in production of official statistics, they can provide auxiliary information for the current practice, or to be combined with macro integration.
High gain: The success of COMBINE could break new ground in official statistics and evidence-based policy making. The production of an error-corrected official statistic provides the opportunity to CBS to present in a relatively cost-effective and efficient way, for each figure, a single estimate with known reliability of social assistance benefits. The method we propose is applicable to other official statistics and therefore several target groups can benefit from it. In the case we deal with in COMBINE, the project will give policy makers the opportunity of deriving evidence-based policies, while policy debate on the crucial issue of the welfare state will be based on these more reliable estimates.
High risk: COMBINE faces three types of risks. Firstly, the statistical models that will be used for the production of error-corrected statistics are computationally demanding and may also face convergence and misspecification problems. These models have never been applied for this specific variable (social assistance). Secondly, the statistical model may produce results that are not convincing for the CBS as an efficient and workable improvement. This means that if it turns out that a small change in the model specification or the data (e.g. data from different years) leads to radically different outcomes, the CBS may be reluctant to adopt this method in the production of official statistics. In general, official statistics is rather conservative in the application of model-based methods in the production of official statistics. Thirdly, it may turn out that users of the statistics find the method too complex and thus not understandable. As a consequence, users may be reluctant to use the new improved statistic. To deal with these risks, we have developed a mitigating strategy (see 1c).
Outline the approach and methodology to explore the innovation potential of your ERC-funded research
So far, we have established that measurement error imposes risks (i.e. bias) to official statistics and, in particular, to official statistics on social assistance. These risks extend further to policy making and public debate that use these figures. Thus, there is a need to seek efficient and robust methods to increase the quality of official statistics on social assistance benefits. Such methods can be applied further to other official statistics and lead to a major breakthrough in the field.
We have also established that HMMs present a solution for correcting for measurement error. These models are suitable for the estimation of the error-corrected number of individuals receiving a social assistance benefit as well as for the transition rate out of social assistance. To realize the objectives of the project and safeguard its impact and feasibility, the PI of the project and his team have reached out to CBS and specifically the Department of Demographic and Socioeconomic Statistics that is responsible for producing statistics on social assistance benefits. The department has committed itself into facilitating in and contributing to this PoC. The aim of the department is to study obtain more understanding of the measurement error inherent in measuring social assistance benefits that may result in the possibility of producing an improved official statistics on social assistance benefits with the use of this error-correction method. Actually, Table 2 (above under section 1A) was produced by the department in preparation for this project. Its results (i.e. the inconsistencies between the two registers) have already led the department in administering an independent study for the sources of the problem. By collaborating with CBS, COMBINE makes a unique contribution by translating scientific research into the production of error-corrected official statistics and also to evidence-based policy making and public debate.
The CBS Department of Demographic and Socioeconomic Statistics and the data owners (municipalities and UWV) will be the first direct end-users of the project. The CBS department will have the option of putting the new method in the production of the official statistic, while the data owners will be provided with aggregated information that they can use directly to improve data quality. However, in case the PoC is successful, there will be a wide array of potential end-users. Specifically, potential end-users will include all sections of CBS as well as of other NSIs that produce official statistics and possess data from more than one datasets. With moderate customization (i.e. adjustment of the HMM for the type of measurement error) the error-correction method can be quickly applied to the production of other categorical official statistics, such as employment status, health status and residential mobility. A second-level of potential end users will include the policy-makers, social partners, professionals and the general public accessing freely-available official statistics. These will form opinions and make decisions on the basis of error-corrected official statistics, i.e. based on better information.
In the context of the GSBPM[2], NSIs perform statistical business processes to create official statistics to satisfy the needs of the users. The PoC will intervene within different parts of the GSBPM, which is used by NSIs to introduce new official statistics. In our case, GSBPM needs further specification as, so far, official statistics are mostly counts of observed data and not model estimates. In this respect, four objectives are defined within COMBINE:
Objective 1: Preliminary assessment of data quality on social assistance benefits. This builds on the preliminary work that has already been conducted at CBS. It will involve the full illustration of inconsistencies of the information on social assistance benefits in the two registers.
Objective 2: Produce error-corrected statistics on social assistance. Most of the attention of COMBINE will be devoted to this objective. By analysing data on social assistance benefits from two registers with our statistical model, we will produce the statistics of interest. Moreover, by using several explanatory variables (e.g. age, gender, household composition, migration status), we will also explore the possible causes of measurement error and derive recommendations for improvement of the source datasets.
Objective 3: Compare the outcome and the process of our error-correction method with alternative approaches, such as editing (which is currently applied) and macro-integration. In this way, we can evaluate the added value (in terms of effectiveness and efficiency) of our error-correction method.
Objective 4: Evaluate and disseminate the results to other NSIs and to policy makers. Within this objective, we will illustrate to policy makers and professionals how the (more) reliable official statistics that we produced can improve policy making and public debate. Moreover, we will address other NSIs to illustrate how the error-correction method that we applied can improve in an efficient and inexpensive way the quality of the official statistics that they produce. To this end, an implementation, and dissemination plan, including stakeholder analysis, are developed under this objective.
[1] This study used data from 2007 from the UWV register and the Labour Force Survey (LFS). According to the LFS, within 3 months, 5.7% of workers with a temporary contract find a job with a permanent contract. According to the UWV register, this percentage is 8.5%. but when correcting for measurement error, this percentage becomes only 3.2%.
[2] GSBPM is developed by the United Nations Economic Commission for Europe (UNECE), on behalf of the international statistical community. For a full description see: https://statswiki.unece.org/display/GSBPM/GSBPM+v5.1
