The problem

• A sophisticated assessment framework, but incomparable institutional assessment results, evaluation scores and justifications unaligned, high-risk malpractice.

Context

A series of national data quality assessments had been conducted in national agencies for data production across Asia-Pacific, using the institutional framework built around environment, statistics processes, and statistical outputs. On paper, the framework was rigorous and highly structured. In practice, however, the retrieved country reports varied widely in how criteria were interpreted, scores justified, and findings translated into evaluative judgments.

Some reports awarded high scores while describing clear shortcomings. Others gave weaker scores despite more positive evidence. In some cases, the justification was barely written out. In others, the tone appeared overly accommodating to institutional counterparts, limiting the visibility of real weaknesses. This meant that even though a common framework was used, the resulting evaluation was not truly comparable across institutions.

My role

I recognised that the problem was not only in the systems being assessed, but in the inconsistency of the assessments themselves. I undertook a deeper re-assessment of the reports to recover comparability across them. This required studying the data quality assessment framework in detail, revisiting its underlying logic and principles, and systematically comparing what the reports actually said against what the scoring criteria required. The DQAF itself drew on UN official statistics principles, the IMF framework, and ISO 9000 quality concepts, which made careful interpretation of meaning and compliance especially important.

What I did

I rebuilt the comparative basis of the assessment by:

• studying the data quality assessment framework in depth, including the meaning of each principle, indicator, and scoring level
• extracting and reviewing the qualitative content from 17 reports rather than relying only on the scores already assigned
• coding passages against the relevant evaluation items to identify what each report actually demonstrated, implied, or failed to justify
• comparing the narrative evidence against the formal compliance requirements in the assessment matrix
• revising scores where the written justification and assigned score clearly did not match
• retaining scores where they were defensible, clarifying them where the interpretation was weak, and using the framework criteria more directly where justification was missing
• standardising interpretation across cases so that the final regional synthesis would reflect a more consistent reading of the framework

The result was a regionally comparable re-assessment across 17 cases, covering core principles such as policy and legal framework, adequacy of resources, relevance, sound methodology, accuracy and reliability, timeliness, consistency, and accessibility. The final report presents this as a regional evaluation of the quality of data for national education statistics across Asia-Pacific.

Outcome

The reassessment produced a more standardised and regionally comparable evaluation of education data quality across Asia-Pacific. Rather than treating the reports at face value, the exercise created a more coherent regional synthesis by aligning scores closely with evidence and with the framework’s own criteria. This made it possible to identify common structural patterns across institutions, including recurring weaknesses in institutional environment, methodological consistency, and user consultation.

What this case shows

This case demonstrates my ability to work one level deeper than surface reporting. I do not only review what assessments conclude; I test whether the conclusions are methodologically defensible and comparable. It shows my strength in framework interpretation, evidence coding, quality control, and cross-country standardisation — especially where formally similar reports are not actually saying the same thing.