Breaking the HISCO Barrier: Automatic Occupational Standardization with OccCANINE

Authors: The paper is written by Christian M. Dahl, Torben S. D. Johansen, and Christian Vedel.

Download: You can access the latest version (February 2026) of the working paper here. The first version was posted in February 2024.

Code: You can find the code for the project here. A YouTube video showing how to use the code is also available.

Data: HISCO codes for the Danish census data (Link-Lives, 1787–1901), produced with OccCANINE, are available here.

Abstract: This paper introduces OccCANINE, an open-source tool that maps occupational descriptions to HISCO codes. Manual coding is slow and error-prone; OccCANINE replaces weeks of work with results in minutes. We fine-tune CANINE on 15.8 million description-code pairs from 29 sources in 13 languages. The model achieves 96 percent accuracy, precision, and recall. We also show that the approach generalizes to three systems – OCC1950, OCCICEM, and ISCO-68 – and release them open source. By breaking the “HISCO barrier,” OccCANINE democratizes access to high-quality occupational coding, enabling broader research in economics, economic history, and related disciplines.

Citing

If you would like to cite our paper, please use

@article{dahl2024hisco,
  title={Breaking the HISCO Barrier: Automatic Occupational Standardization with OccCANINE},
  author={Dahl, Christian M. and Johansen, Torben S. D. and Vedel, Christian},
  journal={arXiv preprint arXiv:2402.13604},
  year={2026}
}