International large-scale educational investigations and the focus on learners' competencies powered a veritable revolution in teaching and learning approaches as well as in educational research methodologies. In the relatively young field of computer science education research, however, there is a considerable lack of empirical studies on the definition and measurement of competencies. The central goal of the presented research project is to identify, describe, and measure competencies for object-oriented programming, in particular for implementing abstract data types.We use an automated assessment system to evaluate and score a large number of students' solutions of programming tasks. Item Response Theory analyses of the results identify subsets of tasks suitable for defining typical programming competencies. Further qualitative analyses reveal the internal structure of the competencies and allow a classification in a competency structure model. This article presents in detail our rigorous methodology and exemplary results for the empirical definition and decomposition of the competency named "Ability to implement the abstract data type Binary Search Tree".