New Ways of Specifying Edits

Data editing is the process by which data that are collected in some way (a statistical survey for example) are examined for errors and corrected with the help of software. Edits, the logical conditions that should be satisfied by the data, are specified by subject-matter experts with a procedure which could be tedious and could lead to mistakes with practical implications. To render the process of edit specification more efficient we provide a new step—the definition of the so-called abstract data model of a survey—which describes the structure of the phenomenon that is studied in a survey. The existence of this model enables experts to identify all combinations of variables which should be checked by edits and to avoid the definition of conflicting edits. Furthermore, we introduce an automatic data validation strategy—TREEVAL—that consists of fast tree growing to derive automatically the functional form of edits and of a statistical criterion to clean the incoming data. The TREEVAL strategy is cast within a total quality management framework. The application of the methodologies proposed is demonstrated with the help of a real life application.