{"id":19375,"date":"2013-03-06T10:08:24","date_gmt":"2013-03-06T08:08:24","guid":{"rendered":"https:\/\/www.clearsy.com\/?p=19375"},"modified":"2023-09-06T10:38:38","modified_gmt":"2023-09-06T08:38:38","slug":"introduction-to-data-validation","status":"publish","type":"post","link":"https:\/\/www.clearsy.com\/en\/uncategorized\/introduction-to-data-validation\/","title":{"rendered":"Introduction to data validation"},"content":{"rendered":"

[vc_row][vc_column]

According to wikipedia, \u00abin computer science, data validation is the process of ensuring that a program operates on clean, correct and useful data. It uses routines, often called validation rules or check routines, that check for correctness, meaningfulness, and security of data that are input to the system. The rules may be implemented through the automated facilities of a data dictionary, or by the inclusion of explicit application program validation logic<\/em> \u00bb.<\/p>\n

In other words, the data are checked against a data model. The data model could be simple or elaborated : it is up to the human validator to define to which extent the data have to be validated.<\/p>\n

Often data validation is understood as validating numbers, dates and text, i.e. verifying types and ranges. For that purpose, spreadsheet dedicated function can do the job. However in a number of applications the data model could be quite complex and verification difficult to automate, hence leaving room for human errors during validation.<\/p>\n

Example :<\/strong><\/p>\n

In the following spreadsheet, A,B,C and D are 4 natural numbers such as D is the sum of A,B and C.<\/p>\n<\/div>[\/vc_column][\/vc_row][vc_row][vc_column]

\"\"<\/div>[\/vc_column][\/vc_row][vc_row][vc_column]

Models 1 to 3 are correct: every predicate appearing in the column is true. These models address diverse levels of verification, model 3 being the most complete \/ most constrained. With model 1, any tuple of natural numbers is acceptable. With model 3, only a finite set of tuples comply.<\/p>\n

On the other hand, one could make mistakes in a data model, leading to situations where the model is partly correct.<\/p>\n

In the previous table :<\/p>\n

    \n
  • model 4 is not verified if A+B+C >= 65536.<\/li>\n
  • model 5 is only verified when C = 0.<\/li>\n<\/ul>\n

    This example is very simple, however it shows that this kind of verification is tricky as data, model or both could be wrong. So a particular attention is to be paid to the following issues:<\/p>\n

      \n
    • how to detect wrong data ?<\/strong><\/span> Validation rules are the key. These rules, expressed using a mathematical language, allow for specifying data domain. Data that doesn\u2019t fit with data domain are wrong. Comma.<\/li>\n
    • how to be sure that data tagged as \u201ccorrect\u201d is correct ?<\/strong><\/span> Human in the loop is a great source of errors (V&V is usually an activity boring as hell), especially when data set is huge and data model is complex. The way to go is to use tools that could be replayed at will . The ideal solution would be redundancy i.e. two independent tools drawing the same conclusion.<\/li>\n
    • to what extent is the model sound ?<\/strong><\/span> However we are not completely safe as we could made mistakes such as our faulty model complies with our faulty data (errare humanum est, persevare diabolicum<\/em>). Another possibility is to have a data model \u201ccorrect most of the time\u201d (but incorrect on some cases that could not happen in a given data set but will surely happen later, thank to Murphy\u2019s laws). Testing validation rules to ensure that wrong data are detected is to be envisaged but it doesn\u2019t allow you to claim that your data are definitely validated as only partial test could be managed most of the time, data domain being too large.<\/li>\n<\/ul>\n<\/div>[\/vc_column][\/vc_row]<\/p>\n<\/div>","protected":false},"excerpt":{"rendered":"

      [vc_row][vc_column][\/vc_column][\/vc_row][vc_row][vc_column][\/vc_column][\/vc_row][vc_row][vc_column][\/vc_column][\/vc_row]<\/p>\n","protected":false},"author":17,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[682,1536,1],"tags":[],"class_list":["post-19375","post","type-post","status-publish","format-standard","hentry","category-railway","category-safety","category-uncategorized"],"acf":[],"yoast_head":"\nIntroduction to data validation - CLEARSY<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.clearsy.com\/en\/uncategorized\/introduction-to-data-validation\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Introduction to data validation - CLEARSY\" \/>\n<meta property=\"og:description\" content=\"[vc_row][vc_column][\/vc_column][\/vc_row][vc_row][vc_column][\/vc_column][\/vc_row][vc_row][vc_column][\/vc_column][\/vc_row]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.clearsy.com\/en\/uncategorized\/introduction-to-data-validation\/\" \/>\n<meta property=\"og:site_name\" content=\"CLEARSY\" \/>\n<meta property=\"article:published_time\" content=\"2013-03-06T08:08:24+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-09-06T08:38:38+00:00\" \/>\n<meta name=\"author\" content=\"Thierry Lecomte\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Thierry Lecomte\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/uncategorized\\\/introduction-to-data-validation\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/uncategorized\\\/introduction-to-data-validation\\\/\"},\"author\":{\"name\":\"Thierry Lecomte\",\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/#\\\/schema\\\/person\\\/62096c067e9d3a553ad93a906e95a673\"},\"headline\":\"Introduction to data validation\",\"datePublished\":\"2013-03-06T08:08:24+00:00\",\"dateModified\":\"2023-09-06T08:38:38+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/uncategorized\\\/introduction-to-data-validation\\\/\"},\"wordCount\":581,\"publisher\":{\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/#organization\"},\"articleSection\":[\"Railway\",\"Safety\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/uncategorized\\\/introduction-to-data-validation\\\/\",\"url\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/uncategorized\\\/introduction-to-data-validation\\\/\",\"name\":\"Introduction to data validation - CLEARSY\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/#website\"},\"datePublished\":\"2013-03-06T08:08:24+00:00\",\"dateModified\":\"2023-09-06T08:38:38+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/uncategorized\\\/introduction-to-data-validation\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.clearsy.com\\\/en\\\/uncategorized\\\/introduction-to-data-validation\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/uncategorized\\\/introduction-to-data-validation\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Introduction to data validation\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/#website\",\"url\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/\",\"name\":\"CLEARSY\",\"description\":\"Safety Solutions Designer\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/#organization\",\"name\":\"CLEARSY\",\"url\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.clearsy.com\\\/wp-content\\\/uploads\\\/2020\\\/04\\\/logo-clearsy.svg\",\"contentUrl\":\"https:\\\/\\\/www.clearsy.com\\\/wp-content\\\/uploads\\\/2020\\\/04\\\/logo-clearsy.svg\",\"width\":652,\"height\":163,\"caption\":\"CLEARSY\"},\"image\":{\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/#\\\/schema\\\/person\\\/62096c067e9d3a553ad93a906e95a673\",\"name\":\"Thierry Lecomte\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1484587b45f9f96afa32a6615796f7bb4738001f5a3e0ebfe061e9a266f268a6?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1484587b45f9f96afa32a6615796f7bb4738001f5a3e0ebfe061e9a266f268a6?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/1484587b45f9f96afa32a6615796f7bb4738001f5a3e0ebfe061e9a266f268a6?s=96&d=mm&r=g\",\"caption\":\"Thierry Lecomte\"},\"url\":\"https:\\\/\\\/www.clearsy.com\\\/en\\\/author\\\/tlecomte\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Introduction to data validation - CLEARSY","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.clearsy.com\/en\/uncategorized\/introduction-to-data-validation\/","og_locale":"en_US","og_type":"article","og_title":"Introduction to data validation - CLEARSY","og_description":"[vc_row][vc_column][\/vc_column][\/vc_row][vc_row][vc_column][\/vc_column][\/vc_row][vc_row][vc_column][\/vc_column][\/vc_row]","og_url":"https:\/\/www.clearsy.com\/en\/uncategorized\/introduction-to-data-validation\/","og_site_name":"CLEARSY","article_published_time":"2013-03-06T08:08:24+00:00","article_modified_time":"2023-09-06T08:38:38+00:00","author":"Thierry Lecomte","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Thierry Lecomte","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.clearsy.com\/en\/uncategorized\/introduction-to-data-validation\/#article","isPartOf":{"@id":"https:\/\/www.clearsy.com\/en\/uncategorized\/introduction-to-data-validation\/"},"author":{"name":"Thierry Lecomte","@id":"https:\/\/www.clearsy.com\/en\/#\/schema\/person\/62096c067e9d3a553ad93a906e95a673"},"headline":"Introduction to data validation","datePublished":"2013-03-06T08:08:24+00:00","dateModified":"2023-09-06T08:38:38+00:00","mainEntityOfPage":{"@id":"https:\/\/www.clearsy.com\/en\/uncategorized\/introduction-to-data-validation\/"},"wordCount":581,"publisher":{"@id":"https:\/\/www.clearsy.com\/en\/#organization"},"articleSection":["Railway","Safety"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.clearsy.com\/en\/uncategorized\/introduction-to-data-validation\/","url":"https:\/\/www.clearsy.com\/en\/uncategorized\/introduction-to-data-validation\/","name":"Introduction to data validation - CLEARSY","isPartOf":{"@id":"https:\/\/www.clearsy.com\/en\/#website"},"datePublished":"2013-03-06T08:08:24+00:00","dateModified":"2023-09-06T08:38:38+00:00","breadcrumb":{"@id":"https:\/\/www.clearsy.com\/en\/uncategorized\/introduction-to-data-validation\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.clearsy.com\/en\/uncategorized\/introduction-to-data-validation\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.clearsy.com\/en\/uncategorized\/introduction-to-data-validation\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.clearsy.com\/en\/"},{"@type":"ListItem","position":2,"name":"Introduction to data validation"}]},{"@type":"WebSite","@id":"https:\/\/www.clearsy.com\/en\/#website","url":"https:\/\/www.clearsy.com\/en\/","name":"CLEARSY","description":"Safety Solutions Designer","publisher":{"@id":"https:\/\/www.clearsy.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.clearsy.com\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.clearsy.com\/en\/#organization","name":"CLEARSY","url":"https:\/\/www.clearsy.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.clearsy.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/www.clearsy.com\/wp-content\/uploads\/2020\/04\/logo-clearsy.svg","contentUrl":"https:\/\/www.clearsy.com\/wp-content\/uploads\/2020\/04\/logo-clearsy.svg","width":652,"height":163,"caption":"CLEARSY"},"image":{"@id":"https:\/\/www.clearsy.com\/en\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.clearsy.com\/en\/#\/schema\/person\/62096c067e9d3a553ad93a906e95a673","name":"Thierry Lecomte","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1484587b45f9f96afa32a6615796f7bb4738001f5a3e0ebfe061e9a266f268a6?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1484587b45f9f96afa32a6615796f7bb4738001f5a3e0ebfe061e9a266f268a6?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1484587b45f9f96afa32a6615796f7bb4738001f5a3e0ebfe061e9a266f268a6?s=96&d=mm&r=g","caption":"Thierry Lecomte"},"url":"https:\/\/www.clearsy.com\/en\/author\/tlecomte\/"}]}},"_links":{"self":[{"href":"https:\/\/www.clearsy.com\/en\/wp-json\/wp\/v2\/posts\/19375","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.clearsy.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.clearsy.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.clearsy.com\/en\/wp-json\/wp\/v2\/users\/17"}],"replies":[{"embeddable":true,"href":"https:\/\/www.clearsy.com\/en\/wp-json\/wp\/v2\/comments?post=19375"}],"version-history":[{"count":1,"href":"https:\/\/www.clearsy.com\/en\/wp-json\/wp\/v2\/posts\/19375\/revisions"}],"predecessor-version":[{"id":19376,"href":"https:\/\/www.clearsy.com\/en\/wp-json\/wp\/v2\/posts\/19375\/revisions\/19376"}],"wp:attachment":[{"href":"https:\/\/www.clearsy.com\/en\/wp-json\/wp\/v2\/media?parent=19375"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.clearsy.com\/en\/wp-json\/wp\/v2\/categories?post=19375"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.clearsy.com\/en\/wp-json\/wp\/v2\/tags?post=19375"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}