Regex doesn't match all foreign characters












2















Here's my regex ^([\p{L}-|a-zA-Z0-9-_]+)$ and it is supposed to allow all foreign letters as well as numeric letter, number. But for some reason, hindi characters cannot match.



I wrote a Xunit test to prove.



[Fact]

        public void test()

        {

            var hindiChar = "इम्तहान";

            var input = "12345ABCDPrüfungテスト中文테스트إسرائيل" + hindiChar;

            var regex = "^([\p{L}-|a-zA-Z0-9-_]+)$";

            Assert.True(new Regex(regex).IsMatch(input));

        }


If you remove the hindiChar, the test will return true; but if you add the hindiChar, the test will return false.



I thought part of the regex is to fit all foreign characters, but not sure why it doesn't match hindi characters.






c# regex xunit







share|improve this question


















  • 1





    It is a known fact that p{L} only matches letters from the BMP plane. Do you want to match diacritics too? Add p{M}. Use @"^[p{M}p{L}-|a-zA-Z0-9-_]+$". What is the | there for? Note the | inside a character class matches a literal | char. It seems to me you wan to use @"^[p{L}p{M}0-9_-]+$"

    – Wiktor Stribiżew
    Nov 15 '18 at 18:47













  • @WiktorStribiżew thanks it worked. The | is or as in this regex allows foreign characters or numeric, numbers.

    – WW pana
    Nov 15 '18 at 18:51






  • 1





    Ok, [|] is not an or operator, and here | must be removed.

    – Wiktor Stribiżew
    Nov 15 '18 at 18:51
















2















Here's my regex ^([\p{L}-|a-zA-Z0-9-_]+)$ and it is supposed to allow all foreign letters as well as numeric letter, number. But for some reason, hindi characters cannot match.



I wrote a Xunit test to prove.



[Fact]

        public void test()

        {

            var hindiChar = "इम्तहान";

            var input = "12345ABCDPrüfungテスト中文테스트إسرائيل" + hindiChar;

            var regex = "^([\p{L}-|a-zA-Z0-9-_]+)$";

            Assert.True(new Regex(regex).IsMatch(input));

        }


If you remove the hindiChar, the test will return true; but if you add the hindiChar, the test will return false.



I thought part of the regex is to fit all foreign characters, but not sure why it doesn't match hindi characters.






c# regex xunit







share|improve this question


















  • 1





    It is a known fact that p{L} only matches letters from the BMP plane. Do you want to match diacritics too? Add p{M}. Use @"^[p{M}p{L}-|a-zA-Z0-9-_]+$". What is the | there for? Note the | inside a character class matches a literal | char. It seems to me you wan to use @"^[p{L}p{M}0-9_-]+$"

    – Wiktor Stribiżew
    Nov 15 '18 at 18:47













  • @WiktorStribiżew thanks it worked. The | is or as in this regex allows foreign characters or numeric, numbers.

    – WW pana
    Nov 15 '18 at 18:51






  • 1





    Ok, [|] is not an or operator, and here | must be removed.

    – Wiktor Stribiżew
    Nov 15 '18 at 18:51














2












2








2








Here's my regex ^([\p{L}-|a-zA-Z0-9-_]+)$ and it is supposed to allow all foreign letters as well as numeric letter, number. But for some reason, hindi characters cannot match.



I wrote a Xunit test to prove.



[Fact]

        public void test()

        {

            var hindiChar = "इम्तहान";

            var input = "12345ABCDPrüfungテスト中文테스트إسرائيل" + hindiChar;

            var regex = "^([\p{L}-|a-zA-Z0-9-_]+)$";

            Assert.True(new Regex(regex).IsMatch(input));

        }


If you remove the hindiChar, the test will return true; but if you add the hindiChar, the test will return false.



I thought part of the regex is to fit all foreign characters, but not sure why it doesn't match hindi characters.






c# regex xunit







share|improve this question














Here's my regex ^([\p{L}-|a-zA-Z0-9-_]+)$ and it is supposed to allow all foreign letters as well as numeric letter, number. But for some reason, hindi characters cannot match.



I wrote a Xunit test to prove.



[Fact]

        public void test()

        {

            var hindiChar = "इम्तहान";

            var input = "12345ABCDPrüfungテスト中文테스트إسرائيل" + hindiChar;

            var regex = "^([\p{L}-|a-zA-Z0-9-_]+)$";

            Assert.True(new Regex(regex).IsMatch(input));

        }


If you remove the hindiChar, the test will return true; but if you add the hindiChar, the test will return false.



I thought part of the regex is to fit all foreign characters, but not sure why it doesn't match hindi characters.






c# regex xunit




c# regex xunit






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 15 '18 at 18:45









WW panaWW pana

474417




474417








  • 1





    It is a known fact that p{L} only matches letters from the BMP plane. Do you want to match diacritics too? Add p{M}. Use @"^[p{M}p{L}-|a-zA-Z0-9-_]+$". What is the | there for? Note the | inside a character class matches a literal | char. It seems to me you wan to use @"^[p{L}p{M}0-9_-]+$"

    – Wiktor Stribiżew
    Nov 15 '18 at 18:47













  • @WiktorStribiżew thanks it worked. The | is or as in this regex allows foreign characters or numeric, numbers.

    – WW pana
    Nov 15 '18 at 18:51






  • 1





    Ok, [|] is not an or operator, and here | must be removed.

    – Wiktor Stribiżew
    Nov 15 '18 at 18:51














  • 1





    It is a known fact that p{L} only matches letters from the BMP plane. Do you want to match diacritics too? Add p{M}. Use @"^[p{M}p{L}-|a-zA-Z0-9-_]+$". What is the | there for? Note the | inside a character class matches a literal | char. It seems to me you wan to use @"^[p{L}p{M}0-9_-]+$"

    – Wiktor Stribiżew
    Nov 15 '18 at 18:47













  • @WiktorStribiżew thanks it worked. The | is or as in this regex allows foreign characters or numeric, numbers.

    – WW pana
    Nov 15 '18 at 18:51






  • 1





    Ok, [|] is not an or operator, and here | must be removed.

    – Wiktor Stribiżew
    Nov 15 '18 at 18:51








1




1





It is a known fact that p{L} only matches letters from the BMP plane. Do you want to match diacritics too? Add p{M}. Use @"^[p{M}p{L}-|a-zA-Z0-9-_]+$". What is the | there for? Note the | inside a character class matches a literal | char. It seems to me you wan to use @"^[p{L}p{M}0-9_-]+$"

– Wiktor Stribiżew
Nov 15 '18 at 18:47







It is a known fact that p{L} only matches letters from the BMP plane. Do you want to match diacritics too? Add p{M}. Use @"^[p{M}p{L}-|a-zA-Z0-9-_]+$". What is the | there for? Note the | inside a character class matches a literal | char. It seems to me you wan to use @"^[p{L}p{M}0-9_-]+$"

– Wiktor Stribiżew
Nov 15 '18 at 18:47















@WiktorStribiżew thanks it worked. The | is or as in this regex allows foreign characters or numeric, numbers.

– WW pana
Nov 15 '18 at 18:51





@WiktorStribiżew thanks it worked. The | is or as in this regex allows foreign characters or numeric, numbers.

– WW pana
Nov 15 '18 at 18:51




1




1





Ok, [|] is not an or operator, and here | must be removed.

– Wiktor Stribiżew
Nov 15 '18 at 18:51





Ok, [|] is not an or operator, and here | must be removed.

– Wiktor Stribiżew
Nov 15 '18 at 18:51












1 Answer
1






active

oldest

votes


















2














It is not enough to use p{L} to match words, you also need to match diacritics. That can be done by adding p{M} to your regex. Note that even the w shorthand "word" character class in .NET regex by default also matches a set of diacritics, p{Mn} (Mark, Nonspacing Unicode char category), see this .NET regex reference. However, here you need p{M} to allow any diacritics.



Note that | inside a character class matches a literal | char, so you need to remove the | from your pattern.



It seems to me you use 



@"^[p{L}p{M}0-9_-]+$"


It will match any string of one or more letters, diacritics, ASCII digits, _ or - chars.



See the regex demo.



Note that in case you want to allow any Unicode digit chars, you may even use



@"^[wp{M}-]+$"


See another demo






share|improve this answer

























    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53326044%2fregex-doesnt-match-all-foreign-characters%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    2














    It is not enough to use p{L} to match words, you also need to match diacritics. That can be done by adding p{M} to your regex. Note that even the w shorthand "word" character class in .NET regex by default also matches a set of diacritics, p{Mn} (Mark, Nonspacing Unicode char category), see this .NET regex reference. However, here you need p{M} to allow any diacritics.



    Note that | inside a character class matches a literal | char, so you need to remove the | from your pattern.



    It seems to me you use 



    @"^[p{L}p{M}0-9_-]+$"


    It will match any string of one or more letters, diacritics, ASCII digits, _ or - chars.



    See the regex demo.



    Note that in case you want to allow any Unicode digit chars, you may even use



    @"^[wp{M}-]+$"


    See another demo






    share|improve this answer






























      2














      It is not enough to use p{L} to match words, you also need to match diacritics. That can be done by adding p{M} to your regex. Note that even the w shorthand "word" character class in .NET regex by default also matches a set of diacritics, p{Mn} (Mark, Nonspacing Unicode char category), see this .NET regex reference. However, here you need p{M} to allow any diacritics.



      Note that | inside a character class matches a literal | char, so you need to remove the | from your pattern.



      It seems to me you use 



      @"^[p{L}p{M}0-9_-]+$"


      It will match any string of one or more letters, diacritics, ASCII digits, _ or - chars.



      See the regex demo.



      Note that in case you want to allow any Unicode digit chars, you may even use



      @"^[wp{M}-]+$"


      See another demo






      share|improve this answer




























        2












        2








        2







        It is not enough to use p{L} to match words, you also need to match diacritics. That can be done by adding p{M} to your regex. Note that even the w shorthand "word" character class in .NET regex by default also matches a set of diacritics, p{Mn} (Mark, Nonspacing Unicode char category), see this .NET regex reference. However, here you need p{M} to allow any diacritics.



        Note that | inside a character class matches a literal | char, so you need to remove the | from your pattern.



        It seems to me you use 



        @"^[p{L}p{M}0-9_-]+$"


        It will match any string of one or more letters, diacritics, ASCII digits, _ or - chars.



        See the regex demo.



        Note that in case you want to allow any Unicode digit chars, you may even use



        @"^[wp{M}-]+$"


        See another demo






        share|improve this answer















        It is not enough to use p{L} to match words, you also need to match diacritics. That can be done by adding p{M} to your regex. Note that even the w shorthand "word" character class in .NET regex by default also matches a set of diacritics, p{Mn} (Mark, Nonspacing Unicode char category), see this .NET regex reference. However, here you need p{M} to allow any diacritics.



        Note that | inside a character class matches a literal | char, so you need to remove the | from your pattern.



        It seems to me you use 



        @"^[p{L}p{M}0-9_-]+$"


        It will match any string of one or more letters, diacritics, ASCII digits, _ or - chars.



        See the regex demo.



        Note that in case you want to allow any Unicode digit chars, you may even use



        @"^[wp{M}-]+$"


        See another demo







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Nov 15 '18 at 18:59

























        answered Nov 15 '18 at 18:54









        Wiktor StribiżewWiktor Stribiżew

        323k16146226




        323k16146226
































            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid …



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53326044%2fregex-doesnt-match-all-foreign-characters%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            -

            Popular posts from this blog

            Bressuire

            Image
            Subprefecture and commune in Nouvelle-Aquitaine, France Bressuire Subprefecture and commune Chateau de Bressuire and the Eglise Notre-Dame Coat of arms Location of Bressuire Bressuire Show map of France Bressuire Show map of Nouvelle-Aquitaine Coordinates: 46°50′27″N 0°29′14″W  /  46.8408°N 0.4872°W  / 46.8408; -0.4872 Coordinates: 46°50′27″N 0°29′14″W  /  46.8408°N 0.4872°W  / 46.8408; -0.4872 Country France Region Nouvelle-Aquitaine Department Deux-Sèvres Arrondissement Bressuire Canton Bressuire Government  • Mayor .mw-parser-output .nobold{font-weight:normal} (2014–20) Jean Michel Bernier Area 1 180.59 km 2 (69.73 sq mi) Population (2014) 2 19,300  • Density 110/km 2 (280/sq mi) Time zone UTC+01:00 (CET)  • Summer (DST) UTC+02:00 (CEST) INSEE/Postal code 79049 /79300 Elevation 98–236 m (322–774 ft) (avg. 173 m or 568 ft) 1 French Land Register data, which exclude...
            Read more

            Vorschmack

            Image
            Vorschmack Ukrainian Jewish-style vorschmack served on rye bread Course Hors d'oeuvre Region or state Eastern Europe Associated national cuisine Ashkenazi Jewish, Finnish, German, Ukrainian, Polish, Russian Main ingredients Ground meat and/or fish Cookbook: Vorschmack   Media: Vorschmack Vorschmack or forshmak (Yiddish: פֿאָרשמאַק ‎, from archaic German Vorschmack , "foretaste" [1] or "appetizer" [2] ) is an originally East European dish made of salty minced fish or meat. Different variants of this dish are especially common in Ashkenazi Jewish and Finnish cuisine. Some varieties are also known in Russian and Polish cuisine. Contents 1 In Jewish cuisine 2 In Russian cuisine 3 In Polish cuisine 4 In Finnish cuisine 5 See also 6 References In Jewish cuisine According to Gil Marks, the German name points to the possible Germanic origin of this dish. [1] William Pokhlyobkin descr...
            Read more

            Quarantine

            Image
            For other uses, see Quarantine (disambiguation). Signal flag "Lima" called the "Yellow Jack" which when flown in harbor means ship is under quarantine. A simple yellow flag (also called the "Yellow Jack") had historically been used to signal quarantine (it stands for Q among signal flags), but now indicates the opposite, as a signal of a ship free of disease that requests boarding and inspection. A quarantine is used to separate and restrict the movement of people; it is 'a restraint upon the activities or communication of persons or the transport of goods designed to prevent the spread of disease or pests', for a certain period of time. [1] This is often used in connection to disease and illness, such as those who may possibly have been exposed to a communicable disease. [2] The term is often erroneously used to mean medical isolation, which is "to separate ill persons who have a communicable disease from those who are healthy...
            Read more