Welcome, Guest
Username: Password: Remember me
This public forum is meant for questions and discussions about Visual FoxPro
  • Page:
  • 1

TOPIC:

X# STRCONV() implementation proposal 04 Feb 2021 20:36 #17403

  • atlopes
  • atlopes's Avatar
  • Topic Author


  • Posts: 84
  • The STRCONV function is a notorious miss from the VFP.Net toolkit list of implemented functions.

    This is a proposal for its implementation. It tries to accommodate the difference in the nature of VFP and X# (.Net) strings. VFP strings are made of 8 bit ANSI or binary characters, X# strings are Unicode.

    The main challenge is to provide the function's nuclear functionality while adding some value to the X# environment.

    Some key points
    • Conversion to Unicode or to DBCS strings return strings.
    • Base64 and hexadecimal representations of data are treated as strings.
    • Single-byte strings and UTF-8 are returned as Binary.
    • Regional identifiers (may) affect single-byte and DBCS strings.
    • Added two types that represent a Unicode string as a Binary (for LE and BE).

    Probably something else will come up with the discussion to follow.
    USING System.Text   
    USING System.IO
    USING System.Globalization
    USING XSharp.Core
    
    FUNCTION Start() AS VOID 
    
        LOCAL TextNr AS Int
        
        TestNr = 1
    
        // Base64
        QuickTest(TestNr++, 'StrConv("Abcd", STRCNV_SB_BASE64)', StrConv("Abcd", STRCNV_SB_BASE64), "QWJjZA==")
        QuickTest(TestNr++, 'StrConv("QWJjZA==", STRCNV_BASE64_SB)', StrConv("QWJjZA==", STRCNV_BASE64_SB), (Binary)"Abcd")
        QuickTest(TestNr++, 'StrConv(0h010203fdfeff, STRCNV_SB_BASE64)', StrConv(0h010203fdfeff, STRCNV_SB_BASE64), "AQID/f7/")
        QuickTest(TestNr++, 'StrConv("AQID/f7/", STRCNV_BASE64_SB)', StrConv("AQID/f7/", STRCNV_BASE64_SB), 0h010203fdfeff)
    
        // Hex
        QuickTest(TestNr++, 'StrConv("Abcd", STRCNV_SB_HEX)', StrConv("Abcd", STRCNV_SB_HEX), "41626364")
        QuickTest(TestNr++, 'StrConv("41626364", STRCNV_HEX_SB)', StrConv("41626364", STRCNV_HEX_SB), (Binary)"Abcd")
        QuickTest(TestNr++, 'StrConv(0h010203fdfeff, STRCNV_SB_HEX)', StrConv(0h010203fdfeff, STRCNV_SB_HEX), "010203FDFEFF")
        QuickTest(TestNr++, 'StrConv("010203FDFEFF", STRCNV_HEX_SB)', StrConv("010203fdfeff", STRCNV_HEX_SB), 0h010203fdfeff)
    
        // Unicode
        QuickTest(TestNr++, 'StrConv(StrConv(StrConv("Привет мир!", STRCNV_UNI_DBCS, 1251, 1), STRCNV_DBCS_SB), STRCNV_SB_HEX)', ;
            StrConv(StrConv(StrConv("Привет мир!", STRCNV_UNI_DBCS, 1251, 1), STRCNV_DBCS_SB), STRCNV_SB_HEX), "CFF0E8E2E5F220ECE8F021")
        QuickTest(TestNr++, 'StrConv("Привет мир!", STRCNV_UNI_UTF8)', StrConv("Привет мир!", STRCNV_UNI_UTF8), 0hD09FD180D0B8D0B2D0B5D18220D0BCD0B8D18021)
        QuickTest(TestNr++, 'StrConv(e"Bel\xc3\xa9m do Par\xc3\xa1", STRCNV_UTF8_UNI)', StrConv(e"Bel\xc3\xa9m do Par\xc3\xa1", STRCNV_UTF8_UNI), "Belém do Pará")
        QuickTest(TestNr++, 'StrConv(0hC8, STRCNV_DBCS_UNI, 1253, 1) + StrConv(0hC8, STRCNV_DBCS_UNI, 1256, 1)', ;
            StrConv(0hC8, STRCNV_DBCS_UNI, 1253, 1) + StrConv(0hC8, STRCNV_DBCS_UNI, 1256, 1), 'Θب')
        // Extras
        QuickTest(TestNr++, 'StrConv("Γειά σου Κόσμε!", STRCNV_UNI_SB)', StrConv("Γειά σου Κόσμε!", STRCNV_UNI_SB), 0h9303B503B903AC032000C303BF03C50320009A03CC03C303BC03B5032100)
    
        WAIT
        
    	RETURN 
    
    FUNCTION QuickTest (Test AS Int, Expression AS String, Result AS USUAL, Expected AS USUAL) AS Void
        
        ? Test, Expression, "->", Result, "(" + IIF(Result == Expected, "Success", "Fail, was expecting " + Expected) + ")"
    
    ENDFUNC
    
    DEFINE STRCNV_SB_DBCS           = 1
    DEFINE STRCNV_DBCS_SB           = 2
    DEFINE STRCNV_KATA_HIRA         = 3
    DEFINE STRCNV_HIRA_KATA         = 4
    DEFINE STRCNV_DBCS_UNI          = 5
    DEFINE STRCNV_UNI_DBCS          = 6
    DEFINE STRCNV_LOWER             = 7
    DEFINE STRCNV_UPPER             = 8
    DEFINE STRCNV_DBCS_UTF8         = 9
    DEFINE STRCNV_UNI_UTF8          = 10
    DEFINE STRCNV_UTF8_DBCS         = 11
    DEFINE STRCNV_UTF8_UNI          = 12
    DEFINE STRCNV_SB_BASE64         = 13
    DEFINE STRCNV_BASE64_SB         = 14
    DEFINE STRCNV_SB_HEX            = 15
    DEFINE STRCNV_HEX_SB            = 16
    DEFINE STRCNV_UNI_SB            = 17
    DEFINE STRCNV_UNIBE_SB          = 18
    
    DEFINE STRCNV_ID_LCID           = 0
    DEFINE STRCNV_ID_CODEPAGE       = 1
    DEFINE STRCNV_ID_CHARSET        = 2
    
    FUNCTION StrConv (Expression AS String, ConversionSetting AS Int, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS USUAL
    
        RETURN StrConv_Helper(Expression, (Binary)Expression, ConversionSetting, RegionalIdentifier, RegionalIDType)
    
    END FUNCTION
    
    FUNCTION StrConv (Expression AS Binary, ConversionSetting AS Int, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS USUAL
    
        RETURN StrConv_Helper((String)Expression, Expression, ConversionSetting, RegionalIdentifier, RegionalIDType)
    
    END FUNCTION
    
    FUNCTION StrConv (Expression AS USUAL, ConversionSetting AS Int, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS USUAL
    
        SWITCH VARTYPE(Expression)
            CASE "C"
                LOCAL StrExpression AS String
            
                StrExpression = Expression
                RETURN StrConv(StrExpression, ConversionSetting, RegionalIdentifier, RegionalIDType)
    
            CASE "Q"
                LOCAL BinExpression AS Binary
            
                BinExpression = Expression
                RETURN StrConv(BinExpression, ConversionSetting, RegionalIdentifier, RegionalIDType)
    
            OTHERWISE
                THROW ArgumentException{"Incorrect parameter.", nameof(Expression)}
        END SWITCH
    
    END FUNCTION
     
    FUNCTION StrConvStr (Expression AS Binary, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS String
    
        VAR enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
        RETURN Encoding.Unicode.GetString(Encoding.Convert(enc, Encoding.Unicode, Expression))
                
    END FUNCTION
    
    STATIC FUNCTION StrConv_helper (Expression AS String, BinaryExpression AS Binary, ConversionSetting AS Int, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS USUAL
        
        LOCAL ReturnStr AS String
        LOCAL ReturnBin AS Binary
        LOCAL ReturnType AS String
    
        LOCAL defenc = Encoding.Default AS System.Text.Encoding
        LOCAL utf8 = Encoding.UTF8 AS System.Text.Encoding
        LOCAL unicode = Encoding.Unicode AS System.Text.Encoding
        LOCAL enc AS System.Text.Encoding
    
        LOCAL Indexer AS Int
    
        SWITCH (ConversionSetting)
    
            CASE STRCNV_SB_DBCS
                 IF RegionalIDType == STRCNV_ID_LCID
                    enc = StrConv_GetEncoding(RegionalIdentifier, STRCNV_ID_LCID)
                    ReturnStr = defenc.GetString(Encoding.Convert(defenc, enc, BinaryExpression))
                    ReturnType = "S"
                ELSE
                    THROW ArgumentException {}
                END IF
    
            CASE STRCNV_DBCS_SB
                 IF RegionalIDType == STRCNV_ID_LCID
                    enc = StrConv_GetEncoding(RegionalIdentifier, STRCNV_ID_LCID)
                    ReturnBin = Encoding.Convert(defenc, enc, BinaryExpression)
                    ReturnType = "B"
                ELSE
                    THROW ArgumentException {}
                END IF
                    
            CASE STRCNV_KATA_HIRA
                ReturnStr = Expression       // not implemented
                ReturnType = "S"
    
            CASE STRCNV_HIRA_KATA
                ReturnStr = Expression       // not implemented
                ReturnType = "S"
    
            CASE STRCNV_DBCS_UNI
                enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
                ReturnStr = unicode.GetString(Encoding.Convert(enc, unicode, BinaryExpression))
                ReturnType = "S"
    
            CASE STRCNV_UNI_DBCS
                enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
                ReturnStr = defenc.GetString(Encoding.Convert(unicode, enc, unicode.GetBytes(Expression)))
                ReturnType = "S"
    
            CASE STRCNV_LOWER
                IF RegionalIDType == STRCNV_ID_LCID
                    ReturnBin = defenc.GetBytes(defenc.GetString(BinaryExpression).ToLower(StrConv_GetCulture(RegionalIdentifier)))
                    ReturnType = "B"
                ELSE
                    THROW ArgumentException {}
                END IF
    
            CASE STRCNV_UPPER
                 IF RegionalIDType == STRCNV_ID_LCID
                    ReturnBin = defenc.GetBytes(defenc.GetString(BinaryExpression).ToUpper(StrConv_GetCulture(RegionalIdentifier)))
                    ReturnType = "B"
                ELSE
                    THROW ArgumentException {}
                END IF
    
            CASE STRCNV_DBCS_UTF8
                enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
                ReturnBin = Encoding.Convert(enc, Encoding.UTF8, BinaryExpression)
                ReturnType = "B"
    
            CASE STRCNV_UNI_UTF8
                ReturnBin = Encoding.Convert(unicode, utf8, unicode.GetBytes(Expression))
                ReturnType = "B"
                
            CASE STRCNV_UTF8_DBCS
                enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
                ReturnStr = defenc.GetString(Encoding.Convert(Encoding.UTF8, enc, BinaryExpression))
                ReturnType = "S"
    
            CASE STRCNV_UTF8_UNI
                ReturnStr = unicode.GetString(Encoding.Convert(Encoding.UTF8, Encoding.Unicode, BinaryExpression))
                ReturnType = "S"
                
            CASE STRCNV_SB_BASE64
                ReturnStr = Convert.ToBase64String(BinaryExpression)
                ReturnType = "S"
                
            CASE STRCNV_BASE64_SB
                ReturnBin = Convert.FromBase64String(defenc.GetString(BinaryExpression))
                ReturnType = "B"
    
            CASE STRCNV_SB_HEX
                ReturnStr = SUBSTR(BinaryExpression.ToString(), 3)
                ReturnType = "S"
    
            CASE STRCNV_HEX_SB
                VAR SingleBytes = Byte[]{Expression.Length / 2}
                LOCAL Hex AS Int
                Hex = 0
                FOR Indexer = 1 TO SingleBytes.Length
                    SingleBytes[Indexer] = Convert.ToByte(Expression.Substring(Hex, 2), 16)
                    Hex += 2
                NEXT
                ReturnBin = SingleBytes
                ReturnType = "B"
                
            CASE STRCNV_UNI_SB
                ReturnBin = unicode.GetBytes(Expression)
                ReturnType = "B"
    
            CASE STRCNV_UNIBE_SB
                ReturnBin = Encoding.BigEndianUnicode.GetBytes(Expression)
                ReturnType = "B"
    
            OTHERWISE
                THROW ArgumentException {}
                
        END SWITCH
    
        SWITCH ReturnType
            CASE "S"
                RETURN ReturnStr
            CASE "B"
                RETURN ReturnBin
            END SWITCH
    
    END FUNCTION
    
    STATIC FUNCTION StrConv_GetEncoding (RegionalIdentifier AS Int, RegionalIDType AS Int) AS Encoding
        
        SWITCH RegionalIDType
            
            CASE STRCNV_ID_LCID
                VAR culture = StrConv_GetCulture(RegionalIdentifier)
                LOCAL CP = culture.TextInfo.ANSICodePage AS Int
     
                RETURN Encoding.GetEncoding(CP)
    
            CASE STRCNV_ID_CODEPAGE
                RETURN Encoding.GetEncoding(RegionalIdentifier)
    
            CASE STRCNV_ID_CHARSET
                LOCAL CP = 1252 AS Int
                SWITCH RegionalIdentifier
                    CASE 128
                        CP = 932
                    CASE 129
                        CP = 949
                    CASE 130
                        CP = 1361
                    CASE 134
                        CP = 936
                    CASE 136
                        CP = 950
                    CASE 161
                        CP = 1253
                    CASE 162
                        CP = 1254
                    CASE 163
                        CP = 1258
                    CASE 177
                        CP = 1255
                    CASE 178
                        CP = 1256
                    CASE 186
                        CP = 1257
                    CASE 204
                        CP = 1251
                    CASE 222
                        CP = 874
                    CASE 238
                        CP = 1250
                END SWITCH
                RETURN Encoding.GetEncoding(CP)
        END SWITCH
    
        RETURN Encoding.Default
    
    END FUNCTION
    
    STATIC FUNCTION StrConv_GetCulture (LCID AS Int) AS CultureInfo
        
        IF LCID != 0
            RETURN CultureInfo.GetCultureInfo(LCID)
        ELSE
            RETURN CultureInfo.CurrentCulture
        END IF
        
    END FUNCTION

    Please Log in or Create an account to join the conversation.

    X# STRCONV() implementation proposal 04 Feb 2021 21:27 #17404

    • robert
    • robert's Avatar


  • Posts: 3289
  • Antonio,
    Thanks for the proposal.
    I will refrain from comments for now: I have no idea what FoxPro does with this function so I can't judge the implementation.
    But what you have done looks very well thought through.
    Maybe some small suggestions:
    - the default codepage in StrConv_GetEncoding could be XSharp.RuntimeState.WinCodepage (this is read from the OS).
    - the argument exceptions could probably include a message
    And we will most certainly copy the "QuickTest" code and add it to our unit tests.

    Robert
    XSharp Development Team
    The Netherlands

    Please Log in or Create an account to join the conversation.

    Last edit: by robert.

    X# STRCONV() implementation proposal 05 Feb 2021 09:58 #17408

    • atlopes
    • atlopes's Avatar
    • Topic Author


  • Posts: 84
  • Robert,

    I hope VFP developers will step in, test the implementation, point out its problems, and propose corrections and enhancements.

    The "final" version, if we come to that, it's for the X# team to integrate with any adjustments you will find fit. As we discussed, that version will be submitted as a pull request.

    Please Log in or Create an account to join the conversation.

    X# STRCONV() implementation proposal 05 Feb 2021 19:14 #17417

    • Eric Selje
    • Eric Selje's Avatar


  • Posts: 31
  • I think this is very cool. I'm trying to figure out a way to create tests so that rather than compare against an "expected value" it would compare against what VFP returned so we could ensure they're the same. If we compiled this into a .NET Class we could call that from DotNetBridge in VFP and have the tests in FoxUnit. That'd be the real test whether it's compatible.

    StrConv() is an odd function in VFP and mostly obsolete because of the native functionality in .NET, but we're going to need it included if we want our apps to compile and run correctly.

    Please Log in or Create an account to join the conversation.

    X# STRCONV() implementation proposal 05 Feb 2021 20:51 #17419

    • atlopes
    • atlopes's Avatar
    • Topic Author


  • Posts: 84
  • Eric,

    The expected results in tests come from values returned by VFP, but going through DotNetBridge and FoxUnit will be much more stressful and, thus, clarifying. I'm looking forward to it. And surely it will serve as a foundation for tests of the implementation of other VFP functions.

    Please Log in or Create an account to join the conversation.

    • Page:
    • 1