Microsoft.Data.SqlClient.SNI.x86.dll, version: 5.2.0.0 crashed host application

[xiangdguo]

Describe the bug

Microsoft.Data.SqlClient.SNI.x86.dll, version: 5.2.0.0 crashed hosted application. The host application BPHost runs multiple backend tasks in parallel in different app domains. Each task run will use the sample code below to retrieve data from a SQL Server database. It crashes multiple times every day randomly.

Application Error from Windows Event Log:
Faulting application name: Eclipse.BPHost.exe, version: 1.5.35.43, time stamp: 0x6740e2c0
Faulting module name: Microsoft.Data.SqlClient.SNI.x86.dll, version: 5.2.0.0, time stamp: 0x65d526aa
Exception code: 0xc0000409
Fault offset: 0x000013fb
Faulting process id: 0x62d0
Faulting application start time: 0x01db41e100d49446
Faulting application path: E:\Smartflow\Halo\SmartFlow.Halo.Agent\Services\Sky2\Eclipse\Eclipse.BPHost.exe
Faulting module path: E:\Smartflow\Halo\SmartFlow.Halo.Agent\Services\Sky2\Eclipse\DTE\Microsoft.Data.SqlClient.SNI.x86.dll
Report Id: 4dbbfafa-baf6-456b-a345-ad2f96b1c3a1

To reproduce

    public static async Task<Dictionary<string, Object>> GetColumnValuesAsync(string connectionString, string userId, SecureString password, List<string> columnNames, string tableName,
                             string whereClauseWithParameterNames, List<(string paramName, SqlDbType dataType, object paramValue, int size, byte scale, byte precision)> parameterNamesWithValues,
                             CommandBehavior commandBehavior = CommandBehavior.Default, int? commandTimeout = null, ILogger? logger = null)
    {

#if NETSTANDARD2_0 || NETSTANDARD2_1 || NETCOREAPP3_1
        _ = connectionString ?? throw new ArgumentNullException(nameof(connectionString), "GetColumnValuesAsync requires connection string");
        _ = userId ?? throw new ArgumentNullException(nameof(userId), "GetColumnValuesAsync requires UserId to access DB");
        _ = password ?? throw new ArgumentNullException(nameof(password), "GetColumnValuesAsync requires Password to access DB");
        _ = tableName ?? throw new ArgumentNullException(nameof(tableName), "GetColumnValuesAsync requires tableName");
        _ = columnNames ?? throw new ArgumentNullException(nameof(columnNames), "GetColumnValuesAsync requires columnNames");
        _ = whereClauseWithParameterNames ?? throw new ArgumentNullException(nameof(whereClauseWithParameterNames), "GetColumnValuesAsync requires whereClauseWithParameterNames");
        _ = parameterNamesWithValues ?? throw new ArgumentNullException(nameof(parameterNamesWithValues), "GetColumnValuesAsync requires parameterNamesWithValues");
#else
        ArgumentNullException.ThrowIfNull(connectionString);
        ArgumentNullException.ThrowIfNull(userId);
        ArgumentNullException.ThrowIfNull(password);
        ArgumentNullException.ThrowIfNull(tableName);
        ArgumentNullException.ThrowIfNull(columnNames);
        ArgumentNullException.ThrowIfNull(whereClauseWithParameterNames);
        ArgumentNullException.ThrowIfNull(parameterNamesWithValues);
#endif

        Dictionary<string, Object> returnDictionary = [];
        StringBuilder sb = new();
        sb.Append("SELECT ");

        foreach (string colName in columnNames)
        {
            sb.AppendFormat(CultureInfo.InvariantCulture, "{0},", colName);
        }
        sb.Length--;

        sb.AppendFormat(CultureInfo.InvariantCulture, " FROM {0} WHERE {1}", tableName, whereClauseWithParameterNames);

        string cmdText = sb.ToString();

        using (SqlConnection conn = new(connectionString) { Credential = new SqlCredential(userId, password) })
        {
            using SqlCommand cmd = new()
            {
                Connection = conn,
                CommandText = cmdText,
                // Also need to specify that this is a stored procedure command (default is Text)
                CommandType = CommandType.Text
            };

            if (commandTimeout.HasValue && commandTimeout.Value >= 0)
                cmd.CommandTimeout = commandTimeout.Value;

            if (parameterNamesWithValues != null)
            {
                foreach (var (paramName, dataType, paramValue, size, scale, precision) in parameterNamesWithValues)
                {
                    if (dataType == SqlDbType.VarChar || dataType == SqlDbType.NVarChar || dataType == SqlDbType.Char || dataType == SqlDbType.NChar || dataType == SqlDbType.Text || dataType == SqlDbType.NText)
                        cmd.Parameters.Add(new SqlParameter { SqlDbType = dataType, ParameterName = paramName, Direction = ParameterDirection.Output, Value = paramValue, Size = size });
                    else if (dataType == SqlDbType.Decimal)
                        cmd.Parameters.Add(new SqlParameter { SqlDbType = dataType, ParameterName = paramName, Direction = ParameterDirection.Output, Value = paramValue, Scale = scale, Precision = precision, });
                    else if (dataType == SqlDbType.DateTime2)
                        cmd.Parameters.Add(new SqlParameter { SqlDbType = dataType, ParameterName = paramName, Direction = ParameterDirection.Output, Value = paramValue, Scale = scale, });
                    else
                        cmd.Parameters.Add(new SqlParameter { SqlDbType = dataType, ParameterName = paramName, Direction = ParameterDirection.Input, Value = paramValue });
                }
            }
            // Create the input parameter, set the properties and add to command.

            await conn.OpenAsync().ConfigureAwait(false);
            using var reader = await cmd.ExecuteReaderAsync(commandBehavior).ConfigureAwait(false);
            await reader.ReadAsync().ConfigureAwait(false);
            returnDictionary = Enumerable.Range(0, reader.FieldCount)
                        .ToDictionary(reader.GetName, reader.GetValue);
        }
        return returnDictionary;
    }

Expected behavior

Do not crash the hosted app and let the tasks continue in next run later even there is an exception in the code.

Further technical details

Microsoft.Data.SqlClient version: 5.2.2
.NET target: .NET Framework 4.7.2,
SQL Server version: SQL Server 2017
Operating system: Windows Server 2019

[mdaigle]

Hi @xiangdguo, is there any more information you can provide on the exception you're seeing? Additional error messages or stack traces would be helpful.

Are there any other characteristics of the failure (e.g. high load on the server) that can help us pinpoint the issue?

[xiangdguo]

Hi @mdaigle, thank you for the reply above! We tried to capture more exception message by adding exception handling code in the host app, but the process crashed the whole host app before reaching the exception handling code.

As I described above, the host app runs multiple processes in parallel, and each process calls MSD 5.2 client to retrieve data from a SQL Server DB. Can a 32 bit host app running on Windows 2019 64 bit OS be a root cause?

[cheenamalhotra]

As I described above, the host app runs multiple processes in parallel, and each process calls MSD 5.2 client to retrieve data from a SQL Server DB. Can a 32 bit host app running on Windows 2019 64 bit OS be a root cause?

Hi @xiangdguo

A few follow up questions:

• How many processes are running in parallel?
• Does modifying that number to a lower count help prevent crashes?
• Do you have any Server side logs you can provide from ERRORLOG file?
• Has this issue always occurred with MDS or this started happening after a recent upgrade? What version worked for you before, if it did?

Would appreciate more details. Thanks!

[xiangdguo]

Hi @cheenamalhotra,

Below are my answers following your questions.

• How many processes are running in parallel?
  It happens with 5 parallel processes in our testing environment.

• Does modifying that number to a lower count help prevent crashes?
  No, it happens in both testing (much less parallel process) and production (many more parallel processes) environment.

• Do you have any Server side logs you can provide from ERRORLOG file?
  We added exception handling and logging code for the code that calls MDS, but nothing has been logged for the crashes.

• Has this issue always occurred with MDS or this started happening after a recent upgrade? What version worked for you before, if it did?
  Always occurred with 5.2. We had not used the lower/older versions of MDS.

Thank you!