Skip to main content
Submit a Request

How To: Use .NET Trace Logging to Troubleshoot a Realm's Web Service, SMTP and SAML Traffic

Ben Watts
Updated
    Applies to:
  • SecureAuth Identity Platform
    Deployment model:
  • Cloud
  • Hybrid
  • On Premises

Version Affected: All

 

Overview

This article explains when and why to use .NET trace logging on a SecureAuth Identity Platform realm, and how to use it to troubleshoot the realm's web service, Simple Mail Transfer Protocol (SMTP) and Security Assertion Markup Language (SAML) traffic.

A realm is a .NET Framework application, and the .NET Framework can write a detailed record of what the realm sends and receives to a text file on the appliance that hosts the realm. Trace logging is turned on by adding a <system.diagnostics> block to the realm's web.config file and turned off by removing it. No other software is needed.

Use trace logging when the realm's own logs show that something failed but not what was actually sent or received. An error such as Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value, for example, says only that a response was not the JSON the realm expected, while the trace shows what the other system actually returned. Because the trace is written inside the realm, it also shows the content of encrypted Hypertext Transfer Protocol Secure (HTTPS) requests, which a network packet capture can't read. Connections to an LDAP directory or to SQL Server use other components and don't appear in any of the traces this article describes.

Before you start:

  • You need administrator access to the appliance that hosts the realm. The trace is written only on that appliance, so if more than one appliance hosts the realm, turn tracing on for each one that may handle the traffic you want to trace.
  • Create a folder on the appliance for the trace file, outside the realm's own folder, because the trace holds credentials. The methods below use C:\Temp\RealmTrace.
  • Give the account that the realm's application pool runs as Modify permission on that folder. In Internet Information Services (IIS) Manager, the realm's Basic Settings show its application pool, and the Identity column on the Application Pools page shows the pool's account. ApplicationPoolIdentity means the account IIS AppPool\<pool name>. If the folder doesn't exist or the account can't write to it, the realm carries on without writing the trace and without reporting an error.

There is one method for each kind of traffic:

  • To trace the realm's calls to web services and APIs, such as a password reset sent to an external service or a request for an OAuth access token, see Method 1. It records each Hypertext Transfer Protocol (HTTP) or HTTPS request's address and headers, the status and headers that came back, and any problem with the server's certificate.
  • To trace the email the realm sends, when sending fails with an error such as OTPHtmlEmailProvider.Send, Exception: Failure sending mail, see Method 2. It records the commands the realm sends to the mail server and the server's replies.
  • To trace SAML logins when a browser-based SAML tracer isn't practical, see Method 3. It records the authentication requests and SAML assertions the realm sends and receives, including those the realm receives at its Assertion Consumer Service (ACS) when it is the service provider (SP) for another identity provider.

In this article

 

Method 1: Trace Web Service and API Calls

Use this method when the realm's call to a web service or API fails, or when you need to confirm exactly what the realm sent and what came back.

  1. Make a backup copy of the realm's web.config file, which is in the realm's folder, for example D:\SecureAuth\SecureAuth<N>\web.config, where <N> is the realm number.
  2. Open web.config in a text editor started with Run as administrator, and search it for system.diagnostics. A web.config file can contain only one <system.diagnostics> element, and a second one causes a configuration error on the realm, so if one is already there, merge the entries from the block below into it instead of pasting a second one. You don't need to decrypt web.config first.
  3. Paste the following block immediately above the closing </configuration> tag at the end of the file. If your trace folder is not C:\Temp\RealmTrace, change the path in initializeData to match.
<system.diagnostics>
  <sources>
    <source name="System.Net" tracemode="includehex" maxdatasize="65536">
      <listeners>
        <add name="TraceFile" />
      </listeners>
    </source>
  </sources>
  <switches>
    <add name="System.Net" value="Verbose" />
  </switches>
  <sharedListeners>
    <add name="TraceFile"
      type="System.Diagnostics.TextWriterTraceListener"
      initializeData="C:\Temp\RealmTrace\network.log"
      traceOutputOptions="DateTime" />
  </sharedListeners>
  <trace autoflush="true" />
</system.diagnostics>
  1. Save web.config. The realm restarts, and network.log appears in the trace folder once the realm has something to record. If no file appears after you reproduce the problem, check that the folder exists and that the application pool's account can write to it.
  2. Reproduce the problem, and note the time.
  3. Turn tracing off: remove the block you added, or the entries you merged in step 2, and save web.config. The realm restarts and stops writing to the trace file.
  4. Open network.log in a text editor. Each entry is followed by a DateTime= line giving its time in Coordinated Universal Time (UTC), so you can find the attempt you reproduced. The following entries show what was exchanged:
  • Request: shows the method and path of each request, for example POST /<path> HTTP/1.1.
  • Sending headers shows the headers sent with the request, including Host and any Authorization header.
  • Received status line shows the status the server returned, for example StatusCode=401, StatusDescription=Unauthorized.
  • Received headers shows the headers that came back, such as Content-Type and Content-Length, and WWW-Authenticate when the server rejected the request's credentials.
  • System.Net Error marks an error, such as a failed connection or a certificate problem.

The body of an error response can be missing from the trace even when Content-Length shows there was one.

For example, a realm that resets passwords through an external web service logged Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value. Its trace showed the two requests involved. The request for an access token returned StatusCode=200 with an application/json body, and the password reset request that followed returned StatusCode=401, StatusDescription=Unauthorized with a short text/html body and a WWW-Authenticate: Bearer header. The web service had rejected the access token, so the next step was to find out from the web service's own logs why it had done so.

  1. When you have finished with it, delete network.log and any copies of it.

 

Method 2: Trace SMTP Email

Use this method when the realm fails to send email, for example with OTPHtmlEmailProvider.Send, Exception: Failure sending mail and no further detail.

  1. Make a backup copy of the realm's web.config file, which is in the realm's folder, for example D:\SecureAuth\SecureAuth<N>\web.config, where <N> is the realm number.
  2. Open web.config in a text editor started with Run as administrator, and search it for system.diagnostics. A web.config file can contain only one <system.diagnostics> element, and a second one causes a configuration error on the realm, so if one is already there, merge the entries from the block below into it instead of pasting a second one. You don't need to decrypt web.config first.
  3. Paste the following block immediately above the closing </configuration> tag at the end of the file. If your trace folder is not C:\Temp\RealmTrace, change the path in initializeData to match.
<system.diagnostics>
  <sources>
    <source name="System.Net">
      <listeners>
        <add name="TraceFile" />
      </listeners>
    </source>
    <source name="System.Net.Sockets">
      <listeners>
        <add name="TraceFile" />
      </listeners>
    </source>
  </sources>
  <switches>
    <add name="System.Net" value="Verbose" />
    <add name="System.Net.Sockets" value="Verbose" />
  </switches>
  <sharedListeners>
    <add name="TraceFile"
      type="System.Diagnostics.TextWriterTraceListener"
      initializeData="C:\Temp\RealmTrace\smtp.log"
      traceOutputOptions="DateTime" />
  </sharedListeners>
  <trace autoflush="true" />
</system.diagnostics>
  1. Save web.config. The realm restarts, and smtp.log appears in the trace folder once the realm has something to record. If no file appears after you reproduce the problem, check that the folder exists and that the application pool's account can write to it.
  2. Reproduce the failure, for example by requesting a one-time passcode by email, and note the time.
  3. Turn tracing off: remove the block you added, or the entries you merged in step 2, and save web.config. The realm restarts and stops writing to the trace file.
  4. Open smtp.log in a text editor. Each entry is followed by a DateTime= line giving its time in UTC. The entries that start with System.Net.Sockets show the data sent and received on each connection. For the connection to the mail server, this data is the SMTP conversation: the realm's commands, such as EHLO, STARTTLS, AUTH, MAIL FROM and RCPT TO, and the server's replies, each of which begins with a three-digit code. A reply that starts with 4 or 5 is a failure, and the text after the code gives the mail server's reason. The file also contains the realm's other network traffic.

If the realm connects to the mail server with Transport Layer Security (TLS), the System.Net.Sockets data is encrypted from the point the server accepts STARTTLS, so the commands sent after that point can't be read in that data.

  1. When you have finished with it, delete smtp.log and any copies of it.

 

Method 3: Trace SAML Logins

When a browser-based SAML tracer isn't practical, use this method to see the authentication requests and SAML assertions the realm sends and receives. It also captures what the realm receives at its ACS when it is the SP for another identity provider.

  1. Make a backup copy of the realm's web.config file, which is in the realm's folder, for example D:\SecureAuth\SecureAuth<N>\web.config, where <N> is the realm number.
  2. Open web.config in a text editor started with Run as administrator, and search it for system.diagnostics. A web.config file can contain only one <system.diagnostics> element, and a second one causes a configuration error on the realm, so if one is already there, merge the entries from the block below into it instead of pasting a second one. You don't need to decrypt web.config first.
  3. Paste the following block immediately above the closing </configuration> tag at the end of the file. If your trace folder is not C:\Temp\RealmTrace, change the path in initializeData to match.
<system.diagnostics>
  <trace autoflush="true">
    <listeners>
      <add name="TextWriter" />
    </listeners>
  </trace>
  <sources>
    <source name="ComponentSpace.SAML2" switchValue="Verbose">
      <listeners>
        <add name="TextWriter" />
      </listeners>
    </source>
  </sources>
  <sharedListeners>
    <add name="TextWriter"
      type="System.Diagnostics.TextWriterTraceListener"
      initializeData="C:\Temp\RealmTrace\saml.log" />
  </sharedListeners>
</system.diagnostics>
  1. Save web.config. The realm restarts, and saml.log appears in the trace folder once the realm has something to record. If no file appears after you reproduce the problem, check that the folder exists and that the application pool's account can write to it.
  2. Reproduce the login, and note the time.
  3. Turn tracing off: remove the block you added, or the entries you merged in step 2, and save web.config. The realm restarts and stops writing to the trace file.
  4. Open saml.log in a text editor. It records the authentication requests and SAML assertions the realm sent and received, so you can compare them with what the other party expects, such as the issuer, the audience, the NameID and the attributes.
  1. When you have finished with it, delete saml.log and any copies of it.



 

Special Considerations

  • The trace holds credentials in readable form. A web service trace records every request header, including an Authorization header carrying a bearer token or a Base64-encoded password, and each request's full address, including any client secret or API key in its query string. It can also record request and response bodies, such as an issued access token or the new password in a password reset request. An SMTP trace can record the realm's mail server credentials when the connection isn't encrypted, and a SAML trace can record the user attributes in each assertion. Store the file as carefully as the credentials it holds, and delete it as soon as you've finished with it.
  • Every save of web.config restarts the realm. Turning tracing on and turning it off each restart the realm's application, and users who are partway through logging in to that realm may have to start again. Make both changes at a quiet time.
  • The trace file grows without limit. It records all of the traffic being traced, from every user, not just the attempt you reproduce, and it is never rolled over or trimmed. On a busy realm it can grow quickly enough to fill the disk, so turn tracing off as soon as you have captured the problem.

 

SecureAuth Knowledge Base Articles provide information based on specific use cases and may not apply to all appliances or configurations. Be advised that these instructions could cause harm to the environment if not followed correctly or if they do not apply to the current use case.

Customers are responsible for their own due diligence prior to utilizing this information and agree that SecureAuth is not liable for any issues caused by misconfiguration directly or indirectly related to SecureAuth products.

Comments

0 comments

Please sign in to leave a comment.