Troubleshooting: WS-Trust / MEX Endpoint Errors

Version Affected: All

 

Overview

The WS-Trust service (WSTrust.svc) and its metadata exchange (MEX) endpoint can fail in a few different, unrelated ways:

• See Cause 1 - Accessing the WS-Trust metadata/MEX endpoint fails with an authentication-schemes error, or that same error fills the logs for a realm using Integrated Windows Authentication (IWA)
• See Cause 2 - The MEX endpoint returns a System.InvalidOperationException saying the ServiceHostBase object is null
• See Cause 3 - WS-Trust endpoints don't appear even though they're enabled on the realm's Post Auth page

These causes are not related, so a fix for one cause will not resolve any of the others.

 

In this article

• Cause 1: Windows Authentication Disabled on the WSTrust.svc Endpoint
• Cause 2: Multiple HTTP/HTTPS Bindings on the Default Web Site
• Cause 3: WSFed Claims Namespace Not Filled Out

Cause 1: Windows Authentication Disabled on the WSTrust.svc Endpoint

Accessing the WS-Trust metadata/MEX endpoint directly, or a realm using Integrated Windows Authentication (IWA), can show this error:

The authentication schemes configured on the host ('Anonymous') do not allow those configured on the binding 'CustomBinding' ('Negotiate'). Please ensure that the SecurityMode is set to Transport or TransportCredentialOnly. Additionally, this may be resolved by changing the authentication schemes for this application through the IIS management tool, through the ServiceHost.Authentication.AuthenticationSchemes property, in the application configuration file at the <serviceAuthenticationManager> element, by updating the ClientCredentialType property on the binding, or by adjusting the AuthenticationScheme property on the HttpTransportBindingElement.

For an IWA realm, this shows up as IWA not working, with the same error filling the logs.

Either way, the cause is the same: Windows Authentication is disabled for the WSTrust.svc endpoint.

Resolution 1:

1. In IIS Manager, expand the realm/application under Default Web Site, expand it, and select WebService.
2. Select the Content View tab at the bottom of the center pane.
3. Select WSTrust.svc.
4. In the Actions pane, click Switch to Features View.
   
5. Open Authentication.
6. Make sure Anonymous Authentication, Forms Authentication, and Windows Authentication are all set to Enabled.
   
7. Test access to the WS-Trust metadata/MEX endpoint again.

If Windows Authentication isn't available as an option (the IIS role feature isn't installed), or you prefer to edit the configuration file directly:

1. Back up D:\SecureAuth\SecureAuthXX\webservice\web.config (substituting the realm's actual virtual directory name for SecureAuthXX).
2. Add the following to the file, inside the existing <configuration> element:
   

   <location path="WSTrust.svc">
     <system.webServer>
       <security>
         <authentication>
           <anonymousAuthentication enabled="true" />
           <windowsAuthentication enabled="true" />
         </authentication>
       </security>
     </system.webServer>
   </location>

3. Save the file and test access to the WS-Trust metadata/MEX endpoint again.

Cause 2: Multiple HTTP/HTTPS Bindings on the Default Web Site

The WS-Trust MEX endpoint (for example https://idp.example.com/SecureAuthXX/webservice/wstrust.svc/mex) returns:

System.InvalidOperationException: The ServiceHostBase object returned from the CreateServiceHost method of type 'SecureAuth.IdentityModel.WSTrust.SASecurityTokenServiceConfiguration' is null.

This happens when the IIS Default Web Site has more than one HTTP and/or HTTPS binding configured.

Resolution 2:

In IIS Manager, open the bindings for the Default Web Site and delete any extra HTTP/HTTPS bindings, leaving only one HTTP binding and/or one HTTPS binding.

Cause 3: WSFed Claims Namespace Not Filled Out

WS-Trust endpoints are enabled on the realm's Post Auth page, but accessing one directly returns the standard custom error page ("An authentication error has occurred"). Turning off custom errors on the realm shows the underlying error is an HTTP 404:

and the warning log shows:

Error Message: Value cannot be null. Parameter name: claimType

This happens when the realm's WS Federation claims are set up without a namespace, which prevents the endpoints from being created at all.

Resolution 3:

1. Open the Admin Console for the WS-Fed/WS-Trust realm.
2. Go to the Post Auth tab.
3. Scroll down to SAML Attributes / WS Federation.
4. Make sure every claim listed there has its Namespace field filled out.
5. Save the changes.

SecureAuth Knowledge Base Articles provide information based on specific use cases and may not apply to all appliances or configurations. Be advised that these instructions could cause harm to the environment if not followed correctly or if they do not apply to the current use case.

Customers are responsible for their own due diligence prior to utilizing this information and agree that SecureAuth is not liable for any issues caused by misconfiguration directly or indirectly related to SecureAuth products.