Sunday 27 September 2015

IIS App Pool Credentials Exposed

Last week I was looking at changing the periodic restart for an app pool using the appcmd tool and I found something very interesting. Using this tool can reveal the username and password used for the app pool.

See below:
PS C:\Windows\system32\inetsrv> whoami
dev\crminstall
PS C:\Windows\system32\inetsrv> .\appcmd list apppool /"apppool.name:CRMApppool" -config
<add name="CRMAppPool" managedRuntimeVersion="v4.0" managedPipelineMode="Classic">
  <processModel identityType="SpecificUser" userName="dev\crmapppool" password="MyPassword" idleTimeout="1.01:00:00" />
  <recycling>
    <periodicRestart time="1.05:00:00">
      <schedule>
      </schedule>
    </periodicRestart>
  </recycling>
  <failure />
  <cpu />
</add>
The user in question was a local administrator (member of the local Administrators group) and the command was run from PowerShell with elevated permissions.

So you might need to be logged in as an administrator but you should under no circumstances be able to see another user's password. This is a pretty big security hole, IMHO.

I've only tried on Windows 2012 and 2012 R2, but the behaviour seems consistent.

Incidentally, this does not seem to be the first case where credentials are exposed like this, see this post. It's fair to mention that the issue on the link was eventually fixed.

Monday 21 September 2015

ID1073: A CryptographicException occurred when attempting to decrypt the cookie using the ProtectedData API (see inner exception for details).

A few weeks back, the performance testers found this issue when doing some resilience testing:
ID1073: A CryptographicException occurred when attempting to decrypt the cookie using the ProtectedData API (see inner exception for details). If you are using IIS 7.5, this could be due to the loadUserProfile setting on the Application Pool being set to false.
In essence, if the user started his session on server 1, then that server was stopped, when he connected to server 2, the issue would occur. Annoyingly, this would require removing cookies, which our testers are not able to do, on their locked down machines.

The deployment uses an NLB flag was checked for this deployment, but this seemed to make no difference, so we decided to encrypt the cookies using a certificate rather than the machine key.

This is a bit annoying as there are few things to consider:
  1. We're going to have to modify the MS Dynamics CRM web.config, which means that every new patch, update, etc.. might overwrite it.
  2. Following from that we need a deployment script to automate it as much as possible.
  3. We'll need to store a way to id the certificate to be used for the encryption somewhere easily accessible (We could hard code it but then when the certificate expires we'd been in trouble).
I decided to use the registry for 3.

This is the code that we've used to encrypt the cookies with a certificate.

using Microsoft.IdentityModel.Tokens;
using Microsoft.IdentityModel.Web;
using Microsoft.Win32;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using System.Threading.Tasks;

namespace CRM.RSASessionCookie
{
    /// <summary>
    /// This class encrypts the session security token using the RSA key
    /// of the relying party's service certificate.
    /// </summary>
    public class RsaEncryptedSessionSecurityTokenHandler : SessionSecurityTokenHandler
    {
        static List<CookieTransform> transforms;

        static RsaEncryptedSessionSecurityTokenHandler()
        {            
            string certThumbprint = (string)Registry.GetValue(@"HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSCRM",
                "CookieCertificateThumbprint",
                null);

            if (!string.IsNullOrEmpty(certThumbprint))
            {
                X509Certificate2 serviceCertificate = CertificateUtil.GetCertificate(StoreName.My,
                                                         StoreLocation.LocalMachine, certThumbprint);

                if (serviceCertificate == null)
                {
                    throw new ApplicationException(string.Format("No certificate was found with thumbprint: {0}", certThumbprint));
                }

                transforms = new List<CookieTransform>() 
                         { 
                             new DeflateCookieTransform(), 
                             new RsaEncryptionCookieTransform(serviceCertificate),
                             new RsaSignatureCookieTransform(serviceCertificate),
                         };
            }
            else
            {
                throw new ApplicationException(
                @"Could not read Registry Key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSCRM\CookieCertificateThumbprint.\nPlease ensure that the key exists and that you have permission to read it.");
            }
        }

        public RsaEncryptedSessionSecurityTokenHandler()
            : base(transforms.AsReadOnly())
        {
        }
    }

    /// <summary>
    /// A utility class which helps to retrieve an x509 certificate
    /// </summary>
    public class CertificateUtil
    {
        /// <summary>
        /// Gets an X.509 certificate given the name, store location and the subject distinguished name of the X.509 certificate.
        /// </summary>
        /// <param name="name">Specifies the name of the X.509 certificate to open.</param>
        /// <param name="location">Specifies the location of the X.509 certificate store.</param>
        /// <param name="thumbprint">Subject distinguished name of the certificate to return.</param>
        /// <returns>The specific X.509 certificate.</returns>
        public static X509Certificate2 GetCertificate(StoreName name, StoreLocation location, string thumbprint)
        {
            X509Store store = null;
            X509Certificate2Collection certificates = null;           
            X509Certificate2 result = null;

            try
            {
                store = new X509Store(name, location);
                store.Open(OpenFlags.ReadOnly);
                //
                // Every time we call store.Certificates property, a new collection will be returned.
                //
                certificates = store.Certificates;

                for (int i = 0; i < certificates.Count; i++)
                {
                    X509Certificate2 cert = certificates[i];

                    if (cert.Thumbprint.Equals(thumbprint, StringComparison.InvariantCultureIgnoreCase))
                    {
                        result = new X509Certificate2(cert);
                        break;
                    }
                }                
            }
            catch (Exception ex)
            {
                throw new ApplicationException(string.Format("An issue occurred opening cert store: {0}\\{1}. Exception:{2}.", name, location, ex));
            }
            finally
            {
                if (certificates != null)
                {
                    for (int i = 0; i < certificates.Count; i++)
                    {
                        X509Certificate2 cert = certificates[i];
                        cert.Reset();
                    }
                }

                if (store != null)
                {
                    store.Close();
                }
            }

            return result;
        }
    }
}

Company standards dictate that this class should be deployed to GAC but it can be deployed to the CRM webpage bin folder instead.

This is the PowerShell function in our script that sets the certificate on the registry:

function SetCookieCertificateThumbprint
{
 param ([string]$value)
 $path = "hklm:\Software\microsoft\mscrm"
 $name = "CookieCertificateThumbprint"
 
 if( -not (Test-Path -Path $path -PathType Container) )
 {
  Write-Error "Cannot find MSCRM Registry Key: " + $path
 }
 else
 {
  $keys = Get-ItemProperty -Path $path

  if ($keys.$name -or $keys.$name -ne $value)
  {
   Set-ItemProperty -path $path -name $name -value $value 
  }
 }
}

I have not automated the rest, which is the really fiddly part, i.e. updating the web.config. Here are the relevant parts though:

Config Sections First:
<configSections>
    <!-- COMMENT:START CRM Titan 28973
   If you add any new section here , please ensure that section name is removed from help/web.config
End COMMENT:END-->
    <section name="crm.authentication" type="Microsoft.Crm.Authentication.AuthenticationSettingsConfigurationSectionHandler, Microsoft.Crm.Authentication, Version=6.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
    <section name="microsoft.identityModel" type="Microsoft.IdentityModel.Configuration.MicrosoftIdentityModelSection, Microsoft.IdentityModel, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
  </configSections>
The actual token handler:
<microsoft.identityModel>
  <service>   
    <securityTokenHandlers>
      <!-- Remove and replace the default SessionSecurityTokenHandler with your own -->
      <remove type="Microsoft.IdentityModel.Tokens.SessionSecurityTokenHandler, Microsoft.IdentityModel, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
      <add type="CRM.RSASessionCookie.RsaEncryptedSessionSecurityTokenHandler, CRM.RSASessionCookie, Version=1.0.0.0, Culture=neutral, PublicKeyToken=d10ca3d28ba8fa6e" />
    </securityTokenHandlers>
  </service>
</microsoft.identityModel>

It should, hopefully be obvious that this will need to be done across all CRM servers that have the Web Server role.

After all this work, we thought we would be ok for our dev and test environments sharing a single ADFS server, as the cookies would be encrypted with the same certificate, but it turns out that this is not supported by MS Dynamics CRM 2013 :(

Monday 14 September 2015

Disabling Autocomplete for ADFS forms sign in page

We've been asked to disable Autocomplete for the sign in page on our MS Dynamics CRM application. We have a sign-in page because we're using IFD.

This turns out to require an unsupported customization of ADFS, as we're using ADFS 2.1, which really doesn't support any customization at all.

Unsupported here, simply means that a patch might overwrite our changes or the page might change completely, no big deal in this case, as it's unlikely that many changes will be rolled out for ADFS 2.1, but it pays to be careful when doing unsupported customization.

Most of our users use IE 9, which means that autocomplete=off will work, however, some of our users don't, which means that we have to have a new solution.

We're are modifying the FormsSignIn.aspx page. This page can normally be found in c:\inetpub\wwwroot\ls\, but it really does depend on how ADFS is installed.

I've done this in a rather verbose way, first the JavaScript functions:

function EnablePasswordField(){
    document.getElementById('<%=PasswordTextBox.ClientID%>').readOnly=false;         
    document.getElementById('<%=PasswordTextBox.ClientID%>').select();
}

function DisablePasswordField(){
    document.getElementById('<%=PasswordTextBox.ClientID%>').readOnly=true;     
}

and then the markup:
<asp:TextBox runat="server" ID="PasswordTextBox" TextMode="Password" onfocus="EnablePasswordField()" onblur="DisablePasswordField()" ReadOnly="true" autocomplete="off"></asp:TextBox>

The key here is to make the password textbox readonly and use the JavaScript functions to make the control writable on focus and readonly when it loses focus, this seems to be enough to thwart autocomplete, for now at least.

This is the complete page:

<%@ Page Language="C#" MasterPageFile="~/MasterPages/MasterPage.master" AutoEventWireup="true" ValidateRequest="false"
    CodeFile="FormsSignIn.aspx.cs" Inherits="FormsSignIn" Title="<%$ Resources:CommonResources, FormsSignInPageTitle%>"
    EnableViewState="false" runat="server" %>

<%@ OutputCache Location="None" %>

<asp:Content ID="FormsSignInContent" ContentPlaceHolderID="ContentPlaceHolder1" runat="server">
        <script>
        
            function EnablePasswordField(){
               document.getElementById('<%=PasswordTextBox.ClientID%>').readOnly=false;
            }
            
   function DisablePasswordField(){
               document.getElementById('<%=PasswordTextBox.ClientID%>').readOnly=true;     
            }
        </script>
    <div class="GroupXLargeMargin">
        <asp:Label Text="<%$ Resources:CommonResources, FormsSignInHeader%>" runat="server" /></div>
    <table class="UsernamePasswordTable">
        <tr>
            <td>
                <span class="Label">
                    <asp:Label Text="<%$ Resources:CommonResources, UsernameLabel%>" runat="server" /></span>
            </td>
            <td>
                <asp:TextBox runat="server" ID="UsernameTextBox" autocomplete="off"></asp:TextBox>
            </td>
            <td class="TextColorSecondary TextSizeSmall">
                <asp:Label Text="<%$ Resources:CommonResources, UsernameExample%>" runat="server" />
            </td>
        </tr>
        <tr>
            <td>
                <span class="Label">
                    <asp:Label Text="<%$ Resources:CommonResources, PasswordLabel%>" runat="server" /></span>
            </td>
            <td>
                 <asp:TextBox runat="server" ID="PasswordTextBox" TextMode="Password" onfocus="EnablePasswordField()" onblur="DisablePasswordField()" ReadOnly="true" autocomplete="off"></asp:TextBox>
            </td>
            <td>&nbsp;</td>
        </tr>
        <tr>
            <td></td>
            <td colspan="2" class="TextSizeSmall TextColorError">
                <asp:Label ID="ErrorTextLabel" runat="server" Text="" Visible="False"></asp:Label>
            </td>
        </tr>
        <tr>
            <td colspan="2">
                <div class="RightAlign GroupXLargeMargin">
                    <asp:Button ID="SubmitButton" runat="server" Text="<%$ Resources:CommonResources, FormsSignInButtonText%>" OnClick="SubmitButton_Click" CssClass="Resizable" />
                </div>
            </td>
            <td>&nbsp;</td>
        </tr>
    </table>
</asp:Content>

Monday 7 September 2015

ARR 3.0 - Bug?

A few weeks back we found a bug in ARR 3.0, if it's not a bug then it's definitely an odd feature.

We have a couple of ARR servers and while configuring, troubleshooting, etc.. we kept one of the servers off. We turned it on and it started failing with the following errors getting logged in the event log.

Application Log

Source: Application Error

Event ID: 1000

Faulting application name: w3wp.exe, version: 8.0.9200.16384, time stamp: 0x50108835

Faulting module name: requestRouter.dll, version: 7.1.1952.0, time stamp: 0x5552511b

Exception code: 0xc0000005

Fault offset: 0x000000000000f2dd

Faulting process id: 0x8bc

Faulting application start time: 0x01d0b89d5edc49ba

Faulting application path: c:\windows\system32\inetsrv\w3wp.exe

Faulting module path: C:\Program Files\IIS\Application Request Routing\requestRouter.dll

Report Id: 9caa10cb-2490-11e5-943b-005056010c6a

Faulting package full name:

Faulting package-relative application ID:


System log

Source: WAS

Event ID: 5009

A process serving application pool 'stark.dev.com' terminated unexpectedly. The process id was '52'. The process exit code was '0xff'.

Source: WAS

Event ID: 5011

A process serving application pool '
stark.dev.com' suffered a fatal communication error with the Windows Process Activation Service. The process id was '2792'. The data field contains the error number.

Source: WAS

Event ID: 5002

Application pool '
stark.dev.com' is being automatically disabled due to a series of failures in the process(es) serving that application pool.

This last one was due to rapid fail being enabled in the app pool

The odd thing is that, Server 1 was working fine, but Server 2 wasn't. Odder still was that they were configured in the same way, at least it look that way, at first.

After a lot of troubleshooting, we found the issue in Server 2, which, surprise, surprise was not configured the same way as Server 1.

This is the offending rule:



Yes, it's a stupid rule, it clearly should've have been Match Any but then again ARR should not have taken the app pool down.

We talked with Microsoft support who said that they were going to talk to the product team but I've not heard anything, so who knows.

Thursday 3 September 2015

How to disable FIPS using PowerShell

I always forget about this, so I thought I would add myself a remainder

FIPS can be disabled by editing the registry and restarting the server:
New-ItemProperty - Path HKLM\System\CurrentControlSet\Control\Lsa\FIPSAlgorithmPolicy -name Enabled -value 0; Restart-Computer -Force