Aug 2, 2017

Mystical GPO change

We are monitoring changes on critical AD objects by event ID 4662. It usually does a good job reporting who initiated the change. Today we have an alert as below:

An operation was performed on an object.
Subject : Security ID: SYSTEM Account Name: DC1$ Account Domain: Logon ID: 0x76AB862E
Object: Object Server: DS Object Type: groupPolicyContainer Object Name: CN={xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx},CN=Policies,CN=System,DC=john,DC=com Handle ID: 0x0
Operation: Operation Type: Object Access Accesses: Write Property
This is a normal event 4662 except that the subject is "System" of DC1. Why would the system account change a GPO? 

Turns out that this is our Default Domain Controllers Policy. There wasn't a direct edit to the GPO. Rather, logon credential for a service on this DC was changed, which in turn added the new service account to "logon as service".

Jun 15, 2016

How to track any attribute change

  1. Assumption #1: DS Access enabled in Group Policy
  2. Assumption #2: audit is enabled on the obejct you want to track ( SACL entries)
  3. run "repadm /showobjmeta DCName DN_of_object"
  4. Check the change time and origin DC of the attribute
  5. Check event id 4662 for that time point on the origin DC
  6. You may see several 4662 events
  7. You should see Subject, Object, and Operation info in the event, and look for operation type "Write Property"
  8. In "properties" section, it list only GUID of the property, so you will have to match it up to AD schema here
  9. Obviously step 8 is near to impossible as there are a few hundreds attributes and you don't want to check them one by one
  10. So you can also just search the GUID from schema context. The query string will look like something like this "schemaIDGUID=#($U@()&%)@" BUT YOU CANNOT USE THE GUID FROM THE EVENT VIREW! You have to convert it first
  11. How to convert a GUID reported in event viewer to a LDAP query filter: this page. Life is hard, isn't it?!

Oct 21, 2015

Using LDP.exe to restore deleted AD objects

The whole process is described here, however, what the article failed to mention:

  • you must use LDAPS to connect to AD
  • When type in new DN, make sure you type in CN portion as well. Most people tend to copy only OU path
  • You need to connect to AD using a user account in local domain, otherwise you won't be see any object under "Deleted Objects" OU

To restore a deleted Active Directory object using Ldp.exe

  1. Open Ldp.exe from an elevated command prompt. Open a command prompt (Cmd.exe) as an administrator. To open a command prompt as an administrator, click Start. In Start Search, type Command Prompt. At the top of the Start menu, right-click Command Prompt, and then click Run as administrator. If the User Account Control dialog box appears, enter the appropriate credentials (if requested), confirm that the action it displays is what you want, and then click Continue.
  2. To connect and bind to the server that hosts the forest root domain of your AD DS environment, under Connections, click Connect, and then click Bind.
  3. On the Options menu, click Controls.
  4. In the Controls dialog box, expand the Load Predefined drop-down list, click Return Deleted Objects, and then click OK.
  5. In the console tree, navigate to the CN=Deleted Objects container.
  6. Locate and right-click the deleted Active Directory object that you want to restore, and then click Modify.
  7. In the Modify dialog box:
    1. In Edit Entry Attribute, type isDeleted.
    2. Leave the Values box empty.
    3. Under Operation, click Delete, and then click Enter.
    4. In Edit Entry Attribute, type distinguishedName.
    5. In Values, type the original distinguished name (also known as DN) of this Active Directory object.
    6. Under Operation, click Replace.
    7. Make sure that the Extended check box is selected, click Enter, and then click Run

Aug 25, 2015

How to create trust between MIT kerberos realm and AD

Copy from this page
Please reference this page from Microsoft

Configuring a Local MIT Kerberos Realm to Trust Active Directory

On the Active Directory Server

  1. Type the following command to specify the local MIT KDC host name (for example, and local realm (for example, YOUR-LOCAL-REALM.COMPANY.COM):
    Run this command on every domain controller that will be referenced by the cluster'skrb5.conf file. If load balancing is being used and a single KDC hostname has to be provided to all domain controllers, refer the Microsoft documentation instead of explicitly using theksetup command on individual domain controllers.
  2. Type the following command to add the local realm trust to Active Directory:
    netdom trust YOUR-LOCAL-REALM.COMPANY.COM /Domain:AD-REALM.COMPANY.COM /add /realm /passwordt:
  3. Type the following command to set the encryption type:
    On Windows 2003 RC2:
    Windows 2003 server installations do not support AES encryption for Kerberos. Therefore RC4 should be used. Please see the Microsoft reference documentation for more information.
    ktpass /MITRealmName YOUR-LOCAL-REALM.COMPANY.COM /TrustEncryp RC4
    On Windows 2008:
      Note: When using AES 256 encryption with Windows 2008 you must update the proper Java Cryptography Extension (JCE) policy files for the version of JDK you are using.
    ksetup /SetEncTypeAttr YOUR-LOCAL-REALM.COMPANY.COM 
    Where the  parameter can be replaced with parameter strings for AES, DES, or RC4 encryption modes. For example, for AES encryption, replace  with AES256-CTS-HMAC-SHA1-96 or AES128-CTS-HMAC-SHA1-96 and for RC4 encryption, replace with RC4-HMAC-MD5. See the Microsoft reference documentation for more information.
    Make sure the encryption type you specify is supported on both, your version of Windows Active Directory and your version of MIT Kerberos.

On the MIT KDC server

Type the following command in the kadmin.local or kadmin shell to add the cross-realm krbtgt principal. Use the same password you used in the netdom command on the Active Directory Server.
where the  parameter specifies the types of encryption this cross-realm krbtgt principal will support: either AES, DES, or RC4 encryption. You can specify multiple encryption types using the parameter in the command above, what's important is that at least one of the encryption types corresponds to the encryption type found in the tickets granted by the KDC in the remote realm.
Examples by Active Directory server type
  • For Windows 2003:
    kadmin: addprinc -e "rc4-hmac:normal" krbtgt/YOUR-LOCAL-REALM.COMPANY.COM@AD-REALM.COMPANY.COM
  • For Windows 2008:
    kadmin: addprinc -e "aes256-cts:normal aes128-cts:normal rc4-hmac:normal" krbtgt/YOUR-LOCAL-REALM.COMPANY.COM@AD-REALM.COMPANY.COM
The cross-realm krbtgt principal that you add in this step must have at least one entry that uses the same encryption type as the tickets that are issued by the remote KDC. If no entries have the same encryption type, then the problem you will see is that authenticating as a principal in the local realm will allow you to successfully run Hadoop commands, but authenticating as a principal in the remote realm will not allow you to run Hadoop commands.

On all of the cluster machines

  1. Verify that both Kerberos realms are configured on all of the cluster boxes. Note that the default realm and the domain realm should remain set as the MIT Kerberos realm which is local to the cluster.
        kdc =
        admin_server =
        default_domain =
        kdc =
        admin_server =
        default_domain =
  2. To properly translate principal names from the Active Directory realm into local names within Hadoop, you must configure the setting in the core-site.xml file on all of the cluster machines. The following example translates all principal names with the realm AD-REALM.CORP.FOO.COM into the first component of the principal name only. It also preserves the standard translation for the default realm (the cluster realm).
For more information about name mapping rules, see:  Configuring the Mapping from Kerberos Principals to Short Names

Apr 22, 2015

How to reduce RDS CALs using WMI privider

Credit: TP (MVP)
Taken from: Technet Forum

You may uninstall license keypacks, reduce the number of install licenses, etc, using the Remote Desktop License Server WMI Provider.  For example, if you just need to reduce the installed count of one of the packs from 30 to 25 you could use the RemoveLicensesWithIdCount method.  Below is the documentation for the Win32_TSLicenseKeyPack Class:
Here are the basic steps for this example.
1. Get a list of installed keypacks.  Open a administrator command prompt and then enter the following commands:
wmic /namespace:\\root\CIMV2 PATH Win32_TSLicenseKeyPack>keypacks.txt
notepad keypacks.txt
2. Let's say you look at the keypacks.txt file and see that the keypack you want to modify is KeyPackId 3, and that you want to uninstall 5 licenses.  In that case you would enter a command similar to this:
wmic /namespace:\\root\CIMV2 PATH Win32_TSLicenseKeyPack CALL RemoveLicensesWithIdCount 3,5
Another example, say you want to uninstall the keypack (id 3 in our example).  Instead of the command in step 2 above, you would enter a command similar to this:
wmic /namespace:\\root\CIMV2 PATH Win32_TSLicenseKeyPack CALL UninstallLicenseKeyPackWithId 3
An alternative to the above is to Deactivate the licensing server, uninstall the RD/TS Licensing Role Service/reinstall the Role Service, Activate, and install your license pack(s) again.  This wipes out the license server database and recreates it.  You may need to contact the Clearinghouse if you get an error when re-installing your license pack(s).

Sep 11, 2014

Kerberos event 4/Target account name is incorrect/ troubleshooting

This error is reported whenever the service provider (e.g. file server) cannot decrypt a service ticket that was sent to it. This is caused by the fact that the ticket was encrypted by one machine then sent to a different machine. In this case, the second machine doesn't have the private key to decrypt the ticket.

Troubleshooting steps

- make sure there is not duplicate SPN for the service you are trying to get. For example, if you want to access \\serverName1\sharePath and get above error; or on a file server you see kerberos event 4, you want to check if there is duplicate SPN host\serverName*. In the past, it takes a little effort to find duplicate SPNs, but started Windows7/2008, you can run setspn -X to find all duplicated SPNs in your domain

- make sure you don't have duplicate DNS entries for same server to different IPs (unless the server does have multiple IP, and all DNS entries are correct).

- if access using FQDN works but not short name (e.g. \\\sharePath works but not \\server\sharePath), and you have WINS in your environment, make sure the target server register correct 20h record (File Server) in WINS

- If your target computer has multiple names (mostly added by command netdom computername), then you should make sure you add SPNs for all the alias you have. For example, if you have a server has both names as server1 and server2, you should below spn registered in AD:

  • host/
  • host/server1
  • host/
  • host/server2
- make sure you don't have hosts file entries that resolve to wrong IP

- Log into the system that actually owns wrong IP, make sure it's not registering wrong name. For DNS, check NIC TCP/IP stack; for WINS, check "netdom computername localhost /enum", as well as check HKLM\CCS\Services\lanmanserver\parameters\OptionalNames

Mar 20, 2014

Script to reverse OU path so it can be sorted from top down - Excel macro & vb procedure

' This excel macro reverse OU path so it can be sorted from domain top to OU trees

' For example: CN=John Lan,OU=IT,OU=accounts,DC=johnlan,DC=com
' Will be reversed to: DC=com,DC=johnlan,OU=accounts,OU=IT,CN=John Lan

' if you want only a function that can reverse OU path (distinguished name), use function ReverseOU
' if you want to use it in excel, copy both ReverseOU and ReverseText as your macro

' How to use it in Excel
' 1. Hold down the ALT + F11 keys, and it opens the Microsoft Visual Basic for Applications window.
' 2. Click Insert > Module, and paste the following macro in the Modulewindow.
' 3. Then press F5, a dialog is displayed on the screen, and you need select a range to work with.
' 4. And then press OK, and all the text strings have been reversed.
' (c)

' ReverseOU can be called from any vb script
Function ReverseOU(s)
    Dim temp
    Dim arrValue
        arrValue = Split(s, ",")
        xLen = UBound(arrValue) + 1
        'wscript.echo xLen
        For i = 0 To ((xLen - 1) / 2)
            'wscript.echo i
            'wscript.echo xLen-i-1
            temp = arrValue(i)
            arrValue(i) = arrValue(xLen - i - 1)
            arrValue(xLen - i - 1) = temp
            'wscript.echo "----------"
        ReverseOU = Join(arrValue, ",")
End Function

'Below is for excel
Sub ReverseText()
    Dim Rng As Range
    Dim WorkRng As Range
    On Error Resume Next
    xTitleId = "KutoolsforExcel"
    Set WorkRng = Application.Selection
    Set WorkRng = Application.InputBox("Range", xTitleId, WorkRng.Address, Type:=8)
    For Each Rng In WorkRng
        xOut = ReverseOU(Rng.Value)
        Rng.Value = xOut
End Sub

Dec 12, 2013

Attributes used in ANR (Ambiguous Name Resolution)

  • GivenName
  • Surname
  • displayName
  • LegacyExchangeDN
  • msExchMailNickname
  • RDN
  • physicalDeliveryOfficeName
  • proxyAddress
  • sAMAccountName
How to run an ANR search:

(anr=John Doe)
(anr=J D)
(anr=J Doe) etc....

Nov 1, 2013

Logon Session (Console & RDP) Tracking Notes

A few notes about how Windows handles session creation/switch when I was writing an app to track user logon time. Maybe useful for somebody.

1. Observation
2. How to handle properly

=  Observations
disconnect from a remote session: remotedisconnect

logoff from console: logoff(user), consoledisconnect(user), consoleconnect(system)

(a system logon screen session can be identified by its "logonUI.exe" process, a subsequent logon will take over this session - meaning same session nuber. In this case, it will be only one event: sessionLogon)

lock a session: sessionLock

switch to an existing session: consoledisconnect(1st user) -> consoleconnect(system) -> consoledisconnect(system) -> consoleconnect(2nd user) -> unlock(2nd user)

unlock directly from orignal screen: consoleUnlock
if "switch user" clicked, then unlock: consoledisconnect(user)->consoleconnect(sytem takes over) -> consoledisconnect(disconnect system)-> consoleconnect(original user) -> sessionunlock(original user).

switch then a new logon: lock old user(sessionlock) -> disconnect old user(DisconnectConsole) -> System takes over(ConsoleConnect) -> new user takes over (sessionLogon)

remote disconnect: RemoteDisconnect

remote New connect:
-> remoteConnect(?)
-> sessionLogon
remote REconnect (existing logon sesseion somewhere):
-> remoteconnect(system)
-> if(existing was on console and open)
-> existingSessionDisconnect(user)
-> then system has to spawn a new session to take over (sessionConnect)
-> remoteDisconnect(system)
-> remoteconnect(user)

=  How to handle properly
There are many session switch that are very confusing at the first sight. For example, for a simple user switch, their may be up to 5 session switch events. The key thing to remember is whethre user has an existing session already.

- once a user obtained session #, he/she keeps that same session # unles she/he logs off. None other session switch events will change this owning relationship;
- Logically, user gets a new session # only when she/he logon
- If a user connects back on a same session or unlock a same session that he obtained prior, then only one switch envent, sessionUnlock or session logon will be fired;
- if a user takes a brand new session, or take session from others, then ConsoleDisconnect/connect will happen multiple times. Basically, TS has to break existing session from another user, connect under System, disconnect System session, then finally users takes the session

Oct 10, 2013

AD CA basic - study notes

  1. There are 3 snapins for certificate-related management
    1. on client, there is Certificates snappin, this manages the actual certificates
    2. on server, you have CA snapin and Certificate Templates snapin
  2. Just because you have a certificate template, doesn't mean CA is going to use it
  3. In order to issue a certain type of certificate, you have to let CA know that you want to issue that type of cert:
    1. In CA snapin, right click server name, New Certificate Template To Issue
    2. To check the current template that your CA can issue, click on "certificate templates"
  4. To manage your templates (in use/not in use), in CA snapin, expand CA server name, right click "Certificate Templates", select manage
  5. After #3, you are in "Certificate Templates" snapin
    1. Templates listed in here can be in use or not in use, depending on if you have perform #3 above
    2. You can directly publish a built-in template, but more typically, you should clone a template, make change on clone, then publish it (not a clone any more since we have made changes)
  6. For client to request:
    1. on a client machine, open Certificate snapin
      Optionally you can use IIS Admin Console or certreq command line
    2. Generate a request, send it to CA admin
    3. CA admin approves it
    4. Client enrolls the cert
  7. You can automate #6 by enabling auto enrollment in group policy
  8. Enterprise CA vs. Standalone CA
    1. standalone CA doesn't have certificate templates
    2. standalone doesn't support auto-enrollment

Apr 26, 2013

KMS, host keys, client keys, etc.

  1. One KMS can host multiple host keys - for example, it can host both Windows 2012 & Office host keys at the same time
  2. Host keys are confidential info for companies who bought the license; while client keys are publicly available from Microsoft's website. Client key for a product is same for all companies who chose to use KMS activation.
  3. Higher host keys is inclusive in that it covers older, lower products in the same product family. For example, once you install Windows Server 2012 Enterprise Edition, you won't need separate keys to cover standard edition or Windows 2008 servers. The same key covers all.

Nov 20, 2012

Replication error'ed out with "no more endpoints"

1.       When right click “replicate now”, and the error message is “error 1753: There are no more endpoints available from the endpoint mapper”, it’s complaining the source DC not able to find a RPC endpoint from target DC. To make it more confusing, these two DCs are replicating to all their other partners - they just don’t want to replication with each other (one direction)
2.       This KB helps: and with good explanation too
3.       In our case, it sounds like root DC(source) was brought to a wrong child DC (target) for replication as per above KB.
4.       However when I check all related A/CNAME records, they are all CORRECT. All WINS records are correct too. Clean DNS cache on both client side and DNS server side didn’t help either
5.       It turns out it’s child’s delegated zone in root zone has incorrect glue record (right click child zone, properties, name servers tab). Windows apparently is capable of detecting such misconfiguration but chose not to auto correct, which is weird. 

Lesson learned: when a DC's CNAME or A is resolved to a wrong IP while all its references in visible zones are correct, please check the Name Servers tab of stud node (of child zone) in parent DNS server. Also, when promote/demote child DCs, or change their IPs, please make sure changes are made in the Name Servers tab too ( I mistakenly assume that dcpromo program would do that automatically)

Sep 17, 2012

Pin point AD object deletion in event log

Ref: Technet blog here

This has been done before object being restored.

  1. Find out DN of the deleted object (using ldifde or adrestore).
    Ldifde –x –d “CN=Deleted Objects,DC=domain,DC=com” –f Deletedobj.ldf
  2. Know when the object was deleted, and on which DC
    Repadmin /Showobjmeta DCname “DN of the deleted object” > Delshowmeta.txt
    In the log, find attribute isDeleted, note the time
  3. Go to the source DC, in security log, find the logs at specific time. Event log IDs are
    630(2k3)/4726(2k8) for user objects
    647(2k3)/4743(2k8) for computer objects

May 31, 2012

Unable to restore deleted AD object

When follow instructions in this link to recover a deleted object, I got error message "illegal modify operation". One of the workaround in the comment worked for me (the restore-adobject method): add a -NewName argument in the restore-adobject statement.

LDAP method didn't work well as it showed only first 1000 objects under "deleted objects" container while we had way more that number.