Tag: ads

vApp IMPS PAM AD Authentication “Jump Server”

Posted on by 2 Comments

The virtual appliance and standalone deployment of Symantec (CA) Identity Suite allow for redirecting authentication for the J2EE tier application through Symantec SSO or directly to an Active Directory domain, instead of the existing userstore for the solution.

Challenge:

The standalone deployment of Symantec (CA) Identity Suite on MS Windows OS allowed for the mid-tier component to utilize PAM modules to redirect to AD authentication for the Global User.

However this PAM feature does not exist for Provisioning Servers on the virtual appliance.

To be clear, there are no expectations this feature will be introduced in the future roadmap for the solution, as the primary UI will be the web browser.

Review:

Symantec (CA) Identity Suite architecture for virtual appliance versus standalone deployment architecture.

  • The standalone deployment architecture has both MS Windows and Linux components of all tiers.
  • The vApp deployment architecture has primary Linux components and few MS Windows components.
    • The vApp MS Windows components do not include the IMPS (Provisioning Server)

Proposal:

To address this requirement of enabling AD authentication to the vApp Provisioning Server, we will introduce the concept of a “jump server”.

The “jump server” will utilize the standalone deployment of Symantec Identity Provisioning Server on an MS Windows OS. This “jump server” will be deployed as an “alternative server” integrated into the existing vApp Provisioning Directory deployment.

We will select deployment configuration ONLY of the Provisioning Server itself. We do not require the embedded CCS Service.

We will integrate this “jump server” deployment with the existing Symantec Identity solution.

Ensure the imps_datakey encryption seed file is in sync between all components vApp and standalone.

To avoid impacting the existing vApp deployment, we will NOT integrate the “jump server” deployment to the IME. The IME’s Directory XML for the Provisioning Directory will not be updated.

Important Note: The Symantec/CA Directory solution is required as a pre-step.

Summary of deployment steps:

  • Select a MS Window OS workstation (clean or with JCS/CCS Services) that may be part of the MS AD Domain
    • Option 1: [RECOMMENDED & PREFERRED] If using a clean OS, install MS .NetFramework 3.5.1 for the provisioning component.
      • Open cmd as administrator to deploy:  DISM /Online /Enable-Feature /All /FeatureName:NetFx3
    • Option 2: [MED-HIGH RISK] If using “side-deployment” on an existing JCS/CCS server (MS Win OS), we will need to make modifications to this server.
      • Will need to rename the file  C:\Windows\vpd.properties   to avoid conflict with the JCS/CCS component naming convention in this “registry” file. (see below screen shot)
      • Will require a post-install execution of the IMPS pwdmgr tool to address an MS Registry path conflict between the CCS and IMPS components.
  • Ensure all CA Directory hostnames are in DNS or in the MS Windows local host file (C:\Windows\System32\drivers\etc\hosts ) otherwise this “jump server” deployment will fail when it tries to validate all possible directory nodes’ hostnames and build the respective Directory knowledge files.
  • Create a reference file for the new IMPS router dxc file on at least one of the existing vApp Identity Suite Directory Server otherwise this “jump server” deployment will fail due to trust issue when testing connections to other directory nodes’ hostnames.
  • Deploy Symantec/CA Directory (if not already done) – default configurations. Otherwise, you will see this error message
  • Deploy IMPS MS Windows – Only IMPS (no CCS) with Alternative Server Selection Configuration & update to latest CP patches. Note: For “side-deployment” only: If the vpd.properties file was not renamed, then a name collision will occur due to this registry file, if using the JCS/CCS server to side-deploy. It is low risk to change this file, as it is used to prevent deployments of lower release version of components over the prior installed higher release versions of the same component. If there is a concern, all components can be reinstalled as needed. Do not forget to install the latest CP patches to ensure this “jump server” is the same binary level as the vApp solution.
  • Review of additional notes during deployment of “jump server”. Note: For “side-deployment” only: On the page that ask for the Identity Suite Directory connection information, you will see the solution attempt to load env variables that do not exist. Override these value and enter the Directory hostname, Port 20394, and the default bind DN credentials for a Directory userID: eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=im,dc=etadb
  • Deploy IMPM Manager GUI if needed.
  • Post-Deployment – Update IMPM Manager GUI preference to ONLY connect to the new IMPS server on MS Windows. Use the “Enable Failover” checkbox and place the IP/hostname first in the list. Hint: Remove the other IMPS servers from this list or you may add an extra digit to the IMPS servers entries you wish to save, but prevent auto-connectivity to them. Confirm able to authenticate directly through the solution using prior credentials for your service ID: etaadmin or imadmin. This will validate connectivity to the existing vApp Identity Suite solution.
  • On the “jump server” under the Provisioning Server\pam\ADS folder copy the etapam.dll to the IMPS \bin folder. Then copy the etapam_id.conf configuration file to the \pam parent folder. Update the parameters in this file. Set the enable= parameter to yes. Set the domain= to either the MS AD Domain or use the FQDN hostname of the ADS Domain Controller (DC). If we use the FQDN hostname of the DC the “jump server” does NOT have to be made a member of the MS AD Domain. Save the file and restart the “CA Identity Manager – Provisioning Server”
  • Validate PAM functionality in the IMPS etatrans log is enabled. We will see two (2) entries: PAM: Initialization started (same for all use-cases) and PAM: Not enabled or No PAM managed endpoint. We want “PAM: No PAM managed endpoint” – That is an extra feature we could enable, but do not require for the “jump server” scenario.
  • Validate PAM functionality with MS Sysinternals. Ensure that we copied the etapam.dll to the bin folder and that the configuration file is being read.
  • Test authentication using IMPM Manager login as IMPS Manager Global User that has same userID format as AD sAMAccountName. Negative Use-Case testing: Create a new AD domain user that does NOT exist as a Global User and attempt to authentication. Test with etaadmin or other Global User that does NOT have a match AD sAMAcccount name entry. Review the IMPS etatrans logs on the “jump server”

Update the IMPS encrypted seed file imps_datakey as needed.

Note: The MS Win version of IMPS encrypted seed file may be different than the vApp seed.

If this step is skipped, there will be no obvious error message with the exception that a bind has failed for communication to the JCS/CCS services.

After this file is updated, we will need to re-install IMPS service to ensure that all prior encryption passwords are replaced with new passwords using the new seed file. Basically, we need to install the MS Win version of IMPS Server twice, e.g. standard install, change the seed file value, re-install with update all components and updated passwords.

CCS Service conflict with “side-loading” IMPS Service {“side-loading” methodology}

The “side loading “process of deploying the “jump server” IMPS Provisioning Server on the JCS/CCS Server will impact starting of the CCS service. The installation will update the MS Registry with extra branches and updated shared attribute values between the CCS service and IMPS service, e.g. ETAHOME.

This challenge is a strong reason why we may choose the “clean” installation methodology, to avoid this conflict and possible support challenge.

To address this concern, update the new registry values that store the embedded reversible encrypted password for the CCS Service. Use the password reset tool “pwdmgr” and reset the “Connector Server” for both “eta” & “im” domain to the prior stored password. If the imps_datakey file is not in sync between all provisioning servers (& ccs service), then we will see failed bind connections error messages in the logs.

We will now be able to stop/start the JCS service, and see the embedded CCS service stop and start as well.

Example of challenge and error messages if imps_datakey is not updated and in sync.

Use the following command, csfconfig.exe, under the newly deployed IMPS bin folder to view the JCS connectors defined to the solution stack.

C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>csfconfig.exe auth=etaadmin show
EtaSSL.initialize: CRYPTO_library_init: 1
EtaSSL.initialize: SSL_library_init: 1
Enter your authentication password:


C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>echo Password01 > c:\imps.pwd


C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>csfconfig.exe auth=etaadmin add name=pamjcs host=192.168.242.143 pass=c:\imps.pwd br-add=@ debug=yes port=20411
EtaSSL.initialize: CRYPTO_library_init: 1
EtaSSL.initialize: SSL_library_init: 1
Enter your authentication password:
Created CS object with name = pamjcs

C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>csfconfig.exe auth=etaadmin remove name=pamjcs 
EtaSSL.initialize: CRYPTO_library_init: 1
EtaSSL.initialize: SSL_library_init: 1
Enter your authentication password:

We will see both error status when the imps_datakey file is out-of-sync with others. Please ensure the Linux & MS Win versions are in sync.

You may view the file imps_datakey being referenced with the pwdmgr tool: 

su - imps
strace -e trace=open,openat  pwdmrg

open(“/opt/CA/IdentityManager/ProvisioningServer/data/tls/keymgmt/imps_datakey”, O_RDONLY) = 5

You wish to monitor what accounts (embedded) are updated with the IMPS pwdmgr tool: su – imps and execute the two commands in a different SSH shell to monitor the pwdmgr.log that was enabled.

eta-env action=set name=eta_pwdmgr_log type=string value=true
tail -f $ETAHOME/bin/pwdmgr.log

Enablement of extra functionality (bypass the no-sync option on Global User password update)

You may wish to keep the Global User and AD password in sync. If they are not, then you will have two passwords that will work for the Global User account. The newer PAM AD authentication credentials, and the older Global User password. The etapam.dll module data path appears to check for PAM AD first, and if it fails, then it will check the Global User eTPassword field as well.

Enable the AD endpoint in the etapam_id.conf file. The type and domain will be as shown, e.g. Active Directory and im (for the vApp). The endpoint-name will be free-form and whatever you may have named your AD endpoint in the IMPS GUI.

Monitor the startup of the PAM module within the IMPS etatrans*.log

Perform a use-case test with changing a Global User account without correlation to an AD endpoint; and then retest with a Global User that is correlated to an AD endpoint. Do both test with NO SYNC operation

If the Global User is already correlated to an AD endpoint account, then we will see a “Child Modify” operation to the correlated AD endpoint account’s Password within the IMPS etatrans*.log.

One “gotcha”. There appears to be a check against the AD password policy. If the new password does not fit the AD password policy, the following error message will appear, “ETA_E_0007 <MGU>, Global user XXXXXXX modification failed: PAM account password updated failed: Account password must match global user password.

ADS Endpoint Configuration Challenges and Hints

Posted on by Leave a comment
  1. Ensure the hostname entry is a FQDN or alias. It can not be an IP address if MS Exchange is to be managed through this connector, due to conflict with Kerberos authentication and IP addresses. If the object was created with an IP address, it may be changed via Jxplorer for two (2) attributes: eTADSprimaryServer and eTADSServerName.

2. General Information on the ADS Endpoint Logging Tab and where this information is stored. Only two (2) the Destination have value with current deployment, e.g. Text File & System Log (MS Windows Event viewer) for Active Directory (ADS). The “Text File” will output data to two (2) files: jcs\logs\ADS\<endpoint-name>.log and ccs\logs\ADS\<endpoint-name>.log

3. Use the MS Event Viewer on the ADS Domain Controller, or use the MS Event Viewer to remotely view the transactions on the remote ADS DC. Select the event codes of 627,628,4723,4724,4738 to start with. Other codes may be added that are useful. Ref: https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/plan/appendix-l–events-to-monitor

4. Additionally, the User ID may be in one of three (3) formats: UPN (serviceid@exchange.lab), NT ( domain\serviceid ), LDAP DN ( cn=serviceid,ou=people,dc=exchange,dc=lab). We recommend UPN or NT format to allow the embedded API features for MS Exchange powershell management to correctly function. If the ID is to be changed, a password update must be done as well, since the User ID is part of the seed for the encrypted password for the service ID to be stored in CA Directory on the ADS endpoint object.

5. SASL versus TLS authentication checkboxes. We can tested the ADS authentication availability using ldapsearch binary. Ports used by Active Directory for authentication by client tools, https://docs.microsoft.com/en-us/troubleshoot/windows-server/identity/config-firewall-for-ad-domains-and-trusts

Note: SASL is encrypted traffic. If wireshark is used to intercept the traffic, the service ID may be seen during initial authentication, but NOT the password nor the payload data.

Notes on SASL validation for Active Directory. {Pro: No need to worry about TLS certificates rotation on client connections – all TLS is managed by the server}

:: Search ADS / LDAP store what is offered for SASL (use -x for simple connection)
ldapsearch -x -h dc2016.exchange.lab -p 389 -b “” -LLL -s base supportedSASLMechanisms

EXAMPLE OUTPUT

[root@oracle ~]# ldapsearch -x -h dc2016.exchange.lab -p 389 -b “” -LLL -s base supportedSASLMechanisms
dn:
supportedSASLMechanisms: GSSAPI
supportedSASLMechanisms: GSS-SPNEGO
supportedSASLMechanisms: EXTERNAL
supportedSASLMechanisms: DIGEST-MD5

:: On Linux OS, execute rpm -qa to search for SASL installed modules/libraries.
rpm -qa | grep cyrus

EXAMPLE OUTPUT

[root@oracle ~]# rpm -qa | grep cyrus
cyrus-sasl-gssapi-2.1.26-23.el7.x86_64
cyrus-sasl-lib-2.1.26-23.el7.x86_64
cyrus-sasl-md5-2.1.26-23.el7.x86_64

:: On Linux OS, install missing SASL libraries & ldapsearch (ldap-client)
yum -y install cyrus-sasl-md5 cyrus-sasl-gssapi openldap-clients

TESTING DIFFERING AUTHENTICATION MECHANISMS #### (may remove -d9 debug switch to view cleaner results)

TLS

LDAPTLS_REQCERT=never ldapsearch -d9 -LLL -H ldaps://dc2016.exchange.lab:636 -w CAdemo123 -D “CN=Administrator,CN=Users,DC=exchange,DC=lab” -b “CN=Administrator,CN=Users,DC=exchange,DC=lab” -s base userAccountControl

Start TLS

LDAPTLS_REQCERT=never ldapsearch -d9 -Z -LLL -H ldap://dc2016.exchange.lab:389 -w CAdemo123 -D “CN=Administrator,CN=Users,DC=exchange,DC=lab” -b “CN=Administrator,CN=Users,DC=exchange,DC=lab” -s base userAccountControl

Digest-MD5

ldapsearch -d9 -LLL -H ldap://dc2016.exchange.lab -w CAdemo123 -Y DIGEST-MD5 -U Administrator -b “CN=Administrator,CN=Users,DC=exchange,DC=lab” -s base userAccountControl

Kerberos (GSS)

ldapsearch -d9 -LLL -H ldap://dc2016.exchange.lab -w CAdemo123 -Y GSSAPI -U Administrator -b “CN=Administrator,CN=Users,DC=exchange,DC=lab” -s base userAccountControl

6. TCP/UDP Ports required for Active Directory Endpoint management per CA Documentation https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-manager/14-4/reference/default-ports-for-ca-identity-manager-and-associated-components.html

SASL appears to connect on TCP 636 briefly, then use TCP 389 extensively. Other ports are 80 (Service), 135 (lsass.exe for home folders), 6405 (lsass.exe). If Kerberos authentication is defined for the service ID, then other ports will be used, e.g. 3268/3269. TCP 4104/4105 are for the legacy CAM/CAFT agents (typically not used any more).

Recommendation: Add these TCP Ports to any Firewall between the IM JCS/CCS Server and the Active Directory Domain Controllers to improve performance and avoid time-out delays.

MS Active Directory References on SASL.

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/989e0748-0953-455d-9d37-d08dfbf3998b

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/a98c1f56-8246-4212-8c4e-d92da1a9563b

Parallel provisioning for Active Directory and MS Exchange mailboxes – Improve Birthright/DayOne Access

Posted on by 1 Comment

One of the challenges that IAM/IAG solutions may have is using single thread processing for select endpoints. For the CA/Symantec Identity Management solution, before IM r14.3cp2, we lived with a single-threaded connector to managed MS Active Directory endpoints.

To address this challenge, we deployed multiple connector servers. We allowed the IM Provisioning Server (IMPS) to use a built-in round-robin approach of load-balancing separate transactions to different connector servers, which would service the same Active Directory endpoints.

The IME may be running as fast as it can with its clustered deployment, but as soon as a task has MS Active Directory, and there is a bottleneck with the CCS Service. We begin to see the IME JMS queue reporting that it is stuck and the IME View Submitted Task reporting “In Progress” for all tasks. If the CCS service is restarted, all IME tasks are then reported as “Failed.”

This is/was the bottleneck for the solution for sites that have MS Active Directory for Birthright/DayOne Access.

We can now avoid this bottleneck. [*** (5/24/2021) – There is an enhancement to CP2 to address im_ccs.exe crashes during peak loads discovered using this testing process. ]

Via the newly delivered enhancement https://community.broadcom.com/participate/ideation-home/viewidea?IdeationKey=7154e15b-085d-469e-bff0-ac588ff6bd5b .

We now have full parallel provisioning to MS Active Directory from a single connector server (JCS/CCS).

The new attribute that regulates this behavior is eTADSMaxConnectionsInPool. This attribute will be applied on every existing ADS endpoint that is currently being managed by the IM Provisioning Server after CP2 is deployed. Note: The default value is 10, but we recommend after much testing, to match the value of the IMPS-> JCS and JCS->CCS to equal 200.

During testing within the IME using Bulk Tasks or the IM BLC, we can see that the CCS-> ADS traffic will reach 20-30 connections if allowed. You may set this attribute to a value of 200 via Jxplorer and/or an ldapmodify/dxmodify script.

echo "############### SET ADS MAX CONNECTIONS IN POOL SIZE ##################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
NAMESPACE=exchange2016
LDAPTLS_REQCERT=never dxmodify -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D "$IMPS_USER" -w "$IMPS_PWD" << EOF
dn: eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta
changetype: modify
eTADSMaxConnectionsInPool: 200
EOF
LDAPTLS_REQCERT=never dxsearch -LLL -H ldap://$IMPS_HOST:$IMPS_PORT -x -D "$IMPS_USER" -w "$IMPS_PWD" -b "eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta" -s base eTADSMaxConnectionsInPool | perl -p00e 's/\r?\n //g'

To confirm the number of open connections is greater than one (1), we can issue a Bulk IM Task or use a performance tool like CA Directory dxsoak.

In this example, we will show case using CA Directory dxsoak to execute 100 parallel threads to create 100 ADS Accounts with MS Exchange Mailboxes. We will also enclose this script for download for others to review and use.

Performance Lab:

Pre-Steps:

  1. Leverage CA Directory samples’ dxsoak binary (performance testing). You may wish to use CA Directory on an existing IM Provisioning Server (Linux OS) or you may deploy CA Directory (MS Windows version) to the JCS/CCS connector. Examples are provided for both OSes.
  2. Create LDIF files for IM Provisioning Server and/or IM Connector Tier. This file is needed to ‘push’ the solution to-failure. The use of the IME Bulk Task and/or etautil scripts to the IM Provisioning Tier, will not provide the transaction speed we need to break the CCS service if possible.
  3. Within the IM Provisioning Manager enable the ADS Endpoint TXT Logs on the Logging TAB, for all checkboxes.
  4. Monitor the IMPS etatrans* logs, monitor the JCS ADS logs, monitor the CCS ADS logs, monitor the number of CCS-> ADS (LDAP/S – TCP 389/636) threads. [Suggest using MS Sysinternals Process Explorer and select im_ccs.exe & then TCP/IP TAB]
  5. Monitor the MS ADS Domain via MS ADUC (AD Users & Computers UI) and MS Exchange Mailbox (Mailbox UI via Browser)

Execution:

6. Perform a UNIT TEST with dxmodify/ldapmodify to confirm the LDIF file input is correct with the correct suffix.

time dxmodify -H ldap://192.168.242.135:20389 -c -x -D "eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta" -w Password01 -f ads_user_with_exchange_dc_eta.ldif

7. Perform the PERFORMANCE TEST with dxsoak binary with the same LDIF file & correct suffix. Rate observed = 23 K ids/hr

./dxsoak -c -l 60 -t 100 -h 192.168.242.135:20389 -D "eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta" -w Password01 -f ads_user_with_exchange_dc_eta.ldif

Observations:

8. IMPS etatrans*.log – Count the number of operations per second. Note any RACE and/or data collisions, e.g. ADS accounts deleted prior to add via 100 threads or ADS account created multiple times attempted in different threads.

9. IM CCS ADS <endpoint>.log – Will only have useful data if the ADS Endpoint Logging TAB has been checked for TXT logs.

10. Finally, validate directly in MS Active Domain with the ADUC or similar tool & MS Exchange mailboxes being created/deleted.

11. Count the number of threads from im_ccs.exe to ADS – Suggest using MS Sysinternals Process Explorer tool and/or Powershell to count the number of connections.

MS Powershell Script to count the number of LDAP (TCP 389) connection from im_ccs.exe. [Note: TCP 389 is used more if the ADS Endpoint is setup to use SASL authentication. TCP 636 is used more if the ADS Endpoint is using the older TLS authentication]

$i=1
Do {
cls
(Get-NetTCPConnection -State Established -OwningProcess (Get-Process -name im_ccs).id -RemotePort 389).count
Start-Sleep -s 1
$i++
}
while ($i -le 5)

Direct Performance Testing to JCS/CCS Service

While this testing has limited value, it can offer satisfaction and assistance to troubleshoot any challenges. We can use the prior LDIF files with a slightly different suffix, dc=etasa (instead of dc=eta), to use dxsoak to push the connector tier to failure. This step helped provide memory dumps back to CA/Symantec Engineering teams to help isolate challenges within the parallel processing. CCS Service is only exposed via localhost. If you wish to test the CCS Service remotely, then update the MS Registry key for the CCS service to use the external IP address of the JCS/CCS Server. Rate observed = 25 K ids/hr

Script to generate 100 ADS Accounts with MS Exchange Mailbox Creation

You may wish to review this script and adjust it for your ADS / MS Exchange domains for testing. You can also create a simple LDIF file with password resets or ADS group membership adds. Just remember that the IMPS Service (TCP 20389/20390) uses the suffix dc=eta, and the IM JCS/CCS Services (TCP 20410/20411) & (TCP 20402/20403) use the suffix dc=etasa. Additionally, if using CA Directory dxsoak, only use the non-TLS ports, as this binary is not equipped for using TLS certs.

generate_ads_users_ldif.sh_Download
#!/bin/bash
#######################################################################################################################
# Name:  Generate ADS Feed Files for IM Solution Provisioning/Connector Tiers
#
# Goal:  Validate the new parallel processes from the IM Connector Tier to Active Directory with MS Exchange
#
#
# Generate ADS User LDIF file(s) for use with unit (dxmodify) and performance testing (dxsoak) to:
#  - {Note: dxsoak will only work with non-TLS ports}
#
# IM JCS (20410)  "dc=etasa"    {Ensure MS Windows Firewall allows this port to be exposed}
# IM CCS (20402)  "dc=etasa"    {This port is localhost only, may open to network traffic via registry update}
# IMPS (20389)    "dc=eta"
#
#
# Monitor:  
#
# The IMPS etatrans*.log  {exclude searches}
# The JCS daily log
# The JCS ADS log {Enable the ADS Endpoint TXT logging for all checkboxes}
# The CCS ADS log {Enable the ADS Endpoint TXT logging for all checkboxes}
#
# Execute per the examples provided during run of this file
#
#
# ANA 05/2021
#######################################################################################################################

# Unique Variables for an ADS Domain
NAMESPACE=exchange2016
ADSDOMAIN=exchange.lab
DCDOMAIN="DC=exchange,DC=lab"
OU=People

#######################################################################################################################


MAX=100
start=00001
counter=$start
echo "###############################################################"
echo "###############################################################"
START=`/bin/date --utc +%Y%m%d%H%M%S,%3N.0Z`
echo `/bin/date --utc +%Y%m%d%H%M%S,%3N.0Z`" = Current OS UTC time stamp"
echo "###############################################################"
FILE1=ads_user_with_exchange_dc_etasa.ldif
FILE2=ads_user_with_exchange_dc_eta.ldif
echo "" > $FILE1
while [ $counter -le $MAX ]
do
    n=$((10000+counter)); n=${n#1}
    tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z`
   echo "Counter with leading zeros = $n   at time:  $tz"


cat << EOF >> $FILE1
dn:  eTADSAccountName=firstname$n aaalastname$n,eTADSOrgUnitName=$OU,eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
changetype: add
objectClass:  eTADSAccount
eTADSobjectClass:  user
eTADSAccountName:  firstname$n aaalastname$n
eTADSgivenName:  firstname$n
eTADSsn:  aaalastname$n
eTADSdisplayName:  firstname$n aaalastname$n
eTADSuserPrincipalName:  aaatestuser$n@$ADSDOMAIN
eTADSsAMAccountName:  aaatestuser$n
eTPassword:  Password01
eTADSpwdLastSet:  -1
eTSuspended:  0
eTADSuserAccountControl:  0000000512
eTADSDescription:  description $tz
eTADSphysicalDeliveryOfficeName:  office
eTADStelephoneNumber:  111-222-3333
eTADSmail:  aaatestuser$n@$ADSDOMAIN
eTADSwwwHomePage:  web.page.lab
eTADSotherTelephone:  111-222-3333
eTADSurl:  other.web.page.lab
eTADSstreetAddress:  street address line01
eTADSpostOfficeBox:  pobox 111
eTADSl:  city
eTADSst:  state
eTADSpostalCode:  11111
eTADSco:  UNITED STATES
eTADSc:  US
eTADScountryCode:  840
eTADSscriptPath:  loginscript.cmd
eTADSprofilePath:  \profile\path\here
eTADShomePhone:  111-222-3333
eTADSpager:  111-222-3333
eTADSmobile:  111-222-3333
eTADSfacsimileTelephoneNumber:  111-222-3333
eTADSipPhone:  111-222-3333
eTADSinfo:  Notes Here
eTADSotherHomePhone:  111-222-3333
eTADSotherPager:  111-222-3333
eTADSotherMobile:  111-222-3333
eTADSotherFacsimileTelephoneNumber:  111-222-3333
eTADSotherIpPhone:  111-222-3333
eTADStitle:  title
eTADSdepartment:  department
eTADScompany:  company
eTADSmanager:  CN=manager_fn manager_ln,OU=$OU,$DCDOMAIN
eTADSmemberOf:  CN=Backup Operators,CN=Builtin,$DCDOMAIN
eTADSlyncSIPAddressOption: 0000000000
eTADSdisplayNamePrintable: aaatestuser$n
eTADSmailNickname: aaatestuser$n
eTADShomeMDB: (Automatic Mailbox Distribution)
eTADShomeMTA: CN=DC001,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,$DCDOMAIN
eTAccountStatus: A
eTADSmsExchRecipientTypeDetails: 0000000001
eTADSmDBUseDefaults: TRUE
eTADSinitials: A
eTADSaccountExpires: 9223372036854775807

EOF
 counter=$(( $counter + 00001 ))
done


#  Create the delete ADS Process
start=00001
counter=$start
while [ $counter -le $MAX ]
do
    n=$((10000+counter)); n=${n#1}
    tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z`
   echo "Counter with leading zeros = $n   at time:  $tz"


cat << EOF >> $FILE1
dn:  eTADSAccountName=firstname$n aaalastname$n,eTADSOrgUnitName=$OU,eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
changetype: delete

EOF
 counter=$(( $counter + 00001 ))
done

echo ""
echo "################################### ADS USER OBJECT STATS ################################################################"
echo "Number of add objects: `grep "changetype: add" $FILE1 | wc -l`"
echo "Number of delete objects: `grep "changetype: delete" $FILE1 | wc -l`"
rm -rf $FILE2
cp -r -p $FILE1 $FILE2
sed -i 's|,dc=im,dc=etasa|,dc=im,dc=eta|g' $FILE2
ls -lart $FILE1
ls -lart $FILE2

echo ""
echo "################################### SET ADS MAX CONNECTIONS IN POOL SIZE ################################################################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
LDAPTLS_REQCERT=never dxmodify  -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D "$IMPS_USER" -w "$IMPS_PWD"  << EOF
dn: eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta
changetype: modify
eTADSMaxConnectionsInPool: 200
EOF
LDAPTLS_REQCERT=never dxsearch -LLL  -H ldap://$IMPS_HOST:$IMPS_PORT -x -D "$IMPS_USER" -w "$IMPS_PWD" -b "eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta" -s base eTADSMaxConnectionsInPool | perl -p00e 's/\r?\n //g'

echo ""
echo "################################### CCS UNIT & PERF TEST ################################################################"
CCS_HOST=192.168.242.80
CCS_PORT=20402
CCS_USER="cn=root,dc=etasa"
CCS_PWD="Password01"
echo "Execute this command to the CCS Service to test single thread with dxmodify or ldapmodify"
echo "dxmodify  -H ldap://$CCS_HOST:$CCS_PORT -c -x -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo "Execute this command to the CCS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $CCS_HOST:$CCS_PORT -D $CCS_USER -w $CCS_PWD -f $FILE1 "

echo ""
echo "################################### JCS UNIT & PERF TEST ################################################################"
CCS_HOST=192.168.242.80
CCS_PORT=20410
CCS_USER="cn=root,dc=etasa"
CCS_PWD="Password01"
echo "Execute this command to the JCS Service to test single thread with dxmodify or ldapmodify "
echo "dxmodify  -H ldap://$CCS_HOST:$CCS_PORT -c -x -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo "Execute this command to the JCS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $CCS_HOST:$CCS_PORT -D $CCS_USER -w $CCS_PWD -f $FILE1 "


echo ""
echo "################################### IMPS UNIT & PERF TEST ################################################################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
echo "Execute this command to the IMPS Service to test single thread with dxmodify or ldapmodify "
echo "dxmodify  -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D \"$IMPS_USER\" -w $IMPS_PWD -f $FILE2 "
echo "Execute this command to the IMPS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $IMPS_HOST:$IMPS_PORT -D \"$IMPS_USER\" -w $IMPS_PWD -f $FILE2 "

Address the new bottleneck of MS Exchange / O365 Provisioning.

After parallel provisioning has been introduced with the new im_ccs.exe service, you may noticed that the number of transactions is still being throttled during performance testing.

Out-of-the-box MS Active Directory Global Throttling Policy has the parameter of PowerShellMaxConcurrency set to a default of 18 connection. Any provisioning that uses MS Powershell for MS Exchange and/or MS O365 will be impacted by this default parameter.

To address this bottleneck, we can create a new Throttling Policy and only assign the service ID that will be managing identities, to avoid a global change.

Example: New-ThrottlingPolicy MaxPowershell -PowerShellMaxConcurrency 100 & Set-Mailbox “User Name” -ThrottlingPolicy MaxPowershell

After this change has been made, restart the IM JCS/CCS Services, and retest again with your performance tools. Review the CCS ADS log for # of creations in 60 seconds, and you will be pleasantly surprise at the rate. The logs are the strong confirmation we are looking for.

Performance test (947 ADS accounts w/Exchange mailboxes in 60 seconds, 08:59:54 to  09:00:53) => Rate of 15 ids/second   (or 54 K ids/hr) with updated MaxPowershell = 100 thottlingpolicy.

The last bottleneck appears to be CPU availability to MS Exchange Supporting Services, w3wp.exe, the MS IIS Service. Which appears to be managing MS Powershell connections per its startup string of 

" c:\windows\system32\inetsrv\w3wp.exe -ap "MSExchangePowerShellAppPool" -v "v4.0" -c "C:\Program Files\Microsoft\Exchange Server\V15\bin\GenericAppPoolConfigWithGCServerEnabledFalse.config" -a \.\pipe\iisipme304c50e-6b42-4b26-83a4-229ee037be5d -h "C:\inetpub\temp\apppools\MSExchangePowerShellAppPool\MSExchangePowerShellAppPool.config" -w "" -m 0"