You are an IBM Connections admin and you would like to know how to fix your problems and open tickets? We will talk about some troubleshooting basics and other useful things to ensure that you have enough skills to get your Connections running smoothly. There are many areas where you need different debugging skills. We will cover backend like DB2, TDI, SSO and some IBM integrations like Docs, CCM, Cognos or Forms Experience Builder. Of course we will cover Connections itself, too. You will also get some helpful best practises which will help you within your daily work. A presentation by Christoph Stoettner & Nico Meisenzahl

1. Toronto, June 6-7 2016
IBM Connections
Troubleshooting or
"get the cow off the ice"
Nico Meisenzahl & Christoph Stoettner
panagenda

2. PLATINUM & SPOTLIGHT SPONSORS
GOLD SPONSORS
SILVER SPONSORS
BRONZE SPONSORS

3. Christoph Stoettner
• Senior Consultant at panagenda
• IBM Notes / Domino since 1999
• IBM Connections since version 2.5 /
2009
• Many years of experience in:
• Migrations
• Administration and installation
• Performance analysis
• Joined panagenda in 2015 focusing in:
• IBM Connections deployment and optimization
• IBM Connections monitoring
• Husband of one & father of two,
Bavarian
@stoeps
linkedin.com/in/christophstoettner
www.stoeps.de
christophstoettner
+49 173 8588719
christoph.stoettner@panagenda.com

4. Nico Meisenzahl
• Consultant at panagenda
• IBM Notes / Domino since 2008
• IBM Connections since version 3.0
/ 2010
• Many years of experience in:
• Consulting
• Migrations & Administration
• Joined panagenda in 2016
focusing in:
• IBM Connections Consulting
• ICS deployment & optimization
@nmeisenzahl
linkedin.com/in/nicomeisenzahl
meisenzahl.org
nico.meisenzahl
+49 170 7355081
nico.meisenzahl@panagenda.com

5. Agenda
• Howto: Troubleshooting
• Troubleshooting…
• Connections itself
• Backend (DB2, TDI, WebSphere, SSO)
• Optional add-ons (Docs, FEB, Cognos, CCM)
• Tools

6. Toronto, June 6-7 2016
Howto: Troubleshooting

7. Reproduce the error
• Reproducible and/or periodically?
• A sequence error?
• Client-side or server-side problem?
• Analyze the root cause

8. Be aware of the big picture
• Client-side problems
• Debug in different Browsers (IE, FF, Chrome)
• Do NOT use a server IE
• Server-side: IBM Connections is based on
many components
• debug on “high level” first
• get an overview which backend service is causing
the error

9. Configuration changes
• Changes in…
• Connections configuration
• Backend (WebSphere, Database, HTTP)
• Firewall or network
• OS, hardware or VM
Tip: Even the smallest configuration change can
have big consequences!

10. Analyze log files and browser
• Analyse log files
• Atom.io, Notepad++ or less/tail
• Baretail or tail –f
• ELK stack
• Tools for client-side problems
• Firebug or Developer-Tools
• BurpSuite or Fiddler

11. Analyze root cause
• Find a hint inside the log
• Network timeout or DNS
• SQL errors
• LDAP errors
• Syntax errors in configuration files „xxx-
config.xml“
• Error stack
Tip: In a clustered environment, start and
analyze only one Node (if possible)

12. Find support
• Knowledge Center http://goo.gl/HFzTmv
• Troubleshooting Tips https://goo.gl/mU1EQ9
• IBM Connections Forum http://goo.gl/CVvQCU
• Community Blogs and/or Chats
• Fix Central
• PMR

13. Toronto, June 6-7 2016
Troubleshooting

14. Get the cow off the ice
How to do this?

15. Agenda

16. WebSphere Application Server
(IBM Connections) logs
• SystemOut.log
• SystemErr.log
• trace.log (if tracing is enabled)
• Log path:
• <wasroot>/profiles/<profilename>/logs/<servername>/

17. Analyze WAS log files
• Time stap: 24h time stamp with milli-seconds
• Thread id: eight character hexadecimal value
• Short name: typically java class name
• Event type: one character only (E, W, I,…)
• Message identifier: String based on component
• Message: Some information

18. WAS Event types
• F - Fatal message
• E - Error message
• W - Warning message
• A - Audit message
• I - Informational message
• C - Configuration message
• D - Detail message
• O - Messages that are written directly to System.out
by an application
• R - Messages that are written directly to System.err by
an application
• Z - Place holder to indicate type was not recognized

19. WAS Message identifier
• Prefix by Application or Server (CLFRW)
• Specific application code (0042)
• Event Type (I)

20. Read trace stack
• First line displays key information
• “Caused by” displays root cause

21. Environment information
• First log lines on server startup
• WebSphere version
• OS version, Process id
• Installation path

22. Enable tracing
• Enable tracing using
ISC
• Runtime or configuration
only
• Define tracing based
on
• App prefix / error stack
• Must gather (PMR)

23. Search issues
• http(s)://<fqdn>/search/serverStatus
• Display index, seedlists, log information
• Data is displayed for one node only
• Using node fqdn to access different nodes

24. Debug Search
• Search queries (runtime)
• com.ibm.connections.search.index.searching.*=all
• Crawling & seedlists
• com.ibm.connections.search.index.indexing.*=all:
com.ibm.connections.search.seedlist.*=all:
com.ibm.connections.httpClient.*=all

25. Recreate Search Index
• SearchService.startBackgroundIndex()
• Crawls seedlists
• Extracts the file content
• Create index
Tip: Use „all_configured“ to index all apps

26. CLFRW0394E: Search indexing of
services ...
• Search index not ready
• interruption at index creation
• CLFRW0283E: Search has encountered a problem while
crawling
• CLFRW0027E: Error Indexing component <app> for
search
• INDEX.READY file not present
• Recreate and enable tracing

27. com.ibm.connections.directory.services
.exception.DSOutOfServiceException
• Access Connections not possible
• DSX not working
• Check SSO settings
• Domain
• LtpaToken Cookie name
Tip: WebSphere FP sometimes resets SSO
domain

28. EJPVJ9284E: Unable to get the groups
from the directory for the user…
• User was not able to access Connections
anymore
• WAS LDAP bind user had no read access
to one of the groups the user was member

29. Debug wsadmin
• Enable trace within wsadmin session
• AdminControl.trace(‘com.ibm.*=all’)
• <wasroot>/profiles/<profilename>/logs/wsadmin.traceout

30. Database connections
• Check datasouces
• ISC – Resources – JDBC – Data sources
• Check logs for more information
• DB2 server log
• <instanceroot>/sqllib/db2dump/
• db2diag.log
• db2diag.xxx.log (log rotation, you should enable this!)
Tip: Oracle users have password expiration
enabled by default!

31. HTTP Server (IHS & Plugins) logs
• IBM HTTP Server
• <installroot>/logs/
• error_log
• access_log
• based on configuration
• WebSphere AppServer Plugins
• <installroot>/logs/<webserver>/http_plugin.log

32. HTTP 404 not found
• Outdated Plugin configuration
• Restart IHS
• WAS Plugin configuration issue
• http_plugin.log

33. HTTP 404 not found
• AppServer or App down
• Network issue
• http_plugin.log

34. HTTP 500 Internal Server Error
• Unexpected error
• http_plugin.log
• Configuration issue
• WAS Root certificate not trusted or missing
• SSL certificate expired

35. SDI / TDI logs
• <tdisol>/log/ibmdi.log
• TDI log file
• <tdisol>/employee.*
• Files include all changed users (adds, update, delete,
error, skip)
• <tdisol>/syncupdates/*
• Temporary files within the sync
• Including database dump and ldiff
• sync_updates_clean_temp_files=false (default: true)
• profiles_tdi.properties
Tip: Check lock file

36. Analyze SDI logs
• Error code prefix
• CLFRN: Profile & User synchronization
• CTGDIS: TDI itself
• Error code suffix
• I, E, W, …

37. Debug SDI
• Profile & User synchronization
(<tdisol>/etc/profiles_tdi.properties)
• source_ldap_debug=true
• debug_update_profile=true
• debug_collect=true
• TDI issues (<tdisol>/etc/log4j.properties)
• log4j.rootCategory=DEBUG, Default

38. Cognos BI logs
• Cognos BI
• SystemOut.log & trace.log
• <installroot>/logs/cogserver.log
• Cognos Transformer
• <installroot>/logs/cogserver.log
• PowerCube build
• <installroot>/metricsmodel/trxschelog.log
• <userhome>/Transformer/Logs/*.log (win only)

39. BMT-MD-6003 No connection to the
data source …
• PowerCubes not created yet
• Check
• Cronjobs or scheduled jobs
• trxschelog.log

40. Debug Cognos BI & Metrics
• Communication between Cognos BI and
Connections Metrics
• SonataHttpUsage=all:
SonataHttpHeader=all:
SonataHttpBody=all:
com.ibm.connections.httpClient.*=all:
com.ibm.connections.metrics.*=all
• Connections Metrics Servlet
• com.ibm.connections.metrics.cognos.servlet.*=all

41. FileNet logs
• SystemOut.log & trace.log
• FileNet logs
• <wasprofile>/<servername>/p8_server_error.log
• <wasprofile>/<servername>/p8_server_trace.log
• <wasprofile>/<servername>/pesvr_system.log
• <wasprofile>/<servername>/pesvr_trace.log

42. FileNet urls
• http://<fqdn>/FileNet/Engine
• Server status, version, sonata/waltz version
• http://<fqdn>/P8CE/Health
• Health checks for authentication, stores and
database
• http://<fqdn>/dm
• FNCS version and configuration

43. Debug CCM Widget
• Widget issues
• com.ibm.quickr.communitylibrary.*=all:
com.ibm.lconn.widgets.service.*=all:
com.ibm.lconn.widgets.actions.*=all
• Authentication issues
• com.ibm.connections.directory.services.*=all:
com.ibm.connections.directory.services.*=all:
com.ibm.connections.httpClient.*=all

44. Debug FileNet using ACCE
• http(s)://<fqdn>/acce

45. Debug FileNet using JVM Properties
• Add generic JVM properties
• -Dlog4j.configuration=file:<path>/log4j.xml
-DskipTLC=true
• Copy & customize sample log4j.xml
• <ContentEngineRoot>/config/samples
• 20 subsystems (db, engine, security, search,…)

46. Docs/Viewer logs & urls
• SystemOut.log & trace.log
• http(s):<fqdn>/vsanity/check
• http(s):<fqdn>/sanity/check?app=all&querytype=report
• http(s):<fqdn>/*/version.txt

47. Debug LTPA between Domino & WAS
• Debug on Domino side (notes.ini)
• Debug_SSO_Trace_Level=2
• Webauth_verbose_trace=1
• WebSess_verbose_trace=1
• Debug_outfile=<logfilepath>
• Debug on WebSphere
• com.ibm.ws.security.ltpa.*=all

48. Debug Kerberos
• Configuration
• com.ibm.ws.security.spnego.*=all:
com.ibm.ws.security.*=all:
com.ibm.issw.spnegoTAI.*=all:
com.ibm.security.krb5.*=all
• Runtime
• com.ibm.connections.httpClient.*=all:
com.ibm.connections.directory.services.*=all:
com.ibm.websphere.wim.*=all:
com.ibm.ws.wim.*=all
• Fiddler & BurpSuite

49. Toronto, June 6-7 2016
Tools

50. Analyze logs
• Analyze logs live
• Baretail
• tail –f
• View logs
• Atom.io, Notepad++
• less, tail
• ELK Stack
• Elasticsearch, Logstash, Kibana
• Small Docker deployment

51. Analyze Client-side
• Browser
• Firebug / Developer Tools
• Intercepting proxies
• Fiddler
• BurpSuite
• VMs with different IE versions
• Without GPO
• https://www.modern.ie/en-us/virtualization-tools

52. SQL Clients
• db2 command
• IBM Datastudio
• Dbeaver, DBVisualizer, squirrelSQL
Tip: Database write access is not supported!

53. LDAP Clients
• ldapsearch command
• Softerra LDAP Browser
• Apache Directory Studio

54. Analyze Network
• Wireshark
• tcpdump
If you ask the * admin:
There is never a problem within the *!

*= Network, SAN, VM, ...

55. Useful Blogs
• http://ibmconnections.com
• http://turtleblog.info
• http://portal2portal.blogspot.de
• https://www.urspringer.de
• http://socialconnections.info
• http://blog.robertfarstad.com
• http://www.curiousmitch.com
• http://www.ramsit.com/category/blog
• http://techblog.gis-ag.info
• https://milanmatejic.wordpress.com
• http://ibmdocs.com
• http://domino.elfworld.org
• https://dontforgetthe0.com
• http://dilf.me.uk/socialshazza
• http://www.stoeps.de
• http://scripting101.org
• http://meisenzahl.org
• http://martin.leyrer.priv.at
• http://kbild.ch
• http://www.notesgoddess.net
• http://www.dominodiva.com
• http://notesbusters.com
• https://rob59blog.wordpress.com
• http://connections101.info
• http://brandlrainer.blogspot.de
• https://collaborationben.com

56. Thank you very much for your attention!
panagenda GmbH – Make Your Data Work for You
Lahnstr. 17 ● 64646 Heppenheim (Germany)
Skype: christophstoettner ● Cell: +49 173 8588719
E-Mail: christoph.stoettner@panagenda.com
Christoph Stoettner
Senior Consultant
panagenda GmbH – Make Your Data Work for You
Lahnstr. 17 ● 64646 Heppenheim (Germany)
Skype: nico.meisenzahl ● Cell: +49 170 7355081
E-Mail: nico.meisenzahl@panagenda.com
Nico Meisenzahl
Consultant

57. PLATINUM & SPOTLIGHT SPONSORS
GOLD SPONSORS
SILVER SPONSORS
BRONZE SPONSORS