Recomendados
Recomendados
Mais conteúdo relacionado
Destaque
Destaque (17)
Semelhante a Tales from the WebLogic Frontline
Semelhante a Tales from the WebLogic Frontline (20)
Último
Último (20)
Tales from the WebLogic Frontline
- 1. Tales from the WebLogic Front Line(What NOT to do with WebLogic) Steve Millidge C2B2 Consulting Limited
- 2. Agenda Health Warning Setting the Scene Common Anti-patterns Ain’t misbehaving This could kill
- 3. C2B2 Consulting Fast Performance and Scalability Reliable High Availability and Recoverability Manageable Monitoring and Supporting Secure Security
- 4. First Day At Work
- 5. It’s All Gone to *#*@!
- 8. Meet the Project Manager
- 11. Philosophy Don’t Panic you didn’t #@#@ it up. Suspect Configuration over Coding Suspect bespoke code over WebLogic code WebLogic is FAST Computers are Logical and Rational
- 14. Too Many Datasources Data Source per Component All to the Same Database JNDI Name per Component C1 C1 C1 DS1 DS1 DS1 Oracle
- 15. Why It’s Bad Requires XA for all Datasources Developers receive strange “XA” exception Untick Honour Global Transactions Makes Exception “go away” Datasources now in autocommit DATA CORRUPTION!! Requires more DB connections BIND same Datasource to multiple JNDI locations
- 17. Inadvertent TX Coupling WLS 1 WLS 2 EJB 1 REQUIRED Invoke EJB 2 REQUIRED Oracle 1 Oracle 2
- 18. Why It’s BAD Results in “Strange Exceptions” if not All XA Is the REALLY what you want XA is very heavyweight And not guaranteed XA requires particular database configuration You have 4 minimum XA resources to coordinate Fix using REQUIRES NEW
- 19. Clustering is TOO Heavy
- 20. WLS Cluster WLS Session Clustering WLS 1 WLS 2 WLS 3 WLS 4 1 2 Don’t Use the Session as a Cache USE A CACHE
- 21. Too Much logging
- 22. Too Much Logging Too Synchronized New Application 30% Slower Tx Timeouts High CPU Many Threads waiting on Logging Monitor Debug turned on, on one node In Memory Buffer Each Thread Logged to in memory queue Writer thread writes messages to file Under heavy load memory queues grew unbounded 1.5Gb heap in log messages
- 23. Ain’tMisbehavin’
- 24. Session Shenanigans Serialization Session Attributes must be Serializable See this one all the time No Failover!!! Working as per Spec No Put You must Put dirtied objects back in the Session Don’t Get modify then no Put No Failover!!! Working as per Spec
- 25. Exceptions and TX What happens to the TX when you throw an Exception from an EJB method? Application ExceptionCommit System ExceptionRollback
- 26. Poisoned JMS
- 27. Poisoned JMS Eats CPU Prevents Delivery Kills Server Performance JMS Q RepeatForever Set Redeliver Delay Set Redelivery Limit Set Error Destination Exception
- 28. Poisoned JMS (Paging) WLS Keeps Messages in Memory Causes OutOfMemory Exceptions Configure Paging Thresholds Configure at JMS Server level
- 29. This Can Kill You
- 30. Too Much RAM -XMS2G –XMX2G 32 bit Can be Too Much O/S Reserves RAM VM Starts up When you need it! HOTSPOT DUMP
- 31. Node Manager Configuration Not Good Out of the Box Auto Kill If Failed = TRUE Auto Restart = TRUE Restart Delay should be set Restart Max should be set Set Panic Action to System.exit
- 32. Disappearing Datasource Start WebLogic when DB not Available Datasource is not created Connection Creation Retry Frequency Seconds should be non-zero
- 33. Summary Production Configuration is Complex Call in the Professionals
- 34. Example Bullet Slide Bullet 1
Notas do Editor
- This is the problem solving philosophy I tell our consultantsConfiguration over Coding: Out of the box WLS is not setup for production needs many changes for production not usually done by developers“Out of the Box” set up for developersWLS is likely to have fewer bugs than custom code. Proven throughout my career. YES there are bugs but I’ve probably only reported a handful compared to customer bugsWLS if fundamentally quick a few ms for a simple JSP page to render. Everything else is your overhead!If you do something once and get a certain result do it again you should get the same resultScientific Method!!
- These are things we’ve seen people do a lot which you shouldn’t do.
- We see many examples of lots and lots of datasourcesShow a real example
- Developer writes a bunch of components in separate EARSEach EAR has it’s own DatasourceAll going to the same databaseC1 calls C2 call C3 all in the same TransactionSimilar antipattern with JMS Connection Factories
- Strange XA exceptions are seen by developers relating to LLR optimisationUntick Honour Global Transactions to make Exception “Go Away”Some Datasources then in “autocommit” mode as they are not enlisted in the Global TXRequires full XA configuration for Oracle!Avoid XA especially when you only have 1 database!!!!!
- This picture of sexual activityCover a scenario we see in quite complex EJB environments with a number of EJB applications
- WE HAVE two APPLICATIONS developedindependantlyLater application 1 needs to retrieve data from 2Initially possibly get a security issueSet Domain credential to be the sameReceive XA Exception again related to LLTX’s inadvertently coupled as WLS will enlist second app in first TX
- Exception will be failure on client side to lookup JDBC Datasource on server side when it is actually there!Often also get security errors relating to anonymous user trying to commit TX depending on WLS versionCan also get difficulties with NATREQUIRES NEW ON SIDE 2Seen a developer try to fix by placing a wrapper bean on side 1 still couples at least removes Oracle 1.
- Typical Architect commentAsk question anyone here think clustering too heavyweight!!!Fallacy why is clustering heavyPerhaps your Session is way too heavySo you don’t need failover???Your customers probably do!!
- Explain Primary SecondaryReplication on attribute changeFailure causes a new primary Seen customers with Mbs of data in the cache including complete XML documentInternet facing watch out for googlebot et al, don’t use cookies so don’t do massive session initialisation or detect bots.20000 sessions created on average by BOTS on large ecommerce site.
- Logging is VERY Heavily SynchronisedGotta Fit all those messages from Multiple Threads into a single FileIn Memory Buffer Logging can Kill youExample next slide
- Tell story 1 about Matt’s case studyTell Story 2 Atos case studySystem Crash every day near 4pm – slow down to unrepsonsiveObserved 100% cpu utilisationTypical of constant GCConsole run out of memoryUse JMS for Logging is more performant
- Things that cause problems but server is doing the right thing as per spec or configuration
- See a couple of other typical session issues a lotSerialization -> Write code to Serialize to a file all your session attributesWATCH OUT FOR 3rd Party web frameworks know what they put in the session (Form Beans)
- System Exception is runtime or RemoteExceptionWe see this in many applications where the behaviour is not what people are expectingInsufficient knowledge of the details of the EJB spec.One company policy wrap all Exceptions into their applications so as not to expose 3rd party apis.Wrapped remotes with application exception -> CAUSING COMMITS
- A couple of typical JMS issues
- Poisoned Message on JMS -> constant spinning CPU (100%)Other messages can never be deliveredSeen this on customer killing all order processingThis slide buildsTalk through the BuildExplain each PointRedeliver Delay allows other messages through watch out if require strict ordering
- Not such a big issue since later versions of WLS 9.2 onwards – still requires tuningMessage headers still in memoryConsider setting quotas and messages can arrive quicker than they are pagedAlso will use 1/3 JVM memory
- 3 issues we have seen take out servers and cause system outages
- Especially on Sun JDKNot as common on JrockitOnly for 32bit platformsSeen one production system unable to restart because of this and numerous system hot spot dump.Large number of JMS messages (see paging from before). Hot Spot crashed couldn’t restart.Reducing JVM heap size fixed the problem
- Node Manager should be set up correctly.Powerful and important – Amazingly some customers don’t use it - NEVER been to a customer where this has been properly configuredBy default will attempt a number of quick restarts and then give up.Configure to try a number of times with a delay for a long period.In 9 and 10 Set Panic action to exit
- Major Government WLI System taken out by this.Regular restart of WLS carried out same time upgrade of Oracle. Oracle outage happened before but never seen a problem (Connection Testing etc. all enabled).WLS restarted while Oracle outSupport team Arrived morning to see that Name Not Found JNDI Exceptions in the log files.ALL ORDER PROCESSING FAILEDDefault setting is to switch this off!! Again suitable for development as quick errors!!
- “Out of the Box” WLS supports Ease of development.Last preproduction domain review I did for a large dual cluster WLS and WLI domain had 80+ recommendations with over 10 Classified as HIGH which means could cause a system outage, significant down time or major security breach.