linux+oralce 11.2.0.3 RAC 应用突然变慢了

liwenchao_yn

从昨天开始应用反应慢了很多，之前都很正常，数据量才大概10G左右

linux+oralce 11.2.0.3 RAC

附件是今天上午的awr，请大神帮忙看看，鹰眼调优视频还是没学到技巧。

Liu Maclean(刘相兵

Elapsed:                  60.07 (mins)                  
DB Time:                  24.41 (mins)                  

db time很低， 如果说慢， 请用户给出慢的应用 模块名或者SQL_ID
否则无意义

liwenchao_yn

应用服务器CPU一直100% 下不来websphere
下面的是Exception-log.log
2013-12-19 08:59:27,056 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==FjV4CyPZh8mRp0xBQJdJJbx
2013-12-19 08:59:42,188 ERROR ExceptionLog - java.lang.IllegalStateException:
Session Object Internals:
id : FjV4CyPZh8mRp0xBQJdJJbx
hashCode : 619128039
create time : Thu Dec 19 08:47:05 GMT 2013
last access : Thu Dec 19 08:58:02 GMT 2013
max inactive interval : 36000
user name : anonymous
valid session : false
new session : false
overflowed : false
app name : default_host/sdpost_fsc
non-serializable app specific session data : null
serializable app specific session data : {}

        at com.ibm.ws.webcontainer.httpsession.SessionData.putValueGuts(SessionData.java:743)
        at com.ibm.ws.webcontainer.httpsession.SessionData.putValue(SessionData.java:1222)
        at com.ibm.ws.webcontainer.httpsession.SessionData.setAttribute(SessionData.java:1149)
        at com.ibm.ws.webcontainer.httpsession.HttpSessionFacade.setAttribute(HttpSessionFacade.java:150)
        at com.jstrd.tfas.workflow.listener.WorkItemQueryListener.doInitAllWorkItemList(WorkItemQueryListener.java:303)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:615)
        at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:216)
        at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:162)
        at com.exp.fcl.reflect.ReflectSupport.invokeMethod(ReflectSupport.java:138)
        at com.exp.web.util.DefaultEventDispatcher.invokeEvent(DefaultEventDispatcher.java:104)
        at com.exp.web.util.DefaultEventDispatcher.doEvent(DefaultEventDispatcher.java:81)
        at com.exp.web.util.DefaultEventDispatcher.dispatchEvent(DefaultEventDispatcher.java:45)
        at com.exp.web.util.BasePage.pageProc(BasePage.java:111)
        at com.ibm._jsp._workItemAllQuery._jspService(_workItemAllQuery.java:514)
        at com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:85)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:966)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:907)
        at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:145)
        at com.jstrd.base.security.SystemEnvSettingWebFilter.doFilter(SystemEnvSettingWebFilter.java:31)
        at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190)
        at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:130)
        at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:87)
        at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:696)
        at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:641)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
        at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:463)
        at com.ibm.wsspi.webcontainer.servlet.GenericServletWrapper.handleRequest(GenericServletWrapper.java:115)
        at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionServletWrapper.handleRequest(AbstractJSPExtensionServletWrapper.java:168)
        at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:92)
        at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:744)
        at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1425)
        at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:92)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:394)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:274)
        at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:984)
        at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:600)
        at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1812)
        at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:152)
        at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:213)
        at com.ibm.io.async.AbstractAsyncFuture.fireCompletionActions(AbstractAsyncFuture.java:195)
        at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:136)
        at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:193)
        at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:725)
        at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:847)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1498)

2013-12-19 08:59:42,219 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==FjV4CyPZh8mRp0xBQJdJJbx
2013-12-19 08:59:44,076 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==O3Yc_IA601fpY7TTZvwkEMQ
2013-12-19 09:01:13,776 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==kn1dhJpXUDHLKd0TnudfgnB
2013-12-19 09:01:29,626 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==EkBVIv6jxBPTssmDWVFsTPz
2013-12-19 09:01:58,907 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==vWKB81L_jeSbpb25AkkFnFB
2013-12-19 09:02:13,976 ERROR ExceptionLog - com.exp.security.LoginException
2013-12-19 09:02:20,497 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==cQPXKDY4ZOkGu8yElyNTdlu
2013-12-19 09:02:22,759 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==iBT0y_DE24vLyJbfybzNW0r
2013-12-19 09:02:42,930 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==UoojxtJ7faN2RiRAa32B57P
2013-12-19 09:03:31,197 ERROR ExceptionLog - com.exp.security.LoginException
2013-12-19 09:03:58,793 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==pVKtAXD6SFhC1G7MGmi9L-h
2013-12-19 09:04:38,183 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==auBbhHCzsgsKlNok7ZLVjBK
2013-12-19 09:05:19,492 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==WPF2udZNf44krwyTFy1iEC1
2013-12-19 09:06:37,492 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==h0mMFOHD-pFqWHiB_6ttXfT
2013-12-19 09:06:56,009 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==Iq_nJ0Pdt6VyPYBiUr3Cmhs
2013-12-19 09:08:29,485 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==Elv1FsuEM_2ZzfBpzqk-pe8
2013-12-19 09:08:37,269 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==B_NNSZg9rhObmmupq2yYp3l
2013-12-19 09:09:23,071 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==QBrWqMD_3Njrin336O1kXne
2013-12-19 09:09:28,312 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==bojiqNGgLw7ijj9N9YWxBYm
2013-12-19 09:10:12,164 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==CKW_gRHUlxVYRRKzazdlTkA
2013-12-19 09:10:33,053 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==9oSftupCeo_LqZMfa0Zx5kc
2013-12-19 09:10:34,301 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==QBrWqMD_3Njrin336O1kXne
2013-12-19 09:10:40,041 ERROR ExceptionLog - session没有com.exp.security.LoginContext_KEYsessionid==-fI5aGVHR4MJqXfBdjvtH6f

Liu Maclean(刘相兵

你应用服务器 CPU高， 瓶颈未必在 DB啊，这样推理是没有依据的