在执行表空间的迁移操作后,RAC数据库出现了上面的错误。
在节点1也就是执行表空间迁移的节点上,alert文件中出现了大量的ORA-600(kjbrchkpkeywait:timeout)错误,同时还出现了ORA-7445(kjxmgmb_breq_header)以及ORA-603错误。
详细信息如下:
Thu Sep 17 14:47:40 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_27466.trc:
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [792720], [61114], [], [], [], []
Thu Sep 17 14:47:41 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_27466.trc:
ORA-07445: exception encountered: core dump [kjxmgmb_breq_header()+136] [SIGSEGV] [Address not mapped to object] [0x000000004] [] []
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [792720], [61114], [], [], [], []
Thu Sep 17 14:47:41 2009
Trace dumping is performing id=[cdmp_20090917144741]
Thu Sep 17 14:48:23 2009
Trace dumping is performing id=[cdmp_20090917144823]
Thu Sep 17 14:49:05 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_3484.trc:
ORA-00600: 内部错误代码, 参数: [kjbrchkpkeywait:timeout], [791819], [792805], [61114], [], [], [], []
Thu Sep 17 14:49:06 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_3484.trc:
ORA-00600: 内部错误代码, 参数: [600], [], [], [], [], [], [], []
Thu Sep 17 14:49:06 2009
Trace dumping is performing id=[cdmp_20090917144906]
Thu Sep 17 14:49:48 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_3484.trc:
ORA-00603: ORACLE 服务器会话因致命错误而终止
ORA-00600: 内部错误代码, 参数: [600], [], [], [], [], [], [], []
Thu Sep 17 14:49:48 2009
Trace dumping is performing id=[cdmp_20090917144948]
Thu Sep 17 14:50:29 2009
Trace dumping is performing id=[cdmp_20090917145029]
Thu Sep 17 14:59:16 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_5801.trc:
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [793416], [61114], [], [], [], []
Thu Sep 17 14:59:17 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_5801.trc:
ORA-00600: internal error code, arguments: [600], [], [], [], [], [], [], []
Thu Sep 17 14:59:17 2009
Trace dumping is performing id=[cdmp_20090917145917]
Thu Sep 17 14:59:18 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_5801.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [600], [], [], [], [], [], [], []
Thu Sep 17 14:59:20 2009
Trace dumping is performing id=[cdmp_20090917145920]
Thu Sep 17 14:59:23 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_17753.trc:
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [793423], [61114], [], [], [], []
Thu Sep 17 14:59:24 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_17753.trc:
ORA-07445: exception encountered: core dump [kjxmgmb_breq_header()+136] [SIGSEGV] [Address not mapped to object] [0x000000004] [] []
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [793423], [61114], [], [], [], []
Thu Sep 17 14:59:24 2009
Trace dumping is performing id=[cdmp_20090917145924]
Thu Sep 17 14:59:35 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_20760.trc:
ORA-00600: internal error code, arguments: [kclcls_8], [], [], [], [], [], [], []
Thu Sep 17 15:00:19 2009
Trace dumping is performing id=[cdmp_20090917150019]
Thu Sep 17 15:00:20 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_23218.trc:
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [793480], [61114], [], [], [], []
Thu Sep 17 15:00:21 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_23218.trc:
ORA-07445: exception encountered: core dump [kjxmgmb_breq_header()+136] [SIGSEGV] [Address not mapped to object] [0x000000004] [] []
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [793480], [61114], [], [], [], []
.
.
.
Thu Sep 17 15:08:43 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_17033.trc:
ORA-07445: exception encountered: core dump [kjxmgmb_breq_header()+136] [SIGSEGV] [Address not mapped to object] [0x000000004] [] []
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [793982], [61114], [], [], [], []
Thu Sep 17 15:09:21 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_7778.trc:
ORA-00600: 内部错误代码, 参数: [kjbrchkpkeywait:timeout], [793120], [794021], [139374], [], [], [], []
Thu Sep 17 15:09:23 2009
Trace dumping is performing id=[cdmp_20090917150923]
Thu Sep 17 15:09:23 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_7778.trc:
ORA-00600: 内部错误代码, 参数: [600], [], [], [], [], [], [], []
Thu Sep 17 15:09:25 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_7778.trc:
ORA-00603: ORACLE 服务器会话因致命错误而终止
ORA-00600: 内部错误代码, 参数: [600], [], [], [], [], [], [], []
Thu Sep 17 15:09:26 2009
Thread 1 advanced to log sequence 49
Current log# 4 seq# 49 mem# 0: +MEMBER/tradedb/tradedb_redo1_4_1_1g
Current log# 4 seq# 49 mem# 1: +MEMBER/tradedb/tradedb_redo1_4_2_1g
Thu Sep 17 15:09:26 2009
Trace dumping is performing id=[cdmp_20090917150926]
Thu Sep 17 15:09:28 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_1851.trc:
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [794028], [61114], [], [], [], []
.
.
.
Thu Sep 17 15:18:25 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_18739.trc:
ORA-00603: ORACLE 服务器会话因致命错误而终止
ORA-00600: 内部错误代码, 参数: [600], [], [], [], [], [], [], []
其中ORA-603的错误也是ORA-600错误所造成的:
SQL> exec dbms_stats.gather_schema_stats(user, cascade => true)
BEGIN dbms_stats.gather_schema_stats(user, cascade => true); END;
*
第 1 行出现错误:
ORA-00603: ORACLE 服务器会话因致命错误而终止
这是收集统计信息报错对应的trace文件信息:
bash-3.00$ more /data/oracle/admin/tradedb/udump/tradedb1_ora_7778.trc
/data/oracle/admin/tradedb/udump/tradedb1_ora_7778.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
ORACLE_HOME = /data/oracle/product/10.2/database
System name: SunOS
Node name: ser1
Release: 5.10
Version: Generic_139555-08
Machine: sun4u
Instance name: tradedb1
Redo thread mounted by this instance: 1
Oracle process number: 34
Unix process pid: 7778, image: oracle@ser1 (TNS V1-V3)
*** ACTION NAME:() 2009-09-17 14:51:45.397
*** MODULE NAME:(SQL*Plus) 2009-09-17 14:51:45.397
*** SERVICE NAME:(SYS$USERS) 2009-09-17 14:51:45.397
*** SESSION ID:(147.6274) 2009-09-17 14:51:45.397
*** 2009-09-17 14:51:45.397
Cleanout Start
Cleanout End
*** 2009-09-17 15:09:21.300
GCS RESOURCE 50f3f92f0 hashq [51057d560,51fb74638] name[0x2d00b.170000] pkey 62260
grant 50c8ba7b8 cvt 0 send 0,0 write 0,0@65535
flag 0x0 mdrole 0x0 mode 0 scan 0 role LOCAL
disk: 0x0103.0706ae1d write: 0x0000.00000000 cnt 0x0 hist 0x7d
xid 0x0000.000.00000000 sid 0 pkwait 901s
pkey 62260
hv 32 [stat 0x0, 0->0, wm 32767, RMno 0, reminc 4, dom 0]
kjga st 0x4, step 0.0.0, cinc 4, rmno 10, flags 0x0
lb 0, hb 0, myb 14849, drmb 14849, apifrz 0
GCS SHADOW 50c8ba7b8,2 sq[50f3f9310,50f3f9310] resp[50f3f92f0,0x2d00b.170000] pkey 62260
grant 0 cvt 0 mdrole 0x20 st 0x40 GRANTQ rl LOCAL
master 0 owner 1 sid 0 remote[526fcda58,3] hist 0xa6752557
history 0x17.0x15.0x12.0x1d.0x26.0x2. cflag 0x0 sender 0 flags 0x0 replay# 0
disk: 0x0000.00000000 write request: 0x0000.00000000
pi scn: 0x0000.00000000
msgseq 0x2 updseq 0x0 reqids[3,0,0] infop 0x0
*** 2009-09-17 15:09:21.300
ksedmp: internal or fatal error
ORA-00600: 内部错误代码, 参数: [kjbrchkpkeywait:timeout], [793120], [794021], [139374], [], [], [], []
Current SQL statement for this session:
select /*+ no_parallel(t) no_parallel_index(t) dbms_stats cursor_sharing_exact use_weak_name_resl dynamic_sampling(0) no_monitoring
*/ count(*),count("REMOTE_USER_CODE"),count(distinct "REMOTE_USER_CODE"),sum(sys_op_opnsize("REMOTE_USER_CODE")),substrb(dump(min(su
bstrb("REMOTE_USER_CODE",1,32)),16,0,32),1,120),substrb(dump(max(substrb("REMOTE_USER_CODE",1,32)),16,0,32),1,120),count("REMOTE_USE
R_PASS"),count(distinct "REMOTE_USER_PASS"),sum(sys_op_opnsize("REMOTE_USER_PASS")),substrb(dump(min(substrb("REMOTE_USER_PASS",1,32
)),16,0,32),1,120),substrb(dump(max(substrb("REMOTE_USER_PASS",1,32)),16,0,32),1,120),count("REMOTE_USER_BROWSER"),count(distinct "R
EMOTE_USER_BROWSER"),sum(sys_op_opnsize("REMOTE_USER_BROWSER")),substrb(dump(min(substrb("REMOTE_USER_BROWSER",1,32)),16,0,32),1,120
),substrb(dump(max(substrb("REMOTE_USER_BROWSER",1,32)),16,0,32),1,120),count("REMOTE_USER_OS"),count(distinct "REMOTE_USER_OS"),sum
(sys_op_opnsize("REMOTE_USER_OS")),substrb(dump(min(substrb("REMOTE_USER_OS",1,32)),16,0,32),1,120),substrb(dump(max(substrb("REMOTE
_USER_OS",1,32)),16,0,32),1,120),count("REMOTE_USER_IP"),count(distinct "REMOTE_USER_IP"),sum(sys_op_opnsize("REMOTE_USER_IP")),subs
trb(dump(min(substrb("REMOTE_USER_IP",1,32)),16,0,32),1,120),substrb(dump(max(substrb("REMOTE_USER_IP",1,32)),16,0,32),1,120),count(
"SERVER_FIRST_MENU"),count(distinct "SERVER_FIRST_MENU"),sum(sys_op_opnsize("SERVER_FIRST_MENU")),substrb(dump(min(substrb("SERVER_F
IRST_MENU",1,32)),16,0,32),1,120),substrb(dump(max(substrb("SERVER_FIRST_MENU",1,32)),16,0,32),1,120),count("SERVER_SECOND_MENU"),co
unt(distinct "SERVER_SECOND_MENU"),sum(sys_op_opnsize("SERVER_SECOND_MENU")),substrb(dump(min(substrb("SERVER_SECOND_MENU",1,32)),16
,0,32),1,120),substrb(dump(max(substrb("SERVER_SECOND_MENU",1,32)),16,0,32),1,120),count("SERVER_HANDLER"),count(distinct "SERVER_HA
NDLER"),sum(sys_op_opnsize("SERVER_HANDLER")),substrb(dump(min(substrb("SERVER_HANDLER",1,32)),16,0,32),1,120),substrb(dump(max(subs
trb("SERVER_HANDLER",1,32)),16,0,32),1,120),count("SERVER_OPERATE"),count(distinct "SERVER_OPERATE"),sum(sys_op_opnsize("SERVER_OPER
ATE")),substrb(dump(min(substrb("SERVER_OPERATE",1,32)),16,0,32),1,120),substrb(dump(max(substrb("SERVER_OPERATE",1,32)),16,0,32),1,
120),count("SERVER_ERROR_MSG"),count(distinct "SERVER_ERROR_MSG"),sum(sys_op_opnsize("SERVER_ERROR_MSG")),substrb(dump(min(substrb("
SERVER_ERROR_MSG",1,32)),16,0,32),1,120),substrb(dump(max(substrb("SERVER_ERROR_MSG",1,32)),16,0,32),1,120),count("WEB_EXPEND_TIME")
,count(distinct "WEB_EXPEND_TIME"),sum(sys_op_opnsize("WEB_EXPEND_TIME")),substrb(dump(min("WEB_EXPEND_TIME"),16,0,32),1,120),substr
b(dump(max("WEB_EXPEND_TIME"),16,0,32),1,120),count(distinct "ERROR_CREATE_DATE"),substrb(dump(min("ERROR_CREATE_DATE"),16,0,32),1,1
20),substrb(dump(max("ERROR_CREATE_DATE"),16,0,32),1,120),sum(sys_op_opnsize("ID")),substrb(dump(min("ID"),16,0,32),1,120),substrb(d
ump(max("ID"),16,0,32),1,120) from "ZHEJIANG"."EMED_WEB_LOG" sample ( .4800581069) t where TBL$OR$IDX$PART$NUM("ZHEJIANG"."EMED_W
EB_LOG",0,4,0,"ROWID") = :objn
----- PL/SQL Call Stack -----
object line object
handle number name
52083e9d0 1218 package body SYS.DBMS_SYS_SQL
52083ef30 334 package body SYS.DBMS_SQL
520974208 10750 package body SYS.DBMS_STATS
520974208 12215 package body SYS.DBMS_STATS
520974208 12494 package body SYS.DBMS_STATS
520974208 12623 package body SYS.DBMS_STATS
520974208 13090 package body SYS.DBMS_STATS
520974208 13291 package body SYS.DBMS_STATS
520974208 13682 package body SYS.DBMS_STATS
520974208 13760 package body SYS.DBMS_STATS
520974208 13719 package body SYS.DBMS_STATS
4d271a018 1 anonymous block
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+744 CALL ksedst() 000000840 ?
FFFFFFFF7FFEF34C ?
000000000 ?
FFFFFFFF7FFEBE40 ?
FFFFFFFF7FFEABA8 ?
FFFFFFFF7FFEB5A8 ?
kgerinv()+200 PTR_CALL 0000000000000000 000106400 ? 10652D264 ?
10652D000 ? 00010652D ?
000106400 ? 10652D264 ?
kgeasnmierr()+28 CALL kgerinv() 106527C18 ? 000000000 ?
105FFA068 ? 000000003 ?
FFFFFFFF7FFEF7F0 ?
000001430 ?
kjbcropen()+4892 CALL kgeasnmierr() 106527C18 ?
FFFFFFFF7C026270 ?
105FFA068 ? 000000003 ?
000000000 ? 0000C1A20 ?
kclscrs()+1200 CALL kjbcropen() 10652D000 ? 1065278D8 ?
000106400 ? 105FFA000 ?
000106400 ? 0000C1DA5 ?
kclgclk()+9936 CALL kclscrs() 000000000 ? 000004000 ?
000000000 ?
FFFFFFFF7FFEFBB0 ?
000000000 ? 000000000 ?
而在另外一个节点,则错误主要以ORA-600()为主:
Thu Sep 17 14:50:10 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb2_ora_18519.trc:
ORA-00600: internal error code, arguments: [kclcls_8], [], [], [], [], [], [], []
Thu Sep 17 14:50:46 2009
Trace dumping is performing id=[cdmp_20090917145046]
Thu Sep 17 14:52:21 2009
Trace dumping is performing id=[cdmp_20090917144741]
Thu Sep 17 14:53:03 2009
Trace dumping is performing id=[cdmp_20090917144823]
Thu Sep 17 14:53:46 2009
Trace dumping is performing id=[cdmp_20090917144906]
Thu Sep 17 14:54:28 2009
Trace dumping is performing id=[cdmp_20090917144948]
Thu Sep 17 14:55:09 2009
Trace dumping is performing id=[cdmp_20090917145029]
Thu Sep 17 15:03:57 2009
Trace dumping is performing id=[cdmp_20090917145917]
Thu Sep 17 15:04:00 2009
Trace dumping is performing id=[cdmp_20090917145920]
Thu Sep 17 15:04:04 2009
Trace dumping is performing id=[cdmp_20090917145924]
Thu Sep 17 15:04:59 2009
Trace dumping is performing id=[cdmp_20090917150019]
Thu Sep 17 15:05:01 2009
Trace dumping is performing id=[cdmp_20090917150021]
Thu Sep 17 15:05:54 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb2_ora_20097.trc:
ORA-00600: internal error code, arguments: [kclcls_8], [], [], [], [], [], [], []
Thu Sep 17 15:05:56 2009
Trace dumping is performing id=[cdmp_20090917150556]
Thu Sep 17 15:05:58 2009
Trace dumping is performing id=[cdmp_20090917150118]
.
.
.
Thu Sep 17 15:51:16 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb2_ora_25090.trc:
ORA-00600: 内部错误代码, 参数: [kclcls_8], [], [], [], [], [], [], []
查询metalink,发现bug描述Doc ID: 5165885.8。虽然在这个bug中,并没有说明是什么导致了这个问题的产生,但是环境和现象的描述都与当前完全一致。这个bug影响10.2.0.3,也就是当前的版本。虽然Oracle提供了单独的补丁集,但是似乎安装这个补丁后有可能导致其他的ORA-600错误的产生,甚至是系统CRASH。
虽然Oracle没有说明是什么导致了问题,但是根据执行的操作可以推断出大致是什么原因。由于在执行迁移表空间的过程中,只是删除了表空间,而没有删除用户,因此应用程序并没有断开连接。由于RAC的特殊性,导致两个节点之间在处理不正确的CURSOR出现了问题,最终造成了两个节点的状态不一致,产生了大量的ORA-600错误。
重启整个数据库后,这个错误不再出现。
来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/4227/viewspace-614937/,如需转载,请注明出处,否则将追究法律责任。
转载于:http://blog.itpub.net/4227/viewspace-614937/

1169

被折叠的 条评论
为什么被折叠?



