EM Express 12c Failing With Error: QERVW*

It’s been a while that I am using Oracle database’s latest release 12c. It’s a very different release from the past versions and IMO, if you are going to delay in learning it thinking that you would do when your company/client/<put-whatever-reason-here> would make you to do so, it would be too late. There are a LOT of new features (~500) with the biggest change in the architecture which came due to the concept of Multitenant databases. So there is a lot that you are required to catch up and delaying it would just make things more and more tough.

This post is not about the new features of Oracle database 12c. I have written few posts already which you would find under the tag 12c. In this post I shall talk about the new Enterprise Manager of database 12c which is now called EM Express. This is a very different enterprise manager from the past versions of 10g, 11g as it is completely based on an awesome technology Application Express. Since APEX runs inside the database, this means that the new EM Express is fast and takes less amount of resources. You may not find it that much feature rich as it’s predecessor was but if you are not willing to use the full blown 12c Cloud Control, this is the free version that you can use now.

There is just a small issue, though!

The database version 12.1.0.1’s EM express crashes with this sort of error messages recorded in the alert log file.

Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x5] [PC:0xB71F161, qervwRowProcedure()+113] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_s000_2751.trc  (incident=12067):
ORA-07445: exception encountered: core dump [qervwRowProcedure()+113] [SIGSEGV] [ADDR:0x5] [PC:0xB71F161] [Address not mapped to object] []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_12067/orcl_s000_2751_i12067.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Sun Mar 23 21:06:27 2014
Dumping diagnostic data in directory=[cdmp_20140323210627], requested by (instance=1, osid=2751 (S000)), summary=[incident=12067].
Sun Mar 23 21:06:56 2014

Well, that doesn’t sound good! Let’s check the mentioned incident file and it’s contents.

Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 – 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing opt
ions
ORACLE_HOME = /u01/app/oracle/product/12.1.0/dbhome_1
System name:    Linux
Node name:      arista.localdomain
Release:        3.8.13-16.2.1.el6uek.x86_64
Version:        #1 SMP Thu Nov 7 17:01:44 PST 2013
Machine:        x86_64
Instance name: orcl
Redo thread mounted by this instance: 1
Oracle process number: 7
Unix process pid: 2751, image: oracle@arista.localdomain (S000)

*** 2014-03-23 21:06:23.277
*** SESSION ID:(19.35) 2014-03-23 21:06:23.277
*** CLIENT ID:() 2014-03-23 21:06:23.277
*** SERVICE NAME:(SYS$USERS) 2014-03-23 21:06:23.277
*** MODULE NAME:() 2014-03-23 21:06:23.277
*** ACTION NAME:() 2014-03-23 21:06:23.277
*** CONTAINER ID:(1) 2014-03-23 21:06:23.277

Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x5] [PC:0xB71F161
, qervwRowProcedure()+113] [flags: 0x0, count: 1]
Incident 12067 created, dump file: /u01/app/oracle/diag/rdbms/orcl/orcl/incident
/incdir_12067/orcl_s000_2751_i12067.trc
ORA-07445: exception encountered: core dump [qervwRowProcedure()+113] [SIGSEGV]
[ADDR:0x5] [PC:0xB71F161] [Address not mapped to object] []

ssexhd: crashing the process…
Shadow_Core_Dump = PARTIAL
ksdbgcra: writing core file to directory ‘/u01/app/oracle/product/12.1.0/dbhome_
1/dbs’

And in the meanwhile, EM Express already reported that “may be” that database is down so the EM is not functional( I didn’t capture the screenshot of it, sorry).

A quick search on https://support.oracle.com got me MOS#1604062.1(Troubleshooting EM Express) . Going through it I came to know about Bug 16527374 : [12100-LIN64-130318]CDB:ORA-07445:EM EXPRESS HOME GOT IO ERROR(unpublished) and to resolve it, the recommendation was to use the patch#16527374. Cool, let’s try it but before that, let’s check our database with OPatch.

[oracle@arista OPatch]$ ./opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.0
Copyright (c) 2012, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.1.0.1.0
OUI version       : 12.1.0.1.0
Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/o
patch2014-03-23_21-21-42PM_1.log

Lsinventory Output file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgto
ollogs/opatch/lsinv/lsinventory2014-03-23_21-21-42PM.txt

——————————————————————————–
Installed Top-level Products (1):

Oracle Database 12c                                                  12.1.0.1.0
There are 1 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

——————————————————————————–

OPatch succeeded.

That was expected. So after unzipping the patch file, I got started using the following command,

[oracle@arista 16527374]$ opatch apply -connectString orcl:sys:oracle

opatch error
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:

Following executables are active :
/u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle

 

Let’s find the process using this executable using the FUSER command and kill it,

[oracle@arista 16527374]$ fuser
/u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle
/u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle:  3620e
[oracle@arista 16527374]$ kill -9 3620
[oracle@arista 16527374]$ fuser
/u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle
[oracle@arista 16527374]$

 

Let’s start again,

[oracle@arista 16527374]$ opatch apply -connectString orcl:sys:oracle
Oracle Interim Patch Installer version 12.1.0.1.0
Copyright (c) 2012, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.1.0.1.0
OUI version       : 12.1.0.1.0
Log file location :
/u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/16527374_Mar_23_2014_21_39_06/apply2014-03-23
_21-39-06PM_1.log

Applying interim patch ‘16527374’ to OH
‘/u01/app/oracle/product/12.1.0/dbhome_1’
Verifying environment and performing prerequisite checks…
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local
system.
(Oracle Home = ‘/u01/app/oracle/product/12.1.0/dbhome_1’)

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files…

Patching component oracle.rdbms, 12.1.0.1.0…

Verifying the update…
Patch 16527374 successfully applied
Log file location:
/u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/16527374_Mar_23_2014_21_39_06/apply2014-03-23
_21-39-06PM_1.log

OPatch succeeded.

 

Let’s check the status of the applied patches in our database again,

 

[oracle@arista 16527374]$ opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.0
Copyright (c) 2012, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.1.0.1.0
OUI version       : 12.1.0.1.0
Log file location :
/u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2014-03-23_21-41-00PM_1.log

Lsinventory Output file location :
/u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2014-03-23_21-41-00PM.txt

——————————————————————————–
Installed Top-level Products (1):

Oracle Database 12c
12.1.0.1.0
There are 1 products installed in this Oracle Home.

Interim patches (1) :

Patch  16527374     : applied on Sun Mar 23 21:40:19 IST 2014
Unique Patch ID:  16551736
   Created on 12 Jul 2013, 13:23:36 hrs PST8PDT
   Bugs fixed:
     16527374

——————————————————————————–

OPatch succeeded.

 

After going for a reboot of the whole machine followed by the startup of the database and EM express, so far, EM has not crashed and I shall hope it would remain the same way.