Sunday, July 5, 2009

Creating a REST web service with PL/SQL (and making pretty URLs for your Apex apps)





If you need to expose the data in your Oracle database to other systems in a standardized and technology-neutral way, web services are a natural choice. Broadly speaking, web services come in two flavors: SOAP and REST. Despite its name, the Simple Object Access Protocol (SOAP) can be complex and overkill for many common scenarios. Representational State Transfer (REST) is considered more lightweight and easy to use.

There is a good introduction to REST here: http://www.xfront.com/REST-Web-Services.html

A key point is that REST is not really a standard, but an architectural style. It is not limited to web services consumed by machines, either. Applications built using Ruby on Rails and the ASP.NET MVC framework typically have user-friendly URLs based on the REST principles.

For example, if you have been to StackOverflow.com, you will see URLs like the following, which are "clean" and friendly both to users and to search engines:



http://stackoverflow.com/users
http://stackoverflow.com/users/1/jeff-atwood
http://stackoverflow.com/questions
http://stackoverflow.com/questions/tagged/oracle
http://stackoverflow.com/questions/1078506/oracle-sql-developer-how-to-view-results-from-a-ref-cursor



So, REST is a good way of exposing resources on the web, and your Oracle database is full of resources (data), but how can you build a REST service using only PL/SQL?

The key to building a REST service in PL/SQL is in a documented, but little-used feature of the Embedded PL/SQL Gateway (and mod_plsql) called "Path Aliasing". I was not aware that this feature existed until I discovered it "by accident" while browsing the mod_plsql documentation:

http://download.oracle.com/docs/cd/A97335_02/apps.102/a90099/feature.htm#1007126

"If the PL/SQL Gateway encounters in an incoming URL the keyword entered in the Path Alias field, it invokes the procedure entered in the Path Alias Procedure field. (...) Applications that use path aliasing must implement the Path Alias Procedure. The procedure receives the rest of the URL (path_alias_URL) after the keyword, URL, as a single parameter, and is therefore responsible and also fully capable of dereferencing the object from the URL. Although there is no restriction on the name and location for this procedure, it can accept only a single parameter, p_path, with the datatype varchar2."

Sounds good, so let's try it out. First, we need to configure the Database Access Descriptor (DAD) to define a PL/SQL procedure which will handle our REST requests. Using the embedded gateway (DBMS_EPG), the attributes are called "path-alias" and "path-alias-procedure" (the corresponding DAD attributes for mod_plsql are "PlsqlPathAlias" and "PlsqlPathAliasProcedure").

I will be using the embedded gateway for this example. Assuming you have an existing DAD called "devtest", run the following as user SYS (or another user who has the privileges to modify the EPG configuration).



begin
dbms_epg.set_dad_attribute (dad_name => 'devtest', attr_name => 'path-alias', attr_value => 'rest-demo');
dbms_epg.set_dad_attribute (dad_name => 'devtest', attr_name => 'path-alias-procedure', attr_value => 'rest_handler.handle_request');

end;
/




Then we need to create the procedure itself. Run the following in the schema associated with the DAD:




create or replace package rest_handler
as

/*

Purpose: A simple example of RESTful web services with PL/SQL (see http://en.wikipedia.org/wiki/Representational_State_Transfer#RESTful_web_services)

Remarks: The DAD must be configured to use a path-alias and path-alias-procedure

Who Date Description
------ ---------- --------------------------------
MBR 05.07.2009 Created

*/

-- the main procedure that will handle all incoming requests
procedure handle_request (p_path in varchar2);

end rest_handler;
/





And then the package body (the example assumes that the EMP and DEPT demo tables exist in your schema; if not, then modify the code accordingly):



create or replace package body rest_handler
as

/*

Purpose: A simple example of RESTful web services with PL/SQL (see http://en.wikipedia.org/wiki/Representational_State_Transfer#RESTful_web_services)

Remarks: The DAD must be configured to use a path-alias and path-alias-procedure

Who Date Description
------ ---------- --------------------------------
MBR 05.07.2009 Created

*/


g_request_method_get constant varchar2(10) := 'GET';
g_request_method_post constant varchar2(10) := 'POST';
g_request_method_put constant varchar2(10) := 'PUT';
g_request_method_delete constant varchar2(10) := 'DELETE';

g_resource_type_employees constant varchar2(255) := 'employees';
g_resource_type_departments constant varchar2(255) := 'departments';


procedure handle_emp (p_request_method in varchar2,
p_id in number)
as
begin

/*

Purpose: Specific handler for Employees

Remarks:

Who Date Description
------ ---------- --------------------------------
MBR 05.07.2009 Created

*/

if (p_id is not null) then

if p_request_method = g_request_method_delete then

delete
from emp
where empno = p_id;

elsif p_request_method = g_request_method_get then

for l_rec in (select * from emp where empno = p_id) loop
htp.p(l_rec.empno || ';' || l_rec.ename || ';' || l_rec.sal);
end loop;

end if;

else

if p_request_method = g_request_method_get then

for l_rec in (select * from emp order by empno) loop
htp.p(l_rec.empno || ';' || l_rec.ename || ';' || l_rec.sal || '
');
end loop;

end if;

end if;

end handle_emp;


procedure handle_dept (p_request_method in varchar2,
p_id in number)
as
begin

/*

Purpose: Specific handler for Departments

Remarks:

Who Date Description
------ ---------- --------------------------------
MBR 05.07.2009 Created

*/

if (p_id is not null) then

if p_request_method = g_request_method_delete then

delete
from dept
where deptno = p_id;

elsif p_request_method = g_request_method_get then

for l_rec in (select * from dept where deptno = p_id) loop
htp.p(l_rec.deptno || ';' || l_rec.dname || ';' || l_rec.loc);
end loop;

end if;

else

if p_request_method = g_request_method_get then

for l_rec in (select * from dept order by deptno) loop
htp.p(l_rec.deptno || ';' || l_rec.dname || ';' || l_rec.loc || '
');
end loop;

end if;

end if;

end handle_dept;


procedure handle_request (p_path in varchar2)
as
l_request_method constant varchar2(10) := owa_util.get_cgi_env('REQUEST_METHOD');
l_path_elements apex_application_global.vc_arr2;
l_resource varchar2(2000);
l_id number;

begin

/*

Purpose: The main procedure that will handle all incoming requests

Remarks: Parses the incoming path and calls a specific handler for each resource type

Who Date Description
------ ---------- --------------------------------
MBR 05.07.2009 Created

*/

-- note that an extra delimiter is added to the path, in case the user leaves out the trailing slash
l_path_elements := apex_util.string_to_table (p_path || '/', '/');

begin
l_resource := l_path_elements(1);
l_id := l_path_elements(2);
exception
when value_error or no_data_found then
l_resource := null;
l_id := null;
end;

case lower(l_resource)
when g_resource_type_employees then
handle_emp (l_request_method, l_id);
when g_resource_type_departments then
handle_dept (l_request_method, l_id);
when 'apex-employees' then
-- we can also use this REST handler to make pretty, search-engine-friendly URLs for Apex applications without having to use Apache mod_rewrite
apex_application.g_flow_id := 104;
owa_util.redirect_url('http://127.0.0.1:8080/apex/f?p=104:2:' || apex_custom_auth.get_session_id_from_cookie || '::::P2_EMPNO:' || l_id, true);
else
owa_util.status_line(404, 'Resource type not recognized.', true);
end case;

end handle_request;


end rest_handler;
/





Now try the following URLs in your browser, and you should be able to see the familiar EMP and DEPT data, in all their RESTful glory! Note that for simplicity, the example code produces simple semicolon-separated values, but depending on your requirements and who will consume the service (machine or human), you will probably want to use XML or HTML as the output format.



http://127.0.0.1:8080/devtest/rest-demo/departments
http://127.0.0.1:8080/devtest/rest-demo/departments/10
http://127.0.0.1:8080/devtest/rest-demo/employees
http://127.0.0.1:8080/devtest/rest-demo/employees/7839






Bonus: Making the URLs in your Apex applications pretty


The URLs generated by Apex are not very friendly to users nor to search engines. You can use Apache with mod_rewrite to set up mapping between REST-style URLs to your Apex pages, but the example code above also shows how this can be accomplished using pure PL/SQL.

The example assumes that you have an Apex application with Application ID = 104, and that you have made a Form on the EMP table on Page 2 of the application.

Try the following link



http://127.0.0.1:8080/devtest/rest-demo/apex-employees/7839




and you should be redirected to the Apex application (and if you were already logged into the Apex application, you don't have to login again as it will reuse your existing session).


REST for inserts and updates

The four HTTP methods are GET, PUT, POST and DELETE. To create a REST web service that can update data as well as query it, we need to inspect the CGI environment variable REQUEST_METHOD and process the request accordingly (see the Wikipedia article for details). The example code implements the GET and DELETE methods for Employees and Departments.

However, I don't see how we can implement POST or PUT processing using the PathAlias technique. The problem is that the webserver/gateway only sends the URL to the PathAliasProcedure. Any data that is POSTed to the URL is simply discarded by the gateway. Ideally, the names and values of the request should be sent to the PathAliasProcedure in name/value arrays (just like the gateway does when using flexible parameter passing). If anyone from Oracle is reading this, it can be considered an enhancement request for the next version of mod_plsql and the embedded gateway!

Even with this limitation, the ability to expose (read-only) data from the database as RESTful web services using just PL/SQL is pretty cool, isn't it? :-)

Tuesday, June 16, 2009

The Fat Database (or Thick Database) Approach

I'm a big believer in the so-called "Fat Database" paradigm for data-centric business applications.



I think I first heard the term "Thick Database" in a presentation by Dr. Paul Dorsey at the ODTUG conference in 2007.

I prefer the slightly more hip term "Fat Database", and offer my own definition of the term:


"Building applications using the Fat Database approach means leveraging the full potential of the database engine and its features, rather than treating the database as a bit bucket. If a problem can be solved using the database, it should be solved using the database, rather than in a programming language outside the database."


In other words, the exact opposite of the current trend, which is to avoid any database feature except basic tables. The enterprise architecture astronauts would rather reinvent the wheel over and over again, using the latest silver bullet in the endless stream of "new and improved" languages and frameworks that appear (and disappear) every few years.

Benefits of the Fat Database approach include reduced cost and complexity, increased performance, and a degree of immunity against the need to constantly rewrite code in a rapidly changing technology landscape.

Here is a collection of links to presentations and papers related to the Fat Database approach:

Dr. Paul Dorsey, co-author of seven Oracle Press books on Designer, Database Design, Developer, and JDeveloper


Toon Koppelaars, co-author of Applied Mathematics for Database Professionals


Others


I will return to this subject in future postings on this blog.

Wednesday, June 10, 2009

The Oracle Database as Development Platform

There is really a lot of amazing stuff that you can do using just the Oracle database and PL/SQL these days. I've made this diagram to illustrate the Oracle database developer's toolbox (click image to enlarge):





Note that this is all native functionality in the database itself, it does not include anything from the Oracle Fusion (Java) technology stack (except JDeveloper, which is free and can be used for general database and web development, not just Java).

Sunday, December 21, 2008

Adventures with Apex, part three: Setting up Apex with Tomcat and the DBPrism plugin

It is common knowledge that you can run Apex using either the Apache-based Oracle HTTP Server (with mod_plsql) or the Embedded PL/SQL Gateway (on 10g XE and 11g), but did you also know that there is a third alternative?

The DBPrism project is a free, open-source plugin for the Tomcat web server that, among other things, can function as a replacement for mod_plsql.

Side Note 1: Interestingly, Oracle itself is also developing the "Apex Listener", what I assume is based on similar [Java] technology. So by Apex 4.0 we will have even more deployment choices.

Side Note 2: My wish is that someone familiar with Microsoft's IIS should write a plugin to mimic mod_plsql on that platform as well, to make it easier to start using Apex in companies that have standardized on Microsoft technology (sneaking it in the back door, so to speak :-).

Anyway, here are instructions for using Apex with Tomcat and DBPrism:

ON THE DATABASE SERVER

1. Download the latest version of Application Express from oracle.com

- http://www.oracle.com/technology/products/database/application_express/download.html

2. Install Apex into the database according to the instructions

- do not worry about configuring either the Embedded PL/SQL Gateway (DBMS_EPG) or Oracle HTTP Server,
- but make sure to unlock the apex_public_user schema and make a note of the password given to apex_public_user (you will need it to configure the JDBC connection on the web server)


ON THE WEB SERVER

Note: This could be the same physical machine as the database, if desired.

1. Download Java Runtime Engine (JRE) from sun.com

- install by running setup program
- create an environment variable called JRE_HOME and set it to the location where JRE was installed

2. Download Tomcat from tomcat.apache.org

- unzip to a folder, for example c:\program files\apache-tomcat-xxx (where xxx is version number)
- follow instructions in RUNNING.txt to start Tomcat and verify installation (http://localhost:8080)

3. Download the Oracle JDBC Thin driver from Oracle.com

- http://www.oracle.com/technology/software/tech/java/sqlj_jdbc/htdocs/jdbc_111060.html
- get the "ojdbc5.jar" file

4. Download DBPrism from http://sourceforge.net/projects/dbprism/

- choose the "dbprism" package (not the "cms" package)
- unzip and get the "dpls.war" file
- put the dpls.war file into the "webapps" folder in Tomcat
- shutdown and restart Tomcat
- this will create a "dpls" folder under "webapps"
- put the "ojdbc5.jar" file from step 3 in the dpls\WEB-INF\lib folder
- edit the prism.xconf file
- under "variables", change the value of the "demo.db" to a valid jdbc connectionstring to the database
- under under category named "DAD_apex", change the "dbpassword" to the correct password of the "apex_public_user" (note that this was set during/after the installation of Apex)

- shutdown and restart Tomcat again

5. Configure Apex images/javascript for Tomcat

- Create a folder called "i" under "webapps" in Tomcat
- shutdown and restart Tomcat
- Extract the files in the "images" folder in the Apex installation zip file and copy it to the "i" folder

6. Verify that everything works

- Go to http://localhost:8080/dpls/apex (this should display the login page of the Apex environment)


Pretty cool, isn't it? :-)

Saturday, April 12, 2008

Adventures with Apex, part two: Setting up the stand-alone Oracle HTTP Server as a Windows service

The latest stand-alone version of Oracle HTTP Server (OHS) for Windows is available on OTN as the file "iAS_101330_Apache2_Modplsql2_win32.zip".

I ran the default setup on a Windows 2003 server (a separate machine from the database server) and encountered no errors during installation. The setup process created a few Start Menu items to start and stop the Apache process (through the Oracle Process Manager, OPMN, and the "opmnctl startall" command, which in turn starts the HTTP Server).

After configuring OHS and mod_plsql on the webserver to connect to Apex on the database server (as per the Apex installation docs), everything seemed to work fine. However, when I logged out of the server, the Apex application (and OHS) stopped responding.

Investigating, I found that the Apache process (apache.exe) is started and runs under the current logged-in user when using the Start Menu item to start it. So once I logged out of the server, the Apache process shuts down...!

As far as I could tell, there should have been a Windows service called OracleProcessManager that runs the OPMN process in the background, without requiring a logged-on user. However, there is no such service created after installation.

I tried creating the service manually myself using the "sc create" command. The service was created successfully but gave an error message ["...did not respond in a timely fashion..."] when trying to start it.)

Fortunately, I had a quick look in the Knowledgebase on Metalink and found the solution there. According to Note:459474.1, entitled "Howto Create a Windows Services Entry for Starting Oracle Application Server Processes", this behavior is by design (!)...:

Most Oracle Application Server installations automatically create a Windows Services entry to allow the OracleAS processes to start up on server startup. By design, certain OracleAS releases such as the SOA Suite 10.1.3.1 do not create the Windows Services entry. In such installations, the user starts OracleAS processes from the Windows Start Menu. When the user logs out of the Windows session, the OracleAS processes will terminate as well. Resulting in an unexpected Application Server shutdown.

This note provides the steps to create a Windows Services entry for starting up OracleAS processes upon reboot/startup of the server. Processes started via Windows Services entries are not affect by a user logging out of the Windows session.

I'll leave the question as to why on earth the setup program doesn't create the necessary Windows services by default, for what is clearly a server-type product?!? up in the air for now.

Instead, I will provide the necessary details here for those who do not have access to Metalink:

Turns out I was right about having to create a Windows service manually using the "sc" command, but there is also a few registry entries that need to be present for it to work.

First, find the "key" value from the file \opmn\bin\oracle.key on the server. In my case, the value of this key was:

SOFTWARE\ORACLE\KEY_oracleas1

Run regedit and add the following two entries (in my case, the first entry already existed):


HKLM / Software / ORACLE / Key_oracleas1 / ORACLE_HOME_KEY = SOFTWARE\ORACLE\KEY_oracleas1


and


HKLM / Software / ORACLE / oracleas1 / ORACLE_OPMN_SERVICE = OracleHTTPServerProcessManager


Then run the following command to create the service:


sc create OracleHTTPServerProcessManager binPath= "c:\OraHome_1\opmn\bin\opmn.exe -s" DisplayName= "OracleHTTPServerProcessManager" start= "auto"


Note that there must be an empty space between the equal sign and the value in the command above. Apparently some kind of quirk with the sc command. By the way, the sc command is part of the Windows server resource kit (but it was already present on my Windows 2003 server).

After creating the service, go to Services and verify that the service starts up. Also verify using Task Manager that apache.exe runs under the SYSTEM user, rather than the logged-in user.

Adventures with Apex, part one: Boosting performance when using DBMS_EPG

We are currently developing a web application using Apex 3.1. There is a handful of developers on the team and we all work in the Apex Application Builder against a shared Oracle 11g database server, which has the Embedded PL/SQL Gateway (DBMS_EPG), rather than a full Apache (Oracle HTTP Server) setup, as per the default database install.

It did not take long before we noticed a significant variation in the time it took to serve Apex pages from the database. Most of the time, pages were served quickly enough, but at uneven intervals pages would just take forever to load. It seemed as if the underlying database or webserver/XDB session was timing out.

After some head-scratching, googling, and experimentation, the following two configuration changes fixed the problem. Page performance is now consistently excellent, and we do not experience timeouts anymore.

  1. Increase the XDB session-timeout parameter (I changed it from the default of 6000 [1 minute] to a new value of 30000 [5 minutes].
  2. Increase the shared_servers database parameter (I changed it from the default of 1 to a new value of 5).

Note: I changed both these settings at the same time, so I did not measure the effect of each individually. I suspect the shared_servers database setting is the most important of the two.

Detailed instructions follow:

Changing the session-timeout parameter in XDB

Connect as SYSDBA and run the following:


declare
l_newconfig xmltype;
begin
select updatexml (dbms_xdb.cfg_get(),
'/xdbconfig/sysconfig/protocolconfig/httpconfig/session-timeout/text()', 30000)
into l_newconfig
from dual;

dbms_xdb.cfg_update (l_newconfig);

commit;
end;
/


Note: The database must be restarted for the change to take effect.

Changing the shared_servers database parameter

Connect as SYSDBA and run the following:


SQL> alter system set shared_servers = 5 scope=both;

System altered.

SQL> alter system register;

System altered.


This will create more background sessions to handle the Apex page requests served through XDB.

Tuesday, March 25, 2008

Implementing table inheritance in the database

Over at sqlteam.com, there is an interesting article about "Implementing table inheritance in SQL Server". But the concept is just as applicable to Oracle, of course:

http://www.sqlteam.com/article/implementing-table-inheritance-in-sql-server