Database@Home - Data Driven : Loading, Indexing, and Searching with Text and JSON

Copyright © 2020, Oracle and/or its affiliates
Loading, Indexing and
Searching for Recommendations
with Text and JSON
Roger Ford
Principal Product Manager
Text & JSON
Starts at 11am ET

2 Copyright © 2020, Oracle and/or its affiliates | Confidential: Internal/Restricted/Highly Restricted [Date]
Roger Ford
Principal Product Manager
Roger Ford is a principle product manager
at Oracle Corporation and has been with the
company since version 5, in 1987. Roger
manages a portfolio of products including
Oracle Text, JSON in the database and the
Database Scheduler. Currently he is
spending most of his time planning the
release of a new Autonomous JSON
product.
About your presenter:
When not at work, Roger is usually found
working on, or racing his Caterham sportscar.
He helps manage the Caterham Graduates
Racing Club, one of the biggest car racing
clubs in the world.

The case for Full Text Indexing
• "Keyword" searching is a familiar concept from internet search engines and web stores
• Yet many companies have massive amounts of valuable information in document formats which are
currently not searchable
• Office documents: MS Word, PDF, Powerpoint
• Email archives
• "How to" documents and training materials
• The database can index these documents and provide full-text content searching
• No need for a separate search engine
• Combine relational and content searching via SQL
• Full text search is available in all versions and editions of the database, and is not a chargable option
Copyright © 2019 Oracle and/or its affiliates.

The case for JSON development
• Traditional SQL development requires pre-defined schema
• Schema changes can be major task
• Limited number of columns available (1000 in 19c)
• JSON development increasingly popular
• Schema-less, agile development style
• Schema evolution is far less costly
• Non-relational data model : hierarchical document/object model
• Data access via many language API
• Key/Value document collection API using QBE
• Textual values in JSON can be full-text indexed

Full Text Indexes
• For text in VARCHAR2 and CLOB columns, eg. a CLOB column called mycol:
create index myind on mytab(mycol) indextype is ctxsys.context;
• Query it with
select * from mytab where contains (mycol, 'hello world' ) > 0;
• For a JSON column jtext, with textual attribute 'description'
create search index myind on mytab (jtext) for json;
• Query it with
select * from mytab where json_textcontains(jtext, '$.description', 'hello world');
Copyright © 2019 Oracle and/or its affiliates.6

Machine learning with a full-text index
• Oracle Text integrates tightly with Oracle Machine Learning (formerly Advanced Analytics)
• Clustering
• Group documents based on shared attributes (words, phrases, themes, etc)
• Classification
• Train the system to recognize particular topic areas
• Sentiment Analysis
• Identify documents which are positive, negative or neutral in tone
• Optionally can specify a topic for sentiment analysis
• eg. Are these restaurant reviews positive about the service (as compared to, say, the food
quality)

Oracle Database with SQL/JSON Support
– bridges the gap between SQL and NoSQL
Database
• JSON Documents are storable, queryable, searchable, updatable,
generatable in Oracle Database
• as varchar, clob, blob Datatype since 12c
• as JSON DataType with efficient binary format in next major release
• Combine schema flexibility of JSON with strengths of the relational model
in one Database System
• Support SQL/JSON Standard – Extension of SQL to query JSON
• Support Partial Update of JSON
• ACID Transaction Model applied to JSON – No Data Loss
Confidential – Oracle Internal/Restricted/Highly Restricted 8

Public9
Oracle as a Document Store
{ "doctype" : "JSON", "count" : 100 }
Oracle Database
Document Collections
Text and JSON Documents
Stored and Managed
Using Oracle DatabaseJSON Data

Public
1
0
All the power of SQL. All the flexibility of schemaless development.
Oracle Database - JSON document store
JSON
Applications
developed
using SODA
APIs
JSON Documents
Stored and Managed
Using Oracle Database
SQL based reporting
and analytical operations
on JSON Documents
Oracle Database 20c
SQL

JSON Query – Oracle Simplified Syntax
Oracle supports a non-standard simplified syntax
SQL> select j.PO_DOCUMENT
2 from J_PURCHASEORDER j
3 where j.PO_DOCUMENT.PONumber = 1600
4 /
SQL> select j.*
2 from CUSTOMER c,
3 JSON_TABLE (c.jcol.orders.lineitems[*]
4 COLUMNS (lineid, quantity, prodid, upc, comments)) j
5 /
SQL> select JSON {cid, firstname, lastname, street, country, zip}
2 from CUSTOMERS
Field Access
Collection
unnesting
JSON Generation

JSON Query – SQL/JSON
SQL/JSON standard
• Joint standard with IBM
• SQL extended with new operators for json e.g. JSON_VALUE
• All operators use JSON Path language for intra document navigation
Confidential – Oracle Internal/Restricted/Highly Restricted
SQL> select JSON_VALUE(PO_DOCUMENT, ‘$.LineItems[0].Part.UnitPrice’
2 returning NUMBER(5,3))
3 from J_PURCHASEORDER p
4 where JSON_VALUE(PO_DOCUMENT, ‘$.PONumber’
5 returning NUMBER(10)) = 1600

JSON Generation - Embedding arrays in documents
Oracle Confidential – Internal/Restricted/Highly Restricted 13
select JSON_OBJECT(
'departmentId' is d.DEPARTMENT_ID,
'name' is d. DEPARTMENT_NAME,
'employees' is (
select JSON_ARRAYAGG(
JSON_OBJECT(
'employeeId' is EMPLOYEE_ID,
'firstName' is FIRST_NAME,
'lastName' is LAST_NAME,
'emailAddress' is EMAIL
)
)
from HR.EMPLOYEES e
where e.DEPARTMENT_ID = d.DEPARTMENT_ID
)
) DEPT_WITH_EMPLOYEES
from HR.DEPARTMENTS d
where DEPARTMENT_NAME = 'Executive'
/
DEPT_WITH_EMPLOYEES
---------------------------------------------------------
-----------------------
{
"departmentId": 90,
"name": "Executive",
"employees": [
{
"employeeId": 100,
"firstName": "Steven",
"lastName": "King",
"emailAddress": "SKING"
}, {
"employeeId": 101,
"firstName": "Neena",
"lastName": "Kochhar",
"emailAddress": "NKOCHHAR"
}, {
"employeeId": 102,
"firstName": "Lex",
"lastName": "De Haan",
"emailAddress": "LDEHAAN"
}
]
}
select JSON_OBJECT(*)
from EMP;

JSON Indexes
• Functional Index
• Index specific fields within a JSON
document
• Search Index
• Universal Index for all fields
• Supports value, range and full
text
• GeoSpatial Index
• JSON_VALUE returns GeoJSON
as SDO_GEOMETRY object
Oracle Confidential – Internal/Restricted/Highly Restricted 14
CREATE SEARCH INDEX po_search_index
ON j_purchaseorder (po_document)
FOR JSON
CREATE UNIQUE INDEX po_ponum_index
ON j_purchaseorder po
(po.po_document.PONumber);

Discover metadata for JSON
• Generate JSON Schema or
• Generate relational schema
Derived relational views
• Declarative procedures to construct
a relational view over a JSON
fragment
Derived virtual columns
• Generated for singleton JSON keys
• Automatically generated for new
keys
Can be used with external data
JSON Dataguide - Schema Discovery
• SQL> select
dbms_json.get_index_DataGuide("REVIEWS',
• 2 'JTEXT',
• 3 dbms_json.FORMAT_HIERARCHICAL,
• 4 dbms_json.PRETTY)
• 5 from dual
{
"type" : "object",
"o:length" : 8192,
"properties" :
{
"text" :
{
"type" : "string",
"o:length" : 1024,
"o:preferred_column_name" : "JTEXT$text"
},
"stars" :
{
"type" : "number",
"o:length" : 4,
"o:preferred_column_name" : "JTEXT$stars"
},
}
}

MongoDB
• data pipelines to move out data for
OLTP, analytics, search, …
• Multiple databases to maintain, patch
• Multiple datasets to backup, administer
• Integration soaks up 30% of project costs
Oracle Database
• Converged Architecture
• All your data managed together
Oracle Database – Converged Data
Relational Full Text
OracleDBmongoDB ElasticSearch …….
…
Oracle
Database
20c
Relation
al
Full Text ….

Microservices with Multitenant
• Multitenant allows creating PDB
for each microservice
• Each PDB offers isolation and
can be scaled independently
• But still preserve unified
administration at the CDB level
Multimodel AND Polyglot
• Each PDB can be used as a
multimodel or specialized store
Oracle Database - Converged Data
Oracle Database
20c
RelationalMultimodel
Micro-
service
Micro-
service
Micro-
service

Workshop Overview
• Loading, indexing and searching for
recommendations in Text and JSON
• On GitHub: Oracle Learning Library
https://oracle.github.io/learning-library/data-
management-library/database/json/freetier/

Workshop Overview
• Aim: Create a simple search microservice
• Find restaurants reviews in the local area that mention particular terms
• eg. "Show me all the reviews for restaurants with zip code "9160%"
which mention "great sushi"
• Workshop will work with the "Yelp" dataset and develop a REST-based
microservice with minimal coding
• Everything will be done using on-line cloud services – no client tools, no
IDE needed

Overview : YELP dataset
• Yelp is a Google-owned website for US business reviews
(shops, restaurants, gyms, etc.)
• Dataset is publicly available for educational, research and personal use
• JSON format
• > 8 million reviews
• > 200,000 businesses
• ~ 2 million users

Workshop Steps
• Upload JSON files to object storage
• Create tables with JSON columns in database
• Copy JSON data from object storage to database tables
• Create indexes on JSON data
• Run queries against JSON data
• Create simple REST API on queries via Oracle Application Express (APEX)

Lab 2: Upload to Oracle Object Storage
• Could upload directly from developer's PC to database
• Requires Oracle Instant Client on PC and download of wallet
• We're doing everything on the cloud so first we need to upload our files to the "cloud file system" –
Oracle Object Storage.
1. Download the Yelp dataset
2. Create object storage 'bucket'
3. Drag-and-drop our files to the bucket

Lab 3: Loading Oracle Database 19c from Object Storage
• We will provision an instance of Autonomous Transaction Processing – an Autonomous Database
• In the ATP database we create simple tables with a JSON column in each
• We then load each table from object storage.
• So the steps are:
1. Provision an Autonomous Transaction Processing Database
2. Connect to SQL Developer Web and create a new user
3. Log in to SDW as the new user and create our tables
eg. create table businesses ( jtext clob constraint busjson check (jtext is json) );
4. Use DBMS_CLOUD.COPY_DATA to load the tables from object storage

Lab 4: Creating indexes and basic queries
• Indexes for JSON take two forms
• FUNCTIONAL indexes to index specific values
• SEARCH indexes to index all values, provide full-text search and optional dataguide
• Step 2: Create functional indexes
• Used when joining tables
• Step 2: Create SEARCH index
• Queries can be run from SQL or from SODA – Simple Oracle Document Architecture
• We won't cover SODA here
• Step 3: Queries
• Create various queries of increasing complexity until we've satisfied our aim:
• "Find all businesses in ZIP codes 8911% which mention "sushi"

Full Text Queries - 1
• Simple query on one table:
select r.jtext.user_id, r.jtext.text
from reviews r
where json_textcontains(jtext, '$.text', 'great sushi')

• Join between two tables:
select u.jtext.name, r.jtext.text
from reviews r, users u
where json_textcontains(r.jtext, '$.text', 'sushi')
and u.jtext.user_id = r.jtext.user_id;

• Full query to join three tables, with column aliases
select u.jtext.name username,
b.jtext.name businessname,
r.jtext.stars rating,
b.jtext.postal_code zip,
r.jtext.text review text
from reviews r, users u, businesses b
where json_textcontains(r.jtext, '$.text', 'sushi')
and u.jtext.user_id = r.jtext.user_id
and b.jtext.business_id = r.jtext.business_id
and b.jtext.postal_code like '8911%';

Lab 5: Creating a RESTful interface
• The final step in our microservices project is to add a REST front-end
• Oracle Application Express (APEX) makes this almost trivially simple
• Step 1: Create an APEX workspace
• Step 2: Create a RESTful module, template and handler
• We will demonstrate the simple "query collection" interface and the PL/SQL procedure interface

Coming up at 12pm ET…
Maps and spatial
analyses: How to use
Them
with
Jayant Sharma &
Nick Salem
Breaktime!
Please complete the feedback
form for the previous session
Join us on the Database@Home
slack channel for more
conversations and to answer
your questions on the database
and labs
https://bit.ly/dbhome-slack

Database@Home - Data Driven : Loading, Indexing, and Searching with Text and JSON

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Database@Home - Data Driven : Loading, Indexing, and Searching with Text and JSON

Similar to Database@Home - Data Driven : Loading, Indexing, and Searching with Text and JSON (20)

Recently uploaded

Recently uploaded (20)

Database@Home - Data Driven : Loading, Indexing, and Searching with Text and JSON