OpenNTF Webinar Series: DQL with John Curtis September 2020

1. OPENNTF WEBINARS September Webinar - What's New with Domino Query Language with John Curtis from HCL

2. AGENDA • Welcome – Howard Greenberg • John Curtis • Questions – Graham Acres

3. ASKING QUESTIONS • First Question – Will this be recorded? • Yes, view on YouTube!!! • https://www.youtube.com/user/OpenNTF • Use the Questions Pane in GoToWebinar • We will get to your questions at the end of the webinar • The speakers will respond to your questions verbally • (not in the Questions pane) • Please keep all questions related to the topics that our speakers are discussing!!! • Unrelated Question => post at: • http://openntf.slack.com/

4. THANKS TO THE OPENNTF SPONSORS • HCL made a significant contribution to help our organization • Funds these webinars! • Contests like Hackathons • Running the organization • Prominic donates all IT related services • Cloud Hosting for OpenNTF • Infrastructure management for HCL Domino and Atlassian Servers • System Administration for day-to-day operation

5. THIS IS OUR COMMUNITY • Join us and get involved! • We are all volunteers • No effort is too small • If your idea is bigger than you can do on your own, we can connect you to a team to work on it • Test or help or modify an existing project • Write guides or documentation • Add reviews on projects / stars on Snippets

6. NEXT WEBINAR • October 22, 2020 • Michael Smith – A Flexible View Control for XPages • Signup at https://openntf.org/webinars

7. UPCOMING EVENTS • Let’s Connect Virtual Event – September 29-30 • https://letsconnect.world/agenda/ • Collabsphere – October 27-29 • https://collabsphere.org/ug/cs2020.nsf/index.html • DNUG – monthly online events • https://dnug.de/en/dnug47online-2/

8. 8 | Copyright © 2019 HCL Technologies Limited | www.hcltech.com8 | Technology for the Next Decade, Today Copyright © 2019 HCL Technologies Limited | www.hcltechsw.com What's New with Domino Query Language (and Domino Appdev) John Curtis Software Architect, Project Lead, Domino Development HCL Digital Solutions @john_d_curtis_ http://www.jdcurtis.blog john_curtis@pnp-hcl.com

9. Copyright © 2020 HCL Technologies Limited | www.hcltechsw.com DQL Evolution October 2018 December 2018 1H-2021 March 2020 • Consolidation of Domino search and results processing • Constant improvement in • Performance • Capability • Multiple venues and programming models December 2019 DQL V1 FT Support Inboard catalog FT Support completed Statistics Performance Results processing (sort, federation, joins) Formula Language inclusion

10. v11.0.1 DQL Features Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • Complete full text index utilization – enable numeric and date terms to take advantage of FT indexes ▪ 5-200X improvement in speed ▪ DQL plan and costing executes FT terms first, then views, then NSF scans ▪ Seamless – existing DQL syntax will work automatically on terms and across Boolean operators ▪ Caveats/limitations: ➢ Single-occurrence fields only (Domino does NOT FT index multiple field values of numbers and dates) ➢ No TIMEs in the datetime values, only dates

11. v11.0.1 DQL Features Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • DQL search strategy by term type Term type Full Text Search View Search NSF Scan field = ‘text’? No Yes Yes field contains (‘Text’) Yes No No field = 123122 Yes * Yes Yes field > ‘text’ No Yes Yes field < 123122 Yes * Yes Yes field = @dt(‘2020-09-17’) Yes * Yes Yes field > @dt(‘2020-09-17’) Yes * Yes Yes field = @dt(‘2020-09-17T11:00:00.00’) No Yes Yes * = single occurrence only (FT search does NOT handle ranges or lists)

12. v11.0.1 DQL Features (continued) Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • DQL statistics ▪ Avg, max, min by term type and overall ▪ “show stat query*” • Enhanced Boolean optimization ▪ Deeper results injection ▪ Reduces view and especially NSF scan processing for much faster query times

13. v12 Early Access September 2020 drop DQL Features Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • DQL client/server ▪ DominoQuery working against remote databases (not supported till now) ▪ Transaction appears in the “show trans” list from the Domino console ▪ Full support for substitution variables, entire DQL syntax

14. v12 Early Access September 2020 drop DQL Features (continued) Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • Embedded Formula Language ▪ Syntax ▪ @FL or @formula (case insensitive) ▪ Embedded syntax enclosed in single quotes @fl(‘@length > 2320) @FORMULA(‘@left(fld1;3) = "abl" or @left(fld2;2) = "ab“’) ▪ Provides ▪ Intra-document terms (between different fields) ▪ Full expression support ▪ HUGE library of existing functionality

15. v12 Early Access September 2020 drop DQL Features (continued) Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • Embedded Formula Language (continued) ▪ Support for substitution variables @fl(‘@length’) > ?lengthvar @formula(‘@left(fld1;3)’) = ?fld1var or @FL(‘@left(fld2;2)’) = ?fld2var ▪ Complete compatibility with the rest of DQL ▪ All formula language terms satisfied via NSF scanning ▪ All ANDed/ORed terms satisfied with a single pass ▪ Current implementation ▪ 256-byte maximum Formula Language syntax allowed ▪ No substitution variable support for an entire FL string

17. v12 Work in progress Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • Results processing ▪ Extensive handling of document collections ➢ NOT just DQL – ALL document collections ▪ Across multiple databases and database schemas ▪ Flexible computed values everywhere ▪ Sorting (including categorization) ▪ Combine rules (joins, lookups, etc.) ▪ Aggregate function support ▪ Impossible to avoid complexity

18. v12 Work in progress Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • Results processing (continued) – the details ▪ (Notes)QueryResultProcessor ➢ New backend (Java/Lotusscript) class ➢ Supplied lists of data and other classes working together ➢ Builds on known classes never combined or used this way before ➢ Extensive capabilities, multiple output formats and objects ➢ Low code? Not so much, but let’s build one to see

19. v12 Work in progress Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • (Notes)QueryResultProcessor (continued) – methods ▪ AddSortColumn(programmatic name, column title, sort order, hidden, categorized) ▪ Akin to creating a view column on the fly ▪ In fact, one of the output types IS a view ▪ May add additional attributes but those shown should be familiar from use in Designer ▪ REMEMBER the programmatic name – it will appear again in other calls ▪ If NOT overridden, will be used as a field name to fetch

20. v12 Work in progress Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • (Notes)QueryResultProcessor (continued) – methods ▪ AddDocumentCollection((Notes)DocumentCollection doccol, collection name) … or … ▪ AddDominoQuery ((Notes)DominoQuery, collection name) ▪ (Notes)DocumentCollection can be created against any database using any means (db.search, ftsearch, getalldocumentsbykey or DQL) ▪ DominoQuery option will also execute DQL for you ▪ Remember collection name – will appear later

21. v12 Work in progress Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • (Notes)QueryResultProcessor (continued) – methods ▪ AddFormula(document collection name, programmatic column name, formula) ➢ Used to override the programmatic name with a formula to compute and fetch values for processing by the results engine ➢ Can use one call to override the formula for multiple document collections using wildcards Addsortcolumn (“name”, “Name Column Title”, “ascending” … ) … AddDocumentCollection(doccol, “doccol1”) .. e.g. from db.search - database 1 AddDocumentCollection(doccol2, “doccol2) .. e.g. from ftsearch - database 2 AddFormula(“doccol*”, “name”, “@Propercase(NameField)”)

22. v12 Work in progress Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • (Notes)QueryResultProcessor (continued) – methods ▪ AddCombineRule (left programmatic name, rule, right programmatic name) ➢ Prior to sort or view create, combine data using the sort columns (with applied formulas) using the specified rule (or method) ➢ “Join” is the most familiar rule type, also keyword lookup ++ ▪ SetOutputType (type, viewname, streamtype) ➢ View (with name) or stream (with streamtype) ▪ Execute (run the result processor)

23. v12 Work in progress Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • (Notes)QueryResultProcessor how it works Sort doccol1 DQL1 ftdoccol Dbsearch doccol DQL2 Combine Fetch View Specified output Input (Notes)DocumentCollections Optional join/lookup Quick fetch including DQL executes (uses formula overrides Domino sort service (with streaming optimization) or

24. v12 Work in progress Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • (Notes)QueryResultProcessor additional information ▪ NEW - Sort package is decoupled from view building ➢ In stream mode, we don’t need to wait for a completed view build ▪ Separate column formulas for each DocumentCollection ▪ Views created can be used and discarded or retained ➢ Given input DocumentCollections, views cannot be refreshed ➢ Document security problem (multi-db documents have multiple readers and authors lists) ▪ Join strategies – sort/merge and nested loop using views or FT indexes

25. Early Access September 2020 drop non-DQL Feature Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • Domino Transaction Support in Lotusscript/Java ▪ (Notes)Database object has 3 new methods ➢ TransactionBegin – starts a unit of work ➢ TransactionCommit – completes a unit of work ➢ TransactionRollback – aborts everything since TransactionBegin ▪ Automatic rollback happens when ➢ Database object is closed or goes out of scope ➢ Agent is terminated ▪ Large, uncommitted transactions can cause log file full (crash) ▪ Transactions cannot nest (one db, one level, one at a time)

26. v12 Work in progress – non-DQL Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • Field length limit relief ▪ Aha idea - https://domino-ideas.hcltechsw.com/ideas/DDXP-I-5 ▪ Traditional limits: ➢ 32k limitation on field size ➢ 62k limitation on document summaries • Other problems: ➢ Better error messages when limits are hit ➢ Support in views and folders for larger fields ➢ Designer support (code size) ➢ Other Domino limitations – like 32k Lotusscript arrays

27. v12 Work in progress – non-DQL Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • Field length limit relief (continued) • (load) compact <db> -ls (or -largesummary) on ➢ First delivered ~ 9.01FP8 ➢ New max sizes: ✓ All fields (text lists or number or time ranges) – 64K ✓ Document summary size - 16MB (had been 62k) ➢ Requires compact to ODS 52 • Have you tried this?

28. v12 Work in progress – non-DQL Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • Field length limit relief (continued) ▪ Multiple core API and persistent data structure changes ▪ Will require a new ODS and large summaries enabled ▪ Only the max summary size (16MB) will remain – ➢ That is, the SUM of the lengths of all summary fields ➢ No size limit within the 16MB for individual summary fields ▪ Nonsummary fields can be up to 4GB (not a typo) ▪ Retain a hard limit on view-indexable fields (64K)

29. v12 Work in progress – non-DQL Copyright © 2020 HC L Technologies Limited | www.hcltechsw.com • Field length limit relief (continued) • “Field is too large (32K) or View's column & selection formulas are too large” ➢ Catch-all, multiple venues and error states throw the error ➢ Often is the SUM of field lengths generating the error – no one field is responsible • Effort for v12 – separate the states into 2 types and improve – ➢ Overall summary size – provide a document summary “map” to show sizes only ➢ Individual fields – include field name in the message

30. 30 Q&A

OpenNTF Webinar Series: DQL with John Curtis September 2020

Esté a ler uma amostra.

Confira estes a seguir

OpenNTF Webinar Series: DQL with John Curtis September 2020

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a OpenNTF Webinar Series: DQL with John Curtis September 2020 (20)

Mais recentes (20)