Workshop eZ Publish Caching Mechanisms

eZ Summer Camp. Setp 2012

eZ Publish Caching
Mechanisms
// WORKSHOP

Gilles Guirand - CTO Kaliop / eZ Publish board member @gandbox
Gilles Ballini – Lead engineer Kaliop
Stephane Sobecki - Lead engineer Kaliop
Jerome Lecocq - Lead engineer Kaliop

:~$ eZ Publish Caching Mechanisms // SPICY DETAILS

AGENDA (1)

TEMPLATE COMPILATION
Concept / exploring the VAR
Ex : HOWTO compile a set of template from a TXT file list

INI CACHE
Ex : HOWTO use dynamic settings with shared compiled templates

VIEWCACHE
Ex : HOWTO log the viewcache activity

2 / 61


AGENDA ( 2 )
CACHE-BLOCK
Ex : HOWTO expire a named cache-block, related to the homepage
expiry, or a contentclass, or whatever else
Ex : HOWTO expire a set of cache-block using a shell script
Ex : HOWTO build the perfect pagelayout

EXPIRY.PHP FILE

EZCACHE.PHP SCRIPT
Ex : HOWTO create a custom ezcache.php --id=myid

PHP CACHE
Ex : HOWTO generate a PHP based cache file
3 / 61


CONTEXT
Common questions :
➔Is it possible to never expire the ENTIRE viewcache ?
… and which actions actually expire the ENTIRE viewcache ?

➔ Is it possible to reduce the I/O (template-blocks) ?

➔ Is it possible to expire a specific set ot template-block files ?

➔ Is it possible to develop a better pagelayout.tpl ?

➔ Is something to improve on high-trafic websites ? Or Websites
factory ( 100 siteaccess... )

4 / 61


CONTEXT
Final questions :
➔ How all these cache mechanisms really work ?
… I mean in details... algorithms, I/O, triggers, exceptions,
undocumented features & settings, unknown bugs, unknown
fixes... and ezfs, ezfs2, ezdfs, ezdb...

Final solutions :
➔ READ THE KERNEL / TEST THE KERNEL... ( be brave )

5 / 61


VM PREPARATION
Copy cache & cache2 extension folders
Activate extension (site.ini.append.php)
Generate autoloads
Clear the cache-all

Add user / login policy for anomymous ( site
access cache & cache 2 )

6 / 61


AGENDA
TEMPLATE
COMPILATION
Ex : HOWTO compile a set of template from a TXT file list

INI CACHE
VIEWCACHE
CACHE-BLOCK
EXPIRY.PHP FILE
EZCACHE.PHP SCRIPT
PHP CACHE
7 / 61


TPL COMPILATION
What for ?
➔ Compile your .tpl files to .php files
➔ Reduce the RAM / CPU (use the stored compiled version)

How to set ? site.ini [templatesettings]
➔ TemplateCompile=enabled
➔ Store compiled version {varDir}/cache/template/compiled/

➔ NodeTreeCaching=enabled ( disabled by default )
➔ Store pre-compiled version {varDir}/cache/template/tree/

Common mistake
➔ TemplateCache=enabled → template-block

8 / 61


TPL COMPILATION
DESIGN {varDir}/cache/template/tree {varDir}/cache/template/compiled

page_mainarea.tpl {hash}-page_mainarea.php page_mainarea-{hash1}.php

NodeTreeCaching=enabled
page_mainarea-{hash2}.php
( disabled by default )

page_mainarea-{hash3}.php

Why eZ generates several
compiled files for the same .tpl?

9 / 61


TPL COMPILATION
How eZ stores the final compiled files ?

{VarDir}/cache/template/compiled/, not clusterized

Filenames : {templatename}-{hash}.php
page_head-49f4458d0b68aec3cf2de63a6918fd61.php
page_head-ec3cf2de63a6918fd6149f4458d0b68a.php

10 / 61


TPL COMPILATION
How eZ calculates the hash ?
$sharedTemplates = false

{VarDir}/cache/template/compiled/

pagelayout-49f4458d0b68aec3cf2de63a6918fd61.php

Md5( filepath ) / internalCharset / language / useFullUrlText /
accessText / pageLayoutVariable / indexFile / extraName

/layout/set/...
11 / 61


TPL COMPILATION
How eZ calculate the hash ?
$sharedTemplates = true

{VarDir}/cache/template/compiled/

pagelayout-aec3cf2de63a6918fd6149f4458d0b68.php

Md5( filepath ) / language

No more siteaccess inside the key...
Ezini operator → 'dynamic' parameter
12 / 61


EXERCISE
How to pre-compile a set of template files
(to avoid : on the fly generating) ?
➔ Force the compilation of 2 template files
php bin/php/eztc.php -s mysiteaccess --force
extension/ezdemo/design/ezdemo/templates/page_head.tpl
extension/ezdemo/design/ezdemo/templates/pagelayout.tpl

➔ Force the compilation of all template files starting with « page » :
find extension/ezdemo/ -name "page_*.tpl" | xargs php bin/php/eztc.php
-s eng --force

➔ Force the compilation of all template files from a TXT file :
cat templates.txt | xargs php bin/php/eztc.php -s eng --force

13 / 61


TPL COMPILATION
The **** layout module
➔ By default ( $shareTemplates = false ), the 'layout' value is used by the
final compiled filename
➔ So if you are using this module with 3 other layouts, you'll produce 4
compiled files (instead of 1)
➔ eztc.php script does not manage the layout parameter

Your 3 additional compiled files ( for each layout ) will
be procude « on the fly », and cannot be pre-compiled
with eztc.php
➔ Suggestion : improve eztc.php, add the –layout=... parameter

14 / 61


AGENDA

INI CACHE
Ex : HOWTO use dynamic settings with shared compiled templates

VIEWCACHE
CACHE-BLOCK
EXPIRY.PHP FILE
EZCACHE.PHP SCRIPT
PHP CACHE

15 / 61


INI CACHE
When eZ generates the INI cache ?
➔ On the fly (if the cache file does not exist) – from an API call
(ezini PHP class), a template call (ezini operator)

➔ By default, eZ automatically check if an INI file has been
changed or not, depending of this setting (config.php)

//define( 'EZP_INI_FILEMTIME_CHECK', false ) ;

/* Set EZP_INI_FILEMTIME_CHECK constant to false to improve performance
( 10% ~ 15% on I/O ) by not checking modified time on ini files. You can also set it to
a string, the name of a ini file you still want to check modified time on, best example
would be to set it to 'site.ini' to make the system still check that but not the rest. */

16 / 61


INI CACHE
How eZ stores the INI cache ?
➔ var/cache/ini ( and not /{varDir}/ini ), not clusterized

➔ {inifilename}-{hash}.php
design-677247a49e428aa0837411b52777b920.php
design-704216b4d0e3ea68c09742504bb366c8.php
design-7333039e7e19aa411f8dd01836555861.php
md5

FileName / RootDir / DirectAccess / overrideDirs /
internalCharset

17 / 61


INI CACHE
How to expire the INI cache ?
➔php bin/php/ezcache.php --clear-id=global_ini
➔ Remove var/cache/ini

➔php bin/php/ezcache.php --clear-tag=ini
➔ Remove var/cache/ini &

var/cache/active_extensions_{hash}.php

➔php bin/php/ezcache.php --clear-id=ini
➔ Nothing (bug) : try to recursively delete INI files in

{varDir}/ini

18 / 61


INI CACHE
How to not compile the INI value ?
lib/ezutils/classes/ezini.php

function eZINI( $fileName = 'site.ini', $rootDir = '', $useTextCodec = null, $useCache
= null, $useLocalOverrides = null, $directAccess = false, $addArrayDefinition = false,
$load = true )
{
...
}
Dynamically load the INI values inside your compiled templates :

site.ini [eZINISettings] / DynamicTemplateMode=enabled
OR
Ezini operator → 'dynamic' parameter

19 / 61


EXERCISE
HOWTO use dynamic settings with shared compiled templates

20 / 61


AGENDA
INI CACHE

VIEWCACHE
Ex : HOWTO log the viewcache activity

CACHE-BLOCK
EXPIRY.PHP FILE
EZCACHE.PHP SCRIPT
PHP CACHE

21 / 61


VIEWCACHE
What for ?
➔ Viewcache = content cache, content
viewcache, …

➔ Viewcache is an HTML (or other) cache of the
$module_result.content outpout, for a node
view call ( content/view/... )

← Illustration : Łukasz Serwatka
http://share.ez.no/learn/ez-publish/ez-publish-performance-optimization-
part-3-of-3-practical-cache-and-template-solutions/(page)/2

22 / 61


VIEWCACHE
How eZ stores the viewcache ?
{VarDir}/cache/content/{siteaccess}/{number}/{number}/{number}

{VarDir}/cache/content/{siteaccess}/
2-ce182234bd70236c6a009578d2a34632.cache
20-ce182234bd70236c6a009578d2a34632.cache

{VarDir}/cache/content/{siteaccess}/1
101-ce182234bd70236c6a009578d2a34632.cache
105-ce182234bd70236c6a009578d2a34632.cache

{VarDir}/cache/content/{siteaccess}/5/2/1
52101-ce182234bd70236c6a009578d2a34632.cache
52144-ce182234bd70236c6a009578d2a34632.cache

23 / 61


VIEWCACHE
How eZ stores the viewcache ?
{nodeid}-{hash}.php
2-ce182234bd70236c6a009578d2a34632.cache
md5
required

nodeID / viewMode / layout / language (view.p.) / offset
(view.p.) / indexFile
optionnal +
roIeIDList / limitValueList / discountList / cacheNameExtra /
pr_user / viewParameters / userPreferences
ViewCacheTweaks[<node_id>]=<setting>[;<setting2>]
ViewCacheTweaks[global]=<setting>[;<setting2>]

24 / 61


VIEWCACHE
How to know the exact matching between a
viewcache file & my URLs ?
{nodeid}-{hash}.php → deduce the nodeID

At the end of file :
"view_parameters";a:6:
{s:6:"offset";b:0;s:4:"year";b:0;s:5:"month";b:0;s:3:"day";b:0;s:10:"na
mefilter";b:0;s:4:"test";s:1:"2";}...

Should be URL : /(test)/2

25 / 61


VIEWCACHE
How eZ expires a set of viewcache ?
Alteration of cache states - Depending of the filehandler ( file.ini )

➔ Ezfsfilehandler : UNLINK file
➔ Ezfs2filehandler : TOUCH file (25/25/1977) ← Star Wars day
➔ Ezdbfilehandler : "ezdbfile" (UPDATE expired=1)
➔ Ezdfsfilehandler : "ezdfsfile" (UPDATE expired=1)

Comparison of cache states - Depending of the filehandler ( file.ini )

➔ Ezfsfilehandler : file does not exist, or is older than the content-view-cache
timestamp
➔ Ezfs2filehandler : file is older than the content-view-cache timestamp
➔ Ezdbfilehandler : "ezdbfile" (expired=1)
➔ Ezdfsfilehandler : "ezdfsfile" (expired=1)

26 / 61


VIEWCACHE
How eZ expires the ENTIRE viewcache ?
➔ eZ does not recursively update / delete the files (too many files... unsafe)

eZ set a global timestamp expiry value :
➔

{VarDir}/cache/expiry.php ← 'content-view-cache' ( current timestamp)

Each time the expiry.php is updated, the ENTIRE viewcache will
expire.
➔ Ok, but my ENTIRE viewcache expires all the
time... i did nothing special ! Don't know why !

27 / 61


VIEWCACHE
Which event / trigger expires a set of
cache ?
By default : on object publication (or back-end feature)
➔ the related nodes & parent node

Willingly, using a script :
➔ bin/php/ezcontentcache.php –clear-node=2

➔ bin/php/ezcontentcache.php –clear-node=2,46,63

➔ bin/php/ezcontentcache.php –clear-node=/company/about

➔ bin/php/ezcontentcache.php –clear-subtree=/company/about

➔ bin/php/ezcontentcache.php –clear-subtree=/company/about,/news

28 / 61


VIEWCACHE
Smart viewcache & cacheThreshold : a set
of cache → the ENTIRE view cache
Automatically, if using the smart view cache / viewcache.ini :

➔ The smart view cache populates a list of related « node_id » viewcache files
to expire, depending of your settings. If this node_id list is larger than the
CacheThreshold limit value (250 by default) → eZ expires the ENTIRE
viewcache !

➔ Some settings could increase the node_id list count & excede the
CacheThreshold limit : ALL, siblings, KeywordNodesCacheClearLimit

➔ Advice : Set KeywordNodesCacheClearLimit = 0, never use ALL, avoid
siblings, log out the CacheThreshold (ezpEvent)

29 / 61


EXERCISE
How to log the viewcache activity ? &
prevent the CacheThreshold limit...

Add an ezpEvent : site.ini.append.php

[Event]
Listeners[]=content/cache@log::logviewcache

# log = your PHP class name
# logviewcache = the static method

30 / 61


EXERCISE
Log the viewcache node_id list to expire,
➔ Create a PHP class / generate autoloads
// extension/extension-name/classes/log.php

<?php
class log
{
static public function logviewcache( $nodeList )
{
$uri = $GLOBALS['_SERVER']['REQUEST_URI'];
eZLog::write('node_id count : '. count( $nodeList ). ' / node_id list : ' . implode(', ', $nodeList ) . ' / URI : '.$uri,
'cachethresold.log');

return $nodeList;
}}
?>

➔ Generate autoloads : php bin/php/ezpgenerateautolaods.php

31 / 61


EXERCISE
● Edit / publish contents in the administration backend

➔ Log output : var/log/cachethreshold.log
[ Sep 03 2012 17:37:33 ] node_id count : 14 / node_id list : 74, 73, 75, 76,
79, 80, 81, 84, 85, 78, 77, 82, 72, 100 / URI :
/administration/content/edit/72/2/eng-GB

[ Sep 05 2012 13:44:54 ] node_id count : 7 / node_id list : 89, 88, 90, 91, 2,
1, 87 / URI : /administration/content/edit/87/2/eng-GB

[ Sep 05 2012 13:45:34 ] node_id count : 2 / node_id list : 2, 1 / URI :

32 / 61


EXERCISE
● Edit / publish contents in the administration backend

➔ Log output : var/log/cachethreshold.log
[ Sep 03 2012 17:37:33 ] node_id count : 14 / node_id list : 74, 73, 75, 76,
79, 80, 81, 84, 85, 78, 77, 82, 72, 100 / URI :

[ Sep 05 2012 13:44:54 ] node_id count : 7 / node_id list : 89, 88, 90, 91, 2,
1, 87 / URI : /administration/content/edit/87/2/eng-GB

[ Sep 05 2012 13:45:34 ] node_id count : 2 / node_id list : 2, 1 / URI :

33 / 61


VIEWCACHE
My ENTIRE viewcache still expire...
Maybe you invoked one of these features (back-end, API) :

➔ Create a new content class ( BUG ? )
➔ Update / delete an existing content class ( or attributes )
➔ Update / delete and existing role ( & policies ) OR role assignement
➔ Section assignement
➔ Update an ezshop rule ( discount rules, currency... )

34 / 61


35 / 61


36 / 61


AGENDA
INI CACHE
VIEWCACHE

CACHE-BLOCK
Ex : HOWTO expire a named cache-block, related to the homepage
expiry, or a contentclass, or whatever else
Ex : HOWTO expire a set of cache-block using a shell script

EXPIRY.PHP FILE
EZCACHE.PHP SCRIPT
PHP CACHE
37 / 61


CACHE-BLOCK
... What for ?
➔ Template-block cache = cache files related to
cache-block, also called « template cache »
( common mistake with the template
compilation )

➔ Template-block cache is an HTML (or other)
cache of a template part ( cache-block )

← Illustration : Łukasz Serwatka
http://share.ez.no/learn/ez-publish/ez-publish-performance-optimization-
part-3-of-3-practical-cache-and-template-solutions/(page)/2

38 / 61


CACHE-BLOCK
Impact of common parameters
The « keys » parameter :

Please AVOID :
{cache-block keys=$uri_string} 1 template-block file for
... each URI !!!
{/cache-block}

Use instead :
{cache-block keys=$my_custom_limited_range_key}
...
{/cache-block}
Only possible states of your content
inside your block
39 / 61


CACHE-BLOCK
The « URI key » parameter : stupid but signifiant simulation

➔ A classic Web site, with 1000 nodes, news folder ( year, month archives ),
products catalog ( filters ), calendar, forum...
➔ Could generate 10 000 « indexable » several URI ( google bot )
➔ A cache-block ( only 1 URI key ) to cache the global menu, who is
generating 300 SQL queries
➔ No « expiry » or « ignore_content_expiry » parameters : so the cache-block
expires every 2 hours (or on content publication)

➔ So... 300 SQL queries * 12 expiration per day * 10 000 URI = 36 millions of
SQL queries per day...

40 / 61


CACHE-BLOCK
The « expiry » parameter :

If possible, always set a custom expiry value & ignore content
expiry :
{cache-block ignore_content_expiry expiry=86400}
...
{/cache-block}

41 / 61


CACHE-BLOCK
The « sutree_expiry » parameter : no more expiry /
ignore_content_expiry needed

{cache-block subtree_expiry='news/'}
...
{/cache-block}

{cache-block subtree_expiry=142}
...
{/cache-block}

42 / 61


CACHE-BLOCK
The « sutree_expiry » parameter : tip of the « magic » node

➔ To expire a cache-block, depending of a custom rule ( per content class, per
PHP algorithm ... )

➔ Create a {cache-block subtree_expiry=$magic_node_id}

➔ Create a node somewhere ( safe location, without FULL view )

➔ Create a workflow or customedithandler to include your custom rules on
publication, and update the content object related to the magic node by the
code

43 / 61


CACHE-BLOCK
Cache-block imbrication
{cache-block expiry=3600}
...

... {* you save … SQL Queries each hours *}

{/cache-block}
...
{cache-block subtree_expiry='news/'}
...
{/cache-block}
...
{/cache-block}

44 / 61


CACHE-BLOCK
I need a « real time » block ( TTL=1m )
A cache-block may not be the solution, use instead :

➔ ESI ( Edge Side Include ) : better for front-end loading & google
bot indexing. But reverse-proxy required ( varnish, squid,
akamai... )

➔ AJAX : easier to implement

45 / 61


CACHE-BLOCK
How eZ stores the template-block cache
{VarDir}/cache/template-block/1/2/3/12345678.cache


{VarDir}/cache/template-block/subtree/1/5/9/cache/1/2/3/12345678.cache

subtree_expiry=159 CRC32
Complex... The
pathname doesn't help

46 / 61


CACHE-BLOCK
How eZ stores the template-block cache

CRC32 ( 32-bit polynomial checksum )

Cache-block position in the .tpl file / template path / keys /
siteaccess

78_1_78_94_extension/ezwebin/design/ezwebin/templates/pagelayout.tpl

78 = {cache-block... } line start
1 = {cache-block... } column start (a simple ' ' before...)
78 = {cache-block... } line end
94 = {cache-block... } column end
47 / 61


CACHE-BLOCK
How to expire the ALL of the template-
block cache files ?
php bin/php/ezcache.php –clear-id=template-block

➔ Expires all the template-block files, by updating the global
cache-block expiry time ( global-template-block-cache ) in the
{varDir}/expiry.php file

48 / 61

CACHE-BLOCK
How to expire a set of specific template-
block cache files ?
➔ Not really possible... exept by searching a specific string inside
each files, and deleting matching files ( ezfs / ezFS2 )

➔ find var/ezdemo_site/cache/template-block/ -name "*.cache" -exec grep -iHl
"string-to-search" {} ; -delete


...
{/cache-block}

49 / 61

CACHE-BLOCK
PULL REQUEST IDEA :
{cache-block id=myfolder ignore_content_expiry expiry=0}

...
{/cache-block}

{VarDir}/cache/template-block/myfolder/1/2/3/12345678.cache

So... to expire your cache-block :
mv myfolder myfolder-to-remove
nice -n 20 rm -r myfolder-to-remove
50 / 61


EXERCISE
Ex : HOWTO expire a named cache-block, related to the
homepage expiry, or a contentclass, or whatever else

Ex : HOWTO expire a set of cache-block using a shell
script


51 / 61


AGENDA
INI CACHE
VIEWCACHE
CACHE-BLOCK

EXPIRY.PHP FILE
(not so) funny quiz

EZCACHE.PHP SCRIPT
PHP CACHE

52 / 61


EXPIRY.PHP FILE
<?php
$Timestamps = array (
What for ?
'state-limitations' => 1321010927,
'user-info-cache' => 1323788789,
'content-view-cache' => 1326299075,
'class-identifier-cache' => 1326298969,
'global-template-block-cache' => 1325856140,
'content-tree-menu' => 1326298969,
'image-manager-alias' => 1323788788,
'active-extensions-cache' => 1325668728,
'ts-translation-cache' => 1323788789,
'content-complex-viewmode-cache' => 1325780763,
'template-block-cache' => 1326298969,
'user-class-cache' => 1326298969,
'sort-key-cache' => 1326298969,
);

53 / 61


AGENDA
INI CACHE
VIEWCACHE
CACHE-BLOCK
EXPIRY.PHP FILE

EZCACHE.PHP
SCRIPT
Ex : HOWTO create a custom ezcache.php --id=myid

PHP CACHE
54 / 61


EZCACHE.PHP
SCRIPT
FAMOUS EZSCRIPT
php bin/php/ezcache.php --clear-id=[ID]
php bin/php/ezcache.php --clear-id=[ID] --purge
php bin/php/ezcache.php --clear-tag=[TAG]
php bin/php/ezcache.php --clear-tag=[TAG] --purge
php bin/php/ezcache.php --clear-all
php bin/php/ezcache.php --clear-all --purge

NEED SOME HELP Does it do something
php bin/php/ezcache.php --help special ?
php bin/php/ezcache.php --list-ids Superstitious parameter
php bin/php/ezcache.php --list-tags
55 / 61


EZCACHE.PHP
SCRIPT
TAG ID
content content / classid / sortey / urlalias / rss_cache /
content_tree_menu / state_limiations / template-block /
ezjscore-packer

template template / template-block / template-override /
texttoimage / design_base / ezjscore-packer

ini ini / global_ini / active_extensions

user user_info_cache
I18n translation / chartrans
codepage codepage
image imagealias
rest rest / rest-routes

56 / 61


EZCACHE.PHP
SCRIPT
MAIN ALGORITHM
Hardcoded in kernel/classes/ezcache.php.
For each ID :

'id' => ID
'tag' => array( TAG LIST )
'enabled' => Cache enabled or not
'is-clustered' => Use the current filehandler or not ?
'path' => Path to purge, if purging (is-clustered=false)
'expiry-key' => Key to set inside the expiry.php file if needed
'function' => Function to apply only on clearing (not purging)
'function-purge' => Function to apply only on purging (not clearing)

57 / 61


EZCACHE.PHP
SCRIPT
MAIN ALGORITHM
Add your own ID for your extensions ( site.ini.append.php )

# Cache item entry (for eZ Publish 4.3 and up)
[Cache]
CacheItems[]=ezjscore

[Cache_ezjscore]
name=eZJSCore Public Packer cache
id=ezjscore-packer
tags[]=content
tags[]=template
path=public
isClustered=true

58 / 61


EXERCISE
How to create a custom ID cache
expiration ?

Use the ezbashcompletion extension :
https://github.com/bdunogier/ezbashcompl
etion

59 / 61


EZCACHE.PHP
SCRIPT
A BUG HELPS TO UNDERSTAND
php bin/php/ezcache.php --clear-id=ini

array( 'name' => ezpI18n::tr( 'kernel/cache', 'INI cache' ),
'id' => 'ini',
'tag' => array( 'ini' ),
'enabled' => true,
'path' => 'ini' ),

Why is it not working ?

60 / 61


AGENDA
INI CACHE
VIEWCACHE
CACHE-BLOCK
EXPIRY.PHP FILE
EZCACHE.PHP SCRIPT

PHP CACHE
Ex : HOWTO generate a PHP based cache file

61 / 61

Workshop eZ Publish Caching Mechanisms

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a Workshop eZ Publish Caching Mechanisms

Semelhante a Workshop eZ Publish Caching Mechanisms (20)

Mais de Kaliop-slide

Mais de Kaliop-slide (20)

Último

Último (20)

Workshop eZ Publish Caching Mechanisms