The document discusses the internals of Twig, a templating language for PHP. It provides an overview of key Twig concepts like the lexer, parser, abstract syntax tree, template compilation process, and extensions. The lexer tokenizes template code into a stream of tokens. The parser processes these tokens to build an abstract syntax tree, representing the template's structure. Twig then compiles templates by transforming the abstract syntax tree into PHP code.
2. MATTHIAS NOBACK
Zeist, the Netherlands
Feature addition in March 2014
Started as a web designer (2003)
Was employed for 6 years at
several companies...
3. Now self employed: Noback's Office
Blog: php-and-symfony.matthiasnoback.nl
Twitter: @matthiasnoback
6. FABIEN POTENCIER
Lead developer of the Symfony project
Was looking for a Django-like templating
engine for Symfony2
Found Twig and started "hacking" on it
15. EXTENSIONS
itraeTi_xesoItrae
nefc wgEtninnefc
{
pbi fnto gtucin(;
ulc ucin eFntos)
pbi fnto gtitr(;
ulc ucin eFles)
pbi fnto gtet(;
ulc ucin eTss)
}
..
.
$n-adxeso(etnin / i tewy
ev>dEtnin$xeso) / s h a
(Symfony2: create a service with a t i . x e s o tag)
wgetnin
16. FUNCTIONS
casMEtninetnswgEtnin
ls yxeso xed Ti_xeso
{
pbi fnto gtucin(
ulc ucin eFntos)
{
rtr ary
eun ra(
nwwgSmlFnto(
e Ti_ipeucin
'yucin,
mFnto'
fnto (tig {
ucin $hn)
rtr srnf'hsi <>y/>%.,$hn)
eun pit(Ti s bm<b s' tig;
}
)
)
;
}
}
{ mFnto(cmue" }
{ yucin"optr) }
17. FILTERS
casMEtninetnswgEtnin
ls yxeso xed Ti_xeso
{
pbi fnto gtitr(
ulc ucin eFles)
{
rtr ary
eun ra(
nwwgSmlFle(
e Ti_ipeitr
'ie,
mn'
fnto (wa,$ie=tu){
ucin $ht mn
re
rtr srnf
eun pit(
's(hc % mn),
% wih s ie'
$ht
wa,
$ie?'s:i nt
mn
i''s o'
)
;
}
)
)
;
}
}
{ tigmn(as)}
{ hn|iefle }
18. TESTS
casMEtninetnswgEtnin
ls yxeso xed Ti_xeso
{
pbi fnto gtet(
ulc ucin eTss)
{
rtr ary
eun ra(
nwwgSmlTs(
e Ti_ipeet
'_ofrne,
acneec'
fnto(nm){
ucin$ae
rtr $ae=='yfnCn;
eun nm = Smoyo'
}
)
)
;
}
}
{ i "yfnCn i acneec %Itl yus{ edf%
% f Smoyo" s _ofrne } od o o% ni }
21. TWIG ENVIRONMENT
$n =nwwgEvrnet)
ev
e Ti_niomn(;
$n-stodrnwwgLae_ieytm_DR_'tmlts);
ev>eLae(e Ti_odrFlsse(_I_./epae')
$otx =ary
cnet
ra(
..
.
)
;
eh $n-rne(idxhm.wg,$otx)
co ev>edr'ne.tlti' cnet;
22. WHAT IS A TEMPLATE?
casTi_niomn
ls wgEvrnet
{
..
.
}
pbi fnto rne(nm,ary$otx =ary)
ulc ucin edr$ae ra cnet
ra()
{
rtr $hs>odepae$ae-rne(cnet;
eun ti-laTmlt(nm)>edr$otx)
}
A template is a class that implements
wgTmltItrae
Ti_epaenefc
l a T m l t ( returns an instance of such a class
odepae)
23. COMPILED TEMPLATE CLASS
/ idxhm.wg*
* ne.tlti /
cas_TiTmlt_12098fe1d89ef1e6etnsTi_epae
ls _wgepaedd753ba3f93c0e59 xed wgTmlt
{
poetdfnto dDslyary$otx,ary$lcs=ary)
rtce ucin oipa(ra cnet ra bok
ra()
{
/ ln 1
/ ie
i (se(cnet"ae]){$nm_=$otx[nm";}es {$nm_=nl
f ist$otx[nm")
_ae
cnet"ae]
le
_ae
u
eh ti_saefle(ti-ev $nm_ "tl,nl,tu)
co wgecp_itr$hs>n, _ae, hm" ul re;
eh "i i afl"
co
s n
ie;
}
}
pbi fnto gtepaeae)
ulc ucin eTmltNm(
{
rtr "ne.tlti"
eun idxhm.wg;
}
24. BEFORE
<>{nm }<p
p{ ae }/>
AFTER
eh 'p'
co <>;
i (se(cnet"ae]){
f ist$otx[nm")
$nm_=$otx[nm";
_ae
cnet"ae]
}es {
le
$nm_=nl;
_ae
ul
}
eh ti_saefle(ti-ev $nm_ "tl,nl,tu)
co wgecp_itr$hs>n, _ae, hm" ul re;
eh '/>;
co <p'
25. HOW TWIG CREATES A
TEMPLATE CLASS
1. Retrieve the source (written in "Twig") from the loader(s)
2. Compile the source to a PHP class
29. FINDING TWIG BLOCKS
The lexer first checks for the position of the main markers:
Start of block:
{
%
Start of variable: {
{
Start of comment: {
#
Then the lexer
1. iterates over the resulting positions, while
2. checking some basic syntax rules, and
3. collecting tokens on its way to E F
O
30. TOKEN TYPES
Tokens have:
a type
a value (optional)
a line number
BOKSAT
LC_TR
BOKED
LC_N
VRSAT
A_TR
VRED
A_N
TX
ET
NM
AE
NME
UBR
SRN
TIG
OEAO
PRTR
PNTAIN
UCUTO
..
.
{
%
%
}
{
{
}
}
raw template data
f r i , etc.
o, f
a number
" . "or ' . '
..
..
+ * ~ etc.
, , ,
| [ { etc.
, , ,
..
.
31. Take this template:
{ edf%
% ni }
<l
u>
{ frie i ies%
% o tm n tm }
<i{ ie|aiaie}<l>
l>{ tmcptlz }/i
{ edo %
% nfr }
<u>
/l
$ee =$n-gtee(;
lxr
ev>eLxr)
$epae=..
tmlt
.;
$oeSra =$ee-tknz(tmlt)
tkntem
lxr>oeie$epae;
32. TOKEN STREAM
eh $oeSra;
co tkntem
BOKSAT
LC_TR
NM(ni)
AEedf
BOKED
LC_N
TX(u>
ET<l)
BOKSAT
LC_TR
NM(o)
AEfr
NM(tm
AEie)
OEAO(n
PRTRi)
NM(tm)
AEies
BOKED
LC_N
..
.
EF
O
{
%
edf
ni
%
}
raw template
data
{
%
fr
o
ie
tm
i
n
ies
tm
%
}
..
.
end of input
{ edf%
% ni }
<l
u>
{ frie i ies%
% o tm n tm }
<i{ ie|aiaie}<l>
l>{ tmcptlz }/i
{ edo %
% nfr }
<u>
/l
33. STATES
To keep track of what the lexer is doing.
DATA
lexing raw template data (start state)
BLOCK lexing a block
VAR
lexing a variable
STRING lexing a string
35. SYNTAX VALIDATION
Each block and variable should be closed
{ fr{ i
% o % f
Brackets ( [should be closed symmetrically
{
{ [a }
{ '' }
Closing brackets ] )can not occur first
}
{ ]}
{
}
37. FROM SYNTAX TO
SEMANTICS
The resulting list of tokens may be semantically incorrect.
In the Twig language, that is...
{ edf%
% ni }
<l
u>
{ frie i ies%
% o tm n tm }
<i{ ie|aiaie}<l>
l>{ tmcptlz }/i
{ edo %
% nfr }
<u>
/l
42. THE ROOT NODE
The parsing process results in a root node containing
the body of the template,
or a collection of blocks,
the link to a parent template,
...
43. THE MAIN TOKEN TYPES
The parser collects nodes based on the token at the current
position in the token stream.
TX
ET
template text
create a T x node with the value of the
et
token
VRSAT
A_TR
variable
parse the expression that follows and
expect V R E D
A_N
B O K S A T block with a tag expect a name, which is the name of the
LC_TR
tag (i.e. for, if, etc.) and call a subparser
44. SUBPARSER == TOKEN
PARSER
Each token parser defines its own rules for the tokens that
should follow the tag:
{ frie i ies%
% o tm n tm }
{ icue'epaehm'wt {fo:'a' %
% nld tmlt.tl ih 'o' br} }
{ stfo br='o' 'a'%
% e o, a
fo, br }
45. FROM TOKENS TO NODES
The token parser returns nodes based on the tokens it finds.
Returned nodes are inserted in the Abstract Syntax Tree
46. A CUSTOM TOKEN PARSER
casCneecTknasretnswgTknasr
ls ofrneoePre xed Ti_oePre
{
pbi fnto prewgTkn$oe)
ulc ucin as(Ti_oe tkn
{
$hs>asr>eSra(-epc(Ti_oe:BOKED;
ti-pre-gttem)>xetwgTkn:LC_N)
$xr=nwwgNd_xrsinCntn(Smoyo'
ep
e Ti_oeEpeso_osat'yfnCn,
$oe-gtie);
tkn>eLn()
}
}
rtr nwwgNd_rn(ep,$oe-gtie) $hs>eTg);
eun e Ti_oePit$xr tkn>eLn(, ti-gta()
pbi fnto gta(
ulc ucin eTg)
{
rtr 'ofrne;
eun cneec'
}
{ cneec %
% ofrne }
47. Teew g,{ cneec %!
hr e o % ofrne }
$epae=..
tmlt
.;
$n =nwwgEvrnet)
ev
e Ti_niomn(;
$n-adoePre(e CneecTknasr);
ev>dTknasrnw ofrneoePre()
eh $n-pre$n-tknz(tmlt);
co ev>as(ev>oeie$epae)
(Better: register them using the g t o e P r e s )of
eTknasr(
your Twig extension class)
48. EXCERPT OF THE ABSTRACT
SYNTAX TREE
Ti_oeMdl(
wgNd_oue
bd:Ti_oeBd(
oy wgNd_oy
0 Ti_oe
: wgNd(
0 Ti_oeTx(
: wgNd_et
dt:'hr w g,'
aa Tee e o
)
1 Ti_oePit
: wgNd_rn(
ep:Ti_oeEpeso_osat
xr wgNd_xrsinCntn(
vle 'yfnCn
au: Smoyo'
)
)
2 Ti_oeTx(
: wgNd_et
dt:''
aa !
)
)
)
)
55. THE ROOT NODE
The node tree contains nodes generated by:
the parser
the expression parser
token parsers
node visitors
56. THE COMPILE STEP
{ frie i ies%
% o tm n tm }
{ ie|aiaie}<r
{ tmcptlz }b>
{ edo %
% nfr }
$epae=..
tmlt
.;
eh $n-cmieore$epae;
co ev>oplSuc(tmlt)
57. EXCERPT OF THE COMPILED
TEMPLATE
cas_TiTmlt_4dc9f0249098c82eetnsTi_epae
ls _wgepaed18d80b0e809ef47 xed wgTmlt
{
poetdfnto dDslyary$otx,ary$lcs=ary)
rtce ucin oipa(ra cnet ra bok
ra()
{
/ ln 1
/ ie
i (se(cnet"tm") {$ies =$otx[ies] }es {$ies =
f ist$otx[ies])
_tm_
cnet"tm";
le
_tm_
$otx[_aet]=(ra)$otx;
cnet'prn'
ary cnet
$otx[_e' =ti_nuetaesbe$ies)
cnet'sq]
wgesr_rvral(_tm_;
frah(cnet'sq]a $otx[_e" = $otx[ie"){
oec $otx[_e' s cnet"ky] > cnet"tm]
/ ln 2
/ ie
eh ""
co
;
i (se(cnet"tm]){$ie_=$otx[ie";}es {$ie_=
f ist$otx[ie")
_tm
cnet"tm]
le
_tm
eh ti_saefle(ti-ev ti_aiaiesrn_itr$hs>n,
co wgecp_itr$hs>n, wgcptlz_tigfle(ti-ev
eh "b>;
co <r"
}
$prn =$otx[_aet]
_aet
cnet'prn';
ust$otx[_e',$otx[_trtd] $otx[_e',$otx[ie'
ne(cnet'sq] cnet'ieae', cnet'ky] cnet'tm]
$otx =arymre$prn,aryitretky$otx,$prn);
cnet
ra_eg(_aet ra_nesc_e(cnet _aet)
}
}
58. RECURSIVE COMPILING
The compiler just calls the c m i e )
opl(
method of the root node.
Which calls the c m i e )of child
opl(
nodes, etc, etc.
59. CUSTOM TOKEN PARSER
REVISITED
casCneecTknasretnswgTknasr
ls ofrneoePre xed Ti_oePre
{
pbi fnto preTi_oe $oe)
ulc ucin as(wgTkn tkn
{
..
.
$xr=nwwgNd_xrsinCntn(Smoyo'
ep
e Ti_oeEpeso_osat'yfnCn,
$oe-gtie);
tkn>eLn()
}
}
rtr nwwgNd_rn(ep,$oe-gtie) $hs>eTg);
eun e Ti_oePit$xr tkn>eLn(, ti-gta()
60. COMPILING A PRINT NODE
casTi_oePit
ls wgNd_rn
{
pbi fnto cmieTi_oplr$oplr
ulc ucin opl(wgCmie cmie)
{
$oplr
cmie
-adeuIf(ti)
>dDbgno$hs
-wie'co'
>rt(eh )
-sbopl(ti-gtoe'xr)
>ucmie$hs>eNd(ep')
-rw""
>a(;n)
;
}
}
The result can be found in the compiled template:
poetdfnto dDslyary$otx,ary$lcs=ary)
rtce ucin oipa(ra cnet ra bok
ra()
{
/ ln 1
/ ie
..
.
eh "yfnCn;
co Smoyo"
..
.
}
61. SOME REFLECTIONS
1. You can put any PHP code you want inside a template
2. You can do (heavy) calculations at compile time (just once)
You only have to create your own node type and
implement its compile method.
62. TESTING AND DEBUGGING
Writing parsers and nodes can be quite difficult, so
1. read the compiled templates in your cache directory and
2. write unit tests for your custom parser and node type
(extend from w g T s _ o e e t a e
Ti_etNdTsCs)