Fast api

W H O A M I
Simone Di Maulo - aka @toretto460
PUGGER
http://toretto.me

A G E N D A
• Performance
• HTTP

A G E N D A
• Performance
• HTTP
• DB Performance

A G E N D A
• Performance
• HTTP
• DB Performance
• Heavy load tasks

— D O N A L D K N U T H —
“ S T R U C T U R E D P R O G R A M M I N G W I T H G O T O S TAT E M E N T S ”
[…] premature optimization is the root of all evil.

— D O N A L D K N U T H —
“ S T R U C T U R E D P R O G R A M M I N G W I T H G O T O S TAT E M E N T S ”
[…] premature optimization is the root of all evil.
K E E P I N
M I N D

M E A S U R E M E A S U R E M E A S U R E M E A S U R E M E A S U R E
M E A S U R E

A PA C H E B E N C H M A R K
ab -n 200 -c 50 -k -g bench.tsv http://my.api.test/v1/pro...
^[1] ^[2] ^[3] ^[4] ^[5]
1. Number of requests
2. Concurrent requests
3. Use the keep-alive connection
4. Write the output ready for gnuplot
5. The URL to call

A PA C H E B E N C H M A R K
## Graph configuration ##
# Set the output format and size
set terminal jpeg size 1280,800
# Set the aspect ratio
set size 1, 1
# Title
set title "Benchmark testing"
# The legend/key position
set key left top
# Draw gridlines on the y axis
set grid y
# Label the x-axis
set xlabel 'seconds'
# Label the y-axis
set ylabel "response time (ms)"
## I/O Configuration
# Specify that the x-series data is time data
set xdata time
# The output file
set output "bench.jpg"
# Specify the *input* format of the time data
set timefmt "%s"
# Specify the *output* format for the x labels
set format x "%S"
# Use tabs as the delimiter instead of spaces
set datafile separator 't'
## Run the Plot
# Plot the data
plot "bench.tsv" every ::2 using 2:5 with points
exit

• https://blackfire.io
• https://tideways.io
• http://xhprof.io/
• https://symfony.com/doc/current/cookbook/profiler/
index.html

H T T P I S Y O U R F R I E N D

STATELESS ❤ ORIZONTAL SCALABILITY

COMPRESSION ❤ SLOW MOBILE NETWORKS

$ curl -I http://my-api.com/users/toretto460

{
“id”: da34gtyu50-lo983,
“username”: “toretto460”,
. . .
}

{
. . .
}
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Sat, 28 Jul 2015 20:12:45 GMT
Cache-Control: private max-age=600
ETag: 88493f3-4afd-507dd8e0aa030

{
. . .
}
HTTP/1.1 200 OK
Date: Sat, 28 Jul 2015 20:12:45 GMT

{
. . .
}
HTTP/1.1 200 OK
Date: Sat, 28 Jul 2015 20:12:45 GMT
Avoid same request until 28 Jul 2015 20:22:45 GMT

P L AY W I T H H E A D E R S
public function getAction($userIdentifier, Request $request)
{
$response = new Response();
$user = $this->fetchUser($userIdentifier);
$response->setETag($user->caclulateETag());
// $response->setLastModified($user->getLastUpdateDate());
if ($response->isNotModified($request)) {
return $response;
}
...
}
T O O M U C H
M A G I C I N S I D E

https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching

HTTP Compression is a set of rules
defined by the protocol which enable network
entities (Client and Server) to exchange
compressed data.

# Apache Configuration File
# ------------------------------------------------------------------------------
# | Compression |
# ------------------------------------------------------------------------------
<IfModule mod_deflate.c>
<IfModule mod_filter.c>
AddOutputFilterByType DEFLATE application/javascript
application/json
text/css
text/html
text/plain
text/xml
</IfModule>
</IfModule>
E N A B L E C O M P R E S S I O N

# Apache Configuration File
# ------------------------------------------------------------------------------
# | Compression |
# ------------------------------------------------------------------------------
<IfModule mod_deflate.c>
<IfModule mod_filter.c>
DeflateFilterNote Input input_info
DeflateFilterNote Output output_info
DeflateFilterNote Ratio ratio_info
LogFormat '"%r" %{output_info}n/%{input_info}n (%{ratio_info}n%%)' deflate
CustomLog /var/log/apache2/deflate_log deflate
</IfModule>
</IfModule>
L O G C O M P R E S S I O N

H T T P C O M P R E S S I O N
GET /users/toretto460 HTTP/1.1
Host: www.example.com
Accept-Encoding: gzip, deflate
HTTP/1.1 200 OK
Date: mon, 27 Jul 2015 22:38:34 GMT
Server: Apache/1.3.3.7 (Unix) (Red-Hat/Linux)
Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
Accept-Ranges: bytes
Content-Length: 438
Connection: close
Content-Type: text/html; charset=UTF-8
Content-Encoding: gzip

H T T P C O M P R E S S I O N
GET /users/toretto460 HTTP/1.1
Host: www.example.com
HTTP/1.1 200 OK
Date: mon, 27 Jul 2015 22:38:34 GMT
Server: Apache/1.3.3.7 (Unix) (Red-Hat/Linux)
Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
Accept-Ranges: bytes
Content-Length: 438
Connection: close
Content-Type: text/html; charset=UTF-8
Accept-Encoding: gzip, deflate
Content-Encoding: gzip

T H E P O W E R O F H T T P
Client ServerISP Proxy

Client cache

Client cache
Public cache

Client cache
Public cache
Reverse Proxy
Data compression

T I P S
• AUTO_INCREMENT / SEQUENCE are not so cheap
• Understanding the TRACKING POLICY
• Use READONLY entities
• Let’s CACHE

A U T O _ I N C R E M E N T & S E Q U E N C E S
A R E N O T S O C H E A P
class User
{
/**
* @ORMId
* @ORMColumn(name="ID", type="string", length=37, nullable=false)
* @ORMGeneratedValue(strategy="AUTO")
*/
private $id;
}

A U T O _ I N C R E M E N T & S E Q U E N C E S
A R E N O T S O C H E A P
class User
{
/**
* @ORMId
* @ORMGeneratedValue(strategy="AUTO")
*/
private $id;
public function __construct(UserInfoDTO $userInfo)
{
$this->id = (string) Uuid::uuid4();
$this->name = $userInfo->name;
...
}
}

U N D E R S TA N D I N G T H E
T R A C K I N G P O L I C Y

U N D E R S TA N D I N G T H E
T R A C K I N G P O L I C Y
— M A R T I N F O W L E R —
“Maintains a list of objects affected by a business
transaction
and coordinates the writing out of changes and the
resolution of concurrency problems.”
UNIT OF WORK

U N I T O F W O R K
D ATA B A S E

$em->persist($user);
U N I T O F W O R K
D ATA B A S E

U N I T O F W O R K
$em->flush();
D ATA B A S E

D E F E R R E D I M P L I C I T
With this policy, Doctrine detects the changes by a
property-by-property comparison at commit time and
also detects changes to entities or new entities that are
referenced by other managed entities …
http://doctrine-orm.readthedocs.org/en/latest/reference/change-tracking-policies.html

D E F E R R E D I M P L I C I T
class User
{
/**
* @ORMId
*/
private $id;
/**
* @ORMColumn(name="FIRST_NAME", type="string", length=100, nullable=false)
*/
private $firstName;
/**
* @ORMColumn(name="LAST_NAME", type="string", length=100, nullable=false)
*/
private $lastName;
/**
* @ORMColumn(name="BIRTH_DATE", type="date", nullable=false)
*/
private $birthDate;
}
UOW
$em->flush();

D E F E R R E D E X P L I C I T
… the difference is that Doctrine 2 only considers
entities that have been explicitly marked for change
detection through a call to
EntityManager::persist(entity)
or through a save cascade.

D E F E R R E D E X P L I C I T
class User
{
/**
* @ORMId
*/
private $id;
/**
* @ORMColumn(name="FIRST_NAME", type="string", length=100, nullable=false)
*/
private $firstName;
/**
* @ORMColumn(name="LAST_NAME", type="string", length=100, nullable=false)
*/
private $lastName;
/**
* @ORMColumn(name="BIRTH_DATE", type="date", nullable=false)
*/
private $birthDate;
}
UOW
$em->flush();

N O T I F Y
This policy is based on the assumption that the entities
notify interested listeners of changes to their properties.
For that purpose, a class that wants to use this policy
needs to implement the NotifyPropertyChanged
interface

N O T I F Y
<?php
use DoctrineCommonNotifyPropertyChanged,
DoctrineCommonPropertyChangedListener;
/**
* @Entity
* @ChangeTrackingPolicy("NOTIFY")
*/
class User implements NotifyPropertyChanged
{
// ...
private $_listeners = array();
public function addPropertyChangedListener(PropertyChangedListener $listener)
{
$this->_listeners[] = $listener;
}
}

N O T I F Y
<?php
// . . .
protected function _onPropertyChanged($propName, $oldValue, $newValue)
{
if ($this->_listeners) {
foreach ($this->_listeners as $listener) {
$listener->propertyChanged($this, $propName, $oldValue, $newValue);
}
}
}
public function setData($data)
{
if ($data != $this->data) {
$this->_onPropertyChanged('data', $this->data, $data);
$this->data = $data;
}
}
}

U S E R E A D O N LY E N T I T I E S
This means that the entity marked as read only is never
considered for updates

U S E R E A D O N LY E N T I T I E S
/**
* @ORMEntity(readOnly=true)
*/
class NewsTag
{
/**
* @ORMId
*/
private $id;
}

L E T ’ S C A C H E
/**
* Configure the Doctrine ORM with a result cache provider.
*/
$config = new DoctrineORMConfiguration();
// The APC way
$apcCacheDriver = new DoctrineCommonCacheApcCache();
$config->setResultCacheImpl($apcCacheDriver);
// The memcache way
$memcache = new Memcache();
$memcache->connect('memcache_host', 11211);
$memCacheDriver = new DoctrineCommonCacheMemcacheCache();
$memCacheDriver->setMemcache($memcache);
$config->setResultCacheImpl($memCacheDriver);

doctrine:
orm:
metadata_cache_driver: apc
query_cache_driver: apc
result_cache_driver:
type: memcache
host: localhost
port: 11211

class UserRepository
{
/**
* @return User[]
*/
protected function findLocked()
{
$qb = $this->createQueryBuilder('user');
$qb->where($qb->expr()->eq('user.locked', true));
$query = $qb->getQuery();
$query->useResultCache(true, 60 * 2 /* 2 minutes */);
return $query->getResult();
}
}

R E A L U S E C A S E - # 1 D ATA I M P O R T
> 2 0 0 K E N T I T I E S
SEQUENCE UUID
$em->flush(get_class($entity));
$em->clear(get_class($entity));

> 2 0 0 K E N T I T I E S

> 2 0 0 K E N T I T I E S
208k Entities
loaded in 2h 25m
😟

> 2 0 0 K E N T I T I E S
AFTER THE CURE
208k Entities
loaded in 28 minutes
😊

S L O W TA S K
The user avatar should be resized to 300x300px
The administrator should upload up to 250k
orders in a single cvs file
Send an email when a new post has been created

S L O W TA S K
A crawler should find and aggregate product info
Every X minutes the rss feed should be updated
The user can export all the orders

R E A L U S E C A S E - # 2 R E P O R T I N G

AS AN analyst
I WANT TO request for a new monthly report

AS AN analyst
CREATE TABLE TMP_CFMS_R7_7844 AS SELECT ROWNUM AS ID, x.* FROM (SELECT COD_DEALER, COD_PDV, TOTAL_ACQUISIZIONI,
CONFRONTO_ACQUISIZIONI, AVG_ACQUISIZIONI, PERCENTUALE_INCREMENTO, CAST( NULL AS NUMBER ) QTA_SOGL, CAST( NULL AS
NUMBER ) QTA_SUPE_SOGL, TOTAL_SOSPENSIONI, SOSPENSIONI_CONFRONTO, AVG_SOSPENSIONI, PERCENTUALE_INCREMENTO_SOSP,
CAST( NULL AS NUMBER ) QTA_SOGL_SOSP, CAST( NULL AS NUMBER ) QTA_SUPE_SOGL_SOSP, PERCENTUALE_SOSPENSIONI_ACQU,
PERC_SOSP_ACQU_CONF, PERCENTUALE_SUPERAMENTO, D_RAG_SOC, D_IND, D_LOCALITA, D_CAP, D_PROVINCIA, D_PIVA, INDIRIZZO
V_IND, LOCALITA V_LOCALITA, CAP V_CAP, PROVINCIA V_PROVINCIA FROM ANAGRAFICA_DEALER, (SELECT COD_DEALER, COD_PDV,
TOTAL_ACQUISIZIONI, CONFRONTO_ACQUISIZIONI, AVG_ACQUISIZIONI, PERCENTUALE_INCREMENTO, CAST( NULL AS NUMBER )
QTA_SOGL, CAST( NULL AS NUMBER ) QTA_SUPE_SOGL, TOTAL_SOSPENSIONI, SOSPENSIONI_CONFRONTO, AVG_SOSPENSIONI,
PERCENTUALE_INCREMENTO_SOSP, CAST( NULL AS NUMBER ) QTA_SOGL_SOSP, CAST( NULL AS NUMBER ) QTA_SUPE_SOGL_SOSP,
PERCENTUALE_SOSPENSIONI_ACQU, PERC_SOSP_ACQU_CONF, PERCENTUALE_SUPERAMENTO, RAGSOC D_RAG_SOC, INDIRIZZO D_IND,
LOCALITA D_LOCALITA, CAP D_CAP, PROVINCIA D_PROVINCIA, PIVA D_PIVA FROM ANAGRAFICA_DEALER, (SELECT
A.COD_DEALER,A.COD_PDV,NVL(A.TOTAL_ACQUISIZIONI,'0') TOTAL_ACQUISIZIONI,NVL(A.CONFRONTO_ACQUISIZIONI,'0')
CONFRONTO_ACQUISIZIONI,NVL(B.AVG_ACQUISIZIONI,'0') AVG_ACQUISIZIONI,NVL(B.PERCENTUALE_INCREMENTO,'0')
PERCENTUALE_INCREMENTO, CAST( NULL AS NUMBER ) QTA_SOGL, CAST( NULL AS NUMBER )
QTA_SUPE_SOGL ,NVL(C.TOTAL_SOSPENSIONI,'0') TOTAL_SOSPENSIONI,NVL(C.SOSPENSIONI_CONFRONTO,'0')
SOSPENSIONI_CONFRONTO, NVL(D.AVG_SOSPENSIONI,'0') AVG_SOSPENSIONI,NVL(D.PERCENTUALE_INCREMENTO_SOSP,'0')
PERCENTUALE_INCREMENTO_SOSP, CAST( NULL AS NUMBER ) QTA_SOGL_SOSP, CAST( NULL AS NUMBER )
QTA_SUPE_SOGL_SOSP ,NVL(E.PERCENTUALE_SOSPENSIONI_ACQU,'0') PERCENTUALE_SOSPENSIONI_ACQU,
NVL(PERC_SOSP_ACQU_CONF,'0') PERC_SOSP_ACQU_CONF,NVL(PERCENTUALE_SUPERAMENTO,'0') PERCENTUALE_SUPERAMENTO FROM
(SELECT ROWNUM AS ID, x.* FROM (SELECT COD_DEALER, COD_PDV, TOTAL_ACQUISIZIONI, CONFRONTO_ACQUISIZIONI, CAST( NULL
AS NUMBER ) QTA_SOGL, CAST( NULL AS NUMBER ) QTA_SUPE_SOGL, D_RAG_SOC, D_IND, D_LOCALITA, D_CAP, D_PROVINCIA,
D_PIVA, INDIRIZZO V_IND, LOCALITA V_LOCALITA, CAP V_CAP, PROVINCIA V_PROVINCIA FROM ANAGRAFICA_DEALER, (SELECT
COD_DEALER, COD_PDV, TOTAL_ACQUISIZIONI, CONFRONTO_ACQUISIZIONI, CAST( NULL AS NUMBER ) QTA_SOGL, CAST( NULL AS
NUMBER ) QTA_SUPE_SOGL, RAGSOC D_RAG_SOC, INDIRIZZO D_IND, LOCALITA D_LOCALITA, CAP D_CAP, PROVINCIA D_PROVINCIA,
PIVA D_PIVA FROM ANAGRAFICA_DEALER, ( SELECT
A.COD_DEALER,A.COD_PDV,A.TOTAL_ACQUISIZIONI,NVL(B.TOTAL_ACQUISIZIONI,'0') CONFRONTO_ACQUISIZIONI, CAST( NULL AS
NUMBER ) QTA_SOGL, CAST( NULL AS NUMBER ) QTA_SUPE_SOGL FROM (SELECT A.COD_DEALER,A.COD_PDV,COUNT(A.COD_DEALER)
TOTAL_ACQUISIZIONI, CAST( NULL AS NUMBER ) QTA_SOGL, CAST( NULL AS NUMBER ) QTA_SUPE_SOGL FROM CFMS_ATT A WHERE
A.DATA_ATTIVAZIONE between TO_DATE('2015-06-15 00:00:00','yyyy-mm-dd HH24:MI:SS') AND TO_DATE('2015-06-15
23:59:59','yyyy-mm-dd HH24:MI:SS')
AND A.COD_PDV IS NOT NULL GROUP BY A.COD_DEALER,A.COD_PDV) A LEFT JOIN ( SELECT
COD_DEALER,COD_PDV,COUNT(COD_DEALER) TOTAL_ACQUISIZIONI FROM CFMS_ATT WHERE DATA_ATTIVAZIONE between
TO_DATE('2015-06-16 00:00:00','yyyy-mm-dd HH24:MI:SS') AND TO_DATE('2015-06-16 23:59:59','yyyy-mm-dd HH24:MI:SS')
AND COD_PDV IS NOT NULL GROUP BY COD_DEALER,COD_PDV ) B ON A.COD_DEALER=B.COD_DEALER AND A.COD_PDV=B.COD_PDV) WHERE
COD_DEALER = COD_ID (+)) WHERE COD_PDV = COD_ID (+) ORDER BY COD_DEALER) x ) A FULL OUTER JOIN (SELECT ROWNUM AS
ID, x.* FROM (SELECT COD_DEALER, COD_PDV, ACQUISIZIONI, AVG_ACQUISIZIONI, ACQUISIZIONI_CONFRONTO,
AVG_ACQUISIZIONI_CONFRONTO, PERCENTUALE_INCREMENTO, D_RAG_SOC, D_IND, D_LOCALITA, D_CAP, D_PROVINCIA, D_PIVA,
INDIRIZZO V_IND, LOCALITA V_LOCALITA, CAP V_CAP, PROVINCIA V_PROVINCIA FROM ANAGRAFICA_DEALER, (SELECT COD_DEALER,
COD_PDV, ACQUISIZIONI, AVG_ACQUISIZIONI, ACQUISIZIONI_CONFRONTO, AVG_ACQUISIZIONI_CONFRONTO, PERCENTUALE_INCREMENTO,
RAGSOC D_RAG_SOC, INDIRIZZO D_IND, LOCALITA D_LOCALITA, CAP D_CAP, PROVINCIA D_PROVINCIA, PIVA D_PIVA FROM
ANAGRAFICA_DEALER, ( SELECT A.COD_DEALER,A.COD_PDV,A.ACQUISIZIONI,TO_CHAR(A.AVG_ACQUISIZIONI,'99990D99')

AS AN analyst
SO THAT the system will send me an email
with the attached csv report

S C A L I N G T H E A S Y N C
Y O U R A P P
I S
H E R E
W O R K E R
W O R K E R
W O R K E R
W O R K E R
Q U E U E
POST /rss
PATCH /users
POST /export

A S Y N C H R O N O U S TA S K
• https://github.com/uecode/qpush-bundle
• https://github.com/videlalvaro/php-amqplib
• https://github.com/videlalvaro/RabbitMqBundle
• http://gearmanbundle.readthedocs.org/en/latest/configuration.html
• https://github.com/chrisboulton/php-resque

Fast api

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Fast api

Semelhante a Fast api (20)

Mais de Simone Di Maulo

Mais de Simone Di Maulo (6)

Último

Último (20)

Fast api