My experience with embedding PostgreSQL

2. About me •  Sr Manager, VMware Data –  Leads Postgres team at VMware –  Blog at hKp://jkshah.blogspot.com •  Previously Staﬀ Engineer, Sun Microsystems –  Team Member of the ﬁrst published SpecJAppServer 2004 benchmark with PostgreSQL –  Performance of PostgreSQL on Solaris/Sun systems •  Working with PostgreSQL circa 2005 © 2013 VMware, Inc. All rights reserved

4. The IniXal Problem •  >30 # products and product components •  >20 requires DB •  Customers have to get their own DB license •  Increases the Total Cost of Deployment –  As database load increases, CPU cores required increases, so does the total license cost of DB © 2013 VMware, Inc. All rights reserved

5. SQL Server Express •  Currently shipped for some Windows based products –  However limits not enough –  Max 10GB database usage •  Standard and Enterprise EdiXons get expensive preKy quickly •  Not Available on Linux © 2013 VMware, Inc. All rights reserved

7. IniXal External Feedback •  Actually got pushback from customers, partners –  PostgreSQL not part of their database standards –  No in-‐house PostgreSQL DBA skills –  Lack of integrated tools , monitor, ease-‐of-‐use –  Large enterprises have to convince their managed hosXng partners to support the database also •  AKendees here are going to be educated about it –  but we are worried about the rest of the world © 2013 VMware, Inc. All rights reserved

8. IniXal Internal Feedback •  DB Support Matrix complexity increases –  Every new DB support increases N plaborm test –  Large scale tests are serialized due to resources © 2013 VMware, Inc. All rights reserved

9. Plan: Embedded Default DB •  Embed PostgreSQL as part of the product •  Make it the default choice opXon for most deployment •  Do not expose database directly to endusers •  ApplicaXon will manage the DB •  Support it for producXon deployment •  Customer can sXll select to use an external database of their choice or save DB License cost with the embedded one © 2013 VMware, Inc. All rights reserved

10. Ground Work: MigraXon •  Products used various databases –  SQL Server –  Oracle –  MySQL •  Few had about 1500 stored procedures •  Some used exoXc features •  For example MySQL used InMemory Engine, InnodB engine, MyISAM engine in the same workload © 2013 VMware, Inc. All rights reserved

11. MigraXon •  Leveraged contractors to migrate stored procedures •  Performance bake-‐oﬀ tests to convince product teams •  Integrated into the build infrastructure to ease pulling components © 2013 VMware, Inc. All rights reserved

13. Plan: Backup & Recovery •  DB not the only thing to be backed up •  ConﬁguraXon ﬁles •  ApplicaXon Status •  All data to be backed up went into a virtual disk backed up with vADP •  pg_dump / pg_restore used in few cases © 2013 VMware, Inc. All rights reserved

14. IniXal Embedded DB Deployments •  Each Product team is diﬀerent –  has diﬀerent levels of DB Development skills •  Each architect comes up with their own design –  Postgres Service designed (Start/Stop) –  own DB Layout design –  own DB Security design –  Some Tuned their DB –  Many did not © 2013 VMware, Inc. All rights reserved

15. Early Results •  YES ! Customer started seeing benefit •  However –  Experience of PostgreSQL different on each product –  Random security holes idenXfied due to lack of knowledge of PostgreSQL nuances –  Performance configuraXon was an issue –  Upgrade to new versions of PostgreSQL was an akerthought (What no in-‐place upgrade?) –  Legal work required for OSS increased –  Resulted in PostgreSQL version sXckiness © 2013 VMware, Inc. All rights reserved

16. Some encounters •  Portable binaries – Shrink wrapped sokware (not compiled on same systems) –  OSS Libraries used for compiling PostgreSQL have to be shipped –  Manage security patches for OSS Libraries •  Need Standard drivers like ODBC, JDBC © 2013 VMware, Inc. All rights reserved

19. InstallaXon •  Extend installaXon to setup a service •  Conﬁgurable to set up as internal service or external service (mostly embedded) •  Security handled as part of service setup to follow best pracXcesfor embedded •  Security based on access and minimum roles deﬁned © 2013 VMware, Inc. All rights reserved

20. InstallaXon •  RPM installaXon –  Dependent OSS Libraries to make RPM portable –  Server and client RPMs –  Service Setup conﬁgurator to setup a default service •  •  •  •  User setup initdb setup pg_hba setup Password handling –  Provide tools to quickly setup DB Owner, DB for various users of the service © 2013 VMware, Inc. All rights reserved

22. Backup & Recovery •  While other backups strategy are available –  Nothing beats standard database backups –  Design cron jobs for criXcal data –  Design Archive On Demand for semng up replicaXon quickly –  Most customer do not want to hack with backup/ restore scripts. –  Should work out of the box © 2013 VMware, Inc. All rights reserved

23. Upgrades •  Seamless upgrade to new version –  RPM Upgrades needs to be transparent for minor versions –  RPM upgrades for major version needs to migrate data from the older version in the default service © 2013 VMware, Inc. All rights reserved

24. Upgrades •  Problems sXll not completely resolved –  Reduce downXme for major release upgrades –  Generally migraXon since in-‐place upgrade does not provide ways to roll back the upgrade © 2013 VMware, Inc. All rights reserved

25. Performance •  Default PostgreSQL semng are conservaXve •  Most deployments use default values •  Started doing auto-‐tuning –  Modiﬁed PostgreSQL core engine with intelligence –  For checkpoint segments, shared_memory •  Added tool to auto tune postgresql.conf –  Based on memory share for DB Service –  Product workload dependent © 2013 VMware, Inc. All rights reserved

26. Performance related tools •  Provided tools to understand what’s happening with Postgres –  pg_top with changes •  Changes already contributed back to community © 2013 VMware, Inc. All rights reserved

29. Availability •  Created a watchdog for PostgreSQL to restart PostgreSQL if database cannot be accessed for long •  If restart fails then it can throw alerts •  Also created plugins for monitoring the service © 2013 VMware, Inc. All rights reserved

30. Related Problems: Crashes •  PostgreSQL started throwing errors with “Out of Memory” errors in stress tests •  MulXple applicaXons can ﬁght for memory •  Made Performance semngs less aggressively © 2013 VMware, Inc. All rights reserved

31. Supportability •  When customers hit a problem –  Hard to get access to the system •  Created support package tool to collect relevant informaXon –  Collect conﬁg, status in a tar ﬁle for analysis © 2013 VMware, Inc. All rights reserved

34. Embedded DBaaS Architecture VMware Product Service Conﬁgurator WebUI Pg_top JDBC PostgreSQL ODBC Linux Based VM Watchdog Perf-‐Tuner Monitoring Support package tool vSphere Plaborm © 2013 VMware, Inc. All rights reserved

37. ReplicaXon between vApps •  Provide simple uXliXes to quickly –  archive_command –  create_replicaXon_user –  run_as_replica –  show_replica_status –  promote_replica_to_primary © 2013 VMware, Inc. All rights reserved

38. In Closing •  Free Sokware is nice •  Experience – What business users care –  User Experience –  DBA Experience –  Support Experience –  etc •  Improve the Experience with the technology © 2013 VMware, Inc. All rights reserved

39. Thanks. QuesXons? Download VMware vFabric Postgres hKp://vmware.com/go/download-‐vfabric-‐postgres Follow us on twitter: @vPostgres Data Platform Blog: http://blogs.vmware.com/dataplatforms © 2013 VMware, Inc. All rights reserved

My experience with embedding PostgreSQL

Esté a ler uma amostra.

Confira estes a seguir

My experience with embedding PostgreSQL

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Quem viu também gostou (20)

Semelhante a My experience with embedding PostgreSQL (20)

Mais recentes (20)