Mais conteúdo relacionado

Apresentações para você(19)

Similar a MidSem(20)



  1. Intern – Balance Tracker Asmita Sharma – MCA 2012-15
  2. Profile  Name – Asmita Sharma  Roll No. – 06  Year – 2012- 2015  University – Delhi University  University Mentor – Ms. Neelima Gupta  Organization – Amazon Inc.  Manager – Mr. Ramesh Krishna Kumar  Mentor – Itiyama Sadana  Team – BTS  Platform – Ecommerce Payments  Role – Software Development Engineer Intern  Internship Start Date – 16 Feb 2015  Internship End Date – 15 Aug 2015
  3. Development Environment JavaS3 Perl JSONJunit Spring Brazil Coral Git Apollo Odin SQS Easy Mock Log4j Dynamo DB Pipelines Version Sets Eclipse
  4. Internship Road Map I am here.
  5. Glossary  Pipeline : A tool for modeling your release process and adding automation to it.  Version Set : A snapshot of application dependencies.  SQS : Simple Queuing Service  DynamoDB : NoSQL Database  S3 : Simple Storage Service  Workflow : Sequence of executable steps.  SWF : Simple Workflow  Environment Stages : Association between hosts and Apollo environments.  Alpha : Environment stage receiving no traffic.  Beta : Environment stage receiving minimal traffic for testing.  Prod : Environment stage receiving real-time traffic.
  6. BTS - Overview  BTS (Balance Tracking System) is a Global Stored Value Platform of Amazon.  BTS is a part of ecommerce payments platform of Amazon.  It allows any business in Amazon to create instruments and track funds movement for them.  It also provides simple APIs for querying balance of an instrument and history of fund movement transactions done on an instrument.
  7. Operational Task - Overview  Objective : To migrate packages from JAVA - 1.7 to JAVA - 1.8  Packages :  PerseusClient  PerseusClientInterface  PerseusTestClient  PerseusTestSuite  MonadArchivalBusinessClient  MonadArchivalLocator  Pipelines :  PerseusClient-development  MonadArchivalPerseusClient/development  SubTasks :  Code Change  Merge Dependencies  Testing  Deployment  Migrate Pipelines
  8. Operational Task – Code Change  Changes to Config file involved adding, removing and updating versions of build and runtime dependencies to latest versions supporting JAVA 8.  Config file changes also involved resolving conflicting dependencies of the package with it’s upstream and downstream packages.  Changes to build.xml involved adding properties for classpath of new packages and few more changes to make upstream JAVA 7 packages compatible with JAVA 8 migrated package.  Changes to Spring configuration xml files involved modification of syntax for dependencies migrated to latest versions supporting JAVA 8.  Changes to Spring configuration xml files also involved using classpath approach for integrating spring packages instead of providing url of spring packages.
  9. Operational Task – Merge dependencies  Live is a version set which contains all the versions of all the packages required within amazon.  Every package corresponds to a version set which contains all the dependencies of the packages in that version set.  To upgrade or add a new version of any dependency, we need to merge that dependency into the version set from live.
  10. Operational Task – Testing  A package is first tested in the workplace itself using brazil CLI commands and targets specified in build.xml.  After these tests are passed, any package dependent on this package is merged to workspace and tested to run with these changes.  When above tests pass successfully, peer review is done and suggested improvements are incorporated.  After peer review is complete , the package is pushed to a temporary branch of the repository.
  11. Operational Task – Deployment  The package from the temporary branch is build into the version set and deployed to the pipeline.  It is tested at various stages (namely Alpha and Beta) of pipeline through regression suites.  After it passes these stages and logs of the hosts at these stages are verified, it is deployed to Prod.
  12. Operational Task – Migrate Pipelines  Migrating pipeline involves merging all packages migrated to Java 8 that belong to that pipeline or are a dependency of the packages belonging to the pipeline.  Any conflicts arising while merging these packages have to be resolved and then merged again.  Logs of hosts at each stage (Alpha , Beta, Prod) have to be verified after this step also.
  13. Minor Project  Objective : To remove reconciliation from PerseusClient/development-clone and MonadBalanceServiceListener pipeline.  Packages :  PerseusClient  PerseusClientInterface  PerseusTestClient  PerseusTestSuite  NotificationParser  ModelConverter  BalanceUpdateBusinessClient  BalanceQueryServiceSAO  SubTasks :  Code Change  Modifying Regressions  Modifying Unit Test Cases  Resetting AWS Credentials  Testing  Major Version Upgrade  Write Change Management (CM) Document  Deployment
  14. Minor Project – Code Change  Removing Tracker Table and Marker Table instances along with their interface and unit test cases.  Removing all SWF (Simple Workflow) workflows and corresponding unit tests.  Removing dependencies from Config file which are no longer needed.  Removing brazil-config file properties that are no longer required.  Removing unnecessary targets and properties from build.xml.  Removing reconciler instances from all packages.  Deleting spring configuration files corresponding to reconciler .  Removing all packages used by reconcilera which are no longer required.
  15. Minor Project – Modifying Regressions  Removing DynamoDB instances previously used by tracker table and marker table.  Modifying regressions to test only notification functionality and neglect reconciliation.  Events in modified regression suite will not be dropped to test reconciler functionality.  All regressions related to reconciler and workflows will be deleted.  In modified regression suite, only notification listener will be tested by updating a dummy instrument and verifying that it’s notification reaches event listener through SQS (Simple Queuing Service).  Reconciler instances and unnecessary dependencies will be removed from Config and xml files.
  16. Minor Project – Modifying Unit Test Cases  Added unit test for Event Details class  Added spring beans to set S3 (Simple Storage Service) and AWS (Amazon Web Services) credentials and configured the same in spring configuration xml files and brazil-config.  Removed unit test cases for workflows and dynamoDB tables.  Removed unit test cases involving missing sequence number tests and updated event listener test for the same.  Modularized a function by shifting s3 upload functionality from the function to a separate method.
  17. Minor Project – Resetting AWS Credentials  Earlier the AWS credentials were set using Odin which did not allow the credentials to be seamlessly retrieved dynamically.  Hence, used AWSCredentialProvider to set AWS credentials for SQS in PerseusClient and PerseusTestClient using spring beans and brazil-config.  This approach eliminates the race condition where dynamic credentials could change in between calls to getAccessKey/getSecretAccessKey/getSessionToken and leave the customer with invalid credentials.  This also makes it very easy for the AWSCredentialsProvider implementation to return different types of credentials objects.
  18. Minor Project – Testing  Created a child environment of the Alpha stage of the PerseusClient-development pipeline and deployed it on local machine using Apollo.  Attached my workspace and modified packages to this child environment using brazil.  Activated the child environment using brazil.  Logged in to the remote host set up at Alpha stage of pipeline using SSH command.  Verified the logs running on the host.  Deactivated the parent environment.  Logged in to the SWF console and verified that the workflows have stopped executing.
  19. Minor Project – Major Version Upgrade  Upgraded the major version of all packages to 2.0.  Changes were made to Config file and build.xml file of all packages.
  20. Major Project  Objective : To support Query APIs on S3.  API:  GetEvent()  GetInstrumentEvents()  GetChildEvents()  GetInitializeStatus()  SubTasks :  Data Analysis  Design Approach  Write Design Document  Environment Setup  Code Development  Writing Unit Tests  Deriving and Coding Regressions  Testing  Deployment
  21. Major Project – Data Analysis  Objective : To analyze up to 3 months old logs to gather requirements.  API :  GetInstrumentEvents()  SubTasks :  Figure out data required from logs  Write Script to gather data from logs  Run script of hosts containing logs  Post-process data for concluding requirements  Represent data graphically  Output :  Domains accessing more than year old events  Total number of events returned by each API call  Number of more than year old , more than six month old and more than three month old events returned by each API call  Number of calls returning more than year old events made by each domain  Difference between API call time and creation time of oldest event returned by each API call
  22. Major Project – Design Approach  Objective : To come up with an approach that best fits the requirements of all the four APIs with optimal use of resources as per the required performance.  SubTasks :  Understand the functionality , input fields and output fields of the APIs  Figure out the required latencies  Analyze available tools for suitability as per the requirements  Devise design approaches  Calculate storage, latencies and cost metrics for each approach  Analyze feasibility, simplicity of development, scalability and maintenance of each approach  Choose between Sync and Async path  Compare candidate approaches  Choose the best fit
  23. Major Project – Design Document Index 1.1 Prior Knowledge 1.1.1 APIs 1.1.2 EventID Format 1.1.3 InstrumentID Format 1.1.4 S3 File Format 1.2 Data Analysis 1.2.1 Script 1.2.2 Requirements 1.3 Design Approach 1.3.1 Storing S3 files on hourly basis and scanning files for required instrument 1.3.2 Creating index on instrument ID per day Index Structure Start_File and End_File field size calculation Metrics Cost of storing indexes in S3 Cost of storing indexes in DynamoDB Comparison 1.3.3 Creating instrument level indexes 1.3.4 Using CloudSearch to index S3 files Costing
  24. Yet To Do…  Minor Project  Deployment  Writing Change Management Document  Major Project  Environment Setup  Code Development  Writing Unit Tests  Deriving and Coding Regressions  Testing  Deployment
  25. Thank you