O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Visualization and forecasting of big time series data

7.213 visualizações

Publicada em

Talk given at the ACEMS big data workshop in Brisbane, 23 February 2015

Publicada em: Educação
  • Well done, also...GROW YOU DOWNLINE OVERNIGHT - Works with any mlm. Have dozens joining whatever mlm your doing today! Go to: www.mlmrc.com
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

Visualization and forecasting of big time series data

  1. 1. Rob J Hyndman Visualizing and forecasting big time series data Victoria: scaled
  2. 2. Outline 1 Examples of biggish time series 2 Time series visualisation 3 BLUF: Best Linear Unbiased Forecasts 4 Application: Australian tourism 5 Fast computation tricks 6 hts package for R 7 References Visualising and forecasting big time series data Examples of biggish time series 2
  3. 3. 1. Australian tourism demand Visualising and forecasting big time series data Examples of biggish time series 3
  4. 4. 1. Australian tourism demand Visualising and forecasting big time series data Examples of biggish time series 3 Quarterly data on visitor night from 1998:Q1 – 2013:Q4 From: National Visitor Survey, based on annual interviews of 120,000 Australians aged 15+, collected by Tourism Research Australia. Split by 7 states, 27 zones and 76 regions (a geographical hierarchy) Also split by purpose of travel Holiday Visiting friends and relatives (VFR) Business Other 304 bottom-level series
  5. 5. 2. Labour market participation Australia and New Zealand Standard Classification of Occupations 8 major groups 43 sub-major groups 97 minor groups – 359 unit groups * 1023 occupations Example: statistician 2 Professionals 22 Business, Human Resource and Marketing Professionals 224 Information and Organisation Professionals 2241 Actuaries, Mathematicians and Statisticians 224113 Statistician Visualising and forecasting big time series data Examples of biggish time series 4
  6. 6. 2. Labour market participation Australia and New Zealand Standard Classification of Occupations 8 major groups 43 sub-major groups 97 minor groups – 359 unit groups * 1023 occupations Example: statistician 2 Professionals 22 Business, Human Resource and Marketing Professionals 224 Information and Organisation Professionals 2241 Actuaries, Mathematicians and Statisticians 224113 Statistician Visualising and forecasting big time series data Examples of biggish time series 4
  7. 7. 3. Spectacle sales Visualising and forecasting big time series data Examples of biggish time series 5 Monthly UK sales data from 2000 – 2014 Provided by a large spectacle manufacturer Split by brand (26), gender (3), price range (6), materials (4), and stores (600) About 1 million bottom-level series
  8. 8. 3. Spectacle sales Visualising and forecasting big time series data Examples of biggish time series 5 Monthly UK sales data from 2000 – 2014 Provided by a large spectacle manufacturer Split by brand (26), gender (3), price range (6), materials (4), and stores (600) About 1 million bottom-level series
  9. 9. 3. Spectacle sales Visualising and forecasting big time series data Examples of biggish time series 5 Monthly UK sales data from 2000 – 2014 Provided by a large spectacle manufacturer Split by brand (26), gender (3), price range (6), materials (4), and stores (600) About 1 million bottom-level series
  10. 10. 3. Spectacle sales Visualising and forecasting big time series data Examples of biggish time series 5 Monthly UK sales data from 2000 – 2014 Provided by a large spectacle manufacturer Split by brand (26), gender (3), price range (6), materials (4), and stores (600) About 1 million bottom-level series
  11. 11. Outline 1 Examples of biggish time series 2 Time series visualisation 3 BLUF: Best Linear Unbiased Forecasts 4 Application: Australian tourism 5 Fast computation tricks 6 hts package for R 7 References Visualising and forecasting big time series data Time series visualisation 6
  12. 12. Kite diagrams 000 Line graph profile Duplicate & flip around the hori- zontal axis Fill the colour Visualising and forecasting big time series data Time series visualisation 7
  13. 13. Kite diagrams: Victorian tourism 20002010 Holiday 20002010 VFR 20002010 Business 20002010 BAA BAB BAC BBA BCA BCB BCC BDA BDB BDC BDD BDE BDF BEA BEB BEC BED BEE BEF Other BEG Victoria Visualising and forecasting big time series data Time series visualisation 8
  14. 14. Kite diagrams: Victorian tourism Visualising and forecasting big time series data Time series visualisation 8
  15. 15. Kite diagrams: Victorian tourism Visualising and forecasting big time series data Time series visualisation 8
  16. 16. Kite diagrams: Victorian tourism 20002010 Holiday 20002010 VFR 20002010 Business 20002010 BAA BAB BAC BBA BCA BCB BCC BDA BDB BDC BDD BDE BDF BEA BEB BEC BED BEE BEF Other BEG Victoria: scaled Visualising and forecasting big time series data Time series visualisation 8
  17. 17. An STL decomposition STL decomposition of tourism demand for holidays in Peninsula 5.06.07.0 data −0.50.5 seasonal 5.86.16.4 trend −0.40.0 2000 2005 2010 remainder Visualising and forecasting big time series data Time series visualisation 9
  18. 18. Seasonal stacked bar chart Place positive values above the origin while negative values below the origin Map the bar length to the magnitude Encode quarters by colours Visualising and forecasting big time series data Time series visualisation 10
  19. 19. Seasonal stacked bar chart Place positive values above the origin while negative values below the origin Map the bar length to the magnitude Encode quarters by colours −1.0 −0.5 0.0 0.5 1.0 Holiday BAA BABBACBBABCABCBBCCBDABDBBDCBDDBDEBDF BEA BEBBECBEDBEE BEFBEG Regions SeasonalComponent Qtr Q1 Q2 Q3 Q4 Visualising and forecasting big time series data Time series visualisation 10
  20. 20. Seasonal stacked bar chart: VIC Visualising and forecasting big time series data Time series visualisation 11
  21. 21. Corrgram of remainder Visualising and forecasting big time series data Time series visualisation 12 Compute the correlations among the remainder components Render both the sign and magnitude using a colour mapping of two hues Order variables according to the first principal component of the correlations.
  22. 22. Corrgram of remainder: VIC Visualising and forecasting big time series data Time series visualisation 13 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 BEEHolBEFOthBEEOthBDEOthBEBOthBEABusBEFBusBDCOthBACHolBEBBusBEAVisBBAHolBDEHolBABOthBAAVisBAAHolBDCHolBBABusBCBHolBEGBusBDDVisBABVisBDAVisBEAOthBDFHolBEEBusBAAOthBACOthBDAOthBDEBusBCBOthBACBusBEBVisBACVisBCAOthBEFVisBCBVisBEDHolBEGOthBDBHolBABBusBEBHolBDFBusBECHolBCAHolBDBOthBEAHolBDCBusBECVisBDBVisBCCHolBBAVisBABHolBBAOthBCCOthBCBBusBCCVisBEGVisBDDHolBECOthBDCVisBAABusBCCBusBECBusBCAVisBDFVisBEGHolBDDOthBEDOthBEDVisBDDBusBDEVisBEFHolBEEVisBDBBusBDABusBDAHolBCABusBDFOthBEDBus BEEHolBEFOthBEEOthBDEOthBEBOthBEABusBEFBusBDCOthBACHolBEBBusBEAVisBBAHolBDEHolBABOthBAAVisBAAHolBDCHolBBABusBCBHolBEGBusBDDVisBABVisBDAVisBEAOthBDFHolBEEBusBAAOthBACOthBDAOthBDEBusBCBOthBACBusBEBVisBACVisBCAOthBEFVisBCBVisBEDHolBEGOthBDBHolBABBusBEBHolBDFBusBECHolBCAHolBDBOthBEAHolBDCBusBECVisBDBVisBCCHolBBAVisBABHolBBAOthBCCOthBCBBusBCCVisBEGVisBDDHolBECOthBDCVisBAABusBCCBusBECBusBCAVisBDFVisBEGHolBDDOthBEDOthBEDVisBDDBusBDEVisBEFHolBEEVisBDBBusBDABusBDAHolBCABusBDFOthBEDBus
  23. 23. Corrgram of remainder: TAS Visualising and forecasting big time series data Time series visualisation 14 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 FCAHol FBBHol FBAHol FAAHol FCBHol FCAVis FBBVis FAAVis FCBBus FAAOth FCAOth FBBOth FBABus FBAOth FCBVis FCABus FBAVis FCBOth FBBBus FAABus FCAHol FBBHol FBAHol FAAHol FCBHol FCAVis FBBVis FAAVis FCBBus FAAOth FCAOth FBBOth FBABus FBAOth FCBVis FCABus FBAVis FCBOth FBBBus FAABus
  24. 24. Feature analysis Summarize each time series with a feature vector: strength of trend lumpiness (variance of annual variances of remainder) strength of seasonality size of seasonal peak size of seasonal trough ACF1 linearity of trend curvature of trend spectral entropy Do PCA on feature matrix Visualising and forecasting big time series data Time series visualisation 15
  25. 25. Feature analysis Visualising and forecasting big time series data Time series visualisation 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 3031 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 6061 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299300 301 302 303304 lum p season spike entropy fo.acf peak trough linearity curvature −2 0 2 4 −2 0 2 standardized PC1 (43.2% explained var.) standardizedPC2(18.6%explainedvar.)
  26. 26. Feature analysis Visualising and forecasting big time series data Time series visualisation 16 0 200 400 600 800 0 250 500 750 0 100 200 300 400 0 500 1000 1500 BEGBusBCCBusCCAOthCACOth 2000 2005 2010 Time value
  27. 27. Feature analysis Visualising and forecasting big time series data Time series visualisation 16 400 800 1200 1600 250 500 750 1000 500 1000 1500 2000 1000 2000 3000 4000 ADAHolBDCHolBBAHolACAHol 2000 2005 2010 Time value
  28. 28. Feature analysis Visualising and forecasting big time series data Time series visualisation 16 q q q q q q qq q q q qq q q q q q q q q q q q q q q qq qq qq q qq q q q q q q q q q qq qqq q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q qq q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q qq q qq q q q q q q q q q q q q qq q q q q q qq q q q q q q qq q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q qq q q q q q qq q q qq q qqq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q qqq q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q qqq q q q q q q q q q q qq q q qq q q q q q qq q q q qq q q q q q q q q q q q q q q qq qq qq q qq q q q q q q q q q qq qqq q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q qq q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q qq q qq q q q q q q q q q q q q qq q q q q q qq q q q q q q qq q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q qq q q q q q qq q q qq q qqq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q qqq q q q q q q q q q q qq q q qq q q q q q qq q q q qq q q q q q q q q q q q q q q qq qq qq q qq q q q q q q q q q qq qqq q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q qq q qq q q q q q q q q q q q q qq q q q q q qq q q q q q q qq q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q qq q q q q q qq q q qq q qqq q qq q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q qqq q q q q q q q q q q qq q q qq q q q q q qq q q q qq q q q q q q q q q q q q q q qq qq qq q qq q q q q q q q q q qq qqq q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q qq q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q qq q q q q q qq q q qq q qqq q q q q q qq q qq q q q q q q q q q q q q qq q q q q q qq q q q q q q qq q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q qqq q q q q q q q q q q qq q q qq q q q q q qq q q q qq q q q q q q q q q q q q q q qq qq qq q qq q q q q q q q q q qq qqq q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q qq q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q qq q qq q q q q q q q q q q q q qq q q q q q qq q q q q q q qq q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q qq q q q q q qq q q qq q qqq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q qqq q q q q q q q q q q qq q q q q qq q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q qq q qq q q q q q q q q q q q q qq q q q q q qq q q q q q q qq q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q qq q q q q q qq q q qq q qqqq q q q q q qq q q q qq q q q q q q q q q q q q q q qq qq qq q qq q q q q q q q q q qq qqq q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q qqq q q q q q q q q q q qq q q qq q q q q q qq q q q qq q q q q q q q q q q q q q q qq qq qq q qq q q q q q q q q q qq qqq q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q qq q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q qq q qq q q q q q q q q q q q q qq q q q q q qq q q q q q q qq q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q qq q q q q q qq q q qq q qqq q q q q q q q q q q q q q q q q q q q q NSW NT QLD SA TAS VIC WA −2 0 2 4 6 −2 0 2 4 6 −7.5−5.0−2.5 0.0 2.5 5.0−7.5−5.0−2.5 0.0 2.5 5.0−7.5−5.0−2.5 0.0 2.5 5.0 PC1 PC2
  29. 29. Feature analysis Visualising and forecasting big time series data Time series visualisation 16 q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q qq q q q q q q q qq q qq q q q q q q q qq q q q q q q q q q q qq q qq q q q q q q q q q q qqq q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q qq q q q q qq q q q q q q q qq q q q q q q q q q q q q qq q q q q q q qq q qq q q qq q q q qq q q q q qq q q q qq q q q qq q q q q q q qq q q q q q q q q qq q q q q q q q q q q q q q qqq q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q qq q q q q q q q q q q qq q q qq qq q q q q q q q q qq q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q qq q q q qqq q q q q q q q q qq q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q qq q qq q q q qq q q q q q q q q qq qq q qq q qq q q q q q q q q q q qq q q qq qqq qq q q q q q q q q q q q q qq q q q q q q q q q q q q q q q qq q q q q q qq q q qq q qqq qq q qq qq q q q q q q q q q q q q q q q q q q q q q q q q qqq q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q qq q q q q q q q q q q qq q q q qq q q q q q q q q q q qq q q q q q q q q qq q q q q q q q q q qq q q q q q q qq q q q q q q q q q q q q q q q qq q q q qq q q q q q q q q qq q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q qq q q q q q q qq q q q q q q q q qq q qq q qq q q q q qq q q q qq q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qqq q q q q q q q qq q qqq q q q q q q qq q q q q q q q q q q qq q qq q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q qq q q q qq q qq q qqq q q q q q qqq q q q qq q qq qq q q q qq q q q q q qq q qq q q q q q q q q q q q qq q q q q q q q q qqqq q q q q q q q q q q q qq q q q q Bus Hol Oth Vis −2 0 2 4 6 −2 0 2 4 6 −7.5 −5.0 −2.5 0.0 2.5 5.0−7.5 −5.0 −2.5 0.0 2.5 5.0 PC1 PC2
  30. 30. Outline 1 Examples of biggish time series 2 Time series visualisation 3 BLUF: Best Linear Unbiased Forecasts 4 Application: Australian tourism 5 Fast computation tricks 6 hts package for R 7 References Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 17
  31. 31. Hierarchical time series A hierarchical time series is a collection of several time series that are linked together in a hierarchical structure. Total A AA AB AC B BA BB BC C CA CB CC Examples Net labour turnover Tourism by state and region Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 18
  32. 32. Hierarchical time series A hierarchical time series is a collection of several time series that are linked together in a hierarchical structure. Total A AA AB AC B BA BB BC C CA CB CC Examples Net labour turnover Tourism by state and region Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 18
  33. 33. Hierarchical time series A hierarchical time series is a collection of several time series that are linked together in a hierarchical structure. Total A AA AB AC B BA BB BC C CA CB CC Examples Net labour turnover Tourism by state and region Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 18
  34. 34. Hierarchical time series Total A B C Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 19 Yt : observed aggregate of all series at time t. YX,t : observation on series X at time t. Bt : vector of all series at bottom level in time t.
  35. 35. Hierarchical time series Total A B C Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 19 Yt : observed aggregate of all series at time t. YX,t : observation on series X at time t. Bt : vector of all series at bottom level in time t.
  36. 36. Hierarchical time series Total A B C yt = [Yt, YA,t, YB,t, YC,t] =     1 1 1 1 0 0 0 1 0 0 0 1       YA,t YB,t YC,t   Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 19 Yt : observed aggregate of all series at time t. YX,t : observation on series X at time t. Bt : vector of all series at bottom level in time t.
  37. 37. Hierarchical time series Total A B C yt = [Yt, YA,t, YB,t, YC,t] =     1 1 1 1 0 0 0 1 0 0 0 1     S   YA,t YB,t YC,t   Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 19 Yt : observed aggregate of all series at time t. YX,t : observation on series X at time t. Bt : vector of all series at bottom level in time t.
  38. 38. Hierarchical time series Total A B C yt = [Yt, YA,t, YB,t, YC,t] =     1 1 1 1 0 0 0 1 0 0 0 1     S   YA,t YB,t YC,t   Bt Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 19 Yt : observed aggregate of all series at time t. YX,t : observation on series X at time t. Bt : vector of all series at bottom level in time t.
  39. 39. Hierarchical time series Total A B C yt = [Yt, YA,t, YB,t, YC,t] =     1 1 1 1 0 0 0 1 0 0 0 1     S   YA,t YB,t YC,t   Bt yt = SBt Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 19 Yt : observed aggregate of all series at time t. YX,t : observation on series X at time t. Bt : vector of all series at bottom level in time t.
  40. 40. Hierarchical time series Total A AX AY AZ B BX BY BZ C CX CY CZ yt =             Yt YA,t YB,t YC,t YAX,t YAY,t YAZ,t YBX,t YBY,t YBZ,t YCX,t YCY,t YCZ,t             =             1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1             S        YAX,t YAY,t YAZ,t YBX,t YBY,t YBZ,t YCX,t YCY,t YCZ,t        Bt Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 20
  41. 41. Hierarchical time series Total A AX AY AZ B BX BY BZ C CX CY CZ yt =             Yt YA,t YB,t YC,t YAX,t YAY,t YAZ,t YBX,t YBY,t YBZ,t YCX,t YCY,t YCZ,t             =             1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1             S        YAX,t YAY,t YAZ,t YBX,t YBY,t YBZ,t YCX,t YCY,t YCZ,t        Bt Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 20
  42. 42. Hierarchical time series Total A AX AY AZ B BX BY BZ C CX CY CZ yt =             Yt YA,t YB,t YC,t YAX,t YAY,t YAZ,t YBX,t YBY,t YBZ,t YCX,t YCY,t YCZ,t             =             1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1             S        YAX,t YAY,t YAZ,t YBX,t YBY,t YBZ,t YCX,t YCY,t YCZ,t        Bt Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 20 yt = SBt
  43. 43. Forecasting notation Let ˆyn(h) be vector of initial h-step forecasts, made at time n, stacked in same order as yt. (They may not add up.) Reconciled forecasts are of the form: ˜yn(h) = SPˆyn(h) for some matrix P. P extracts and combines base forecasts ˆyn(h) to get bottom-level forecasts. S adds them up Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 21
  44. 44. Forecasting notation Let ˆyn(h) be vector of initial h-step forecasts, made at time n, stacked in same order as yt. (They may not add up.) Reconciled forecasts are of the form: ˜yn(h) = SPˆyn(h) for some matrix P. P extracts and combines base forecasts ˆyn(h) to get bottom-level forecasts. S adds them up Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 21
  45. 45. Forecasting notation Let ˆyn(h) be vector of initial h-step forecasts, made at time n, stacked in same order as yt. (They may not add up.) Reconciled forecasts are of the form: ˜yn(h) = SPˆyn(h) for some matrix P. P extracts and combines base forecasts ˆyn(h) to get bottom-level forecasts. S adds them up Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 21
  46. 46. Forecasting notation Let ˆyn(h) be vector of initial h-step forecasts, made at time n, stacked in same order as yt. (They may not add up.) Reconciled forecasts are of the form: ˜yn(h) = SPˆyn(h) for some matrix P. P extracts and combines base forecasts ˆyn(h) to get bottom-level forecasts. S adds them up Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 21
  47. 47. Forecasting notation Let ˆyn(h) be vector of initial h-step forecasts, made at time n, stacked in same order as yt. (They may not add up.) Reconciled forecasts are of the form: ˜yn(h) = SPˆyn(h) for some matrix P. P extracts and combines base forecasts ˆyn(h) to get bottom-level forecasts. S adds them up Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 21
  48. 48. General properties: bias ˜yn(h) = SPˆyn(h) Assume: base forecasts ˆyn(h) are unbiased: E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn] Let ˆBn(h) be bottom level base forecasts with βn(h) = E[ˆBn(h)|y1, . . . , yn]. Then E[ˆyn(h)] = Sβn(h). We want the revised forecasts to be unbiased: E[˜yn(h)] = SPSβn(h) = Sβn(h). Revised forecasts are unbiased iff SPS = S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 22
  49. 49. General properties: bias ˜yn(h) = SPˆyn(h) Assume: base forecasts ˆyn(h) are unbiased: E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn] Let ˆBn(h) be bottom level base forecasts with βn(h) = E[ˆBn(h)|y1, . . . , yn]. Then E[ˆyn(h)] = Sβn(h). We want the revised forecasts to be unbiased: E[˜yn(h)] = SPSβn(h) = Sβn(h). Revised forecasts are unbiased iff SPS = S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 22
  50. 50. General properties: bias ˜yn(h) = SPˆyn(h) Assume: base forecasts ˆyn(h) are unbiased: E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn] Let ˆBn(h) be bottom level base forecasts with βn(h) = E[ˆBn(h)|y1, . . . , yn]. Then E[ˆyn(h)] = Sβn(h). We want the revised forecasts to be unbiased: E[˜yn(h)] = SPSβn(h) = Sβn(h). Revised forecasts are unbiased iff SPS = S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 22
  51. 51. General properties: bias ˜yn(h) = SPˆyn(h) Assume: base forecasts ˆyn(h) are unbiased: E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn] Let ˆBn(h) be bottom level base forecasts with βn(h) = E[ˆBn(h)|y1, . . . , yn]. Then E[ˆyn(h)] = Sβn(h). We want the revised forecasts to be unbiased: E[˜yn(h)] = SPSβn(h) = Sβn(h). Revised forecasts are unbiased iff SPS = S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 22
  52. 52. General properties: bias ˜yn(h) = SPˆyn(h) Assume: base forecasts ˆyn(h) are unbiased: E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn] Let ˆBn(h) be bottom level base forecasts with βn(h) = E[ˆBn(h)|y1, . . . , yn]. Then E[ˆyn(h)] = Sβn(h). We want the revised forecasts to be unbiased: E[˜yn(h)] = SPSβn(h) = Sβn(h). Revised forecasts are unbiased iff SPS = S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 22
  53. 53. General properties: bias ˜yn(h) = SPˆyn(h) Assume: base forecasts ˆyn(h) are unbiased: E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn] Let ˆBn(h) be bottom level base forecasts with βn(h) = E[ˆBn(h)|y1, . . . , yn]. Then E[ˆyn(h)] = Sβn(h). We want the revised forecasts to be unbiased: E[˜yn(h)] = SPSβn(h) = Sβn(h). Revised forecasts are unbiased iff SPS = S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 22
  54. 54. General properties: bias ˜yn(h) = SPˆyn(h) Assume: base forecasts ˆyn(h) are unbiased: E[ˆyn(h)|y1, . . . , yn] = E[yn+h|y1, . . . , yn] Let ˆBn(h) be bottom level base forecasts with βn(h) = E[ˆBn(h)|y1, . . . , yn]. Then E[ˆyn(h)] = Sβn(h). We want the revised forecasts to be unbiased: E[˜yn(h)] = SPSβn(h) = Sβn(h). Revised forecasts are unbiased iff SPS = S. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 22
  55. 55. General properties: variance ˜yn(h) = SPˆyn(h) Let variance of base forecasts ˆyn(h) be given by Σh = Var[ˆyn(h)|y1, . . . , yn] Then the variance of the revised forecasts is given by Var[˜yn(h)|y1, . . . , yn] = SPΣhP S . Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 23
  56. 56. General properties: variance ˜yn(h) = SPˆyn(h) Let variance of base forecasts ˆyn(h) be given by Σh = Var[ˆyn(h)|y1, . . . , yn] Then the variance of the revised forecasts is given by Var[˜yn(h)|y1, . . . , yn] = SPΣhP S . Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 23
  57. 57. General properties: variance ˜yn(h) = SPˆyn(h) Let variance of base forecasts ˆyn(h) be given by Σh = Var[ˆyn(h)|y1, . . . , yn] Then the variance of the revised forecasts is given by Var[˜yn(h)|y1, . . . , yn] = SPΣhP S . Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 23
  58. 58. BLUF via trace minimization Theorem For any P satisfying SPS = S, then min P = trace[SPΣhP S ] has solution P = (S Σ† hS)−1 S Σ† h. Σ† h is generalized inverse of Σh. ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h) Revised forecasts Base forecasts Equivalent to GLS estimate of regression ˆyn(h) = Sβn(h) + εh where ε ∼ N(0, Σh). Problem: Σh hard to estimate. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 24
  59. 59. BLUF via trace minimization Theorem For any P satisfying SPS = S, then min P = trace[SPΣhP S ] has solution P = (S Σ† hS)−1 S Σ† h. Σ† h is generalized inverse of Σh. ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h) Revised forecasts Base forecasts Equivalent to GLS estimate of regression ˆyn(h) = Sβn(h) + εh where ε ∼ N(0, Σh). Problem: Σh hard to estimate. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 24
  60. 60. BLUF via trace minimization Theorem For any P satisfying SPS = S, then min P = trace[SPΣhP S ] has solution P = (S Σ† hS)−1 S Σ† h. Σ† h is generalized inverse of Σh. ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h) Revised forecasts Base forecasts Equivalent to GLS estimate of regression ˆyn(h) = Sβn(h) + εh where ε ∼ N(0, Σh). Problem: Σh hard to estimate. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 24
  61. 61. BLUF via trace minimization Theorem For any P satisfying SPS = S, then min P = trace[SPΣhP S ] has solution P = (S Σ† hS)−1 S Σ† h. Σ† h is generalized inverse of Σh. ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h) Revised forecasts Base forecasts Equivalent to GLS estimate of regression ˆyn(h) = Sβn(h) + εh where ε ∼ N(0, Σh). Problem: Σh hard to estimate. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 24
  62. 62. BLUF via trace minimization Theorem For any P satisfying SPS = S, then min P = trace[SPΣhP S ] has solution P = (S Σ† hS)−1 S Σ† h. Σ† h is generalized inverse of Σh. ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h) Revised forecasts Base forecasts Equivalent to GLS estimate of regression ˆyn(h) = Sβn(h) + εh where ε ∼ N(0, Σh). Problem: Σh hard to estimate. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 24
  63. 63. BLUF via trace minimization Theorem For any P satisfying SPS = S, then min P = trace[SPΣhP S ] has solution P = (S Σ† hS)−1 S Σ† h. Σ† h is generalized inverse of Σh. ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h) Revised forecasts Base forecasts Equivalent to GLS estimate of regression ˆyn(h) = Sβn(h) + εh where ε ∼ N(0, Σh). Problem: Σh hard to estimate. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 24
  64. 64. Optimal combination forecasts Revised forecasts Base forecasts Solution 1: OLS Assume εh ≈ SεB,h where εB,h is the forecast error at bottom level. Then Σh ≈ SΩhS where Ωh = Var(εB,h). If Moore-Penrose generalized inverse used, then (S Σ† hS)−1 S Σ† h = (S S)−1 S . ˜yn(h) = S(S S)−1 S ˆyn(h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 25 ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h)
  65. 65. Optimal combination forecasts Revised forecasts Base forecasts Solution 1: OLS Assume εh ≈ SεB,h where εB,h is the forecast error at bottom level. Then Σh ≈ SΩhS where Ωh = Var(εB,h). If Moore-Penrose generalized inverse used, then (S Σ† hS)−1 S Σ† h = (S S)−1 S . ˜yn(h) = S(S S)−1 S ˆyn(h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 25 ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h)
  66. 66. Optimal combination forecasts Revised forecasts Base forecasts Solution 1: OLS Assume εh ≈ SεB,h where εB,h is the forecast error at bottom level. Then Σh ≈ SΩhS where Ωh = Var(εB,h). If Moore-Penrose generalized inverse used, then (S Σ† hS)−1 S Σ† h = (S S)−1 S . ˜yn(h) = S(S S)−1 S ˆyn(h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 25 ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h)
  67. 67. Optimal combination forecasts Revised forecasts Base forecasts Solution 1: OLS Assume εh ≈ SεB,h where εB,h is the forecast error at bottom level. Then Σh ≈ SΩhS where Ωh = Var(εB,h). If Moore-Penrose generalized inverse used, then (S Σ† hS)−1 S Σ† h = (S S)−1 S . ˜yn(h) = S(S S)−1 S ˆyn(h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 25 ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h)
  68. 68. Optimal combination forecasts Revised forecasts Base forecasts Solution 1: OLS Assume εh ≈ SεB,h where εB,h is the forecast error at bottom level. Then Σh ≈ SΩhS where Ωh = Var(εB,h). If Moore-Penrose generalized inverse used, then (S Σ† hS)−1 S Σ† h = (S S)−1 S . ˜yn(h) = S(S S)−1 S ˆyn(h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 25 ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h)
  69. 69. Optimal combination forecasts Revised forecasts Base forecasts Solution 1: OLS Assume εh ≈ SεB,h where εB,h is the forecast error at bottom level. Then Σh ≈ SΩhS where Ωh = Var(εB,h). If Moore-Penrose generalized inverse used, then (S Σ† hS)−1 S Σ† h = (S S)−1 S . ˜yn(h) = S(S S)−1 S ˆyn(h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 25 ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h)
  70. 70. Optimal combination forecasts Revised forecasts Base forecasts Solution 2: WLS Suppose we approximate Σ1 by its diagonal. Easy to estimate, and places weight where we have best forecasts. Empirically, it gives better forecasts than other available methods. ˜yn(h) = S(S ΛS)−1 S Λˆyn(h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26 ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h)
  71. 71. Optimal combination forecasts Revised forecasts Base forecasts Solution 2: WLS Suppose we approximate Σ1 by its diagonal. Easy to estimate, and places weight where we have best forecasts. Empirically, it gives better forecasts than other available methods. ˜yn(h) = S(S ΛS)−1 S Λˆyn(h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26 ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h)
  72. 72. Optimal combination forecasts Revised forecasts Base forecasts Solution 2: WLS Suppose we approximate Σ1 by its diagonal. Easy to estimate, and places weight where we have best forecasts. Empirically, it gives better forecasts than other available methods. ˜yn(h) = S(S ΛS)−1 S Λˆyn(h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26 ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h)
  73. 73. Optimal combination forecasts Revised forecasts Base forecasts Solution 2: WLS Suppose we approximate Σ1 by its diagonal. Easy to estimate, and places weight where we have best forecasts. Empirically, it gives better forecasts than other available methods. ˜yn(h) = S(S ΛS)−1 S Λˆyn(h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26 ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h)
  74. 74. Optimal combination forecasts Revised forecasts Base forecasts Solution 2: WLS Suppose we approximate Σ1 by its diagonal. Easy to estimate, and places weight where we have best forecasts. Empirically, it gives better forecasts than other available methods. ˜yn(h) = S(S ΛS)−1 S Λˆyn(h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26 ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h)
  75. 75. Optimal combination forecasts Revised forecasts Base forecasts Solution 2: WLS Suppose we approximate Σ1 by its diagonal. Easy to estimate, and places weight where we have best forecasts. Empirically, it gives better forecasts than other available methods. ˜yn(h) = S(S ΛS)−1 S Λˆyn(h) Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 26 ˜yn(h) = S(S Σ† hS)−1 S Σ† hˆyn(h)
  76. 76. Challenges Computational difficulties in big hierarchies due to size of the S matrix and singular behavior of (S ΛS). Loss of information in ignoring covariance matrix in computing point forecasts. Still need to estimate covariance matrix to produce prediction intervals. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 27 ˜yn(h) = S(S ΛS)−1 S Λˆyn(h)
  77. 77. Challenges Computational difficulties in big hierarchies due to size of the S matrix and singular behavior of (S ΛS). Loss of information in ignoring covariance matrix in computing point forecasts. Still need to estimate covariance matrix to produce prediction intervals. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 27 ˜yn(h) = S(S ΛS)−1 S Λˆyn(h)
  78. 78. Challenges Computational difficulties in big hierarchies due to size of the S matrix and singular behavior of (S ΛS). Loss of information in ignoring covariance matrix in computing point forecasts. Still need to estimate covariance matrix to produce prediction intervals. Visualising and forecasting big time series data BLUF: Best Linear Unbiased Forecasts 27 ˜yn(h) = S(S ΛS)−1 S Λˆyn(h)
  79. 79. Outline 1 Examples of biggish time series 2 Time series visualisation 3 BLUF: Best Linear Unbiased Forecasts 4 Application: Australian tourism 5 Fast computation tricks 6 hts package for R 7 References Visualising and forecasting big time series data Application: Australian tourism 28
  80. 80. Australian tourism Visualising and forecasting big time series data Application: Australian tourism 29
  81. 81. Australian tourism Visualising and forecasting big time series data Application: Australian tourism 29 Hierarchy: States (7) Zones (27) Regions (82)
  82. 82. Australian tourism Visualising and forecasting big time series data Application: Australian tourism 29 Hierarchy: States (7) Zones (27) Regions (82) Base forecasts ETS (exponential smoothing) models
  83. 83. Base forecasts Visualising and forecasting big time series data Application: Australian tourism 30 Domestic tourism forecasts: Total Year Visitornights 1998 2000 2002 2004 2006 2008 600006500070000750008000085000
  84. 84. Base forecasts Visualising and forecasting big time series data Application: Australian tourism 30 Domestic tourism forecasts: NSW Year Visitornights 1998 2000 2002 2004 2006 2008 18000220002600030000
  85. 85. Base forecasts Visualising and forecasting big time series data Application: Australian tourism 30 Domestic tourism forecasts: VIC Year Visitornights 1998 2000 2002 2004 2006 2008 1000012000140001600018000
  86. 86. Base forecasts Visualising and forecasting big time series data Application: Australian tourism 30 Domestic tourism forecasts: Nth.Coast.NSW Year Visitornights 1998 2000 2002 2004 2006 2008 50006000700080009000
  87. 87. Base forecasts Visualising and forecasting big time series data Application: Australian tourism 30 Domestic tourism forecasts: Metro.QLD Year Visitornights 1998 2000 2002 2004 2006 2008 800090001100013000
  88. 88. Base forecasts Visualising and forecasting big time series data Application: Australian tourism 30 Domestic tourism forecasts: Sth.WA Year Visitornights 1998 2000 2002 2004 2006 2008 400600800100012001400
  89. 89. Base forecasts Visualising and forecasting big time series data Application: Australian tourism 30 Domestic tourism forecasts: X201.Melbourne Year Visitornights 1998 2000 2002 2004 2006 2008 40004500500055006000
  90. 90. Base forecasts Visualising and forecasting big time series data Application: Australian tourism 30 Domestic tourism forecasts: X402.Murraylands Year Visitornights 1998 2000 2002 2004 2006 2008 0100200300
  91. 91. Base forecasts Visualising and forecasting big time series data Application: Australian tourism 30 Domestic tourism forecasts: X809.Daly Year Visitornights 1998 2000 2002 2004 2006 2008 020406080100
  92. 92. Reconciled forecasts Visualising and forecasting big time series data Application: Australian tourism 31 Total 2000 2005 2010 650008000095000
  93. 93. Reconciled forecasts Visualising and forecasting big time series data Application: Australian tourism 31 NSW 2000 2005 2010 180002400030000 VIC 2000 2005 2010 100001400018000 QLD 2000 2005 2010 1400020000 Other 2000 2005 2010 1800024000
  94. 94. Reconciled forecasts Visualising and forecasting big time series data Application: Australian tourism 31 Sydney 2000 2005 2010 40007000 OtherNSW 2000 2005 2010 1400022000 Melbourne 2000 2005 2010 40005000 OtherVIC 2000 2005 2010 600012000 GCandBrisbane 2000 2005 2010 60009000 OtherQLD 2000 2005 2010 600012000 Capitalcities 2000 2005 2010 1400020000 Other 2000 2005 2010 55007500
  95. 95. Forecast evaluation Select models using all observations; Re-estimate models using first 12 observations and generate 1- to 8-step-ahead forecasts; Increase sample size one observation at a time, re-estimate models, generate forecasts until the end of the sample; In total 24 1-step-ahead, 23 2-steps-ahead, up to 17 8-steps-ahead for forecast evaluation. Visualising and forecasting big time series data Application: Australian tourism 32
  96. 96. Forecast evaluation Select models using all observations; Re-estimate models using first 12 observations and generate 1- to 8-step-ahead forecasts; Increase sample size one observation at a time, re-estimate models, generate forecasts until the end of the sample; In total 24 1-step-ahead, 23 2-steps-ahead, up to 17 8-steps-ahead for forecast evaluation. Visualising and forecasting big time series data Application: Australian tourism 32
  97. 97. Forecast evaluation Select models using all observations; Re-estimate models using first 12 observations and generate 1- to 8-step-ahead forecasts; Increase sample size one observation at a time, re-estimate models, generate forecasts until the end of the sample; In total 24 1-step-ahead, 23 2-steps-ahead, up to 17 8-steps-ahead for forecast evaluation. Visualising and forecasting big time series data Application: Australian tourism 32
  98. 98. Forecast evaluation Select models using all observations; Re-estimate models using first 12 observations and generate 1- to 8-step-ahead forecasts; Increase sample size one observation at a time, re-estimate models, generate forecasts until the end of the sample; In total 24 1-step-ahead, 23 2-steps-ahead, up to 17 8-steps-ahead for forecast evaluation. Visualising and forecasting big time series data Application: Australian tourism 32
  99. 99. Hierarchy: states, zones, regions MAPE h = 1 h = 2 h = 4 h = 6 h = 8 Average Top Level: Australia Bottom-up 3.79 3.58 4.01 4.55 4.24 4.06 OLS 3.83 3.66 3.88 4.19 4.25 3.94 WLS 3.68 3.56 3.97 4.57 4.25 4.04 Level: States Bottom-up 10.70 10.52 10.85 11.46 11.27 11.03 OLS 11.07 10.58 11.13 11.62 12.21 11.35 WLS 10.44 10.17 10.47 10.97 10.98 10.67 Level: Zones Bottom-up 14.99 14.97 14.98 15.69 15.65 15.32 OLS 15.16 15.06 15.27 15.74 16.15 15.48 WLS 14.63 14.62 14.68 15.17 15.25 14.94 Bottom Level: Regions Bottom-up 33.12 32.54 32.26 33.74 33.96 33.18 OLS 35.89 33.86 34.26 36.06 37.49 35.43 WLS 31.68 31.22 31.08 32.41 32.77 31.89 Visualising and forecasting big time series data Application: Australian tourism 33
  100. 100. Outline 1 Examples of biggish time series 2 Time series visualisation 3 BLUF: Best Linear Unbiased Forecasts 4 Application: Australian tourism 5 Fast computation tricks 6 hts package for R 7 References Visualising and forecasting big time series data Fast computation tricks 34
  101. 101. Fast computation: hierarchical data Total A AX AY AZ B BX BY BZ C CX CY CZ yt =             Yt YA,t YB,t YC,t YAX,t YAY,t YAZ,t YBX,t YBY,t YBZ,t YCX,t YCY,t YCZ,t             =             1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1             S        YAX,t YAY,t YAZ,t YBX,t YBY,t YBZ,t YCX,t YCY,t YCZ,t        Bt Visualising and forecasting big time series data Fast computation tricks 35 yt = SBt
  102. 102. Fast computation: hierarchical data Total A AX AY AZ B BX BY BZ C CX CY CZ yt =             Yt YA,t YAX,t YAY,t YAZ,t YB,t YBX,t YBY,t YBZ,t YC,t YCX,t YCY,t YCZ,t             =             1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1             S        YAX,t YAY,t YAZ,t YBX,t YBY,t YBZ,t YCX,t YCY,t YCZ,t        Bt Visualising and forecasting big time series data Fast computation tricks 36 yt = SBt
  103. 103. Fast computation: hierarchies Think of the hierarchy as a tree of trees: Total T1 T2 . . . TK Then the summing matrix contains k smaller summing matrices: S =       1n1 1n2 · · · 1nK S1 0 · · · 0 0 S2 · · · 0 ... ... ... ... 0 0 · · · SK       where 1n is an n-vector of ones and tree Ti has ni terminal nodes. Visualising and forecasting big time series data Fast computation tricks 37
  104. 104. Fast computation: hierarchies Think of the hierarchy as a tree of trees: Total T1 T2 . . . TK Then the summing matrix contains k smaller summing matrices: S =       1n1 1n2 · · · 1nK S1 0 · · · 0 0 S2 · · · 0 ... ... ... ... 0 0 · · · SK       where 1n is an n-vector of ones and tree Ti has ni terminal nodes. Visualising and forecasting big time series data Fast computation tricks 37
  105. 105. Fast computation: hierarchies SΛS =     S1Λ1S1 0 · · · 0 0 S2Λ2S2 · · · 0 ... ... ... ... 0 0 · · · SKΛKSK    +λ0 Jn λ0 is the top left element of Λ; Λk is a block of Λ, corresponding to tree Tk; Jn is a matrix of ones; n = k nk. Now apply the Sherman-Morrison formula . . . Visualising and forecasting big time series data Fast computation tricks 38
  106. 106. Fast computation: hierarchies SΛS =     S1Λ1S1 0 · · · 0 0 S2Λ2S2 · · · 0 ... ... ... ... 0 0 · · · SKΛKSK    +λ0 Jn λ0 is the top left element of Λ; Λk is a block of Λ, corresponding to tree Tk; Jn is a matrix of ones; n = k nk. Now apply the Sherman-Morrison formula . . . Visualising and forecasting big time series data Fast computation tricks 38
  107. 107. Fast computation: hierarchies (SΛS)−1 =      (S1Λ1S1)−1 0 · · · 0 0 (S2Λ2S2)−1 · · · 0 ... ... ... ... 0 0 · · · (SKΛKSK)−1      −cS0 S0 can be partitioned into K2 blocks, with the (k, ) block (of dimension nk × n ) being (SkΛkSk)−1 Jnk,n (S Λ S )−1 Jnk,n is a nk × n matrix of ones. c−1 = λ−1 0 + k 1nk (SkΛkSk)−1 1nk . Each SkΛkSk can be inverted similarly. SΛy can also be computed recursively. Visualising and forecasting big time series data Fast computation tricks 39
  108. 108. Fast computation: hierarchies (SΛS)−1 =      (S1Λ1S1)−1 0 · · · 0 0 (S2Λ2S2)−1 · · · 0 ... ... ... ... 0 0 · · · (SKΛKSK)−1      −cS0 S0 can be partitioned into K2 blocks, with the (k, ) block (of dimension nk × n ) being (SkΛkSk)−1 Jnk,n (S Λ S )−1 Jnk,n is a nk × n matrix of ones. c−1 = λ−1 0 + k 1nk (SkΛkSk)−1 1nk . Each SkΛkSk can be inverted similarly. SΛy can also be computed recursively. Visualising and forecasting big time series data Fast computation tricks 39 The recursive calculations can be done in such a way that we never store any of the large matrices involved.
  109. 109. Fast computation A similar algorithm has been developed for grouped time series with two groups. When the time series are not strictly hierarchical and have more than two grouping variables: Use sparse matrix storage and arithmetic. Use iterative approximation for inverting large sparse matrices. Paige & Saunders (1982) ACM Trans. Math. Software Visualising and forecasting big time series data Fast computation tricks 40
  110. 110. Fast computation A similar algorithm has been developed for grouped time series with two groups. When the time series are not strictly hierarchical and have more than two grouping variables: Use sparse matrix storage and arithmetic. Use iterative approximation for inverting large sparse matrices. Paige & Saunders (1982) ACM Trans. Math. Software Visualising and forecasting big time series data Fast computation tricks 40
  111. 111. Fast computation A similar algorithm has been developed for grouped time series with two groups. When the time series are not strictly hierarchical and have more than two grouping variables: Use sparse matrix storage and arithmetic. Use iterative approximation for inverting large sparse matrices. Paige & Saunders (1982) ACM Trans. Math. Software Visualising and forecasting big time series data Fast computation tricks 40
  112. 112. Outline 1 Examples of biggish time series 2 Time series visualisation 3 BLUF: Best Linear Unbiased Forecasts 4 Application: Australian tourism 5 Fast computation tricks 6 hts package for R 7 References Visualising and forecasting big time series data hts package for R 41
  113. 113. hts package for R Visualising and forecasting big time series data hts package for R 42 hts: Hierarchical and grouped time series Methods for analysing and forecasting hierarchical and grouped time series Version: 4.5 Depends: forecast (≥ 5.0), SparseM Imports: parallel, utils Published: 2014-12-09 Author: Rob J Hyndman, Earo Wang and Alan Lee Maintainer: Rob J Hyndman <Rob.Hyndman at monash.edu> BugReports: https://github.com/robjhyndman/hts/issues License: GPL (≥ 2)
  114. 114. Example using R library(hts) # bts is a matrix containing the bottom level time series # nodes describes the hierarchical structure y <- hts(bts, nodes=list(2, c(3,2))) Visualising and forecasting big time series data hts package for R 43
  115. 115. Example using R library(hts) # bts is a matrix containing the bottom level time series # nodes describes the hierarchical structure y <- hts(bts, nodes=list(2, c(3,2))) Visualising and forecasting big time series data hts package for R 43 Total A AX AY AZ B BX BY
  116. 116. Example using R library(hts) # bts is a matrix containing the bottom level time series # nodes describes the hierarchical structure y <- hts(bts, nodes=list(2, c(3,2))) # Forecast 10-step-ahead using WLS combination method # ETS used for each series by default fc <- forecast(y, h=10) Visualising and forecasting big time series data hts package for R 44
  117. 117. forecast.gts function Usage forecast(object, h, method = c("comb", "bu", "mo", "tdgsf", "tdgsa", "tdfp"), fmethod = c("ets", "rw", "arima"), weights = c("sd", "none", "nseries"), positive = FALSE, parallel = FALSE, num.cores = 2, ...) Arguments object Hierarchical time series object of class gts. h Forecast horizon method Method for distributing forecasts within the hierarchy. fmethod Forecasting method to use positive If TRUE, forecasts are forced to be strictly positive weights Weights used for "optimal combination" method. When weights = "sd", it takes account of the standard deviation of forecasts. parallel If TRUE, allow parallel processing num.cores If parallel = TRUE, specify how many cores are going to be used Visualising and forecasting big time series data hts package for R 45
  118. 118. Outline 1 Examples of biggish time series 2 Time series visualisation 3 BLUF: Best Linear Unbiased Forecasts 4 Application: Australian tourism 5 Fast computation tricks 6 hts package for R 7 References Visualising and forecasting big time series data References 46
  119. 119. References RJ Hyndman, RA Ahmed, G Athanasopoulos, and HL Shang (2011). “Optimal combination forecasts for hierarchical time series”. Computational statistics & data analysis 55(9), 2579–2589. RJ Hyndman, AJ Lee, and E Wang (2014). Fast computation of reconciled forecasts for hierarchical and grouped time series. Working paper 17/14. Department of Econometrics & Business Statistics, Monash University RJ Hyndman, AJ Lee, and E Wang (2014). hts: Hierarchical and grouped time series. cran.r-project.org/package=hts. RJ Hyndman and G Athanasopoulos (2014). Forecasting: principles and practice. OTexts. OTexts.org/fpp/. Visualising and forecasting big time series data References 47
  120. 120. References RJ Hyndman, RA Ahmed, G Athanasopoulos, and HL Shang (2011). “Optimal combination forecasts for hierarchical time series”. Computational statistics & data analysis 55(9), 2579–2589. RJ Hyndman, AJ Lee, and E Wang (2014). Fast computation of reconciled forecasts for hierarchical and grouped time series. Working paper 17/14. Department of Econometrics & Business Statistics, Monash University RJ Hyndman, AJ Lee, and E Wang (2014). hts: Hierarchical and grouped time series. cran.r-project.org/package=hts. RJ Hyndman and G Athanasopoulos (2014). Forecasting: principles and practice. OTexts. OTexts.org/fpp/. Visualising and forecasting big time series data References 47 ¯ Papers and R code: robjhyndman.com ¯ Email: Rob.Hyndman@monash.edu

×