Your Hive honeymoon can be cut short if you don't take the necessary precautions. In this talk I'll share my experience with Hive in the last 3 years (in Elastic MapReduce and Cloudera CDH3), describing what I got wrong the first time around, and what eventually saved the day. I've used Hive in environments with a number of events ranging from a few million to a few billion a day, so hopefully there'll be something for everyone.

1. My life as a
beekeeper
@89clouds

2. Who am I?
Pedro Figueiredo (pfig@89clouds.com)
Hadoop et al
SocialFacebook games, media (TV,
publishing)
Elastic MapReduce, Cloudera
NoSQL, as in “Not a SQL guy”

3. The problem with
Hive
It looks like SQL

4. No, seriously
SELECT
CONCAT(vishi,vislo),
SUM(
CASE WHEN searchengine = 'google'
THEN 1
ELSE 0
END
) AS google_searches
FROM omniture
WHERE
year(hittime) = 2011 AND
month(hittime) = 8 AND
is_search = 'Y'
GROUP BY CONCAT(vishi,vislo);

5. “It’s just like
Oracle!”
Analysts will be very happy
At least until they join with that 30
billion-record table
Pro tip: explain MapReduce and then
MAPJOIN
set
hive.mapjoin.smalltable.filesize=xxx;

6. Your first interview
question
“Explain the difference
between CREATE TABLE and
CREATE EXTERNAL TABLE”

7. Dynamic partitions
Partitions are the poor person’s
indexes
Unstructured data is full of surprises
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.dynamic.partition=true;
set hive.exec.max.dynamic.partitions=100000;
set hive.exec.max.dynamic.partitions.pernode=100000;
Plan your partitions ahead

8. Multi-vitamins
You can minimise input scans by using
multi-table INSERTs:
FROM input
INSERT INTO TABLE output1 SELECT foo
INSERT INTO TABLE output2 SELECT bar;

9. Persistence, do you
speak it?
External Hive metastore
Avoid the pain of cluster set up
Use an RDS metastore if on AWS, RDBMS
otherwise.
10GB will get you a long way, this
thing is tiny

10. Now you have 2
problems
Regular expressions are great, if
you’re using a real programming
language.
WHERE foo RLIKE ‘(a|b|c)’ will hurt
WHERE foo=‘a’ OR foo=‘b’ OR foo=‘c’
Generate these statements, if needs
be, it will pay off.

11. Avro
Serialisation framework (think
Thrift/Protocol Buffers).
Avro container files are
SequenceFile-like, splittable.
Support for snappy built-in.
If using the LinkedIn SerDe, the
table creation syntax changes.

12. Avro
CREATE EXTERNAL TABLE IF NOT EXISTS mytable
PARTITIONED BY (ds STRING)
ROW FORMAT SERDE
'com.linkedin.haivvreo.AvroSerDe'
WITH SERDEPROPERTIES ('schema.url'='hdfs:///user/
hadoop/avro/myschema.avsc')
STORED AS
INPUTFORMAT
'com.linkedin.haivvreo.AvroContainerInputFormat'
OUTPUTFORMAT
'com.linkedin.haivvreo.AvroContainerOutputFormat'
LOCATION '/data/mytable'
;

13. MAKE! MONEY! FAST!
Use spot instances in EMR
Usually stick around until America
wakes up
Brilliant for worker nodes

14. Bag of tricks
set hive.optimize.s3.query=true;
set hive.cli.print.header=true;
set hive.exec.max.created.files=xxx;
set mapred.reduce.tasks=xxx;
hive.exec.compress.intermediate=true;
hive.exec.parallel=true;

15. Bag of tricks
set hive.optimize.s3.query=true;
set hive.cli.print.header=true;
set hive.exec.max.created.files=xxx;
set mapred.reduce.tasks=xxx;
hive.exec.compress.intermediate=true;
hive.exec.parallel=true;

16. Bag of tricks
set hive.optimize.s3.query=true;
set hive.cli.print.header=true;
set hive.exec.max.created.files=xxx;
set mapred.reduce.tasks=xxx;
hive.exec.compress.intermediate=true;
hive.exec.parallel=true;

17. Bag of tricks
set hive.optimize.s3.query=true;
set hive.cli.print.header=true;
set hive.exec.max.created.files=xxx;
set mapred.reduce.tasks=xxx;
hive.exec.compress.intermediate=true;
hive.exec.parallel=true;

18. Bag of tricks
set hive.optimize.s3.query=true;
set hive.cli.print.header=true;
set hive.exec.max.created.files=xxx;
set mapred.reduce.tasks=xxx;
hive.exec.compress.intermediate=true;
hive.exec.parallel=true;

19. Bag of tricks
set hive.optimize.s3.query=true;
set hive.cli.print.header=true;
set hive.exec.max.created.files=xxx;
set mapred.reduce.tasks=xxx;
hive.exec.compress.intermediate=true;
hive.exec.parallel=true;

20. To be or not to be
“Consider a traditional RDBMS”
At what size should we do this?
Hive is not an end, it’s the means
Data on HDFS/S3 is simply available,
not “available to Hive”
Hive isn’t suitable for near real
time

21. Hive != MapReduce
Don’t use Hive instead of Native/
Streaming
“I know, I’ll just stream this bit
through a shell script!”
Imo, Hive excels at analysis and
aggregation, so use it for that

22. Thank you
Fred Easey (@poppa_f)
Peter Hanlon

23. Questions?
pfig@89clouds.com
@pfig / @89clouds
http://89clouds.com/