NOSQL also means RDF stores: an Android case study

NoSQL also means
RDF stores:
an Android case study
Fabrizio Giudici, Senior Java Architect
Tidalwave s.a.s - fabrizio.giudici@tidalwave.it

Agenda

NoSQL also means RDF stores: an Android case study 2

Agenda

• Meaning and reasons for NoSQL


Agenda

• Introducing the blueBill (Mobile) project


Agenda

• What’s a RDF store? Why a RDF store?


Agenda

• RDF open solutions in Java: OpenSesame, Elmo


Agenda

• RDF design issues


Agenda

• Running an RDF store on Android


Agenda

• Running an RDF store on Android
• Conclusions


About the speaker


About the speaker

• Senior Software Architect, Mentor, Technical Writer
• Fourteen years of Java Experience (JSE, JEE, JME, etc...)
• Sun Consultant since 1998, Oracle consultant since 2010
• Author of a number of open source projects
• Speaker at JavaOne, Devoxx, Jazoon, JAX Italia and other events
• Member of the NetBeans Dream Team
• Co-leader of JUG Milano
• Java.Net blogger at http://www.java.net/blogs/fabriziogiudici


NoSQL? What?


NoSQL? What?

• SQL: query language, also refers to “relational model” (RDBM)


NoSQL? What?

• SQL/RDBM are the de-facto standards for ACID persistence


NoSQL? What?

• Tired of that?


NoSQL? What?

• Tired of that?
• OO impedance


NoSQL? What?

• Tired of that?
• OO impedance
• Scalability, elastic scalability (performance)


NoSQL? What?

• Tired of that?
• OO impedance
• Flexibility (data schema)


NoSQL? What?

• Tired of that?
• OO impedance
• Distributed data model, distributed computation


NoSQL? What?

• Tired of that?
• OO impedance
• Distributed data model, distributed computation
• etc


NoSQL


NoSQL

• Interpretations


NoSQL

• Interpretations
• #1: No SQL (at all)


NoSQL

• Interpretations
• #2: Not Only SQL (maybe on top of it)


NoSQL

• Interpretations
• ACID or not?


NoSQL
Five advantages of NoSQL
• Interpretations scaling
1. Elastic
2. Big data
• #1: No SQL Goodbye DBAs (see you later?)
3.
(at all)
4. Economics
• #2: Not Only SQL data models top of
5. Flexible (maybe on it)
• ACID orFive challenges of NoSQL
not?
1. Maturity
2. Support
3. Analytics and business intelligence
4. Administration
5. Expertise

Guy Harrison


NoSQL

• Interpretations
• ACID or not?

• Examples


NoSQL

• Interpretations
• ACID or not?

• Examples
• BigTable (Google), Cassandra (Facebook), Dynamo (Amazon), ...


NoSQL

• Interpretations
• ACID or not?

• Examples
• RDF stores


NoSQL

• Interpretations
• ACID or not?

• Examples
• RDF stores

• This talk is only about the latter topic


Introducing blueBill (Mobile)



• Take ﬁeld notes of bird observations



• Learn about birds



• Aggregating many different knowledge sources




• Collaborative, open, community-oriented




• Subtle requirements




• Complex data structure




• Complex data structure

• Most advanced mobile version currently running on Android


Complex data structure


• Checklists / Taxonomies


• List of species in a given geographic area


• Compiled and vetted by ornithologist and birdwatcher associations


• Deﬁne ofﬁcial common names (with variants)


• Continuously evolving



• Taxonomies



• Taxonomies
• Scientiﬁc names, hierarchical



• Taxonomies
• Not universal consensus



• Taxonomies
• Not universal consensus
• Split & lump

What’s RDF?


What’s RDF?

• RDF is the core of the Semantic Web family of technologies


What’s RDF?

• Semantic Web: give semantic features to the web


What’s RDF?


• Semantic technologies != Semantic Web


What’s RDF?


• They can be used even when the web doesn’t matter


What’s RDF?


• Just a better way to model data


What’s RDF?


• Just a better way to model data
• RDF can be useful to you even though you don’t endorse the
Semantic Web model


RDF principles


RDF principles

• Data are modeled with a single “schema”


RDF principles

• subject - predicate - object (statement, triple)


RDF principles


• Formal ways to represent them


RDF principles


• URNs, URIs, URLs


RDF principles


• namespaces


RDF principles


• namespaces

• RDF != XML


RDF principles


• namespaces

• RDF != XML
• RDF is a model


RDF principles


• namespaces

• RDF != XML
• RDF is a model
• RDF-XML is just one of the many ways to serialize RDF data


Ontologies

• Standard “vocabularies” of terms with agreed-on semantics
• Identiﬁed by their own namespaces
• Can be mixed together
• Semantic equivalence
• Declaring that two concepts from two different ontologies are the same thing
• Can be enforced locally


RDF examples

• Show some examples


What’s a RDF store?



• A software facility for managing data modeled in RDF



• Persistent storage



• CRUD operations



• CRUD operations
• SPARQL and other query languages



• CRUD operations

• Indipendent implementation, or sitting upon a SQL database



• CRUD operations

• Transactions, in the standard way



• CRUD operations


• Typically suitable for distributed, web-oriented queries



• CRUD operations


• Typically suitable for distributed, web-oriented queries
• Not necessarily relevant for one’s purposes


Why a RDF store?


Why a RDF store?

• Most important point: no schema constraints


Why a RDF store?

• “It’s easier to add a row in a table than changing a schema”


Why a RDF store?


• Perfectly suitable for data models that:


Why a RDF store?


• have got a “fuzzy” schema, initially or forever


Why a RDF store?


• are subject to a (strong) iterative development


Why a RDF store?


• need to be integrated with other data models


Why a RDF store?


• need to be integrated with other data models

• The storage model is the same thing as the serialization model


Designing RDF


Designing RDF

• RDF has got its own design rules


Designing RDF

• Validation rules, best practices, etc...


Designing RDF


• Not necessarily easy, but may be incremental


Designing RDF


• Not relevant if used internally, apart your own maintenance burden


Designing RDF


• Very relevant if used in a distributed data scenario


Designing RDF



• The big challenge is to properly reuse others’ ontologies


Designing RDF



• RDFs, OWL, Dublin Core, Darwin Core


Designing RDF



• RDFs, OWL, Dublin Core, Darwin Core
• Which ones are established and which ones not?


blueBill Specs


blueBill Specs

• Integrate data from different sources


blueBill Specs

• Wikipedia, media data bases, other information sources


blueBill Specs


• A common model core can be deﬁned


blueBill Specs


• But many things are “discovered” by working on them


blueBill Specs



• Iterative, incremental approach


blueBill Specs



• Iterative, incremental approach
• Manipulating a SQL schema is cumbersome


OpenSesame, Elmo


OpenSesame, Elmo

• Two major players in the Java ecosystem


OpenSesame, Elmo

• Jena (donated by HP)


OpenSesame, Elmo

• OpenSesame (by Aduna)


OpenSesame, Elmo

• Both FLOSS


OpenSesame, Elmo

• Both FLOSS

• My preference for OpenSesame, seems more modular


OpenSesame, Elmo

• Both FLOSS

• My preference for OpenSesame, seems more modular
• Fundamental feature for Android development...


OpenSesame

• RDF store implementation in 100% Java
• Provides store persistence, query support
• Small store can be manipulated in memory (Graph)
• Regular stores can be memory-based or persistent (Repository)
• Can be backed by ﬂat ﬁle or SQL database


Elmo


Elmo

• Annotation-based persistence for OpenSesame


Elmo

• Annotation-based persistence for OpenSesame
• Similar (and partially compatible) with JPA


Repository repository = new SailRepository(new MemoryStore());
ElmoModule elmoModule = new ElmoModule();
Elmo
SesameManagerFactory smf = new SesameManagerFactory(elmoModule, repository);
ElmoManager em = smf.createElmoManager(); // similar to EntityManager, thread-bound

////////////////////////////////

RepositoryConnection conn = repository.getConnection();
ValueFactory valueFactory = conn.getValueFactory();
RepositoryResult<Statement> statements = conn.getStatements(null,
valueFactory.createURI(...), null, false);

while (statements.hasNext())
{
Statement statement = statements.next();
Object subject = statement.getSubject();
Object object = statement.getObject();

•
// ...
} Annotation-based persistence for OpenSesame
conn.close();
• Similar (and partially compatible) with JPA
//////////////////////////////// Elmo annotations

@rdf("http://http://rs.tdwg.org/dwc/2009-12-07/terms/Taxon)
public class Taxon
{
@rdf("http://http://rs.tdwg.org/dwc/2009-12-07/terms/scientificNameID")
private String scientificNameID;

@rdf("http://http://rs.tdwg.org/dwc/2009-12-07/terms/taxonRank")
private String rank;

// getters and setters
}

Running OpenSesame on Android



• Q: Is Android Java? Java™? JSE? A JDK? Something else?



• Not interested (here) in legal aspects: developer’s point of view




• A: It can run most of Java bytecode, with differences




• Obvious missing parts: UI (not a problem for RDF)




• Not-so-obvious differences: XML, JDBC (depending also on version)




• Not-so-obvious differences: XML, JDBC (depending also on version)

• Much, much better than JME, but not plain JSE


blueBill Android issues



• Targeting Android 1.5 (still 10% of appliances)



• Some missing XML APIs



• e.g. Duration, XMLGregorianCalendar, QName




• Reﬂection works, but Google warns it’s slow




• Things might have changed with 2.2





• OpenSesame doesn’t support sqlLite out-of-the-box





• OpenSesame doesn’t support sqlLite out-of-the-box
• Performance (?)


Fixing XML


Fixing XML

• Android apps can’t bundle java.*, javax.* packages


Fixing XML

• Obvious security reasons


Fixing XML


• Solution: Maven Shade plugin


Fixing XML


• Missing classes from Harmony into it.tidalwave.android.xml.*


Fixing XML



• Shade plugin renames references in existing bytecode


Fixing XML



• Shade plugin renames references in existing bytecode

• Similar solution would work with Ant


Fixing XML
<groupId>it.tidalwave.bluebill</groupId>
<artifactId>it-tidalwave-android-org-openrdf-sesame-model</artifactId>
<version>2.2.3.8-SNAPSHOT</version>
<packaging>jar</packaging>
<name>Android Patch: OpenSesame Model</name>

<dependencies>
<dependency>
<groupId>org.openrdf.sesame</groupId>
<artifactId>sesame-model</artifactId>
</dependency>
<dependency>
<groupId>it.tidalwave.bluebill</groupId>
<artifactId>it-tidalwave-android-javax-xml-datatype</artifactId>
</dependency>

</dependencies>

<build>
<plugins>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>

<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>

•
</goals>
Missing classes from Harmony into it.tidalwave.android.xml.*
<configuration>
....
<relocations>

•
<relocation>
Shade plugin renames references in existing bytecode
<pattern>javax.xml.namespace.QName</pattern>
<shadedPattern>it.tidalwave.android.javax.xml.namespace.QName</shadedPattern>
</relocation>

•
<relocation>

Similar solution would work with Ant
<pattern>javax.xml.datatype</pattern>
<shadedPattern>it.tidalwave.android.javax.xml.datatype</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

Hand-written marshalling



• Replaces Elmo reﬂected-based support



• Write a custom *Marshaller and *Unmarshaller for
each entity



each entity
• Might go away



each entity
• Might go away
• If Android 2.2 proves to support fast reﬂection



each entity
• Might go away
• When 2.2 becomes the target Android version


public class SimpleObservationMarshaller extends AsStatementMarshaller
{
@Override @Nonnull
public void marshal (final @Nonnull As entity, final @Nonnull Context context)
{
super.marshal(entity, context);
final Observation observation = (Observation)entity;
context.addStatement(observation, Vocabulary.RDF_TYPE, Vocabulary.OBSERVATION);
context.addStatement(observation, Vocabulary.DC_DATE, observation.getDate());
context.addStatement(observation, Vocabulary.MADE_AT, observation.getLocation());
context.addStatement(observation, Vocabulary.DC_PUBLISHER, observation.getSource());

for (final Observer observer : observation.findObservers().results())

•
{
Write a custom *Marshaller and *Unmarshaller for
}
context.addStatement(observation, Vocabulary.DC_CREATOR, observer);

each entityObservationItem observationItem : observation.findObservationItems().results())
for (final
{

•
context.addStatement(observation, Vocabulary.SKOS_NARROWER, observationItem);
Might go away
}

try
{
context.addStatement(observation.as(Media.class), Vocabulary.FOAF_TOPIC, observation);
}

catch (AsException e)
{
// ok, no attachment
}
}
}


public class SimpleObservationUnmarshaller extends StatementUnmarshallerSupport
public class SimpleObservationMarshaller extends AsStatementMarshaller
{
{@Override @Nonnull
public As unmarshal (final @Nonnull List<Statement> statements, final @Nonnull Context context)
@Override @Nonnull
{
public SimpleObservationSet observationSet = (SimpleObservationSet)context.peek();
final
void marshal (final @Nonnull As entity, final @Nonnull Context context)
{
final Date date = (Date)getLiteral(statements, Vocabulary.DC_DATE);
super.marshal(entity, context);
final Location location = (Location)findEntity(context, statements, Vocabulary.MADE_AT);
final Source source = (Source)findEntity(context, statements, Vocabulary.DC_PUBLISHER);
final Observation observation = (Observation)entity;
final Set<Observer> observers = new HashSet<Observer>();
context.addStatement(observation, Vocabulary.RDF_TYPE, Vocabulary.OBSERVATION);
final Set<ObservationItem> observationItems = new HashSet<ObservationItem>();
context.addStatement(observation, Vocabulary.DC_DATE, observation.getDate());
final List<Object> extras = new ArrayList<Object>();
extras.add(new DefaultIdentifiable(Converter.valueToId(statements.get(0).getSubject())));
context.addStatement(observation, Vocabulary.MADE_AT, observation.getLocation());
• Replaces Elmo reﬂected-based support observation.getSource());
context.addStatement(observation, Vocabulary.DC_PUBLISHER,
final String displayName = (String)getLiteral(statements, Vocabulary.RDFS_LABEL);

if for (final!= null)
(displayName Observer observer : observation.findObservers().results())

•
{ {

Write a custom *Marshaller and *Unmarshaller for
}
}
extras.add(new DefaultDisplayable(displayName, "SimpleObservation"));
context.addStatement(observation, Vocabulary.DC_CREATOR, observer);

each entityObservationItem observationItem : observation.findObservationItems().results())
for (final Statement statement : context.findStatements(Vocabulary.FOAF_TOPIC,
for (final
{
Converter.valueToId(statements.get(0).getSubject())))

{
extras.add(context.find(statement.getSubject()));

•
context.addStatement(observation, Vocabulary.SKOS_NARROWER, observationItem);
Might go away
}
}
final SimpleObservation observation =
new SimpleObservation(observationSet, date, location, observers, observationItems, source, extras);
try
context.push(observation);

{
for (final Statement statement : findStatementsWithPredicate(statements, Vocabulary.SKOS_NARROWER))
{
context.addStatement(observation.as(Media.class), Vocabulary.FOAF_TOPIC, observation);
}
observationItems.add((SimpleObservationItem)context.find(statement.getObject()));

{
}catch (AsException e)

for (final Statement statement : findStatementsWithPredicate(statements, Vocabulary.DC_CREATOR))
// ok, no attachment
{
}
observers.add((SimpleObserver)context.find(statement.getObject()));
}}
} context.pop();

return observation;
}
}

Iterations


Iterations

• Iteration #1: no RDF, just ad hoc code (and a few data)


Iterations

• Current iteration #2: RDF, OpenSesame, custom marshalling


Iterations

• 4400+ birds, about 500.000 statements


Iterations

• Statements grouped per-bird in N3 syntax, zipped


Iterations

• Zip provides random access


Iterations


• Future iteration #3: Using OpenSesame on a single ﬂat ﬁle


Iterations


• Future iteration #4: Using OpenSesame on a single sqlLite


Iterations


• Future iteration #4: Using OpenSesame on a single sqlLite
• Future iteration #5: Eventually use Elmo


Challenges


Challenges

• How to import (once) 500.000 statements


Challenges

• Given that 50” are an acceptable only-once initialization time...


Challenges

• ... it would require 0.1ms per statement (probably impossible!)


Challenges


• Writing sqlLite integration code for OpenSesame


Challenges


• Writing sqlLite integration code for OpenSesame
• Query performance of a single database is unknown


Conclusion


Conclusion
• RDF stores are a NoSQL approach for ﬂexible data schemata


Conclusion
• Not necessarily related to the full Semantic Web


Conclusion

• 100% Java, FLOSS solutions exist


Conclusion

• Can be at least partially deployed with Android smartphones


Conclusion

• Full deployment being researched


Conclusion


• Proved to be effective in production


Conclusion


• blueBill Mobile works


Conclusion



• The Semantic Web has actually got good points


Conclusion



• The Semantic Web has actually got good points
• blueBill effort cut in half if people provided data in (good) RDF

Question time


Question time

• Follow me
• at my Java.Net blog
• at DZone

• Useful links
• http://bluebill.tidalwave.it/mobile
• http://www.openrdf.org


NOSQL also means RDF stores: an Android case study

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a NOSQL also means RDF stores: an Android case study

Semelhante a NOSQL also means RDF stores: an Android case study (20)

Mais de Fabrizio Giudici

Mais de Fabrizio Giudici (13)

Último

Último (20)

NOSQL also means RDF stores: an Android case study

Notas do Editor