Loading a lot of data into a graph database is not a trivial exercise. TypeDB Loader (formerly known as GraMi) was developed to allow large-scale data import into TypeDB, a strongly-typed database. Recent improvements have immensely simplified the configuration interface to allow for easier data importing, while maintaining features and the promise of loading huge amounts of data into TypeDB as fast as possible.
5. why TypeDB?
Most of biology can be modeled as a networks/graphs
data are highly context-dependent
inference can do A LOT for us
backend for AI/ML to “complete” our graph (attribute
and link prediction)
19. migration status / stop & restart
Things just got a lot more tricky…
And it doesn’t yet work!
BUT:
- you cannot change block size
- you cannot re-order data
- you can change threads
23. how to use typeDB Loader: as a Java Dependency
24. significant changes summary
Significant Updates:
• Data config + processor config now just one config
• Config syntax
• Config validation and reporting
• Failure reports on row-level
• UTF8 encoding enforcement
• RegEx-based preprocessing of data
25. Next Steps
• Config validation continued…
• Migration status re-implemention in progress
• Improved error reporting
• Schema-based config template generation
• JSON schema/data handling
26. Use me!
resources
TypeDB Loader
Github
https://github.com/typedb-osi/typedb-loader
Wiki https://github.com/typedb-osi/typedb-loader/wiki
Medium Tutorial in progress…
Example Project in progress…
Licensing
Above repositories include software developed at Bayer AG. They are released under the Apache License 2.0.
Credits
Icon in banner by Smashicons from Flaticon