Using AsciiArt to Analyse your SourceCode with Neo4j and OSS Tools

Who the hell is this guy?

Michael Hunger
Developer Advocate Neo Technology
Love People and Graphs
@mesirii | michael@neotechnology.com

What will he talk about?

What is this Neo4j Graphdatabase thing?
Ascii-Art Rocks
Graphs in Your Code - The Idea
Having Fun with your Code and jQAssistant
Gimme More

What is a Graph Database ?

labeled Nodes
directed, typed Relationships
arbitrary Properties on each

Property Graph Model

What makes it special ?

close to the object model
prematerialize relationships
traversals in linear time
sparse, heterogenous data + schema free
local queries - explore the neighbourhood
whiteboard-friendly

Where can/should I use it ?

Impact Analysis (Network, Software)
Routing / Logistics
Recommendation, Dating, Job-Search
Sciene (Metadata, Drug Research)
Masterdata, Hierarchy-Mgmt
Fraud-Detection, Market-Analysis
Social, …. and many more

(Graphs)-[:LOVE]→(Whiteboards)

Ascii-Art Rocks

Turn text into pictures
Turn picture into text
The Power of Symbols
Graph Patterns Made easy
Hacker and Mudder Friendly
Diffs, VCS

Ascii-Art Rocks

_                         )      ((   ))     (
(@)                      /|\      ))_((     /|\                        _
|-|`\                   / | \    (/\|/\)   / | \                      (@)
| | -------------------/--|-voV---\`|'/--Vov-|--\---------------------|-|
|-|                         '^`   (o o)  '^`                          | |
| |                               `\Y/'                               |-|
|-|          Welcome to the Mages Lair Multiple User Dungeon          | |
| |                    Welcome to the Addiction.                      |-|
|-|                                                                   | |
| |                                                                   |-|
|_|___________________________________________________________________| |
(@)               l   /\ /        ( (       \ /\   l                `\|-|
                  l /   V          \ \       V   \ l                  (@)
                  l/               _) )_          \I
                                   `\ /'
                                     `

Cypher

(Cypher)-[:USES]→(Ascii-Art)

Declarative Graph Query Language
Graph Pattern Matching
Humane, Readable
Expressive
Read and Write Graphs
Tabular Results

Cypher Query - Example Geekout

Setup

CREATE (:Year {year:2014})<-[:IN_YEAR]-(geekout:Conference {name:"Geekout"})
        -[:LOCATION]->(:City {name:"Tallinn"})
CREATE (track:Track {name:"Room 1"})-[:TRACK_OF]->(geekout)

MERGE (speaker:Attendee:Speaker {name:"Hadi Hariri"}) MERGE (geekout)<-[:ATTENDS]-(speaker)

CREATE (speaker)-[:PRESENTS]->(session:Session {title:"Mouseless IDE"})<-[:IN_TRACK]-(track)

FOREACH (name in ["Java","IDE","Development"] |
   MERGE (topic:Topic {name:name})
   CREATE (session)-[:HAS_TOPIC]->(topic)))

Query

// Which Speakers entertain you with Java ?
MATCH (t:Topic {name:"Java"})<-[:HAS_TOPIC]-(session)<-[:PRESENTS]-(speaker),
      (session)<-[:IN_TRACK]-(track)

RETURN speaker.name as speaker, {time:session.time, session:session.title} as session,
       track.name as track
ORDER BY session.time

Geekout 2014 GraphGist

Interactive GraphGist Document WorldCup

Software Analytics - Approach

look at one interesting aspect
which insights would be cool?
model it as a graph
get data
import into graph model
enrich graph model with concepts / structure
query for insights

(Code)-[:IS_A]→(Graph)

AST, ByteCode, Source-Code
Inheritance, Composition, Dependencies
Transitive Module and Library dependencies
Dependency injection config
Data model (db) <→ object model
Runtime characteristics: call graph, heap
Version control, repositories, issues

jQAssistant

Open Source Software Analytics Tool
Plugins for Java, JEE, JPA, Maven, Gradle, SonarJ …
All Cypher based
Technical and Domain Concept Definitions
Compute Software Metrics
Declare and Validate Architectural Rules
Integrated in Build Process

jQAssistant

Actively Looking for Contributions

http://github.com/buschmais/jqassistant

Modeling Source Code as a Graph

Java software graph model : Nodes

Artifact
Package
Type, Class, Interface, Annotation, Enum
Method, Constructor, Parameter
Field
Value, Class, Annotation, Enum, Primitive, Array

Java software graph model: Relationships

CONTAINS, DECLARES
EXTENDS, IMPLEMENTS
RETURNS, THROWS, INVOKES, HAS, IS
ANNOTATED_BY, OF_TYPE

Approach

Scan your project with
Plugins for Code (Java-ASM), Config, Metadata
Import into Neo4j
Enrich with declared technical and domain concepts
On top of those concepts
Software-Metrics queries
Architectural-Rules queries

Query the Data

Demo

Analyzing A Maven Repository

Rickard Öberg

This little nifty tool will allow you to import your local Maven repository information into a Neo4j graph, in particular dependencies between artifacts.

You can then take this graph and put it into a Neo4j server, and perform Cypher queries on it.

Or whatever else awesome you want to do.

mvn compile exec:java -Dexec.mainClass=com.github.rickardoberg.neomvn.Main \
  -Dexec.arguments="$HOME/.m2/repository"

https://github.com/rickardoberg/neomvn#example-queries

NeoMVN: Example Queries

Find all transitive dependencies of all artifacts with "org.neo4j" groups

MATCH (group:Group {groupId:'org.neo4j'}),
      (group)-[:HAS_ARTIFACT]->(artifact)-[:HAS_VERSION]->(version)<-[:HAS_DEPENDENCY]-(dependent)

WHERE left(dependent.groupId,9)<>group.groupId
RETURN DISTINCT dependent.artifactId, dependent.groupId

Which version of JUnit is the most popular

MATCH (group:Group {groupId:'junit'})
MATCH (group)-[:HAS_ARTIFACT]->(artifact)-[:HAS_VERSION]->(version)<-[:HAS_DEPENDENCY]-(dependent)
RETURN version.version, count(dependent) as depCount
ORDER BY depCount DESC

Query a JVM Heapdump

Get a heap-dump with jmap
jmap -dump:format=b,file=dump.hprof <pid>
Use jhat dump.hprof find the /oql endpoint
run the OQL script to generate Cypher code
Import into Neo4j
Run arbitrary queries
Visualize

Graph Domain Model

OQL to Generate Cypher

select (function () {
  var entries = filter(p.waitingTxMap.data.table, function(it) { return it; });
  var resources = "";

  var idSeq = 0;

  for ( var i = 0; i < entries.length; i++ ) {
    var resource = entries[i].value.resource;
    var index = resource.index.toString();
    var key = resource.key.toString();
    var indexNameIndexKey = index + "_" + key;
    var indexName = "n" + idSeq++;

    var resourceName = "";

    var tx = entries[i].key;
    var txName = "n" + idSeq++;

    var cypher =  "MERGE (" + txName + ":Tx {txId:" + tx.eventIdentifier + "})<br />";
    cypher     += "MERGE (" + indexName + ":IndexLock {indexLockId:'" + indexNameIndexKey + "'})<br />";
    cypher     += "CREATE (" + txName + ")-[:WAITING_ON]->(" + indexName + ")<br />";

    resources += cypher + "<br />";
  }

...

  return resources;
})()
from org.neo4j.kernel.impl.transaction.RagManager p

Visualization of a Deadlock Scenario

JVM heap analysis using Neo4j

Nat Pryce, James Richardson (Software Engineers, Sky)

Use Neo4J for ad-hoc analysis of heap use in a proprietary embedded JVM that’s deployed in one of the most widely used consumer products in the UK (Sky Box).
Used Cypher queries that uncovered surprising aspects of their code, platform and the Java compiler.
And finding a memory leaking JSON parser with a Cypher query.

Heap Model

From Nat’s Graph Schema Modeling Approach

Talk at Accu 2014: Looking for Smoking Guns in a Haystack - using a graph database for JVM heap analysis

Using graphs for source code analysis

Raoul-Gabriel Urma (PhD in Computer Science, The University of Cambridge)

Did you know that the source code of software that you engineer every day can also be represented as graphs?

In this talk, we demonstrate how you can perform program analysis using Neo4j.

We describe a prototype that stores a graph representation of the source code of Java programs in Neo4j. We can then query the graph using Cypher and we’ll show various examples of possible queries such as “find me all recursive methods” and “find me all subtypes of a given type”.

This talk is based on Raoul-Gabriel’s research paper Source-Code Queries with Graph Databases—with Application to Programming Language Usage and Evolution

Source Control, Issues, Social Coding

What can you learn from commits
about the code
Class Toxicity, Frequency of Change (Feathers)
about the people
checkin times, collaboration, commit-size, commit-frequency
Issues
bug-rich classes (separation of concerns?)

Example: Import Git Commit Logs into Neo4j

git log --format emits CSV
Graph Model
LOAD CSV with Cypher
Create / Update complex Graph Structure

Blog Post

Visualization

Finally: Some Eye Candy

Isaac & Nash (Software Engineers at Leap Motion)

Leap Motion Software
Inheritance Hierarchy, Call Graph
Render to .dot file
Use dotparse.js to read it in
WebGL enabled Three.js rendering
LeapMotion SDK 2.x beta

Let’s have a look: Demo Source

Questions ? Thank You!

← →

Using AsciiArt to Analyse your SourceCode with Neo4j and OSS Tools

Who the hell is this guy?

What will he talk about?

What is a Graph Database ?

Property Graph Model

What makes it special ?

Where can/should I use it ?

Ascii-Art Rocks

Ascii-Art Rocks

Cypher

(Cypher)-[:USES]→(Ascii-Art)

Cypher Query - Example Geekout

Setup

Query

Geekout 2014 GraphGist

Software Analytics - Approach

(Code)-[:IS_A]→(Graph)

jQAssistant

jQAssistant

Modeling Source Code as a Graph

Java software graph model : Nodes

Java software graph model: Relationships

Approach

Query the Data

Demo

Analyzing A Maven Repository

NeoMVN: Example Queries

Find all transitive dependencies of all artifacts with "org.neo4j" groups

Which version of JUnit is the most popular

Query a JVM Heapdump

Graph Domain Model

OQL to Generate Cypher

Visualization of a Deadlock Scenario

JVM heap analysis using Neo4j

Heap Model

Using graphs for source code analysis

Source Control, Issues, Social Coding

Example: Import Git Commit Logs into Neo4j

Visualization

Finally: Some Eye Candy

Questions ? Thank You!

Status of the presenation

Cypher queries execution