DN v5 : JPQL FROM JOIN to another “root” entity

A vendor extension now available in DataNucleus v5 is for JPQL’s ability to JOIN to other entities. In standard JPQL you can only add a JOIN through a relation. For example

SELECT p FROM Person p 
    JOIN p.address a WHERE a.street = 'Main Street'

In this example we have a candidate “root” entity of Person, and are joining to the Address entity via the “address” relation field in Person.

The vendor extension provides the ability to join to a new “root” entity. For example

SELECT p FROM Person p 
    LEFT OUTER JOIN Address a ON p.addressName = a.name

In this example we have a candidate “root” entity of Person, and we are joining to the Address entity using a manually specified ON clause. That is, there doesn’t need to be a relation between Person and Address to make this join, just a condition that we are imposing in the ON clause.

This is not part of the JPA 2.1 spec, but may be in a future JPA spec, but you can make use of it in DataNucleus v5+.

Posted in Uncategorized | Leave a comment

JDO 3.2 JDOQLTypedQuery : improvements relative to DN extension

Now that JDO 3.2 standardises the “typed query” mechanism that we prototyped in DataNucleus v3.x and v4.0/v4.1 it is worth mentioning the additions to the API that are present in the JDO 3.2 variant.


This has gained the following methods to providing JDOQL “matches” operation capability

BooleanExpression matches(StringExpression expr);
BooleanExpression matches(String str);



The following methods are added, for bitwise operations support

NumericExpression bAnd(NumericExpression bitExpr);
NumericExpression bOr(NumericExpression bitExpr);
NumericExpression bXor(NumericExpression bitExpr);

In addition it also gains these methods to match the equivalent JDOQL string based operators (-, ~)

NumericExpression neg();
NumericExpression com();



This also gains the JDOQL string based operators (-, ~)

NumericExpression neg();
NumericExpression com();



This gains the JDOQL string based operator (-)

BooleanExpression neg();

Java 8

In addition to the above you can now also make use of Java8 Time types in JDOQLTypedQuery, whereas this was not possible in the previous DataNucleus extension.



The JDOQLTypedQuery class has some slight changes to make it more consistent with the JDOQL standard API.

  • Added method result(…) rather than specifying result clauses in the execute call.
  • Changed addExtension/setExtensions methods to extension(…)/extensions(…).
  • Added unmodifiable, serializeRead, datastoreReadTimeout, datastoreWriteTimeout, ignoreCache, saveAsNamedQuery.
Posted in JDO, JDOQL, LINQ, Uncategorized | Leave a comment

JDO 3.2 JDOQLTypedQuery

JDO 3.2 is now standardising what we initially developed as a DataNucleus extension for JDO querying, namely typesafe queries. In DataNucleus upto and including v4.1 it was referred to as JDOQL Typesafe Query, and in JDO 3.2 (DataNucleus v4.2+) it is called JDOQLTypedQuery. Here we present some examples, the same examples as a previous blog post about the original DataNucleus extension but now updated to the JDOQLTypedQuery API.

In these examples we have a class Product. Consequently we have a “query” class autogenerated – QProduct. Here is the QProduct class

public class QProduct extends PersistableExpressionImpl implements PersistableExpression
    public static QProduct candidate() {...} // candidate "this"
    public static QProduct candidate(String name) {...}
    public static QProduct parameter(String name) {...}
    public static QProduct variable(String name) {...}
    public NumericExpression id;
    public StringExpression name;
    public NumericExpression value;

So, as you can see we have access to the persistable fields/properties of Product by way of public fields in its query class. Now on to some examples

Filter only
Here we select all Products that have a value less than 40.0

JDOQLTypedQuery tq = pm.newJDOQLTypedQuery(Product.class);
List<Product> results = tq.filter(QProduct.candidate().value.lt(40.0))

Filter + Order
Here we select all Products that have a value less than 40.0 and ordering by their name

JDOQLTypedQuery tq = pm.newJDOQLTypedQuery(Product.class);
QProduct cand = QProduct.candidate();
List<Product> results = tq.filter(cand.value.lt(40.0))

Filter + Order + Result
Here we select the product name and product value for all Products that have a value less than 40.0 , ordering by their name

JDOQLTypedQuery tq = pm.newJDOQLTypedQuery(Product.class);
QProduct cand = QProduct.candidate();
List results = tq.filter(cand.value.lt(40.0)).orderBy(cand.name.asc())
    .result(true, cand.name, cand.value)

Filter with Methods
Here we select all Products that have a value less than 40.0 and name starting with “Wal”, and ordering by their name

JDOQLTypedQuery tq = pm.newJDOQLTypedQuery(Product.class);
QProduct cand = QProduct.candidate();
List<Product> results = 

Filter, using Parameters
Here we select all Products that have a value less than 40.0, using a parameter

JDOQLTypedQuery tq = pm.newJDOQLTypedQuery(Product.class);
QProduct cand = QProduct.candidate()
List<Product> results = 
    .setParameter("param1", 40.0)

Order + Range
Here we select Products, ordering by the name and restrict to the first two

JDOQLTypedQuery tq = pm.newJDOQLTypedQuery(Product.class);
QProduct cand = QProduct.candidate();
List<Product> results = tq.orderBy(cand.name.asc()).range(0,2)

Filter with Variable
Here we select Inventory (which has a collection of Products) that contain a Product with a particular name

JDOQLTypedQuery tq = pm.newJDOQLTypedQuery(Inventory.class);
QInventory cand = QInventory.candidate();
QProduct var = QProduct.variable("var");
List<Product> results = 

Filter with Subquery
Here we select Products with a value less than the average value of any Product

JDOQLtypedQuery tq = pm.newJDOQLTypedQuery(Product.class);
QProduct cand = QProduct.candidate();
JDOQLTypedSubquery tqsub = tq.subquery(Product.class, "p");
QProduct candsub = QProduct.candidate("p");
List<Product> results = 

As you can see from the above queries we have no hard-coded class or field names, providing better refactorability. We can obviously make use of all JDOQL supported methods in the above queries, and define parameters and variables just like in the standard (string-based) JDOQL API.

It should be noted that this JDOQL facility is much more elegant and requires less code than the equivalent queries using JPA “Criteria”.

Posted in Criteria, DataNucleus, JDO, JDOQL, JPA, JPQL, Typesafe | Leave a comment

JDO 3.2 Revisions to Query API

Here we present some changes to the JDO Query API that are coming in version 3.2 of the spec. Comments should be directed ASAP to the Apache JDO mailing list if you want any changes to make it in to this next revision.

Fluent API
When you are specifying a JDOQL query using the assorted setter methods you can end up with code like this

Query q = pm.newQuery(Manager.class);
q.setFilter("theDepts.contains(dept0) && dept0.manager == this");
q.declareParameters("mydomain.model.Department[] theDepts");
q.declareVariables("mydomain.model.Department dept0");
q.setOrdering("this.firstName ascending");

Clearly this leaves a lot to be desired in terms of API. In JDO 3.2 you can now do the following

Query q = pm.newQuery(Manager.class)
    .filter("theDepts.contains(dept0) && dept0.manager == this")
    .parameters("mydomain.model.Department[] theDepts")
    .variables("mydomain.model.Department dept0")
    .ordering("this.firstName ascending")

So you can chain your specification(s). The other thing of note is that you can now set the parameter values prior to execution. The reason for this comes next.

No casting on execute()
With the former query above, we would execute it as follows

Collection<Manager> managers = (Collection<Manager>) q.execute(depts);

With JDO 3.2 we can execute it like this

Collection<Manager> managers = q.executeList();

I think we can conclude that this is much cleaner. There is no need to cast the result from the execute method, simply using a different execute method based on what your query is expected to return.

Save as a NamedQuery
You can now construct a query, and save it as a named query, so you don’t need to regenerate it. You do that as follows

Query q = pm.newQuery("SELECT FROM Person WHERE this.lastName == :surname");

So from that point forwards when you need that query, you can do the following

Query q = pm.newNamedQuery(Person.class, "FindPeopleWithSurname");
Posted in Uncategorized | Leave a comment

JDO : querying between classes without relation

Let’s say we have the following classes and are using RDBMS for persistence
public class Country
long id;
String name;

public class Region
long id;
String name;
Long countryId;

public class District
long id;
String name;
Long regionId;

So we ignored all good object-oriented guidance and didn’t use real relations. Now lets suppose I want to query for the Country that has a district called “Cross Gates”. We need to write some JDOQL, but don’t have any real relations … just countryId and regionId to navigate through. So we introduce 2 variables to navigate across the 2 “pseudo relations”.

Query q = pm.newQuery("SELECT FROM mydomain.Country " +
"WHERE this.id == reg.countryId && reg.id == dist.regionId && dist.name == 'Cross Gates' " +
"VARIABLES mydomain.Region reg; mydomain.District dist;");

This creates the following SQL
WHERE VAR_DIST."NAME" = 'Cross Gates'

In our particular case we know that there are no NULL pseudo relations, so would like “INNER JOIN” for performance. So we make use of a DataNucleus extension, like this

Query q = pm.newQuery("... query as before ...");
q.addExtension("datanucleus.query.jdoql.reg.join", "INNERJOIN");
q.addExtension("datanucleus.query.jdoql.dist.join", "INNERJOIN");

So we set joins for both variables to INNER JOIN, and the SQL becomes

WHERE VAR_DIST."NAME" = 'Cross Gates'

Easy really 😉 Would clearly have been easier if we had designed our classes around good O-O practice though. And if using some other type of datastore then your datastore would have to allow joins to even attempt this

Posted in JDO, JDOQL | 10 Comments

Want to assist in the development of JDO 4.0?

We have, since 2006, been reliant on the Apache JDO project to push forward the JDO standard. Politics have finally become too much for this arrangement, with Oracle involving lawyers to prohibit progress, and the Apache organisation not being as rapid as it should be at getting releases out of the door. I have now forked their API (now present in DN GitHub). The aim is for this repository to be used to develop JDO 4.0 (and maybe later dependent on people’s involvement), initially to get a TypeSafe query mechanism into the JDO API (originally contributed back in April 2010!!), and bring the API up to date wrt generics in queries themselves. There are also many other features that were requested in Apache JDO JIRA over the last few years that have been simply left, but are needed.

If you, or your company uses JDO this is your chance to get involved and push it forward to cater for these required features. Do you really want to have to cast the return from your Query to be a List etc. You could offer your real-world experience with the JDO API and what you think could be done to make it easier to use.

Should the Apache JDO project “wake up” at some point in the future we could easily enough merge our changes back into their API and discontinue this fork.

Posted in JDO, JDOQL | 7 Comments

Bytecode Enhancement contract in DataNucleus AccessPlatform v4.0

Now in GitHub master, for DataNucleus AccessPlatform v4.0, we have changed the bytecode enhancement contract.

Since the days of JPOX we’ve always used the JDO bytecode enhancement contract as defined in the JDO spec. This has always been adequate to provide the necessary hooks into the object to allow for “transparent persistence“. Saying that though, it does mean that anyone using DataNucleus would always have to have jdo-api.jar in their CLASSPATH, even when using JPA. This was clearly undesirable, but not a large price to pay for easy provision of JPA.

In v4.0 onwards we will enhance classes to implement org.datanucleus.enhancer.Persistable. This is very similar in terms of structure, just that methods are now prefixed “dn” instead of “jdo“, and there is now a method to get the ExecutionContext that is managing the object (whereas before it was a PersistenceManager, which made very little sense for JPA usage).

Why the change?

Oracle is putting significant obstacles in the way of having further releases of the JDO standard, involving lawyers etc. Additionally, following the Apache way, the Apache JDO project has not exactly operated very efficiently in terms of getting releases out of the door. Moreover we want to remove the requirement of having to have jdo-api.jar in the CLASSPATH for JPA usage. This change will also mean that we can, in principle, improve the bytecode enhancement contract to make things more efficient or add on more information to enhance the persistence process without being restricted by what JDO has bothered to standardise.

What does this mean for a typical user?

It means very little in reality, and the majority of applications will work unchanged (apart from having to re-enhance the classes). Some minor things that will change

  • Wherever you use enhanced classes, you will need datanucleus-core.jar in the CLASSPATH
  • JPA users won’t need to have jdo-api.jar in the CLASSPATH
  • Internally DataNucleus now uses its own builtin single-field identity classes, and if you refer to javax.jdo.identity.* classes (for JDO contexts) will auto-convert to our own class for internal use. See this package for the DN internal identity classes. There is really no need to use these JDO builtin classes directly since DataNucleus will always select the most appropriate id type when you have a single PK field.
  • You no longer check if a returned object is of type javax.jdo.spi.PersistenceCapable since it won’t be, instead being a org.datanucleus.enhancer.Persistable.

Please register any concerns/queries in the comments section

Posted in Bytecode, JDO, JPA | 6 Comments

Configuring persistence of fields/properties using TypeConverters with JDO

JPA 2.1 allows a user to specify a converter on the value of a field/property for how it is persisted in the datastore. The way we implement that in DataNucleus is to have the JPA converter as a wrapper to our own internal TypeConverter class. This means that we can make the TypeConverter mechanism available to JDO users too. Here’s how it works. The first thing to do is to define a TypeConverter. As you can see, it requires implementation of just 2 methods, one for use in converting the (raw) field value into the datastore value, and one for use in converting the datastore value back into the (raw) field value. This means that the user has significant flexibility on how their values are stored, and can define their own converters and not rely on them being part of DataNucleus. If we take an example, here we want to store a field as serialised, but not standard Java serialised, instead using the Kryo library. So we define our TypeConverter as

public class KryoSerialiseStringConverter implements TypeConverter
    ThreadLocal kryo = new ThreadLocal();
    public Kryo getKryo()
        Object value = this.kryo.get();
        if (value == null)
            value = new Kryo();
        return (Kryo)value;

    public String toDatastoreType(Serializable memberValue)
        if (memberValue == null)
            return null;

        Kryo kryo = getKryo();
        String str = null;
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        Output output = null;
            output = new Output(baos);
            kryo.writeClassAndObject(output, memberValue);
            str = new String(Base64.encode(baos.toByteArray()));
        return str;

    public Serializable toMemberType(String datastoreValue)
        if (datastoreValue == null)
            return null;

        Kryo kryo = getKryo();
        byte[] bytes = Base64.decode(datastoreValue);
        Object obj = null;
        ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
            Input input = null;
                input = new Input(bais);
                obj = kryo.readClassAndObject(input);
                    if (input != null)
        catch (Exception e)
            throw new NucleusException("Error Kryo deserialising " + datastoreValue, e);
        return (Serializable)obj;

Now we need to register this converter under a name with DataNucleus runtime. Here we define a plugin.xml at the root of the plugin jar as

<type-converter name="kryo-serialise" member-type="java.lang.Serializable" datastore-type="java.lang.String"

So this converter is internally known as kryo-serialise. We add a MANIFEST.MF to our plugin jar as (something like)

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: DataNucleus Converter using Kryo
Bundle-SymbolicName: org.datanucleus.store.types.converters.kryo;singleton:=true
Bundle-Version: 3.2
Bundle-Localization: plugin
Bundle-Vendor: DataNucleus
Require-Bundle: org.datanucleus
Import-Package: com.esotericsoftware.kryo;version="2.21",

just defining the dependencies of this plugin for OSGi and the plugin mechanism. Now we just need to use it on a sample class and apply the conversion to a field/property.

public class Sample
    long id;

    @Extension(vendorName="datanucleus", key="type-converter-name", value="kryo-serialise")
    String longString;


Now whenever instances of Sample are persisted the field longString will be converted using our kryo-serialise converter. Easy! You can get the code for this example converter at DataNucleus GitHub repo

Posted in DataNucleus, java, JDO, Persistence | Leave a comment

AccessPlatform 3.3 and JPA 2.1

We will soon be releasing AccessPlatform 3.3. This is coming very soon after 3.2, and the reason for this is that it is simply AccessPlatform 3.2 plus full support for JPA 2.1 (i.e an upgraded datanucleus-api-jpa plugin). From that point both of these “versions” of AccessPlatform will be maintained for a period of time. So what is provided in JPA 2.1 ?

Stored Procedures

You can execute RDBMS Stored Procedures using the JPA API. This API should give you quite complete control over execution of any stored procedure, setting of IN/OUT/INOUT parameters and obtaining result set(s). To give an example

StoredProcedureQuery spq = 
Integer.class, ParameterMode.OUT);
boolean val = spq.execute();
Object paramVal = spq.getOutputParameterValue("PARAM1");

So here we have a stored proc that we register an Integer output parameter, and retrieve the value when we execute it. This is just one mode of operation, and you can see more usages in the documentation.

Entity Graphs

JPA has only had an equivalent of the JDO “default fetch group” since its inception. In this release it finally gets some degree of control over what fields are fetched when fetching an object from the datastore. To give an example, we have a class Person and want to pull in a field “bestFriend” under some circumstances but not by default. We define a named EntityGraph in metadata

@NamedEntityGraph(name="includeFriend", attributeNodes= {@NamedAttributeNode("bestFriend")})
public class Person
Person bestFriend;


and now we want to use this entity graph when loading an object of this type. We do this as follows.

EntityGraph friendGraph = em.getEntityGraph("includeFriend");
Properties props = new Properties();
props.put("javax.persistence.loadgraph", friendGraph);
MyClass myObj = em.find(Person.class, id, props);

So we retrieved the EntityGraph, and then used it in the find method. Equally we could have used it in a Query. You can read more about this topic in the documentation.

Schema Generation

JPA 2.1 allows generation of the schema as an up front task, or via Persistence.generateSchema(). You can specify this by making use of persistence properties, for example javax.persistence.schema-generation.database.action set to create. See the available persistence properties for details.

Foreign-Keys and Indexes

JPA 2.1 adds on the ability to specify RDBMS schema foreign-keys and indexes, for use during schema generation. By default a JPA implementation is free to generate whatever foreign-keys it decides are appropriate, but this ability allows a user to override this and control what is generated.

@JoinColumn(name="BESTFRIEND_ID", foreignKey=
@ForeignKey(name="BESTFRIEND_FK", foreignKeyDefinition=
Person bestFriend;

which will create a foreign key called “BESTFRIEND_FK” for this purposes. Similarly we can define indexes on a table of a class.

@Table(indexes={@Index(name="FIRSTNAME_IDX", columnList="FIRST_NAME")})
public class Person
String firstName;


so the firstName field is now indexed. You can read more about this topic in the documentation.

Criteria UPDATE/DELETE queries

Whilst the JPA Criteria API is overengineered and verbose it now has the ability to generate UPDATE and DELETE queries. For example

CriteriaUpdate crit = qb.createCriteriaUpdate(Person.class);
Root candidate = crit.from(Person.class);
crit.set(candidate.get(Person_.firstName), "Freddie");
Predicate teamName = qb.equal(candidate.get(Person.firstName), "Fred");
Query q = em.createQuery(crit);
int num = q.executeUpdate();

which will create the JPQL “UPDATE Person p SET p.firstName = ‘Freddie’ WHERE p.firstName = ‘Fred'”. You can do similar things for DELETE queries. You can read more about this topic in the documentation

Attribute Converters

By default a JPA implementation will persist a field in a datastore column of its chosen type. You can now override this to use a converter, performing the conversion from the field type to the datastore type in your converter class. The example we use in the documentation is where we have a field in our class of type URL and want to persist this as a String-type in the datastore (VARCHAR, CHAR etc).

public class URLStringConverter implements AttributeConverter
public URL convertToEntityAttribute(String str)
if (str == null)
return null;

URL url = null;
url = new java.net.URL(str.trim());
catch (MalformedURLException mue)
throw new IllegalStateException("Error converting the URL", mue);
return url;

public String convertToDatabaseColumn(URL url)
return url != null ? url.toString() : null;

and then in our class that has a URL field we mark the field to use this converter

URL url;

You can read more about this topic in the documentation.

JPQL FROM “ON” clauses

When joining in JPQL previously we could not add additional constraints on the join. You can now do this using the “ON” clause.

List result = em.createQuery(
"SELECT Object(A) FROM Account A LEFT OUTER JOIN A.login L ON L.userName = 'fred'").getResultList();


No matter what features you put in a query language some people will always want to make use of SQL functions specific to a particular datastore. With JPQL you can now do this using the FUNCTION keyword, like this

Query q = em.createQuery(
"SELECT p FROM Person p WHERE FUNCTION('UPPER', p.firstName) = 'FRED'");

As you can see, JPA 2.1 is a minor iteration on JPA, and you can now benefit from all of these features in DataNucleus AccessPlatform 3.3

Posted in AccessPlatform, JPA | Leave a comment

What does open-source owe you?

If you ever embark on a period of time writing open source (free) software you’ll almost certainly come across many different attitudes in the people who choose to use this software.

There will be some people who’ll actually say thanks for providing this software, that it’s helped them with their project, saved them many hours of time that otherwise they would have had to spend writing something similar. The software was suitable for their project needs, so great, your effort in making the software open source has benefitted people. Feel good.

There’ll be people who maybe say thanks but typically just accept the software as it is, some sort of given, and when they have a problem they take the source code and try to work out where the problem is. They may ask you questions about how the code works, or where to look to get started in resolving the problem. And some time later they may come back with a patch for their problem, so this makes the software better. Again, a good thing.

There will also be a group who take the software and use it for their projects. If a problem occurs they will report it. Their report may provide a way of reproducing the problem, or it may not. If a problem is reported with a way of being reproduced then it can be fixed when the people writing the software have some spare time to do it. Another good thing. If there is no way provided to demonstrate it then the problem report is of little use to anyone … unless the person who has the problem is willing to get their hands dirty, get the code and fix it (with help where necessary) since only they can see it.

A final group will take the software and if a problem occurs they keep it to themselves; it’s like they expect you to be aware of everything that could possibly happen with the software, the developer is a crystal ball wielder. The people who develop that software only have a certain amount of time, and they typically will use it and test it against what their own project requires. That doesn’t mean that their use-cases are the same as yours. So don’t expect open source developers testing to cover what you need for your project; you could contribute tests to their suite, or donate for their time to run against other datastores if this is important to you.

You may find people who ask the question “should I ditch use of your software?” when faced with a problem, something that your software doesn’t cater for, or fails on. Maybe this is in some kind of “threat” sense, fix this problem or I leave? Well the answer to that is simple really. People should do what is right for their project. They’ve demonstrated one way or another whether they wish to contribute anything to the open source software (problem reports, testcases, patches, documentation, blogs, testimonials, donations, etc, there are many ways). If they haven’t demonstrated the willingness to do anything for the project then their input won’t be missed if they go off somewhere else. Do they pay the people who develop the software ? well no. Does the license of that software imply any guarantee that all problems will be fixed immediately when the toys are thrown from the pram? nope. Maybe this software is not the correct tool for their project? in which case use the correct tool for the job, and don’t vent your frustration at your choices on the people who have provided something for nothing. Further to this, stick to the old addage “don’t ask someone to do what you wouldn’t be prepared to do yourself“, didn’t your mum teach you that?

People seem to have got accustomed to having an open source solution these days, and that somehow it’s their “right” to have it and their “right” to have any problems found fixed. While open source (free) software gives projects a leg up in reaching their end goal more rapidly and is a great thing for software developers, open source software owes the end user nothing. Best understand this. The end user has the opportunity to do many things to contribute to that software, make it better, repay those people who put their time into developing it. The time of these people who wrote it is important to them, even if it isn’t to you; at least respect that.

Some things are for sure, when you embark on writing open source software, it can be very rewarding, very beneficial if you want a way to demonstrate to potential employers of your coding skills, excellent possibilities for exploring other technologies and gaining experience, working with other people with different viewpoints, but don’t go into it for the gratitude 🙂

[Disclaimer : while there is such a thing as commercial open source software, providing the source code yet charging for the software, what is being discussed here is the much more common open source free software]

Posted in Uncategorized | Leave a comment