JDO 3.2 JDOQLTypedQuery

JDO 3.2 is now standardising what we initially developed as a DataNucleus extension for JDO querying, namely typesafe queries. In DataNucleus upto and including v4.1 it was referred to as JDOQL Typesafe Query, and in JDO 3.2 (DataNucleus v4.2+) it is called JDOQLTypedQuery. Here we present some examples, the same examples as a previous blog post about the original DataNucleus extension but now updated to the JDOQLTypedQuery API.

In these examples we have a class Product. Consequently we have a “query” class autogenerated – QProduct. Here is the QProduct class

public class QProduct extends PersistableExpressionImpl implements PersistableExpression
{
    public static QProduct candidate() {...} // candidate "this"
    public static QProduct candidate(String name) {...}
    public static QProduct parameter(String name) {...}
    public static QProduct variable(String name) {...}
    public NumericExpression id;
    public StringExpression name;
    public NumericExpression value;
    ...
}

So, as you can see we have access to the persistable fields/properties of Product by way of public fields in its query class. Now on to some examples

Filter only
Here we select all Products that have a value less than 40.0

JDOQLTypedQuery tq = pm.newJDOQLTypedQuery(Product.class);
List<Product> results = tq.filter(QProduct.candidate().value.lt(40.0))
    .executeList();

Filter + Order
Here we select all Products that have a value less than 40.0 and ordering by their name

JDOQLTypedQuery tq = pm.newJDOQLTypedQuery(Product.class);
QProduct cand = QProduct.candidate();
List<Product> results = tq.filter(cand.value.lt(40.0))
    .orderBy(cand.name.asc())
    .executeList();

Filter + Order + Result
Here we select the product name and product value for all Products that have a value less than 40.0 , ordering by their name

JDOQLTypedQuery tq = pm.newJDOQLTypedQuery(Product.class);
QProduct cand = QProduct.candidate();
List results = tq.filter(cand.value.lt(40.0)).orderBy(cand.name.asc())
    .result(true, cand.name, cand.value)
    .executeResultList();

Filter with Methods
Here we select all Products that have a value less than 40.0 and name starting with “Wal”, and ordering by their name

JDOQLTypedQuery tq = pm.newJDOQLTypedQuery(Product.class);
QProduct cand = QProduct.candidate();
List<Product> results = 
    tq.filter(cand.value.lt(40.0).and(cand.name.startsWith("Wal")))
    .executeList();

Filter, using Parameters
Here we select all Products that have a value less than 40.0, using a parameter

JDOQLTypedQuery tq = pm.newJDOQLTypedQuery(Product.class);
QProduct cand = QProduct.candidate()
List<Product> results = 
    tq.filter(cand.value.lt(tq.doubleParameter("param1")))
    .setParameter("param1", 40.0)
    .executeList();

Order + Range
Here we select Products, ordering by the name and restrict to the first two

JDOQLTypedQuery tq = pm.newJDOQLTypedQuery(Product.class);
QProduct cand = QProduct.candidate();
List<Product> results = tq.orderBy(cand.name.asc()).range(0,2)
    .executeList();

Filter with Variable
Here we select Inventory (which has a collection of Products) that contain a Product with a particular name

JDOQLTypedQuery tq = pm.newJDOQLTypedQuery(Inventory.class);
QInventory cand = QInventory.candidate();
QProduct var = QProduct.variable("var");
List<Product> results = 
    tq.filter(cand.products.contains(var).and(var.name.startsWith("Wal")))
    .executeList();

Filter with Subquery
Here we select Products with a value less than the average value of any Product

JDOQLtypedQuery tq = pm.newJDOQLTypedQuery(Product.class);
QProduct cand = QProduct.candidate();
JDOQLTypedSubquery tqsub = tq.subquery(Product.class, "p");
QProduct candsub = QProduct.candidate("p");
List<Product> results = 
    tq.filter(cand.value.lt(tqsub.select(candsub.value.avg())))
    .executeList();

As you can see from the above queries we have no hard-coded class or field names, providing better refactorability. We can obviously make use of all JDOQL supported methods in the above queries, and define parameters and variables just like in the standard (string-based) JDOQL API.

It should be noted that this JDOQL facility is much more elegant and requires less code than the equivalent queries using JPA “Criteria”.

Posted in Criteria, DataNucleus, JDO, JDOQL, JPA, JPQL, Typesafe | Leave a comment

JDO 3.2 Revisions to Query API

Here we present some changes to the JDO Query API that are coming in version 3.2 of the spec. Comments should be directed ASAP to the Apache JDO mailing list if you want any changes to make it in to this next revision.

Fluent API
When you are specifying a JDOQL query using the assorted setter methods you can end up with code like this

Query q = pm.newQuery(Manager.class);
q.setFilter("theDepts.contains(dept0) && dept0.manager == this");
q.declareParameters("mydomain.model.Department[] theDepts");
q.declareVariables("mydomain.model.Department dept0");
q.setOrdering("this.firstName ascending");

Clearly this leaves a lot to be desired in terms of API. In JDO 3.2 you can now do the following

Query q = pm.newQuery(Manager.class)
    .filter("theDepts.contains(dept0) && dept0.manager == this")
    .parameters("mydomain.model.Department[] theDepts")
    .variables("mydomain.model.Department dept0")
    .ordering("this.firstName ascending")
    .setParameters(depts);

So you can chain your specification(s). The other thing of note is that you can now set the parameter values prior to execution. The reason for this comes next.

No casting on execute()
With the former query above, we would execute it as follows

Collection<Manager> managers = (Collection<Manager>) q.execute(depts);

With JDO 3.2 we can execute it like this

Collection<Manager> managers = q.executeList();

I think we can conclude that this is much cleaner. There is no need to cast the result from the execute method, simply using a different execute method based on what your query is expected to return.

Save as a NamedQuery
You can now construct a query, and save it as a named query, so you don’t need to regenerate it. You do that as follows

Query q = pm.newQuery("SELECT FROM Person WHERE this.lastName == :surname");
q.saveAsNamedQuery("FindPeopleWithSurname");

So from that point forwards when you need that query, you can do the following

Query q = pm.newNamedQuery(Person.class, "FindPeopleWithSurname");
Posted in Uncategorized | Leave a comment

JDO : querying between classes without relation

Let’s say we have the following classes and are using RDBMS for persistence
public class Country
{
long id;
String name;
}

public class Region
{
long id;
String name;
Long countryId;
}

public class District
{
long id;
String name;
Long regionId;
}

So we ignored all good object-oriented guidance and didn’t use real relations. Now lets suppose I want to query for the Country that has a district called “Cross Gates”. We need to write some JDOQL, but don’t have any real relations … just countryId and regionId to navigate through. So we introduce 2 variables to navigate across the 2 “pseudo relations”.

Query q = pm.newQuery("SELECT FROM mydomain.Country " +
"WHERE this.id == reg.countryId && reg.id == dist.regionId && dist.name == 'Cross Gates' " +
"VARIABLES mydomain.Region reg; mydomain.District dist;");

This creates the following SQL
SELECT DISTINCT 'mydomain.Country' AS NUCLEUS_TYPE,A0.ID,A0."NAME"
FROM COUNTRY A0
CROSS JOIN REGION VAR_REG ON A0.ID = VAR_REG.COUNTRYID
CROSS JOIN DISTRICT VAR_DIST ON VAR_REG.ID = VAR_DIST.REGIONID
WHERE VAR_DIST."NAME" = 'Cross Gates'

In our particular case we know that there are no NULL pseudo relations, so would like “INNER JOIN” for performance. So we make use of a DataNucleus extension, like this

Query q = pm.newQuery("... query as before ...");
q.addExtension("datanucleus.query.jdoql.reg.join", "INNERJOIN");
q.addExtension("datanucleus.query.jdoql.dist.join", "INNERJOIN");

So we set joins for both variables to INNER JOIN, and the SQL becomes

SELECT DISTINCT 'mydomain.Country' AS NUCLEUS_TYPE,A0.ID,A0."NAME"
FROM COUNTRY A0
INNER JOIN REGION VAR_REG ON A0.ID = VAR_REG.COUNTRYID
INNER JOIN DISTRICT VAR_DIST ON VAR_REG.ID = VAR_DIST.REGIONID
WHERE VAR_DIST."NAME" = 'Cross Gates'

Easy really ;-) Would clearly have been easier if we had designed our classes around good O-O practice though. And if using some other type of datastore then your datastore would have to allow joins to even attempt this

Posted in JDO, JDOQL | 6 Comments

Want to assist in the development of JDO 4.0?

We have, since 2006, been reliant on the Apache JDO project to push forward the JDO standard. Politics have finally become too much for this arrangement, with Oracle involving lawyers to prohibit progress, and the Apache organisation not being as rapid as it should be at getting releases out of the door. I have now forked their API (now present in DN GitHub). The aim is for this repository to be used to develop JDO 4.0 (and maybe later dependent on people’s involvement), initially to get a TypeSafe query mechanism into the JDO API (originally contributed back in April 2010!!), and bring the API up to date wrt generics in queries themselves. There are also many other features that were requested in Apache JDO JIRA over the last few years that have been simply left, but are needed.

If you, or your company uses JDO this is your chance to get involved and push it forward to cater for these required features. Do you really want to have to cast the return from your Query to be a List etc. You could offer your real-world experience with the JDO API and what you think could be done to make it easier to use.

Should the Apache JDO project “wake up” at some point in the future we could easily enough merge our changes back into their API and discontinue this fork.

Posted in JDO, JDOQL | 7 Comments

Bytecode Enhancement contract in DataNucleus AccessPlatform v4.0

Now in GitHub master, for DataNucleus AccessPlatform v4.0, we have changed the bytecode enhancement contract.

Since the days of JPOX we’ve always used the JDO bytecode enhancement contract as defined in the JDO spec. This has always been adequate to provide the necessary hooks into the object to allow for “transparent persistence“. Saying that though, it does mean that anyone using DataNucleus would always have to have jdo-api.jar in their CLASSPATH, even when using JPA. This was clearly undesirable, but not a large price to pay for easy provision of JPA.

In v4.0 onwards we will enhance classes to implement org.datanucleus.enhancer.Persistable. This is very similar in terms of structure, just that methods are now prefixed “dn” instead of “jdo“, and there is now a method to get the ExecutionContext that is managing the object (whereas before it was a PersistenceManager, which made very little sense for JPA usage).

Why the change?

Oracle is putting significant obstacles in the way of having further releases of the JDO standard, involving lawyers etc. Additionally, following the Apache way, the Apache JDO project has not exactly operated very efficiently in terms of getting releases out of the door. Moreover we want to remove the requirement of having to have jdo-api.jar in the CLASSPATH for JPA usage. This change will also mean that we can, in principle, improve the bytecode enhancement contract to make things more efficient or add on more information to enhance the persistence process without being restricted by what JDO has bothered to standardise.

What does this mean for a typical user?

It means very little in reality, and the majority of applications will work unchanged (apart from having to re-enhance the classes). Some minor things that will change

  • Wherever you use enhanced classes, you will need datanucleus-core.jar in the CLASSPATH
  • JPA users won’t need to have jdo-api.jar in the CLASSPATH
  • Internally DataNucleus now uses its own builtin single-field identity classes, and if you refer to javax.jdo.identity.* classes (for JDO contexts) will auto-convert to our own class for internal use. See this package for the DN internal identity classes. There is really no need to use these JDO builtin classes directly since DataNucleus will always select the most appropriate id type when you have a single PK field.
  • You no longer check if a returned object is of type javax.jdo.spi.PersistenceCapable since it won’t be, instead being a org.datanucleus.enhancer.Persistable.

Please register any concerns/queries in the comments section

Posted in Bytecode, JDO, JPA | 6 Comments

Configuring persistence of fields/properties using TypeConverters with JDO

JPA 2.1 allows a user to specify a converter on the value of a field/property for how it is persisted in the datastore. The way we implement that in DataNucleus is to have the JPA converter as a wrapper to our own internal TypeConverter class. This means that we can make the TypeConverter mechanism available to JDO users too. Here’s how it works. The first thing to do is to define a TypeConverter. As you can see, it requires implementation of just 2 methods, one for use in converting the (raw) field value into the datastore value, and one for use in converting the datastore value back into the (raw) field value. This means that the user has significant flexibility on how their values are stored, and can define their own converters and not rely on them being part of DataNucleus. If we take an example, here we want to store a field as serialised, but not standard Java serialised, instead using the Kryo library. So we define our TypeConverter as

public class KryoSerialiseStringConverter implements TypeConverter
{
    ThreadLocal kryo = new ThreadLocal();
    public Kryo getKryo()
    {
        Object value = this.kryo.get();
        if (value == null)
        {
            value = new Kryo();
            this.kryo.set(value);
        }
        return (Kryo)value;
    }

    public String toDatastoreType(Serializable memberValue)
    {
        if (memberValue == null)
        {
            return null;
        }

        Kryo kryo = getKryo();
        String str = null;
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        Output output = null;
        try
        {
            output = new Output(baos);
            kryo.writeClassAndObject(output, memberValue);
        }
        finally
        {
            output.close();
            str = new String(Base64.encode(baos.toByteArray()));
        }
        return str;
    }

    public Serializable toMemberType(String datastoreValue)
    {
        if (datastoreValue == null)
        {
            return null;
        }

        Kryo kryo = getKryo();
        byte[] bytes = Base64.decode(datastoreValue);
        Object obj = null;
        ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
        try
        {
            Input input = null;
            try
            {
                input = new Input(bais);
                obj = kryo.readClassAndObject(input);
            }
            finally
            {
                try
                {
                    bais.close();
                }
                finally
                {
                    if (input != null)
                    {
                        input.close();
                    }
                }
            }
        }
        catch (Exception e)
        {
            throw new NucleusException("Error Kryo deserialising " + datastoreValue, e);
        }
        return (Serializable)obj;
    }
}

Now we need to register this converter under a name with DataNucleus runtime. Here we define a plugin.xml at the root of the plugin jar as

<type-converter name="kryo-serialise" member-type="java.lang.Serializable" datastore-type="java.lang.String"
converter-class="org.datanucleus.store.types.converters.KryoSerialiseStringConverter"/>

So this converter is internally known as kryo-serialise. We add a MANIFEST.MF to our plugin jar as (something like)

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: DataNucleus Converter using Kryo
Bundle-SymbolicName: org.datanucleus.store.types.converters.kryo;singleton:=true
Bundle-Version: 3.2
Bundle-Localization: plugin
Bundle-Vendor: DataNucleus
Require-Bundle: org.datanucleus
Import-Package: com.esotericsoftware.kryo;version="2.21",
 com.esotericsoftware.kryo.io;version="2.21"

just defining the dependencies of this plugin for OSGi and the plugin mechanism. Now we just need to use it on a sample class and apply the conversion to a field/property.

@PersistenceCapable
public class Sample
{
    @PrimaryKey
    long id;

    @Extension(vendorName="datanucleus", key="type-converter-name", value="kryo-serialise")
    String longString;

    ...
}

Now whenever instances of Sample are persisted the field longString will be converted using our kryo-serialise converter. Easy! You can get the code for this example converter at DataNucleus GitHub repo

Posted in DataNucleus, java, JDO, Persistence | Leave a comment

AccessPlatform 3.3 and JPA 2.1

We will soon be releasing AccessPlatform 3.3. This is coming very soon after 3.2, and the reason for this is that it is simply AccessPlatform 3.2 plus full support for JPA 2.1 (i.e an upgraded datanucleus-api-jpa plugin). From that point both of these “versions” of AccessPlatform will be maintained for a period of time. So what is provided in JPA 2.1 ?

Stored Procedures

You can execute RDBMS Stored Procedures using the JPA API. This API should give you quite complete control over execution of any stored procedure, setting of IN/OUT/INOUT parameters and obtaining result set(s). To give an example

StoredProcedureQuery spq = 
em.createStoredProcedureQuery(procName);
spq.registerStoredProcedureParameter("PARAM1",
Integer.class, ParameterMode.OUT);
boolean val = spq.execute();
Object paramVal = spq.getOutputParameterValue("PARAM1");

So here we have a stored proc that we register an Integer output parameter, and retrieve the value when we execute it. This is just one mode of operation, and you can see more usages in the documentation.

Entity Graphs

JPA has only had an equivalent of the JDO “default fetch group” since its inception. In this release it finally gets some degree of control over what fields are fetched when fetching an object from the datastore. To give an example, we have a class Person and want to pull in a field “bestFriend” under some circumstances but not by default. We define a named EntityGraph in metadata


@Entity
@NamedEntityGraph(name="includeFriend", attributeNodes= {@NamedAttributeNode("bestFriend")})
public class Person
{
@OneToOne;
Person bestFriend;

...
}

and now we want to use this entity graph when loading an object of this type. We do this as follows.

EntityGraph friendGraph = em.getEntityGraph("includeFriend");
Properties props = new Properties();
props.put("javax.persistence.loadgraph", friendGraph);
MyClass myObj = em.find(Person.class, id, props);

So we retrieved the EntityGraph, and then used it in the find method. Equally we could have used it in a Query. You can read more about this topic in the documentation.

Schema Generation

JPA 2.1 allows generation of the schema as an up front task, or via Persistence.generateSchema(). You can specify this by making use of persistence properties, for example javax.persistence.schema-generation.database.action set to create. See the available persistence properties for details.

Foreign-Keys and Indexes

JPA 2.1 adds on the ability to specify RDBMS schema foreign-keys and indexes, for use during schema generation. By default a JPA implementation is free to generate whatever foreign-keys it decides are appropriate, but this ability allows a user to override this and control what is generated.


@OneToOne
@JoinColumn(name="BESTFRIEND_ID", foreignKey=
@ForeignKey(name="BESTFRIEND_FK", foreignKeyDefinition=
"FOREIGN KEY (BESTFRIEND_ID) REFERENCES PERSON (PERSON_ID) ]"))
Person bestFriend;

which will create a foreign key called “BESTFRIEND_FK” for this purposes. Similarly we can define indexes on a table of a class.

@Entity
@Table(indexes={@Index(name="FIRSTNAME_IDX", columnList="FIRST_NAME")})
public class Person
{
@Column(name="FIRST_NAME")
String firstName;

...
}

so the firstName field is now indexed. You can read more about this topic in the documentation.

Criteria UPDATE/DELETE queries

Whilst the JPA Criteria API is overengineered and verbose it now has the ability to generate UPDATE and DELETE queries. For example


CriteriaUpdate crit = qb.createCriteriaUpdate(Person.class);
Root candidate = crit.from(Person.class);
candidate.alias("p");
crit.set(candidate.get(Person_.firstName), "Freddie");
Predicate teamName = qb.equal(candidate.get(Person.firstName), "Fred");
crit.where(teamName);
Query q = em.createQuery(crit);
int num = q.executeUpdate();


which will create the JPQL “UPDATE Person p SET p.firstName = ‘Freddie’ WHERE p.firstName = ‘Fred'”. You can do similar things for DELETE queries. You can read more about this topic in the documentation

Attribute Converters

By default a JPA implementation will persist a field in a datastore column of its chosen type. You can now override this to use a converter, performing the conversion from the field type to the datastore type in your converter class. The example we use in the documentation is where we have a field in our class of type URL and want to persist this as a String-type in the datastore (VARCHAR, CHAR etc).


public class URLStringConverter implements AttributeConverter
{
public URL convertToEntityAttribute(String str)
{
if (str == null)
{
return null;
}

URL url = null;
try
{
url = new java.net.URL(str.trim());
}
catch (MalformedURLException mue)
{
throw new IllegalStateException("Error converting the URL", mue);
}
return url;
}

public String convertToDatabaseColumn(URL url)
{
return url != null ? url.toString() : null;
}
}

and then in our class that has a URL field we mark the field to use this converter


@Basic
@Convert(converter=URLStringConverter.class)
URL url;

You can read more about this topic in the documentation.

JPQL FROM “ON” clauses

When joining in JPQL previously we could not add additional constraints on the join. You can now do this using the “ON” clause.


List result = em.createQuery(
"SELECT Object(A) FROM Account A LEFT OUTER JOIN A.login L ON L.userName = 'fred'").getResultList();

JPQL “FUNCTION”

No matter what features you put in a query language some people will always want to make use of SQL functions specific to a particular datastore. With JPQL you can now do this using the FUNCTION keyword, like this


Query q = em.createQuery(
"SELECT p FROM Person p WHERE FUNCTION('UPPER', p.firstName) = 'FRED'");

As you can see, JPA 2.1 is a minor iteration on JPA, and you can now benefit from all of these features in DataNucleus AccessPlatform 3.3

Posted in AccessPlatform, JPA | Leave a comment