The blog has moved to a new address

New blog address is https://blog.rpuch.com

Testing an Aspect in a Spring application

Sometimes, an Aspect is the best tool to solve a task at hand. But how can we unit-test it? An obvious solution is to start up the whole application context, including the Aspect, and then test how it behaves, but this will be actually an integration test; such tests are heavy, and you have less control over the situations you can model in such a test (for example, it may be pretty difficult to simulate an exceptional situation).

Actually, there are two things that are interesting from the testing perspective an an aspect:

1. The business logic that the aspect executes when triggered
2. The pointcut expression(s) which trigger aspect execution

Even the first of them is not so easy to test because you need an instance of ProceedingJoinPoint which is cumbersome to implement or mock (and it is not recommended to mock external interfaces, as it is explained in Growing Object-Oriented Software, Guided by Tests, for example).

The solution

Let's imagine that we have an aspect that must throw an exception if a method's first argument is null, otherwise allow the method invocation proceed.

It should only be applied to controllers annotated with our custom @ThrowOnNullFirstArg annotation.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

@Aspect public class ThrowOnNullFirstArgAspect { @Pointcut("" + "within(@org.springframework.stereotype.Controller *) || " + "within(@(@org.springframework.stereotype.Controller *) *)") private void isController() {} @Around("isController()") public Object executeAroundController(ProceedingJoinPoint point) throws Throwable { throwIfNullFirstArgIsPassed(point); return point.proceed(); } private void throwIfNullFirstArgIsPassed(ProceedingJoinPoint point) { if (!(point.getSignature() instanceof MethodSignature)) { return; } if (point.getArgs().length > 0 && point.getArgs()[0] == null) { throw new IllegalStateException("The first argument is not allowed to be null"); } } }

We could test it like so:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

public class ThrowOnNullFirstArgAspectTest { private final ThrowOnNullFirstArgAspect aspect = new ThrowOnNullFirstArgAspect(); private TestController controllerProxy; @Before public void setUp() { AspectJProxyFactory aspectJProxyFactory = new AspectJProxyFactory(new TestController()); aspectJProxyFactory.addAspect(aspect); DefaultAopProxyFactory proxyFactory = new DefaultAopProxyFactory(); AopProxy aopProxy = proxyFactory.createAopProxy(aspectJProxyFactory); controllerProxy = (TestController) aopProxy.getProxy(); } @Test public void whenInvokingWithNullFirstArg_thenExceptionShouldBeThrown() { try { controllerProxy.someMethod(null); fail("An exception should be thrown"); } catch (IllegalStateException e) { assertThat(e.getMessage(), is("The first argument is not allowed to be null")); } } @Test public void whenInvokingWithNonNullFirstArg_thenNothingShouldBeThrown() { String result = controllerProxy.someMethod(Descriptor.builder().externalId("id").build()); assertThat(result, is("ok")); } @Controller @ThrowOnNullFirstArg private static class TestController { @SuppressWarnings("unused") String someMethod(Descriptor descriptor) { return "ok"; } } }

The key part is inside the setUp() method. Please note that it also allows to verify the correctness of your pointcut expression, so it solves both problems.

Of course, in a real project it is better to extract the 'proxy contstruction' code to some helper class to avoid code duplication and make the intentions clearer.

Cassandra cqlsh client, OperationTimedOut and request timeouts

I'm used to mysql command line client. If a query runs for a long time, it just keeps running. This is very useful if you feed a script (i.e. a sequence of queries) to mysql client for execution: no matter how long each query executes, the queries are run serially. If nothing breaks in the middle, they all execute successfully

It turned out that with the default settings Cassandra's cqlsh (command line client) behaves differently. All of a sudden, my script (a sequence of DDL queries run in the beginning of an integration test to prepare database) has failed. The first error was OperationTimedOut, but the following ones were caused by the fact that the first query did not yet finish. For example, in my case the first query was DROP KEYSPACE, while the second was CREATE KEYSPACE with the same name. Of course, if failed, and the following CREATE TABLE  queries failed as well.

Why does this happen? Because cqlsh has a limit (by default it is 10 seconds, according to documentation). If your query runs more than this limit, the client just fails with OperationTimedOut error message, but the query is still running on the server.

OK, how do we disable this limit, or at least configure it to be long enough?

Good news: cqlsh in Cassandra 2.1.16 has --request-timeout command line parameter and you can specify the limit there (in seconds). --request-timeout 3600 would be a good start.

Bad news: cqlsh in Cassandra 2.1.12 does NOT have that parameter yet, so this parameter is not that universal.

By the way, version reported by cqlsh (with the usual --version) is strange. I tried it with cqlsh included into Cassandra distribution for Cassandra 2.1.8, 2.1.12, 2.1.16, and in all these cases the version was reported as 5.0.1, even though 2.1.16 reports support for --request-timeout (and really supports it) and the other two versions don't.

But let's return to out limit.

Good news: ~/.cassandra/cqlshrc file allows to define this timeout in [connection] section.

Bad news: the documentation is not accurate. Although it says that the option was added in version 2.1.1 and is called request_timeout, and this is true for 2.1.16, it is NOT true for 2.1.12. In it, you have to call the option client_timeout. Moreover: in 2.1.12, according to this article, you could completely disable the timeout by assigning None. Alas, in 2.1.16 (with request_timeout) this does not work.

It is not possible to (reliably) completely disable the timeout. If you set request_timeout to 0, this will mean that any request will timeout. Negative values cause errors. So the only option is to set it to some large value (like the abovementioned 3600 seconds).

So, a kinda universal way to make sure your integration tests don't stumble upon this, is to put the following in your ~/.cassandra/cqlshrc:

[connection]
request_timeout = 3600
client_timeout = 3600

BTW, how come that DROP KEYSPACE for a keyspace with a few tables with no data in them where the cluster contains just one node could not fit into the default timeout (presumably 10 seconds) on a machine with a decent HDD which was not overloaded? It's a different story...

Peculiarities of @ControllerAdvice in Spring MVC

What is @ControllerAdvice?

@ControllerAdvice annotation is used to define some attributes for many Spring controllers (@Controller) at a time.

For example, it can be used for a centralized exception control (with @ExceptionHandler annotation). This post will concentrate on this use.

Global and specific advices

If a @ControllerAdvice does not have any selectors specified via annotation attritubes, it defines a global advice which affects all the controllers, and its @ExceptionHandler will catch all the exceptions thrown with handle methods (and not just these exceptions, see below).

In Spring 4.0, the ability to define specific advices was added. @ControllerAdvice now has some attributes with which we can define advice selectors. These selectors define the scope of the advice, i.e. the exact set of controllers which will be adviced by it.

Global @ExceptionHandler catches 'no man's' exceptions

'No man's' exceptions are exceptions which occur before the handler to process the request is obtained.

So, if we don't specify any attributes at @ControllerAdvice annotation, its @ExceptionHandler method will catch even HttpRequestMethodNotSupportedException when someone tries to issue a GET request to our POST-only controller.

But if we define a class, annotation or an existing basePackage, then the advice will not be global anymore and will not catch 'no man's' exceptions.

basePackages does not work for controllers proxied with Proxy

To have the ability to intercept controller invocations (for example, to handle exceptions), our advice has to wrap controller instance in a proxy. Proxy creation options:

1. If the controller class has at least one implemented interface, an interface-based proxy is created using Proxy class.
2. If the controller class does not implement any inferfaces, then CGLIB is used; proxy class is created at runtime; this class extends our initial controller class.

For CGLIB option, Spring's ControllerAdviceBean determines correctly the controller class package: it just takes the superclass of CGLIB-generated class (this will be the real controller class as written by us) and then takes its package. In this case, basePackages of @ControllerAdvice works correctly.

But for the interface-proxy based option Spring has no ability to determine the real class of an instance wrapped with Proxy, so it tries to take the package of the Proxy instance. But proxy.getClass().getPackage() returns null! In Spring 4.0.5 this even causes a NullPointerException. In 4.0.9 the NPE was fixed, but the package will not be determined correctly, so basePackages will not work.

To sum up:

1. If the controller class has at least one implemented interface, an interface-based proxy is created using Proxy class, and basePackages of @ControllerAdvice DOES NOT work.
2. If the controller class does not implement any inferfaces, then CGLIB is used and basePackages attribute works.

What can we do?

Use annotations attribute. First, let's create an annotation like

@ControllerAdvicedByMyAdvice

Then we annotate with this annotation all the controllers to which we want to apply the advice. And then we annotate the advice:

@ControllerAdvice(annotations = {ControllerAdvicedByMyAdvice.class})

This approach seems more reliable than the utilization of basePackages attribute.

Clean, safe and concise read-only Wicket Model with Java 8 Lambdas

Wicket framework uses models (IModel implementations) to bind data to components. Let's say you want to display properties of some object. Here is the data class:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

public class User implements Serializable { private final String name; private final int age; public User(String name, int age) { this.name = name; this.age = age; } public String getName() { return name; } public int getAge() { return age; } }

You can do the following to display both its properties using Label components:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

public class AbstractReadOnlyModelPanel extends Panel { public AbstractReadOnlyModelPanel(String id, IModel<User> model) { super(id, model); add(new Label("name", new AbstractReadOnlyModel<String>() { @Override public String getObject() { return model.getObject().getName(); } })); add(new Label("age", new AbstractReadOnlyModel<Integer>() { @Override public Integer getObject() { return model.getObject().getAge(); } })); } }

Straight-forward, type-safe, but not too concise: each label requires 6 lines of code! Of course, we can reduce this count using some optimized coding conventions and so on, but anyway, anonymous classes are very verbose.

A more economical way (in terms of lines and characters to type and read) is PropertyModel.

1 2 3 4 5 6 7 8

public class PropertyModelPanel extends Panel { public PropertyModelPanel(String id, IModel<User> model) { super(id, model); add(new Label("name", PropertyModel.of(model, "name"))); add(new Label("age", PropertyModel.of(model, "age"))); } }

It is way shorter and still pretty intuitive. But it has drawbacks:

• First of all, it is not safe as the compiler does not check whether property named "age" exists at all!
• And it uses reflection which does not make your web-application faster. This does not seem to be critical, but it is still a little drawback.

Luckily, Java 8 introduced lambdas and method references which allow us to create another model implementation. Here it is:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

public class GetterModel<E, P> extends AbstractReadOnlyModel<P> { private final E entity; private final IModel<E> entityModel; private final IPropertyGetter<E, P> getter; private GetterModel(E entity, IModel<E> entityModel, IPropertyGetter<E, P> getter) { this.entity = entity; this.entityModel = entityModel; this.getter = getter; } public static <E, P> GetterModel<E, P> ofObject(E entity, IPropertyGetter<E, P> getter) { Objects.requireNonNull(entity, "Entity cannot be null"); Objects.requireNonNull(getter, "Getter cannot be null"); return new GetterModel<>(entity, null, getter); } public static <E, P> GetterModel<E, P> ofModel(IModel<E> entityModel, IPropertyGetter<E, P> getter) { Objects.requireNonNull(entityModel, "Entity model cannot be null"); Objects.requireNonNull(getter, "Getter cannot be null"); return new GetterModel<>(null, entityModel, getter); } @Override public P getObject() { return getter.getPropertyValue(getEntity()); } private E getEntity() { return entityModel != null ? entityModel.getObject() : entity; } }

... along with its support interface:

1 2 3

public interface IPropertyGetter<E, P> { P getPropertyValue(E entity); }

And here is the same panel example rewritten using the new model class:

1 2 3 4 5 6 7 8

public class GetterModelPanel extends Panel { public GetterModelPanel(String id, IModel<User> model) { super(id, model); add(new Label("name", GetterModel.ofModel(model, User::getName))); add(new Label("age", GetterModel.ofModel(model, User::getAge))); } }

The code is almost as concise as the version using PropertyModel, but it is:

• type-safe: the compiler will check the actual getter type
• defends you from typos better, because compiler will check that the getter actually exists
• fast as it just uses regular method calls (2 per getObject() call in this case) instead of parsing property expression and using reflection

Here are the drawbacks of the described approach in comparison with PropertyModel:

• It's read-only while PropertyModel allows to write to the property. It's easy to add ability to write using setter, but it will make code pretty clumsy, and we'll have to be careful and not use getter from one property and setter from another one.
• PropertyModel allows to reference nested properties using the dot operator, for instance using "outerObject.itsProperty.propertyOfProperty" property expression.

But anyway, when you just need read-only models, GetterModel seems to be an interesting alternative to the PropertyModel.

And here is a little bonus: this model implementation allows to use both models and plain data objects as sources. We just need two factory methods: ofModel() and ofObject(), and we mimic the magical universality of PropertyModel (which accepts both models and POJOs as first argument) with no magic tricks at all.

XSLT to convert log4j.xml config to logback.xml config

Logback is a modern logging framework widely used nowadays. It has a drop-in replacement for a log4j 1.2 (the previous favorite): logback-classic. But a little problem arises if you have an application which uses log4j and want to migrate it to logback: configuration.

log4j has two configuration formats: log4j.properties and log4j.xml. For the former, everything is fine: there is log4j.properties translator script.
But for log4j.xml there doesn't seem to be any convertion tool available, and logback.xml does not understand the unconverted log4j.xml files.
So here is an XSLT which allows to convert log4j.xml files to the corresponding logback.xml configurations.
And here is an example. We have the following log4j.xml file:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

<?xml version="1.0" encoding="UTF-8"?> <log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/"> <appender name="default" class="org.apache.log4j.ConsoleAppender"> <param name="target" value="System.out"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%d %t %p [%c] - %m%n"/> </layout> </appender> <appender name="log4jremote" class="org.apache.log4j.net.SocketAppender"> <param name="RemoteHost" value="10.0.1.10"/> <param name="Port" value="4712"/> <param name="ReconnectionDelay" value="10000"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="[my-host][%d{ISO8601}]%c{1}%n%m%n"/> </layout> <filter class="org.apache.log4j.varia.LevelRangeFilter"> <param name="LevelMin" value="ERROR"/> <param name="LevelMax" value="FATAL"/> </filter> </appender> <logger name="com.somepackage"> <level value="INFO"/> </logger> <root> <level value="INFO"/> <appender-ref ref="default"/> </root> </log4j:configuration>

Using Xalan to convert it:

java -cp xalan.jar:xercesImpl.jar:serializer.jar:xml-apis.jar org.apache.xalan.xslt.Process -IN log4j.xml -XSL log4j-to-logback.xsl -OUT logback.xml

And here is the result:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

<?xml version="1.0" encoding="UTF-8"?><configuration scanPeriod="10 seconds" scan="true"> <appender name="default" class="ch.qos.logback.core.ConsoleAppender"> <target>System.out</target> <encoder> <pattern>%d %t %p [%c] - %m%n</pattern> </encoder> </appender> <appender name="log4jremote" class="ch.qos.logback.classic.net.SocketAppender"> <remoteHost>10.0.1.10</remoteHost> <port>4712</port> <reconnectionDelay>10000</reconnectionDelay> <!-- this is NOT needed tor this logger, so it is commented out --> <!-- <layout> <pattern>[my-host][%d{ISO8601}]%c{1}%n%m%n</pattern> </layout> --> <filter class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>ERROR</level> </filter> </appender> <logger name="com.somepackage" level="INFO"/> <root level="INFO"> <appender-ref ref="default"/> </root> </configuration>

The script translates the basic loggers (console/email/file/syslog/socket).

Here is github repository: https://github.com/rpuch/log4j2logback

Spring Security 3.2+ defaults break Wicket Ajax-based file uploads

A couple of days ago we have run into a bug: we found that file uploads in our Wicket application has broken. Instead of working as expected, upload button did not work, instead a message appeared in the browser console (this one is for Chrome):

Refused to display 'http://localhost:8084/paynet-ui/L7ExSNbPC4sb6TPJDblCAkN0baRJxw3q6-_dANoYsTD…QK61FV9bCONpyleIKW61suSWRondDQjTs8tjqJJOpCEaXXCL_A%2FL7E59%2FTs858%2F9QS3a' in a frame because it set 'X-Frame-Options' to 'DENY'.

That seemed strange, because X-Frame-Options relates to frames which we didn't use explicitly. But when file upload is made using Ajax, Wicket carries this out using an implicit Frame.
Spring Security started adding this header starting with version 3.2, so it was actually an upgrade to Spring Security 3.2 that broke file uploads. To sort this out, it was sufficiently to change the X-Frame-Options value from DENY to SAMEORIGIN using the following snippet in web security configuration (created using @Configuration-based approach):

http
    .headers()
        .contentTypeOptions()
        .xssProtection()
        .cacheControl()
        .httpStrictTransportSecurity()
        .addHeaderWriter(new XFrameOptionsHeaderWriter(XFrameOptionsHeaderWriter.XFrameOptionsMode.SAMEORIGIN))

File uploads work now, the quest is finished.

ssh reverse tunnel for external internet

Today I needed to create an ssh reverse tunnel, so when someone connects to a server's port, he is forwarded to my PC's port. A usual thing, nothing special.
I googled a bit and found a lot of examples.

ssh -R 8888:localhost:80 user@host.com

So when a TCP connection is opened to host.com:8888, it actually gets forwarded to my localhost:80.

Pretty simple. But it did not work.

After googling and googling and finding all the same simple but not working examples, I finally did the right thing: reading manual.
It turned out that when server-side bind address is not specified (as it is in my case) it binds to 127.0.0.1 which is not visible from the outside internet. You can specify * or empty string as bind address to make sshd bind to all interfaces, or even specify an IP address to bind to, but any of these will work only if sshd server has GatewayPorts enabled.

Okay, after adding the following line

GatewayPorts on

to the /etc/ssh/sshd_config and restarting sshd it has started to work. Command is:

ssh -R *:8888:localhost:80 user@host.com

One more little thing learned.

Working around a Selenium Maven plugin problem on Firefox 3

A couple of days ago I tried Spring Roo. Its ten-minute tutorial offers you to run Selenium tests using the following command:

mvn selenium:selenese

I launched it, but console window freezed and nothing happened.
Looks like it's a problem with Selenium+Firefox 3 (possibly also Ubuntu).
For me, a solution was simple: just change selenium-maven-plugin version to 1.0.1 (which is the most recent stable version at the time of writing). So plugin definition in Roo-generated application's pom.xml will look like

<plugin>
   <groupid>org.codehaus.mojo</groupid>
   <artifactid>selenium-maven-plugin</artifactid>
   <version>1.0.1</version>
   <configuration>
     <suite>src/main/webapp/selenium/test-suite.xhtml</suite>
     <browser>*firefox</browser>
     <results>${project.build.directory}/target/selenium.txt</results>
     <starturl>http://localhost:4444/</starturl>
   </configuration>
</plugin>

The same applies to using Selenium on Ubuntu+Firefox 3 through JUnit tests: I had to use version 1.0.1 of selenium-maven-plugin to make this work.

Testing RTMP streaming with JW Player

When configuring an RTMP streaming in Flash Player (see previous post for an example), I will use the same examples in this post), it may be hard to tell why it does not work: was URL incorrect? Or you just missed something in your player? In such a case it may be helpful to use JW Player online player constructor.

1. Go to http://www.longtailvideo.com/support/jw-player-setup-wizard
2. Select flvplayer with an rtmp stream
3. In File properties section, set File to your stream name, like 'video1.flv' or 'mp4:video2.mp4' (without quotes)
4. In External Communitation section, set streamer to the part of URL without your stream name, like rtmp://abcdefghijklmn.cloudfront.net/cfx/st
5. Press Update Preview & Code button
6. Press on the player below. If it plays your video correctly, then your bucket+distribution are configured correctly and URL is correct, too

Note that values for streamer on step 4 and for file on step 3 give you the full streaming URL when concatenated through a slash (/) as described in the aforementioned post.

Streaming h264 video from Cloudfront to Flash Player: how-to

Flash makes it easy to stream a video, at least in theory. With Amazon Cloudfront it's even easier - you just create a bucket, upload a video to it, and it's ready to be streamed via Flash Player.
Or it seems so.
There're some details which are documented poorly. I will try to fill this hole.

Brief algorithm

To stream a video (including h264) from Cloudfront, you need to do the following:

1. Create an AWS account
2. Create an S3 bucket for you videos
3. Create a Cloudfront distribution for it

Then, for each file you wish to stream, you will have to do the following:

1. Upload it to your S3 bucket as a public resource
2. Stream it through Flash Player using correct URL

Let's go!

Configuring software

To work with S3 and Cloudfront you need some special software. For instance, there's an extension for Firefox called S3Fox. I personally prefer using s3cmd command line program to manage S3, I will provide examples for s3cmd.
Anyway, when registering in AWS, you will get a couple of pretty long and ugly strings. Those are Access Key and Secret Key. You have to configure your AWS-related software to use these keys for authentication. For s3cmd, the command is

s3cmd --configure

Creating an AWS account

To create an AWS (Amazon Web Services) account.

Creating an S3 bucket for your videos

First, you have to create an S3 (Simple Storage Service) bucket for your videos, it will feed them to your Cloudfront distributions.

s3cmd mb s3://your-bucket-name.you-site.com

Bucket name must be unique among all buckets, so it may seem reasonable to call your bucket based on your domain name.

Creating a Cloudfront distribution for your videos

A distribution must be bound to a bucket. Distribution will distribute any file with public access from a bucket bound to it.
Distribution may be created using mentioned software or using AWS Console web interface.
When creating, select 'Streaming' as type as we're going to stream video. Note that you can create several distributions for your bucket, so you may also have access to your bucket via HTTP if you wish. Of course, specify your bucket to which distribution will be bound.
Distribution takes some time to create. After it completes, you will get a public domain name for it, this will be domain name you will need to construct video URLs.

Uploading a video to bucket

When uploading a video to your bucket, make sure that you make it publicly readable.

s3cmd --acl-public put video.mp4 s3://your-bucket-name.your-site.com/video.mp4

Note that you don't need to make your whole bucket publicly readable, only files.

Constructing a URL

It's not a trivial task to construct a correct URL for streaming, because RTMP (and we are forced to use RTMP for streaming) has its own logic of constructing URLs. And there's still another catch with correct format prefix which you must use if you stream mp4 (or mp3). So I will provide some examples.
The common schema for Cloudfront is the following:

rtmp://distribution-domain-name/cfx/st/[type_prefix]stream_name

• for flv, you don't need a type prefix, and you may supply extension name while it's recommented to omit it.
• for mp4, you have to use type prefix 'mp4:' (with no quotes) and you must supply extension (mov, mp4, aac, f4v, m4v, m4a - any of these).
• for mp3, you have to use type prefix 'mp3:'.

Let's assume you have some video and audio files stored as s3://your-bucket-name.your-site.com/video1.flv, s3://your-bucket-name.your-site.com/video2.mp4 and s3://your-bucket-name.your-site.com/audio.mp3. Assuming that your Cloudfront distribution domain name is abcdefghijklmn.cloudfront.net, you'll have to use the following URLs:

rtmp://abcdefghijklmn.cloudfront.net/cfx/st/video1
rtmp://abcdefghijklmn.cloudfront.net/cfx/st/mp4:video2.mp4
rtmp://abcdefghijklmn.cloudfront.net/cfx/st/mp3:audio.mp3

Links

http://livedocs.adobe.com/flex/3/langref/flash/net/NetStream.html#play() - here is more info on type prefixes