JSON Assert 0.2.1 released
I’m pleased to announce that JSON Assert version 0.2.1 got released! The following features have been added
- AssertJ support
- Bumped up JSON Path to 2.2.0
- Fixed a bug related to chars escaping
- I’ve changed README.md to README.ascidoc
Breaking changes
I’ve changed the groupId and packages of the library so you have to do the migration of both.
GroupId
Was:
com.blogspot.toomuchcoding
Is:
com.toomuchcoding.jsonassert
Package
Was:
com.blogspot.toomuchcoding.jsonassert
Is:
com.toomuchcoding.jsonassert
BTW
I’ve created a new page on the website OSS. You can check out which OSS projects I’ve created or have contributed sth there. If you don’t care - just ignore it ;)
Goodbye Blogspot welcome Octopress!
I’m more than happy to announce that I’ve finally migrated from Blogspot to a decent blogging technology - Octopress! Thanks to Tomek Dziurko who was the one that suggested to choose that technology.
Also as you can see I’ve finally bought a domain for the Too Much Coding blog which is https://toomuchcoding.com. I don’t even know why I’m writing it since you can see the address in your browser ;) You can also send me an email at blog (at) toomuchcoding.com.
Even though initially I had some doubts about choosing Octopress I have to admit that it seems like an awesome technology and you should definitely give it a try!
P.S. We’re looking for sponsors for the upcoming Warsaw GR8 Day Conference (19.03.2016). Over 60 people have already registered! Contact us to be a part of this gr8 event!
JSON Assert lib released
Rationale
In AccuREST (the Consumer Driven Contracts implementation library) we're creating tests of the server side. For more information on what is AccuREST and what Consumer Driven Contracts is check the AccurREST wiki. Anyways, we're checking if the response from the server matches the one described in the contract.So having such a Groovy DSL:
io.codearte.accurest.dsl.GroovyDsl.make {
priority 1
request {
method 'POST'
url '/users/password'
headers {
header 'Content-Type': 'application/json'
}
body(
email: $(stub(optional(regex(email()))), test('abc@abc.com')),
callback_url: $(stub(regex(hostname())), test('https://partners.com'))
)
}
response {
status 404
headers {
header 'Content-Type': 'application/json'
}
body(
code: value(stub("123123"), test(optional("123123"))),
message: "User not found by email = [${value(test(regex(email())), stub('not.existing@user.com'))}]"
)
}
}
Resulted in creation of the following server side response verification
given:
def request = given()
.header('Content-Type', 'application/json')
.body('{"email":"abc@abc.com","callback_url":"https://partners.com"}')
when:
def response = given().spec(request)
.post("/users/password")
then:
response.statusCode == 404
response.header('Content-Type') == 'application/json'
and:
DocumentContext parsedJson = JsonPath.parse(response.body.asString())
!parsedJson.read('''$[?(@.code =~ /(123123)?/)]''', JSONArray).empty
!parsedJson.read('''$[?(@.message =~ /User not found by email = \\[[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,4}\\]/)]''', JSONArray).empty
AccuREST users stated that their biggest problem is this part:
!parsedJson.read('''$[?(@.code =~ /(123123)?/)]''', JSONArray).empty
!parsedJson.read('''$[?(@.message =~ /User not found by email = \\[[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,4}\\]/)]''', JSONArray).empty
They said that JSON Paths are too difficult for them to read.
That's why I've created the JSON Assert library. So that instead of the aforementioned code one gets sth like this:
assertThatJson(parsedJson).field('code').matches('123123?')
assertThatJson(parsedJson).field('message').matches('User not found by email = \\[[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,4}\\]/)]');
How to add it to your project
If your using Gradle just add (check the latest version number):
testCompile `com.blogspot.toomuchcoding:jsonassert:0.1.2`
and if Maven just add:
<dependency>
<groupId>com.blogspot.toomuchcoding</groupId>
<artifactId>jsonassert</artifactId>
<version>0.1.2</version>
</dependency>
How to use it
Since almost everything in JSON Assert is package scoped you have access to two public classes. One of which is the JsonAssertion class. It gives you a couple of public methods that give you the entry point to the fluent interface of the library.
You can check the JavaDocs of the JsonVerifiable interface in order to see what kind of methods can be used.
Examples
Best examples are tests. I'll show you a couple of them here.
Example 1
Having a JSON:
[ {
"some" : {
"nested" : {
"json" : "with value",
"anothervalue": 4,
"withlist" : [
{ "name" :"name1"} , {"name": "name2"}, {"anothernested": { "name": "name3"} }
]
}
}
},
{
"someother" : {
"nested" : {
"json" : "with value",
"anothervalue": 4,
"withlist" : [
{ "name" :"name1"} , {"name": "name2"}
]
}
}
}
]
$[*].some.nested.withlist[*].anothernested[?(@.name == 'name3')]
assertThat(json).array().field("some").field("nested").array("withlist").field("anothernested").field("name").isEqualTo("name3")
Example 2
Having a JSON:
{
"property1": [
{ "property2": "test1"},
{ "property3": "test2"}
]
}
$.property1[*][?(@.property2 == 'test1')]
assertThat(json).array("property1").contains("property2").isEqualTo("test1")
Future plans
It would be nice to:
- integrate with AssertJ
- add more JSON Path features (functions, filters etc.)
Links
Post on Spring blog
Since December I'm working for Pivotal and the Spring Cloud team. I've just posted my first blog post on the Spring blog. Check it out!
Testing Spring Cloud Projects
Business value gone wild
- the more and more frequent "it's not my problem" approach in the IT industry running in a corporation.
- the "business value" frenzy of the management
TL;DR : the more you care in a corporate enterprise the worse for you. Eventually some developers will hate your ideas of quality and standards because they are paid to tap the keys. Your management will fire you for not bringing "business value". The faster you embrace it, the better for you - you'll start searching for a new job sooner.
Features are not only functionalities
Let's define some facts: IT is paid by the business. Business wants features. IT has to deliver features to gain money. That's a fact and our reality. Even if you hear from your managers that "cleaning technical debt is a necessity" what they really think is:
- they don't get technical aspects, but they trust the engineers
- they don't care about technical aspects and they won't listen to any of the programmers' advice
If you have the latter business people then most likely you're in this situation:
and actually you should be doing such a shift:
https://www.technalytical.com/2012/04/aesthetical-cable-management-before-and.html |
Now for the tricky part. Now change the word "business" to "developer" and everything is still valid.
"Delivering a feature" it's not only coding some functions in whatever language you are using. It's not taking a keyboard and pressing the keys to make the functionality work. If this is your approach then you're a key tapper. Tapping keys to get things done.
Programming is more than tapping keys
I hope that nobody feels offended by this term "key tapper". I'm not trying to be offensive - I'm just describing what I saw in my career. In my opinion there are a couple of different types of IT guys:
- there are people for whom programming is a passion. They put a lot of energy and effort to make things better
- there are also IT guys for whom programming isn't a passion, but still put (sucessfully) a lot of energy and effort in order to make things better just because they want to be honest and valuable employees (thanks Michal Szostek)
- there are people for whom programming is not a passion and they just come to work and tap the keys
- there are others who would love to do stuff properly but the business is breathing at their necks to do stuff in a bad way because the "deadlines are coming".
- there are positions where people last. They come and simulate work. They lie, talk a lot and delegate work so that there is some impression of progress
Regardless of the position, if one doesn't focus on quality and just taps in the functionality then:
- even if he provides the business feature it might badly influence other people (introducing coupling between modules, breaking encapsulation etc.)
- the functionality might be written in such a way that you will result in the global timeout of the whole system
- you're not thinking about the company standards (passing of CorrelationID for instance), that will break the approaches set in the company. This in effect will lead in increased time needed to provide support
- writing the next functionality will take more time than the previous one
I don't have time for this - it's not my problem. I've delivered my business feature and this is what I'm paid for. What you're referring to is not of my interest.Now imagine that you join a project which is full of such developers and you're asked to fix a bug:
Technical changes are not bringing money
We have to educate both the business and the developers: writing features and providing business value is actually a sum of a coded and tested functionality with technical advancement. What are those? Code refactoring, introduction of new approaches, migrations from one way of doing things in one way to another. For example:
- version control system (e.g. SVN to Git)
- build system (e.g Maven to Gradle)
- UI framework (e.g. Vaadin to AngularJS)
- library versions (e.g. Spring 3.0 to Spring 4.0)
- going from deployment to application servers to embedded servlet containers (e.g. Glassifsh to embedded JAR with Jetty)
https://abdulinnewzealand.wordpress.com/2012/12/03/new-things-from-my-visit-to-new-zeland/ |
- libraries
- approach to testing
- approach to deployment
- approach to running the application
Then you can tell your business that they will pay A LOT of money for the support. The learning curve will be gigantic for the newcomers. But hey! It's better to code a new functionality in the meantime right?
Seemingly all the developers would like to see the effect of those migrations and standardization. Everybody wants this to happen but who should actually do it? When asked about this you might hear:
I don't have time for this - it's not my problem. I've delivered my business feature and this is what I'm paid for. What you're referring to is not of my interest.
Stupid idea
Introduce the following flow of working in IT:
- the "coding team" writes a business feature and pushes it to master
- the "clean code team" rewrites the code according to the clean code standards
- the "technical team" introduces the technical standards for the written piece of code
- the "migration team" migrates the code from one approach to another
Introduce... caring! Invest a lot of time and effort in educating business and developers that you have to take care of the code quality. Imagine where your company would be if every programmer would focus for 1 hour per day to manage the technical debt. If your managers don't understand the importance of clearing that debt, then you should consider changing jobs cause it's going to get worse with every single push to the repo.
You are an engineer!
Developing a feature is not just typing in code that compiles and makes the tests pass. Maybe the constant breathing of the project manager on your neck made you forget about this but you are an engineer. Following Wikipedia:
An engineer is a professional practitioner of engineering, concerned with applying scientific knowledge, mathematics, and ingenuity to develop solutions for technical, societal and commercial problems. Engineers design materials, structures, and systems while considering the limitations imposed by practicality, regulation, safety, and cost.[1][2] The word engineer is derived from the Latin words ingeniare ("to contrive, devise") and ingenium("cleverness").[3][4]
I don't have time for this - it's not my problem. I've delivered my business feature and this is what I'm paid for. What you're referring to is not of my interest.
My schedule is tight but I'll fix the issues that you suggested. I understand that delivering business value means writing a feature and making the technical progress as a company. This is what I'm paid for and what you are referring to is part of my duties.Unfortunately there is one problem with this approach...
Are you an engineer that has a say? You're gonna get fired!
Yes, if you start caring in a corporate enterprise you will eventually get fired. Business prefers people who nod their heads and agree to everything. After some time quality becomes a burden for the management. It becomes a cost that doesn't bring "business value".
So you will start fighting for the quality because this is the very meaning of your programming life. Deliver quality software that satisfies the business requirements, bearing in mind technical consequences. You will defend your developers against the growing pressure from the business to deliver features at a larger pace. The corporate axe will come closer to your neck with every single fight to defend the very meaning of being an engineer.
In the meantime your fellow developers that don't agree with your permanent interference in the key tapping due to buzzwords like "resilience", "fail-fast", "latency" or "tests" will continue to dislike you. They will constantly show their lack of support to what you're doing. Their mediocrity and lack of willingness to stand to what they believe in will allow them to remain in the company for years to come.
Then one day you will have to pack your stuff in a box and you will be escorted out of the office because you will get fired. The reason will be simple: "not delivering business value".
Epilogue
Additional reading
Microservice Deployment
The goal
Our goal was to:Enforce standards
Have a unique way of deploying alll microservices - we need to enforce standards
Tackle the microservice dependencies complexity issue
Make the deployment process maintainable from the infrastructure and operations perspective
Make the pipeline fast and certain
Have the greatest possible certainty that our features are working fine.
We wanted to make the deployment pipeline as fast as possible.
It was crucial to add the possibility to automatically rollback if something goes wrong.
Enforce standards
It is crucial that if you're starting with microservices you start introducing standards. Standards of running applications, configuring them (externalized properties) but also you should enforce standards in how you deploy your applications. At some point in time we have seen that different applications do common tasks in different ways.Why should we bother - we have business value to deliver and not waste time on enforcing standards - your manager might say. Actually he is really wrong because you're wasting plenty of time (thus money) on supporting such nonstandard applications. Imagine how much it needs for the new developers to understand how exactly the rules are set in this particular process.
The same relates to deployment and deployment pipelines. That's why we decided to enforce one, single way of deploying microservices.
Tackle the microservice dependencies complexity issue
If you have two monolithic applications talking to each other and not too many developers working on the codebases you can queue deployment of both apps and always perform end to end tests.
Two monolithic applications deployed for end to end testing |
Many microservices deployed in different versions |
- Should I queue deployments of microservices on one testing environment or should I have an environment per microservice?
- If I queue deployments people will have to wait for hours to have their tests ran - that can be a problem
- To remove that issue I can have an environment per microservice
- Who will pay the bills (imagine 100 microservices - each having each own environment).
- Who will support each of those environments?
- Should we spawn a new environment each time we execute a new pipeline and then wrap it up or should we have them up and running for the whole day?
- In which versions should I deploy the dependent microservices - development or production versions?
- If I have development versions then I can test my application against a feature that is not yet on production. That can lead to exceptions on production
- If I test against production versions then I'll never be able to test against a feature under development anytime before deployment to production.
Make the pipeline fast and certain
- For monolithic applications we had plenty of unit, integration and end to end tests
- The different types of tests covered the same functionality up to three times
- The tests took a lot of time to run
Simplify the infrastructure complexity
Execute tests on a deployed microservice on stubbed dependencies |
We're testing microservices in isolation |
- A lot of unit tests executed during build time
- Some integration tests running on stubs of dependent services executed during build time
- Not many acceptance tests running on stubs of dependent services executed during build time (these can be treated as special case of integration tests)
- A handful of smoke tests executed on a deployed application to see if the app is really packaged properly
- No need to deploy dependent services
- The stubs used for the tests ran on a deployed microservice are the same as those used during integration tests
- Those stubs have been tested against the application that produces them (check Accurest for more information)
- We don't have many slow tests running on a deployed application - thus the pipeline gets executed much faster
- We don't have to queue deployments - we're testing in isolation thus pipelines don't interfere with each other
- We don't have to spawn virtual machines each time for deployment purposes
- No end to end tests before production - you don't have the full certainty that a feature is working
- Due to this certainty that the functionality is working decreases
- First time the applications will talk in a real way will be on production
Overcoming fear of uncertainty
The technical overview of the solution
Microservice deployment pipeline (without A/B testing) |
- unit and integration tests are ran
- validity of declared stubs specifications is tested against the application itself
Summary
How to speed up your Gradle build from 90 to 8 minutes
At one of the companies that I've been working we've faced a big problem related to pull request build times. We have one monolithic application that we are in progress of slicing into microservices but still until this process is finished we have to build that big app for each PR. We needed to change things to have really fast feedback from our build so that pull request builds don't get queued up endlessly in our CI. You can only imagine the frustration of developers who can't have their branches merged to master because of the waiting time.
Current build time: ~90 minutes.
That way Gradle created awesome stats for our build. If you are doing any sort of optimization then it's crucial to gather measurements and statistics. Check out this Gradle page about profiling your build for more info on that switch and features.
- excluded JS minification
- benefit: 13 countries * ~60 secs * at least 2 modules where minification occurred ~ 26 minutes
- optimized GWT compilation:
- have permutations done for only 1 browser (by default it's done for multiple browsers)
- disable optimization of the compilation (-optimize 0)
- add the -draftCompile switch to to compile quickly with minimal optimizations
- benefit: about 2 minutes less on GWT compilation * sth like 5 projects with GWT ~ 10 minutes
Benefit: By simple disabling this test we gained about 1 minute.
Overall gain: ~ 41 minutes.
That's why we decided to go parallel! Let's build the projects (over 200) in parallel and we'll gain a lot of time on that. When you execute the Gradle build with the --parallel flag Gradle calculates how many threads can be used to concurrently build the modules. For more info go to the Gradle's documentation on parallel project execution.
It's an incubating feature so wen we started to get BindExceptions on port allocation we initially thought that most likely it's Gradle's fault. Then we had a chat with Szczepan Faberwho worked for Gradleware and it turns out that the feature is actually really mature (thx Szczepan for the help BTW :) ).
We needed quick gains so instead of fixing the port binding stuff we decided only to compile everything in parallel and then run tests sequentially.
Benefit: By doing this lame looking hack we gained ~4 mintues (on my 8 core laptop).
Overall gain: ~ 45 minutes.
So we went through the code, randomized all the fixed ports, patched micro-infra-spring so it does the same upon Wiremock and Zookeeper instantiation and just ran the building of the project like this:
We were sure that this is the killer feature that we were lacking and we're going to win the lottery. Much to our surprise the result was really disappointing.
Benefit: Concurrent project build decreased the time by ~5 minutes.
Overall gain: ~ 50 minutes.
We've opened up htop, iotop and all the possible tools including vmstat to see what the hell was going on. It turned out that context switching is at an acceptable level whereas at some point of the build only part of the cores are used as if sth was executed sequentially!
The answer to that mystery was pretty simple. We had a wrong project structure.
We had a module that ended up as a test-jar in testCompile dependency of other projects. That means that the vast majority of modules where waiting for this project to be built. Built means compiled and tested. It turned out that this test-jar module had also plenty of slow integration tests in it so only after those tests were executed could other modules be actually built!
Now we could do further optimization - we've split the slow integration tests into two modules to make all the modules in the whole project be built in more or less equal time (around 3,5 minutes).
.
Benefit: Fixing the project structure decreased the time by ~10 minutes
Overall gain: ~ 60 minutes.
If someone form the management tells you that that machine costs a lot of money and the company can't afford it you can actually do a fast calculation. You can ask this manager what is more expensive - paying for the machine or paying the developer for 77 minutes * number of builds of waiting?
Benefit: Paying for a really good machine on AWS decreased the build time by ~22 minutes
Overall gain: ~ 82 minutes.
Possible solutions are:
- Go through all of the tests and check why some of them take so long to run
- Go through the integration tests and check if don't duplicate the logic - we will remove them
- We're using Liquibase for schema versioning and we haven't merged the changests for some time thus sth like 100 changesets are executed each time we boot up Spring context (it takes more or less 30 seconds)
- We could limit the Spring context scope for different parts of our applications so that Spring boots up faster
- Buy a more powerful machine ;)
Spock Subject Collaborators Extension 1.0.1 released!
I'm really happy to say that I've just released a new version 1.0.1 of the Spock Subject Collaborators Extension.
The changelog is as follows:
1.0.1
1.0.0
As you can see now you'll be able to use this extension together with Spock in version 1.0.0 (assuming that nothing will change until then).
How to get it?
For Maven:
<repositories>
<repository>
<snapshots>
<enabled>false</enabled>
</snapshots>
<id>central</id>
<name>bintray</name>
<url>https://jcenter.bintray.com</url>
</repository>
</repositories>
<dependency>
<groupId>com.blogspot.toomuchcoding</groupId>
<artifactId>spock-subjects-collaborators-extension</artifactId>
<version>1.0.1</version>
<scope>test</scope>
</dependency>
For Gradle:
repositories {
jcenter()
}
dependencies {
testCompile 'com.blogspot.toomuchcoding:spock-subjects-collaborators-extension:1.0.1'
}
How to use it?
package com.blogspot.toomuchcoding.spock.subjcollabs
import spock.lang.Specification
import com.blogspot.toomuchcoding.spock.subjcollabs.Collaborator
import com.blogspot.toomuchcoding.spock.subjcollabs.Subject
class ConstructorInjectionSpec extends Specification {
public static final String TEST_METHOD_1 = "Test method 1"
SomeOtherClass someOtherClassNotToBeInjected = Mock()
@Collaborator
SomeOtherClass someOtherClass = Mock()
@Subject
SomeClass systemUnderTest
def "should inject collaborator into subject"() {
given:
someOtherClass.someMethod() >> TEST_METHOD_1
when:
String firstResult = systemUnderTest.someOtherClass.someMethod()
then:
firstResult == TEST_METHOD_1
systemUnderTest.someOtherClass == someOtherClass
}
class SomeClass {
SomeOtherClass someOtherClass
SomeClass(SomeOtherClass someOtherClass) {
this.someOtherClass = someOtherClass
}
}
class SomeOtherClass {
String someMethod() {
"Some other class"
}
}
}
Disclaimer
Remember that if you're using this extension as a way to hack your way through an awful design of your application then you should do your best to fix your code in the first place! You've been warned ;)New personal website
Mockito Cookbook is out!!
I'm very pleased to announce that you can order my new book Mockito Cookbook here https://www.packtpub.com/mockito-cookbook/book . I hope that you will enjoy it :)